Compare commits
64 Commits
v0.9.1
...
9b88be4dd2
| Author | SHA1 | Date | |
|---|---|---|---|
| 9b88be4dd2 | |||
| 456a46e01b | |||
| 3740c779eb | |||
| 6943a5ebc9 | |||
| 9a09454621 | |||
| b8c6708439 | |||
| a62cb87b78 | |||
| 0b5eaf46f4 | |||
| 882059014c | |||
| 0a3e52d0e5 | |||
| 2b19a121fa | |||
| 29c65971eb | |||
| 3ecf4c5b7f | |||
| f402e27542 | |||
| f720b6cdc0 | |||
| 50215d2105 | |||
| 74411e3f54 | |||
| 4358a7cdce | |||
| dcea5261ab | |||
| 97141c7a9b | |||
| 2e2d36024e | |||
| 4d2cddf801 | |||
| 21a35a5205 | |||
| 435d1ade07 | |||
| 819ac24aed | |||
| 2c472449e8 | |||
| 5d7f604a2c | |||
| 8041f99782 | |||
| c417c892d9 | |||
| 481215c5db | |||
| 5d94d3f32d | |||
| f8f1b8d4a5 | |||
| 06d5973746 | |||
| 8045b27173 | |||
| 985468c1b9 | |||
| 6a0b30b7c7 | |||
| 86ce4eb68c | |||
| 33541a71fc | |||
| b7277e0c02 | |||
| dec05821bf | |||
| 1a9f462fbf | |||
| a990da957b | |||
| a4cd63ca4e | |||
| fba519cab0 | |||
| da8ff81692 | |||
| f91a185f9d | |||
| e966276c40 | |||
| 1e2db468ea | |||
| cefd4dfc7c | |||
| 42c646c328 | |||
| b2c429983c | |||
| 0f20d1e4cb | |||
| 979d754d18 | |||
| 002f9d0ba6 | |||
| ec115ba152 | |||
| 47176520bb | |||
| 9e27a431a1 | |||
| f0605781a2 | |||
| abb8f218ec | |||
| 4ac4e401ed | |||
| 77bd59cb87 | |||
| bb5abf798b | |||
| fd820fd964 | |||
| cca873cafb |
@@ -1,7 +1,8 @@
|
|||||||
.git
|
.git
|
||||||
.gitignore
|
.gitignore
|
||||||
**/*.md
|
**/*.md
|
||||||
|
!docs/*.md
|
||||||
|
!docs/**/*.md
|
||||||
dist/
|
dist/
|
||||||
tmp/
|
tmp/
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|
||||||
|
|||||||
4
AGENTS.md
Normal file
4
AGENTS.md
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
Please carefully review the documents in `docs/policy` before making any changes to this repository.
|
||||||
|
- `architecture.md` provides the canonical high-level architecture policy for this repository.
|
||||||
|
- `development.md` provides more granular development policy for this repository.
|
||||||
|
- `documentation.md` provides the canonical documentation policy for this repository.
|
||||||
339
API.md
339
API.md
@@ -1,339 +0,0 @@
|
|||||||
# weatherfeeder API (Wire Contract)
|
|
||||||
|
|
||||||
This document defines the stable, consumer-facing JSON contract emitted by weatherfeeder sinks.
|
|
||||||
|
|
||||||
weatherfeeder emits **events** encoded as JSON. Each event has:
|
|
||||||
- an **envelope** (metadata + schema identifier), and
|
|
||||||
- a **payload** whose shape is determined by `schema`.
|
|
||||||
|
|
||||||
Downstream consumers should:
|
|
||||||
1. parse the event envelope,
|
|
||||||
2. switch on `schema`, then
|
|
||||||
3. decode `payload` into the matching schema.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Event envelope
|
|
||||||
|
|
||||||
All events are JSON objects with these fields:
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `id` | string | yes | Stable event identifier. Treat as opaque. |
|
|
||||||
| `schema` | string | yes | Schema identifier (e.g. `weather.observation.v1`). |
|
|
||||||
| `source` | string | yes | Provider/source identifier (stable within configuration). |
|
|
||||||
| `effectiveAt` | string (timestamp) | yes | RFC3339Nano timestamp indicating when this event is effective. |
|
|
||||||
| `payload` | object | yes | Schema-specific payload (see below). |
|
|
||||||
|
|
||||||
### Timestamp format
|
|
||||||
|
|
||||||
All timestamps are encoded as JSON strings using Go’s `time.Time` JSON encoding (RFC3339Nano).
|
|
||||||
|
|
||||||
Examples:
|
|
||||||
- `"2026-01-17T14:27:00Z"`
|
|
||||||
- `"2026-01-17T08:27:00-06:00"`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Canonical schemas
|
|
||||||
|
|
||||||
weatherfeeder emits four canonical domain schemas:
|
|
||||||
|
|
||||||
- `weather.observation.v1`
|
|
||||||
- `weather.forecast.v1`
|
|
||||||
- `weather.forecast_discussion.v1`
|
|
||||||
- `weather.alert.v1`
|
|
||||||
|
|
||||||
Each payload is described below using the JSON field names as the contract.
|
|
||||||
|
|
||||||
### Raw upstream schemas
|
|
||||||
|
|
||||||
weatherfeeder sources also emit provider-specific raw schemas before normalization.
|
|
||||||
For this feature, the raw source schema is:
|
|
||||||
|
|
||||||
- `raw.nws.forecast_discussion.v1`
|
|
||||||
- payload type: string
|
|
||||||
- payload contents: exact fetched HTML response body
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Shared Conventions
|
|
||||||
|
|
||||||
- Timestamps are JSON strings in RFC3339Nano format.
|
|
||||||
- Optional fields are omitted when unknown (`omitempty` behavior).
|
|
||||||
- Numeric measurements are normalized to metric units:
|
|
||||||
- `*C` = Celsius
|
|
||||||
- `*Kmh` = kilometers/hour
|
|
||||||
- `*Pa` = Pascals
|
|
||||||
- `*Meters` = meters
|
|
||||||
- `*Mm` = millimeters
|
|
||||||
- `*Percent` = percent (0-100)
|
|
||||||
- `conditionCode` is a WMO weather interpretation code (`int`).
|
|
||||||
- Unknown/unmappable is `-1`.
|
|
||||||
- Downstream consumers should treat unknown codes as “unknown conditions” rather than failing decoding.
|
|
||||||
- For readability and stability, weatherfeeder rounds floating-point values in canonical payloads to
|
|
||||||
**4 digits after the decimal** during normalization.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Schema: `weather.observation.v1`
|
|
||||||
|
|
||||||
Payload type: `WeatherObservation`
|
|
||||||
|
|
||||||
A `WeatherObservation` represents a point-in-time observation for a station/location.
|
|
||||||
|
|
||||||
### Fields
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `stationId` | string | no | Provider station/location identifier |
|
|
||||||
| `stationName` | string | no | Human station name |
|
|
||||||
| `timestamp` | timestamp string | yes | Observation timestamp |
|
|
||||||
| `conditionCode` | int | yes | WMO code (`-1` unknown) |
|
|
||||||
| `isDay` | bool | no | Day/night hint |
|
|
||||||
| `textDescription` | string | no | Human-facing short description |
|
|
||||||
| `temperatureC` | number | no | Celsius |
|
|
||||||
| `dewpointC` | number | no | Celsius |
|
|
||||||
| `windDirectionDegrees` | number | no | Degrees |
|
|
||||||
| `windSpeedKmh` | number | no | km/h |
|
|
||||||
| `windGustKmh` | number | no | km/h |
|
|
||||||
| `barometricPressurePa` | number | no | Pascals |
|
|
||||||
| `visibilityMeters` | number | no | Meters |
|
|
||||||
| `relativeHumidityPercent` | number | no | Percent |
|
|
||||||
| `apparentTemperatureC` | number | no | Celsius |
|
|
||||||
| `presentWeather` | array | no | Provider-specific structured weather fragments |
|
|
||||||
|
|
||||||
### Nested: `presentWeather[]`
|
|
||||||
|
|
||||||
Each `presentWeather[]` element:
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `raw` | object | no | Provider-specific JSON object |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Schema: `weather.forecast.v1`
|
|
||||||
|
|
||||||
Payload type: `WeatherForecastRun`
|
|
||||||
|
|
||||||
A `WeatherForecastRun` is a single issued forecast snapshot for a location and a specific product
|
|
||||||
(hourly / narrative / daily). The run contains an ordered list of forecast periods.
|
|
||||||
|
|
||||||
### `product` values
|
|
||||||
|
|
||||||
`product` is one of:
|
|
||||||
|
|
||||||
- `"hourly"`
|
|
||||||
- `"narrative"`
|
|
||||||
- `"daily"`
|
|
||||||
|
|
||||||
### Fields
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `locationId` | string | no | Provider location identifier |
|
|
||||||
| `locationName` | string | no | Human name, if available |
|
|
||||||
| `issuedAt` | string (timestamp) | yes | When this run was generated/issued |
|
|
||||||
| `updatedAt` | string (timestamp) | no | Optional later update time |
|
|
||||||
| `product` | string | yes | One of `hourly`, `narrative`, `daily` |
|
|
||||||
| `latitude` | number | no | Degrees |
|
|
||||||
| `longitude` | number | no | Degrees |
|
|
||||||
| `elevationMeters` | number | no | meters |
|
|
||||||
| `periods` | array | yes | Chronological forecast periods |
|
|
||||||
|
|
||||||
### Nested: `periods[]` (`WeatherForecastPeriod`)
|
|
||||||
|
|
||||||
A `WeatherForecastPeriod` is valid for `[startTime, endTime)`.
|
|
||||||
|
|
||||||
| Field | Type | Required | Units / Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `startTime` | string (timestamp) | yes | Period start |
|
|
||||||
| `endTime` | string (timestamp) | yes | Period end |
|
|
||||||
| `name` | string | no | Human label (often empty for hourly) |
|
|
||||||
| `isDay` | bool | no | Day/night hint |
|
|
||||||
| `conditionCode` | int | yes | WMO code (`-1` for unknown) |
|
|
||||||
| `textDescription` | string | no | Human-facing short phrase |
|
|
||||||
| `temperatureC` | number | no | °C |
|
|
||||||
| `temperatureCMin` | number | no | °C (aggregated products) |
|
|
||||||
| `temperatureCMax` | number | no | °C (aggregated products) |
|
|
||||||
| `dewpointC` | number | no | °C |
|
|
||||||
| `relativeHumidityPercent` | number | no | percent |
|
|
||||||
| `windDirectionDegrees` | number | no | degrees |
|
|
||||||
| `windSpeedKmh` | number | no | km/h |
|
|
||||||
| `windGustKmh` | number | no | km/h |
|
|
||||||
| `barometricPressurePa` | number | no | Pa |
|
|
||||||
| `visibilityMeters` | number | no | meters |
|
|
||||||
| `apparentTemperatureC` | number | no | °C |
|
|
||||||
| `cloudCoverPercent` | number | no | percent |
|
|
||||||
| `probabilityOfPrecipitationPercent` | number | no | percent |
|
|
||||||
| `precipitationAmountMm` | number | no | mm (liquid equivalent) |
|
|
||||||
| `snowfallDepthMm` | number | no | mm |
|
|
||||||
| `uvIndex` | number | no | unitless index |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Schema: `weather.alert.v1`
|
|
||||||
|
|
||||||
Payload type: `WeatherAlertRun`
|
|
||||||
|
|
||||||
A `WeatherAlertRun` is a snapshot of *active* alerts for a location as-of a point in time.
|
|
||||||
A run may contain zero, one, or many alerts.
|
|
||||||
|
|
||||||
### Fields
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `locationId` | string | no | Provider location identifier |
|
|
||||||
| `locationName` | string | no | Human name, if available |
|
|
||||||
| `asOf` | string (timestamp) | yes | When the provider asserted this snapshot is current |
|
|
||||||
| `latitude` | number | no | Degrees |
|
|
||||||
| `longitude` | number | no | Degrees |
|
|
||||||
| `alerts` | array | yes | Active alerts (order provider-dependent) |
|
|
||||||
|
|
||||||
### Nested: `alerts[]` (`WeatherAlert`)
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `id` | string | yes | Provider-stable identifier (often a URL/URI) |
|
|
||||||
| `event` | string | no | Classification/event label |
|
|
||||||
| `headline` | string | no | Alert headline |
|
|
||||||
| `severity` | string | no | Example: Extreme/Severe/Moderate/Minor/Unknown |
|
|
||||||
| `urgency` | string | no | Example: Immediate/Expected/Future/Past/Unknown |
|
|
||||||
| `certainty` | string | no | Example: Observed/Likely/Possible/Unlikely/Unknown |
|
|
||||||
| `status` | string | no | Example: Actual/Exercise/Test/System/Unknown |
|
|
||||||
| `messageType` | string | no | Example: Alert/Update/Cancel |
|
|
||||||
| `category` | string | no | Example: Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other |
|
|
||||||
| `response` | string | no | Example: Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None |
|
|
||||||
| `response` | string | no | e.g. Shelter/Evacuate/Prepare/... |
|
|
||||||
| `description` | string | no | Narrative |
|
|
||||||
| `instruction` | string | no | What to do |
|
|
||||||
| `sent` | string (timestamp) | no | Provider-dependent |
|
|
||||||
| `effective` | string (timestamp) | no | Provider-dependent |
|
|
||||||
| `onset` | string (timestamp) | no | Provider-dependent |
|
|
||||||
| `expires` | string (timestamp) | no | Provider-dependent |
|
|
||||||
| `areaDescription` | string | no | Often a provider string |
|
|
||||||
| `senderName` | string | no | Provenance |
|
|
||||||
| `references` | array | no | Related alert references |
|
|
||||||
|
|
||||||
### Nested: `references[]` (`AlertReference`)
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `id` | string | no | Provider reference ID/URI |
|
|
||||||
| `identifier` | string | no | Provider identifier string, if distinct |
|
|
||||||
| `sender` | string | no | Sender |
|
|
||||||
| `sent` | string (timestamp) | no | Timestamp |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Schema: `weather.forecast_discussion.v1`
|
|
||||||
|
|
||||||
Payload type: `WeatherForecastDiscussion`
|
|
||||||
|
|
||||||
A `WeatherForecastDiscussion` is an issued narrative bulletin for an NWS office.
|
|
||||||
It is distinct from `weather.forecast.v1`, which is period-based.
|
|
||||||
|
|
||||||
### Fields
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `officeId` | string | no | NWS office identifier, e.g. `LSX` |
|
|
||||||
| `officeName` | string | no | Human office name |
|
|
||||||
| `product` | string | yes | Currently `afd` |
|
|
||||||
| `issuedAt` | string (timestamp) | yes | Bulletin issue time |
|
|
||||||
| `updatedAt` | string (timestamp) | no | Optional page/update timestamp |
|
|
||||||
| `keyMessages` | array | no | Ordered key-message bullet list |
|
|
||||||
| `shortTerm` | object | no | Short-term discussion section |
|
|
||||||
| `longTerm` | object | no | Long-term discussion section |
|
|
||||||
|
|
||||||
### Nested: `shortTerm` / `longTerm`
|
|
||||||
|
|
||||||
| Field | Type | Required | Notes |
|
|
||||||
|---|---:|:---:|---|
|
|
||||||
| `qualifier` | string | no | Header qualifier such as `(Through Late Sunday Night)` |
|
|
||||||
| `issuedAt` | string (timestamp) | no | Optional section-local issue time |
|
|
||||||
| `text` | string | no | Paragraph-preserved prose text |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Compatibility rules
|
|
||||||
|
|
||||||
- Consumers **must** ignore unknown fields.
|
|
||||||
- Producers (weatherfeeder) prefer **additive changes** within a schema version.
|
|
||||||
- Renames/removals/semantic breaks normally require a **schema version bump** (`weather.*.v2`); pre-1.0 projects may choose in-place changes.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Examples
|
|
||||||
|
|
||||||
### Observation event (`weather.observation.v1`)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "nws:KSTL:2026-01-17T14:00:00Z",
|
|
||||||
"schema": "weather.observation.v1",
|
|
||||||
"source": "nws_observation",
|
|
||||||
"effectiveAt": "2026-01-17T14:00:00Z",
|
|
||||||
"payload": {
|
|
||||||
"stationId": "KSTL",
|
|
||||||
"timestamp": "2026-01-17T14:00:00Z",
|
|
||||||
"conditionCode": 1,
|
|
||||||
"textDescription": "Mainly Sunny",
|
|
||||||
"temperatureC": 3.25,
|
|
||||||
"windSpeedKmh": 18.5
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Forecast event (`weather.forecast.v1`)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "openmeteo:38.63,-90.20:2026-01-17T13:00:00Z",
|
|
||||||
"schema": "weather.forecast.v1",
|
|
||||||
"source": "openmeteo_forecast",
|
|
||||||
"effectiveAt": "2026-01-17T13:00:00Z",
|
|
||||||
"payload": {
|
|
||||||
"locationName": "St. Louis, MO",
|
|
||||||
"issuedAt": "2026-01-17T13:00:00Z",
|
|
||||||
"product": "hourly",
|
|
||||||
"latitude": 38.63,
|
|
||||||
"longitude": -90.2,
|
|
||||||
"periods": [
|
|
||||||
{
|
|
||||||
"startTime": "2026-01-17T14:00:00Z",
|
|
||||||
"endTime": "2026-01-17T15:00:00Z",
|
|
||||||
"conditionCode": 2,
|
|
||||||
"textDescription": "Partly Cloudy",
|
|
||||||
"temperatureC": 3.5,
|
|
||||||
"probabilityOfPrecipitationPercent": 10
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Alert event (`weather.alert.v1`)
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"id": "nws:alerts:2026-01-17T14:10:00Z",
|
|
||||||
"schema": "weather.alert.v1",
|
|
||||||
"source": "nws_alerts",
|
|
||||||
"effectiveAt": "2026-01-17T14:10:00Z",
|
|
||||||
"payload": {
|
|
||||||
"asOf": "2026-01-17T14:05:00Z",
|
|
||||||
"alerts": [
|
|
||||||
{
|
|
||||||
"id": "https://api.weather.gov/alerts/abc123",
|
|
||||||
"event": "Winter Weather Advisory",
|
|
||||||
"headline": "Winter Weather Advisory issued January 17 at 8:05AM CST",
|
|
||||||
"severity": "Moderate",
|
|
||||||
"description": "Mixed precipitation expected...",
|
|
||||||
"expires": "2026-01-18T06:00:00Z"
|
|
||||||
}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
59
README.md
59
README.md
@@ -1,35 +1,42 @@
|
|||||||
# weatherfeeder
|
# weatherfeeder
|
||||||
|
|
||||||
weatherfeeder is a small daemon that polls weather observations, forecasts, and alerts from multiple upstream
|
`weatherfeeder` is a config-driven daemon that polls weather providers, normalizes
|
||||||
providers, normalizes them into a provider-independent format, and emits them to a sink.
|
provider-specific responses into canonical weather events, and routes those
|
||||||
|
events to configured sinks.
|
||||||
|
|
||||||
Today, the only implemented sink is `stdout`, which prints JSON-encoded events.
|
It currently supports NWS observations, alerts, hourly forecasts, narrative
|
||||||
|
forecasts, forecast discussions, and weather stories; SPC Day 1-3 convective
|
||||||
|
outlooks; Open-Meteo observations and hourly forecasts; and OpenWeather
|
||||||
|
observations. Implemented sinks are stdout, NATS, and Postgres.
|
||||||
|
|
||||||
## What weatherfeeder emits
|
## Quickstart
|
||||||
|
|
||||||
weatherfeeder emits **feed events** encoded as JSON. Each event includes a schema identifier and a payload.
|
Run the checked-in sample config:
|
||||||
Downstream consumers should key off the `schema` value and decode the `payload` accordingly.
|
|
||||||
|
|
||||||
Canonical domain schemas emitted after normalization:
|
```sh
|
||||||
|
cd cmd/weatherfeeder
|
||||||
|
go run .
|
||||||
|
```
|
||||||
|
|
||||||
- `weather.observation.v1` → `WeatherObservation`
|
The sample config at `cmd/weatherfeeder/config.yml` is load-tested and can be
|
||||||
- `weather.forecast.v1` → `WeatherForecastRun`
|
used as a starting point. The executable always reads `config.yml` from its
|
||||||
- `weather.forecast_discussion.v1` → `WeatherForecastDiscussion`
|
current working directory.
|
||||||
- `weather.alert.v1` → `WeatherAlertRun`
|
|
||||||
|
|
||||||
For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see:
|
## Documentation
|
||||||
|
|
||||||
- **API.md**
|
- [CLI reference](docs/cli.md)
|
||||||
|
- [Configuration reference](docs/config.md)
|
||||||
## Upstream providers (current MVP)
|
- [Operations guide](docs/operations.md)
|
||||||
|
- [Troubleshooting guide](docs/troubleshooting.md)
|
||||||
- NWS: observations, hourly forecasts, narrative forecasts, forecast discussions, alerts
|
- [Example configs](examples/)
|
||||||
- Open-Meteo: observations, hourly forecasts
|
- [Go consumer guide](docs/consumers/api.md)
|
||||||
- OpenWeather: observations
|
- [Event wire contract](docs/integrations/events.md)
|
||||||
|
- [Postgres table contract](docs/integrations/postgres.md)
|
||||||
## Versioning & compatibility
|
- [Feedkit integration notes](docs/integrations/feedkit.md)
|
||||||
|
- [NWS integration notes](docs/integrations/nws.md)
|
||||||
The JSON field names on canonical payload types are treated as part of the wire contract.
|
- [SPC integration notes](docs/integrations/spc.md)
|
||||||
Additive changes are preferred. Renames/removals require a schema version bump.
|
- [Open-Meteo integration notes](docs/integrations/openmeteo.md)
|
||||||
|
- [OpenWeather integration notes](docs/integrations/openweather.md)
|
||||||
See **API.md** for details.
|
- [Architecture policy](docs/policy/architecture.md)
|
||||||
|
- [Development policy](docs/policy/development.md)
|
||||||
|
- [Documentation policy](docs/policy/documentation.md)
|
||||||
|
|||||||
@@ -24,7 +24,7 @@ sources:
|
|||||||
# driver: openweather_observation
|
# driver: openweather_observation
|
||||||
# every: 10m
|
# every: 10m
|
||||||
# params:
|
# params:
|
||||||
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&appid=c954f2566cb7ccb56b43737b52e88fc6&units=metric"
|
# url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&units=metric"
|
||||||
# user_agent: "HomeOps (eric@maximumdirect.net)"
|
# user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||||
|
|
||||||
# - name: NWSObservationKSUS
|
# - name: NWSObservationKSUS
|
||||||
@@ -72,6 +72,15 @@ sources:
|
|||||||
url: "https://forecast.weather.gov/product.php?site=LSX&issuedby=LSX&product=AFD&format=TXT&version=1&glossary=0"
|
url: "https://forecast.weather.gov/product.php?site=LSX&issuedby=LSX&product=AFD&format=TXT&version=1&glossary=0"
|
||||||
user_agent: "HomeOps (eric@maximumdirect.net)"
|
user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||||
|
|
||||||
|
- name: NWSWeatherStoriesSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["weather_story"]
|
||||||
|
driver: nws_weatherstories
|
||||||
|
every: 30m
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/offices/LSX/weatherstories"
|
||||||
|
user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||||
|
|
||||||
- name: OpenMeteoHourlyForecastSTL
|
- name: OpenMeteoHourlyForecastSTL
|
||||||
mode: poll
|
mode: poll
|
||||||
kinds: ["forecast"]
|
kinds: ["forecast"]
|
||||||
@@ -90,6 +99,18 @@ sources:
|
|||||||
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
|
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
|
||||||
user_agent: "HomeOps (eric@maximumdirect.net)"
|
user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||||
|
|
||||||
|
- name: SPCConvectiveOutlookSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["outlook"]
|
||||||
|
driver: spc_convective_outlook
|
||||||
|
every: 30m
|
||||||
|
params:
|
||||||
|
latitude: 38.6239
|
||||||
|
longitude: -90.3571
|
||||||
|
location_id: "stl"
|
||||||
|
location_name: "St. Louis, MO"
|
||||||
|
user_agent: "HomeOps (eric@maximumdirect.net)"
|
||||||
|
|
||||||
sinks:
|
sinks:
|
||||||
- name: stdout
|
- name: stdout
|
||||||
driver: stdout
|
driver: stdout
|
||||||
@@ -106,7 +127,7 @@ sinks:
|
|||||||
# params:
|
# params:
|
||||||
# uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
|
# uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
|
||||||
# username: weatherdb
|
# username: weatherdb
|
||||||
# password: weatherdb
|
# password: <database_password>
|
||||||
# prune: 3d
|
# prune: 3d
|
||||||
# # Prunes rows older than now-3d on each write transaction.
|
# # Prunes rows older than now-3d on each write transaction.
|
||||||
|
|
||||||
@@ -117,13 +138,13 @@ sinks:
|
|||||||
|
|
||||||
routes:
|
routes:
|
||||||
- sink: stdout
|
- sink: stdout
|
||||||
kinds: ["observation", "forecast", "forecast_discussion", "alert"]
|
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
|
||||||
|
|
||||||
- sink: nats_weatherfeeder
|
- sink: nats_weatherfeeder
|
||||||
kinds: ["observation", "forecast", "forecast_discussion", "alert"]
|
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
|
||||||
|
|
||||||
# - sink: pg_weatherfeeder
|
# - sink: pg_weatherfeeder
|
||||||
# kinds: ["observation", "forecast", "forecast_discussion", "alert"]
|
# kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
|
||||||
|
|
||||||
# - sink: logfile
|
# - sink: logfile
|
||||||
# kinds: ["observation", "alert", "forecast", "forecast_discussion"]
|
# kinds: ["observation", "alert", "forecast", "forecast_discussion", "weather_story", "outlook"]
|
||||||
|
|||||||
@@ -2,7 +2,9 @@ package main
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
|
"sort"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -18,6 +20,7 @@ import (
|
|||||||
|
|
||||||
wfnormalizers "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers"
|
wfnormalizers "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers"
|
||||||
wfsources "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources"
|
wfsources "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
)
|
)
|
||||||
|
|
||||||
type testInput struct {
|
type testInput struct {
|
||||||
@@ -34,10 +37,10 @@ type testKindsSource struct {
|
|||||||
func (s testKindsSource) Kinds() []fkevent.Kind { return s.kinds }
|
func (s testKindsSource) Kinds() []fkevent.Kind { return s.kinds }
|
||||||
|
|
||||||
func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) {
|
func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) {
|
||||||
sc := config.SourceConfig{Kinds: []string{"observation"}}
|
sc := config.SourceConfig{Kinds: []string{standards.KindObservation}}
|
||||||
in := testKindsSource{
|
in := testKindsSource{
|
||||||
testInput: testInput{name: "test"},
|
testInput: testInput{name: "test"},
|
||||||
kinds: []fkevent.Kind{"observation", "forecast"},
|
kinds: []fkevent.Kind{fkevent.Kind(standards.KindObservation), fkevent.Kind(standards.KindForecast)},
|
||||||
}
|
}
|
||||||
|
|
||||||
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
|
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
|
||||||
@@ -46,10 +49,10 @@ func TestValidateSourceExpectedKindsSubsetAllowed(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateSourceExpectedKindsMismatchFails(t *testing.T) {
|
func TestValidateSourceExpectedKindsMismatchFails(t *testing.T) {
|
||||||
sc := config.SourceConfig{Kinds: []string{"alert"}}
|
sc := config.SourceConfig{Kinds: []string{standards.KindAlert}}
|
||||||
in := testKindsSource{
|
in := testKindsSource{
|
||||||
testInput: testInput{name: "test"},
|
testInput: testInput{name: "test"},
|
||||||
kinds: []fkevent.Kind{"observation", "forecast"},
|
kinds: []fkevent.Kind{fkevent.Kind(standards.KindObservation), fkevent.Kind(standards.KindForecast)},
|
||||||
}
|
}
|
||||||
|
|
||||||
err := fksources.ValidateExpectedKinds(sc, in)
|
err := fksources.ValidateExpectedKinds(sc, in)
|
||||||
@@ -62,7 +65,7 @@ func TestValidateSourceExpectedKindsMismatchFails(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestValidateSourceExpectedKindsNoMetadataSkipsCheck(t *testing.T) {
|
func TestValidateSourceExpectedKindsNoMetadataSkipsCheck(t *testing.T) {
|
||||||
sc := config.SourceConfig{Kinds: []string{"alert"}}
|
sc := config.SourceConfig{Kinds: []string{standards.KindAlert}}
|
||||||
in := testInput{name: "test"}
|
in := testInput{name: "test"}
|
||||||
|
|
||||||
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
|
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
|
||||||
@@ -82,6 +85,37 @@ func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
|
|||||||
t.Fatalf("config.Load(config.yml) unexpected error: %v", err)
|
t.Fatalf("config.Load(config.yml) unexpected error: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
assertConfigSourcesBuildSchedulerJobs(t, cfg)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMaintainedConfigExamplesLoad(t *testing.T) {
|
||||||
|
paths, err := filepath.Glob("../../examples/*.yml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("filepath.Glob examples: %v", err)
|
||||||
|
}
|
||||||
|
sort.Strings(paths)
|
||||||
|
if len(paths) == 0 {
|
||||||
|
t.Fatalf("expected maintained config examples")
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, path := range paths {
|
||||||
|
t.Run(filepath.Base(path), func(t *testing.T) {
|
||||||
|
cfg, err := config.Load(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("config.Load(%s) unexpected error: %v", path, err)
|
||||||
|
}
|
||||||
|
assertConfigSourcesBuildSchedulerJobs(t, cfg)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertConfigSourcesBuildSchedulerJobs(t *testing.T, cfg *config.Config) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if len(cfg.Sources) == 0 {
|
||||||
|
t.Fatalf("config has no sources")
|
||||||
|
}
|
||||||
|
|
||||||
reg := fksources.NewRegistry()
|
reg := fksources.NewRegistry()
|
||||||
wfsources.RegisterBuiltins(reg)
|
wfsources.RegisterBuiltins(reg)
|
||||||
|
|
||||||
@@ -91,6 +125,10 @@ func TestExampleConfigSourcesBuildSchedulerJobs(t *testing.T) {
|
|||||||
t.Fatalf("BuildInput(sources[%d]) error = %v", i, err)
|
t.Fatalf("BuildInput(sources[%d]) error = %v", i, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if err := fksources.ValidateExpectedKinds(sc, in); err != nil {
|
||||||
|
t.Fatalf("ValidateExpectedKinds(sources[%d]) error = %v", i, err)
|
||||||
|
}
|
||||||
|
|
||||||
job, err := fkscheduler.JobFromSourceConfig(in, sc)
|
job, err := fkscheduler.JobFromSourceConfig(in, sc)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("JobFromSourceConfig(sources[%d]) error = %v", i, err)
|
t.Fatalf("JobFromSourceConfig(sources[%d]) error = %v", i, err)
|
||||||
@@ -125,7 +163,7 @@ func TestNormalizeNoMatchPassThrough(t *testing.T) {
|
|||||||
pl := &fkpipeline.Pipeline{Processors: chain}
|
pl := &fkpipeline.Pipeline{Processors: chain}
|
||||||
in := fkevent.Event{
|
in := fkevent.Event{
|
||||||
ID: "evt-no-match",
|
ID: "evt-no-match",
|
||||||
Kind: fkevent.Kind("observation"),
|
Kind: fkevent.Kind(standards.KindObservation),
|
||||||
Source: "test",
|
Source: "test",
|
||||||
EmittedAt: time.Now().UTC(),
|
EmittedAt: time.Now().UTC(),
|
||||||
Schema: "raw.weatherfeeder.unknown.v1",
|
Schema: "raw.weatherfeeder.unknown.v1",
|
||||||
@@ -155,7 +193,7 @@ func TestDedupeDropsSecondEventWithSameID(t *testing.T) {
|
|||||||
pl := &fkpipeline.Pipeline{Processors: chain}
|
pl := &fkpipeline.Pipeline{Processors: chain}
|
||||||
in := fkevent.Event{
|
in := fkevent.Event{
|
||||||
ID: "evt-dedupe-1",
|
ID: "evt-dedupe-1",
|
||||||
Kind: fkevent.Kind("observation"),
|
Kind: fkevent.Kind(standards.KindObservation),
|
||||||
Source: "test",
|
Source: "test",
|
||||||
EmittedAt: time.Now().UTC(),
|
EmittedAt: time.Now().UTC(),
|
||||||
Schema: "raw.weatherfeeder.unknown.v1",
|
Schema: "raw.weatherfeeder.unknown.v1",
|
||||||
|
|||||||
72
docs/cli.md
Normal file
72
docs/cli.md
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
# CLI Reference
|
||||||
|
|
||||||
|
## Shortest Useful Command
|
||||||
|
|
||||||
|
Run `weatherfeeder` from a directory containing `config.yml`:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd cmd/weatherfeeder
|
||||||
|
go run .
|
||||||
|
```
|
||||||
|
|
||||||
|
When using a built binary:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
./weatherfeeder
|
||||||
|
```
|
||||||
|
|
||||||
|
## Command Overview
|
||||||
|
|
||||||
|
`weatherfeeder` starts a long-running polling daemon. On startup it:
|
||||||
|
|
||||||
|
1. reads `config.yml` from the current working directory;
|
||||||
|
2. builds configured sources, sinks, and routes;
|
||||||
|
3. starts polling sources on their configured intervals;
|
||||||
|
4. normalizes and deduplicates events;
|
||||||
|
5. dispatches matching events to configured sinks.
|
||||||
|
|
||||||
|
The command logs startup, runtime, and shutdown messages to stderr using the Go
|
||||||
|
standard logger.
|
||||||
|
|
||||||
|
## Flags
|
||||||
|
|
||||||
|
There are currently no CLI flags, subcommands, or environment-variable based
|
||||||
|
configuration controls.
|
||||||
|
|
||||||
|
The config path is fixed at `config.yml` relative to the process current working
|
||||||
|
directory. To run with a different config, change the working directory or place
|
||||||
|
the desired file at that path.
|
||||||
|
|
||||||
|
## Common Workflows
|
||||||
|
|
||||||
|
Run the checked-in sample config:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
cd cmd/weatherfeeder
|
||||||
|
go run .
|
||||||
|
```
|
||||||
|
|
||||||
|
Maintained copyable configs are available under [`examples/`](../examples/).
|
||||||
|
|
||||||
|
Build and run a local binary:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go build -o weatherfeeder ./cmd/weatherfeeder
|
||||||
|
cp cmd/weatherfeeder/config.yml .
|
||||||
|
./weatherfeeder
|
||||||
|
```
|
||||||
|
|
||||||
|
Run in the project container image with a mounted config:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
docker run --rm -v "$PWD/config.yml:/weatherfeeder/config.yml:ro" weatherfeeder
|
||||||
|
```
|
||||||
|
|
||||||
|
The Docker image sets `/weatherfeeder` as the working directory, so the mounted
|
||||||
|
file must appear at `/weatherfeeder/config.yml`.
|
||||||
|
|
||||||
|
## Shutdown
|
||||||
|
|
||||||
|
Stop the daemon with `Ctrl-C` or `SIGTERM`. The process uses context-aware
|
||||||
|
shutdown for scheduler, dispatcher, processors, sources, and sinks, then logs
|
||||||
|
`shutdown complete`.
|
||||||
273
docs/config.md
Normal file
273
docs/config.md
Normal file
@@ -0,0 +1,273 @@
|
|||||||
|
# Configuration Reference
|
||||||
|
|
||||||
|
## Config File
|
||||||
|
|
||||||
|
`weatherfeeder` reads exactly one YAML file named `config.yml` from the current
|
||||||
|
working directory. There is no config path flag and no search path.
|
||||||
|
|
||||||
|
YAML decoding is strict for config struct fields: misspelled fields such as
|
||||||
|
`sources[].drviver` fail startup. Driver-specific `params` maps are validated by
|
||||||
|
the source or sink constructor that consumes them.
|
||||||
|
|
||||||
|
The top-level file contains:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sources:
|
||||||
|
- name: NWSObservationKSTL
|
||||||
|
mode: poll
|
||||||
|
driver: nws_observation
|
||||||
|
every: 10m
|
||||||
|
kinds: ["observation"]
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||||
|
user_agent: "Example weatherfeeder operator (ops@example.com)"
|
||||||
|
|
||||||
|
sinks:
|
||||||
|
- name: stdout
|
||||||
|
driver: stdout
|
||||||
|
params: {}
|
||||||
|
|
||||||
|
routes:
|
||||||
|
- sink: stdout
|
||||||
|
kinds: ["observation"]
|
||||||
|
```
|
||||||
|
|
||||||
|
`sources` and `sinks` must each contain at least one entry. `routes` is optional.
|
||||||
|
When `routes` is omitted, every configured sink receives every event kind.
|
||||||
|
Maintained copyable configs are available under [`examples/`](../examples/).
|
||||||
|
|
||||||
|
## Production-Oriented Shape
|
||||||
|
|
||||||
|
A typical deployment uses multiple polling sources and sends the same canonical
|
||||||
|
event stream to a broker or database:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sources:
|
||||||
|
- name: NWSAlertsLocal
|
||||||
|
mode: poll
|
||||||
|
driver: nws_alerts
|
||||||
|
every: 1m
|
||||||
|
kinds: ["alert"]
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
|
||||||
|
user_agent: "Example weatherfeeder operator (ops@example.com)"
|
||||||
|
|
||||||
|
- name: SPCConvectiveOutlookLocal
|
||||||
|
mode: poll
|
||||||
|
driver: spc_convective_outlook
|
||||||
|
every: 30m
|
||||||
|
kinds: ["outlook"]
|
||||||
|
params:
|
||||||
|
latitude: 38.6239
|
||||||
|
longitude: -90.3571
|
||||||
|
location_id: "local"
|
||||||
|
location_name: "Configured point"
|
||||||
|
user_agent: "Example weatherfeeder operator (ops@example.com)"
|
||||||
|
|
||||||
|
sinks:
|
||||||
|
- name: nats_weather
|
||||||
|
driver: nats
|
||||||
|
params:
|
||||||
|
url: nats://nats:4222
|
||||||
|
subject: weatherfeeder
|
||||||
|
|
||||||
|
- name: pg_weather
|
||||||
|
driver: postgres
|
||||||
|
params:
|
||||||
|
uri: postgres://weatherdb:5432/weatherdb?sslmode=disable
|
||||||
|
username: weatherdb
|
||||||
|
password: <database_password>
|
||||||
|
prune: 3d
|
||||||
|
|
||||||
|
routes:
|
||||||
|
- sink: nats_weather
|
||||||
|
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
|
||||||
|
|
||||||
|
- sink: pg_weather
|
||||||
|
kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert", "outlook"]
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not commit real API keys, database passwords, or personal contact addresses in
|
||||||
|
copyable configs.
|
||||||
|
|
||||||
|
## Top-Level Fields
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `sources` | yes | List of configured input sources. |
|
||||||
|
| `sinks` | yes | List of configured output sinks. |
|
||||||
|
| `routes` | no | List of sink routing rules. If omitted, all sinks receive all kinds. |
|
||||||
|
|
||||||
|
## Source Fields
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `name` | yes | Unique source name. Used as the event source identifier. |
|
||||||
|
| `driver` | yes | Source driver name. |
|
||||||
|
| `mode` | no | `poll`, `stream`, or omitted for auto. Current weatherfeeder drivers are polling drivers. |
|
||||||
|
| `every` | yes | Poll interval for current weatherfeeder source drivers. |
|
||||||
|
| `kinds` | no | Expected event kinds. If present, startup verifies they match the source driver. |
|
||||||
|
| `params` | driver-specific | Driver parameters. See the source-specific sections below. |
|
||||||
|
|
||||||
|
Current event kinds are `observation`, `forecast`, `forecast_discussion`,
|
||||||
|
`weather_story`, `alert`, and `outlook`.
|
||||||
|
|
||||||
|
## Source Drivers
|
||||||
|
|
||||||
|
| Driver | Kind | Upstream product |
|
||||||
|
|---|---|---|
|
||||||
|
| `nws_observation` | `observation` | NWS station latest observation. |
|
||||||
|
| `nws_alerts` | `alert` | NWS alerts collection. |
|
||||||
|
| `nws_forecast_hourly` | `forecast` | NWS hourly gridpoint forecast. |
|
||||||
|
| `nws_forecast_narrative` | `forecast` | NWS narrative gridpoint forecast. |
|
||||||
|
| `nws_forecast_discussion` | `forecast_discussion` | NWS forecast discussion HTML product. |
|
||||||
|
| `nws_weatherstories` | `weather_story` | NWS office weather stories. |
|
||||||
|
| `openmeteo_observation` | `observation` | Open-Meteo current conditions. |
|
||||||
|
| `openmeteo_forecast` | `forecast` | Open-Meteo hourly forecast. |
|
||||||
|
| `openweather_observation` | `observation` | OpenWeather current weather. |
|
||||||
|
| `spc_convective_outlook` | `outlook` | SPC Day 1-3 convective outlooks. |
|
||||||
|
|
||||||
|
## HTTP Source Params
|
||||||
|
|
||||||
|
Most source drivers use the shared HTTP polling helper.
|
||||||
|
|
||||||
|
| Param | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `url` | yes | Full upstream request URL. `URL` is also accepted by the helper. |
|
||||||
|
| `user_agent` | yes | User-Agent sent to the upstream provider. `userAgent` is also accepted by the helper. |
|
||||||
|
| `conditional` | no | Boolean. Defaults to `true`; enables ETag and Last-Modified conditional requests. |
|
||||||
|
| `http_timeout` | no | Positive duration for the HTTP client timeout. |
|
||||||
|
| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. |
|
||||||
|
|
||||||
|
When `conditional` is enabled and the upstream returns `304 Not Modified`, the
|
||||||
|
source emits no events for that poll.
|
||||||
|
|
||||||
|
OpenWeather observation URLs must include `units=metric`. Startup fails if the
|
||||||
|
URL omits it or sets another unit system.
|
||||||
|
|
||||||
|
## SPC Convective Outlook Params
|
||||||
|
|
||||||
|
`spc_convective_outlook` fetches the nine required Day 1-3 GeoJSON outlook
|
||||||
|
products and the three required Day 1-3 print pages as one atomic bundle.
|
||||||
|
|
||||||
|
| Param | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `latitude` | yes | Location latitude in decimal degrees. |
|
||||||
|
| `longitude` | yes | Location longitude in decimal degrees. |
|
||||||
|
| `user_agent` | yes | User-Agent sent to SPC. `userAgent` is also accepted. |
|
||||||
|
| `location_id` | no | Operator-defined location identifier copied into canonical outlook runs. |
|
||||||
|
| `location_name` | no | Human location label copied into canonical outlook runs. |
|
||||||
|
| `geojson_urls` | no | Map of product key to override URL. Used for tests and upstream URL changes. |
|
||||||
|
| `discussion_urls` | no | Map of day key to override print-page URL. Used for tests and upstream URL changes. |
|
||||||
|
| `rss_url` | no | Optional RSS URL. RSS is not fetched unless this is configured. |
|
||||||
|
| `http_timeout` | no | Positive duration for the HTTP client timeout. |
|
||||||
|
| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. |
|
||||||
|
|
||||||
|
GeoJSON product keys are `day1_categorical`, `day1_tornado`, `day1_hail`,
|
||||||
|
`day1_wind`, `day2_categorical`, `day2_tornado`, `day2_hail`, `day2_wind`,
|
||||||
|
and `day3_categorical`. SPC does not provide Day 3 tornado, hail, or wind
|
||||||
|
GeoJSON products. Discussion keys are `day1`, `day2`, and `day3`.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sources:
|
||||||
|
- name: SPCConvectiveOutlookSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["outlook"]
|
||||||
|
driver: spc_convective_outlook
|
||||||
|
every: 30m
|
||||||
|
params:
|
||||||
|
latitude: 38.6239
|
||||||
|
longitude: -90.3571
|
||||||
|
location_id: "stl"
|
||||||
|
location_name: "St. Louis, MO"
|
||||||
|
user_agent: "Example weatherfeeder operator (ops@example.com)"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Sink Fields
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `name` | yes | Unique sink name. Routes refer to this value. |
|
||||||
|
| `driver` | yes | Sink driver name. |
|
||||||
|
| `params` | driver-specific | Sink parameters. |
|
||||||
|
|
||||||
|
## Sink Drivers
|
||||||
|
|
||||||
|
### `stdout`
|
||||||
|
|
||||||
|
Prints each event as JSON to stdout.
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
sinks:
|
||||||
|
- name: stdout
|
||||||
|
driver: stdout
|
||||||
|
params: {}
|
||||||
|
```
|
||||||
|
|
||||||
|
### `nats`
|
||||||
|
|
||||||
|
Publishes each event as JSON to a NATS subject.
|
||||||
|
|
||||||
|
| Param | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `url` | yes | NATS server URL, such as `nats://localhost:4222`. |
|
||||||
|
| `subject` | yes | Subject to publish events to. |
|
||||||
|
|
||||||
|
### `postgres`
|
||||||
|
|
||||||
|
Writes supported canonical weather events to Postgres using weatherfeeder's
|
||||||
|
registered schema mapping. The table contract is documented in
|
||||||
|
[Postgres integration](integrations/postgres.md).
|
||||||
|
|
||||||
|
| Param | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `uri` | yes | PostgreSQL connection URI. |
|
||||||
|
| `username` | yes | Database username. |
|
||||||
|
| `password` | yes | Database password. |
|
||||||
|
| `prune` | no | Retention window. If set, rows older than the window are pruned on each write transaction. |
|
||||||
|
|
||||||
|
`prune` accepts Go duration strings such as `72h`, plus day and week suffixes
|
||||||
|
such as `3d` and `2w`.
|
||||||
|
|
||||||
|
## Routes
|
||||||
|
|
||||||
|
Routes connect event kinds to sinks:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
routes:
|
||||||
|
- sink: stdout
|
||||||
|
kinds: ["observation", "alert"]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
|---|:---:|---|
|
||||||
|
| `sink` | yes | Name of a configured sink. |
|
||||||
|
| `kinds` | no | Event kinds to send to that sink. Omit or use an empty list to match all kinds. |
|
||||||
|
|
||||||
|
Route `kinds` values are trimmed and lowercased by the dispatcher. Blank entries
|
||||||
|
are rejected.
|
||||||
|
|
||||||
|
## Duration Formats
|
||||||
|
|
||||||
|
Top-level source `every` accepts:
|
||||||
|
|
||||||
|
- Go duration strings such as `30s`, `10m`, or `1h`;
|
||||||
|
- integer values, interpreted as minutes;
|
||||||
|
- numeric strings such as `"15"`, also interpreted as minutes.
|
||||||
|
|
||||||
|
HTTP param durations such as `http_timeout` accept Go duration strings. Numeric
|
||||||
|
values and numeric strings are interpreted as seconds.
|
||||||
|
|
||||||
|
Postgres `prune` must be a string duration.
|
||||||
|
|
||||||
|
## Secrets
|
||||||
|
|
||||||
|
The config file is read directly from disk and has no built-in secret expansion.
|
||||||
|
Keep real credentials out of repository-tracked configs. Use deployment tooling
|
||||||
|
to render `config.yml` with the needed secret values before starting the daemon.
|
||||||
|
|
||||||
|
## Maintained Examples
|
||||||
|
|
||||||
|
- [Minimal stdout config](../examples/config.minimal.yml)
|
||||||
|
- [NATS publishing config](../examples/config.nats.yml)
|
||||||
|
- [Postgres persistence config](../examples/config.postgres.yml)
|
||||||
92
docs/consumers/api.md
Normal file
92
docs/consumers/api.md
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
# Consumer API Guide
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This guide is for developers and LLM coding agents integrating `weatherfeeder`
|
||||||
|
from another Go codebase.
|
||||||
|
|
||||||
|
`weatherfeeder` is primarily a daemon, not an SDK. Its public integration
|
||||||
|
surface is intentionally narrow:
|
||||||
|
|
||||||
|
- `model`: canonical weather payload structs.
|
||||||
|
- `standards`: schema strings, event kind strings, and shared WMO constants.
|
||||||
|
- JSON event output from stdout and NATS sinks.
|
||||||
|
- Postgres tables written by the optional Postgres sink.
|
||||||
|
|
||||||
|
Packages under `internal/` are implementation details and are not public
|
||||||
|
integration surfaces.
|
||||||
|
|
||||||
|
## Recommended Workflow
|
||||||
|
|
||||||
|
Consumers should switch on the event `schema` value and decode `payload` into
|
||||||
|
the matching `model` type.
|
||||||
|
|
||||||
|
Minimal example:
|
||||||
|
|
||||||
|
```go
|
||||||
|
package consumer
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Event struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Kind string `json:"kind"`
|
||||||
|
Schema string `json:"schema"`
|
||||||
|
Payload json.RawMessage `json:"payload"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func Decode(payload []byte) (any, error) {
|
||||||
|
var evt Event
|
||||||
|
if err := json.Unmarshal(payload, &evt); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
switch evt.Schema {
|
||||||
|
case standards.SchemaWeatherObservationV1:
|
||||||
|
var out model.WeatherObservation
|
||||||
|
return &out, json.Unmarshal(evt.Payload, &out)
|
||||||
|
case standards.SchemaWeatherForecastV1:
|
||||||
|
var out model.WeatherForecastRun
|
||||||
|
return &out, json.Unmarshal(evt.Payload, &out)
|
||||||
|
case standards.SchemaWeatherForecastDiscussionV1:
|
||||||
|
var out model.WeatherForecastDiscussion
|
||||||
|
return &out, json.Unmarshal(evt.Payload, &out)
|
||||||
|
case standards.SchemaWeatherStoryV1:
|
||||||
|
var out model.WeatherStoryRun
|
||||||
|
return &out, json.Unmarshal(evt.Payload, &out)
|
||||||
|
case standards.SchemaWeatherAlertV1:
|
||||||
|
var out model.WeatherAlertRun
|
||||||
|
return &out, json.Unmarshal(evt.Payload, &out)
|
||||||
|
case standards.SchemaWeatherOutlookV2:
|
||||||
|
var out model.WeatherOutlookRun
|
||||||
|
return &out, json.Unmarshal(evt.Payload, &out)
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("unsupported weatherfeeder schema %q", evt.Schema)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Consumer Responsibilities
|
||||||
|
|
||||||
|
- Treat event IDs as opaque.
|
||||||
|
- Treat absent `omitempty` fields as unknown, not zero.
|
||||||
|
- Prefer schema constants from `standards` over string literals in Go code.
|
||||||
|
- Expect canonical numeric measurements to use metric units.
|
||||||
|
- Expect canonical timestamps from normalizers to be UTC unless a field-specific
|
||||||
|
contract says otherwise.
|
||||||
|
- Handle additive fields within the same schema version.
|
||||||
|
- Do not import `internal/...` packages.
|
||||||
|
|
||||||
|
## Canonical References
|
||||||
|
|
||||||
|
- Public payload package: [`pkg-model.md`](pkg-model.md).
|
||||||
|
- Public constants package: [`pkg-standards.md`](pkg-standards.md).
|
||||||
|
- JSON event wire contract: [`../integrations/events.md`](../integrations/events.md).
|
||||||
|
- Postgres table contract: [`../integrations/postgres.md`](../integrations/postgres.md).
|
||||||
|
- Runtime and adapter architecture: [`../policy/architecture.md`](../policy/architecture.md).
|
||||||
64
docs/consumers/pkg-model.md
Normal file
64
docs/consumers/pkg-model.md
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
# Package `model`
|
||||||
|
|
||||||
|
## Import Path
|
||||||
|
|
||||||
|
```go
|
||||||
|
import "gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Package `model` defines `weatherfeeder`'s canonical weather payload structs.
|
||||||
|
These structs are emitted as the `payload` of canonical `weather.*` events and
|
||||||
|
are also the domain types consumed by downstream applications such as
|
||||||
|
`weatherapi`.
|
||||||
|
|
||||||
|
The JSON field tags on these structs are part of the wire contract. For the full
|
||||||
|
field-by-field JSON contract, use the [event wire contract](../integrations/events.md).
|
||||||
|
|
||||||
|
## Payload Types
|
||||||
|
|
||||||
|
Current canonical schema families map to these public types:
|
||||||
|
|
||||||
|
| Schema | Primary type |
|
||||||
|
|---|---|
|
||||||
|
| `weather.observation.v1` | `WeatherObservation` |
|
||||||
|
| `weather.forecast.v1` | `WeatherForecastRun` |
|
||||||
|
| `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` |
|
||||||
|
| `weather.weather_story.v1` | `WeatherStoryRun` |
|
||||||
|
| `weather.alert.v1` | `WeatherAlertRun` |
|
||||||
|
| `weather.outlook.v2` | `WeatherOutlookRun` |
|
||||||
|
|
||||||
|
Related child types include:
|
||||||
|
|
||||||
|
- `WeatherObservationPresentWeather`
|
||||||
|
- `WeatherForecastPeriod`
|
||||||
|
- `WeatherForecastDiscussionSection`
|
||||||
|
- `WeatherStory`
|
||||||
|
- `WeatherAlert`
|
||||||
|
- `WeatherAlertReference`
|
||||||
|
- `WeatherOutlook`
|
||||||
|
- `WeatherOutlookDiscussion`
|
||||||
|
- `WMOCode`
|
||||||
|
|
||||||
|
## Wire And Compatibility Rules
|
||||||
|
|
||||||
|
- JSON tags define canonical payload field names.
|
||||||
|
- Pointer fields and fields tagged `omitempty` are optional on the wire.
|
||||||
|
- Missing optional fields mean unknown or not applicable.
|
||||||
|
- Canonical measurements use metric units.
|
||||||
|
- Canonical timestamps are `time.Time` values encoded by Go's JSON encoder.
|
||||||
|
- Normalized canonical timestamps are UTC unless a field-specific contract says
|
||||||
|
otherwise.
|
||||||
|
- Additive fields are compatible within a schema version.
|
||||||
|
- Removing, renaming, or changing the meaning of a field requires a new schema
|
||||||
|
identifier.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
`model` should not depend on source adapters, sinks, SQL column names, provider
|
||||||
|
HTTP shapes, or runtime configuration.
|
||||||
|
|
||||||
|
Consumers should not rely on packages under `internal/...`. Use `model` with
|
||||||
|
schema constants from [`standards`](pkg-standards.md) and the JSON contract in
|
||||||
|
[`docs/integrations/events.md`](../integrations/events.md).
|
||||||
96
docs/consumers/pkg-standards.md
Normal file
96
docs/consumers/pkg-standards.md
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
# Package `standards`
|
||||||
|
|
||||||
|
## Import Path
|
||||||
|
|
||||||
|
```go
|
||||||
|
import "gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
```
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Package `standards` defines stable identifiers and shared weather constants used
|
||||||
|
by `weatherfeeder` producers and Go consumers.
|
||||||
|
|
||||||
|
Use this package when switching on event schemas, comparing event kinds, or
|
||||||
|
working with canonical WMO condition codes.
|
||||||
|
|
||||||
|
## Event Kind Constants
|
||||||
|
|
||||||
|
Current event kind constants are:
|
||||||
|
|
||||||
|
| Constant | Value |
|
||||||
|
|---|---|
|
||||||
|
| `KindObservation` | `observation` |
|
||||||
|
| `KindForecast` | `forecast` |
|
||||||
|
| `KindForecastDiscussion` | `forecast_discussion` |
|
||||||
|
| `KindWeatherStory` | `weather_story` |
|
||||||
|
| `KindAlert` | `alert` |
|
||||||
|
| `KindOutlook` | `outlook` |
|
||||||
|
|
||||||
|
These are plain string constants. Convert them at adapter boundaries when using
|
||||||
|
feedkit's `event.Kind` type.
|
||||||
|
|
||||||
|
## Canonical Schema Constants
|
||||||
|
|
||||||
|
Canonical schemas emitted after normalization:
|
||||||
|
|
||||||
|
| Constant | Value |
|
||||||
|
|---|---|
|
||||||
|
| `SchemaWeatherObservationV1` | `weather.observation.v1` |
|
||||||
|
| `SchemaWeatherForecastV1` | `weather.forecast.v1` |
|
||||||
|
| `SchemaWeatherForecastDiscussionV1` | `weather.forecast_discussion.v1` |
|
||||||
|
| `SchemaWeatherStoryV1` | `weather.weather_story.v1` |
|
||||||
|
| `SchemaWeatherAlertV1` | `weather.alert.v1` |
|
||||||
|
| `SchemaWeatherOutlookV2` | `weather.outlook.v2` |
|
||||||
|
|
||||||
|
Historical canonical schema constant:
|
||||||
|
|
||||||
|
| Constant | Value |
|
||||||
|
|---|---|
|
||||||
|
| `SchemaWeatherOutlookV1` | `weather.outlook.v1` |
|
||||||
|
|
||||||
|
## Raw Schema Constants
|
||||||
|
|
||||||
|
Raw source schemas emitted by current registered sources:
|
||||||
|
|
||||||
|
| Constant | Value |
|
||||||
|
|---|---|
|
||||||
|
| `SchemaRawNWSObservationV1` | `raw.nws.observation.v1` |
|
||||||
|
| `SchemaRawOpenMeteoCurrentV1` | `raw.openmeteo.current.v1` |
|
||||||
|
| `SchemaRawOpenWeatherCurrentV1` | `raw.openweather.current.v1` |
|
||||||
|
| `SchemaRawNWSHourlyForecastV1` | `raw.nws.hourly.forecast.v1` |
|
||||||
|
| `SchemaRawNWSNarrativeForecastV1` | `raw.nws.narrative.forecast.v1` |
|
||||||
|
| `SchemaRawNWSForecastDiscussionV1` | `raw.nws.forecast_discussion.v1` |
|
||||||
|
| `SchemaRawNWSWeatherStoriesV1` | `raw.nws.weatherstories.v1` |
|
||||||
|
| `SchemaRawOpenMeteoHourlyForecastV1` | `raw.openmeteo.hourly.forecast.v1` |
|
||||||
|
| `SchemaRawNWSAlertsV1` | `raw.nws.alerts.v1` |
|
||||||
|
| `SchemaRawSPCConvectiveOutlookV1` | `raw.spc.convective_outlook.v1` |
|
||||||
|
|
||||||
|
Additional raw schema constant:
|
||||||
|
|
||||||
|
| Constant | Value |
|
||||||
|
|---|---|
|
||||||
|
| `SchemaRawOpenWeatherHourlyForecastV1` | `raw.openweather.hourly.forecast.v1` |
|
||||||
|
|
||||||
|
`SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current registered
|
||||||
|
source emits it. Consumers should not expect that raw schema unless a later
|
||||||
|
registered source documents it as part of the current event contract.
|
||||||
|
|
||||||
|
## WMO Constants And Text
|
||||||
|
|
||||||
|
`standards` also defines the canonical `WMOCode` vocabulary and text helpers
|
||||||
|
used by normalized observations and forecasts.
|
||||||
|
|
||||||
|
Consumer guidance:
|
||||||
|
|
||||||
|
- Treat `WMOUnknown` as unknown condition data.
|
||||||
|
- Observation `conditionCode` is required in the current event contract.
|
||||||
|
- Forecast period `conditionCode` is optional because some forecast products do
|
||||||
|
not provide a meaningful WMO condition.
|
||||||
|
- Prefer WMO constants and helper functions from this package instead of
|
||||||
|
duplicating code tables in consumers.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
`standards` is provider-agnostic. Provider-specific parsing belongs in
|
||||||
|
`weatherfeeder` internals, not in this package and not in consumers.
|
||||||
327
docs/integrations/events.md
Normal file
327
docs/integrations/events.md
Normal file
@@ -0,0 +1,327 @@
|
|||||||
|
# Event Wire Contract
|
||||||
|
|
||||||
|
This document is the canonical JSON contract for events emitted by
|
||||||
|
`weatherfeeder` JSON sinks, including stdout and NATS. Postgres stores the same
|
||||||
|
event envelope fields in parent table columns; see
|
||||||
|
[Postgres integration](postgres.md).
|
||||||
|
|
||||||
|
Downstream consumers should read the envelope, switch on `schema`, and decode
|
||||||
|
`payload` according to that schema.
|
||||||
|
|
||||||
|
## Envelope
|
||||||
|
|
||||||
|
Every emitted event is a JSON object with these fields:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `id` | string | yes | Stable event identifier. Treat as opaque. |
|
||||||
|
| `kind` | string | yes | Routing kind, such as `observation` or `alert`. |
|
||||||
|
| `source` | string | yes | Configured source name. |
|
||||||
|
| `emitted_at` | timestamp | yes | When the daemon emitted the event. |
|
||||||
|
| `effective_at` | timestamp | no | Timestamp the payload is about, when known. |
|
||||||
|
| `schema` | string | no | Schema identifier. Weatherfeeder sources and normalizers set this. |
|
||||||
|
| `payload` | object, array, string, or scalar | yes | Schema-specific payload. |
|
||||||
|
|
||||||
|
Timestamps are JSON strings using Go `time.Time` JSON encoding, which is
|
||||||
|
RFC3339Nano-compatible. Weatherfeeder normalizers use UTC timestamps for
|
||||||
|
canonical payloads.
|
||||||
|
|
||||||
|
## Kinds And Schemas
|
||||||
|
|
||||||
|
Canonical schemas emitted after normalization:
|
||||||
|
|
||||||
|
| Kind | Schema | Payload |
|
||||||
|
|---|---|---|
|
||||||
|
| `observation` | `weather.observation.v1` | `WeatherObservation` |
|
||||||
|
| `forecast` | `weather.forecast.v1` | `WeatherForecastRun` |
|
||||||
|
| `forecast_discussion` | `weather.forecast_discussion.v1` | `WeatherForecastDiscussion` |
|
||||||
|
| `weather_story` | `weather.weather_story.v1` | `WeatherStoryRun` |
|
||||||
|
| `alert` | `weather.alert.v1` | `WeatherAlertRun` |
|
||||||
|
| `outlook` | `weather.outlook.v2` | `WeatherOutlookRun` |
|
||||||
|
|
||||||
|
Raw upstream schemas emitted by current sources:
|
||||||
|
|
||||||
|
| Kind | Schema | Payload |
|
||||||
|
|---|---|---|
|
||||||
|
| `observation` | `raw.nws.observation.v1` | NWS observation JSON |
|
||||||
|
| `observation` | `raw.openmeteo.current.v1` | Open-Meteo current JSON |
|
||||||
|
| `observation` | `raw.openweather.current.v1` | OpenWeather current JSON |
|
||||||
|
| `forecast` | `raw.nws.hourly.forecast.v1` | NWS hourly forecast JSON |
|
||||||
|
| `forecast` | `raw.nws.narrative.forecast.v1` | NWS narrative forecast JSON |
|
||||||
|
| `forecast_discussion` | `raw.nws.forecast_discussion.v1` | NWS forecast discussion HTML string |
|
||||||
|
| `weather_story` | `raw.nws.weatherstories.v1` | NWS weather stories JSON |
|
||||||
|
| `forecast` | `raw.openmeteo.hourly.forecast.v1` | Open-Meteo hourly forecast JSON |
|
||||||
|
| `alert` | `raw.nws.alerts.v1` | NWS alerts JSON |
|
||||||
|
| `outlook` | `raw.spc.convective_outlook.v1` | SPC convective outlook raw bundle |
|
||||||
|
|
||||||
|
`standards.SchemaRawOpenWeatherHourlyForecastV1` exists in code, but no current
|
||||||
|
registered source emits it.
|
||||||
|
|
||||||
|
## Shared Conventions
|
||||||
|
|
||||||
|
- Canonical numeric measurements use metric units.
|
||||||
|
- Floating-point values in canonical payloads are rounded to 4 digits after the
|
||||||
|
decimal point during normalization.
|
||||||
|
- Optional fields use JSON `omitempty`; absent fields should be treated as
|
||||||
|
unknown.
|
||||||
|
- `conditionCode` is a WMO weather interpretation code. Unknown observation
|
||||||
|
conditions use `-1`. Forecast period `conditionCode` is optional.
|
||||||
|
- Additive fields are compatible within a schema version. Removing, renaming, or
|
||||||
|
changing the meaning of a field requires a new schema identifier.
|
||||||
|
|
||||||
|
## `weather.observation.v1`
|
||||||
|
|
||||||
|
Payload type: `WeatherObservation`.
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `stationId` | string | no | Provider station/location identifier. |
|
||||||
|
| `stationName` | string | no | Human station name. |
|
||||||
|
| `timestamp` | timestamp | yes | Observation timestamp. |
|
||||||
|
| `conditionCode` | integer | yes | WMO code; `-1` means unknown. |
|
||||||
|
| `isDay` | boolean | no | Day/night hint. |
|
||||||
|
| `textDescription` | string | no | Short human description. |
|
||||||
|
| `temperatureC` | number | no | Celsius. |
|
||||||
|
| `dewpointC` | number | no | Celsius. |
|
||||||
|
| `windDirectionDegrees` | number | no | Degrees. |
|
||||||
|
| `windSpeedKmh` | number | no | Kilometers per hour. |
|
||||||
|
| `windGustKmh` | number | no | Kilometers per hour. |
|
||||||
|
| `barometricPressurePa` | number | no | Pascals. |
|
||||||
|
| `visibilityMeters` | number | no | Meters. |
|
||||||
|
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
|
||||||
|
| `apparentTemperatureC` | number | no | Celsius. |
|
||||||
|
| `presentWeather` | array | no | Provider-specific present weather fragments. |
|
||||||
|
|
||||||
|
`presentWeather[]` entries contain optional `raw` objects.
|
||||||
|
|
||||||
|
## `weather.forecast.v1`
|
||||||
|
|
||||||
|
Payload type: `WeatherForecastRun`.
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `locationId` | string | no | Provider location identifier. |
|
||||||
|
| `locationName` | string | no | Human location name. |
|
||||||
|
| `issuedAt` | timestamp | yes | When the forecast run was generated or issued. |
|
||||||
|
| `updatedAt` | timestamp | no | Subsequent provider update time. |
|
||||||
|
| `product` | string | yes | Current emitted values are `hourly` and `narrative`. |
|
||||||
|
| `latitude` | number | no | Degrees. |
|
||||||
|
| `longitude` | number | no | Degrees. |
|
||||||
|
| `elevationMeters` | number | no | Meters. |
|
||||||
|
| `periods` | array | yes | Ordered forecast periods. |
|
||||||
|
|
||||||
|
`periods[]` entries:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `startTime` | timestamp | yes | Period start. |
|
||||||
|
| `endTime` | timestamp | yes | Period end. |
|
||||||
|
| `name` | string | no | Human label. |
|
||||||
|
| `isDay` | boolean | no | Day/night hint. |
|
||||||
|
| `conditionCode` | integer | no | WMO code when applicable. |
|
||||||
|
| `textDescription` | string | no | Human summary. |
|
||||||
|
| `temperatureC` | number | no | Celsius. |
|
||||||
|
| `temperatureCMin` | number | no | Celsius. |
|
||||||
|
| `temperatureCMax` | number | no | Celsius. |
|
||||||
|
| `dewpointC` | number | no | Celsius. |
|
||||||
|
| `relativeHumidityPercent` | number | no | Percent from 0 to 100. |
|
||||||
|
| `windDirectionDegrees` | number | no | Degrees. |
|
||||||
|
| `windSpeedKmh` | number | no | Kilometers per hour. |
|
||||||
|
| `windGustKmh` | number | no | Kilometers per hour. |
|
||||||
|
| `barometricPressurePa` | number | no | Pascals. |
|
||||||
|
| `visibilityMeters` | number | no | Meters. |
|
||||||
|
| `apparentTemperatureC` | number | no | Celsius. |
|
||||||
|
| `cloudCoverPercent` | number | no | Percent from 0 to 100. |
|
||||||
|
| `probabilityOfPrecipitationPercent` | number | no | Percent from 0 to 100. |
|
||||||
|
| `precipitationAmountMm` | number | no | Liquid-equivalent millimeters. |
|
||||||
|
| `snowfallDepthMm` | number | no | Millimeters. |
|
||||||
|
| `uvIndex` | number | no | Unitless index. |
|
||||||
|
|
||||||
|
## `weather.forecast_discussion.v1`
|
||||||
|
|
||||||
|
Payload type: `WeatherForecastDiscussion`.
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `officeId` | string | no | NWS office identifier. |
|
||||||
|
| `officeName` | string | no | Office name. |
|
||||||
|
| `product` | string | yes | Current value is `afd`. |
|
||||||
|
| `issuedAt` | timestamp | yes | Bulletin issue time. |
|
||||||
|
| `updatedAt` | timestamp | no | Subsequent update time. |
|
||||||
|
| `keyMessages` | array of strings | no | Extracted key messages. |
|
||||||
|
| `shortTerm` | object | no | Short-term section. |
|
||||||
|
| `longTerm` | object | no | Long-term section. |
|
||||||
|
|
||||||
|
`shortTerm` and `longTerm` sections contain optional `qualifier`, `issuedAt`,
|
||||||
|
and `text` fields.
|
||||||
|
|
||||||
|
## `weather.weather_story.v1`
|
||||||
|
|
||||||
|
Payload type: `WeatherStoryRun`.
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `officeId` | string | no | NWS office identifier. |
|
||||||
|
| `asOf` | timestamp | yes | Snapshot time. |
|
||||||
|
| `stories` | array | yes | Ordered story cards. |
|
||||||
|
|
||||||
|
`stories[]` entries:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `officeId` | string | no | Office identifier. |
|
||||||
|
| `startTime` | timestamp | yes | Story start. |
|
||||||
|
| `endTime` | timestamp | yes | Story end. |
|
||||||
|
| `updatedAt` | timestamp | yes | Story update time. |
|
||||||
|
| `title` | string | no | Story title. |
|
||||||
|
| `description` | string | no | Story description. |
|
||||||
|
| `altText` | string | no | Image alternate text. |
|
||||||
|
| `priority` | boolean | yes | Provider priority flag. |
|
||||||
|
| `order` | integer | yes | Provider display order. |
|
||||||
|
| `downloadUrl` | string | no | Story image URL. |
|
||||||
|
|
||||||
|
## `weather.alert.v1`
|
||||||
|
|
||||||
|
Payload type: `WeatherAlertRun`.
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `locationId` | string | no | Provider location identifier. |
|
||||||
|
| `locationName` | string | no | Human location name. |
|
||||||
|
| `asOf` | timestamp | yes | Snapshot time. |
|
||||||
|
| `latitude` | number | no | Degrees. |
|
||||||
|
| `longitude` | number | no | Degrees. |
|
||||||
|
| `alerts` | array | yes | Active alerts. |
|
||||||
|
|
||||||
|
`alerts[]` entries:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `id` | string | yes | Provider-stable alert identifier. |
|
||||||
|
| `event` | string | no | Alert event label. |
|
||||||
|
| `headline` | string | no | Alert headline. |
|
||||||
|
| `severity` | string | no | Provider severity. |
|
||||||
|
| `urgency` | string | no | Provider urgency. |
|
||||||
|
| `certainty` | string | no | Provider certainty. |
|
||||||
|
| `status` | string | no | Alert status. |
|
||||||
|
| `messageType` | string | no | Alert message type. |
|
||||||
|
| `category` | string | no | Alert category. |
|
||||||
|
| `response` | string | no | Recommended response. |
|
||||||
|
| `description` | string | no | Alert description. |
|
||||||
|
| `instruction` | string | no | Alert instruction. |
|
||||||
|
| `sent` | timestamp | no | Provider sent time. |
|
||||||
|
| `effective` | timestamp | no | Effective time. |
|
||||||
|
| `onset` | timestamp | no | Alert period start. |
|
||||||
|
| `ends` | timestamp | no | Alert period end. |
|
||||||
|
| `expires` | timestamp | no | Provider expiration metadata; not necessarily the alert period end. |
|
||||||
|
| `areaDescription` | string | no | Affected area description. |
|
||||||
|
| `senderName` | string | no | Provider sender name. |
|
||||||
|
| `references` | array | no | Related alerts. |
|
||||||
|
|
||||||
|
`references[]` entries contain optional `id`, `identifier`, `sender`, and
|
||||||
|
`sent` fields.
|
||||||
|
|
||||||
|
## `weather.outlook.v2`
|
||||||
|
|
||||||
|
Payload type: `WeatherOutlookRun`.
|
||||||
|
|
||||||
|
The current producer is the SPC convective outlook normalizer. It emits Day 1-3
|
||||||
|
convective outlook snapshots for categorical, tornado, hail, and wind products
|
||||||
|
that apply to the configured forecast point. Raw SPC bundles remain complete;
|
||||||
|
canonical outlook payloads are filtered to local polygons. All timestamps are
|
||||||
|
UTC.
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `locationId` | string | no | Operator-configured location identifier. |
|
||||||
|
| `locationName` | string | no | Operator-configured location label. |
|
||||||
|
| `latitude` | number | no | Configured point latitude in decimal degrees. |
|
||||||
|
| `longitude` | number | no | Configured point longitude in decimal degrees. |
|
||||||
|
| `asOf` | timestamp | yes | Snapshot time. For SPC, this is the latest outlook issue time when available. |
|
||||||
|
| `issuedAt` | timestamp | no | Latest issue time across outlook features when any feature exists. |
|
||||||
|
| `outlooks` | array | yes | Ordered outlook polygons containing the configured point. |
|
||||||
|
| `discussions` | array | yes | Run-level day discussions for retained outlook days. |
|
||||||
|
|
||||||
|
`outlooks[]` entries:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `id` | string | yes | Deterministic weatherfeeder outlook identifier. |
|
||||||
|
| `provider` | string | yes | Current value is `spc`. |
|
||||||
|
| `product` | string | yes | Current value is `convective`. |
|
||||||
|
| `day` | integer | yes | SPC outlook day, currently `1`, `2`, or `3`. |
|
||||||
|
| `outlookType` | string | yes | `categorical`, `tornado`, `hail`, or `wind`. |
|
||||||
|
| `label` | string | yes | SPC outlook label such as `SLGT` or `15`. |
|
||||||
|
| `labelText` | string | no | Human label text from SPC, when present. |
|
||||||
|
| `severityRank` | integer | no | SPC `DN` value, when present. |
|
||||||
|
| `validFrom` | timestamp | yes | Valid period start. |
|
||||||
|
| `validTo` | timestamp | yes | Valid period end. |
|
||||||
|
| `issuedAt` | timestamp | yes | Feature issue time. |
|
||||||
|
| `expiresAt` | timestamp | yes | Expiration time; currently equal to `validTo`. |
|
||||||
|
| `forecaster` | string | no | SPC forecaster text, when present. |
|
||||||
|
| `sourceUrl` | string | no | GeoJSON product URL for this outlook feature. |
|
||||||
|
| `imageUrl` | string | no | Reserved for provider image URLs; currently empty. |
|
||||||
|
| `containsLocation` | boolean | yes | Always `true` for emitted v2 outlooks. |
|
||||||
|
| `geometry` | object | yes | Compact GeoJSON `Polygon` or `MultiPolygon` geometry. |
|
||||||
|
|
||||||
|
`geometry` preserves the SPC feature geometry as compact GeoJSON using
|
||||||
|
`[longitude, latitude]` coordinate order. `containsLocation` is computed with
|
||||||
|
that geometry and the configured source `latitude`/`longitude`; boundary points
|
||||||
|
count as contained. Polygons that do not contain the configured point are not
|
||||||
|
included in canonical v2 payloads.
|
||||||
|
|
||||||
|
`discussions[]` entries:
|
||||||
|
|
||||||
|
| Field | Type | Required | Notes |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `day` | integer | yes | SPC outlook day, currently `1`, `2`, or `3`. |
|
||||||
|
| `headline` | string | no | Matching Day 1-3 print-page product title. |
|
||||||
|
| `summary` | string | no | Text from the print-page `...SUMMARY...` section. |
|
||||||
|
| `discussion` | string | no | Cleaned full print-page product text. |
|
||||||
|
| `updatedAt` | timestamp | no | Print-page update time, when present. |
|
||||||
|
|
||||||
|
When no SPC polygons apply locally, the run is still emitted with `outlooks: []`
|
||||||
|
and `discussions: []`. Discussions are included only for days represented by at
|
||||||
|
least one retained outlook, and multiple retained outlook types for the same day
|
||||||
|
share one discussion entry.
|
||||||
|
|
||||||
|
### SPC Outlook Supersession
|
||||||
|
|
||||||
|
Consumers should prefer latest-run semantics for current conditions: read the
|
||||||
|
latest `WeatherOutlookRun` for the configured location and use its `outlooks`
|
||||||
|
and `discussions` arrays together.
|
||||||
|
|
||||||
|
Historical SQL consumers that collapse older rows should identify superseded
|
||||||
|
outlooks by `provider`, `product`, `outlookType`, `validFrom`, and `validTo`,
|
||||||
|
then keep rows with the greatest `issuedAt`. `day` and `label` are not identity
|
||||||
|
fields. When multiple retained polygons share that latest `issuedAt`, preserve
|
||||||
|
the full group.
|
||||||
|
|
||||||
|
## Legacy `weather.outlook.v1`
|
||||||
|
|
||||||
|
`weather.outlook.v1` is a historical canonical schema retained as a standards
|
||||||
|
constant for older data and consumers. Current SPC normalization emits
|
||||||
|
`weather.outlook.v2`.
|
||||||
|
|
||||||
|
The v1 payload used `WeatherOutlookRun` and placed `headline`, `summary`, and
|
||||||
|
`discussion` on each `outlooks[]` polygon. It also represented the pre-v2 SPC
|
||||||
|
canonical behavior, where national polygons were preserved in canonical output.
|
||||||
|
|
||||||
|
## Compact Example
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "NWSObservationKSTL:2026-06-10T12:00:00Z",
|
||||||
|
"kind": "observation",
|
||||||
|
"source": "NWSObservationKSTL",
|
||||||
|
"emitted_at": "2026-06-10T12:00:05Z",
|
||||||
|
"effective_at": "2026-06-10T12:00:00Z",
|
||||||
|
"schema": "weather.observation.v1",
|
||||||
|
"payload": {
|
||||||
|
"stationId": "KSTL",
|
||||||
|
"timestamp": "2026-06-10T12:00:00Z",
|
||||||
|
"conditionCode": 0,
|
||||||
|
"temperatureC": 22.5
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
106
docs/integrations/feedkit.md
Normal file
106
docs/integrations/feedkit.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# Feedkit Integration
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes the feedkit runtime behavior that `weatherfeeder`
|
||||||
|
currently depends on. It is for maintainers and LLM coding agents changing
|
||||||
|
runtime wiring, config behavior, source construction, processing, routing, or
|
||||||
|
sink behavior.
|
||||||
|
|
||||||
|
Weather-domain behavior belongs in `weatherfeeder`. Generic daemon mechanics
|
||||||
|
belong to feedkit.
|
||||||
|
|
||||||
|
## Current Dependency
|
||||||
|
|
||||||
|
`weatherfeeder` imports feedkit as its daemon framework dependency. The exact
|
||||||
|
module version is declared in `go.mod`.
|
||||||
|
|
||||||
|
Feedkit provides:
|
||||||
|
|
||||||
|
- YAML config loading and validation.
|
||||||
|
- Source, processor, and sink registries.
|
||||||
|
- HTTP source helper behavior.
|
||||||
|
- Scheduler polling.
|
||||||
|
- Normalize and dedupe processors.
|
||||||
|
- Route compilation and sink dispatch.
|
||||||
|
- Built-in stdout, NATS, and Postgres sink mechanics.
|
||||||
|
|
||||||
|
## Config Contract
|
||||||
|
|
||||||
|
`cmd/weatherfeeder` calls feedkit config loading for `config.yml` in the current
|
||||||
|
working directory.
|
||||||
|
|
||||||
|
Implemented behavior relied on by weatherfeeder docs and tests:
|
||||||
|
|
||||||
|
- Top-level config contains `sources`, `sinks`, and optional `routes`.
|
||||||
|
- Config struct fields are decoded strictly, so misspelled struct fields fail
|
||||||
|
startup.
|
||||||
|
- Driver-specific `params` maps are decoded generically and validated by the
|
||||||
|
source or sink constructor that consumes them.
|
||||||
|
- Source `kinds` can be validated against a source's advertised `Kinds()`.
|
||||||
|
|
||||||
|
## Source And HTTP Contract
|
||||||
|
|
||||||
|
Most weatherfeeder sources use feedkit's single-document HTTP source helper for:
|
||||||
|
|
||||||
|
- request construction;
|
||||||
|
- `User-Agent` and `Accept` headers;
|
||||||
|
- optional conditional GET validators;
|
||||||
|
- response body size limits;
|
||||||
|
- context-aware HTTP work;
|
||||||
|
- unchanged `304 Not Modified` responses that emit no events.
|
||||||
|
|
||||||
|
The SPC convective outlook source fetches multiple documents itself, but it uses
|
||||||
|
feedkit transport helpers for HTTP clients and response body limits.
|
||||||
|
|
||||||
|
## Scheduler And Processing Contract
|
||||||
|
|
||||||
|
Weatherfeeder builds feedkit scheduler jobs from source configs. Current source
|
||||||
|
drivers are polling drivers and use the configured `every` interval.
|
||||||
|
|
||||||
|
Events flow through a feedkit pipeline in this order:
|
||||||
|
|
||||||
|
1. normalize processor;
|
||||||
|
2. dedupe processor.
|
||||||
|
|
||||||
|
The normalize processor is configured with `RequireMatch=false`, so unmatched
|
||||||
|
schemas pass through unchanged. Weatherfeeder registers its built-in normalizers
|
||||||
|
and owns the provider-to-canonical mapping.
|
||||||
|
|
||||||
|
The dedupe processor stores a bounded in-memory set of recent event IDs. The
|
||||||
|
bound is configured in `cmd/weatherfeeder`.
|
||||||
|
|
||||||
|
## Dispatch And Sink Contract
|
||||||
|
|
||||||
|
Feedkit compiles routes from config and dispatches processed events to matching
|
||||||
|
sinks. If `routes` is omitted, every configured sink receives every event kind.
|
||||||
|
|
||||||
|
Feedkit owns sink fanout mechanics, per-sink workers, queueing, context-aware
|
||||||
|
shutdown, and sink error logging. Weatherfeeder owns the event kinds and schemas
|
||||||
|
that make routes meaningful.
|
||||||
|
|
||||||
|
Built-in feedkit sinks used by weatherfeeder:
|
||||||
|
|
||||||
|
- `stdout`: validates and writes JSON events to stdout.
|
||||||
|
- `nats`: publishes JSON events to a configured subject.
|
||||||
|
- generic `postgres` sink factory: opens the database, ensures tables/indexes,
|
||||||
|
runs transactions, inserts mapped rows, and prunes when configured.
|
||||||
|
|
||||||
|
Weatherfeeder supplies its Postgres table schema and event mapper to feedkit's
|
||||||
|
Postgres sink factory. The table contract is documented in
|
||||||
|
[`postgres.md`](postgres.md).
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
Do not move weather-domain policy into feedkit. Weatherfeeder owns:
|
||||||
|
|
||||||
|
- provider source drivers;
|
||||||
|
- raw and canonical schema constants;
|
||||||
|
- event kind meaning;
|
||||||
|
- canonical payload structs;
|
||||||
|
- normalizers;
|
||||||
|
- Postgres table shape and row mapping.
|
||||||
|
|
||||||
|
Do not duplicate generic feedkit mechanics in weatherfeeder unless there is a
|
||||||
|
narrow weather-specific reason. Runtime composition details are documented in
|
||||||
|
[`../internal/runtime.md`](../internal/runtime.md).
|
||||||
123
docs/integrations/nws.md
Normal file
123
docs/integrations/nws.md
Normal file
@@ -0,0 +1,123 @@
|
|||||||
|
# NWS Integration Notes
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes the NWS products that `weatherfeeder` currently polls
|
||||||
|
and normalizes. It is for developers and operators maintaining NWS source URLs,
|
||||||
|
normalizers, fixtures, and tests.
|
||||||
|
|
||||||
|
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||||
|
events are documented in [event wire contract](events.md).
|
||||||
|
|
||||||
|
## Implemented Drivers
|
||||||
|
|
||||||
|
| Driver | Kind | Raw schema | Canonical schema |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `nws_observation` | `observation` | `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||||
|
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||||
|
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||||
|
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||||
|
|
||||||
|
## Config Requirements
|
||||||
|
|
||||||
|
All NWS drivers require HTTP source params:
|
||||||
|
|
||||||
|
- `url`
|
||||||
|
- `user_agent`
|
||||||
|
|
||||||
|
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
||||||
|
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
||||||
|
an upstream `304 Not Modified` response emits no event for that poll.
|
||||||
|
|
||||||
|
NWS expects a descriptive `User-Agent`. Do not use anonymous or placeholder
|
||||||
|
contact values in production configs.
|
||||||
|
|
||||||
|
## Upstream Shapes Used
|
||||||
|
|
||||||
|
`nws_observation` expects the latest station observation GeoJSON shape. The
|
||||||
|
normalizer uses fields under `properties` such as `stationId`, `stationName`,
|
||||||
|
`timestamp`, `textDescription`, measured values, `presentWeather`, and
|
||||||
|
`cloudLayers`, plus point geometry for day/night inference.
|
||||||
|
|
||||||
|
`nws_alerts` expects an alerts FeatureCollection. The normalizer uses the
|
||||||
|
collection `updated` timestamp, `title`, each feature ID, alert classification
|
||||||
|
fields, narrative fields, timing fields, sender fields, and references.
|
||||||
|
`properties.onset` and `properties.ends` map to the canonical alert period
|
||||||
|
start and end. `properties.expires` maps only to canonical `expires` provider
|
||||||
|
metadata and is not treated as the alert period end.
|
||||||
|
|
||||||
|
`nws_forecast_hourly` and `nws_forecast_narrative` expect gridpoint forecast
|
||||||
|
GeoJSON with `properties.generatedAt`, `properties.updateTime`, elevation,
|
||||||
|
polygon geometry, and ordered `periods`.
|
||||||
|
|
||||||
|
`nws_forecast_discussion` expects an HTML page containing the discussion text in
|
||||||
|
a `<pre>` block. The provider helper extracts office identity, product, issue
|
||||||
|
time, update time, key messages, and short/long term sections.
|
||||||
|
|
||||||
|
`nws_weatherstories` expects a JSON response with a `stories` array. The
|
||||||
|
normalizer uses office ID, start/end/update times, title, description, alt text,
|
||||||
|
priority, order, and download URL.
|
||||||
|
|
||||||
|
## Accept Headers
|
||||||
|
|
||||||
|
NWS JSON sources request:
|
||||||
|
|
||||||
|
```text
|
||||||
|
application/geo+json, application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
The forecast discussion source requests:
|
||||||
|
|
||||||
|
```text
|
||||||
|
text/html, application/xhtml+xml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Effective Time
|
||||||
|
|
||||||
|
Source events set `effective_at` from the best metadata available:
|
||||||
|
|
||||||
|
- observations: `properties.timestamp`;
|
||||||
|
- alerts: collection `updated`, otherwise latest per-alert timestamp;
|
||||||
|
- hourly and narrative forecasts: `properties.generatedAt`, otherwise update
|
||||||
|
time;
|
||||||
|
- forecast discussions: parsed issue time;
|
||||||
|
- weather stories: latest story update time, otherwise latest story start time.
|
||||||
|
|
||||||
|
Normalizers use canonical payload time as the normalized event effective time.
|
||||||
|
Alerts and weather stories fall back to the incoming event envelope when the
|
||||||
|
payload does not provide a better snapshot time.
|
||||||
|
|
||||||
|
## Mapping Notes
|
||||||
|
|
||||||
|
Observations preserve raw `presentWeather` fragments and infer WMO condition
|
||||||
|
codes from METAR phenomena, provider text, and cloud-layer fallback. Sea-level
|
||||||
|
pressure is preferred over barometric pressure when present.
|
||||||
|
|
||||||
|
Hourly forecasts infer WMO condition codes from `shortForecast` and icon tokens.
|
||||||
|
Narrative forecasts preserve text but intentionally leave period condition codes
|
||||||
|
unset. Forecast temperatures are converted to Celsius when NWS supplies
|
||||||
|
Fahrenheit, and wind speed strings are converted to kilometers per hour.
|
||||||
|
|
||||||
|
Alert timing fields are parsed best-effort. Invalid per-alert timestamps are
|
||||||
|
left unset rather than failing the whole alert run. NWS `ends` is preserved
|
||||||
|
separately from `expires`; `expires` does not fall back to `ends`. Missing alert
|
||||||
|
IDs are synthesized from the run snapshot time and array position.
|
||||||
|
|
||||||
|
Forecast discussion parsing requires an issue time. Weather story entries require
|
||||||
|
start time, end time, and update time.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Constructor validation failures stop daemon startup. Polling failures are
|
||||||
|
returned to the scheduler. JSON sources still emit raw payloads when only
|
||||||
|
minimal metadata decoding fails. Forecast discussion polling fails if the HTML
|
||||||
|
cannot be parsed enough to determine the issue time.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sources/nws/*_test.go`
|
||||||
|
- `internal/normalizers/nws/*_test.go`
|
||||||
|
- `internal/providers/nws/*_test.go`
|
||||||
|
- fixtures under `internal/providers/nws/testdata`
|
||||||
101
docs/integrations/openmeteo.md
Normal file
101
docs/integrations/openmeteo.md
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
# Open-Meteo Integration Notes
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes the Open-Meteo API usage currently implemented by
|
||||||
|
`weatherfeeder`. It is for developers and operators maintaining Open-Meteo
|
||||||
|
source URLs, normalizers, fixtures, and tests.
|
||||||
|
|
||||||
|
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||||
|
events are documented in [event wire contract](events.md).
|
||||||
|
|
||||||
|
## Implemented Drivers
|
||||||
|
|
||||||
|
| Driver | Kind | Raw schema | Canonical schema |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||||
|
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
|
||||||
|
## Config Requirements
|
||||||
|
|
||||||
|
Both drivers require HTTP source params:
|
||||||
|
|
||||||
|
- `url`
|
||||||
|
- `user_agent`
|
||||||
|
|
||||||
|
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
||||||
|
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
||||||
|
an upstream `304 Not Modified` response emits no event for that poll.
|
||||||
|
|
||||||
|
## Upstream Shapes Used
|
||||||
|
|
||||||
|
`openmeteo_observation` expects a JSON response with top-level location/timezone
|
||||||
|
metadata and a `current` object. The normalizer uses:
|
||||||
|
|
||||||
|
- `latitude`, `longitude`, `timezone`, `utc_offset_seconds`;
|
||||||
|
- `current.time`;
|
||||||
|
- current temperature, apparent temperature, relative humidity, weather code,
|
||||||
|
wind speed/direction/gusts, pressure, and `is_day`.
|
||||||
|
|
||||||
|
`openmeteo_forecast` expects top-level location/timezone metadata and an
|
||||||
|
array-oriented `hourly` object. The normalizer uses:
|
||||||
|
|
||||||
|
- `hourly.time`;
|
||||||
|
- hourly temperature, apparent temperature, dew point, relative humidity,
|
||||||
|
precipitation probability, precipitation amount, snowfall, weather code,
|
||||||
|
pressure, wind speed/direction/gusts, `is_day`, cloud cover, visibility, and
|
||||||
|
UV index.
|
||||||
|
|
||||||
|
Open-Meteo field presence is allowed to vary. Missing optional arrays produce
|
||||||
|
nil canonical fields for the affected periods.
|
||||||
|
|
||||||
|
## Accept Header
|
||||||
|
|
||||||
|
Open-Meteo sources request:
|
||||||
|
|
||||||
|
```text
|
||||||
|
application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Time Handling
|
||||||
|
|
||||||
|
Open-Meteo timestamps often omit an explicit offset. The provider helper parses
|
||||||
|
times by using the returned `timezone` or `utc_offset_seconds` when needed.
|
||||||
|
|
||||||
|
Observation source events set `effective_at` from `current.time` when it can be
|
||||||
|
parsed. Hourly forecast source events prefer `current.time`, then the first
|
||||||
|
non-empty `hourly.time` entry.
|
||||||
|
|
||||||
|
The hourly forecast normalizer sets canonical `issuedAt` from the incoming event
|
||||||
|
`emitted_at` when present, otherwise from the first hourly period start.
|
||||||
|
Normalized forecast `effective_at` matches `issuedAt`.
|
||||||
|
|
||||||
|
## Mapping Notes
|
||||||
|
|
||||||
|
Open-Meteo is not a station feed. Weatherfeeder synthesizes canonical
|
||||||
|
station/location IDs from latitude and longitude when both are available.
|
||||||
|
|
||||||
|
Open-Meteo weather codes are WMO codes and are treated as authoritative.
|
||||||
|
Canonical text is derived from the WMO code and day/night hint.
|
||||||
|
|
||||||
|
Wind speed and gust fields are treated as kilometers per hour. Pressure values
|
||||||
|
are treated as hPa and converted to Pa. Snowfall values are treated as
|
||||||
|
centimeters and converted to millimeters.
|
||||||
|
|
||||||
|
Hourly forecast period end time is the next period start. The last period uses
|
||||||
|
the previous interval length, or one hour when there is no previous interval.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Constructor validation failures stop daemon startup. Polling failures are
|
||||||
|
returned to the scheduler. Metadata decoding failures in sources still allow raw
|
||||||
|
payload emission when the HTTP response itself succeeded.
|
||||||
|
|
||||||
|
Normalization fails when required time data is missing or invalid, such as an
|
||||||
|
empty `hourly.time` array for hourly forecasts.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sources/openmeteo/source_test.go`
|
||||||
|
- `internal/normalizers/openmeteo/*_test.go`
|
||||||
|
- `internal/providers/openmeteo/*_test.go`
|
||||||
101
docs/integrations/openweather.md
Normal file
101
docs/integrations/openweather.md
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
# OpenWeather Integration Notes
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes the OpenWeather current-weather usage implemented by
|
||||||
|
`weatherfeeder`. It is for developers and operators maintaining OpenWeather
|
||||||
|
source URLs, normalizers, fixtures, and tests.
|
||||||
|
|
||||||
|
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||||
|
events are documented in [event wire contract](events.md).
|
||||||
|
|
||||||
|
## Implemented Driver
|
||||||
|
|
||||||
|
| Driver | Kind | Raw schema | Canonical schema |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `openweather_observation` | `observation` | `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||||
|
|
||||||
|
Only current-weather observation polling is registered for OpenWeather.
|
||||||
|
|
||||||
|
## Config Requirements
|
||||||
|
|
||||||
|
The driver requires HTTP source params:
|
||||||
|
|
||||||
|
- `url`
|
||||||
|
- `user_agent`
|
||||||
|
|
||||||
|
The shared HTTP helper also accepts `conditional`, `http_timeout`, and
|
||||||
|
`http_response_body_limit_bytes`. Conditional requests are enabled by default;
|
||||||
|
an upstream `304 Not Modified` response emits no event for that poll.
|
||||||
|
|
||||||
|
The configured URL must include:
|
||||||
|
|
||||||
|
```text
|
||||||
|
units=metric
|
||||||
|
```
|
||||||
|
|
||||||
|
Startup fails if `units` is omitted or set to another value. Keep OpenWeather
|
||||||
|
API keys out of committed configs. Use local config management or deployment
|
||||||
|
secrets for the `appid` query parameter.
|
||||||
|
|
||||||
|
## Upstream Shape Used
|
||||||
|
|
||||||
|
The source emits the full current-weather JSON payload as a raw event. The
|
||||||
|
normalizer uses:
|
||||||
|
|
||||||
|
- `coord.lat`, `coord.lon`;
|
||||||
|
- primary `weather[0]` condition ID, description, and icon;
|
||||||
|
- `main.temp`, `main.feels_like`, `main.pressure`, `main.humidity`, and
|
||||||
|
optional `main.sea_level`;
|
||||||
|
- `visibility`;
|
||||||
|
- `wind.speed`, `wind.deg`, and `wind.gust`;
|
||||||
|
- `dt`;
|
||||||
|
- `sys.sunrise` and `sys.sunset`;
|
||||||
|
- `id` and `name`.
|
||||||
|
|
||||||
|
## Accept Header
|
||||||
|
|
||||||
|
OpenWeather sources request:
|
||||||
|
|
||||||
|
```text
|
||||||
|
application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
## Time Handling
|
||||||
|
|
||||||
|
Source events set `effective_at` from `dt` when it is present and positive.
|
||||||
|
The normalizer also uses `dt` as the canonical observation timestamp and
|
||||||
|
normalized effective time.
|
||||||
|
|
||||||
|
## Mapping Notes
|
||||||
|
|
||||||
|
Metric units are required so canonical unit conversion is deterministic:
|
||||||
|
|
||||||
|
- `main.temp` and `main.feels_like` are treated as Celsius;
|
||||||
|
- `wind.speed` and `wind.gust` are treated as meters per second and converted to
|
||||||
|
kilometers per hour;
|
||||||
|
- pressure values are treated as hPa and converted to Pa.
|
||||||
|
|
||||||
|
The primary condition is `weather[0]`. OpenWeather condition IDs are mapped into
|
||||||
|
the canonical WMO code vocabulary. The human text description is preserved from
|
||||||
|
the provider description.
|
||||||
|
|
||||||
|
Day/night is inferred from the OpenWeather icon suffix when available, otherwise
|
||||||
|
from sunrise and sunset bounds.
|
||||||
|
|
||||||
|
The station ID uses the OpenWeather city ID when present. If no city ID is
|
||||||
|
present, weatherfeeder synthesizes an ID from coordinates. The station name uses
|
||||||
|
the provider `name`, falling back to `OpenWeatherMap` when blank.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Constructor validation failures stop daemon startup. Polling also re-checks the
|
||||||
|
metric-unit requirement before fetching. HTTP failures are returned to the
|
||||||
|
scheduler. Metadata decoding failures in the source still allow raw payload
|
||||||
|
emission when the HTTP response itself succeeded.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sources/openweather/source_test.go`
|
||||||
|
- `internal/normalizers/openweather/*_test.go`
|
||||||
|
- `internal/providers/openweather/*_test.go`
|
||||||
502
docs/integrations/postgres.md
Normal file
502
docs/integrations/postgres.md
Normal file
@@ -0,0 +1,502 @@
|
|||||||
|
# Postgres Integration
|
||||||
|
|
||||||
|
This document is the canonical table contract for the optional `postgres` sink.
|
||||||
|
It describes the schema created and written by weatherfeeder through feedkit's
|
||||||
|
Postgres sink.
|
||||||
|
|
||||||
|
Configure the sink as described in [configuration](../config.md#postgres).
|
||||||
|
|
||||||
|
## Initialization And Writes
|
||||||
|
|
||||||
|
At startup, each configured Postgres sink opens the database and runs
|
||||||
|
`CREATE TABLE IF NOT EXISTS` for every weatherfeeder table, followed by
|
||||||
|
`CREATE INDEX IF NOT EXISTS` for every configured index.
|
||||||
|
|
||||||
|
This initialization creates missing tables and indexes only. It does not alter
|
||||||
|
existing tables, migrate column definitions, drop old objects, or backfill data.
|
||||||
|
Schema changes require operator-managed database migration.
|
||||||
|
|
||||||
|
Events are mapped only for canonical weather schemas:
|
||||||
|
|
||||||
|
- `weather.observation.v1`
|
||||||
|
- `weather.forecast.v1`
|
||||||
|
- `weather.forecast_discussion.v1`
|
||||||
|
- `weather.weather_story.v1`
|
||||||
|
- `weather.alert.v1`
|
||||||
|
- `weather.outlook.v2`
|
||||||
|
|
||||||
|
Unsupported schemas produce no writes for this sink. Mapped events are inserted
|
||||||
|
transactionally. Inserts use ordinary `INSERT`; duplicate primary keys fail the
|
||||||
|
write.
|
||||||
|
|
||||||
|
## Shared Envelope Columns
|
||||||
|
|
||||||
|
Parent tables store the feed event envelope:
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
|
||||||
|
## Table Overview
|
||||||
|
|
||||||
|
| Table | Primary key | Prune column |
|
||||||
|
|---|---|---|
|
||||||
|
| `observations` | `event_id` | `observed_at` |
|
||||||
|
| `observation_present_weather` | `event_id`, `weather_index` | `observed_at` |
|
||||||
|
| `forecasts` | `event_id` | `issued_at` |
|
||||||
|
| `forecast_periods` | `run_event_id`, `period_index` | `issued_at` |
|
||||||
|
| `forecast_discussions` | `event_id` | `issued_at` |
|
||||||
|
| `forecast_discussion_key_messages` | `run_event_id`, `message_index` | `issued_at` |
|
||||||
|
| `weather_story_runs` | `event_id` | `as_of` |
|
||||||
|
| `weather_stories` | `run_event_id`, `story_index` | `as_of` |
|
||||||
|
| `alert_runs` | `event_id` | `as_of` |
|
||||||
|
| `alerts` | `run_event_id`, `alert_index` | `as_of` |
|
||||||
|
| `alert_references` | `run_event_id`, `alert_index`, `reference_index` | `as_of` |
|
||||||
|
| `outlook_runs` | `event_id` | `as_of` |
|
||||||
|
| `outlooks` | `run_event_id`, `outlook_index` | `as_of` |
|
||||||
|
| `outlook_discussions` | `run_event_id`, `discussion_index` | `as_of` |
|
||||||
|
|
||||||
|
## Table Contract
|
||||||
|
|
||||||
|
### `observations`
|
||||||
|
|
||||||
|
Primary key: `event_id`
|
||||||
|
|
||||||
|
Prune column: `observed_at`
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_obs_station_observed_at` on `station_id`, `observed_at`
|
||||||
|
- `idx_wf_obs_observed_at` on `observed_at`
|
||||||
|
- `idx_wf_obs_condition_code` on `condition_code`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
| `station_id` | `TEXT` | yes | `payload.stationId` |
|
||||||
|
| `station_name` | `TEXT` | yes | `payload.stationName` |
|
||||||
|
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
|
||||||
|
| `condition_code` | `INTEGER` | no | `payload.conditionCode` |
|
||||||
|
| `is_day` | `BOOLEAN` | yes | `payload.isDay` |
|
||||||
|
| `text_description` | `TEXT` | yes | `payload.textDescription` |
|
||||||
|
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.temperatureC` |
|
||||||
|
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.dewpointC` |
|
||||||
|
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.windDirectionDegrees` |
|
||||||
|
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.windSpeedKmh` |
|
||||||
|
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.windGustKmh` |
|
||||||
|
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.barometricPressurePa` |
|
||||||
|
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.visibilityMeters` |
|
||||||
|
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.relativeHumidityPercent` |
|
||||||
|
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.apparentTemperatureC` |
|
||||||
|
|
||||||
|
### `observation_present_weather`
|
||||||
|
|
||||||
|
Primary key: `event_id`, `weather_index`
|
||||||
|
|
||||||
|
Prune column: `observed_at`
|
||||||
|
|
||||||
|
Foreign key: `event_id` references `observations(event_id)` with cascade delete.
|
||||||
|
|
||||||
|
Index: `idx_wf_obs_present_observed_at` on `observed_at`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT REFERENCES observations(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `weather_index` | `INTEGER` | no | `payload.presentWeather[]` index. |
|
||||||
|
| `observed_at` | `TIMESTAMPTZ` | no | `payload.timestamp` |
|
||||||
|
| `raw_text` | `TEXT` | yes | Compact JSON text from `payload.presentWeather[].raw` |
|
||||||
|
|
||||||
|
### `forecasts`
|
||||||
|
|
||||||
|
Primary key: `event_id`
|
||||||
|
|
||||||
|
Prune column: `issued_at`
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_fc_location_product_issued_at` on `location_id`, `product`, `issued_at`
|
||||||
|
- `idx_wf_fc_issued_at` on `issued_at`
|
||||||
|
- `idx_wf_fc_product_issued_at` on `product`, `issued_at`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
| `location_id` | `TEXT` | yes | `payload.locationId` |
|
||||||
|
| `location_name` | `TEXT` | yes | `payload.locationName` |
|
||||||
|
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
|
||||||
|
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
|
||||||
|
| `product` | `TEXT` | no | `payload.product` |
|
||||||
|
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
|
||||||
|
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
|
||||||
|
| `elevation_meters` | `DOUBLE PRECISION` | yes | `payload.elevationMeters` |
|
||||||
|
| `period_count` | `INTEGER` | no | `len(payload.periods)` |
|
||||||
|
|
||||||
|
### `forecast_periods`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `period_index`
|
||||||
|
|
||||||
|
Prune column: `issued_at`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `forecasts(event_id)` with cascade delete.
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_fc_period_start_time` on `start_time`
|
||||||
|
- `idx_wf_fc_period_end_time` on `end_time`
|
||||||
|
- `idx_wf_fc_period_run_start` on `run_event_id`, `start_time`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES forecasts(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `period_index` | `INTEGER` | no | `payload.periods[]` index. |
|
||||||
|
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
|
||||||
|
| `start_time` | `TIMESTAMPTZ` | no | `payload.periods[].startTime` |
|
||||||
|
| `end_time` | `TIMESTAMPTZ` | no | `payload.periods[].endTime` |
|
||||||
|
| `name` | `TEXT` | yes | `payload.periods[].name` |
|
||||||
|
| `is_day` | `BOOLEAN` | yes | `payload.periods[].isDay` |
|
||||||
|
| `condition_code` | `INTEGER` | yes | `payload.periods[].conditionCode` |
|
||||||
|
| `text_description` | `TEXT` | yes | `payload.periods[].textDescription` |
|
||||||
|
| `temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureC` |
|
||||||
|
| `temperature_c_min` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMin` |
|
||||||
|
| `temperature_c_max` | `DOUBLE PRECISION` | yes | `payload.periods[].temperatureCMax` |
|
||||||
|
| `dewpoint_c` | `DOUBLE PRECISION` | yes | `payload.periods[].dewpointC` |
|
||||||
|
| `relative_humidity_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].relativeHumidityPercent` |
|
||||||
|
| `wind_direction_degrees` | `DOUBLE PRECISION` | yes | `payload.periods[].windDirectionDegrees` |
|
||||||
|
| `wind_speed_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windSpeedKmh` |
|
||||||
|
| `wind_gust_kmh` | `DOUBLE PRECISION` | yes | `payload.periods[].windGustKmh` |
|
||||||
|
| `barometric_pressure_pa` | `DOUBLE PRECISION` | yes | `payload.periods[].barometricPressurePa` |
|
||||||
|
| `visibility_meters` | `DOUBLE PRECISION` | yes | `payload.periods[].visibilityMeters` |
|
||||||
|
| `apparent_temperature_c` | `DOUBLE PRECISION` | yes | `payload.periods[].apparentTemperatureC` |
|
||||||
|
| `cloud_cover_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].cloudCoverPercent` |
|
||||||
|
| `probability_of_precipitation_percent` | `DOUBLE PRECISION` | yes | `payload.periods[].probabilityOfPrecipitationPercent` |
|
||||||
|
| `precipitation_amount_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].precipitationAmountMm` |
|
||||||
|
| `snowfall_depth_mm` | `DOUBLE PRECISION` | yes | `payload.periods[].snowfallDepthMm` |
|
||||||
|
| `uv_index` | `DOUBLE PRECISION` | yes | `payload.periods[].uvIndex` |
|
||||||
|
|
||||||
|
### `forecast_discussions`
|
||||||
|
|
||||||
|
Primary key: `event_id`
|
||||||
|
|
||||||
|
Prune column: `issued_at`
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_discussion_office_product_issued_at` on `office_id`, `product`, `issued_at`
|
||||||
|
- `idx_wf_discussion_issued_at` on `issued_at`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
| `office_id` | `TEXT` | yes | `payload.officeId` |
|
||||||
|
| `office_name` | `TEXT` | yes | `payload.officeName` |
|
||||||
|
| `issued_at` | `TIMESTAMPTZ` | no | `payload.issuedAt` |
|
||||||
|
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.updatedAt` |
|
||||||
|
| `product` | `TEXT` | no | `payload.product` |
|
||||||
|
| `short_term_qualifier` | `TEXT` | yes | `payload.shortTerm.qualifier` |
|
||||||
|
| `short_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.shortTerm.issuedAt` |
|
||||||
|
| `short_term_text` | `TEXT` | yes | `payload.shortTerm.text` |
|
||||||
|
| `long_term_qualifier` | `TEXT` | yes | `payload.longTerm.qualifier` |
|
||||||
|
| `long_term_issued_at` | `TIMESTAMPTZ` | yes | `payload.longTerm.issuedAt` |
|
||||||
|
| `long_term_text` | `TEXT` | yes | `payload.longTerm.text` |
|
||||||
|
| `key_message_count` | `INTEGER` | no | `len(payload.keyMessages)` |
|
||||||
|
|
||||||
|
### `forecast_discussion_key_messages`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `message_index`
|
||||||
|
|
||||||
|
Prune column: `issued_at`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `forecast_discussions(event_id)` with
|
||||||
|
cascade delete.
|
||||||
|
|
||||||
|
Index: `idx_wf_discussion_message_issued_at` on `issued_at`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES forecast_discussions(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `message_index` | `INTEGER` | no | `payload.keyMessages[]` index. |
|
||||||
|
| `issued_at` | `TIMESTAMPTZ` | no | Parent `payload.issuedAt` |
|
||||||
|
| `message_text` | `TEXT` | yes | `payload.keyMessages[]` value |
|
||||||
|
|
||||||
|
### `weather_story_runs`
|
||||||
|
|
||||||
|
Primary key: `event_id`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_story_run_office_as_of` on `office_id`, `as_of`
|
||||||
|
- `idx_wf_story_run_as_of` on `as_of`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
| `office_id` | `TEXT` | yes | `payload.officeId` |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
|
||||||
|
| `story_count` | `INTEGER` | no | `len(payload.stories)` |
|
||||||
|
|
||||||
|
### `weather_stories`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `story_index`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `weather_story_runs(event_id)` with
|
||||||
|
cascade delete.
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_stories_start_time` on `start_time`
|
||||||
|
- `idx_wf_stories_end_time` on `end_time`
|
||||||
|
- `idx_wf_stories_updated_at` on `updated_at`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `story_index` | `INTEGER` | no | `payload.stories[]` index. |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||||
|
| `office_id` | `TEXT` | yes | `payload.stories[].officeId` |
|
||||||
|
| `start_time` | `TIMESTAMPTZ` | no | `payload.stories[].startTime` |
|
||||||
|
| `end_time` | `TIMESTAMPTZ` | no | `payload.stories[].endTime` |
|
||||||
|
| `updated_at` | `TIMESTAMPTZ` | no | `payload.stories[].updatedAt` |
|
||||||
|
| `title` | `TEXT` | yes | `payload.stories[].title` |
|
||||||
|
| `description` | `TEXT` | yes | `payload.stories[].description` |
|
||||||
|
| `alt_text` | `TEXT` | yes | `payload.stories[].altText` |
|
||||||
|
| `priority` | `BOOLEAN` | no | `payload.stories[].priority` |
|
||||||
|
| `story_order` | `INTEGER` | no | `payload.stories[].order` |
|
||||||
|
| `download_url` | `TEXT` | yes | `payload.stories[].downloadUrl` |
|
||||||
|
|
||||||
|
### `alert_runs`
|
||||||
|
|
||||||
|
Primary key: `event_id`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_alert_run_location_as_of` on `location_id`, `as_of`
|
||||||
|
- `idx_wf_alert_run_as_of` on `as_of`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
| `location_id` | `TEXT` | yes | `payload.locationId` |
|
||||||
|
| `location_name` | `TEXT` | yes | `payload.locationName` |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
|
||||||
|
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
|
||||||
|
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
|
||||||
|
| `alert_count` | `INTEGER` | no | `len(payload.alerts)` |
|
||||||
|
|
||||||
|
### `alerts`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `alert_index`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
|
||||||
|
delete.
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_alerts_alert_id` on `alert_id`
|
||||||
|
- `idx_wf_alerts_severity_expires` on `severity`, `expires`
|
||||||
|
- `idx_wf_alerts_as_of` on `as_of`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `alert_index` | `INTEGER` | no | `payload.alerts[]` index. |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||||
|
| `alert_id` | `TEXT` | no | `payload.alerts[].id` |
|
||||||
|
| `event` | `TEXT` | yes | `payload.alerts[].event` |
|
||||||
|
| `headline` | `TEXT` | yes | `payload.alerts[].headline` |
|
||||||
|
| `severity` | `TEXT` | yes | `payload.alerts[].severity` |
|
||||||
|
| `urgency` | `TEXT` | yes | `payload.alerts[].urgency` |
|
||||||
|
| `certainty` | `TEXT` | yes | `payload.alerts[].certainty` |
|
||||||
|
| `status` | `TEXT` | yes | `payload.alerts[].status` |
|
||||||
|
| `message_type` | `TEXT` | yes | `payload.alerts[].messageType` |
|
||||||
|
| `category` | `TEXT` | yes | `payload.alerts[].category` |
|
||||||
|
| `response` | `TEXT` | yes | `payload.alerts[].response` |
|
||||||
|
| `description` | `TEXT` | yes | `payload.alerts[].description` |
|
||||||
|
| `instruction` | `TEXT` | yes | `payload.alerts[].instruction` |
|
||||||
|
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].sent` |
|
||||||
|
| `effective` | `TIMESTAMPTZ` | yes | `payload.alerts[].effective` |
|
||||||
|
| `onset` | `TIMESTAMPTZ` | yes | `payload.alerts[].onset` |
|
||||||
|
| `ends` | `TIMESTAMPTZ` | yes | `payload.alerts[].ends` |
|
||||||
|
| `expires` | `TIMESTAMPTZ` | yes | `payload.alerts[].expires` |
|
||||||
|
| `area_description` | `TEXT` | yes | `payload.alerts[].areaDescription` |
|
||||||
|
| `sender_name` | `TEXT` | yes | `payload.alerts[].senderName` |
|
||||||
|
| `reference_count` | `INTEGER` | no | `len(payload.alerts[].references)` |
|
||||||
|
|
||||||
|
### `alert_references`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `alert_index`, `reference_index`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `alert_runs(event_id)` with cascade
|
||||||
|
delete.
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_alert_refs_as_of` on `as_of`
|
||||||
|
- `idx_wf_alert_refs_sent` on `sent`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES alert_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `alert_index` | `INTEGER` | no | Parent alert index. |
|
||||||
|
| `reference_index` | `INTEGER` | no | `payload.alerts[].references[]` index. |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||||
|
| `id` | `TEXT` | yes | `payload.alerts[].references[].id` |
|
||||||
|
| `identifier` | `TEXT` | yes | `payload.alerts[].references[].identifier` |
|
||||||
|
| `sender` | `TEXT` | yes | `payload.alerts[].references[].sender` |
|
||||||
|
| `sent` | `TIMESTAMPTZ` | yes | `payload.alerts[].references[].sent` |
|
||||||
|
|
||||||
|
### `outlook_runs`
|
||||||
|
|
||||||
|
Primary key: `event_id`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_outlook_run_location_as_of` on `location_id`, `as_of`
|
||||||
|
- `idx_wf_outlook_run_as_of` on `as_of`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `event_id` | `TEXT` | no | `event.id` |
|
||||||
|
| `event_kind` | `TEXT` | no | `event.kind` |
|
||||||
|
| `event_source` | `TEXT` | no | `event.source` |
|
||||||
|
| `event_schema` | `TEXT` | no | `event.schema` |
|
||||||
|
| `event_emitted_at` | `TIMESTAMPTZ` | no | `event.emitted_at` |
|
||||||
|
| `event_effective_at` | `TIMESTAMPTZ` | yes | `event.effective_at` |
|
||||||
|
| `location_id` | `TEXT` | yes | `payload.locationId` |
|
||||||
|
| `location_name` | `TEXT` | yes | `payload.locationName` |
|
||||||
|
| `latitude` | `DOUBLE PRECISION` | yes | `payload.latitude` |
|
||||||
|
| `longitude` | `DOUBLE PRECISION` | yes | `payload.longitude` |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | `payload.asOf` |
|
||||||
|
| `issued_at` | `TIMESTAMPTZ` | yes | `payload.issuedAt` |
|
||||||
|
| `outlook_count` | `INTEGER` | no | `len(payload.outlooks)` |
|
||||||
|
| `discussion_count` | `INTEGER` | no | `len(payload.discussions)` |
|
||||||
|
|
||||||
|
### `outlooks`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `outlook_index`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `outlook_runs(event_id)` with cascade
|
||||||
|
delete.
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_outlooks_contains_valid` on `contains_location`, `valid_from`, `valid_to`
|
||||||
|
- `idx_wf_outlooks_day_type_label` on `day`, `outlook_type`, `label`
|
||||||
|
- `idx_wf_outlooks_valid` on `valid_from`, `valid_to`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `outlook_index` | `INTEGER` | no | `payload.outlooks[]` index. |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||||
|
| `outlook_id` | `TEXT` | no | `payload.outlooks[].id` |
|
||||||
|
| `provider` | `TEXT` | no | `payload.outlooks[].provider` |
|
||||||
|
| `product` | `TEXT` | no | `payload.outlooks[].product` |
|
||||||
|
| `day` | `INTEGER` | no | `payload.outlooks[].day` |
|
||||||
|
| `outlook_type` | `TEXT` | no | `payload.outlooks[].outlookType` |
|
||||||
|
| `label` | `TEXT` | no | `payload.outlooks[].label` |
|
||||||
|
| `label_text` | `TEXT` | yes | `payload.outlooks[].labelText` |
|
||||||
|
| `severity_rank` | `INTEGER` | yes | `payload.outlooks[].severityRank` |
|
||||||
|
| `valid_from` | `TIMESTAMPTZ` | no | `payload.outlooks[].validFrom` |
|
||||||
|
| `valid_to` | `TIMESTAMPTZ` | no | `payload.outlooks[].validTo` |
|
||||||
|
| `issued_at` | `TIMESTAMPTZ` | no | `payload.outlooks[].issuedAt` |
|
||||||
|
| `expires_at` | `TIMESTAMPTZ` | no | `payload.outlooks[].expiresAt` |
|
||||||
|
| `forecaster` | `TEXT` | yes | `payload.outlooks[].forecaster` |
|
||||||
|
| `source_url` | `TEXT` | yes | `payload.outlooks[].sourceUrl` |
|
||||||
|
| `image_url` | `TEXT` | yes | `payload.outlooks[].imageUrl` |
|
||||||
|
| `contains_location` | `BOOLEAN` | no | `payload.outlooks[].containsLocation` |
|
||||||
|
| `geometry_json` | `TEXT` | no | Compact JSON from `payload.outlooks[].geometry` |
|
||||||
|
|
||||||
|
### `outlook_discussions`
|
||||||
|
|
||||||
|
Primary key: `run_event_id`, `discussion_index`
|
||||||
|
|
||||||
|
Prune column: `as_of`
|
||||||
|
|
||||||
|
Foreign key: `run_event_id` references `outlook_runs(event_id)` with cascade
|
||||||
|
delete.
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
|
||||||
|
- `idx_wf_outlook_discussions_day_as_of` on `day`, `as_of`
|
||||||
|
- unique `idx_wf_outlook_discussions_run_day` on `run_event_id`, `day`
|
||||||
|
|
||||||
|
| Column | Type | Null | Source |
|
||||||
|
|---|---|:---:|---|
|
||||||
|
| `run_event_id` | `TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE` | no | Parent event ID. |
|
||||||
|
| `discussion_index` | `INTEGER` | no | `payload.discussions[]` index. |
|
||||||
|
| `as_of` | `TIMESTAMPTZ` | no | Parent `payload.asOf` |
|
||||||
|
| `day` | `INTEGER` | no | `payload.discussions[].day` |
|
||||||
|
| `headline` | `TEXT` | yes | `payload.discussions[].headline` |
|
||||||
|
| `summary` | `TEXT` | yes | `payload.discussions[].summary` |
|
||||||
|
| `discussion` | `TEXT` | yes | `payload.discussions[].discussion` |
|
||||||
|
| `updated_at` | `TIMESTAMPTZ` | yes | `payload.discussions[].updatedAt` |
|
||||||
|
|
||||||
|
## Retention
|
||||||
|
|
||||||
|
When sink param `prune` is set, every successful write transaction deletes rows
|
||||||
|
older than `now - prune` from every table using that table's prune column.
|
||||||
|
|
||||||
|
The sink also exposes manual prune helpers in code, but the `weatherfeeder`
|
||||||
|
binary does not provide CLI commands for them.
|
||||||
|
|
||||||
|
## Reconstructing Canonical Payloads
|
||||||
|
|
||||||
|
- `WeatherObservation`: read `observations`, then join
|
||||||
|
`observation_present_weather` by `event_id` ordered by `weather_index`.
|
||||||
|
- `WeatherForecastRun`: read `forecasts`, then join `forecast_periods` by
|
||||||
|
`run_event_id` ordered by `period_index`.
|
||||||
|
- `WeatherForecastDiscussion`: read `forecast_discussions`, then join
|
||||||
|
`forecast_discussion_key_messages` by `run_event_id` ordered by
|
||||||
|
`message_index`.
|
||||||
|
- `WeatherStoryRun`: read `weather_story_runs`, then join `weather_stories` by
|
||||||
|
`run_event_id` ordered by `story_index`.
|
||||||
|
- `WeatherAlertRun`: read `alert_runs`, join `alerts` by `run_event_id` ordered
|
||||||
|
by `alert_index`, then join `alert_references` by `run_event_id` and
|
||||||
|
`alert_index` ordered by `reference_index`.
|
||||||
|
- `WeatherOutlookRun`: read `outlook_runs`, join `outlooks` by `run_event_id`
|
||||||
|
ordered by `outlook_index`, then join `outlook_discussions` by
|
||||||
|
`run_event_id` ordered by `discussion_index`.
|
||||||
144
docs/integrations/spc.md
Normal file
144
docs/integrations/spc.md
Normal file
@@ -0,0 +1,144 @@
|
|||||||
|
# SPC Integration Notes
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes the Storm Prediction Center convective outlook usage
|
||||||
|
implemented by `weatherfeeder`. It is for developers and operators maintaining
|
||||||
|
SPC source configuration, provider helpers, fixtures, and tests.
|
||||||
|
|
||||||
|
General config syntax belongs in [configuration](../config.md). Emitted JSON
|
||||||
|
events are documented in [event wire contract](events.md).
|
||||||
|
|
||||||
|
## Implemented Driver
|
||||||
|
|
||||||
|
| Driver | Kind | Raw schema | Canonical schema |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` | `weather.outlook.v2` |
|
||||||
|
|
||||||
|
## Config Requirements
|
||||||
|
|
||||||
|
The driver requires:
|
||||||
|
|
||||||
|
- `latitude`
|
||||||
|
- `longitude`
|
||||||
|
- `user_agent`
|
||||||
|
|
||||||
|
Optional params are:
|
||||||
|
|
||||||
|
- `location_id`
|
||||||
|
- `location_name`
|
||||||
|
- `geojson_urls`
|
||||||
|
- `discussion_urls`
|
||||||
|
- `rss_url`
|
||||||
|
- `http_timeout`
|
||||||
|
- `http_response_body_limit_bytes`
|
||||||
|
|
||||||
|
RSS is not fetched unless `rss_url` is configured. URL override maps are intended
|
||||||
|
for tests and upstream URL changes; the default driver configuration owns the
|
||||||
|
current Day 1-3 SPC product URLs.
|
||||||
|
|
||||||
|
## Upstream Products Used
|
||||||
|
|
||||||
|
The source fetches nine required GeoJSON products every poll:
|
||||||
|
|
||||||
|
- Day 1 categorical, tornado, hail, and wind
|
||||||
|
- Day 2 categorical, tornado, hail, and wind
|
||||||
|
- Day 3 categorical
|
||||||
|
|
||||||
|
It also fetches three required print pages:
|
||||||
|
|
||||||
|
- Day 1 convective outlook print page
|
||||||
|
- Day 2 convective outlook print page
|
||||||
|
- Day 3 convective outlook print page
|
||||||
|
|
||||||
|
GeoJSON products are authoritative for outlook polygons, valid windows, issue
|
||||||
|
times, labels, and severity rank. Print pages are authoritative for run-level
|
||||||
|
day discussion headline, summary, and discussion text.
|
||||||
|
|
||||||
|
## Accept Headers
|
||||||
|
|
||||||
|
GeoJSON requests use:
|
||||||
|
|
||||||
|
```text
|
||||||
|
application/geo+json, application/json
|
||||||
|
```
|
||||||
|
|
||||||
|
Print-page requests use:
|
||||||
|
|
||||||
|
```text
|
||||||
|
text/html, application/xhtml+xml
|
||||||
|
```
|
||||||
|
|
||||||
|
RSS requests, when configured, use:
|
||||||
|
|
||||||
|
```text
|
||||||
|
application/rss+xml, application/xml, text/xml
|
||||||
|
```
|
||||||
|
|
||||||
|
## Polling And Raw Events
|
||||||
|
|
||||||
|
The source polls all required GeoJSON and print-page products as one bundle. If
|
||||||
|
any required request fails or returns a non-2xx response, the poll returns an
|
||||||
|
error and emits no partial event.
|
||||||
|
|
||||||
|
The raw payload contains fetched bodies plus configured location metadata and
|
||||||
|
per-product metadata. The source parses only the timestamp metadata needed for
|
||||||
|
event effective time selection; canonical mapping belongs to the normalizer.
|
||||||
|
|
||||||
|
The source emits no event when a complete fetched bundle is unchanged from the
|
||||||
|
previous successful poll. It does this with a source-local hash of the fetched
|
||||||
|
document bodies.
|
||||||
|
|
||||||
|
## Time Handling
|
||||||
|
|
||||||
|
Raw source `effective_at` prefers:
|
||||||
|
|
||||||
|
1. the latest valid GeoJSON `ISSUE_ISO`;
|
||||||
|
2. the latest print-page `Updated:` timestamp;
|
||||||
|
3. RSS `lastBuildDate` when RSS was fetched and parseable;
|
||||||
|
4. fetch time.
|
||||||
|
|
||||||
|
The normalizer sets canonical `asOf` and normalized event `effective_at` from
|
||||||
|
the latest valid GeoJSON issue time across the complete raw bundle, including
|
||||||
|
empty no-risk placeholders, with fallback to print-page update time and then the
|
||||||
|
incoming event envelope.
|
||||||
|
|
||||||
|
## Mapping Notes
|
||||||
|
|
||||||
|
The raw source fetches and envelopes the complete SPC bundle. The normalizer
|
||||||
|
decodes every configured GeoJSON product, skips empty no-risk
|
||||||
|
`GeometryCollection` placeholders, and emits canonical outlooks only when the
|
||||||
|
configured point is inside or on the boundary of a real feature. Products are
|
||||||
|
ordered by day, then categorical, tornado, hail, and wind. Retained feature order
|
||||||
|
is preserved within each product.
|
||||||
|
|
||||||
|
The normalizer computes `containsLocation` with the configured latitude and
|
||||||
|
longitude against compact GeoJSON `Polygon` or `MultiPolygon` geometry.
|
||||||
|
Coordinates use GeoJSON order, `[longitude, latitude]`, and boundary points
|
||||||
|
count as contained.
|
||||||
|
|
||||||
|
Canonical outlook runs are emitted even when no polygons apply locally. In that
|
||||||
|
case the payload contains empty `outlooks` and `discussions` arrays.
|
||||||
|
|
||||||
|
Print-page prose is represented as run-level day discussions. Discussions are
|
||||||
|
included only for days represented by at least one retained outlook. Multiple
|
||||||
|
retained outlook types for the same day share one discussion entry.
|
||||||
|
|
||||||
|
For downstream current-state and historical supersession guidance, see the
|
||||||
|
[event wire contract](events.md#spc-outlook-supersession).
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Constructor validation failures stop daemon startup. Polling failures are
|
||||||
|
returned to the scheduler and emit no event for that poll.
|
||||||
|
|
||||||
|
Normalization fails when required GeoJSON timestamps, labels, geometry, or
|
||||||
|
configured coordinates are missing or invalid. Print-page extraction failures
|
||||||
|
also fail normalization because print pages are required inputs.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/providers/spc/*_test.go`
|
||||||
|
- `internal/sources/spc/*_test.go`
|
||||||
|
- `internal/normalizers/spc/*_test.go`
|
||||||
|
- fixtures under `internal/providers/spc/testdata`
|
||||||
101
docs/internal/normalizers.md
Normal file
101
docs/internal/normalizers.md
Normal file
@@ -0,0 +1,101 @@
|
|||||||
|
# Normalizer Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Normalizers convert raw provider events into canonical weather events. They are
|
||||||
|
weather-domain mapping code and should stay independent of runtime wiring,
|
||||||
|
source polling, and sink persistence.
|
||||||
|
|
||||||
|
Detailed package conventions live in `internal/normalizers/doc.go`.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Inputs are raw feed events whose schemas identify provider payload shape.
|
||||||
|
Outputs are canonical feed events using `model` payloads and `weather.*`
|
||||||
|
schemas.
|
||||||
|
|
||||||
|
Current mappings:
|
||||||
|
|
||||||
|
| Raw schema | Canonical schema |
|
||||||
|
| --- | --- |
|
||||||
|
| `raw.nws.observation.v1` | `weather.observation.v1` |
|
||||||
|
| `raw.openmeteo.current.v1` | `weather.observation.v1` |
|
||||||
|
| `raw.openweather.current.v1` | `weather.observation.v1` |
|
||||||
|
| `raw.nws.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `raw.nws.narrative.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `raw.openmeteo.hourly.forecast.v1` | `weather.forecast.v1` |
|
||||||
|
| `raw.nws.forecast_discussion.v1` | `weather.forecast_discussion.v1` |
|
||||||
|
| `raw.nws.weatherstories.v1` | `weather.weather_story.v1` |
|
||||||
|
| `raw.nws.alerts.v1` | `weather.alert.v1` |
|
||||||
|
| `raw.spc.convective_outlook.v1` | `weather.outlook.v2` |
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Normalizers match by `Event.Schema`.
|
||||||
|
- Normalizers decode raw payloads into provider structs.
|
||||||
|
- Normalizers map provider data into canonical `model` payloads.
|
||||||
|
- Normalizers do not fetch network data, read config, route events, or write
|
||||||
|
sinks.
|
||||||
|
- Shared cross-provider behavior belongs in `internal/normalizers/common`.
|
||||||
|
- Provider-specific helper logic shared with sources belongs in
|
||||||
|
`internal/providers/<provider>`.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Normalizers do not read config. They operate only on incoming events.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
Runtime composition creates feedkit's normalize processor with
|
||||||
|
`RequireMatch=false`. Events without a matching normalizer pass through
|
||||||
|
unchanged.
|
||||||
|
|
||||||
|
Weatherfeeder registers normalizers in a stable order:
|
||||||
|
|
||||||
|
1. NWS
|
||||||
|
2. Open-Meteo
|
||||||
|
3. OpenWeather
|
||||||
|
4. SPC
|
||||||
|
|
||||||
|
The current normalizers avoid ambiguous matches by using schema equality.
|
||||||
|
|
||||||
|
The SPC outlook normalizer decodes the raw multi-document bundle, maps
|
||||||
|
location-containing GeoJSON features to canonical outlooks, and adds one
|
||||||
|
run-level print-page discussion per retained outlook day. It preserves compact
|
||||||
|
GeoJSON feature geometry and computes `containsLocation` with
|
||||||
|
`internal/geo.ContainsPoint` using the source-configured point. Boundary points
|
||||||
|
count as contained. Polygons that do not contain the point are omitted from the
|
||||||
|
canonical run.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
Normalizers should be stateless. Shared helpers should be deterministic and free
|
||||||
|
of I/O.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Malformed required raw payload data should produce contextual errors from the
|
||||||
|
owning normalizer. Successful normalization validates the output event before it
|
||||||
|
continues through the pipeline.
|
||||||
|
|
||||||
|
`internal/normalizers/common.Finalize` preserves the input event envelope except
|
||||||
|
for schema, payload, and effective time. It also rounds canonical float values
|
||||||
|
to four digits after the decimal point.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/normalizers/builtins_test.go`
|
||||||
|
- provider normalizer tests under `internal/normalizers/nws`
|
||||||
|
- provider normalizer tests under `internal/normalizers/openmeteo`
|
||||||
|
- provider normalizer tests under `internal/normalizers/openweather`
|
||||||
|
- provider normalizer tests under `internal/normalizers/spc`
|
||||||
|
- common helper tests under `internal/normalizers/common`
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Match by schema constants from `standards`.
|
||||||
|
- Preserve the event envelope except for intentional canonical changes.
|
||||||
|
- Produce canonical payload structs from `model`.
|
||||||
|
- Validate normalized events before returning them.
|
||||||
|
- Keep normalizers independent of sources, sinks, config loading, and runtime
|
||||||
|
composition.
|
||||||
133
docs/internal/postgres-sink.md
Normal file
133
docs/internal/postgres-sink.md
Normal file
@@ -0,0 +1,133 @@
|
|||||||
|
# Postgres Sink Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`internal/sinks/postgres` defines weatherfeeder's canonical-event-to-Postgres
|
||||||
|
mapping. It supplies a schema definition and mapper to feedkit's generic
|
||||||
|
Postgres sink.
|
||||||
|
|
||||||
|
The consumer-facing table contract is
|
||||||
|
[`docs/integrations/postgres.md`](../integrations/postgres.md). This document
|
||||||
|
describes the internal ownership boundary.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Inputs are canonical feed events. The mapper currently handles these schemas:
|
||||||
|
|
||||||
|
- `weather.observation.v1`
|
||||||
|
- `weather.forecast.v1`
|
||||||
|
- `weather.forecast_discussion.v1`
|
||||||
|
- `weather.weather_story.v1`
|
||||||
|
- `weather.alert.v1`
|
||||||
|
- `weather.outlook.v2`
|
||||||
|
|
||||||
|
Outputs are feedkit `PostgresWrite` values for weatherfeeder-owned tables.
|
||||||
|
Unsupported schemas produce no writes and no error.
|
||||||
|
|
||||||
|
Outlook runs are written to `outlook_runs`, retained local polygons are written
|
||||||
|
to `outlooks`, and run-level day discussions are written to
|
||||||
|
`outlook_discussions`. The parent run row stores `outlook_count` and
|
||||||
|
`discussion_count`.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Weatherfeeder owns table definitions in `schema.go`.
|
||||||
|
- Weatherfeeder owns canonical payload mapping in `map.go`.
|
||||||
|
- Feedkit owns database opening, table and index creation, transactions,
|
||||||
|
inserts, context-aware consumption, and prune execution.
|
||||||
|
- Postgres mapping consumes canonical events only. It should not understand raw
|
||||||
|
provider schemas.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Weatherfeeder registers the `postgres` sink by passing `PostgresSchema()` to
|
||||||
|
feedkit. Feedkit parses sink params:
|
||||||
|
|
||||||
|
- `uri`
|
||||||
|
- `username`
|
||||||
|
- `password`
|
||||||
|
- `prune`, optional duration
|
||||||
|
|
||||||
|
Weatherfeeder-owned mapper code does not read config directly.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
The runtime registers the sink with:
|
||||||
|
|
||||||
|
```go
|
||||||
|
sinkReg.Register("postgres", fksinks.PostgresFactory(wfpgsink.PostgresSchema()))
|
||||||
|
```
|
||||||
|
|
||||||
|
Feedkit validates events at the sink boundary, calls the weatherfeeder mapper,
|
||||||
|
validates writes against the compiled schema, inserts rows in a transaction, and
|
||||||
|
optionally prunes rows older than the configured window.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
The mapper is stateless. Durable state is stored in Postgres through feedkit's
|
||||||
|
sink implementation.
|
||||||
|
|
||||||
|
## Mapping Rules
|
||||||
|
|
||||||
|
Parent rows preserve event envelope fields where the table supports them:
|
||||||
|
|
||||||
|
- `event_id`
|
||||||
|
- `event_kind`
|
||||||
|
- `event_source`
|
||||||
|
- `event_schema`
|
||||||
|
- `event_emitted_at`
|
||||||
|
- `event_effective_at`
|
||||||
|
|
||||||
|
Child rows use positional indexes to preserve canonical array order:
|
||||||
|
|
||||||
|
- `weather_index`
|
||||||
|
- `period_index`
|
||||||
|
- `message_index`
|
||||||
|
- `story_index`
|
||||||
|
- `alert_index`
|
||||||
|
- `reference_index`
|
||||||
|
- `outlook_index`
|
||||||
|
- `discussion_index`
|
||||||
|
|
||||||
|
Required canonical fields are validated before writes are returned:
|
||||||
|
|
||||||
|
- observations require `timestamp`;
|
||||||
|
- forecasts require `issuedAt` and `product`, and each period requires
|
||||||
|
`startTime` and `endTime`;
|
||||||
|
- forecast discussions require `issuedAt` and `product`;
|
||||||
|
- weather story runs require `asOf`, and each story requires `startTime`,
|
||||||
|
`endTime`, and `updatedAt`;
|
||||||
|
- alert runs require `asOf`, and each alert requires `id`;
|
||||||
|
- outlook runs require `asOf`, and each outlook requires `id`, `provider`,
|
||||||
|
`product`, `day`, `outlookType`, `label`, `validFrom`, `validTo`, `issuedAt`,
|
||||||
|
`expiresAt`, `containsLocation: true`, and `geometry`;
|
||||||
|
- outlook discussions require day `1`, `2`, or `3`; at least one of
|
||||||
|
`headline`, `summary`, or `discussion`; and no duplicate discussion day in the
|
||||||
|
same run.
|
||||||
|
|
||||||
|
Nullable canonical values are converted to SQL nulls by mapper helpers.
|
||||||
|
Observation present-weather raw values and outlook geometry values are stored as
|
||||||
|
compact JSON text.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Payload decode failures, missing required fields, invalid compact JSON values,
|
||||||
|
or schema/write mismatches return errors to feedkit's sink. Feedkit rolls back
|
||||||
|
the transaction when a write fails.
|
||||||
|
|
||||||
|
Unsupported canonical schemas are ignored by this mapper so other routed events
|
||||||
|
can use different sinks without Postgres-specific failures.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sinks/postgres/schema_test.go`
|
||||||
|
- `internal/sinks/postgres/map_test.go`
|
||||||
|
- feedkit Postgres sink tests when changing generic sink behavior assumptions
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Persist only canonical schemas.
|
||||||
|
- Preserve event envelope fields in parent rows.
|
||||||
|
- Preserve array order with child positional indexes.
|
||||||
|
- Validate required fields before writing.
|
||||||
|
- Keep table-contract docs synchronized with schema and mapper changes.
|
||||||
125
docs/internal/runtime.md
Normal file
125
docs/internal/runtime.md
Normal file
@@ -0,0 +1,125 @@
|
|||||||
|
# Runtime Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
`cmd/weatherfeeder` wires the daemon together. It owns process setup and runtime
|
||||||
|
composition; provider mapping, source fetching details, and sink persistence
|
||||||
|
rules stay in their owning packages.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
The executable reads `config.yml` from the current working directory through
|
||||||
|
feedkit config loading. It builds configured sources, scheduler jobs, sinks, and
|
||||||
|
routes, then runs source polling and sink dispatch until shutdown.
|
||||||
|
|
||||||
|
Inputs are configured source polls. Outputs are feed events delivered to the
|
||||||
|
configured sinks.
|
||||||
|
|
||||||
|
## Runtime Flow
|
||||||
|
|
||||||
|
The implemented flow is:
|
||||||
|
|
||||||
|
1. load `config.yml`;
|
||||||
|
2. register weatherfeeder source drivers;
|
||||||
|
3. register feedkit built-in sinks and the weatherfeeder Postgres sink;
|
||||||
|
4. build source inputs and scheduler jobs;
|
||||||
|
5. validate configured expected kinds against source-advertised kinds;
|
||||||
|
6. build sinks and compile routes;
|
||||||
|
7. run the processor chain `normalize`, then `dedupe`;
|
||||||
|
8. run the scheduler and dispatcher concurrently;
|
||||||
|
9. shut down on signal or fatal scheduler/dispatcher error.
|
||||||
|
|
||||||
|
The in-process event channel is buffered to 256 events. The dedupe processor is
|
||||||
|
bounded by `dedupeMaxEntries`, currently 2048.
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Runtime composition belongs in `cmd/weatherfeeder/main.go`.
|
||||||
|
- Source driver behavior belongs under `internal/sources`.
|
||||||
|
- Normalizer behavior belongs under `internal/normalizers`.
|
||||||
|
- Canonical payloads and schema strings belong in `model` and `standards`.
|
||||||
|
- Postgres mapping belongs under `internal/sinks/postgres`.
|
||||||
|
|
||||||
|
`cmd/weatherfeeder` should stay thin and should not contain provider parsing,
|
||||||
|
canonical mapping, or table-mapping rules.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Runtime wiring consumes the feedkit top-level config sections:
|
||||||
|
|
||||||
|
- `sources`: source driver selection, source name, mode, cadence, expected kinds,
|
||||||
|
and driver params;
|
||||||
|
- `sinks`: sink driver selection, sink name, and sink params;
|
||||||
|
- `routes`: event-kind routing to named sinks.
|
||||||
|
|
||||||
|
The executable does not expose CLI flags or config path discovery.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
Runtime composition uses feedkit for:
|
||||||
|
|
||||||
|
- config loading;
|
||||||
|
- source registry and expected-kind validation;
|
||||||
|
- scheduler job construction;
|
||||||
|
- processor registry and chain execution;
|
||||||
|
- normalization and dedupe processors;
|
||||||
|
- sink registry and built-in sinks;
|
||||||
|
- route compilation and dispatch.
|
||||||
|
|
||||||
|
Weatherfeeder registers its own source drivers and its Postgres schema mapper.
|
||||||
|
|
||||||
|
Responsibility split:
|
||||||
|
|
||||||
|
| Runtime concern | Owner |
|
||||||
|
| --- | --- |
|
||||||
|
| Config loading and generic validation | feedkit |
|
||||||
|
| Source, processor, and sink registries | feedkit mechanics; weatherfeeder registrations |
|
||||||
|
| Source polling and stream supervision | feedkit scheduler |
|
||||||
|
| Raw weather data fetching | weatherfeeder source adapters |
|
||||||
|
| Normalizer execution order and pass-through behavior | feedkit normalize processor |
|
||||||
|
| Weather raw-to-canonical mapping | weatherfeeder normalizers |
|
||||||
|
| Dedupe mechanics | feedkit dedupe processor |
|
||||||
|
| Route compilation and sink fanout | feedkit dispatch |
|
||||||
|
| Weather Postgres table shape and row mapping | weatherfeeder Postgres adapter |
|
||||||
|
| Postgres connection, DDL, inserts, transactions, and pruning | feedkit Postgres sink |
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
Weatherfeeder-owned runtime state is in process:
|
||||||
|
|
||||||
|
- event channel contents;
|
||||||
|
- the bounded dedupe key set;
|
||||||
|
- source instances and their in-memory unchanged-content state;
|
||||||
|
- scheduler and dispatcher goroutines.
|
||||||
|
|
||||||
|
There is no weatherfeeder-owned durable scheduler state, checkpoint, replay log,
|
||||||
|
or resume marker. Durable persistence is owned by configured external sinks.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Startup failures are fatal and include context such as config index, source name,
|
||||||
|
sink name, driver name, or the operation that failed.
|
||||||
|
|
||||||
|
At runtime, scheduler and dispatcher errors are sent to a shared error channel.
|
||||||
|
Context cancellation and deadline errors are treated as normal shutdown. Any
|
||||||
|
other scheduler or dispatcher error is logged as fatal and cancels the process
|
||||||
|
context.
|
||||||
|
|
||||||
|
The daemon handles `os.Interrupt` and `SIGTERM` with `signal.NotifyContext`.
|
||||||
|
After both runtime goroutines return, it logs `shutdown complete`.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `cmd/weatherfeeder/main_test.go`
|
||||||
|
- source registry tests under `internal/sources`
|
||||||
|
- normalizer registration tests under `internal/normalizers`
|
||||||
|
- feedkit scheduler, processor, dispatch, and sink tests when changing runtime
|
||||||
|
infrastructure usage
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Keep normalization before dedupe.
|
||||||
|
- Keep queue sizes and dedupe bounds explicit.
|
||||||
|
- Preserve context-aware shutdown.
|
||||||
|
- Keep runtime wiring separate from domain mapping and persistence rules.
|
||||||
|
- Keep startup validation failures loud and contextual.
|
||||||
122
docs/internal/sources.md
Normal file
122
docs/internal/sources.md
Normal file
@@ -0,0 +1,122 @@
|
|||||||
|
# Source Internals
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Source packages poll upstream weather providers and emit raw feed events. They
|
||||||
|
are adapters, not canonical mappers.
|
||||||
|
|
||||||
|
Sources should decode only the metadata needed for event identity, effective
|
||||||
|
time, and routing policy. Full provider payload interpretation belongs in
|
||||||
|
normalizers.
|
||||||
|
|
||||||
|
## Inputs And Outputs
|
||||||
|
|
||||||
|
Inputs are feedkit `config.SourceConfig` values and upstream HTTP responses.
|
||||||
|
Outputs are feed events whose payloads are raw provider JSON and whose schemas
|
||||||
|
come from `standards`.
|
||||||
|
|
||||||
|
Current drivers:
|
||||||
|
|
||||||
|
| Driver | Kind | Raw schema |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `nws_observation` | `observation` | `raw.nws.observation.v1` |
|
||||||
|
| `nws_alerts` | `alert` | `raw.nws.alerts.v1` |
|
||||||
|
| `nws_forecast_hourly` | `forecast` | `raw.nws.hourly.forecast.v1` |
|
||||||
|
| `nws_forecast_narrative` | `forecast` | `raw.nws.narrative.forecast.v1` |
|
||||||
|
| `nws_forecast_discussion` | `forecast_discussion` | `raw.nws.forecast_discussion.v1` |
|
||||||
|
| `nws_weatherstories` | `weather_story` | `raw.nws.weatherstories.v1` |
|
||||||
|
| `openmeteo_observation` | `observation` | `raw.openmeteo.current.v1` |
|
||||||
|
| `openmeteo_forecast` | `forecast` | `raw.openmeteo.hourly.forecast.v1` |
|
||||||
|
| `openweather_observation` | `observation` | `raw.openweather.current.v1` |
|
||||||
|
| `spc_convective_outlook` | `outlook` | `raw.spc.convective_outlook.v1` |
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
- Source constructors validate source-specific params.
|
||||||
|
- Sources use feedkit HTTP helpers for HTTP polling.
|
||||||
|
- Sources emit raw events and should not build canonical `model` payloads.
|
||||||
|
- Provider helper packages under `internal/providers/<provider>` hold shared
|
||||||
|
parsing and validation helpers.
|
||||||
|
- Registration is centralized in `internal/sources/builtins.go`.
|
||||||
|
|
||||||
|
## Config Fields Used
|
||||||
|
|
||||||
|
Most source drivers use feedkit `HTTPSource`.
|
||||||
|
|
||||||
|
Required params:
|
||||||
|
|
||||||
|
- `url`
|
||||||
|
- `user_agent`
|
||||||
|
|
||||||
|
Optional params:
|
||||||
|
|
||||||
|
- `conditional`, default `true`;
|
||||||
|
- `http_timeout`;
|
||||||
|
- `http_response_body_limit_bytes`.
|
||||||
|
|
||||||
|
OpenWeather observation sources additionally require the configured URL to use
|
||||||
|
metric units. This is enforced by `internal/providers/openweather`.
|
||||||
|
|
||||||
|
The SPC convective outlook source is a multi-document poll source rather than a
|
||||||
|
single-URL `HTTPSource`. It requires `latitude`, `longitude`, and `user_agent`;
|
||||||
|
accepts optional `location_id`, `location_name`, `geojson_urls`,
|
||||||
|
`discussion_urls`, and `rss_url`; and supports `http_timeout` and
|
||||||
|
`http_response_body_limit_bytes`.
|
||||||
|
|
||||||
|
Source-level `kinds`, when configured, are validated against the source's
|
||||||
|
advertised `Kinds()`.
|
||||||
|
|
||||||
|
## External Adapters Used
|
||||||
|
|
||||||
|
Most sources use feedkit's HTTP helper for:
|
||||||
|
|
||||||
|
- request construction;
|
||||||
|
- `User-Agent` and `Accept` headers;
|
||||||
|
- optional conditional GET validators;
|
||||||
|
- response body size limits;
|
||||||
|
- JSON raw-message fetches.
|
||||||
|
|
||||||
|
NWS helpers parse NWS timestamps. Open-Meteo helpers parse provider-local times
|
||||||
|
with timezone or UTC-offset data. OpenWeather helpers enforce metric-unit URLs.
|
||||||
|
SPC helpers define Day 1-3 product metadata, parse GeoJSON timestamps, extract
|
||||||
|
cleaned print-page discussion text, and parse optional RSS metadata.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
HTTP conditional validators are held in each single-document HTTP source
|
||||||
|
instance. They are not persisted across process restarts. The SPC outlook source
|
||||||
|
keeps only a source-local hash of the most recent complete required product
|
||||||
|
bundle and emits no event when a later complete bundle is unchanged.
|
||||||
|
|
||||||
|
## Failure Behavior
|
||||||
|
|
||||||
|
Constructor failures are returned during startup and stop the daemon. Polling
|
||||||
|
failures are returned to the scheduler.
|
||||||
|
|
||||||
|
If a source cannot decode minimal metadata from an otherwise successful upstream
|
||||||
|
response, it still emits the raw event when possible. The event then falls back
|
||||||
|
to default ID/effective-time behavior from feedkit source helpers.
|
||||||
|
|
||||||
|
Unchanged conditional responses return no events and no error.
|
||||||
|
|
||||||
|
The SPC outlook source fetches all configured Day 1-3 GeoJSON products and print
|
||||||
|
pages atomically. If any required GeoJSON or print-page request fails, the poll
|
||||||
|
returns an error and emits no partial bundle. RSS is fetched only when `rss_url`
|
||||||
|
is configured.
|
||||||
|
|
||||||
|
## Tests To Inspect
|
||||||
|
|
||||||
|
- `internal/sources/builtins_test.go`
|
||||||
|
- provider source tests under `internal/sources/nws`
|
||||||
|
- provider source tests under `internal/sources/openmeteo`
|
||||||
|
- provider source tests under `internal/sources/openweather`
|
||||||
|
- provider source tests under `internal/sources/spc`
|
||||||
|
- provider helper tests under `internal/providers`
|
||||||
|
|
||||||
|
## Invariants
|
||||||
|
|
||||||
|
- Emit raw provider schemas from `standards`.
|
||||||
|
- Keep provider-to-canonical mapping out of sources.
|
||||||
|
- Keep HTTP behavior context-aware.
|
||||||
|
- Keep driver registration explicit and stable.
|
||||||
|
- Keep source tests independent of live upstream services.
|
||||||
163
docs/operations.md
Normal file
163
docs/operations.md
Normal file
@@ -0,0 +1,163 @@
|
|||||||
|
# Operations
|
||||||
|
|
||||||
|
This document describes how to run and observe the `weatherfeeder` daemon in its
|
||||||
|
current form. For configuration syntax, see [configuration](config.md). For the
|
||||||
|
CLI surface, see [CLI reference](cli.md).
|
||||||
|
|
||||||
|
## Normal Workflow
|
||||||
|
|
||||||
|
1. Prepare `config.yml` in the process working directory.
|
||||||
|
2. Start the daemon with `./weatherfeeder` or `go run .` from
|
||||||
|
`cmd/weatherfeeder`.
|
||||||
|
3. Watch stderr logs for startup or runtime errors.
|
||||||
|
4. Consume events from the configured sinks.
|
||||||
|
5. Stop the process with `Ctrl-C` or `SIGTERM`.
|
||||||
|
|
||||||
|
The daemon has no admin subcommands and no runtime reload command. Change the
|
||||||
|
config file and restart the process to apply configuration changes.
|
||||||
|
|
||||||
|
Maintained copyable configs are available under [`examples/`](../examples/).
|
||||||
|
|
||||||
|
## Runtime Lifecycle
|
||||||
|
|
||||||
|
On startup, `weatherfeeder`:
|
||||||
|
|
||||||
|
1. loads `config.yml` from the current working directory;
|
||||||
|
2. registers built-in source drivers;
|
||||||
|
3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
|
||||||
|
4. builds sources and validates configured `kinds` against source metadata;
|
||||||
|
5. builds sinks and compiles routes;
|
||||||
|
6. starts the scheduler and dispatcher;
|
||||||
|
7. processes events through normalization, then in-memory dedupe;
|
||||||
|
8. routes processed events to configured sinks.
|
||||||
|
|
||||||
|
Startup errors are fatal and terminate the process. Runtime poll, pipeline, and
|
||||||
|
sink errors are logged and the process continues unless the scheduler or
|
||||||
|
dispatcher returns a fatal error.
|
||||||
|
|
||||||
|
## Logs
|
||||||
|
|
||||||
|
The process uses the Go standard logger with date, time, and microseconds. Logs
|
||||||
|
go to stderr.
|
||||||
|
|
||||||
|
Common log prefixes:
|
||||||
|
|
||||||
|
| Prefix | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| `config load failed` | `config.yml` could not be read, parsed, or validated. |
|
||||||
|
| `build source failed` | A source driver or its params are invalid. |
|
||||||
|
| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. |
|
||||||
|
| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. |
|
||||||
|
| `compile routes failed` | Routes reference invalid sinks or kinds. |
|
||||||
|
| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. |
|
||||||
|
| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. |
|
||||||
|
| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. |
|
||||||
|
| `shutdown complete` | Scheduler and dispatcher have exited. |
|
||||||
|
|
||||||
|
## Scheduling And Polling
|
||||||
|
|
||||||
|
Current weatherfeeder sources are polling sources. Each source uses its
|
||||||
|
configured `every` interval. The scheduler applies jitter before the first poll
|
||||||
|
and before each interval tick. If no jitter is configured in code, feedkit uses
|
||||||
|
`min(every/10, 30s)`, capped at half the interval.
|
||||||
|
|
||||||
|
Poll failures are logged and do not stop the daemon. A failed poll emits no
|
||||||
|
events for that source until a subsequent poll succeeds.
|
||||||
|
|
||||||
|
## Unchanged Upstream Content
|
||||||
|
|
||||||
|
Most sources use feedkit's single-document HTTP polling helper. By default,
|
||||||
|
`params.conditional` is `true` for those sources, so the helper keeps ETag and
|
||||||
|
Last-Modified validators in memory for each source instance.
|
||||||
|
|
||||||
|
If the upstream returns `304 Not Modified`, the source emits no events for that
|
||||||
|
poll. Validator state is in memory only; restarting the process starts with no
|
||||||
|
cached validators.
|
||||||
|
|
||||||
|
The SPC convective outlook source polls multiple required documents as one
|
||||||
|
bundle. It emits no event when a later complete bundle has the same fetched
|
||||||
|
document bodies as the previous successful poll.
|
||||||
|
|
||||||
|
## Processing And Dedupe
|
||||||
|
|
||||||
|
Every event passes through normalization first and dedupe second.
|
||||||
|
|
||||||
|
Normalizers match raw source schemas and produce versioned canonical `weather.*`
|
||||||
|
payloads. If an event has no matching normalizer, the normalize processor passes
|
||||||
|
it through unchanged.
|
||||||
|
|
||||||
|
Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs.
|
||||||
|
Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts
|
||||||
|
with an empty dedupe set.
|
||||||
|
|
||||||
|
## Routing And Sink Fanout
|
||||||
|
|
||||||
|
Routes choose sinks by event kind. If `routes` is omitted, every sink receives
|
||||||
|
every event kind. If a route omits `kinds`, that route also matches all kinds.
|
||||||
|
|
||||||
|
The dispatcher creates one queue and one worker goroutine per sink. The default
|
||||||
|
per-sink queue size is 64. `weatherfeeder` does not currently expose config
|
||||||
|
fields for sink queue size, enqueue timeout, or consume timeout.
|
||||||
|
|
||||||
|
Sink errors are logged per event. A sink failure does not stop other sinks from
|
||||||
|
receiving the same event.
|
||||||
|
|
||||||
|
## Sink Behavior
|
||||||
|
|
||||||
|
### stdout
|
||||||
|
|
||||||
|
The stdout sink validates each event and prints one JSON object per line to
|
||||||
|
stdout. This is useful for local inspection and log forwarding.
|
||||||
|
|
||||||
|
### NATS
|
||||||
|
|
||||||
|
The NATS sink connects lazily on the first event, reuses the connection while it
|
||||||
|
is open, and publishes each event as JSON to the configured subject. Connection,
|
||||||
|
marshal, and publish failures are logged by the dispatch worker.
|
||||||
|
|
||||||
|
### Postgres
|
||||||
|
|
||||||
|
The Postgres sink opens the database during startup. It creates missing tables
|
||||||
|
and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`.
|
||||||
|
It does not modify existing table definitions.
|
||||||
|
|
||||||
|
Each mapped canonical event is written in one transaction. If `params.prune` is
|
||||||
|
set, the sink deletes rows older than `now - prune` from every weatherfeeder
|
||||||
|
table in that same transaction. See the
|
||||||
|
[Postgres table contract](integrations/postgres.md).
|
||||||
|
|
||||||
|
## State And Recovery
|
||||||
|
|
||||||
|
`weatherfeeder` keeps only runtime state in process memory:
|
||||||
|
|
||||||
|
- scheduler goroutines and timers;
|
||||||
|
- HTTP conditional request validators;
|
||||||
|
- source-local unchanged-content state;
|
||||||
|
- event channel buffers;
|
||||||
|
- per-sink fanout queues;
|
||||||
|
- the dedupe ID set.
|
||||||
|
|
||||||
|
Durable state is external sink state: NATS broker state outside this process and
|
||||||
|
Postgres tables managed by the configured database.
|
||||||
|
|
||||||
|
There is no internal checkpoint, replay log, or resume marker. To recover from a
|
||||||
|
process failure, fix the underlying issue and restart the daemon from a working
|
||||||
|
directory containing the desired `config.yml`.
|
||||||
|
|
||||||
|
## Shutdown
|
||||||
|
|
||||||
|
`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the
|
||||||
|
shared context is canceled. Scheduler jobs stop polling, dispatch workers stop,
|
||||||
|
and the process logs `shutdown complete`.
|
||||||
|
|
||||||
|
Queued sink work may be dropped when shutdown context cancellation reaches the
|
||||||
|
fanout workers. Use external sink durability, such as Postgres or broker
|
||||||
|
retention, for durable downstream state.
|
||||||
|
|
||||||
|
## Caveats
|
||||||
|
|
||||||
|
- There is no health-check endpoint.
|
||||||
|
- There is no runtime config reload.
|
||||||
|
- There are no built-in metrics.
|
||||||
|
- Source conditional request state and dedupe state are reset by restart.
|
||||||
|
- Existing Postgres schemas are not migrated automatically.
|
||||||
225
docs/policy/architecture.md
Normal file
225
docs/policy/architecture.md
Normal file
@@ -0,0 +1,225 @@
|
|||||||
|
# Architecture Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document defines `weatherfeeder`'s development architecture and invariants for maintainers and LLM coding agents. It describes how the implemented system is built and how subsequent changes should preserve its boundaries.
|
||||||
|
|
||||||
|
This is an inward-facing policy document. User-facing wire contracts belong in
|
||||||
|
[`docs/integrations/events.md`](../integrations/events.md), and roadmap items
|
||||||
|
belong under [`docs/roadmap/`](../roadmap/).
|
||||||
|
|
||||||
|
## Project Shape
|
||||||
|
|
||||||
|
`weatherfeeder` is a config-driven Go daemon that polls upstream weather providers, emits feed events, normalizes provider-specific raw payloads into canonical weather payloads, and routes those events to configured sinks.
|
||||||
|
|
||||||
|
The implemented runtime flow is:
|
||||||
|
|
||||||
|
1. `cmd/weatherfeeder` loads `config.yml` from the working directory.
|
||||||
|
2. Source drivers are built through the source registry.
|
||||||
|
3. Feedkit scheduler jobs poll sources and publish raw events onto an in-process event channel.
|
||||||
|
4. A pipeline runs normalization, then in-memory dedupe.
|
||||||
|
5. The dispatcher routes processed events to configured sinks by event kind.
|
||||||
|
6. Sinks consume events independently through feedkit fanout workers.
|
||||||
|
|
||||||
|
Canonical payload structs live in `model`. Schema identifiers and cross-provider wire conventions live in `standards`. Source adapters live under `internal/sources`. Normalizers live under `internal/normalizers`. Provider-specific parsing helpers shared by sources and normalizers live under `internal/providers`. Sink-specific persistence mapping lives under `internal/sinks`.
|
||||||
|
|
||||||
|
## Architecture Style
|
||||||
|
|
||||||
|
`weatherfeeder` uses a pragmatic ports-and-adapters architecture rather than a
|
||||||
|
formal framework. Provider APIs, config loading, scheduling, dispatch, and sinks
|
||||||
|
sit outside the weather domain model and normalization rules.
|
||||||
|
|
||||||
|
The implementation style is:
|
||||||
|
|
||||||
|
- Pipeline-oriented: events flow from source polling through normalization,
|
||||||
|
dedupe, routing, and sink fanout.
|
||||||
|
- Schema-routed: normalizers select raw payloads by explicit schema strings, not
|
||||||
|
source names or configured routes.
|
||||||
|
- Provider-isolated: NWS, Open-Meteo, OpenWeather, and SPC quirks stay in
|
||||||
|
provider-specific source, provider-helper, and normalizer packages.
|
||||||
|
- Registry-based: built-in source drivers, normalizers, processors, and sinks
|
||||||
|
are assembled explicitly through registries instead of dynamic plugin loading.
|
||||||
|
- Adapter-clean: persistence and external-system details stay behind source and
|
||||||
|
sink adapters, not in `model` or normalizers.
|
||||||
|
- Direct Go: prefer small package-level constructors and straightforward code
|
||||||
|
over broad abstractions.
|
||||||
|
|
||||||
|
## Core Design Principles
|
||||||
|
|
||||||
|
- Hexagonal boundaries: provider APIs, config loading, scheduling, dispatch, and sinks are external mechanisms around the weather domain model and normalization logic.
|
||||||
|
- Raw-to-canonical flow: sources should fetch and envelope raw provider payloads; normalizers should own provider-to-canonical mapping.
|
||||||
|
- Location-focused canonical data: canonical weather events represent data relevant to the configured forecast location or configured provider object; `weatherfeeder` is not a national provider-data archive.
|
||||||
|
- Schema-based routing: normalizers match on event schema, not source name or event kind.
|
||||||
|
- Composable registries: source drivers, normalizers, processors, and sinks are assembled explicitly through registries.
|
||||||
|
- Bounded concurrency: scheduling and sink fanout are concurrent, but the application should keep queues, goroutine ownership, logging, and cancellation behavior visible.
|
||||||
|
- Standard-library-first: use the Go standard library unless a narrow dependency materially improves maintainability or interoperability.
|
||||||
|
- Current-behavior docs: outside roadmap files, document only implemented behavior.
|
||||||
|
|
||||||
|
## Architectural Boundaries
|
||||||
|
|
||||||
|
Core/domain logic:
|
||||||
|
|
||||||
|
- `model` defines canonical payload structs and JSON field names.
|
||||||
|
- `standards` defines schema strings, versioning conventions, WMO constants, and shared wire policy.
|
||||||
|
- Normalizer mapping code is domain logic and should stay independent of CLI setup, filesystem paths, sink details, and runtime orchestration.
|
||||||
|
|
||||||
|
Adapters:
|
||||||
|
|
||||||
|
- Source adapters under `internal/sources/<provider>` fetch upstream data and produce raw feed events.
|
||||||
|
- Sink adapters under `internal/sinks/<sink>` map canonical events to external systems.
|
||||||
|
- Provider helper packages under `internal/providers/<provider>` contain pure parsing or provider-specific helper logic shared by sources and normalizers.
|
||||||
|
|
||||||
|
Runtime composition:
|
||||||
|
|
||||||
|
- `cmd/weatherfeeder/main.go` owns process wiring: config load, registry setup, scheduler jobs, processor chain, dispatcher, signal cancellation, and logging.
|
||||||
|
- It should remain thin. Do not move provider mapping or sink persistence rules into `cmd/weatherfeeder`.
|
||||||
|
|
||||||
|
Tests and examples:
|
||||||
|
|
||||||
|
- The sample `cmd/weatherfeeder/config.yml` is executable test input and is load-tested.
|
||||||
|
- Tests should keep exercising package contracts directly rather than relying only on full-daemon execution.
|
||||||
|
|
||||||
|
## Feedkit Boundary
|
||||||
|
|
||||||
|
`feedkit` provides reusable daemon infrastructure. `weatherfeeder` provides the
|
||||||
|
weather-domain adapters, models, schemas, and mapping policy.
|
||||||
|
|
||||||
|
| Area | Feedkit owns | Weatherfeeder owns |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| Config | Generic YAML shape: sources, sinks, routes, modes, cadence, and params. | Driver-specific config rules such as NWS `user_agent`, OpenWeather `units=metric`, and SPC coordinates. |
|
||||||
|
| Events | Domain-agnostic event envelope: ID, kind, source, emitted/effective times, schema, and payload. | Event kind meaning, schema strings, and canonical weather payloads. |
|
||||||
|
| Sources | Source interfaces, registry, expected-kind validation, HTTP helper, and default event ID helper. | NWS/Open-Meteo/OpenWeather/SPC source drivers and raw schema emission. |
|
||||||
|
| Processing | Processor registry, normalize processor, dedupe processor, and pipeline execution. | Weather normalizers and schema-specific raw-to-canonical mapping. |
|
||||||
|
| Dispatch | Route compilation and sink fanout mechanics. | Which weather event kinds are configured and meaningful. |
|
||||||
|
| Sinks | Generic stdout, NATS, and Postgres sink mechanics. | Weather-specific Postgres schema and canonical event-to-row mapping. |
|
||||||
|
|
||||||
|
Do not move weather-domain policy into `feedkit`, and do not duplicate generic
|
||||||
|
daemon mechanics in `weatherfeeder` when feedkit already provides the boundary.
|
||||||
|
|
||||||
|
## Modules Or Processing Steps
|
||||||
|
|
||||||
|
The implemented processing steps are source polling, normalization, dedupe, and sink dispatch.
|
||||||
|
|
||||||
|
Source contract:
|
||||||
|
|
||||||
|
- Build from `feedkit/config.SourceConfig`.
|
||||||
|
- Validate required driver params inside the source constructor.
|
||||||
|
- Advertise emitted event kinds through `Kinds()` when possible.
|
||||||
|
- Emit raw schemas from `standards`.
|
||||||
|
- Decode only minimal metadata needed for event identity and effective time; leave full provider decoding to normalizers.
|
||||||
|
- Respect `context.Context` during network work.
|
||||||
|
|
||||||
|
Normalizer contract:
|
||||||
|
|
||||||
|
- One normalizer type per normalizer file.
|
||||||
|
- Match by `Event.Schema`.
|
||||||
|
- Decode raw payloads into provider structs.
|
||||||
|
- Map to canonical `model` payloads.
|
||||||
|
- Preserve the incoming event envelope except for intentional schema, payload, and effective-time changes.
|
||||||
|
- Use shared helpers in `internal/normalizers/common` for cross-provider behavior.
|
||||||
|
- Follow the detailed normalizer conventions in `internal/normalizers/doc.go`.
|
||||||
|
|
||||||
|
Sink contract:
|
||||||
|
|
||||||
|
- Consume canonical schemas, not provider raw schemas.
|
||||||
|
- Keep sink mapping isolated from normalizers and sources.
|
||||||
|
- Preserve event envelope fields in durable storage where the sink schema supports it.
|
||||||
|
- Validate required canonical fields before writing.
|
||||||
|
|
||||||
|
## State, Inputs, and Outputs
|
||||||
|
|
||||||
|
Inputs are configured source polls. The daemon currently uses feedkit's YAML config model with sources, sinks, and routes.
|
||||||
|
|
||||||
|
Outputs are feed events sent to configured sinks. Implemented sink support comes from feedkit built-ins plus weatherfeeder's Postgres schema mapping. The sample config includes stdout and NATS routes; Postgres is configured as an optional commented example.
|
||||||
|
|
||||||
|
The daemon's own state is in-process:
|
||||||
|
|
||||||
|
- the event channel buffers events during runtime;
|
||||||
|
- the dedupe processor stores a bounded in-memory key set;
|
||||||
|
- source instances may keep HTTP conditional request state through feedkit HTTP source helpers;
|
||||||
|
- scheduler and dispatcher state is not persisted by `weatherfeeder`.
|
||||||
|
|
||||||
|
Durable persistence is an external sink concern. The Postgres table contract is
|
||||||
|
documented in [`docs/integrations/postgres.md`](../integrations/postgres.md);
|
||||||
|
the consumer-facing event contract is documented in
|
||||||
|
[`docs/integrations/events.md`](../integrations/events.md).
|
||||||
|
|
||||||
|
## Configuration and CLI Boundaries
|
||||||
|
|
||||||
|
The implemented executable reads `config.yml` from the current working directory. It does not currently expose CLI flags or config path discovery.
|
||||||
|
|
||||||
|
Configuration shape is owned by feedkit's config package:
|
||||||
|
|
||||||
|
- `sources` define named source drivers, mode, poll cadence, expected kinds, and driver params.
|
||||||
|
- `sinks` define named sink drivers and sink params.
|
||||||
|
- `routes` connect event kinds to sinks.
|
||||||
|
|
||||||
|
Weatherfeeder-specific config policy belongs in source and sink constructors, registry setup, and tests. Do not spread config parsing through domain model or normalizer packages.
|
||||||
|
|
||||||
|
[`docs/config.md`](../config.md) and [`docs/cli.md`](../cli.md) are the
|
||||||
|
canonical user/operator references. This policy should stay architectural and
|
||||||
|
avoid duplicating those references.
|
||||||
|
|
||||||
|
## Errors, Logging, and Diagnostics
|
||||||
|
|
||||||
|
`cmd/weatherfeeder` uses the standard library `log` package with timestamps and microseconds.
|
||||||
|
|
||||||
|
Startup errors are fatal and include config index, source/sink name, driver name, or operation context where available. Runtime scheduler and dispatcher errors are reported through logs; context cancellation and deadline errors are treated as shutdown conditions.
|
||||||
|
|
||||||
|
Normalizers and sink mappers should wrap errors with operation and payload context, for example decode, parse, map, scan, or required-field context. Avoid logging or returning whole upstream payloads by default.
|
||||||
|
|
||||||
|
The daemon handles `os.Interrupt` and `SIGTERM` through `signal.NotifyContext`. Sources, sinks, scheduler jobs, dispatcher, and processors should respect `context.Context`.
|
||||||
|
|
||||||
|
## Testing Expectations
|
||||||
|
|
||||||
|
When changing behavior, inspect or add focused tests in the owning package.
|
||||||
|
|
||||||
|
Expected coverage by change type:
|
||||||
|
|
||||||
|
- Source drivers: constructor behavior, advertised kinds, poll event schema/kind, effective-time policy, unchanged responses, and malformed metadata handling.
|
||||||
|
- Normalizers: schema matching, canonical schema output, key field mapping, effective time, malformed required fields, and wire-shape regressions.
|
||||||
|
- Provider helpers: parsing edge cases and fixtures.
|
||||||
|
- Runtime wiring: config loading, source registry build, scheduler job creation, processor ordering, pass-through behavior, and dedupe behavior.
|
||||||
|
- Postgres sink: schema shape, mapper writes, required-field validation, nullable handling, and compact JSON behavior.
|
||||||
|
- Documentation-sensitive examples: keep sample config loadable.
|
||||||
|
|
||||||
|
Use local test servers and fixtures rather than real upstream services. Full-package tests should remain fast and deterministic.
|
||||||
|
|
||||||
|
## Dependency Policy
|
||||||
|
|
||||||
|
Prefer the Go standard library for parsing, HTTP handling, time handling, logging, and tests where reasonable.
|
||||||
|
|
||||||
|
Existing broad runtime composition is delegated to `feedkit`, which provides config, sources, scheduler, processors, dispatch, and sinks. Keep weatherfeeder-specific code from depending directly on low-level external clients when feedkit or a small adapter can contain that dependency.
|
||||||
|
|
||||||
|
Third-party dependencies should be narrow, justified, and preferably de facto standard for their purpose. YAML parsing through feedkit is an acceptable example. Do not add dependencies for small conveniences, and do not let dependency-specific types leak across package boundaries unless that dependency is the package's explicit contract.
|
||||||
|
|
||||||
|
## Documentation Expectations
|
||||||
|
|
||||||
|
Documentation must follow [`docs/policy/documentation.md`](documentation.md).
|
||||||
|
|
||||||
|
Rules for architecture-related docs:
|
||||||
|
|
||||||
|
- Current-behavior docs must describe implemented behavior only.
|
||||||
|
- Roadmap or speculative work belongs only under `docs/roadmap/`.
|
||||||
|
- Prefer links to canonical docs over repeated reference material.
|
||||||
|
- Update docs in the same change when modifying schemas, config behavior, runtime behavior, adapters, or persistence contracts.
|
||||||
|
|
||||||
|
## Architectural Invariants
|
||||||
|
|
||||||
|
- Keep `cmd/weatherfeeder` as composition code, not domain logic.
|
||||||
|
- Keep source fetching separate from normalization.
|
||||||
|
- Keep normalizers matched by schema constants from `standards`.
|
||||||
|
- Keep canonical payload structs in `model` and treat JSON tags as wire contract.
|
||||||
|
- Keep provider-specific helpers under `internal/providers/<provider>` when shared by sources and normalizers.
|
||||||
|
- Keep cross-provider normalizer helpers pure and deterministic.
|
||||||
|
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
|
||||||
|
- Preserve explicit registry-based extension points for sources and normalizers.
|
||||||
|
- Preserve context-aware shutdown and bounded in-process queues.
|
||||||
|
- Avoid broad dependencies without clear architectural value.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
- `weatherfeeder` is not an HTTP API. API serving belongs to separate consumers such as `weatherapi`.
|
||||||
|
- `weatherfeeder` does not own long-term durable state except through configured external sinks.
|
||||||
|
- `weatherfeeder` does not provide a general plugin runtime; new built-in providers and sinks are registered in code.
|
||||||
|
- Architecture policy is not a CLI, config, or wire-contract reference.
|
||||||
230
docs/policy/development.md
Normal file
230
docs/policy/development.md
Normal file
@@ -0,0 +1,230 @@
|
|||||||
|
# Development Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes how to change `weatherfeeder` safely. It is for
|
||||||
|
maintainers and coding agents working in the repository.
|
||||||
|
|
||||||
|
Use this alongside the [architecture policy](architecture.md). User-facing CLI,
|
||||||
|
configuration, operations, and wire-contract details belong in their canonical
|
||||||
|
docs, not here.
|
||||||
|
|
||||||
|
## Repository Layout
|
||||||
|
|
||||||
|
- `cmd/weatherfeeder/`: executable wiring, sample `config.yml`, and runtime
|
||||||
|
composition tests.
|
||||||
|
- `model/`: canonical weather payload structs. JSON tags are part of the wire
|
||||||
|
contract.
|
||||||
|
- `standards/`: schema strings, versioning conventions, WMO constants, and
|
||||||
|
shared wire-format policy.
|
||||||
|
- `internal/sources/`: source adapters that poll upstream providers and emit raw
|
||||||
|
feed events.
|
||||||
|
- `internal/normalizers/`: raw-to-canonical event transforms.
|
||||||
|
- `internal/providers/`: pure provider helper code shared by sources and
|
||||||
|
normalizers.
|
||||||
|
- `internal/sinks/postgres/`: weatherfeeder-owned Postgres schema and canonical
|
||||||
|
event mapper.
|
||||||
|
- `docs/`: current behavior, consumer guides, integration contracts, policies,
|
||||||
|
and roadmap files.
|
||||||
|
- `examples/`: maintained, copyable configuration examples.
|
||||||
|
|
||||||
|
## Build And Test
|
||||||
|
|
||||||
|
Run the full test suite before committing behavior or documentation changes that
|
||||||
|
depend on code behavior:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./...
|
||||||
|
```
|
||||||
|
|
||||||
|
Use narrower commands while iterating:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go test ./cmd/weatherfeeder
|
||||||
|
go test ./internal/sources/...
|
||||||
|
go test ./internal/normalizers/...
|
||||||
|
go test ./internal/sinks/postgres
|
||||||
|
```
|
||||||
|
|
||||||
|
Format Go code before committing:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w <changed-go-files>
|
||||||
|
```
|
||||||
|
|
||||||
|
Do not require live upstream weather services, NATS, or Postgres for unit tests.
|
||||||
|
Use fixtures, local test servers, and package-level tests.
|
||||||
|
|
||||||
|
## Coding Conventions
|
||||||
|
|
||||||
|
- Keep `cmd/weatherfeeder` focused on composition: config load, registry setup,
|
||||||
|
scheduler jobs, processor chain, dispatch, signal handling, and logging.
|
||||||
|
- Keep source fetching separate from normalizer mapping.
|
||||||
|
- Match normalizers by schema constants from `standards`, not source names.
|
||||||
|
- Keep provider-specific helper code under `internal/providers/<provider>` when
|
||||||
|
both sources and normalizers use it.
|
||||||
|
- Keep cross-provider normalizer helpers pure and deterministic under
|
||||||
|
`internal/normalizers/common`.
|
||||||
|
- Keep sink persistence mapping isolated under `internal/sinks/<sink>`.
|
||||||
|
- Wrap errors with operation context, but do not include whole upstream payloads
|
||||||
|
in errors or logs by default.
|
||||||
|
- Prefer explicit registries and small package-level constructors over hidden
|
||||||
|
global behavior.
|
||||||
|
|
||||||
|
## Architecture-Preserving Changes
|
||||||
|
|
||||||
|
When changing `weatherfeeder`, preserve the split between feedkit
|
||||||
|
infrastructure and weather-domain behavior.
|
||||||
|
|
||||||
|
Do:
|
||||||
|
|
||||||
|
- keep generic scheduling, dispatch, processor, config, and sink mechanics in
|
||||||
|
feedkit;
|
||||||
|
- keep weather provider rules in source adapters, provider helpers, and
|
||||||
|
normalizers;
|
||||||
|
- keep canonical weather payloads in `model` and schema/wire identifiers in
|
||||||
|
`standards`;
|
||||||
|
- keep Postgres table and row mapping under `internal/sinks/postgres`;
|
||||||
|
- use explicit registries for built-in sources and normalizers.
|
||||||
|
|
||||||
|
Do not:
|
||||||
|
|
||||||
|
- move provider parsing, WMO mapping, or canonical weather policy into
|
||||||
|
`cmd/weatherfeeder`;
|
||||||
|
- move weather-specific constants, schemas, or validation rules into feedkit;
|
||||||
|
- put database column metadata or sink-specific tags on canonical model structs;
|
||||||
|
- replace explicit registries with dynamic plugin loading;
|
||||||
|
- introduce broad abstractions when a small provider-specific helper preserves
|
||||||
|
clarity.
|
||||||
|
|
||||||
|
## Dependency Policy
|
||||||
|
|
||||||
|
Prefer the Go standard library unless a dependency materially improves
|
||||||
|
maintainability or interoperability.
|
||||||
|
|
||||||
|
`feedkit` owns generic daemon infrastructure for config, HTTP source helpers,
|
||||||
|
scheduling, processors, dispatch, and sinks. Weatherfeeder code should contain
|
||||||
|
weather-domain behavior and narrow adapter logic rather than duplicating feedkit
|
||||||
|
infrastructure.
|
||||||
|
|
||||||
|
Do not add broad dependencies for small conveniences. Do not let
|
||||||
|
dependency-specific types leak across package boundaries unless that dependency
|
||||||
|
is the package contract.
|
||||||
|
|
||||||
|
## Adding Config Fields
|
||||||
|
|
||||||
|
Generic config shape is owned by feedkit. Weatherfeeder-specific config behavior
|
||||||
|
belongs in source or sink constructors, registry setup, and tests.
|
||||||
|
|
||||||
|
When adding config behavior:
|
||||||
|
|
||||||
|
- validate required params at the adapter boundary;
|
||||||
|
- keep secrets in environment variables or placeholders, not committed values;
|
||||||
|
- update [configuration docs](../config.md);
|
||||||
|
- update maintained examples when the change affects normal operation;
|
||||||
|
- add or update config-load tests for example files when practical.
|
||||||
|
|
||||||
|
## Adding CLI Flags
|
||||||
|
|
||||||
|
The executable currently reads `config.yml` from the current working directory.
|
||||||
|
If CLI flags are added:
|
||||||
|
|
||||||
|
- keep parsing in `cmd/weatherfeeder`;
|
||||||
|
- avoid moving config policy into domain packages;
|
||||||
|
- update [CLI docs](../cli.md);
|
||||||
|
- update tests that exercise command behavior.
|
||||||
|
|
||||||
|
## Adding A Source Driver
|
||||||
|
|
||||||
|
Source drivers should fetch upstream data and emit raw events with minimal
|
||||||
|
metadata decoding.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- implement the driver under `internal/sources/<provider>`;
|
||||||
|
- build from `config.SourceConfig`;
|
||||||
|
- validate required params in the constructor;
|
||||||
|
- use feedkit HTTP helpers for HTTP polling when applicable;
|
||||||
|
- emit raw schema constants from `standards`;
|
||||||
|
- advertise emitted kinds through `Kinds()`;
|
||||||
|
- decode only metadata needed for event ID and effective time;
|
||||||
|
- register the driver in `internal/sources/builtins.go`;
|
||||||
|
- add constructor, kind, and polling tests;
|
||||||
|
- update config docs and examples when operators need new configuration;
|
||||||
|
- add provider integration notes when the provider contract needs maintenance
|
||||||
|
context.
|
||||||
|
|
||||||
|
## Adding A Normalizer
|
||||||
|
|
||||||
|
Normalizers own provider-to-canonical mapping.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- add one normalizer type per normalizer file;
|
||||||
|
- match using `Event.Schema`;
|
||||||
|
- decode raw payloads into provider structs;
|
||||||
|
- map to canonical `model` payloads;
|
||||||
|
- use `internal/normalizers/common.Finalize` so envelope handling and float
|
||||||
|
rounding stay consistent;
|
||||||
|
- preserve input envelope fields except schema, payload, and effective time;
|
||||||
|
- register through the provider package and `internal/normalizers/builtins.go`;
|
||||||
|
- add tests for schema matching, key payload fields, effective time, malformed
|
||||||
|
required data, and output validation.
|
||||||
|
|
||||||
|
## Adding Canonical Models Or Schemas
|
||||||
|
|
||||||
|
Canonical event changes affect multiple contracts.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- update payload structs in `model`;
|
||||||
|
- add or update schema constants in `standards`;
|
||||||
|
- update [event wire contract docs](../integrations/events.md);
|
||||||
|
- update normalizers that produce the schema;
|
||||||
|
- update Postgres mapping if the schema is persisted;
|
||||||
|
- add tests for wire shape and mapper behavior.
|
||||||
|
|
||||||
|
## Adding Postgres Mapping
|
||||||
|
|
||||||
|
Weatherfeeder owns the canonical-event-to-table mapping. Feedkit owns the
|
||||||
|
generic Postgres sink mechanics.
|
||||||
|
|
||||||
|
Checklist:
|
||||||
|
|
||||||
|
- update `internal/sinks/postgres/schema.go`;
|
||||||
|
- update `internal/sinks/postgres/map.go`;
|
||||||
|
- preserve event envelope columns in parent rows when the table supports them;
|
||||||
|
- validate required canonical fields before writing;
|
||||||
|
- use positional indexes for child rows that represent arrays;
|
||||||
|
- update mapper and schema tests;
|
||||||
|
- update [Postgres integration docs](../integrations/postgres.md) when the table
|
||||||
|
contract changes.
|
||||||
|
|
||||||
|
## Examples And Documentation
|
||||||
|
|
||||||
|
Documentation must follow the [documentation policy](documentation.md).
|
||||||
|
|
||||||
|
When behavior changes, update the canonical docs in the same change:
|
||||||
|
|
||||||
|
- config shape: `docs/config.md`;
|
||||||
|
- CLI behavior: `docs/cli.md`;
|
||||||
|
- operations and recovery: `docs/operations.md`;
|
||||||
|
- troubleshooting: `docs/troubleshooting.md`;
|
||||||
|
- public Go package consumption: `docs/consumers/`;
|
||||||
|
- external contracts: `docs/integrations/`;
|
||||||
|
- internal component behavior: `docs/internal/`;
|
||||||
|
- copyable configs: `examples/`.
|
||||||
|
|
||||||
|
Keep roadmap content under `docs/roadmap/`. Current-behavior docs must describe
|
||||||
|
implemented behavior only.
|
||||||
|
|
||||||
|
## Review Checklist
|
||||||
|
|
||||||
|
Before committing:
|
||||||
|
|
||||||
|
- run focused tests for changed packages;
|
||||||
|
- run `go test ./...` for broad behavior or documentation changes tied to code;
|
||||||
|
- verify maintained examples still load when examples or config docs changed;
|
||||||
|
- check links in changed docs;
|
||||||
|
- search for stale paths, unsupported features, and secret-like values;
|
||||||
|
- keep unrelated refactors out of the change.
|
||||||
444
docs/policy/documentation.md
Normal file
444
docs/policy/documentation.md
Normal file
@@ -0,0 +1,444 @@
|
|||||||
|
# Go Project Documentation Policy
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
Project documentation must help five audiences:
|
||||||
|
|
||||||
|
1. users who need to run the application;
|
||||||
|
2. administrators/operators who need to configure and operate it;
|
||||||
|
3. developers who need to understand and change it safely;
|
||||||
|
4. LLM coding agents that need clear scope, boundaries, and invariants;
|
||||||
|
5. developers and LLM coding agents integrating this project from another codebase.
|
||||||
|
|
||||||
|
Docs should be accurate, concise, task-oriented, and organized by audience. Prefer links to canonical docs over repetition.
|
||||||
|
|
||||||
|
## Core Rules
|
||||||
|
|
||||||
|
### 1. Keep docs concise
|
||||||
|
|
||||||
|
Each document should cover a defined scope and only the essentials for that scope.
|
||||||
|
|
||||||
|
Avoid:
|
||||||
|
- long background explanations;
|
||||||
|
- repeated reference material;
|
||||||
|
- implementation detail in user-facing docs;
|
||||||
|
- aspirational language outside roadmap docs;
|
||||||
|
- verbose examples where one minimal example is clearer.
|
||||||
|
|
||||||
|
### 2. Document only implemented behavior outside roadmap files
|
||||||
|
|
||||||
|
Unimplemented, planned, aspirational, experimental, or future work may be described only under:
|
||||||
|
|
||||||
|
- `docs/roadmap/`
|
||||||
|
|
||||||
|
No other documentation file, including `README.md`, should describe code, features, modules, stages, commands, config fields, or behaviors that do not currently exist.
|
||||||
|
|
||||||
|
If a feature is partial, non-roadmap docs may describe only the implemented portion and its current boundary.
|
||||||
|
|
||||||
|
### 3. Use canonical homes
|
||||||
|
|
||||||
|
Each type of information should have one canonical location.
|
||||||
|
|
||||||
|
Canonical homes:
|
||||||
|
|
||||||
|
- project purpose and quickstart: `README.md`
|
||||||
|
- development principles: `docs/policy/architecture.md`
|
||||||
|
- public HTTP API reference: `docs/api.md`
|
||||||
|
- configuration reference: `docs/config.md`
|
||||||
|
- CLI reference: `docs/cli.md`
|
||||||
|
- operations and recovery: `docs/operations.md`
|
||||||
|
- troubleshooting: `docs/troubleshooting.md`
|
||||||
|
- public API/package consumer guidance: `docs/consumers/`
|
||||||
|
- implemented internals: `docs/internal/`
|
||||||
|
- external protocol, service, and file-format contracts: `docs/integrations/`
|
||||||
|
- future work: `docs/roadmap/`
|
||||||
|
- contributor workflow: `docs/policy/development.md`
|
||||||
|
- copyable examples: `examples/`
|
||||||
|
|
||||||
|
Other files should summarize briefly and link to the canonical source.
|
||||||
|
|
||||||
|
### 4. Keep examples real
|
||||||
|
|
||||||
|
Examples should be valid, maintained, and free of secrets.
|
||||||
|
|
||||||
|
Where practical:
|
||||||
|
- example configs should load successfully;
|
||||||
|
- example commands should match real CLI syntax;
|
||||||
|
- important examples should be covered by tests.
|
||||||
|
|
||||||
|
## Documentation Profiles
|
||||||
|
|
||||||
|
All projects require:
|
||||||
|
|
||||||
|
- `README.md`
|
||||||
|
- `docs/policy/architecture.md`
|
||||||
|
|
||||||
|
Additional docs depend on the project.
|
||||||
|
|
||||||
|
### Small library
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/policy/development.md`, if contributor conventions are non-obvious
|
||||||
|
|
||||||
|
### Simple CLI
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Config-driven CLI
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`
|
||||||
|
- `docs/config.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `examples/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Stateful or operator-facing application
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- `examples/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
### Modular, service-oriented, or orchestration application
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
- `docs/internal/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- validated examples under `examples/`
|
||||||
|
|
||||||
|
### Public HTTP API service
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/api.md`
|
||||||
|
- `docs/cli.md`, if CLI-based
|
||||||
|
- `docs/config.md`, if config-driven
|
||||||
|
- `docs/operations.md`
|
||||||
|
- `docs/internal/`
|
||||||
|
- `docs/policy/development.md`
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- `docs/troubleshooting.md`
|
||||||
|
- `docs/consumers/`, for task-oriented client integration guides
|
||||||
|
- `docs/integrations/`, for upstream/downstream service contracts
|
||||||
|
- validated examples under `examples/`
|
||||||
|
|
||||||
|
### Project with public packages or consumer APIs
|
||||||
|
|
||||||
|
Required:
|
||||||
|
- `docs/consumers/api.md`
|
||||||
|
- one `docs/consumers/pkg-<name>.md` file per public package, if public packages exist
|
||||||
|
|
||||||
|
Recommended:
|
||||||
|
- copyable consumer examples under `examples/`, if practical
|
||||||
|
|
||||||
|
## Required Documents
|
||||||
|
|
||||||
|
### README.md
|
||||||
|
|
||||||
|
**Audience:** users, administrators, operators
|
||||||
|
|
||||||
|
The README is the outward-facing project orientation page.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. concise description;
|
||||||
|
2. elevator pitch;
|
||||||
|
3. shortest useful command or usage example;
|
||||||
|
4. links to targeted docs.
|
||||||
|
|
||||||
|
The README should be short. It is not a manual.
|
||||||
|
|
||||||
|
The “shortest useful command” means the simplest command that performs the project’s core use case. (It does not mean `app --help`.)
|
||||||
|
|
||||||
|
### docs/policy/architecture.md
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
`docs/policy/architecture.md` is required for every project.
|
||||||
|
|
||||||
|
It is an inward-facing development policy document. It should describe how the project is intended to be built and changed.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
- project shape;
|
||||||
|
- core design principles;
|
||||||
|
- package and boundary philosophy;
|
||||||
|
- state/persistence philosophy, if applicable;
|
||||||
|
- external integration philosophy, if applicable;
|
||||||
|
- error-handling and logging principles;
|
||||||
|
- testing expectations;
|
||||||
|
- documentation expectations;
|
||||||
|
- architectural invariants;
|
||||||
|
- explicit non-goals, if useful.
|
||||||
|
|
||||||
|
For small projects, this file may be brief. It may simply state that the project is intentionally narrow, monolithic, and dependency-light.
|
||||||
|
|
||||||
|
### docs/api.md
|
||||||
|
|
||||||
|
**Audience:** external HTTP API consumers, developers, LLM coding agents integrating by HTTP
|
||||||
|
|
||||||
|
Required for projects whose primary public interface is HTTP.
|
||||||
|
|
||||||
|
`docs/api.md` is the canonical public HTTP API contract. It should be normative for external consumers and should not be duplicated by README, operations docs, consumer guides, or integration docs.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
1. base URL conventions;
|
||||||
|
2. authentication and authorization behavior, if implemented;
|
||||||
|
3. response envelope;
|
||||||
|
4. supported media types and content negotiation behavior;
|
||||||
|
5. shared query parameters;
|
||||||
|
6. endpoint reference grouped by route family;
|
||||||
|
7. request parameters and validation rules;
|
||||||
|
8. response fields, units, nullability, and optionality;
|
||||||
|
9. error response shape and status codes;
|
||||||
|
10. pagination, caching, rate-limit, idempotency, and retry behavior, if implemented;
|
||||||
|
11. compact request and response examples.
|
||||||
|
|
||||||
|
It must document only implemented endpoints and behavior. Planned endpoints, proposed fields, future filters, and experimental response shapes belong only under `docs/roadmap/`.
|
||||||
|
|
||||||
|
For HTTP API projects, `docs/consumers/` may provide task-oriented client integration guides, but those guides should link to `docs/api.md` for the authoritative endpoint contract.
|
||||||
|
|
||||||
|
### docs/policy/development.md
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for projects maintained by humans and LLM coding agents.
|
||||||
|
|
||||||
|
It should include:
|
||||||
|
|
||||||
|
- repository layout;
|
||||||
|
- build/test commands;
|
||||||
|
- coding conventions;
|
||||||
|
- dependency policy;
|
||||||
|
- how to add config fields;
|
||||||
|
- how to add CLI flags;
|
||||||
|
- how to add modules or adapters, if applicable;
|
||||||
|
- how to update examples;
|
||||||
|
- documentation update expectations.
|
||||||
|
|
||||||
|
### docs/config.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators, advanced users
|
||||||
|
|
||||||
|
Required for applications with configuration files.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. config file locations and discovery precedence;
|
||||||
|
2. minimal working config;
|
||||||
|
3. production-oriented config;
|
||||||
|
4. full configuration reference;
|
||||||
|
5. secrets handling, if applicable;
|
||||||
|
6. links to maintained examples.
|
||||||
|
|
||||||
|
The full configuration reference should be canonical.
|
||||||
|
|
||||||
|
### docs/cli.md
|
||||||
|
|
||||||
|
**Audience:** users, administrators, operators
|
||||||
|
|
||||||
|
Required for CLI applications.
|
||||||
|
|
||||||
|
It should include, in order:
|
||||||
|
|
||||||
|
1. shortest useful command;
|
||||||
|
2. command overview;
|
||||||
|
3. complete flag reference;
|
||||||
|
4. common workflows;
|
||||||
|
5. diagnostic or recovery commands, if applicable.
|
||||||
|
|
||||||
|
Explain when commands are useful, not just their syntax.
|
||||||
|
|
||||||
|
### docs/operations.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators
|
||||||
|
|
||||||
|
Required for applications that maintain state, support resume behavior, run multi-step workflows, write durable artifacts, use remote storage, or require recovery procedures.
|
||||||
|
|
||||||
|
It should cover:
|
||||||
|
|
||||||
|
- normal workflow;
|
||||||
|
- filesystem layout;
|
||||||
|
- remote storage layout, if applicable;
|
||||||
|
- logs and manifests;
|
||||||
|
- resume/retry behavior;
|
||||||
|
- cleanup behavior;
|
||||||
|
- archive/backup behavior;
|
||||||
|
- safe recovery procedures;
|
||||||
|
- operational caveats.
|
||||||
|
|
||||||
|
### docs/troubleshooting.md
|
||||||
|
|
||||||
|
**Audience:** administrators, operators
|
||||||
|
|
||||||
|
Recommended once recurring failure modes exist.
|
||||||
|
|
||||||
|
Each entry should include:
|
||||||
|
|
||||||
|
- symptom;
|
||||||
|
- likely cause;
|
||||||
|
- diagnostic command or inspection step;
|
||||||
|
- safe fix;
|
||||||
|
- relevant links.
|
||||||
|
|
||||||
|
### docs/consumers/
|
||||||
|
|
||||||
|
**Audience:** developers and LLM coding agents integrating this project from another codebase
|
||||||
|
|
||||||
|
Required for projects with public packages, SDKs, client APIs, plugin APIs, or other application-facing integration surfaces.
|
||||||
|
|
||||||
|
This directory describes how an external codebase should consume the project's public API. It should be task-oriented and copyable where useful. It is not the place for internal implementation details or operator procedures.
|
||||||
|
|
||||||
|
For projects whose public API is HTTP, `docs/consumers/` is not required, and it should not duplicate the endpoint reference in `docs/api.md`. If present, it may provide practical integration workflows, client-specific examples, or migration notes that link back to `docs/api.md`.
|
||||||
|
|
||||||
|
`docs/consumers/api.md` should provide the consumer-facing overview and primary implementation workflow. It should include:
|
||||||
|
|
||||||
|
1. intended consumer audience and use cases;
|
||||||
|
2. required inputs supplied by operators or deployment configuration;
|
||||||
|
3. recommended public package or API workflow;
|
||||||
|
4. minimal copyable example;
|
||||||
|
5. consumer responsibilities and boundaries;
|
||||||
|
6. retry, idempotency, or status behavior, if applicable;
|
||||||
|
7. links to package-specific docs and canonical integration contracts.
|
||||||
|
|
||||||
|
Package-specific docs should be named `pkg-<name>.md` and should include:
|
||||||
|
|
||||||
|
1. import path;
|
||||||
|
2. intended use cases;
|
||||||
|
3. primary types and functions needed by consumers;
|
||||||
|
4. minimal examples;
|
||||||
|
5. validation, error, retry, and boundary behavior;
|
||||||
|
6. links to canonical file-format or wire-protocol contracts.
|
||||||
|
|
||||||
|
### docs/internal/
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for modular, service-oriented, or orchestration projects.
|
||||||
|
|
||||||
|
This directory describes implemented internal components. It is not the roadmap.
|
||||||
|
|
||||||
|
Use one file per major component where useful.
|
||||||
|
|
||||||
|
Each component doc should include:
|
||||||
|
|
||||||
|
1. purpose;
|
||||||
|
2. inputs and outputs;
|
||||||
|
3. boundaries;
|
||||||
|
4. config fields used;
|
||||||
|
5. external adapters used;
|
||||||
|
6. state or manifest behavior, if applicable;
|
||||||
|
7. skip/resume behavior, if applicable;
|
||||||
|
8. failure behavior;
|
||||||
|
9. tests to inspect before changing;
|
||||||
|
10. architectural invariants.
|
||||||
|
|
||||||
|
### docs/roadmap/
|
||||||
|
|
||||||
|
**Audience:** maintainers, developers, LLM coding agents
|
||||||
|
|
||||||
|
This is the only place for planned, future, aspirational, experimental, or unimplemented work.
|
||||||
|
|
||||||
|
Roadmap docs should clearly distinguish:
|
||||||
|
|
||||||
|
- proposed work;
|
||||||
|
- accepted plans;
|
||||||
|
- deferred ideas;
|
||||||
|
- rejected ideas;
|
||||||
|
- implementation prompts or task breakdowns, if useful.
|
||||||
|
|
||||||
|
Roadmap docs should not be confused with current behavior.
|
||||||
|
|
||||||
|
### docs/integrations/
|
||||||
|
|
||||||
|
**Audience:** developers, LLM coding agents
|
||||||
|
|
||||||
|
Required for projects that depend on external CLIs, APIs, services, protocols, or file formats where the integration contract is important to maintain.
|
||||||
|
|
||||||
|
This directory contains concise, versioned reference notes for external integration contracts. It should document only the parts of the external system that this project actually uses or exposes.
|
||||||
|
|
||||||
|
For public HTTP API services, `docs/integrations/` should document upstream, downstream, storage, protocol, or runtime contracts that the service depends on or bridges. It should not become a second copy of the public HTTP endpoint reference; that belongs in `docs/api.md`.
|
||||||
|
|
||||||
|
Use one file per integration where useful.
|
||||||
|
|
||||||
|
## Examples Directory
|
||||||
|
|
||||||
|
Projects with non-trivial configuration or workflows should include `examples/`.
|
||||||
|
|
||||||
|
Useful examples include:
|
||||||
|
|
||||||
|
- minimal working config;
|
||||||
|
- production-oriented config;
|
||||||
|
- full annotated config;
|
||||||
|
- local development config;
|
||||||
|
- remote/object-storage config;
|
||||||
|
- minimal session/input file.
|
||||||
|
|
||||||
|
Examples should be valid, maintained, tested when practical, and linked from relevant docs.
|
||||||
|
|
||||||
|
## Security and Privacy
|
||||||
|
|
||||||
|
Docs and examples must not include:
|
||||||
|
|
||||||
|
- real API keys;
|
||||||
|
- tokens;
|
||||||
|
- passwords;
|
||||||
|
- private keys;
|
||||||
|
- private environment dumps;
|
||||||
|
- sensitive user data;
|
||||||
|
- raw private transcripts;
|
||||||
|
- private infrastructure details unless intentionally public.
|
||||||
|
|
||||||
|
Document secret-handling mechanisms, not actual secret values.
|
||||||
|
|
||||||
|
## Maintenance Rules
|
||||||
|
|
||||||
|
When docs change, verify the affected behavior.
|
||||||
|
|
||||||
|
Where practical:
|
||||||
|
|
||||||
|
- load example config files in tests;
|
||||||
|
- test CLI examples or command parser behavior;
|
||||||
|
- validate documented flags against real flags;
|
||||||
|
- remove stale references;
|
||||||
|
- update links after renames;
|
||||||
|
- keep roadmap content out of non-roadmap docs.
|
||||||
|
|
||||||
|
If documentation and code disagree, fix the documentation and/or open a roadmap item; do not leave aspirational behavior in current-behavior docs.
|
||||||
|
|
||||||
|
Documentation is complete only when it matches the current code.
|
||||||
|
|
||||||
|
## Documentation Change Checklist
|
||||||
|
|
||||||
|
Before merging documentation changes, verify:
|
||||||
|
|
||||||
|
- README is concise and orientation-focused.
|
||||||
|
- `docs/policy/architecture.md` describes development principles.
|
||||||
|
- `docs/api.md` is the canonical HTTP contract for HTTP API services.
|
||||||
|
- Future work appears only under `docs/roadmap/`.
|
||||||
|
- User-facing docs avoid unnecessary internals.
|
||||||
|
- Consumer-facing docs explain public APIs without duplicating HTTP endpoint or integration contracts.
|
||||||
|
- Developer-facing docs preserve boundaries and invariants.
|
||||||
|
- Config examples match the schema.
|
||||||
|
- CLI examples match real commands and flags.
|
||||||
|
- Defaults appear in the canonical config reference.
|
||||||
|
- No secrets or private data are included.
|
||||||
|
- Links are accurate.
|
||||||
99
docs/roadmap/future.md
Normal file
99
docs/roadmap/future.md
Normal file
@@ -0,0 +1,99 @@
|
|||||||
|
# Future Work
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document is the catch-all roadmap for planned, deferred, aspirational, experimental, or unimplemented weatherfeeder work. Current behavior belongs in the canonical docs outside `docs/roadmap/`.
|
||||||
|
|
||||||
|
## NWS AFD Parsing Resilience
|
||||||
|
|
||||||
|
The current parser handles the concrete RAH, LWX, and MFR variants that
|
||||||
|
motivated these ideas. Future work should keep those extension points
|
||||||
|
maintainable as additional evidence appears.
|
||||||
|
|
||||||
|
### Generalize Wrapper-Scoped Embedded Sections
|
||||||
|
|
||||||
|
The scanner currently permits undotted nested headings only inside `PREV
|
||||||
|
DISCUSSION`. If other wrapper identities are observed, replace the single
|
||||||
|
wrapper check with a small explicit provider-local registry and add a fixture
|
||||||
|
for each wrapper family. Do not make the leading dot globally optional: wrapper
|
||||||
|
scope is the safeguard against classifying uppercase prose as a section.
|
||||||
|
|
||||||
|
### Extend Conservative Preamble Classification
|
||||||
|
|
||||||
|
Leading key-message metadata currently supports validated `Issued at`, `Updated
|
||||||
|
at`, and `As of <clock> <weekday>...` forms. Add future wording variants as
|
||||||
|
small, ordered classifiers with strict label boundaries and value grammars.
|
||||||
|
Every addition should include collision tests proving that similar message prose
|
||||||
|
and malformed metadata remain canonical content.
|
||||||
|
|
||||||
|
### Keep List-Marker Recognition Extensible
|
||||||
|
|
||||||
|
The marker parser currently supports hyphens, asterisks, `N)`, `N.`, `(N)`, and
|
||||||
|
composite forms such as `- (N)`. If new decorators appear, evolve the helper
|
||||||
|
toward an explicit marker grammar or typed classification result rather than a
|
||||||
|
broad punctuation heuristic. Preserve positive-number and whitespace-boundary
|
||||||
|
checks so ordinary prose is not stripped.
|
||||||
|
|
||||||
|
### Maintain a Cross-Office Fixture Corpus
|
||||||
|
|
||||||
|
The compact RAH, LWX, and current MFR fixtures seed regression coverage for the
|
||||||
|
observed layouts. Future parser changes should add concise, deterministic HTML
|
||||||
|
fixtures for materially distinct office formats and exercise them through both
|
||||||
|
the provider parser and normalizer. Fixture comments should identify the format
|
||||||
|
family and state that edited prose is not an archived product; tests must remain
|
||||||
|
offline and assert both intended extraction and adjacent-section isolation.
|
||||||
|
|
||||||
|
## SPC Convective Outlook Follow-Ups
|
||||||
|
|
||||||
|
### Weatherapi Outlook Endpoints
|
||||||
|
|
||||||
|
Expose persisted SPC convective outlooks through `weatherapi` after the weatherfeeder storage contract is stable.
|
||||||
|
|
||||||
|
Likely endpoints:
|
||||||
|
|
||||||
|
- `GET /outlooks/convective`
|
||||||
|
- `GET /outlooks/convective/active`
|
||||||
|
- `GET /outlooks/convective/location`
|
||||||
|
|
||||||
|
Recommended behavior:
|
||||||
|
|
||||||
|
- Return the latest outlook run by default.
|
||||||
|
- Support active outlook filtering by current time and `containsLocation=true`.
|
||||||
|
- Consider optional query filters for `day`, `outlookType`, and `containsLocation`.
|
||||||
|
- Preserve canonical outlook geometry for downstream display and audit use.
|
||||||
|
|
||||||
|
### SPC Day 4-8 Outlooks
|
||||||
|
|
||||||
|
Add SPC Day 4-8 convective outlook support as a schema-compatible extension only after Day 1-3 operation is proven.
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
|
||||||
|
- Day 4-8 products have different semantics from Day 1-3 categorical/tornado/hail/wind products.
|
||||||
|
- Avoid forcing Day 4-8 assumptions into the current Day 1-3 model until the source shapes and consumer needs are reviewed.
|
||||||
|
- Prefer reusing `weather.outlook.v2` if the fields remain accurate; otherwise write a separate roadmap before changing the canonical contract.
|
||||||
|
|
||||||
|
### Degraded SPC Bundle Mode
|
||||||
|
|
||||||
|
Evaluate whether the SPC source should support degraded partial bundles when one required upstream product fails.
|
||||||
|
|
||||||
|
Current behavior should remain atomic: if a required GeoJSON or print-page fetch fails, emit no event for that poll.
|
||||||
|
|
||||||
|
Future degraded mode would need a clear contract for:
|
||||||
|
|
||||||
|
- distinguishing "no risk polygon" from "product missing";
|
||||||
|
- exposing per-product fetch errors without leaking raw provider internals into canonical events;
|
||||||
|
- deciding whether downstream sinks and APIs should store or serve partial snapshots.
|
||||||
|
|
||||||
|
### Richer SPC Page Assets And Tables
|
||||||
|
|
||||||
|
Evaluate whether to parse additional SPC print-page metadata beyond the current discussion text.
|
||||||
|
|
||||||
|
Possible additions:
|
||||||
|
|
||||||
|
- archive GeoJSON/shapefile/KML links;
|
||||||
|
- image URLs;
|
||||||
|
- page risk tables;
|
||||||
|
- city tables;
|
||||||
|
- richer discussion section metadata.
|
||||||
|
|
||||||
|
Keep GeoJSON products authoritative for polygons, validity windows, and point matching unless a future roadmap explicitly changes that contract.
|
||||||
87
docs/roadmap/outlook-schema-transition.md
Normal file
87
docs/roadmap/outlook-schema-transition.md
Normal file
@@ -0,0 +1,87 @@
|
|||||||
|
# SPC Outlook Postgres Schema Transition
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
This document describes how to reset existing Postgres outlook tables from the
|
||||||
|
`weather.outlook.v1` storage shape to the `weather.outlook.v2` compatible
|
||||||
|
storage shape.
|
||||||
|
|
||||||
|
Updated `weatherfeeder` versions create outlook tables with run-level
|
||||||
|
discussion storage. Existing databases that already contain the old outlook
|
||||||
|
table family need a manual reset because the Postgres sink creates tables with
|
||||||
|
`CREATE TABLE IF NOT EXISTS`.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
This reset drops only the outlook table family and lets updated `weatherfeeder`
|
||||||
|
recreate it:
|
||||||
|
|
||||||
|
- `outlook_discussions`
|
||||||
|
- `outlooks`
|
||||||
|
- `outlook_runs`
|
||||||
|
|
||||||
|
Other weather tables are not affected.
|
||||||
|
|
||||||
|
## Warning
|
||||||
|
|
||||||
|
These commands delete stored SPC outlook history. Existing `weather.outlook.v1`
|
||||||
|
outlook rows are intentionally removed. Downstream readers should be updated
|
||||||
|
intentionally for the new outlook shape.
|
||||||
|
|
||||||
|
## Deployment Order
|
||||||
|
|
||||||
|
1. Stop `weatherfeeder`.
|
||||||
|
2. Drop the existing outlook tables.
|
||||||
|
3. Deploy updated `weatherfeeder`.
|
||||||
|
4. Start `weatherfeeder` so the Postgres sink recreates the new outlook tables.
|
||||||
|
5. Deploy updated downstream consumers such as `weatherapi`.
|
||||||
|
|
||||||
|
## Reset SQL
|
||||||
|
|
||||||
|
```sql
|
||||||
|
DROP TABLE IF EXISTS outlook_discussions;
|
||||||
|
DROP TABLE IF EXISTS outlooks;
|
||||||
|
DROP TABLE IF EXISTS outlook_runs;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Verification SQL
|
||||||
|
|
||||||
|
Before or after the updated daemon starts, this query shows which outlook tables
|
||||||
|
exist:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT table_name
|
||||||
|
FROM information_schema.tables
|
||||||
|
WHERE table_name IN ('outlook_runs', 'outlooks', 'outlook_discussions')
|
||||||
|
ORDER BY table_name;
|
||||||
|
```
|
||||||
|
|
||||||
|
After the updated daemon has started and recreated the tables, verify the new
|
||||||
|
run column:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT column_name, is_nullable, data_type
|
||||||
|
FROM information_schema.columns
|
||||||
|
WHERE table_name = 'outlook_runs'
|
||||||
|
AND column_name = 'discussion_count';
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify the discussion table indexes:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT indexname
|
||||||
|
FROM pg_indexes
|
||||||
|
WHERE tablename = 'outlook_discussions'
|
||||||
|
ORDER BY indexname;
|
||||||
|
```
|
||||||
|
|
||||||
|
Verify that legacy polygon-level prose columns are gone from `outlooks`:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT column_name
|
||||||
|
FROM information_schema.columns
|
||||||
|
WHERE table_name = 'outlooks'
|
||||||
|
AND column_name IN ('headline', 'summary', 'discussion');
|
||||||
|
```
|
||||||
|
|
||||||
|
The final query should return zero rows.
|
||||||
251
docs/troubleshooting.md
Normal file
251
docs/troubleshooting.md
Normal file
@@ -0,0 +1,251 @@
|
|||||||
|
# Troubleshooting
|
||||||
|
|
||||||
|
Use this guide with [configuration](config.md) and [operations](operations.md).
|
||||||
|
Messages are emitted through the standard logger on stderr.
|
||||||
|
|
||||||
|
## `config load failed: ... read "config.yml"`
|
||||||
|
|
||||||
|
Symptom: startup exits before building sources or sinks.
|
||||||
|
|
||||||
|
Likely cause: the process working directory does not contain `config.yml`, or
|
||||||
|
the runtime user cannot read it.
|
||||||
|
|
||||||
|
Diagnostic: run `pwd` in the same working directory used by the process, then
|
||||||
|
check `ls -l config.yml`.
|
||||||
|
|
||||||
|
Safe fix: place the intended config at `./config.yml`, change the working
|
||||||
|
directory, or mount the file at `/weatherfeeder/config.yml` when using the
|
||||||
|
provided container image.
|
||||||
|
|
||||||
|
## `config load failed: ... parse YAML`
|
||||||
|
|
||||||
|
Symptom: startup exits with a YAML parse error or an unknown field error.
|
||||||
|
|
||||||
|
Likely cause: invalid YAML syntax, multiple YAML documents, or a misspelled
|
||||||
|
config struct field.
|
||||||
|
|
||||||
|
Diagnostic: inspect the line and field in the error. Feedkit uses strict YAML
|
||||||
|
field decoding for config struct fields.
|
||||||
|
|
||||||
|
Safe fix: correct the YAML and compare the shape with
|
||||||
|
[configuration](config.md). Driver-specific `params` keys are validated by their
|
||||||
|
source or sink constructors.
|
||||||
|
|
||||||
|
## `config validation failed`
|
||||||
|
|
||||||
|
Symptom: startup exits and prints one or more validation messages.
|
||||||
|
|
||||||
|
Likely cause: missing `sources` or `sinks`, blank names, duplicate source or sink
|
||||||
|
names, invalid `mode`, missing `every` for a polling source with `mode: poll`, or
|
||||||
|
a route that references an unknown sink.
|
||||||
|
|
||||||
|
Diagnostic: read every bullet under `config validation failed`; the loader sorts
|
||||||
|
these messages so multiple issues can be fixed in one edit.
|
||||||
|
|
||||||
|
Safe fix: update the top-level config fields as documented in
|
||||||
|
[configuration](config.md).
|
||||||
|
|
||||||
|
## `unknown source driver`
|
||||||
|
|
||||||
|
Symptom: startup exits with `build source failed`.
|
||||||
|
|
||||||
|
Likely cause: `sources[].driver` does not match a registered weatherfeeder
|
||||||
|
source driver.
|
||||||
|
|
||||||
|
Diagnostic: compare the configured driver with the source driver table in
|
||||||
|
[configuration](config.md#source-drivers).
|
||||||
|
|
||||||
|
Safe fix: correct the driver name. Current drivers include `nws_observation`,
|
||||||
|
`nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`,
|
||||||
|
`nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`,
|
||||||
|
`openmeteo_forecast`, `openweather_observation`, and
|
||||||
|
`spc_convective_outlook`.
|
||||||
|
|
||||||
|
## `unknown sink driver`
|
||||||
|
|
||||||
|
Symptom: startup exits with `build sink failed`.
|
||||||
|
|
||||||
|
Likely cause: `sinks[].driver` is not registered.
|
||||||
|
|
||||||
|
Diagnostic: compare the configured driver with the sink driver table in
|
||||||
|
[configuration](config.md#sink-drivers).
|
||||||
|
|
||||||
|
Safe fix: use `stdout`, `nats`, or `postgres`.
|
||||||
|
|
||||||
|
## `source expected kinds validation failed`
|
||||||
|
|
||||||
|
Symptom: startup exits after building a source.
|
||||||
|
|
||||||
|
Likely cause: `sources[].kinds` declares a kind the source does not emit.
|
||||||
|
|
||||||
|
Diagnostic: compare the configured `kinds` list with the source driver kind in
|
||||||
|
[configuration](config.md#source-drivers).
|
||||||
|
|
||||||
|
Safe fix: remove `kinds` or set it to the kind emitted by that driver.
|
||||||
|
|
||||||
|
## `params.url is required` Or `params.user_agent is required`
|
||||||
|
|
||||||
|
Symptom: startup exits with `build source failed`.
|
||||||
|
|
||||||
|
Likely cause: a source is missing required HTTP params, or the values are blank
|
||||||
|
or not strings.
|
||||||
|
|
||||||
|
Diagnostic: inspect the named source in the error and check its `params`.
|
||||||
|
|
||||||
|
Safe fix: add non-empty `url` and `user_agent` values. See
|
||||||
|
[HTTP source params](config.md#http-source-params).
|
||||||
|
|
||||||
|
SPC convective outlook sources do not use `params.url`; they require
|
||||||
|
`latitude`, `longitude`, and `user_agent`. See
|
||||||
|
[SPC convective outlook params](config.md#spc-convective-outlook-params).
|
||||||
|
|
||||||
|
## `params.latitude is required` Or `params.longitude is required`
|
||||||
|
|
||||||
|
Symptom: startup exits for an `spc_convective_outlook` source.
|
||||||
|
|
||||||
|
Likely cause: the source is missing one of the configured point coordinates.
|
||||||
|
|
||||||
|
Diagnostic: inspect the named SPC source in the error and check its `params`.
|
||||||
|
|
||||||
|
Safe fix: add numeric `latitude` and `longitude` values in decimal degrees.
|
||||||
|
|
||||||
|
## `url must include units=metric`
|
||||||
|
|
||||||
|
Symptom: startup exits for an `openweather_observation` source.
|
||||||
|
|
||||||
|
Likely cause: the OpenWeather URL omits `units=metric` or sets another unit
|
||||||
|
system.
|
||||||
|
|
||||||
|
Diagnostic: inspect the query string in `params.url`.
|
||||||
|
|
||||||
|
Safe fix: add `units=metric` to the OpenWeather current-weather URL. Keep API
|
||||||
|
keys out of committed configs.
|
||||||
|
|
||||||
|
## `source ... sources[].every must be > 0 for polling sources`
|
||||||
|
|
||||||
|
Symptom: startup exits while building scheduler jobs.
|
||||||
|
|
||||||
|
Likely cause: a current weatherfeeder polling source has no usable `every`
|
||||||
|
interval.
|
||||||
|
|
||||||
|
Diagnostic: inspect the named `sources[]` entry and check `every`.
|
||||||
|
|
||||||
|
Safe fix: set a positive duration such as `1m`, `10m`, or `1h`.
|
||||||
|
|
||||||
|
## `build sink failed ... params.url is required`
|
||||||
|
|
||||||
|
Symptom: startup exits while building a NATS sink.
|
||||||
|
|
||||||
|
Likely cause: the NATS sink is missing `params.url`, or the value is blank or
|
||||||
|
not a string.
|
||||||
|
|
||||||
|
Diagnostic: inspect the named sink in the error and check its `params`.
|
||||||
|
|
||||||
|
Safe fix: set a NATS URL such as `nats://localhost:4222`.
|
||||||
|
|
||||||
|
## `build sink failed ... params.subject is required`
|
||||||
|
|
||||||
|
Symptom: startup exits while building a NATS sink.
|
||||||
|
|
||||||
|
Likely cause: the NATS sink is missing `params.subject`, or the value is blank
|
||||||
|
or not a string.
|
||||||
|
|
||||||
|
Diagnostic: inspect the named sink in the error and check its `params`.
|
||||||
|
|
||||||
|
Safe fix: set a non-empty subject such as `weatherfeeder`.
|
||||||
|
|
||||||
|
## `dispatch: sink ... failed consuming event ... NATS sink: connect`
|
||||||
|
|
||||||
|
Symptom: the daemon starts, but NATS events are not published.
|
||||||
|
|
||||||
|
Likely cause: the NATS server URL is unreachable, the server is not accepting
|
||||||
|
connections, or the configured URL is wrong for the runtime network.
|
||||||
|
|
||||||
|
Diagnostic: from the same runtime environment, check that the host and port in
|
||||||
|
`sinks[].params.url` are reachable.
|
||||||
|
|
||||||
|
Safe fix: correct the NATS URL or restore broker connectivity. Other configured
|
||||||
|
sinks continue receiving events.
|
||||||
|
|
||||||
|
## `postgres sink ... open db`
|
||||||
|
|
||||||
|
Symptom: startup exits while building a Postgres sink.
|
||||||
|
|
||||||
|
Likely cause: the database URI, username, password, network path, or database
|
||||||
|
availability is wrong.
|
||||||
|
|
||||||
|
Diagnostic: inspect `sinks[].params.uri`, `username`, and `password`; verify
|
||||||
|
that the same runtime environment can reach the database.
|
||||||
|
|
||||||
|
Safe fix: correct the credentials or URI, restore database connectivity, then
|
||||||
|
restart the daemon.
|
||||||
|
|
||||||
|
## `postgres sink ... ensure table` Or `ensure index`
|
||||||
|
|
||||||
|
Symptom: startup exits during Postgres initialization.
|
||||||
|
|
||||||
|
Likely cause: the database user cannot create required tables or indexes, an
|
||||||
|
existing object conflicts with weatherfeeder's expected table contract, or the
|
||||||
|
database is unavailable during initialization.
|
||||||
|
|
||||||
|
Diagnostic: inspect the named table or index in the error and compare existing
|
||||||
|
database objects with the [Postgres table contract](integrations/postgres.md).
|
||||||
|
|
||||||
|
Safe fix: grant the needed database privileges, create a compatible schema, or
|
||||||
|
perform an operator-managed migration before restarting.
|
||||||
|
|
||||||
|
## `postgres sink: insert into ...`
|
||||||
|
|
||||||
|
Symptom: the daemon starts, but Postgres writes for some events fail.
|
||||||
|
|
||||||
|
Likely cause: a duplicate primary key, incompatible existing table definition,
|
||||||
|
database constraint error, or connection failure during a write transaction.
|
||||||
|
|
||||||
|
Diagnostic: inspect the table name and database error in the log. Compare the
|
||||||
|
table with [Postgres integration](integrations/postgres.md).
|
||||||
|
|
||||||
|
Safe fix: repair the database schema or address the duplicate/connection issue.
|
||||||
|
Other configured sinks continue receiving events.
|
||||||
|
|
||||||
|
## No Events Appear On A Sink
|
||||||
|
|
||||||
|
Symptom: the daemon is running but the expected sink receives no events.
|
||||||
|
|
||||||
|
Likely cause: the route does not match the event kind, the source has not
|
||||||
|
emitted changed content, or the sink is failing per event.
|
||||||
|
|
||||||
|
Diagnostic: check `routes`, source `kinds`, and logs for `scheduler: poll
|
||||||
|
failed`, `dispatcher: pipeline error`, or `dispatch: sink ... failed consuming
|
||||||
|
event`.
|
||||||
|
|
||||||
|
Safe fix: correct the route or source configuration. If the source uses
|
||||||
|
conditional HTTP and the upstream has not changed, no event is emitted for a
|
||||||
|
`304 Not Modified` response; wait for changed upstream content or temporarily
|
||||||
|
set `params.conditional: false` for diagnosis.
|
||||||
|
|
||||||
|
## `scheduler: poll failed`
|
||||||
|
|
||||||
|
Symptom: one source logs poll failures while the daemon keeps running.
|
||||||
|
|
||||||
|
Likely cause: upstream HTTP error, bad URL, timeout, response body limit, or
|
||||||
|
provider response shape that the source cannot parse.
|
||||||
|
|
||||||
|
Diagnostic: inspect the source name in the log and review its HTTP params.
|
||||||
|
|
||||||
|
Safe fix: correct the URL, user agent, timeout, or body limit. The next
|
||||||
|
scheduled poll will retry.
|
||||||
|
|
||||||
|
## `dispatcher: pipeline error`
|
||||||
|
|
||||||
|
Symptom: source polling succeeds, but one event is dropped before sinks.
|
||||||
|
|
||||||
|
Likely cause: a normalizer could not decode or map the raw payload, or dedupe
|
||||||
|
received an invalid event ID.
|
||||||
|
|
||||||
|
Diagnostic: inspect the error text and the source/schema that produced the
|
||||||
|
event. Review the [event wire contract](integrations/events.md) for expected
|
||||||
|
canonical fields.
|
||||||
|
|
||||||
|
Safe fix: correct source configuration if it points to the wrong upstream
|
||||||
|
product. If the upstream payload changed shape, update the relevant normalizer
|
||||||
|
and tests.
|
||||||
19
examples/config.minimal.yml
Normal file
19
examples/config.minimal.yml
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
---
|
||||||
|
sources:
|
||||||
|
- name: NWSObservationKSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["observation"]
|
||||||
|
driver: nws_observation
|
||||||
|
every: 10m
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
sinks:
|
||||||
|
- name: stdout
|
||||||
|
driver: stdout
|
||||||
|
params: {}
|
||||||
|
|
||||||
|
routes:
|
||||||
|
- sink: stdout
|
||||||
|
kinds: ["observation"]
|
||||||
42
examples/config.nats.yml
Normal file
42
examples/config.nats.yml
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
---
|
||||||
|
sources:
|
||||||
|
- name: NWSObservationKSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["observation"]
|
||||||
|
driver: nws_observation
|
||||||
|
every: 10m
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
- name: NWSAlertsSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["alert"]
|
||||||
|
driver: nws_alerts
|
||||||
|
every: 1m
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
- name: SPCConvectiveOutlookSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["outlook"]
|
||||||
|
driver: spc_convective_outlook
|
||||||
|
every: 30m
|
||||||
|
params:
|
||||||
|
latitude: 38.6239
|
||||||
|
longitude: -90.3571
|
||||||
|
location_id: "stl"
|
||||||
|
location_name: "St. Louis, MO"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
sinks:
|
||||||
|
- name: nats_weather
|
||||||
|
driver: nats
|
||||||
|
params:
|
||||||
|
url: nats://localhost:4222
|
||||||
|
subject: weatherfeeder.events
|
||||||
|
|
||||||
|
routes:
|
||||||
|
- sink: nats_weather
|
||||||
|
kinds: ["observation", "alert", "outlook"]
|
||||||
44
examples/config.postgres.yml
Normal file
44
examples/config.postgres.yml
Normal file
@@ -0,0 +1,44 @@
|
|||||||
|
---
|
||||||
|
sources:
|
||||||
|
- name: NWSObservationKSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["observation"]
|
||||||
|
driver: nws_observation
|
||||||
|
every: 10m
|
||||||
|
params:
|
||||||
|
url: "https://api.weather.gov/stations/KSTL/observations/latest"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
- name: OpenMeteoHourlyForecastSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["forecast"]
|
||||||
|
driver: openmeteo_forecast
|
||||||
|
every: 1h
|
||||||
|
params:
|
||||||
|
url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&hourly=temperature_2m,relative_humidity_2m,dew_point_2m,apparent_temperature,precipitation_probability,precipitation,snowfall,weather_code,surface_pressure,wind_speed_10m,wind_direction_10m&forecast_days=3"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
- name: SPCConvectiveOutlookSTL
|
||||||
|
mode: poll
|
||||||
|
kinds: ["outlook"]
|
||||||
|
driver: spc_convective_outlook
|
||||||
|
every: 30m
|
||||||
|
params:
|
||||||
|
latitude: 38.6239
|
||||||
|
longitude: -90.3571
|
||||||
|
location_id: "stl"
|
||||||
|
location_name: "St. Louis, MO"
|
||||||
|
user_agent: "weatherfeeder example (operator@example.com)"
|
||||||
|
|
||||||
|
sinks:
|
||||||
|
- name: pg_weather
|
||||||
|
driver: postgres
|
||||||
|
params:
|
||||||
|
uri: "postgres://postgres.example.invalid:5432/weatherfeeder?sslmode=disable"
|
||||||
|
username: <database_username>
|
||||||
|
password: <database_password>
|
||||||
|
prune: 3d
|
||||||
|
|
||||||
|
routes:
|
||||||
|
- sink: pg_weather
|
||||||
|
kinds: ["observation", "forecast", "outlook"]
|
||||||
107
internal/geo/geojson.go
Normal file
107
internal/geo/geojson.go
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
package geo
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
)
|
||||||
|
|
||||||
|
type geometry struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Coordinates json.RawMessage `json:"coordinates"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// ContainsPoint reports whether a GeoJSON Polygon or MultiPolygon contains p.
|
||||||
|
// GeoJSON coordinate order is [longitude, latitude].
|
||||||
|
func ContainsPoint(raw []byte, p Point) (bool, error) {
|
||||||
|
if len(raw) == 0 {
|
||||||
|
return false, fmt.Errorf("geojson geometry is empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
var geom geometry
|
||||||
|
if err := json.Unmarshal(raw, &geom); err != nil {
|
||||||
|
return false, fmt.Errorf("decode geojson geometry: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
switch geom.Type {
|
||||||
|
case "Polygon":
|
||||||
|
polygon, err := decodePolygon(geom.Coordinates)
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("decode polygon: %w", err)
|
||||||
|
}
|
||||||
|
return polygonContainsPoint(polygon, p), nil
|
||||||
|
case "MultiPolygon":
|
||||||
|
multiPolygon, err := decodeMultiPolygon(geom.Coordinates)
|
||||||
|
if err != nil {
|
||||||
|
return false, fmt.Errorf("decode multipolygon: %w", err)
|
||||||
|
}
|
||||||
|
for _, polygon := range multiPolygon {
|
||||||
|
if polygonContainsPoint(polygon, p) {
|
||||||
|
return true, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false, nil
|
||||||
|
case "":
|
||||||
|
return false, fmt.Errorf("geojson geometry type is required")
|
||||||
|
default:
|
||||||
|
return false, fmt.Errorf("unsupported geojson geometry type %q", geom.Type)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodePolygon(raw json.RawMessage) (Polygon, error) {
|
||||||
|
var coords [][][]float64
|
||||||
|
if err := json.Unmarshal(raw, &coords); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return polygonFromCoordinates(coords)
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeMultiPolygon(raw json.RawMessage) ([]Polygon, error) {
|
||||||
|
var coords [][][][]float64
|
||||||
|
if err := json.Unmarshal(raw, &coords); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(coords) == 0 {
|
||||||
|
return nil, fmt.Errorf("multipolygon has no polygons")
|
||||||
|
}
|
||||||
|
|
||||||
|
out := make([]Polygon, 0, len(coords))
|
||||||
|
for i, polygonCoords := range coords {
|
||||||
|
polygon, err := polygonFromCoordinates(polygonCoords)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("polygons[%d]: %w", i, err)
|
||||||
|
}
|
||||||
|
out = append(out, polygon)
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func polygonFromCoordinates(coords [][][]float64) (Polygon, error) {
|
||||||
|
if len(coords) == 0 {
|
||||||
|
return nil, fmt.Errorf("polygon has no rings")
|
||||||
|
}
|
||||||
|
|
||||||
|
polygon := make(Polygon, 0, len(coords))
|
||||||
|
for i, ringCoords := range coords {
|
||||||
|
ring, err := ringFromCoordinates(ringCoords)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("rings[%d]: %w", i, err)
|
||||||
|
}
|
||||||
|
polygon = append(polygon, ring)
|
||||||
|
}
|
||||||
|
return polygon, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ringFromCoordinates(coords [][]float64) (Ring, error) {
|
||||||
|
if len(coords) == 0 {
|
||||||
|
return nil, fmt.Errorf("ring has no points")
|
||||||
|
}
|
||||||
|
|
||||||
|
ring := make(Ring, 0, len(coords))
|
||||||
|
for i, pair := range coords {
|
||||||
|
if len(pair) < 2 {
|
||||||
|
return nil, fmt.Errorf("points[%d] has %d values, need longitude and latitude", i, len(pair))
|
||||||
|
}
|
||||||
|
ring = append(ring, Point{Longitude: pair[0], Latitude: pair[1]})
|
||||||
|
}
|
||||||
|
return ring, nil
|
||||||
|
}
|
||||||
105
internal/geo/point.go
Normal file
105
internal/geo/point.go
Normal file
@@ -0,0 +1,105 @@
|
|||||||
|
package geo
|
||||||
|
|
||||||
|
import "math"
|
||||||
|
|
||||||
|
const epsilon = 1e-9
|
||||||
|
|
||||||
|
// Point is a geographic coordinate in decimal degrees.
|
||||||
|
type Point struct {
|
||||||
|
Longitude float64
|
||||||
|
Latitude float64
|
||||||
|
}
|
||||||
|
|
||||||
|
// Ring is one GeoJSON linear ring.
|
||||||
|
type Ring []Point
|
||||||
|
|
||||||
|
// Polygon is a GeoJSON polygon. The first ring is the exterior ring; subsequent
|
||||||
|
// rings are holes.
|
||||||
|
type Polygon []Ring
|
||||||
|
|
||||||
|
func polygonContainsPoint(polygon Polygon, p Point) bool {
|
||||||
|
if len(polygon) == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if pointOnRing(polygon[0], p) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if !ringContainsPoint(polygon[0], p) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, hole := range polygon[1:] {
|
||||||
|
if pointOnRing(hole, p) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if ringContainsPoint(hole, p) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
func ringContainsPoint(ring Ring, p Point) bool {
|
||||||
|
inside := false
|
||||||
|
n := len(ring)
|
||||||
|
if n == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, j := 0, n-1; i < n; j, i = i, i+1 {
|
||||||
|
a := ring[j]
|
||||||
|
b := ring[i]
|
||||||
|
if pointOnSegment(p, a, b) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
intersects := (a.Latitude > p.Latitude) != (b.Latitude > p.Latitude)
|
||||||
|
if intersects {
|
||||||
|
x := (b.Longitude-a.Longitude)*(p.Latitude-a.Latitude)/(b.Latitude-a.Latitude) + a.Longitude
|
||||||
|
if almostEqual(x, p.Longitude) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
if x > p.Longitude {
|
||||||
|
inside = !inside
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return inside
|
||||||
|
}
|
||||||
|
|
||||||
|
func pointOnRing(ring Ring, p Point) bool {
|
||||||
|
n := len(ring)
|
||||||
|
if n == 0 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for i, j := 0, n-1; i < n; j, i = i, i+1 {
|
||||||
|
if pointOnSegment(p, ring[j], ring[i]) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func pointOnSegment(p, a, b Point) bool {
|
||||||
|
cross := (p.Latitude-a.Latitude)*(b.Longitude-a.Longitude) - (p.Longitude-a.Longitude)*(b.Latitude-a.Latitude)
|
||||||
|
if math.Abs(cross) > epsilon {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
minLon, maxLon := minMax(a.Longitude, b.Longitude)
|
||||||
|
minLat, maxLat := minMax(a.Latitude, b.Latitude)
|
||||||
|
return p.Longitude >= minLon-epsilon &&
|
||||||
|
p.Longitude <= maxLon+epsilon &&
|
||||||
|
p.Latitude >= minLat-epsilon &&
|
||||||
|
p.Latitude <= maxLat+epsilon
|
||||||
|
}
|
||||||
|
|
||||||
|
func minMax(a, b float64) (float64, float64) {
|
||||||
|
if a < b {
|
||||||
|
return a, b
|
||||||
|
}
|
||||||
|
return b, a
|
||||||
|
}
|
||||||
|
|
||||||
|
func almostEqual(a, b float64) bool {
|
||||||
|
return math.Abs(a-b) <= epsilon
|
||||||
|
}
|
||||||
181
internal/geo/point_test.go
Normal file
181
internal/geo/point_test.go
Normal file
@@ -0,0 +1,181 @@
|
|||||||
|
package geo
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
const squarePolygon = `{
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [[
|
||||||
|
[-91.0, 38.0],
|
||||||
|
[-90.0, 38.0],
|
||||||
|
[-90.0, 39.0],
|
||||||
|
[-91.0, 39.0],
|
||||||
|
[-91.0, 38.0]
|
||||||
|
]]
|
||||||
|
}`
|
||||||
|
|
||||||
|
func TestContainsPointInsideSimplePolygon(t *testing.T) {
|
||||||
|
got, err := ContainsPoint([]byte(squarePolygon), Point{Longitude: -90.5, Latitude: 38.5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if !got {
|
||||||
|
t.Fatalf("ContainsPoint() = false, want true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointAcceptsRawMessage(t *testing.T) {
|
||||||
|
got, err := ContainsPoint(json.RawMessage(squarePolygon), Point{Longitude: -90.5, Latitude: 38.5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if !got {
|
||||||
|
t.Fatalf("ContainsPoint() = false, want true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointOutsideSimplePolygon(t *testing.T) {
|
||||||
|
got, err := ContainsPoint([]byte(squarePolygon), Point{Longitude: -89.5, Latitude: 38.5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if got {
|
||||||
|
t.Fatalf("ContainsPoint() = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointOnBoundary(t *testing.T) {
|
||||||
|
got, err := ContainsPoint([]byte(squarePolygon), Point{Longitude: -91.0, Latitude: 38.5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if !got {
|
||||||
|
t.Fatalf("ContainsPoint() = false, want true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointInHoleReturnsFalse(t *testing.T) {
|
||||||
|
const polygonWithHole = `{
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [
|
||||||
|
[[0,0],[10,0],[10,10],[0,10],[0,0]],
|
||||||
|
[[4,4],[6,4],[6,6],[4,6],[4,4]]
|
||||||
|
]
|
||||||
|
}`
|
||||||
|
|
||||||
|
got, err := ContainsPoint([]byte(polygonWithHole), Point{Longitude: 5, Latitude: 5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if got {
|
||||||
|
t.Fatalf("ContainsPoint() = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointOnHoleBoundaryReturnsTrue(t *testing.T) {
|
||||||
|
const polygonWithHole = `{
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [
|
||||||
|
[[0,0],[10,0],[10,10],[0,10],[0,0]],
|
||||||
|
[[4,4],[6,4],[6,6],[4,6],[4,4]]
|
||||||
|
]
|
||||||
|
}`
|
||||||
|
|
||||||
|
got, err := ContainsPoint([]byte(polygonWithHole), Point{Longitude: 4, Latitude: 5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if !got {
|
||||||
|
t.Fatalf("ContainsPoint() = false, want true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointInsideOneMultiPolygonMember(t *testing.T) {
|
||||||
|
const multiPolygon = `{
|
||||||
|
"type": "MultiPolygon",
|
||||||
|
"coordinates": [
|
||||||
|
[[[0,0],[1,0],[1,1],[0,1],[0,0]]],
|
||||||
|
[[[10,10],[12,10],[12,12],[10,12],[10,10]]]
|
||||||
|
]
|
||||||
|
}`
|
||||||
|
|
||||||
|
got, err := ContainsPoint([]byte(multiPolygon), Point{Longitude: 11, Latitude: 11})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if !got {
|
||||||
|
t.Fatalf("ContainsPoint() = false, want true")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointUsesLongitudeLatitudeOrder(t *testing.T) {
|
||||||
|
const narrowPolygon = `{
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [[
|
||||||
|
[-91.0, 38.0],
|
||||||
|
[-90.0, 38.0],
|
||||||
|
[-90.0, 39.0],
|
||||||
|
[-91.0, 39.0],
|
||||||
|
[-91.0, 38.0]
|
||||||
|
]]
|
||||||
|
}`
|
||||||
|
|
||||||
|
got, err := ContainsPoint([]byte(narrowPolygon), Point{Longitude: -90.5, Latitude: 38.5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = %v", err)
|
||||||
|
}
|
||||||
|
if !got {
|
||||||
|
t.Fatalf("ContainsPoint() = false, want true")
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err = ContainsPoint([]byte(narrowPolygon), Point{Longitude: 38.5, Latitude: -90.5})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ContainsPoint() reversed error = %v", err)
|
||||||
|
}
|
||||||
|
if got {
|
||||||
|
t.Fatalf("ContainsPoint() with reversed coordinate values = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointUnsupportedGeometryError(t *testing.T) {
|
||||||
|
_, err := ContainsPoint([]byte(`{"type":"Point","coordinates":[-90,38]}`), Point{Longitude: -90, Latitude: 38})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), `unsupported geojson geometry type "Point"`) {
|
||||||
|
t.Fatalf("ContainsPoint() error = %q", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointInvalidJSONError(t *testing.T) {
|
||||||
|
_, err := ContainsPoint([]byte(`{"type":"Polygon"`), Point{Longitude: -90, Latitude: 38})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "decode geojson geometry") {
|
||||||
|
t.Fatalf("ContainsPoint() error = %q", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointMalformedCoordinatesError(t *testing.T) {
|
||||||
|
_, err := ContainsPoint([]byte(`{"type":"Polygon","coordinates":[[[1]]]}`), Point{Longitude: 1, Latitude: 1})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "need longitude and latitude") {
|
||||||
|
t.Fatalf("ContainsPoint() error = %q", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContainsPointEmptyRingError(t *testing.T) {
|
||||||
|
_, err := ContainsPoint([]byte(`{"type":"Polygon","coordinates":[[]]}`), Point{Longitude: 1, Latitude: 1})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("ContainsPoint() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "ring has no points") {
|
||||||
|
t.Fatalf("ContainsPoint() error = %q", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -7,12 +7,14 @@ import (
|
|||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/nws"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/nws"
|
||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openmeteo"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openmeteo"
|
||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/spc"
|
||||||
)
|
)
|
||||||
|
|
||||||
var builtinRegistrations = []func([]fknormalize.Normalizer) []fknormalize.Normalizer{
|
var builtinRegistrations = []func([]fknormalize.Normalizer) []fknormalize.Normalizer{
|
||||||
nws.Register,
|
nws.Register,
|
||||||
openmeteo.Register,
|
openmeteo.Register,
|
||||||
openweather.Register,
|
openweather.Register,
|
||||||
|
spc.Register,
|
||||||
}
|
}
|
||||||
|
|
||||||
// RegisterBuiltins registers all normalizers shipped with this binary.
|
// RegisterBuiltins registers all normalizers shipped with this binary.
|
||||||
|
|||||||
@@ -8,6 +8,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/nws"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/nws"
|
||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openmeteo"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openmeteo"
|
||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/openweather"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/spc"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterBuiltinsOrder(t *testing.T) {
|
func TestRegisterBuiltinsOrder(t *testing.T) {
|
||||||
@@ -20,10 +21,12 @@ func TestRegisterBuiltinsOrder(t *testing.T) {
|
|||||||
nws.ObservationNormalizer{},
|
nws.ObservationNormalizer{},
|
||||||
nws.ForecastNormalizer{},
|
nws.ForecastNormalizer{},
|
||||||
nws.ForecastDiscussionNormalizer{},
|
nws.ForecastDiscussionNormalizer{},
|
||||||
|
nws.WeatherStoriesNormalizer{},
|
||||||
nws.AlertsNormalizer{},
|
nws.AlertsNormalizer{},
|
||||||
openmeteo.ObservationNormalizer{},
|
openmeteo.ObservationNormalizer{},
|
||||||
openmeteo.ForecastNormalizer{},
|
openmeteo.ForecastNormalizer{},
|
||||||
openweather.ObservationNormalizer{},
|
openweather.ObservationNormalizer{},
|
||||||
|
spc.ConvectiveOutlookNormalizer{},
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(got) != len(want) {
|
if len(got) != len(want) {
|
||||||
|
|||||||
@@ -9,12 +9,12 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
// Finalize builds the output event envelope by copying the input and applying the
|
// Finalize builds the output event envelope by copying the input and applying the
|
||||||
// canonical schema/payload, plus (optionally) EffectiveAt.
|
// canonical schema/payload, plus an optional effective time.
|
||||||
//
|
//
|
||||||
// Important behavior:
|
// Important behavior:
|
||||||
// - ID/Kind/Source/EmittedAt are preserved by copying the input event.
|
// - ID/Kind/Source/EmittedAt are preserved by copying the input event.
|
||||||
// - EffectiveAt is only overwritten when effectiveAt is non-zero.
|
// - EffectiveAt is only overwritten when the supplied effective time is non-zero.
|
||||||
// If effectiveAt is zero, any existing in.EffectiveAt is preserved.
|
// If the supplied time is zero, any existing in.EffectiveAt is preserved.
|
||||||
// - Payload floats are rounded to a stable wire-friendly precision (see round.go).
|
// - Payload floats are rounded to a stable wire-friendly precision (see round.go).
|
||||||
func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) {
|
func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) {
|
||||||
// Enforce stable numeric presentation for weather payloads before delegating to feedkit's
|
// Enforce stable numeric presentation for weather payloads before delegating to feedkit's
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/ejr/feedkit/event"
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestFinalizeRoundsWeatherPayloadFloats(t *testing.T) {
|
func TestFinalizeRoundsWeatherPayloadFloats(t *testing.T) {
|
||||||
@@ -14,7 +15,7 @@ func TestFinalizeRoundsWeatherPayloadFloats(t *testing.T) {
|
|||||||
|
|
||||||
in := event.Event{
|
in := event.Event{
|
||||||
ID: "evt-1",
|
ID: "evt-1",
|
||||||
Kind: event.Kind("observation"),
|
Kind: event.Kind(standards.KindObservation),
|
||||||
Source: "source-a",
|
Source: "source-a",
|
||||||
EmittedAt: time.Date(2026, 3, 28, 12, 0, 0, 0, time.UTC),
|
EmittedAt: time.Date(2026, 3, 28, 12, 0, 0, 0, time.UTC),
|
||||||
Schema: "raw.example.v1",
|
Schema: "raw.example.v1",
|
||||||
|
|||||||
@@ -15,24 +15,36 @@ import (
|
|||||||
// - sources emit raw JSON payloads (typically json.RawMessage)
|
// - sources emit raw JSON payloads (typically json.RawMessage)
|
||||||
// - normalizers decode into provider structs
|
// - normalizers decode into provider structs
|
||||||
//
|
//
|
||||||
// Errors include a small amount of stage context ("extract payload", "decode raw payload").
|
// Errors include a small amount of operation context ("extract payload", "decode raw payload").
|
||||||
// Callers typically wrap these with a provider/kind label.
|
// Callers typically wrap these with a provider/kind label.
|
||||||
func DecodeJSONPayload[T any](in event.Event) (T, error) {
|
func DecodeJSONPayload[T any](in event.Event) (T, error) {
|
||||||
|
var zero T
|
||||||
|
|
||||||
|
if typed, ok := in.Payload.(T); ok {
|
||||||
|
return typed, nil
|
||||||
|
}
|
||||||
|
if ptr, ok := in.Payload.(*T); ok {
|
||||||
|
if ptr == nil {
|
||||||
|
return zero, fmt.Errorf("extract payload: payload pointer is nil")
|
||||||
|
}
|
||||||
|
return *ptr, nil
|
||||||
|
}
|
||||||
|
|
||||||
return fknormalize.DecodeJSONPayload[T](in)
|
return fknormalize.DecodeJSONPayload[T](in)
|
||||||
}
|
}
|
||||||
|
|
||||||
// NormalizeJSON is a convenience wrapper for the common JSON-normalizer pattern:
|
// NormalizeJSON is a convenience wrapper for the common JSON-normalizer pattern:
|
||||||
//
|
//
|
||||||
// 1. Decode raw JSON payload into provider struct T
|
// 1. Decode raw JSON payload into provider struct T
|
||||||
// 2. Map T into canonical payload P (plus an EffectiveAt timestamp)
|
// 2. Map T into canonical payload P (plus an effective time)
|
||||||
// 3. Finalize the event envelope (schema/payload/effectiveAt) + Validate
|
// 3. Finalize the event envelope (schema/payload/effective time) + Validate
|
||||||
//
|
//
|
||||||
// label should be short and specific, e.g. "openweather observation".
|
// label should be short and specific, e.g. "openweather observation".
|
||||||
// outSchema should be the canonical schema constant.
|
// outSchema should be the canonical schema constant.
|
||||||
// build should contain ONLY provider/domain mapping logic.
|
// build should contain ONLY provider/domain mapping logic.
|
||||||
//
|
//
|
||||||
// Error policy:
|
// Error policy:
|
||||||
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <stage>: ..."
|
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <operation>: ..."
|
||||||
// - build() should return specific errors without repeating the label prefix.
|
// - build() should return specific errors without repeating the label prefix.
|
||||||
func NormalizeJSON[T any, P any](
|
func NormalizeJSON[T any, P any](
|
||||||
in event.Event,
|
in event.Event,
|
||||||
|
|||||||
39
internal/normalizers/common/json_test.go
Normal file
39
internal/normalizers/common/json_test.go
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
package common
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDecodeJSONPayloadAcceptsTypedPayload(t *testing.T) {
|
||||||
|
type rawPayload struct {
|
||||||
|
Value string `json:"value"`
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := DecodeJSONPayload[rawPayload](event.Event{
|
||||||
|
Payload: rawPayload{Value: "ok"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DecodeJSONPayload() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Value != "ok" {
|
||||||
|
t.Fatalf("Value = %q, want ok", got.Value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecodeJSONPayloadAcceptsTypedPointerPayload(t *testing.T) {
|
||||||
|
type rawPayload struct {
|
||||||
|
Value string `json:"value"`
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := DecodeJSONPayload[rawPayload](event.Event{
|
||||||
|
Payload: &rawPayload{Value: "ok"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DecodeJSONPayload() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Value != "ok" {
|
||||||
|
t.Fatalf("Value = %q, want ok", got.Value)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -70,7 +70,7 @@
|
|||||||
//
|
//
|
||||||
// weather.<kind>.vN
|
// weather.<kind>.vN
|
||||||
//
|
//
|
||||||
// weatherfeeder centralizes schema strings in internal/standards/schema.go.
|
// weatherfeeder centralizes schema strings in standards/schema.go.
|
||||||
// Always use those constants (do not inline schema strings).
|
// Always use those constants (do not inline schema strings).
|
||||||
//
|
//
|
||||||
// Example mappings:
|
// Example mappings:
|
||||||
@@ -101,8 +101,8 @@
|
|||||||
// Every normalizer type must have a doc comment that states:
|
// Every normalizer type must have a doc comment that states:
|
||||||
//
|
//
|
||||||
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
// - what it converts (e.g., “OpenWeather current -> WeatherObservation”)
|
||||||
// - which raw schema it matches (constant identifier from internal/standards)
|
// - which raw schema it matches (constant identifier from standards)
|
||||||
// - which canonical schema it produces (constant identifier from internal/standards)
|
// - which canonical schema it produces (constant identifier from standards)
|
||||||
// - any special caveats (units, day/night inference, missing fields, etc.)
|
// - any special caveats (units, day/night inference, missing fields, etc.)
|
||||||
//
|
//
|
||||||
// Including literal schema string values is optional,
|
// Including literal schema string values is optional,
|
||||||
|
|||||||
@@ -29,7 +29,7 @@ import (
|
|||||||
// 2. Alert timing fields are best-effort parsed; invalid timestamps do not fail the
|
// 2. Alert timing fields are best-effort parsed; invalid timestamps do not fail the
|
||||||
// entire normalization (they are left nil).
|
// entire normalization (they are left nil).
|
||||||
// 3. Some fields are intentionally passed through as strings (severity/urgency/etc.)
|
// 3. Some fields are intentionally passed through as strings (severity/urgency/etc.)
|
||||||
// since canonical vocabularies may evolve later.
|
// because the canonical model currently preserves provider vocabulary there.
|
||||||
type AlertsNormalizer struct{}
|
type AlertsNormalizer struct{}
|
||||||
|
|
||||||
func (AlertsNormalizer) Match(e event.Event) bool {
|
func (AlertsNormalizer) Match(e event.Event) bool {
|
||||||
@@ -37,7 +37,7 @@ func (AlertsNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
|
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
|
||||||
fallbackAsOf := in.EmittedAt.UTC()
|
fallbackAsOf := in.EmittedAt.UTC()
|
||||||
@@ -93,12 +93,8 @@ func buildAlerts(parsed nwsAlertsResponse, fallbackAsOf time.Time) (model.Weathe
|
|||||||
sent := nwscommon.ParseTimePtr(p.Sent)
|
sent := nwscommon.ParseTimePtr(p.Sent)
|
||||||
effective := nwscommon.ParseTimePtr(p.Effective)
|
effective := nwscommon.ParseTimePtr(p.Effective)
|
||||||
onset := nwscommon.ParseTimePtr(p.Onset)
|
onset := nwscommon.ParseTimePtr(p.Onset)
|
||||||
|
ends := nwscommon.ParseTimePtr(p.Ends)
|
||||||
// Expires: prefer "expires"; fall back to "ends" if present.
|
|
||||||
expires := nwscommon.ParseTimePtr(p.Expires)
|
expires := nwscommon.ParseTimePtr(p.Expires)
|
||||||
if expires == nil {
|
|
||||||
expires = nwscommon.ParseTimePtr(p.Ends)
|
|
||||||
}
|
|
||||||
|
|
||||||
refs := parseNWSAlertReferences(p.References)
|
refs := parseNWSAlertReferences(p.References)
|
||||||
|
|
||||||
@@ -123,6 +119,7 @@ func buildAlerts(parsed nwsAlertsResponse, fallbackAsOf time.Time) (model.Weathe
|
|||||||
Sent: sent,
|
Sent: sent,
|
||||||
Effective: effective,
|
Effective: effective,
|
||||||
Onset: onset,
|
Onset: onset,
|
||||||
|
Ends: ends,
|
||||||
Expires: expires,
|
Expires: expires,
|
||||||
|
|
||||||
AreaDescription: strings.TrimSpace(p.AreaDesc),
|
AreaDescription: strings.TrimSpace(p.AreaDesc),
|
||||||
|
|||||||
136
internal/normalizers/nws/alerts_test.go
Normal file
136
internal/normalizers/nws/alerts_test.go
Normal file
@@ -0,0 +1,136 @@
|
|||||||
|
package nws
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAlertsNormalizerMapsEndsSeparatelyFromExpires(t *testing.T) {
|
||||||
|
raw := []byte(`{
|
||||||
|
"updated":"2026-06-16T10:00:00+00:00",
|
||||||
|
"title":"Current watches, warnings, and advisories for St. Louis",
|
||||||
|
"features":[{
|
||||||
|
"id":"https://api.weather.gov/alerts/alert-1",
|
||||||
|
"properties":{
|
||||||
|
"event":"Flood Warning",
|
||||||
|
"headline":"Flood Warning issued",
|
||||||
|
"sent":"2026-06-16T09:55:00+00:00",
|
||||||
|
"effective":"2026-06-16T10:00:00+00:00",
|
||||||
|
"onset":"2026-06-16T10:15:00+00:00",
|
||||||
|
"ends":"2026-06-16T14:00:00+00:00",
|
||||||
|
"expires":"2026-06-16T11:00:00+00:00"
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}`)
|
||||||
|
|
||||||
|
out, err := AlertsNormalizer{}.Normalize(context.Background(), alertRawEvent(raw))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
run := decodeAlertRun(t, out)
|
||||||
|
if len(run.Alerts) != 1 {
|
||||||
|
t.Fatalf("expected 1 alert, got %d", len(run.Alerts))
|
||||||
|
}
|
||||||
|
alert := run.Alerts[0]
|
||||||
|
wantEnds := time.Date(2026, 6, 16, 14, 0, 0, 0, time.UTC)
|
||||||
|
wantExpires := time.Date(2026, 6, 16, 11, 0, 0, 0, time.UTC)
|
||||||
|
if alert.Ends == nil || !alert.Ends.Equal(wantEnds) {
|
||||||
|
t.Fatalf("ends = %v, want %s", alert.Ends, wantEnds)
|
||||||
|
}
|
||||||
|
if alert.Expires == nil || !alert.Expires.Equal(wantExpires) {
|
||||||
|
t.Fatalf("expires = %v, want %s", alert.Expires, wantExpires)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAlertsNormalizerDoesNotFallbackExpiresToEnds(t *testing.T) {
|
||||||
|
raw := []byte(`{
|
||||||
|
"updated":"2026-06-16T10:00:00+00:00",
|
||||||
|
"features":[{
|
||||||
|
"id":"alert-ends-only",
|
||||||
|
"properties":{
|
||||||
|
"event":"Heat Advisory",
|
||||||
|
"ends":"2026-06-16T22:00:00+00:00"
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}`)
|
||||||
|
|
||||||
|
out, err := AlertsNormalizer{}.Normalize(context.Background(), alertRawEvent(raw))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
run := decodeAlertRun(t, out)
|
||||||
|
alert := run.Alerts[0]
|
||||||
|
if alert.Ends == nil {
|
||||||
|
t.Fatal("expected ends to be populated")
|
||||||
|
}
|
||||||
|
if alert.Expires != nil {
|
||||||
|
t.Fatalf("expected expires nil when upstream expires is absent, got %v", alert.Expires)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAlertsNormalizerIgnoresInvalidEnds(t *testing.T) {
|
||||||
|
raw := []byte(`{
|
||||||
|
"updated":"2026-06-16T10:00:00+00:00",
|
||||||
|
"features":[{
|
||||||
|
"id":"alert-invalid-ends",
|
||||||
|
"properties":{
|
||||||
|
"event":"Special Weather Statement",
|
||||||
|
"ends":"not-a-time",
|
||||||
|
"expires":"2026-06-16T11:00:00+00:00"
|
||||||
|
}
|
||||||
|
}]
|
||||||
|
}`)
|
||||||
|
|
||||||
|
out, err := AlertsNormalizer{}.Normalize(context.Background(), alertRawEvent(raw))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
run := decodeAlertRun(t, out)
|
||||||
|
alert := run.Alerts[0]
|
||||||
|
if alert.Ends != nil {
|
||||||
|
t.Fatalf("expected invalid ends to map nil, got %v", alert.Ends)
|
||||||
|
}
|
||||||
|
if alert.Expires == nil {
|
||||||
|
t.Fatal("expected expires to remain populated")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func alertRawEvent(raw []byte) event.Event {
|
||||||
|
emittedAt := time.Date(2026, 6, 16, 10, 5, 0, 0, time.UTC)
|
||||||
|
return event.Event{
|
||||||
|
ID: "raw-alerts",
|
||||||
|
Kind: event.Kind(standards.KindAlert),
|
||||||
|
Source: "NWSAlerts",
|
||||||
|
Schema: standards.SchemaRawNWSAlertsV1,
|
||||||
|
EmittedAt: emittedAt,
|
||||||
|
Payload: json.RawMessage(raw),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func decodeAlertRun(t *testing.T, e *event.Event) model.WeatherAlertRun {
|
||||||
|
t.Helper()
|
||||||
|
if e == nil {
|
||||||
|
t.Fatal("expected normalized event")
|
||||||
|
}
|
||||||
|
if e.Schema != standards.SchemaWeatherAlertV1 {
|
||||||
|
t.Fatalf("schema = %q, want %q", e.Schema, standards.SchemaWeatherAlertV1)
|
||||||
|
}
|
||||||
|
var run model.WeatherAlertRun
|
||||||
|
raw, err := json.Marshal(e.Payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal alert payload: %v", err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &run); err != nil {
|
||||||
|
t.Fatalf("decode alert payload: %v", err)
|
||||||
|
}
|
||||||
|
return run
|
||||||
|
}
|
||||||
@@ -23,10 +23,11 @@ import (
|
|||||||
// builders by raw schema.
|
// builders by raw schema.
|
||||||
//
|
//
|
||||||
// Caveats / policy:
|
// Caveats / policy:
|
||||||
// 1. NWS forecast periods do not include METAR presentWeather phenomena, so ConditionCode
|
// 1. Hourly NWS forecast periods do not include METAR presentWeather phenomena, so
|
||||||
// is inferred from period.shortForecast (with a conservative icon-based fallback).
|
// ConditionCode is inferred from period.shortForecast (with a conservative icon fallback).
|
||||||
// 2. Temperature is converted to °C when NWS supplies °F.
|
// 2. Narrative NWS periods intentionally leave ConditionCode unset.
|
||||||
// 3. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
|
// 3. Temperature is converted to °C when NWS supplies °F.
|
||||||
|
// 4. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
|
||||||
type ForecastNormalizer struct{}
|
type ForecastNormalizer struct{}
|
||||||
|
|
||||||
func (ForecastNormalizer) Match(e event.Event) bool {
|
func (ForecastNormalizer) Match(e event.Event) bool {
|
||||||
@@ -41,7 +42,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normalizeForecastEventBySchema(in)
|
return normalizeForecastEventBySchema(in)
|
||||||
}
|
}
|
||||||
@@ -224,6 +225,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
|
|||||||
// Infer WMO from shortForecast (and fall back to icon token).
|
// Infer WMO from shortForecast (and fall back to icon token).
|
||||||
providerDesc := strings.TrimSpace(p.ShortForecast)
|
providerDesc := strings.TrimSpace(p.ShortForecast)
|
||||||
wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC)
|
wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC)
|
||||||
|
wmoPtr := wmoCodePtr(wmo)
|
||||||
|
|
||||||
return model.WeatherForecastPeriod{
|
return model.WeatherForecastPeriod{
|
||||||
StartTime: start,
|
StartTime: start,
|
||||||
@@ -232,7 +234,7 @@ func mapHourlyForecastPeriod(idx int, p nwsHourlyForecastPeriod) (model.WeatherF
|
|||||||
Name: strings.TrimSpace(p.Name),
|
Name: strings.TrimSpace(p.Name),
|
||||||
IsDay: isDay,
|
IsDay: isDay,
|
||||||
|
|
||||||
ConditionCode: wmo,
|
ConditionCode: wmoPtr,
|
||||||
|
|
||||||
// For forecasts, keep provider short forecast text as the human-facing description.
|
// For forecasts, keep provider short forecast text as the human-facing description.
|
||||||
TextDescription: providerDesc,
|
TextDescription: providerDesc,
|
||||||
@@ -264,9 +266,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
|
|||||||
|
|
||||||
tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit)
|
tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit)
|
||||||
|
|
||||||
// Infer WMO from shortForecast (and fall back to icon token).
|
|
||||||
shortForecast := strings.TrimSpace(p.ShortForecast)
|
shortForecast := strings.TrimSpace(p.ShortForecast)
|
||||||
wmo := wmoFromNWSForecast(shortForecast, p.Icon, tempC)
|
|
||||||
|
|
||||||
textDescription := strings.TrimSpace(p.DetailedForecast)
|
textDescription := strings.TrimSpace(p.DetailedForecast)
|
||||||
if textDescription == "" {
|
if textDescription == "" {
|
||||||
@@ -280,7 +280,7 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
|
|||||||
Name: strings.TrimSpace(p.Name),
|
Name: strings.TrimSpace(p.Name),
|
||||||
IsDay: isDay,
|
IsDay: isDay,
|
||||||
|
|
||||||
ConditionCode: wmo,
|
ConditionCode: nil,
|
||||||
|
|
||||||
TextDescription: textDescription,
|
TextDescription: textDescription,
|
||||||
|
|
||||||
@@ -292,3 +292,8 @@ func mapNarrativeForecastPeriod(idx int, p nwsNarrativeForecastPeriod) (model.We
|
|||||||
ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value,
|
ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value,
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
|
||||||
|
out := code
|
||||||
|
return &out
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"encoding/json"
|
"encoding/json"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -18,7 +19,7 @@ func TestForecastDiscussionNormalizerProducesCanonicalSchema(t *testing.T) {
|
|||||||
|
|
||||||
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
||||||
ID: "evt-discussion-1",
|
ID: "evt-discussion-1",
|
||||||
Kind: event.Kind("forecast_discussion"),
|
Kind: event.Kind(standards.KindForecastDiscussion),
|
||||||
Source: "nws-discussion-test",
|
Source: "nws-discussion-test",
|
||||||
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
||||||
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
||||||
@@ -33,7 +34,7 @@ func TestForecastDiscussionNormalizerProducesCanonicalSchema(t *testing.T) {
|
|||||||
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
|
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
|
||||||
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
|
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
|
||||||
}
|
}
|
||||||
if out.Kind != event.Kind("forecast_discussion") {
|
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
|
||||||
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
|
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -71,10 +72,281 @@ func TestForecastDiscussionNormalizerProducesCanonicalSchema(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestForecastDiscussionNormalizerSupportsMixedHeadingFormats(t *testing.T) {
|
||||||
|
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
||||||
|
ID: "evt-discussion-mixed-format",
|
||||||
|
Kind: event.Kind(standards.KindForecastDiscussion),
|
||||||
|
Source: "nws-discussion-test",
|
||||||
|
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
||||||
|
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
||||||
|
Payload: loadMixedFormatForecastDiscussionSampleHTML(t),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
if out == nil {
|
||||||
|
t.Fatalf("Normalize() returned nil output")
|
||||||
|
}
|
||||||
|
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
|
||||||
|
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
|
||||||
|
}
|
||||||
|
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
|
||||||
|
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
|
||||||
|
}
|
||||||
|
wantEffectiveAt := time.Date(2026, 3, 28, 19, 24, 0, 0, time.UTC)
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantEffectiveAt) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
|
||||||
|
}
|
||||||
|
if payload.ShortTerm == nil || payload.LongTerm == nil {
|
||||||
|
t.Fatalf("ShortTerm=%v LongTerm=%v, want both populated", payload.ShortTerm, payload.LongTerm)
|
||||||
|
}
|
||||||
|
if payload.ShortTerm.Qualifier != "Through Late Sunday Night" {
|
||||||
|
t.Fatalf("ShortTerm.Qualifier = %q", payload.ShortTerm.Qualifier)
|
||||||
|
}
|
||||||
|
if !strings.Contains(payload.ShortTerm.Text, "After a chilly morning") {
|
||||||
|
t.Fatalf("ShortTerm.Text missing expected prose: %q", payload.ShortTerm.Text)
|
||||||
|
}
|
||||||
|
if payload.LongTerm.Qualifier != "Monday through Next Saturday" {
|
||||||
|
t.Fatalf("LongTerm.Qualifier = %q", payload.LongTerm.Qualifier)
|
||||||
|
}
|
||||||
|
if !strings.Contains(payload.LongTerm.Text, "The peak of the warmth arrives Monday and Tuesday") {
|
||||||
|
t.Fatalf("LongTerm.Text missing expected prose: %q", payload.LongTerm.Text)
|
||||||
|
}
|
||||||
|
if strings.Contains(payload.LongTerm.Text, "AVIATION") || strings.Contains(payload.LongTerm.Text, "VFR conditions are expected") {
|
||||||
|
t.Fatalf("LongTerm.Text includes aviation content: %q", payload.LongTerm.Text)
|
||||||
|
}
|
||||||
|
|
||||||
|
b, err := json.Marshal(out.Payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
var fields map[string]any
|
||||||
|
if err := json.Unmarshal(b, &fields); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"aviation", "sections"} {
|
||||||
|
if _, ok := fields[key]; ok {
|
||||||
|
t.Fatalf("unexpected key %q in canonical payload", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestForecastDiscussionNormalizerSupportsCrossOfficeLayout(t *testing.T) {
|
||||||
|
in := event.Event{
|
||||||
|
ID: "evt-discussion-bou",
|
||||||
|
Kind: event.Kind(standards.KindForecastDiscussion),
|
||||||
|
Source: "nws-discussion-bou-test",
|
||||||
|
EmittedAt: time.Date(2026, 4, 7, 19, 1, 0, 0, time.UTC),
|
||||||
|
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
||||||
|
Payload: loadForecastDiscussionBOUSampleHTML(t),
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, in)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
if out == nil {
|
||||||
|
t.Fatalf("Normalize() returned nil output")
|
||||||
|
}
|
||||||
|
if out.ID != in.ID || out.Source != in.Source || !out.EmittedAt.Equal(in.EmittedAt) {
|
||||||
|
t.Fatalf("envelope = %#v, want ID/source/emittedAt from input", out)
|
||||||
|
}
|
||||||
|
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
|
||||||
|
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
|
||||||
|
}
|
||||||
|
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
|
||||||
|
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
|
||||||
|
}
|
||||||
|
wantEffectiveAt := time.Date(2026, 4, 7, 19, 0, 0, 0, time.UTC)
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantEffectiveAt) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantEffectiveAt.Format(time.RFC3339))
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
|
||||||
|
}
|
||||||
|
if payload.OfficeID != "BOU" || payload.OfficeName != "National Weather Service Denver CO" {
|
||||||
|
t.Fatalf("OfficeID=%q OfficeName=%q", payload.OfficeID, payload.OfficeName)
|
||||||
|
}
|
||||||
|
wantMessages := []string{
|
||||||
|
"Strong winds are expected along the Front Range this evening.",
|
||||||
|
"Cooler temperatures arrive on Wednesday.",
|
||||||
|
}
|
||||||
|
if len(payload.KeyMessages) != len(wantMessages) {
|
||||||
|
t.Fatalf("KeyMessages = %#v, want %#v", payload.KeyMessages, wantMessages)
|
||||||
|
}
|
||||||
|
for i := range wantMessages {
|
||||||
|
if payload.KeyMessages[i] != wantMessages[i] {
|
||||||
|
t.Fatalf("KeyMessages[%d] = %q, want %q", i, payload.KeyMessages[i], wantMessages[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if payload.ShortTerm == nil || payload.LongTerm == nil {
|
||||||
|
t.Fatalf("ShortTerm=%v LongTerm=%v, want both populated", payload.ShortTerm, payload.LongTerm)
|
||||||
|
}
|
||||||
|
if payload.ShortTerm.Qualifier != "(Tonight through Wednesday)" || payload.ShortTerm.Text != "Gusty west winds will continue through the evening before decreasing overnight." {
|
||||||
|
t.Fatalf("ShortTerm = %#v", payload.ShortTerm)
|
||||||
|
}
|
||||||
|
if payload.LongTerm.Qualifier != "(Thursday through Saturday)" || payload.LongTerm.Text != "Warmer and drier conditions return Thursday, followed by a chance of showers Friday." {
|
||||||
|
t.Fatalf("LongTerm = %#v", payload.LongTerm)
|
||||||
|
}
|
||||||
|
if payload.ShortTerm.IssuedAt == nil || payload.LongTerm.IssuedAt == nil ||
|
||||||
|
!payload.ShortTerm.IssuedAt.Equal(wantEffectiveAt) || !payload.LongTerm.IssuedAt.Equal(wantEffectiveAt) {
|
||||||
|
t.Fatalf("section issue times = short %v long %v, want %s", payload.ShortTerm.IssuedAt, payload.LongTerm.IssuedAt, wantEffectiveAt.Format(time.RFC3339))
|
||||||
|
}
|
||||||
|
|
||||||
|
b, err := json.Marshal(out.Payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
var fields map[string]any
|
||||||
|
if err := json.Unmarshal(b, &fields); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"aviation", "discussion", "sections"} {
|
||||||
|
if _, ok := fields[key]; ok {
|
||||||
|
t.Fatalf("unexpected key %q in canonical payload", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestForecastDiscussionNormalizerSupportsCrossOfficeKeyMessageFixtures(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
filename string
|
||||||
|
id string
|
||||||
|
source string
|
||||||
|
emittedAt time.Time
|
||||||
|
effectiveAt time.Time
|
||||||
|
messages []string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "numbered key messages",
|
||||||
|
filename: "forecast_discussion_bgm_numbered_sample.html",
|
||||||
|
id: "evt-discussion-bgm",
|
||||||
|
source: "nws-discussion-bgm-test",
|
||||||
|
emittedAt: time.Date(2026, 4, 10, 17, 31, 0, 0, time.UTC),
|
||||||
|
effectiveAt: time.Date(2026, 4, 10, 17, 30, 0, 0, time.UTC),
|
||||||
|
messages: []string{
|
||||||
|
"Periods of rain are expected through Saturday, with locally heavier amounts possible.",
|
||||||
|
"Cooler temperatures return late this weekend.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "key points alias",
|
||||||
|
filename: "forecast_discussion_mfr_key_points_sample.html",
|
||||||
|
id: "evt-discussion-mfr",
|
||||||
|
source: "nws-discussion-mfr-test",
|
||||||
|
emittedAt: time.Date(2026, 4, 10, 19, 46, 0, 0, time.UTC),
|
||||||
|
effectiveAt: time.Date(2026, 4, 10, 19, 45, 0, 0, time.UTC),
|
||||||
|
messages: []string{
|
||||||
|
"Gusty winds will develop over exposed ridges, especially during the afternoon.",
|
||||||
|
"Inland valleys remain dry through Saturday.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "as of preamble",
|
||||||
|
filename: "forecast_discussion_rah_as_of_sample.html",
|
||||||
|
id: "evt-discussion-rah",
|
||||||
|
source: "nws-discussion-rah-test",
|
||||||
|
emittedAt: time.Date(2026, 8, 2, 16, 36, 0, 0, time.UTC),
|
||||||
|
effectiveAt: time.Date(2026, 8, 2, 16, 35, 0, 0, time.UTC),
|
||||||
|
messages: []string{
|
||||||
|
"Scattered storms may produce locally heavy rain this afternoon.",
|
||||||
|
"Drier weather arrives Monday.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "parenthesized numeric markers",
|
||||||
|
filename: "forecast_discussion_lwx_parenthesized_number_sample.html",
|
||||||
|
id: "evt-discussion-lwx",
|
||||||
|
source: "nws-discussion-lwx-test",
|
||||||
|
emittedAt: time.Date(2026, 8, 2, 18, 1, 0, 0, time.UTC),
|
||||||
|
effectiveAt: time.Date(2026, 8, 2, 18, 0, 0, 0, time.UTC),
|
||||||
|
messages: []string{
|
||||||
|
"Thunderstorms remain possible near the Blue Ridge this evening.",
|
||||||
|
"Seasonably warm conditions continue Monday.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "embedded key messages in previous discussion",
|
||||||
|
filename: "forecast_discussion_mfr_prev_discussion_sample.html",
|
||||||
|
id: "evt-discussion-mfr-previous",
|
||||||
|
source: "nws-discussion-mfr-previous-test",
|
||||||
|
emittedAt: time.Date(2026, 8, 2, 22, 20, 0, 0, time.UTC),
|
||||||
|
effectiveAt: time.Date(2026, 8, 2, 22, 19, 0, 0, time.UTC),
|
||||||
|
messages: []string{
|
||||||
|
"Heat returns to inland valleys Monday.",
|
||||||
|
"Gusty afternoon winds develop east of the Cascades.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
in := event.Event{
|
||||||
|
ID: tt.id,
|
||||||
|
Kind: event.Kind(standards.KindForecastDiscussion),
|
||||||
|
Source: tt.source,
|
||||||
|
EmittedAt: tt.emittedAt,
|
||||||
|
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
||||||
|
Payload: loadForecastDiscussionFixtureHTML(t, tt.filename),
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, in)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
if out == nil {
|
||||||
|
t.Fatalf("Normalize() returned nil output")
|
||||||
|
}
|
||||||
|
if out.ID != in.ID || out.Source != in.Source || !out.EmittedAt.Equal(in.EmittedAt) {
|
||||||
|
t.Fatalf("envelope = %#v, want ID/source/emittedAt from input", out)
|
||||||
|
}
|
||||||
|
if out.Kind != event.Kind(standards.KindForecastDiscussion) {
|
||||||
|
t.Fatalf("Kind = %q, want forecast_discussion", out.Kind)
|
||||||
|
}
|
||||||
|
if out.Schema != standards.SchemaWeatherForecastDiscussionV1 {
|
||||||
|
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherForecastDiscussionV1)
|
||||||
|
}
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(tt.effectiveAt) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, tt.effectiveAt.Format(time.RFC3339))
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, ok := out.Payload.(model.WeatherForecastDiscussion)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Payload type = %T, want model.WeatherForecastDiscussion", out.Payload)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(payload.KeyMessages, tt.messages) {
|
||||||
|
t.Fatalf("KeyMessages = %#v, want %#v", payload.KeyMessages, tt.messages)
|
||||||
|
}
|
||||||
|
|
||||||
|
b, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("json.Marshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
var fields map[string]any
|
||||||
|
if err := json.Unmarshal(b, &fields); err != nil {
|
||||||
|
t.Fatalf("json.Unmarshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"nearTerm", "discussion", "aviation", "sections"} {
|
||||||
|
if _, ok := fields[key]; ok {
|
||||||
|
t.Fatalf("unexpected key %q in canonical payload", key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestForecastDiscussionNormalizerRejectsMissingIssueTime(t *testing.T) {
|
func TestForecastDiscussionNormalizerRejectsMissingIssueTime(t *testing.T) {
|
||||||
_, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
_, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
||||||
ID: "evt-discussion-bad",
|
ID: "evt-discussion-bad",
|
||||||
Kind: event.Kind("forecast_discussion"),
|
Kind: event.Kind(standards.KindForecastDiscussion),
|
||||||
Source: "nws-discussion-test",
|
Source: "nws-discussion-test",
|
||||||
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
||||||
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
||||||
@@ -93,7 +365,7 @@ func TestForecastDiscussionNormalizerWireShapeHasNoUnexpectedKeys(t *testing.T)
|
|||||||
|
|
||||||
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
out, err := (ForecastDiscussionNormalizer{}).Normalize(nil, event.Event{
|
||||||
ID: "evt-discussion-2",
|
ID: "evt-discussion-2",
|
||||||
Kind: event.Kind("forecast_discussion"),
|
Kind: event.Kind(standards.KindForecastDiscussion),
|
||||||
Source: "nws-discussion-test",
|
Source: "nws-discussion-test",
|
||||||
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
EmittedAt: time.Date(2026, 3, 28, 19, 25, 0, 0, time.UTC),
|
||||||
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
Schema: standards.SchemaRawNWSForecastDiscussionV1,
|
||||||
@@ -128,3 +400,56 @@ func loadForecastDiscussionSampleHTML(t *testing.T) string {
|
|||||||
}
|
}
|
||||||
return string(b)
|
return string(b)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func loadForecastDiscussionBOUSampleHTML(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
path := filepath.Join("..", "..", "providers", "nws", "testdata", "forecast_discussion_bou_sample.html")
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadForecastDiscussionFixtureHTML(t *testing.T, filename string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
path := filepath.Join("..", "..", "providers", "nws", "testdata", filename)
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("os.ReadFile(%q) error = %v", path, err)
|
||||||
|
}
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadMixedFormatForecastDiscussionSampleHTML(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
raw := loadForecastDiscussionSampleHTML(t)
|
||||||
|
replacements := []struct {
|
||||||
|
original string
|
||||||
|
replacement string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
original: ".SHORT TERM... (Through Late Sunday Night)",
|
||||||
|
replacement: ".SHORT TERM /Through Late Sunday Night/...",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
original: ".LONG TERM... (Monday through Next Saturday)",
|
||||||
|
replacement: ".LONG TERM /Monday through Next Saturday/...",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
original: ".AVIATION... (For the 18z TAFs through 18z Sunday Afternoon)",
|
||||||
|
replacement: ".AVIATION /For the 18z TAFs through 18z Sunday Afternoon/...",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, replacement := range replacements {
|
||||||
|
if !strings.Contains(raw, replacement.original) {
|
||||||
|
t.Fatalf("fixture missing heading %q", replacement.original)
|
||||||
|
}
|
||||||
|
raw = strings.Replace(raw, replacement.original, replacement.replacement, 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
|||||||
@@ -35,6 +35,9 @@ func TestBuildHourlyForecastUsesShortForecastAsTextDescription(t *testing.T) {
|
|||||||
if got, want := run.Periods[0].TextDescription, "Mostly Cloudy"; got != want {
|
if got, want := run.Periods[0].TextDescription, "Mostly Cloudy"; got != want {
|
||||||
t.Fatalf("TextDescription = %q, want %q", got, want)
|
t.Fatalf("TextDescription = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
|
if run.Periods[0].ConditionCode == nil {
|
||||||
|
t.Fatalf("ConditionCode is nil, want inferred hourly WMO code")
|
||||||
|
}
|
||||||
|
|
||||||
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
|
wantIssued := time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC)
|
||||||
if !run.IssuedAt.Equal(wantIssued) {
|
if !run.IssuedAt.Equal(wantIssued) {
|
||||||
@@ -179,7 +182,7 @@ func TestNormalizeForecastEventBySchemaProducesCanonicalWeatherForecastSchema(t
|
|||||||
t.Run(tt.name, func(t *testing.T) {
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
out, err := normalizeForecastEventBySchema(event.Event{
|
out, err := normalizeForecastEventBySchema(event.Event{
|
||||||
ID: "evt-1",
|
ID: "evt-1",
|
||||||
Kind: event.Kind("forecast"),
|
Kind: event.Kind(standards.KindForecast),
|
||||||
Source: "nws-test",
|
Source: "nws-test",
|
||||||
EmittedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
|
EmittedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
|
||||||
Schema: tt.schema,
|
Schema: tt.schema,
|
||||||
@@ -261,6 +264,9 @@ func TestBuildNarrativeForecastMapsExpectedFields(t *testing.T) {
|
|||||||
if p.ProbabilityOfPrecipitationPercent == nil || *p.ProbabilityOfPrecipitationPercent != 20 {
|
if p.ProbabilityOfPrecipitationPercent == nil || *p.ProbabilityOfPrecipitationPercent != 20 {
|
||||||
t.Fatalf("ProbabilityOfPrecipitationPercent = %v, want 20", p.ProbabilityOfPrecipitationPercent)
|
t.Fatalf("ProbabilityOfPrecipitationPercent = %v, want 20", p.ProbabilityOfPrecipitationPercent)
|
||||||
}
|
}
|
||||||
|
if p.ConditionCode != nil {
|
||||||
|
t.Fatalf("ConditionCode = %v, want nil for narrative period", p.ConditionCode)
|
||||||
|
}
|
||||||
|
|
||||||
wantIssued := time.Date(2026, 3, 27, 15, 17, 1, 0, time.UTC)
|
wantIssued := time.Date(2026, 3, 27, 15, 17, 1, 0, time.UTC)
|
||||||
if !run.IssuedAt.Equal(wantIssued) {
|
if !run.IssuedAt.Equal(wantIssued) {
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normcommon.NormalizeJSON(
|
return normcommon.NormalizeJSON(
|
||||||
in,
|
in,
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ var builtins = []fknormalize.Normalizer{
|
|||||||
ObservationNormalizer{},
|
ObservationNormalizer{},
|
||||||
ForecastNormalizer{},
|
ForecastNormalizer{},
|
||||||
ForecastDiscussionNormalizer{},
|
ForecastDiscussionNormalizer{},
|
||||||
|
WeatherStoriesNormalizer{},
|
||||||
AlertsNormalizer{},
|
AlertsNormalizer{},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -262,6 +262,25 @@ type nwsAlertProperties struct {
|
|||||||
References json.RawMessage `json:"references"`
|
References json.RawMessage `json:"references"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// nwsWeatherStoriesResponse is a minimal representation of the NWS /weatherstories
|
||||||
|
// payload needed for mapping into model.WeatherStoryRun.
|
||||||
|
type nwsWeatherStoriesResponse struct {
|
||||||
|
Stories []nwsWeatherStory `json:"stories"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type nwsWeatherStory struct {
|
||||||
|
OfficeID string `json:"officeId"`
|
||||||
|
StartTime string `json:"startTime"`
|
||||||
|
EndTime string `json:"endTime"`
|
||||||
|
UpdateTime string `json:"updateTime"`
|
||||||
|
Title string `json:"title"`
|
||||||
|
Description string `json:"description"`
|
||||||
|
AltText string `json:"altText"`
|
||||||
|
Priority bool `json:"priority"`
|
||||||
|
Order int `json:"order"`
|
||||||
|
Download string `json:"download"`
|
||||||
|
}
|
||||||
|
|
||||||
type nwsAlertReference struct {
|
type nwsAlertReference struct {
|
||||||
ID string `json:"id"`
|
ID string `json:"id"`
|
||||||
Identifier string `json:"identifier"`
|
Identifier string `json:"identifier"`
|
||||||
|
|||||||
107
internal/normalizers/nws/weatherstories.go
Normal file
107
internal/normalizers/nws/weatherstories.go
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
package nws
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
||||||
|
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
)
|
||||||
|
|
||||||
|
// WeatherStoriesNormalizer converts:
|
||||||
|
//
|
||||||
|
// standards.SchemaRawNWSWeatherStoriesV1 -> standards.SchemaWeatherStoryV1
|
||||||
|
//
|
||||||
|
// It maps the NWS /weatherstories JSON response into a canonical story snapshot.
|
||||||
|
type WeatherStoriesNormalizer struct{}
|
||||||
|
|
||||||
|
func (WeatherStoriesNormalizer) Match(e event.Event) bool {
|
||||||
|
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSWeatherStoriesV1
|
||||||
|
}
|
||||||
|
|
||||||
|
func (WeatherStoriesNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
|
_ = ctx
|
||||||
|
|
||||||
|
fallbackAsOf := in.EmittedAt.UTC()
|
||||||
|
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
|
||||||
|
fallbackAsOf = in.EffectiveAt.UTC()
|
||||||
|
}
|
||||||
|
|
||||||
|
return normcommon.NormalizeJSON(
|
||||||
|
in,
|
||||||
|
"nws weatherstories",
|
||||||
|
standards.SchemaWeatherStoryV1,
|
||||||
|
func(parsed nwsWeatherStoriesResponse) (model.WeatherStoryRun, time.Time, error) {
|
||||||
|
return buildWeatherStories(parsed, fallbackAsOf)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildWeatherStories(parsed nwsWeatherStoriesResponse, fallbackAsOf time.Time) (model.WeatherStoryRun, time.Time, error) {
|
||||||
|
stories := make([]model.WeatherStory, 0, len(parsed.Stories))
|
||||||
|
|
||||||
|
var officeID string
|
||||||
|
var asOf time.Time
|
||||||
|
for i, raw := range parsed.Stories {
|
||||||
|
startTime, err := parseRequiredNWSTime(raw.StartTime, fmt.Sprintf("stories[%d].startTime", i))
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherStoryRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
endTime, err := parseRequiredNWSTime(raw.EndTime, fmt.Sprintf("stories[%d].endTime", i))
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherStoryRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
updatedAt, err := parseRequiredNWSTime(raw.UpdateTime, fmt.Sprintf("stories[%d].updateTime", i))
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherStoryRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
storyOfficeID := strings.TrimSpace(raw.OfficeID)
|
||||||
|
if officeID == "" && storyOfficeID != "" {
|
||||||
|
officeID = storyOfficeID
|
||||||
|
}
|
||||||
|
if asOf.IsZero() || updatedAt.After(asOf) {
|
||||||
|
asOf = updatedAt
|
||||||
|
}
|
||||||
|
|
||||||
|
stories = append(stories, model.WeatherStory{
|
||||||
|
OfficeID: storyOfficeID,
|
||||||
|
StartTime: startTime,
|
||||||
|
EndTime: endTime,
|
||||||
|
UpdatedAt: updatedAt,
|
||||||
|
Title: strings.TrimSpace(raw.Title),
|
||||||
|
Description: strings.TrimSpace(raw.Description),
|
||||||
|
AltText: strings.TrimSpace(raw.AltText),
|
||||||
|
Priority: raw.Priority,
|
||||||
|
Order: raw.Order,
|
||||||
|
DownloadURL: strings.TrimSpace(raw.Download),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
if asOf.IsZero() {
|
||||||
|
asOf = fallbackAsOf.UTC()
|
||||||
|
}
|
||||||
|
|
||||||
|
run := model.WeatherStoryRun{
|
||||||
|
OfficeID: officeID,
|
||||||
|
AsOf: asOf,
|
||||||
|
Stories: stories,
|
||||||
|
}
|
||||||
|
return run, asOf, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseRequiredNWSTime(raw, field string) (time.Time, error) {
|
||||||
|
if strings.TrimSpace(raw) == "" {
|
||||||
|
return time.Time{}, fmt.Errorf("%s is required", field)
|
||||||
|
}
|
||||||
|
t, err := nwscommon.ParseTime(raw)
|
||||||
|
if err != nil {
|
||||||
|
return time.Time{}, fmt.Errorf("%s: %w", field, err)
|
||||||
|
}
|
||||||
|
return t.UTC(), nil
|
||||||
|
}
|
||||||
146
internal/normalizers/nws/weatherstories_test.go
Normal file
146
internal/normalizers/nws/weatherstories_test.go
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
package nws
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestWeatherStoriesNormalizerProducesCanonicalSchemaAndMapsSample(t *testing.T) {
|
||||||
|
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(weatherStoriesSamplePayload()))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
if out == nil {
|
||||||
|
t.Fatalf("Normalize() returned nil output")
|
||||||
|
}
|
||||||
|
if out.Schema != standards.SchemaWeatherStoryV1 {
|
||||||
|
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherStoryV1)
|
||||||
|
}
|
||||||
|
if out.Kind != event.Kind(standards.KindWeatherStory) {
|
||||||
|
t.Fatalf("Kind = %q, want weather_story", out.Kind)
|
||||||
|
}
|
||||||
|
|
||||||
|
payload, ok := out.Payload.(model.WeatherStoryRun)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
|
||||||
|
}
|
||||||
|
if payload.OfficeID != "LSX" {
|
||||||
|
t.Fatalf("OfficeID = %q, want LSX", payload.OfficeID)
|
||||||
|
}
|
||||||
|
wantAsOf := time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC)
|
||||||
|
if !payload.AsOf.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("AsOf = %s, want %s", payload.AsOf, wantAsOf)
|
||||||
|
}
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantAsOf)
|
||||||
|
}
|
||||||
|
if len(payload.Stories) != 1 {
|
||||||
|
t.Fatalf("Stories len = %d, want 1", len(payload.Stories))
|
||||||
|
}
|
||||||
|
story := payload.Stories[0]
|
||||||
|
if story.Title != "Several Chances for Rain Through Monday" {
|
||||||
|
t.Fatalf("Title = %q", story.Title)
|
||||||
|
}
|
||||||
|
if story.Description != "A stagnant weather pattern." {
|
||||||
|
t.Fatalf("Description = %q", story.Description)
|
||||||
|
}
|
||||||
|
if story.AltText != "This slide shows the forecast." {
|
||||||
|
t.Fatalf("AltText = %q", story.AltText)
|
||||||
|
}
|
||||||
|
if story.Priority {
|
||||||
|
t.Fatalf("Priority = true, want false")
|
||||||
|
}
|
||||||
|
if story.Order != 1 {
|
||||||
|
t.Fatalf("Order = %d, want 1", story.Order)
|
||||||
|
}
|
||||||
|
if story.DownloadURL != "https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f" {
|
||||||
|
t.Fatalf("DownloadURL = %q", story.DownloadURL)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWeatherStoriesNormalizerEmptyStoriesUsesFallbackAsOf(t *testing.T) {
|
||||||
|
effectiveAt := time.Date(2026, 5, 30, 12, 0, 0, 0, time.UTC)
|
||||||
|
in := weatherStoriesRawEvent(`{"stories":[]}`)
|
||||||
|
in.EffectiveAt = &effectiveAt
|
||||||
|
|
||||||
|
out, err := (WeatherStoriesNormalizer{}).Normalize(nil, in)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
payload, ok := out.Payload.(model.WeatherStoryRun)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Payload type = %T, want model.WeatherStoryRun", out.Payload)
|
||||||
|
}
|
||||||
|
if !payload.AsOf.Equal(effectiveAt) {
|
||||||
|
t.Fatalf("AsOf = %s, want fallback %s", payload.AsOf, effectiveAt)
|
||||||
|
}
|
||||||
|
if payload.Stories == nil {
|
||||||
|
t.Fatalf("Stories = nil, want empty slice")
|
||||||
|
}
|
||||||
|
if len(payload.Stories) != 0 {
|
||||||
|
t.Fatalf("Stories len = %d, want 0", len(payload.Stories))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWeatherStoriesNormalizerRejectsInvalidRequiredStoryTime(t *testing.T) {
|
||||||
|
_, err := (WeatherStoriesNormalizer{}).Normalize(nil, weatherStoriesRawEvent(`{
|
||||||
|
"stories": [{
|
||||||
|
"officeId": "LSX",
|
||||||
|
"startTime": "bad",
|
||||||
|
"endTime": "2026-05-31T11:00:00+00:00",
|
||||||
|
"updateTime": "2026-05-30T09:00:34+00:00"
|
||||||
|
}]
|
||||||
|
}`))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Normalize() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "stories[0].startTime") {
|
||||||
|
t.Fatalf("error = %q, want field context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWeatherStoriesNormalizerMatch(t *testing.T) {
|
||||||
|
n := WeatherStoriesNormalizer{}
|
||||||
|
if !n.Match(event.Event{Schema: standards.SchemaRawNWSWeatherStoriesV1}) {
|
||||||
|
t.Fatalf("Match(raw weatherstories) = false, want true")
|
||||||
|
}
|
||||||
|
if n.Match(event.Event{Schema: standards.SchemaRawNWSAlertsV1}) {
|
||||||
|
t.Fatalf("Match(raw alerts) = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func weatherStoriesRawEvent(payload string) event.Event {
|
||||||
|
return event.Event{
|
||||||
|
ID: "evt-weatherstories-1",
|
||||||
|
Kind: event.Kind(standards.KindWeatherStory),
|
||||||
|
Source: "nws-weatherstories-test",
|
||||||
|
EmittedAt: time.Date(2026, 5, 30, 9, 5, 0, 0, time.UTC),
|
||||||
|
Schema: standards.SchemaRawNWSWeatherStoriesV1,
|
||||||
|
Payload: json.RawMessage(payload),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func weatherStoriesSamplePayload() string {
|
||||||
|
return `{
|
||||||
|
"stories": [
|
||||||
|
{
|
||||||
|
"officeId": " LSX ",
|
||||||
|
"startTime": "2026-05-30T08:46:00+00:00",
|
||||||
|
"endTime": "2026-05-31T11:00:00+00:00",
|
||||||
|
"updateTime": "2026-05-30T09:00:34+00:00",
|
||||||
|
"title": " Several Chances for Rain Through Monday ",
|
||||||
|
"description": " A stagnant weather pattern. ",
|
||||||
|
"altText": " This slide shows the forecast. ",
|
||||||
|
"priority": false,
|
||||||
|
"order": 1,
|
||||||
|
"download": " https://api.weather.gov/offices/LSX/weatherstories/download/3228e499-2aae-45a8-9ff9-1c060311026f "
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}`
|
||||||
|
}
|
||||||
@@ -33,7 +33,7 @@ func (ForecastNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
// If present, prefer the existing event EmittedAt as IssuedAt.
|
// If present, prefer the existing event EmittedAt as IssuedAt.
|
||||||
var fallbackIssued time.Time
|
var fallbackIssued time.Time
|
||||||
@@ -98,6 +98,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
|
|||||||
}
|
}
|
||||||
|
|
||||||
wmo := wmoAt(parsed.Hourly.WeatherCode, i)
|
wmo := wmoAt(parsed.Hourly.WeatherCode, i)
|
||||||
|
wmoPtr := wmoCodePtr(wmo)
|
||||||
canonicalText := standards.WMOText(wmo, isDay)
|
canonicalText := standards.WMOText(wmo, isDay)
|
||||||
|
|
||||||
period := model.WeatherForecastPeriod{
|
period := model.WeatherForecastPeriod{
|
||||||
@@ -107,7 +108,7 @@ func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.W
|
|||||||
Name: "",
|
Name: "",
|
||||||
IsDay: isDay,
|
IsDay: isDay,
|
||||||
|
|
||||||
ConditionCode: wmo,
|
ConditionCode: wmoPtr,
|
||||||
TextDescription: canonicalText,
|
TextDescription: canonicalText,
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -237,3 +238,8 @@ func wmoAt(vals []*int, idx int) model.WMOCode {
|
|||||||
}
|
}
|
||||||
return model.WMOUnknown
|
return model.WMOUnknown
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func wmoCodePtr(code model.WMOCode) *model.WMOCode {
|
||||||
|
out := code
|
||||||
|
return &out
|
||||||
|
}
|
||||||
|
|||||||
@@ -35,6 +35,9 @@ func TestBuildForecastUsesCanonicalTextDescription(t *testing.T) {
|
|||||||
if got := run.Periods[0].TextDescription; got != expectedText {
|
if got := run.Periods[0].TextDescription; got != expectedText {
|
||||||
t.Fatalf("TextDescription = %q, want %q", got, expectedText)
|
t.Fatalf("TextDescription = %q, want %q", got, expectedText)
|
||||||
}
|
}
|
||||||
|
if run.Periods[0].ConditionCode == nil {
|
||||||
|
t.Fatalf("ConditionCode is nil, want mapped WMO code")
|
||||||
|
}
|
||||||
|
|
||||||
wantIssued := time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC)
|
wantIssued := time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC)
|
||||||
if !run.IssuedAt.Equal(wantIssued) {
|
if !run.IssuedAt.Equal(wantIssued) {
|
||||||
|
|||||||
@@ -40,7 +40,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normcommon.NormalizeJSON(
|
return normcommon.NormalizeJSON(
|
||||||
in,
|
in,
|
||||||
|
|||||||
@@ -8,8 +8,7 @@ import (
|
|||||||
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
||||||
)
|
)
|
||||||
|
|
||||||
// This file holds provider-specific helpers that are shared across multiple
|
// This file holds provider-specific helpers for OpenWeather normalizers.
|
||||||
// OpenWeather normalizers (observations today; forecasts/alerts later).
|
|
||||||
// Keeping these out of observation.go helps preserve the "one normalizer per file"
|
// Keeping these out of observation.go helps preserve the "one normalizer per file"
|
||||||
// convention while avoiding duplication.
|
// convention while avoiding duplication.
|
||||||
|
|
||||||
|
|||||||
@@ -37,7 +37,7 @@ func (ObservationNormalizer) Match(e event.Event) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
func (ObservationNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
|
_ = ctx // normalization is pure/CPU; keep signature aligned with Normalizer.
|
||||||
|
|
||||||
return normcommon.NormalizeJSON(
|
return normcommon.NormalizeJSON(
|
||||||
in,
|
in,
|
||||||
|
|||||||
365
internal/normalizers/spc/convective_outlook.go
Normal file
365
internal/normalizers/spc/convective_outlook.go
Normal file
@@ -0,0 +1,365 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"math"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/geo"
|
||||||
|
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
|
||||||
|
spcprovider "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/spc"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
providerSPC = "spc"
|
||||||
|
productConvective = "convective"
|
||||||
|
outlookNormalizer = "spc convective outlook"
|
||||||
|
outlookTypeUnknown = 99
|
||||||
|
)
|
||||||
|
|
||||||
|
var idTokenRE = regexp.MustCompile(`[^a-z0-9]+`)
|
||||||
|
|
||||||
|
// ConvectiveOutlookNormalizer converts:
|
||||||
|
//
|
||||||
|
// standards.SchemaRawSPCConvectiveOutlookV1 -> standards.SchemaWeatherOutlookV2
|
||||||
|
//
|
||||||
|
// It maps SPC GeoJSON outlook features containing the configured location into
|
||||||
|
// canonical outlook polygons and adds matching day-level print-page discussions.
|
||||||
|
type ConvectiveOutlookNormalizer struct{}
|
||||||
|
|
||||||
|
func (ConvectiveOutlookNormalizer) Match(e event.Event) bool {
|
||||||
|
return strings.TrimSpace(e.Schema) == standards.SchemaRawSPCConvectiveOutlookV1
|
||||||
|
}
|
||||||
|
|
||||||
|
func (ConvectiveOutlookNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
|
||||||
|
_ = ctx
|
||||||
|
|
||||||
|
fallbackAsOf := in.EmittedAt.UTC()
|
||||||
|
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
|
||||||
|
fallbackAsOf = in.EffectiveAt.UTC()
|
||||||
|
}
|
||||||
|
|
||||||
|
return normcommon.NormalizeJSON(
|
||||||
|
in,
|
||||||
|
outlookNormalizer,
|
||||||
|
standards.SchemaWeatherOutlookV2,
|
||||||
|
func(parsed spcprovider.RawConvectiveOutlookBundle) (model.WeatherOutlookRun, time.Time, error) {
|
||||||
|
return buildConvectiveOutlook(parsed, fallbackAsOf)
|
||||||
|
},
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildConvectiveOutlook(bundle spcprovider.RawConvectiveOutlookBundle, fallbackAsOf time.Time) (model.WeatherOutlookRun, time.Time, error) {
|
||||||
|
if err := validateCoordinates(bundle.Latitude, bundle.Longitude); err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
discussions, latestDiscussionUpdated, err := parseDiscussions(bundle.Discussions)
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
products := orderedProducts(bundle.Products)
|
||||||
|
point := geo.Point{Latitude: bundle.Latitude, Longitude: bundle.Longitude}
|
||||||
|
outlooks := make([]model.WeatherOutlook, 0)
|
||||||
|
var latestIssue time.Time
|
||||||
|
|
||||||
|
for _, product := range products {
|
||||||
|
if err := validateProductMetadata(product); err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
if _, ok := discussions[product.Day]; !ok {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, fmt.Errorf("product %s: discussion for day %d is required", product.Key, product.Day)
|
||||||
|
}
|
||||||
|
|
||||||
|
collection, err := spcprovider.DecodeGeoJSON(product.Body)
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, fmt.Errorf("product %s: %w", product.Key, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, feature := range collection.Features {
|
||||||
|
if spcprovider.IsEmptyGeometryCollection(feature.Geometry) {
|
||||||
|
issuedAt, err := parseRequiredSPCTime(feature.Properties.IssueISO, fmt.Sprintf("product %s feature %d.ISSUE_ISO", product.Key, i))
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
if latestIssue.IsZero() || issuedAt.After(latestIssue) {
|
||||||
|
latestIssue = issuedAt
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
outlook, err := mapFeature(product, feature, i, point)
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
if latestIssue.IsZero() || outlook.IssuedAt.After(latestIssue) {
|
||||||
|
latestIssue = outlook.IssuedAt
|
||||||
|
}
|
||||||
|
if !outlook.ContainsLocation {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
outlooks = append(outlooks, outlook)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
runDiscussions, err := buildOutlookDiscussions(outlooks, discussions)
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlookRun{}, time.Time{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
asOf := latestIssue
|
||||||
|
if asOf.IsZero() {
|
||||||
|
asOf = latestDiscussionUpdated
|
||||||
|
}
|
||||||
|
if asOf.IsZero() {
|
||||||
|
asOf = fallbackAsOf.UTC()
|
||||||
|
}
|
||||||
|
|
||||||
|
var issuedAt *time.Time
|
||||||
|
if !latestIssue.IsZero() {
|
||||||
|
t := latestIssue.UTC()
|
||||||
|
issuedAt = &t
|
||||||
|
}
|
||||||
|
|
||||||
|
lat := bundle.Latitude
|
||||||
|
lon := bundle.Longitude
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
LocationID: strings.TrimSpace(bundle.LocationID),
|
||||||
|
LocationName: strings.TrimSpace(bundle.LocationName),
|
||||||
|
Latitude: &lat,
|
||||||
|
Longitude: &lon,
|
||||||
|
AsOf: asOf.UTC(),
|
||||||
|
IssuedAt: issuedAt,
|
||||||
|
Outlooks: outlooks,
|
||||||
|
Discussions: runDiscussions,
|
||||||
|
}
|
||||||
|
return run, run.AsOf, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type parsedDiscussion struct {
|
||||||
|
Headline string
|
||||||
|
Summary string
|
||||||
|
Discussion string
|
||||||
|
UpdatedAt *time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseDiscussions(pages []spcprovider.RawDiscussionPage) (map[int]parsedDiscussion, time.Time, error) {
|
||||||
|
out := map[int]parsedDiscussion{}
|
||||||
|
var latestUpdated time.Time
|
||||||
|
for _, page := range pages {
|
||||||
|
parsed, err := spcprovider.ParseDiscussionHTML(page.Body)
|
||||||
|
if err != nil {
|
||||||
|
return nil, time.Time{}, fmt.Errorf("discussion %s: %w", page.Key, err)
|
||||||
|
}
|
||||||
|
day := page.Day
|
||||||
|
if day == 0 {
|
||||||
|
if meta, ok := spcprovider.DiscussionProductByKey(page.Key); ok {
|
||||||
|
day = meta.Day
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if day < 1 || day > 3 {
|
||||||
|
return nil, time.Time{}, fmt.Errorf("discussion %s: day must be 1, 2, or 3, got %d", page.Key, page.Day)
|
||||||
|
}
|
||||||
|
disc := parsedDiscussion{
|
||||||
|
Headline: strings.TrimSpace(parsed.Headline),
|
||||||
|
Summary: strings.TrimSpace(parsed.Summary),
|
||||||
|
Discussion: strings.TrimSpace(parsed.Discussion),
|
||||||
|
UpdatedAt: parsed.UpdatedAt,
|
||||||
|
}
|
||||||
|
out[day] = disc
|
||||||
|
if parsed.UpdatedAt != nil && (latestUpdated.IsZero() || parsed.UpdatedAt.After(latestUpdated)) {
|
||||||
|
latestUpdated = parsed.UpdatedAt.UTC()
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out, latestUpdated, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildOutlookDiscussions(outlooks []model.WeatherOutlook, discussions map[int]parsedDiscussion) ([]model.WeatherOutlookDiscussion, error) {
|
||||||
|
daysWithOutlooks := map[int]bool{}
|
||||||
|
for _, outlook := range outlooks {
|
||||||
|
daysWithOutlooks[outlook.Day] = true
|
||||||
|
}
|
||||||
|
|
||||||
|
days := make([]int, 0, len(daysWithOutlooks))
|
||||||
|
for day := range daysWithOutlooks {
|
||||||
|
days = append(days, day)
|
||||||
|
}
|
||||||
|
sort.Ints(days)
|
||||||
|
|
||||||
|
out := make([]model.WeatherOutlookDiscussion, 0, len(days))
|
||||||
|
for _, day := range days {
|
||||||
|
disc, ok := discussions[day]
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("discussion for retained day %d is required", day)
|
||||||
|
}
|
||||||
|
out = append(out, model.WeatherOutlookDiscussion{
|
||||||
|
Day: day,
|
||||||
|
Headline: disc.Headline,
|
||||||
|
Summary: disc.Summary,
|
||||||
|
Discussion: disc.Discussion,
|
||||||
|
UpdatedAt: utcTimePtr(disc.UpdatedAt),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return out, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func utcTimePtr(t *time.Time) *time.Time {
|
||||||
|
if t == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tt := t.UTC()
|
||||||
|
return &tt
|
||||||
|
}
|
||||||
|
|
||||||
|
func orderedProducts(products []spcprovider.RawOutlookProduct) []spcprovider.RawOutlookProduct {
|
||||||
|
out := make([]spcprovider.RawOutlookProduct, len(products))
|
||||||
|
copy(out, products)
|
||||||
|
sort.SliceStable(out, func(i, j int) bool {
|
||||||
|
if out[i].Day != out[j].Day {
|
||||||
|
return out[i].Day < out[j].Day
|
||||||
|
}
|
||||||
|
left := outlookTypeOrder(out[i].OutlookType)
|
||||||
|
right := outlookTypeOrder(out[j].OutlookType)
|
||||||
|
if left != right {
|
||||||
|
return left < right
|
||||||
|
}
|
||||||
|
return out[i].Key < out[j].Key
|
||||||
|
})
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func outlookTypeOrder(outlookType string) int {
|
||||||
|
switch strings.TrimSpace(outlookType) {
|
||||||
|
case spcprovider.OutlookTypeCategorical:
|
||||||
|
return 0
|
||||||
|
case spcprovider.OutlookTypeTornado:
|
||||||
|
return 1
|
||||||
|
case spcprovider.OutlookTypeHail:
|
||||||
|
return 2
|
||||||
|
case spcprovider.OutlookTypeWind:
|
||||||
|
return 3
|
||||||
|
default:
|
||||||
|
return outlookTypeUnknown
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateProductMetadata(product spcprovider.RawOutlookProduct) error {
|
||||||
|
if product.Day < 1 || product.Day > 3 {
|
||||||
|
return fmt.Errorf("product %s: day must be 1, 2, or 3, got %d", product.Key, product.Day)
|
||||||
|
}
|
||||||
|
switch strings.TrimSpace(product.OutlookType) {
|
||||||
|
case spcprovider.OutlookTypeCategorical, spcprovider.OutlookTypeTornado, spcprovider.OutlookTypeHail, spcprovider.OutlookTypeWind:
|
||||||
|
return nil
|
||||||
|
default:
|
||||||
|
return fmt.Errorf("product %s: unsupported outlook type %q", product.Key, product.OutlookType)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mapFeature(product spcprovider.RawOutlookProduct, feature spcprovider.GeoJSONFeature, index int, point geo.Point) (model.WeatherOutlook, error) {
|
||||||
|
fieldPrefix := fmt.Sprintf("product %s feature %d", product.Key, index)
|
||||||
|
props := feature.Properties
|
||||||
|
|
||||||
|
validFrom, err := parseRequiredSPCTime(props.ValidISO, fieldPrefix+".VALID_ISO")
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlook{}, err
|
||||||
|
}
|
||||||
|
validTo, err := parseRequiredSPCTime(props.ExpireISO, fieldPrefix+".EXPIRE_ISO")
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlook{}, err
|
||||||
|
}
|
||||||
|
issuedAt, err := parseRequiredSPCTime(props.IssueISO, fieldPrefix+".ISSUE_ISO")
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlook{}, err
|
||||||
|
}
|
||||||
|
label := strings.TrimSpace(props.Label)
|
||||||
|
if label == "" {
|
||||||
|
return model.WeatherOutlook{}, fmt.Errorf("%s.LABEL is required", fieldPrefix)
|
||||||
|
}
|
||||||
|
if len(feature.Geometry) == 0 {
|
||||||
|
return model.WeatherOutlook{}, fmt.Errorf("%s.geometry is required", fieldPrefix)
|
||||||
|
}
|
||||||
|
containsLocation, err := geo.ContainsPoint(feature.Geometry, point)
|
||||||
|
if err != nil {
|
||||||
|
return model.WeatherOutlook{}, fmt.Errorf("%s.geometry: %w", fieldPrefix, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
geometry := make(json.RawMessage, len(feature.Geometry))
|
||||||
|
copy(geometry, feature.Geometry)
|
||||||
|
|
||||||
|
return model.WeatherOutlook{
|
||||||
|
ID: outlookID(product.Day, product.OutlookType, label, issuedAt, validFrom, index),
|
||||||
|
Provider: providerSPC,
|
||||||
|
Product: productConvective,
|
||||||
|
Day: product.Day,
|
||||||
|
OutlookType: strings.TrimSpace(product.OutlookType),
|
||||||
|
Label: label,
|
||||||
|
LabelText: strings.TrimSpace(props.Label2),
|
||||||
|
SeverityRank: props.DN,
|
||||||
|
ValidFrom: validFrom,
|
||||||
|
ValidTo: validTo,
|
||||||
|
IssuedAt: issuedAt,
|
||||||
|
ExpiresAt: validTo,
|
||||||
|
Forecaster: strings.TrimSpace(props.Forecaster),
|
||||||
|
SourceURL: strings.TrimSpace(product.URL),
|
||||||
|
ImageURL: "",
|
||||||
|
ContainsLocation: containsLocation,
|
||||||
|
Geometry: geometry,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseRequiredSPCTime(value, field string) (time.Time, error) {
|
||||||
|
if strings.TrimSpace(value) == "" {
|
||||||
|
return time.Time{}, fmt.Errorf("%s is required", field)
|
||||||
|
}
|
||||||
|
t, err := spcprovider.ParseISOTimestamp(value)
|
||||||
|
if err != nil {
|
||||||
|
return time.Time{}, fmt.Errorf("%s: %w", field, err)
|
||||||
|
}
|
||||||
|
return t.UTC(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func outlookID(day int, outlookType, label string, issuedAt time.Time, validFrom time.Time, index int) string {
|
||||||
|
return fmt.Sprintf(
|
||||||
|
"spc-convective-day%d-%s-%s-%s-%s-%d",
|
||||||
|
day,
|
||||||
|
safeIDToken(outlookType),
|
||||||
|
safeIDToken(label),
|
||||||
|
issuedAt.UTC().Format(time.RFC3339),
|
||||||
|
validFrom.UTC().Format(time.RFC3339),
|
||||||
|
index,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
func safeIDToken(value string) string {
|
||||||
|
value = strings.ToLower(strings.TrimSpace(value))
|
||||||
|
value = idTokenRE.ReplaceAllString(value, "-")
|
||||||
|
value = strings.Trim(value, "-")
|
||||||
|
if value == "" {
|
||||||
|
return "unknown"
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateCoordinates(latitude, longitude float64) error {
|
||||||
|
switch {
|
||||||
|
case math.IsNaN(latitude) || math.IsInf(latitude, 0):
|
||||||
|
return fmt.Errorf("latitude must be finite")
|
||||||
|
case math.IsNaN(longitude) || math.IsInf(longitude, 0):
|
||||||
|
return fmt.Errorf("longitude must be finite")
|
||||||
|
case latitude < -90 || latitude > 90:
|
||||||
|
return fmt.Errorf("latitude must be between -90 and 90, got %v", latitude)
|
||||||
|
case longitude < -180 || longitude > 180:
|
||||||
|
return fmt.Errorf("longitude must be between -180 and 180, got %v", longitude)
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
526
internal/normalizers/spc/convective_outlook_test.go
Normal file
526
internal/normalizers/spc/convective_outlook_test.go
Normal file
@@ -0,0 +1,526 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/event"
|
||||||
|
spcprovider "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/spc"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerMatch(t *testing.T) {
|
||||||
|
n := ConvectiveOutlookNormalizer{}
|
||||||
|
if !n.Match(event.Event{Schema: standards.SchemaRawSPCConvectiveOutlookV1}) {
|
||||||
|
t.Fatalf("Match(raw SPC outlook) = false, want true")
|
||||||
|
}
|
||||||
|
if n.Match(event.Event{Schema: standards.SchemaRawNWSAlertsV1}) {
|
||||||
|
t.Fatalf("Match(raw NWS alerts) = true, want false")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerProducesCanonicalSchemaAndMapsSample(t *testing.T) {
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 38.5, -90.5)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
if out.Schema != standards.SchemaWeatherOutlookV2 {
|
||||||
|
t.Fatalf("Schema = %q, want %q", out.Schema, standards.SchemaWeatherOutlookV2)
|
||||||
|
}
|
||||||
|
if out.Kind != event.Kind(standards.KindOutlook) {
|
||||||
|
t.Fatalf("Kind = %q, want outlook", out.Kind)
|
||||||
|
}
|
||||||
|
|
||||||
|
run, ok := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("Payload type = %T, want model.WeatherOutlookRun", out.Payload)
|
||||||
|
}
|
||||||
|
wantAsOf := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
|
||||||
|
if !run.AsOf.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("AsOf = %s, want %s", run.AsOf, wantAsOf)
|
||||||
|
}
|
||||||
|
if run.IssuedAt == nil || !run.IssuedAt.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("IssuedAt = %v, want %s", run.IssuedAt, wantAsOf)
|
||||||
|
}
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want %s", out.EffectiveAt, wantAsOf)
|
||||||
|
}
|
||||||
|
if run.LocationID != "stl" || run.LocationName != "St. Louis, MO" {
|
||||||
|
t.Fatalf("location metadata = %q/%q", run.LocationID, run.LocationName)
|
||||||
|
}
|
||||||
|
if run.Latitude == nil || *run.Latitude != 38.5 || run.Longitude == nil || *run.Longitude != -90.5 {
|
||||||
|
t.Fatalf("coordinates = %v,%v", run.Latitude, run.Longitude)
|
||||||
|
}
|
||||||
|
if len(run.Outlooks) != 4 {
|
||||||
|
t.Fatalf("Outlooks length = %d, want 4", len(run.Outlooks))
|
||||||
|
}
|
||||||
|
assertAllOutlooksContainLocation(t, run.Outlooks)
|
||||||
|
assertDiscussionDays(t, run.Discussions, 1)
|
||||||
|
day1Discussion := run.Discussions[0]
|
||||||
|
if day1Discussion.Headline != "Day 1 Convective Outlook" {
|
||||||
|
t.Fatalf("day 1 Headline = %q", day1Discussion.Headline)
|
||||||
|
}
|
||||||
|
if !strings.Contains(day1Discussion.Summary, "central Plains") {
|
||||||
|
t.Fatalf("day 1 Summary = %q", day1Discussion.Summary)
|
||||||
|
}
|
||||||
|
if !strings.Contains(day1Discussion.Discussion, "...DISCUSSION...") {
|
||||||
|
t.Fatalf("day 1 Discussion missing product text: %q", day1Discussion.Discussion)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(day1Discussion.Discussion, "SPC AC 111234") {
|
||||||
|
t.Fatalf("day 1 Discussion = %q, want SPC product code prefix", day1Discussion.Discussion)
|
||||||
|
}
|
||||||
|
|
||||||
|
got := run.Outlooks[0]
|
||||||
|
if got.Provider != "spc" || got.Product != "convective" {
|
||||||
|
t.Fatalf("provider/product = %q/%q", got.Provider, got.Product)
|
||||||
|
}
|
||||||
|
if got.Day != 1 || got.OutlookType != spcprovider.OutlookTypeCategorical {
|
||||||
|
t.Fatalf("day/type = %d/%q", got.Day, got.OutlookType)
|
||||||
|
}
|
||||||
|
if got.Label != "SLGT" || got.LabelText != "Slight Risk" {
|
||||||
|
t.Fatalf("label fields = %q/%q", got.Label, got.LabelText)
|
||||||
|
}
|
||||||
|
if got.SeverityRank == nil || *got.SeverityRank != 3 {
|
||||||
|
t.Fatalf("SeverityRank = %v, want 3", got.SeverityRank)
|
||||||
|
}
|
||||||
|
assertTime(t, "ValidFrom", got.ValidFrom, 2026, 6, 11, 13, 0, 0)
|
||||||
|
assertTime(t, "ValidTo", got.ValidTo, 2026, 6, 12, 12, 0, 0)
|
||||||
|
assertTime(t, "IssuedAt", got.IssuedAt, 2026, 6, 11, 12, 34, 56)
|
||||||
|
assertTime(t, "ExpiresAt", got.ExpiresAt, 2026, 6, 12, 12, 0, 0)
|
||||||
|
if got.Forecaster != "SMITH" {
|
||||||
|
t.Fatalf("Forecaster = %q, want SMITH", got.Forecaster)
|
||||||
|
}
|
||||||
|
if got.SourceURL != "https://example.invalid/day1_categorical.geojson" {
|
||||||
|
t.Fatalf("SourceURL = %q", got.SourceURL)
|
||||||
|
}
|
||||||
|
wantGeometry := `{"type":"Polygon","coordinates":[[[-91.0,38.0],[-90.0,38.0],[-90.0,39.0],[-91.0,39.0],[-91.0,38.0]]]}`
|
||||||
|
if string(got.Geometry) != wantGeometry {
|
||||||
|
t.Fatalf("Geometry = %s, want %s", got.Geometry, wantGeometry)
|
||||||
|
}
|
||||||
|
if !got.ContainsLocation {
|
||||||
|
t.Fatalf("ContainsLocation = false, want true")
|
||||||
|
}
|
||||||
|
if got.ID != "spc-convective-day1-categorical-slgt-2026-06-11T12:34:56Z-2026-06-11T13:00:00Z-0" {
|
||||||
|
t.Fatalf("ID = %q", got.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerAcceptsTypedSourcePayload(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 38.5, -90.5)
|
||||||
|
in := spcRawEvent(t, bundle)
|
||||||
|
in.Payload = bundle
|
||||||
|
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, in)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
if len(run.Outlooks) != 4 {
|
||||||
|
t.Fatalf("Outlooks length = %d, want 4", len(run.Outlooks))
|
||||||
|
}
|
||||||
|
assertAllOutlooksContainLocation(t, run.Outlooks)
|
||||||
|
assertDiscussionDays(t, run.Discussions, 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerEmitsEmptyLocalRunOutsidePolygons(t *testing.T) {
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 0, 0)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
if len(run.Outlooks) != 0 {
|
||||||
|
t.Fatalf("Outlooks length = %d, want 0", len(run.Outlooks))
|
||||||
|
}
|
||||||
|
if len(run.Discussions) != 0 {
|
||||||
|
t.Fatalf("Discussions length = %d, want 0", len(run.Discussions))
|
||||||
|
}
|
||||||
|
wantAsOf := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
|
||||||
|
if !run.AsOf.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("AsOf = %s, want latest product issue time %s", run.AsOf, wantAsOf)
|
||||||
|
}
|
||||||
|
if run.IssuedAt == nil || !run.IssuedAt.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("IssuedAt = %v, want latest product issue time %s", run.IssuedAt, wantAsOf)
|
||||||
|
}
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(run.AsOf) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want run AsOf %s", out.EffectiveAt, run.AsOf)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerOrdersProductsByDayAndType(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 38.5, -90.5)
|
||||||
|
for i, j := 0, len(bundle.Products)-1; i < j; i, j = i+1, j-1 {
|
||||||
|
bundle.Products[i], bundle.Products[j] = bundle.Products[j], bundle.Products[i]
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
got := []string{
|
||||||
|
run.Outlooks[0].OutlookType,
|
||||||
|
run.Outlooks[1].OutlookType,
|
||||||
|
run.Outlooks[2].OutlookType,
|
||||||
|
run.Outlooks[3].OutlookType,
|
||||||
|
}
|
||||||
|
want := []string{
|
||||||
|
spcprovider.OutlookTypeCategorical,
|
||||||
|
spcprovider.OutlookTypeTornado,
|
||||||
|
spcprovider.OutlookTypeHail,
|
||||||
|
spcprovider.OutlookTypeWind,
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if got[i] != want[i] || run.Outlooks[i].Day != 1 {
|
||||||
|
t.Fatalf("outlook[%d] = day %d type %q, want day 1 type %q", i, run.Outlooks[i].Day, got[i], want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerMapsProbabilisticOutlookTypes(t *testing.T) {
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 38.5, -90.5)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
assertAllOutlooksContainLocation(t, run.Outlooks)
|
||||||
|
for _, outlookType := range []string{
|
||||||
|
spcprovider.OutlookTypeTornado,
|
||||||
|
spcprovider.OutlookTypeHail,
|
||||||
|
spcprovider.OutlookTypeWind,
|
||||||
|
} {
|
||||||
|
if findOutlook(run.Outlooks, 1, outlookType) == nil {
|
||||||
|
t.Fatalf("missing day 1 outlook type %q", outlookType)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerSkipsEmptyGeometryCollectionPlaceholder(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 36, -99)
|
||||||
|
replaced := false
|
||||||
|
for i := range bundle.Products {
|
||||||
|
if bundle.Products[i].Day == 2 && bundle.Products[i].OutlookType == spcprovider.OutlookTypeTornado {
|
||||||
|
bundle.Products[i].Body = json.RawMessage(emptyGeometryCollectionGeoJSON())
|
||||||
|
replaced = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !replaced {
|
||||||
|
t.Fatalf("test setup did not find day 2 tornado product")
|
||||||
|
}
|
||||||
|
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
if len(run.Outlooks) != 3 {
|
||||||
|
t.Fatalf("Outlooks length = %d, want 3", len(run.Outlooks))
|
||||||
|
}
|
||||||
|
assertAllOutlooksContainLocation(t, run.Outlooks)
|
||||||
|
assertDiscussionDays(t, run.Discussions, 2)
|
||||||
|
if got := findOutlook(run.Outlooks, 2, spcprovider.OutlookTypeTornado); got != nil {
|
||||||
|
t.Fatalf("day 2 tornado outlook = %+v, want nil placeholder skipped", *got)
|
||||||
|
}
|
||||||
|
wantAsOf := time.Date(2026, 6, 12, 10, 0, 0, 0, time.UTC)
|
||||||
|
if !run.AsOf.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("AsOf = %s, want placeholder ISSUE_ISO %s", run.AsOf, wantAsOf)
|
||||||
|
}
|
||||||
|
if out.EffectiveAt == nil || !out.EffectiveAt.Equal(wantAsOf) {
|
||||||
|
t.Fatalf("EffectiveAt = %v, want placeholder ISSUE_ISO %s", out.EffectiveAt, wantAsOf)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerIncludesOnlyDayWithContainingPolygons(t *testing.T) {
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 36, -99)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
if len(run.Outlooks) == 0 {
|
||||||
|
t.Fatalf("Outlooks length = 0, want retained day 2 outlooks")
|
||||||
|
}
|
||||||
|
assertAllOutlooksContainLocation(t, run.Outlooks)
|
||||||
|
for i, outlook := range run.Outlooks {
|
||||||
|
if outlook.Day != 2 {
|
||||||
|
t.Fatalf("outlook[%d].Day = %d, want 2", i, outlook.Day)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assertDiscussionDays(t, run.Discussions, 2)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerUsesOneDiscussionForMultipleSameDayOutlooks(t *testing.T) {
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 38.5, -90.5)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
if got := countOutlooksByDay(run.Outlooks, 1); got < 2 {
|
||||||
|
t.Fatalf("day 1 outlook count = %d, want multiple", got)
|
||||||
|
}
|
||||||
|
assertAllOutlooksContainLocation(t, run.Outlooks)
|
||||||
|
assertDiscussionDays(t, run.Discussions, 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerPreservesCorrectionMarker(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 36, -99)
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
run := out.Payload.(model.WeatherOutlookRun)
|
||||||
|
got := findOutlook(run.Outlooks, 2, spcprovider.OutlookTypeTornado)
|
||||||
|
if got == nil {
|
||||||
|
t.Fatalf("missing day 2 tornado outlook")
|
||||||
|
}
|
||||||
|
assertDiscussionDays(t, run.Discussions, 2)
|
||||||
|
if !strings.Contains(run.Discussions[0].Headline, "CORR 1") {
|
||||||
|
t.Fatalf("day 2 headline = %q, want correction marker", run.Discussions[0].Headline)
|
||||||
|
}
|
||||||
|
if !strings.Contains(run.Discussions[0].Discussion, "CORR 1") {
|
||||||
|
t.Fatalf("day 2 discussion = %q, want correction marker", run.Discussions[0].Discussion)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerMissingRSSNormalizes(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 0, 0)
|
||||||
|
bundle.RSS = nil
|
||||||
|
if _, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle)); err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerInvalidRequiredTimestampFailsWithContext(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 0, 0)
|
||||||
|
bundle.Products[0].Body = json.RawMessage(strings.Replace(
|
||||||
|
string(bundle.Products[0].Body),
|
||||||
|
`"ISSUE_ISO": "2026-06-11T12:34:56Z"`,
|
||||||
|
`"ISSUE_ISO": "bad"`,
|
||||||
|
1,
|
||||||
|
))
|
||||||
|
|
||||||
|
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Normalize() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "product day1_categorical feature 0.ISSUE_ISO") {
|
||||||
|
t.Fatalf("error = %q, want product and feature context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerInvalidGeometryFailsWithContext(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 0, 0)
|
||||||
|
bundle.Products[0].Body = json.RawMessage(strings.Replace(
|
||||||
|
string(bundle.Products[0].Body),
|
||||||
|
`"geometry": {`,
|
||||||
|
`"geometry": {"type":"LineString","coordinates":[[-91,38],[-90,39]]}, "oldGeometry": {`,
|
||||||
|
1,
|
||||||
|
))
|
||||||
|
|
||||||
|
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Normalize() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "product day1_categorical feature 0.geometry") {
|
||||||
|
t.Fatalf("error = %q, want product and feature context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerRejectsMissingLabel(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 0, 0)
|
||||||
|
bundle.Products[0].Body = json.RawMessage(strings.Replace(
|
||||||
|
string(bundle.Products[0].Body),
|
||||||
|
`"LABEL": "SLGT"`,
|
||||||
|
`"LABEL": ""`,
|
||||||
|
1,
|
||||||
|
))
|
||||||
|
|
||||||
|
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Normalize() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "product day1_categorical feature 0.LABEL") {
|
||||||
|
t.Fatalf("error = %q, want label context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerRejectsMissingDiscussion(t *testing.T) {
|
||||||
|
bundle := spcBundle(t, 38.5, -90.5)
|
||||||
|
bundle.Discussions = bundle.Discussions[1:]
|
||||||
|
|
||||||
|
_, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, bundle))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Normalize() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "product day1_categorical: discussion for day 1 is required") {
|
||||||
|
t.Fatalf("error = %q, want missing discussion context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConvectiveOutlookNormalizerOutputJSONShape(t *testing.T) {
|
||||||
|
out, err := (ConvectiveOutlookNormalizer{}).Normalize(nil, spcRawEvent(t, spcBundle(t, 38.5, -90.5)))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v", err)
|
||||||
|
}
|
||||||
|
raw, err := json.Marshal(out.Payload)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal(payload) error = %v", err)
|
||||||
|
}
|
||||||
|
got := string(raw)
|
||||||
|
for _, want := range []string{`"asOf"`, `"outlooks"`, `"discussions"`, `"containsLocation"`, `"geometry"`} {
|
||||||
|
if !strings.Contains(got, want) {
|
||||||
|
t.Fatalf("payload JSON missing %s: %s", want, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
outlookStart := strings.Index(got, `"outlooks"`)
|
||||||
|
discussionStart := strings.Index(got, `"discussions"`)
|
||||||
|
if outlookStart == -1 || discussionStart == -1 || discussionStart <= outlookStart {
|
||||||
|
t.Fatalf("payload JSON has unexpected outlook/discussion order: %s", got)
|
||||||
|
}
|
||||||
|
outlookJSON := got[outlookStart:discussionStart]
|
||||||
|
for _, unwanted := range []string{`"headline"`, `"summary"`, `"discussion"`} {
|
||||||
|
if strings.Contains(outlookJSON, unwanted) {
|
||||||
|
t.Fatalf("outlook JSON exposed polygon-level prose key %s: %s", unwanted, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, unwanted := range []string{`"products"`, `"fetchedAt"`, `"body"`} {
|
||||||
|
if strings.Contains(got, unwanted) {
|
||||||
|
t.Fatalf("payload JSON exposed raw key %s: %s", unwanted, got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func emptyGeometryCollectionGeoJSON() []byte {
|
||||||
|
return []byte(`{
|
||||||
|
"type": "FeatureCollection",
|
||||||
|
"features": [
|
||||||
|
{
|
||||||
|
"type": "Feature",
|
||||||
|
"properties": {
|
||||||
|
"VALID_ISO": "2026-06-12T12:00:00Z",
|
||||||
|
"EXPIRE_ISO": "2026-06-13T12:00:00Z",
|
||||||
|
"ISSUE_ISO": "2026-06-12T10:00:00Z",
|
||||||
|
"FORECASTER": "DOE",
|
||||||
|
"LABEL": "Less Than 2% All Areas",
|
||||||
|
"LABEL2": "",
|
||||||
|
"DN": 0
|
||||||
|
},
|
||||||
|
"geometry": {
|
||||||
|
"type": "GeometryCollection",
|
||||||
|
"geometries": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}`)
|
||||||
|
}
|
||||||
|
|
||||||
|
func spcRawEvent(t *testing.T, bundle spcprovider.RawConvectiveOutlookBundle) event.Event {
|
||||||
|
t.Helper()
|
||||||
|
raw, err := json.Marshal(bundle)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal(bundle) error = %v", err)
|
||||||
|
}
|
||||||
|
effectiveAt := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
|
||||||
|
return event.Event{
|
||||||
|
ID: "evt-spc-outlook-1",
|
||||||
|
Kind: event.Kind(standards.KindOutlook),
|
||||||
|
Source: "spc-test",
|
||||||
|
EmittedAt: time.Date(2026, 6, 11, 20, 5, 0, 0, time.UTC),
|
||||||
|
EffectiveAt: &effectiveAt,
|
||||||
|
Schema: standards.SchemaRawSPCConvectiveOutlookV1,
|
||||||
|
Payload: json.RawMessage(raw),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func spcBundle(t *testing.T, latitude, longitude float64) spcprovider.RawConvectiveOutlookBundle {
|
||||||
|
t.Helper()
|
||||||
|
fetchedAt := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
|
||||||
|
products := make([]spcprovider.RawOutlookProduct, 0, len(spcprovider.GeoJSONProducts()))
|
||||||
|
for _, product := range spcprovider.GeoJSONProducts() {
|
||||||
|
products = append(products, spcprovider.RawOutlookProduct{
|
||||||
|
Key: product.Key,
|
||||||
|
Day: product.Day,
|
||||||
|
OutlookType: product.OutlookType,
|
||||||
|
URL: "https://example.invalid/" + product.Key + ".geojson",
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
Body: json.RawMessage(geoJSONFixtureForProduct(t, product.Key)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return spcprovider.RawConvectiveOutlookBundle{
|
||||||
|
LocationID: "stl",
|
||||||
|
LocationName: "St. Louis, MO",
|
||||||
|
Latitude: latitude,
|
||||||
|
Longitude: longitude,
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
Products: products,
|
||||||
|
Discussions: []spcprovider.RawDiscussionPage{
|
||||||
|
{Key: "day1", Day: 1, URL: "https://example.invalid/day1.html", FetchedAt: fetchedAt, Body: string(readSPCTestFixture(t, "day1_prt.html"))},
|
||||||
|
{Key: "day2", Day: 2, URL: "https://example.invalid/day2.html", FetchedAt: fetchedAt, Body: string(readSPCTestFixture(t, "day2_prt_corr.html"))},
|
||||||
|
{Key: "day3", Day: 3, URL: "https://example.invalid/day3.html", FetchedAt: fetchedAt, Body: string(readSPCTestFixture(t, "day3_prt.html"))},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func geoJSONFixtureForProduct(t *testing.T, key string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
switch {
|
||||||
|
case strings.HasPrefix(key, "day1_"):
|
||||||
|
return readSPCTestFixture(t, "day1_cat.geojson")
|
||||||
|
case strings.HasPrefix(key, "day2_"):
|
||||||
|
return readSPCTestFixture(t, "day2_torn.geojson")
|
||||||
|
case strings.HasPrefix(key, "day3_"):
|
||||||
|
return readSPCTestFixture(t, "day3_cat.geojson")
|
||||||
|
default:
|
||||||
|
t.Fatalf("unknown product key %q", key)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func findOutlook(outlooks []model.WeatherOutlook, day int, outlookType string) *model.WeatherOutlook {
|
||||||
|
for i := range outlooks {
|
||||||
|
if outlooks[i].Day == day && outlooks[i].OutlookType == outlookType {
|
||||||
|
return &outlooks[i]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func countOutlooksByDay(outlooks []model.WeatherOutlook, day int) int {
|
||||||
|
count := 0
|
||||||
|
for _, outlook := range outlooks {
|
||||||
|
if outlook.Day == day {
|
||||||
|
count++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return count
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertAllOutlooksContainLocation(t *testing.T, outlooks []model.WeatherOutlook) {
|
||||||
|
t.Helper()
|
||||||
|
for i, outlook := range outlooks {
|
||||||
|
if !outlook.ContainsLocation {
|
||||||
|
t.Fatalf("outlook[%d].ContainsLocation = false, want true", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertDiscussionDays(t *testing.T, discussions []model.WeatherOutlookDiscussion, want ...int) {
|
||||||
|
t.Helper()
|
||||||
|
if len(discussions) != len(want) {
|
||||||
|
t.Fatalf("Discussions length = %d, want %d", len(discussions), len(want))
|
||||||
|
}
|
||||||
|
for i, day := range want {
|
||||||
|
if discussions[i].Day != day {
|
||||||
|
t.Fatalf("Discussions[%d].Day = %d, want %d", i, discussions[i].Day, day)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTime(t *testing.T, name string, got time.Time, year int, month time.Month, day int, hour int, minute int, second int) {
|
||||||
|
t.Helper()
|
||||||
|
want := time.Date(year, month, day, hour, minute, second, 0, time.UTC)
|
||||||
|
if !got.Equal(want) {
|
||||||
|
t.Fatalf("%s = %s, want %s", name, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
17
internal/normalizers/spc/fixture_test.go
Normal file
17
internal/normalizers/spc/fixture_test.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func readSPCTestFixture(t *testing.T, name string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join("..", "..", "providers", "spc", "testdata", name)
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read fixture %s: %v", path, err)
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
}
|
||||||
14
internal/normalizers/spc/register.go
Normal file
14
internal/normalizers/spc/register.go
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
fknormalize "gitea.maximumdirect.net/ejr/feedkit/processors/normalize"
|
||||||
|
)
|
||||||
|
|
||||||
|
var builtins = []fknormalize.Normalizer{
|
||||||
|
ConvectiveOutlookNormalizer{},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Register appends SPC normalizers in stable order.
|
||||||
|
func Register(in []fknormalize.Normalizer) []fknormalize.Normalizer {
|
||||||
|
return append(in, builtins...)
|
||||||
|
}
|
||||||
@@ -27,11 +27,34 @@ type ForecastDiscussionSection struct {
|
|||||||
Text string
|
Text string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type forecastDiscussionSectionRole uint8
|
||||||
|
|
||||||
|
const (
|
||||||
|
forecastDiscussionSectionRoleKeyMessages forecastDiscussionSectionRole = iota
|
||||||
|
forecastDiscussionSectionRoleShortTerm
|
||||||
|
forecastDiscussionSectionRoleLongTerm
|
||||||
|
)
|
||||||
|
|
||||||
|
type forecastDiscussionSectionHeading struct {
|
||||||
|
section string
|
||||||
|
qualifier string
|
||||||
|
}
|
||||||
|
|
||||||
|
type forecastDiscussionSectionBlock struct {
|
||||||
|
heading forecastDiscussionSectionHeading
|
||||||
|
body []string
|
||||||
|
}
|
||||||
|
|
||||||
var (
|
var (
|
||||||
forecastDiscussionHeaderRE = regexp.MustCompile(`^\.(KEY MESSAGES|SHORT TERM|LONG TERM|AVIATION)\.\.\.(.*)$`)
|
forecastDiscussionSectionRoles = map[string]forecastDiscussionSectionRole{
|
||||||
forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`)
|
"KEY MESSAGES": forecastDiscussionSectionRoleKeyMessages,
|
||||||
forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`)
|
"KEY POINTS": forecastDiscussionSectionRoleKeyMessages,
|
||||||
forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`)
|
"SHORT TERM": forecastDiscussionSectionRoleShortTerm,
|
||||||
|
"LONG TERM": forecastDiscussionSectionRoleLongTerm,
|
||||||
|
}
|
||||||
|
forecastDiscussionAFDRE = regexp.MustCompile(`^AFD([A-Z]{3})$`)
|
||||||
|
forecastDiscussionWMORE = regexp.MustCompile(`\bK([A-Z]{3})\b`)
|
||||||
|
forecastDiscussionSigRE = regexp.MustCompile(`^[A-Z]{2,6}$`)
|
||||||
)
|
)
|
||||||
|
|
||||||
func ParseForecastDiscussionHTML(raw string) (ForecastDiscussion, error) {
|
func ParseForecastDiscussionHTML(raw string) (ForecastDiscussion, error) {
|
||||||
@@ -99,22 +122,30 @@ func ParseForecastDiscussionText(text string) (ForecastDiscussion, error) {
|
|||||||
IssuedAt: issuedAt.UTC(),
|
IssuedAt: issuedAt.UTC(),
|
||||||
}
|
}
|
||||||
|
|
||||||
if block, ok := extractForecastDiscussionSection(lines, "KEY MESSAGES"); ok {
|
seenRoles := make(map[forecastDiscussionSectionRole]bool, len(forecastDiscussionSectionRoles))
|
||||||
out.KeyMessages = parseForecastDiscussionKeyMessages(block)
|
for _, block := range parseForecastDiscussionSectionBlocks(lines) {
|
||||||
}
|
role, ok := forecastDiscussionSectionRoles[block.heading.section]
|
||||||
if block, ok := extractForecastDiscussionSection(lines, "SHORT TERM"); ok {
|
if !ok || seenRoles[role] {
|
||||||
section, err := parseForecastDiscussionTextSection(block)
|
continue
|
||||||
if err != nil {
|
|
||||||
return ForecastDiscussion{}, fmt.Errorf("parse SHORT TERM: %w", err)
|
|
||||||
}
|
}
|
||||||
out.ShortTerm = §ion
|
seenRoles[role] = true
|
||||||
}
|
|
||||||
if block, ok := extractForecastDiscussionSection(lines, "LONG TERM"); ok {
|
switch role {
|
||||||
section, err := parseForecastDiscussionTextSection(block)
|
case forecastDiscussionSectionRoleKeyMessages:
|
||||||
if err != nil {
|
out.KeyMessages = parseForecastDiscussionKeyMessages(block.body)
|
||||||
return ForecastDiscussion{}, fmt.Errorf("parse LONG TERM: %w", err)
|
case forecastDiscussionSectionRoleShortTerm:
|
||||||
|
section, err := parseForecastDiscussionTextSection(block)
|
||||||
|
if err != nil {
|
||||||
|
return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", block.heading.section, err)
|
||||||
|
}
|
||||||
|
out.ShortTerm = §ion
|
||||||
|
case forecastDiscussionSectionRoleLongTerm:
|
||||||
|
section, err := parseForecastDiscussionTextSection(block)
|
||||||
|
if err != nil {
|
||||||
|
return ForecastDiscussion{}, fmt.Errorf("parse %s: %w", block.heading.section, err)
|
||||||
|
}
|
||||||
|
out.LongTerm = §ion
|
||||||
}
|
}
|
||||||
out.LongTerm = §ion
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return out, nil
|
return out, nil
|
||||||
@@ -285,8 +316,9 @@ func parseForecastDiscussionHeader(lines []string) (string, time.Time, error) {
|
|||||||
|
|
||||||
func parseForecastDiscussionIssueTime(line string) (time.Time, error) {
|
func parseForecastDiscussionIssueTime(line string) (time.Time, error) {
|
||||||
line = strings.TrimSpace(line)
|
line = strings.TrimSpace(line)
|
||||||
line = strings.TrimPrefix(line, "Issued at ")
|
if isForecastDiscussionIssuedAtLine(line) {
|
||||||
line = strings.TrimSpace(line)
|
line = strings.TrimSpace(line[len("Issued at"):])
|
||||||
|
}
|
||||||
|
|
||||||
parts := strings.Fields(line)
|
parts := strings.Fields(line)
|
||||||
if len(parts) != 7 {
|
if len(parts) != 7 {
|
||||||
@@ -386,40 +418,217 @@ func forecastDiscussionLocation(abbrev string) (*time.Location, error) {
|
|||||||
return time.FixedZone(abbr, offset), nil
|
return time.FixedZone(abbr, offset), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func extractForecastDiscussionSection(lines []string, section string) ([]string, bool) {
|
func parseForecastDiscussionSectionHeading(line string) (forecastDiscussionSectionHeading, bool) {
|
||||||
target := "." + section + "..."
|
line = strings.TrimSpace(line)
|
||||||
for i, raw := range lines {
|
if len(line) < 2 || line[0] != '.' {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.HasSuffix(line, "/...") {
|
||||||
|
return parseForecastDiscussionSlashQualifiedHeading(line)
|
||||||
|
}
|
||||||
|
if strings.HasSuffix(line, "...") {
|
||||||
|
if heading, ok := parseForecastDiscussionParenthesizedTerminalHeading(line); ok {
|
||||||
|
return heading, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return parseForecastDiscussionEllipsisHeading(line)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseForecastDiscussionSlashQualifiedHeading(line string) (forecastDiscussionSectionHeading, bool) {
|
||||||
|
content := strings.TrimSuffix(line[1:], "/...")
|
||||||
|
separator := -1
|
||||||
|
for i := 1; i < len(content); i++ {
|
||||||
|
if content[i] == '/' && isForecastDiscussionHorizontalWhitespace(content[i-1]) {
|
||||||
|
separator = i
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if separator < 0 {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
section, ok := normalizeForecastDiscussionSectionIdentity(content[:separator])
|
||||||
|
if !ok {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
qualifier := strings.TrimSpace(content[separator+1:])
|
||||||
|
if qualifier == "" {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
return forecastDiscussionSectionHeading{section: section, qualifier: qualifier}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseForecastDiscussionParenthesizedTerminalHeading(line string) (forecastDiscussionSectionHeading, bool) {
|
||||||
|
if len(line) < 4 || line[0] != '.' || !strings.HasSuffix(line, "...") {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
content := strings.TrimRight(line[1:len(line)-3], " \t")
|
||||||
|
if !strings.HasSuffix(content, ")") {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
separator := -1
|
||||||
|
for i := 1; i < len(content); i++ {
|
||||||
|
if content[i] == '(' && isForecastDiscussionHorizontalWhitespace(content[i-1]) {
|
||||||
|
separator = i
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if separator < 0 {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
section, ok := normalizeForecastDiscussionSectionIdentity(content[:separator])
|
||||||
|
if !ok {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
qualifier := content[separator:]
|
||||||
|
if len(qualifier) <= 2 || strings.TrimSpace(qualifier[1:len(qualifier)-1]) == "" {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
return forecastDiscussionSectionHeading{section: section, qualifier: qualifier}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseForecastDiscussionEllipsisHeading(line string) (forecastDiscussionSectionHeading, bool) {
|
||||||
|
content := line[1:]
|
||||||
|
delimiter := strings.Index(content, "...")
|
||||||
|
if delimiter < 0 {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
section, ok := normalizeForecastDiscussionSectionIdentity(content[:delimiter])
|
||||||
|
if !ok {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
return forecastDiscussionSectionHeading{
|
||||||
|
section: section,
|
||||||
|
qualifier: strings.TrimSpace(content[delimiter+3:]),
|
||||||
|
}, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeForecastDiscussionSectionIdentity(raw string) (string, bool) {
|
||||||
|
var normalized strings.Builder
|
||||||
|
pendingSpace := false
|
||||||
|
hasLetterOrDigit := false
|
||||||
|
|
||||||
|
for i := 0; i < len(raw); i++ {
|
||||||
|
b := raw[i]
|
||||||
|
switch {
|
||||||
|
case isForecastDiscussionIdentityLetterOrDigit(b):
|
||||||
|
hasLetterOrDigit = true
|
||||||
|
case b == ' ' || b == '\t':
|
||||||
|
pendingSpace = normalized.Len() > 0
|
||||||
|
continue
|
||||||
|
case b == '/' && i > 0 && isForecastDiscussionHorizontalWhitespace(raw[i-1]):
|
||||||
|
return "", false
|
||||||
|
case b != '/' && b != '&' && b != '\'' && b != '-':
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
if pendingSpace {
|
||||||
|
normalized.WriteByte(' ')
|
||||||
|
pendingSpace = false
|
||||||
|
}
|
||||||
|
normalized.WriteByte(b)
|
||||||
|
}
|
||||||
|
if !hasLetterOrDigit {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return normalized.String(), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func isForecastDiscussionIdentityLetterOrDigit(b byte) bool {
|
||||||
|
return b >= 'A' && b <= 'Z' || b >= '0' && b <= '9'
|
||||||
|
}
|
||||||
|
|
||||||
|
func isForecastDiscussionHorizontalWhitespace(b byte) bool {
|
||||||
|
return b == ' ' || b == '\t'
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseForecastDiscussionSectionBlocks(lines []string) []forecastDiscussionSectionBlock {
|
||||||
|
var blocks []forecastDiscussionSectionBlock
|
||||||
|
var active *forecastDiscussionSectionBlock
|
||||||
|
embeddedHeadings := false
|
||||||
|
|
||||||
|
finish := func() {
|
||||||
|
if active == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
blocks = append(blocks, *active)
|
||||||
|
active = nil
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, raw := range lines {
|
||||||
line := strings.TrimSpace(raw)
|
line := strings.TrimSpace(raw)
|
||||||
if !strings.HasPrefix(line, target) {
|
if line == "$$" {
|
||||||
|
finish()
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if line == "&&" || strings.Contains(line, "WATCHES/WARNINGS/ADVISORIES") {
|
||||||
|
finish()
|
||||||
|
embeddedHeadings = false
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
out := []string{line}
|
heading, ok := parseForecastDiscussionSectionHeading(raw)
|
||||||
for j := i + 1; j < len(lines); j++ {
|
if ok {
|
||||||
next := strings.TrimSpace(lines[j])
|
finish()
|
||||||
if next == "&&" || next == "$$" || strings.Contains(next, "WATCHES/WARNINGS/ADVISORIES") {
|
active = &forecastDiscussionSectionBlock{heading: heading}
|
||||||
break
|
embeddedHeadings = isForecastDiscussionEmbeddedSectionWrapper(heading.section)
|
||||||
}
|
continue
|
||||||
if j > i+1 && isForecastDiscussionSectionHeader(next) {
|
}
|
||||||
break
|
if embeddedHeadings {
|
||||||
}
|
heading, ok = parseForecastDiscussionEmbeddedSectionHeading(raw)
|
||||||
out = append(out, lines[j])
|
if ok {
|
||||||
|
finish()
|
||||||
|
active = &forecastDiscussionSectionBlock{heading: heading}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if active != nil {
|
||||||
|
active.body = append(active.body, raw)
|
||||||
}
|
}
|
||||||
return out, true
|
|
||||||
}
|
}
|
||||||
return nil, false
|
finish()
|
||||||
|
|
||||||
|
return blocks
|
||||||
}
|
}
|
||||||
|
|
||||||
func isForecastDiscussionSectionHeader(line string) bool {
|
func isForecastDiscussionEmbeddedSectionWrapper(section string) bool {
|
||||||
return forecastDiscussionHeaderRE.MatchString(strings.TrimSpace(line))
|
return section == "PREV DISCUSSION"
|
||||||
}
|
}
|
||||||
|
|
||||||
func parseForecastDiscussionKeyMessages(block []string) []string {
|
func parseForecastDiscussionEmbeddedSectionHeading(line string) (forecastDiscussionSectionHeading, bool) {
|
||||||
if len(block) <= 1 {
|
line = strings.TrimSpace(line)
|
||||||
|
if line == "" || line[0] == '.' {
|
||||||
|
return forecastDiscussionSectionHeading{}, false
|
||||||
|
}
|
||||||
|
return parseForecastDiscussionSectionHeading("." + line)
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseForecastDiscussionKeyMessages(body []string) []string {
|
||||||
|
body = removeForecastDiscussionPresentationMarkers(body)
|
||||||
|
body = trimBlankLines(body)
|
||||||
|
if len(body) > 0 && isForecastDiscussionKeyMessageMetadataLine(body[0]) {
|
||||||
|
body = trimBlankLines(body[1:])
|
||||||
|
}
|
||||||
|
if len(body) == 0 {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
hasMarkers := false
|
||||||
|
for _, raw := range body {
|
||||||
|
line := strings.TrimSpace(raw)
|
||||||
|
if line == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := stripForecastDiscussionKeyMessageMarker(line); ok {
|
||||||
|
hasMarkers = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
body := trimBlankLines(block[1:])
|
|
||||||
var messages []string
|
var messages []string
|
||||||
var current strings.Builder
|
var current strings.Builder
|
||||||
|
|
||||||
@@ -431,15 +640,21 @@ func parseForecastDiscussionKeyMessages(block []string) []string {
|
|||||||
current.Reset()
|
current.Reset()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
seenMarker := false
|
||||||
for _, raw := range body {
|
for _, raw := range body {
|
||||||
line := strings.TrimSpace(raw)
|
line := strings.TrimSpace(raw)
|
||||||
if line == "" {
|
if line == "" {
|
||||||
|
if !hasMarkers || !seenMarker {
|
||||||
|
flush()
|
||||||
|
}
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if strings.HasPrefix(line, "-") {
|
if stripped, ok := stripForecastDiscussionKeyMessageMarker(line); ok {
|
||||||
flush()
|
flush()
|
||||||
line = strings.TrimSpace(strings.TrimPrefix(line, "-"))
|
seenMarker = true
|
||||||
current.WriteString(line)
|
line = stripped
|
||||||
|
}
|
||||||
|
if line == "" {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
if current.Len() > 0 {
|
if current.Len() > 0 {
|
||||||
@@ -452,25 +667,24 @@ func parseForecastDiscussionKeyMessages(block []string) []string {
|
|||||||
return messages
|
return messages
|
||||||
}
|
}
|
||||||
|
|
||||||
func parseForecastDiscussionTextSection(block []string) (ForecastDiscussionSection, error) {
|
func parseForecastDiscussionTextSection(block forecastDiscussionSectionBlock) (ForecastDiscussionSection, error) {
|
||||||
if len(block) == 0 {
|
|
||||||
return ForecastDiscussionSection{}, fmt.Errorf("empty section")
|
|
||||||
}
|
|
||||||
|
|
||||||
section := ForecastDiscussionSection{
|
section := ForecastDiscussionSection{
|
||||||
Qualifier: parseForecastDiscussionQualifier(strings.TrimSpace(block[0])),
|
Qualifier: block.heading.qualifier,
|
||||||
}
|
}
|
||||||
|
|
||||||
body := trimBlankLines(block[1:])
|
body := trimBlankLines(removeForecastDiscussionPresentationMarkers(block.body))
|
||||||
|
if section.Qualifier == "" && len(body) > 0 && isForecastDiscussionStandaloneParenthetical(body[0]) {
|
||||||
|
section.Qualifier = strings.TrimSpace(body[0])
|
||||||
|
body = trimBlankLines(body[1:])
|
||||||
|
}
|
||||||
if len(body) == 0 {
|
if len(body) == 0 {
|
||||||
return section, nil
|
return section, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
first := strings.TrimSpace(body[0])
|
if isForecastDiscussionIssuedAtLine(body[0]) {
|
||||||
if strings.HasPrefix(first, "Issued at ") {
|
issuedAt, err := parseForecastDiscussionIssueTime(body[0])
|
||||||
issuedAt, err := parseForecastDiscussionIssueTime(first)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return ForecastDiscussionSection{}, fmt.Errorf("parse section issuedAt %q: %w", first, err)
|
return ForecastDiscussionSection{}, fmt.Errorf("parse section issuedAt %q: %w", strings.TrimSpace(body[0]), err)
|
||||||
}
|
}
|
||||||
tt := issuedAt.UTC()
|
tt := issuedAt.UTC()
|
||||||
section.IssuedAt = &tt
|
section.IssuedAt = &tt
|
||||||
@@ -482,12 +696,147 @@ func parseForecastDiscussionTextSection(block []string) (ForecastDiscussionSecti
|
|||||||
return section, nil
|
return section, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func parseForecastDiscussionQualifier(header string) string {
|
func isForecastDiscussionPresentationMarker(line string) bool {
|
||||||
m := forecastDiscussionHeaderRE.FindStringSubmatch(header)
|
switch {
|
||||||
if len(m) != 3 {
|
case strings.EqualFold(strings.TrimSpace(line), "-- Changed Discussion --"):
|
||||||
return ""
|
return true
|
||||||
|
case strings.EqualFold(strings.TrimSpace(line), "-- End Changed Discussion --"):
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
return strings.TrimSpace(m[2])
|
}
|
||||||
|
|
||||||
|
func removeForecastDiscussionPresentationMarkers(lines []string) []string {
|
||||||
|
body := make([]string, 0, len(lines))
|
||||||
|
for _, line := range lines {
|
||||||
|
if !isForecastDiscussionPresentationMarker(line) {
|
||||||
|
body = append(body, line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return body
|
||||||
|
}
|
||||||
|
|
||||||
|
func isForecastDiscussionStandaloneParenthetical(line string) bool {
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
return len(line) > 2 && line[0] == '(' && line[len(line)-1] == ')' && strings.TrimSpace(line[1:len(line)-1]) != ""
|
||||||
|
}
|
||||||
|
|
||||||
|
func isForecastDiscussionIssuedAtLine(line string) bool {
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
return len(line) > len("Issued at") &&
|
||||||
|
strings.EqualFold(line[:len("Issued at")], "Issued at") &&
|
||||||
|
isForecastDiscussionHorizontalWhitespace(line[len("Issued at")])
|
||||||
|
}
|
||||||
|
|
||||||
|
func isForecastDiscussionKeyMessageMetadataLine(line string) bool {
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
for _, label := range []string{"Issued at", "Updated at"} {
|
||||||
|
if !hasForecastDiscussionASCIIPrefix(line, label) || len(line) == len(label) || !isForecastDiscussionHorizontalWhitespace(line[len(label)]) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, err := parseForecastDiscussionIssueTime(strings.TrimSpace(line[len(label):])); err == nil {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return isForecastDiscussionKeyMessageAsOfLine(line)
|
||||||
|
}
|
||||||
|
|
||||||
|
func isForecastDiscussionKeyMessageAsOfLine(line string) bool {
|
||||||
|
const label = "As of"
|
||||||
|
|
||||||
|
if !hasForecastDiscussionASCIIPrefix(line, label) || len(line) == len(label) || !isForecastDiscussionHorizontalWhitespace(line[len(label)]) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
remainder := strings.TrimSpace(line[len(label):])
|
||||||
|
if !strings.HasSuffix(remainder, "...") {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
fields := strings.Fields(strings.TrimSpace(strings.TrimSuffix(remainder, "...")))
|
||||||
|
if len(fields) != 3 {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if _, _, err := parseForecastDiscussionClock(fields[0], fields[1]); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
switch strings.ToLower(fields[2]) {
|
||||||
|
case "monday", "tuesday", "wednesday", "thursday", "friday", "saturday", "sunday":
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func stripForecastDiscussionKeyMessageMarker(line string) (string, bool) {
|
||||||
|
if line == "" {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
if line[0] == '-' || line[0] == '*' {
|
||||||
|
content := line[1:]
|
||||||
|
hadWhitespace := len(content) > 0 && isForecastDiscussionHorizontalWhitespace(content[0])
|
||||||
|
content = strings.TrimLeft(content, " \t")
|
||||||
|
if hadWhitespace {
|
||||||
|
if stripped, ok := stripForecastDiscussionKeyMessageNumericMarker(content); ok {
|
||||||
|
content = stripped
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return content, true
|
||||||
|
}
|
||||||
|
|
||||||
|
return stripForecastDiscussionKeyMessageNumericMarker(line)
|
||||||
|
}
|
||||||
|
|
||||||
|
func stripForecastDiscussionKeyMessageNumericMarker(line string) (string, bool) {
|
||||||
|
digitStart := 0
|
||||||
|
digitEnd := 0
|
||||||
|
parenthesized := len(line) > 0 && line[0] == '('
|
||||||
|
if parenthesized {
|
||||||
|
digitStart = 1
|
||||||
|
digitEnd = 1
|
||||||
|
}
|
||||||
|
for digitEnd < len(line) && line[digitEnd] >= '0' && line[digitEnd] <= '9' {
|
||||||
|
digitEnd++
|
||||||
|
}
|
||||||
|
if digitEnd == digitStart || digitEnd == len(line) {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
if parenthesized && line[digitEnd] != ')' {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
if !parenthesized && line[digitEnd] != ')' && line[digitEnd] != '.' {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
|
||||||
|
markerEnd := digitEnd + 1
|
||||||
|
if markerEnd < len(line) && !isForecastDiscussionHorizontalWhitespace(line[markerEnd]) {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
value, err := strconv.ParseUint(line[digitStart:digitEnd], 10, 0)
|
||||||
|
if err != nil || value == 0 {
|
||||||
|
return "", false
|
||||||
|
}
|
||||||
|
return strings.TrimLeft(line[markerEnd:], " \t"), true
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasForecastDiscussionASCIIPrefix(line, prefix string) bool {
|
||||||
|
if len(line) < len(prefix) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for i := range prefix {
|
||||||
|
actual := line[i]
|
||||||
|
if actual >= 'A' && actual <= 'Z' {
|
||||||
|
actual += 'a' - 'A'
|
||||||
|
}
|
||||||
|
expected := prefix[i]
|
||||||
|
if expected >= 'A' && expected <= 'Z' {
|
||||||
|
expected += 'a' - 'A'
|
||||||
|
}
|
||||||
|
if actual != expected {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
func trimBlankLines(lines []string) []string {
|
func trimBlankLines(lines []string) []string {
|
||||||
|
|||||||
File diff suppressed because it is too large
Load Diff
26
internal/providers/nws/testdata/forecast_discussion_bgm_numbered_sample.html
vendored
Normal file
26
internal/providers/nws/testdata/forecast_discussion_bgm_numbered_sample.html
vendored
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<!-- Representative BGM/CTP-style layout; prose is concise edited test data, not an archived product. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<pre class="glossaryProduct">
|
||||||
|
FXUS61 KBGM 101730
|
||||||
|
AFDBGM
|
||||||
|
|
||||||
|
Area Forecast Discussion
|
||||||
|
National Weather Service Binghamton NY
|
||||||
|
130 PM EDT Fri Apr 10 2026
|
||||||
|
|
||||||
|
.KEY MESSAGES...
|
||||||
|
1) Periods of rain are expected through Saturday,
|
||||||
|
with locally heavier amounts possible.
|
||||||
|
2. Cooler temperatures return late this weekend.
|
||||||
|
|
||||||
|
.DISCUSSION...
|
||||||
|
Discussion details remain boundary-only content.
|
||||||
|
|
||||||
|
$$
|
||||||
|
|
||||||
|
WFO BGM
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
48
internal/providers/nws/testdata/forecast_discussion_bou_sample.html
vendored
Normal file
48
internal/providers/nws/testdata/forecast_discussion_bou_sample.html
vendored
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<pre class="glossaryProduct">
|
||||||
|
FXUS65 KBOU 071900
|
||||||
|
AFDBOU
|
||||||
|
|
||||||
|
Area Forecast Discussion
|
||||||
|
National Weather Service Denver CO
|
||||||
|
100 PM MDT Tue Apr 7 2026
|
||||||
|
|
||||||
|
.KEY MESSAGES...
|
||||||
|
-- Changed Discussion --
|
||||||
|
Updated at 100 PM MDT Tue Apr 7 2026
|
||||||
|
- Strong winds are expected along the Front Range this evening.
|
||||||
|
- Cooler temperatures arrive on Wednesday.
|
||||||
|
-- End Changed Discussion --
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
.SHORT TERM...
|
||||||
|
(Tonight through Wednesday)
|
||||||
|
Issued at 100 PM MDT Tue Apr 7 2026
|
||||||
|
|
||||||
|
Gusty west winds will continue through the evening before decreasing overnight.
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
.LONG TERM...
|
||||||
|
(Thursday through Saturday)
|
||||||
|
ISSUED AT 100 PM MDT Tue Apr 7 2026
|
||||||
|
|
||||||
|
Warmer and drier conditions return Thursday, followed by a chance of showers Friday.
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
.AVIATION...
|
||||||
|
|
||||||
|
VFR conditions are expected at Denver-area terminals through Wednesday morning.
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
$$
|
||||||
|
|
||||||
|
WFO BOU
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
27
internal/providers/nws/testdata/forecast_discussion_lwx_parenthesized_number_sample.html
vendored
Normal file
27
internal/providers/nws/testdata/forecast_discussion_lwx_parenthesized_number_sample.html
vendored
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<!-- Representative LWX-style layout; prose is concise edited test data, not an archived product. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<pre class="glossaryProduct">
|
||||||
|
FXUS61 KLWX 021800
|
||||||
|
AFDLWX
|
||||||
|
|
||||||
|
Area Forecast Discussion
|
||||||
|
National Weather Service Baltimore MD/Washington DC
|
||||||
|
200 PM EDT Sun Aug 2 2026
|
||||||
|
|
||||||
|
.KEY MESSAGES...
|
||||||
|
- (1) Thunderstorms remain possible near the Blue Ridge this evening.
|
||||||
|
- (2) Seasonably warm conditions continue Monday.
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
.AVIATION...
|
||||||
|
Aviation details remain boundary-only content.
|
||||||
|
|
||||||
|
$$
|
||||||
|
|
||||||
|
WFO LWX
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
25
internal/providers/nws/testdata/forecast_discussion_mfr_key_points_sample.html
vendored
Normal file
25
internal/providers/nws/testdata/forecast_discussion_mfr_key_points_sample.html
vendored
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<!-- Representative MFR-style layout; prose is concise edited test data, not an archived product. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<pre class="glossaryProduct">
|
||||||
|
FXUS66 KMFR 101945
|
||||||
|
AFDMFR
|
||||||
|
|
||||||
|
Area Forecast Discussion
|
||||||
|
National Weather Service Medford OR
|
||||||
|
1245 PM PDT Fri Apr 10 2026
|
||||||
|
|
||||||
|
.KEY POINTS...
|
||||||
|
* Gusty winds will develop over exposed ridges,
|
||||||
|
especially during the afternoon.
|
||||||
|
* Inland valleys remain dry through Saturday.
|
||||||
|
.DISCUSSION (Today through Thursday)...
|
||||||
|
Discussion details must not be included with key points.
|
||||||
|
|
||||||
|
$$
|
||||||
|
|
||||||
|
WFO MFR
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
33
internal/providers/nws/testdata/forecast_discussion_mfr_prev_discussion_sample.html
vendored
Normal file
33
internal/providers/nws/testdata/forecast_discussion_mfr_prev_discussion_sample.html
vendored
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<!-- Representative current MFR previous-discussion wrapper layout; prose is concise edited test data, not an archived product. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<pre class="glossaryProduct">
|
||||||
|
FXUS66 KMFR 022219
|
||||||
|
AFDMFR
|
||||||
|
|
||||||
|
Area Forecast Discussion
|
||||||
|
National Weather Service Medford OR
|
||||||
|
319 PM PDT Sun Aug 2 2026
|
||||||
|
|
||||||
|
.PREV DISCUSSION... /Issued 319 PM PDT Sun Aug 2 2026/
|
||||||
|
|
||||||
|
KEY MESSAGES...
|
||||||
|
|
||||||
|
* Heat returns to inland valleys Monday.
|
||||||
|
* Gusty afternoon winds develop east of the Cascades.
|
||||||
|
|
||||||
|
DISCUSSION...
|
||||||
|
Discussion details must not be included with key messages.
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
.MFR WATCHES/WARNINGS/ADVISORIES...
|
||||||
|
None.
|
||||||
|
|
||||||
|
$$
|
||||||
|
|
||||||
|
WFO MFR
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
29
internal/providers/nws/testdata/forecast_discussion_rah_as_of_sample.html
vendored
Normal file
29
internal/providers/nws/testdata/forecast_discussion_rah_as_of_sample.html
vendored
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
<!DOCTYPE html>
|
||||||
|
<!-- Representative RAH-style layout; prose is concise edited test data, not an archived product. -->
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<pre class="glossaryProduct">
|
||||||
|
FXUS62 KRAH 021635
|
||||||
|
AFDRAH
|
||||||
|
|
||||||
|
Area Forecast Discussion
|
||||||
|
National Weather Service Raleigh NC
|
||||||
|
1235 PM EDT Sun Aug 2 2026
|
||||||
|
|
||||||
|
.KEY MESSAGES...
|
||||||
|
As of 1235 PM Sunday...
|
||||||
|
|
||||||
|
1) Scattered storms may produce locally heavy rain this afternoon.
|
||||||
|
2) Drier weather arrives Monday.
|
||||||
|
|
||||||
|
&&
|
||||||
|
|
||||||
|
.DISCUSSION...
|
||||||
|
Discussion details remain boundary-only content.
|
||||||
|
|
||||||
|
$$
|
||||||
|
|
||||||
|
WFO RAH
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
// Package openweather contains provider-specific helper code for OpenWeather used by
|
// Package openweather contains provider-specific helper code for OpenWeather
|
||||||
// both sources and normalizers.
|
// used by sources and normalizers.
|
||||||
//
|
//
|
||||||
// Rules:
|
// Rules:
|
||||||
// - No network I/O here.
|
// - No network I/O here.
|
||||||
|
|||||||
188
internal/providers/spc/discussion.go
Normal file
188
internal/providers/spc/discussion.go
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"html"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
scriptBlockRE = regexp.MustCompile(`(?is)<script\b[^>]*>.*?</script>`)
|
||||||
|
preBlockRE = regexp.MustCompile(`(?is)<pre\b[^>]*>(.*?)</pre>`)
|
||||||
|
tagRE = regexp.MustCompile(`(?is)<[^>]+>`)
|
||||||
|
updatedRE = regexp.MustCompile(`(?im)^\s*Updated:\s*(.+?)\s*$`)
|
||||||
|
pageUpdatedRE = regexp.MustCompile(`(?i)\bUpdated:\s*((?:\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z)|(?:[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{2}:\d{2}:\d{2}\s+UTC\s+\d{4})|(?:\d{4}\s+UTC\s+[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{4})|(?:\d{4}Z\s+[A-Z][a-z]{2}\s+[A-Z][a-z]{2}\s+\d{1,2}\s+\d{4}))`)
|
||||||
|
productCodeRE = regexp.MustCompile(`(?i)^SPC\s+AC\s+\d+\s*$`)
|
||||||
|
sectionRE = regexp.MustCompile(`^\s*\.\.\.[A-Z0-9 /-]+\.{3}\s*$`)
|
||||||
|
)
|
||||||
|
|
||||||
|
// DiscussionText contains parsed text from an SPC print page.
|
||||||
|
type DiscussionText struct {
|
||||||
|
ProductTitle string
|
||||||
|
Headline string
|
||||||
|
Summary string
|
||||||
|
Discussion string
|
||||||
|
UpdatedAt *time.Time
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExtractProductText extracts and cleans the first useful preformatted SPC
|
||||||
|
// product text block from a print page.
|
||||||
|
func ExtractProductText(rawHTML string) (string, error) {
|
||||||
|
matches := preBlockRE.FindAllStringSubmatch(rawHTML, -1)
|
||||||
|
for _, match := range matches {
|
||||||
|
if len(match) < 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
text := cleanHTMLText(match[1])
|
||||||
|
if strings.TrimSpace(text) != "" {
|
||||||
|
return text, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("no useful pre block found")
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseDiscussionHTML extracts SPC product text and page-level metadata from a
|
||||||
|
// print-page HTML document.
|
||||||
|
func ParseDiscussionHTML(rawHTML string) (DiscussionText, error) {
|
||||||
|
text, err := ExtractProductText(rawHTML)
|
||||||
|
if err != nil {
|
||||||
|
return DiscussionText{}, err
|
||||||
|
}
|
||||||
|
parsed := ParseDiscussionText(text)
|
||||||
|
if updatedAt := ParsePageUpdatedTimestamp(rawHTML); updatedAt != nil {
|
||||||
|
parsed.UpdatedAt = updatedAt
|
||||||
|
}
|
||||||
|
return parsed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseDiscussionText extracts common SPC narrative metadata from cleaned
|
||||||
|
// product text.
|
||||||
|
func ParseDiscussionText(text string) DiscussionText {
|
||||||
|
text = trimBlankLines(normalizeNewlines(text))
|
||||||
|
title := ParseProductTitle(text)
|
||||||
|
return DiscussionText{
|
||||||
|
ProductTitle: title,
|
||||||
|
Headline: title,
|
||||||
|
Summary: ExtractSummary(text),
|
||||||
|
Discussion: text,
|
||||||
|
UpdatedAt: ParseUpdatedTimestamp(text),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParsePageUpdatedTimestamp parses the page-level Updated row from an SPC print
|
||||||
|
// page. SPC currently places this outside the product <pre> block.
|
||||||
|
func ParsePageUpdatedTimestamp(rawHTML string) *time.Time {
|
||||||
|
text := cleanHTMLText(rawHTML)
|
||||||
|
text = strings.ReplaceAll(text, "\u00a0", " ")
|
||||||
|
text = strings.Join(strings.Fields(text), " ")
|
||||||
|
match := pageUpdatedRE.FindStringSubmatch(text)
|
||||||
|
if len(match) != 2 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return parseUpdatedValue(match[1])
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseUpdatedTimestamp parses an SPC print-page Updated line when present.
|
||||||
|
func ParseUpdatedTimestamp(text string) *time.Time {
|
||||||
|
match := updatedRE.FindStringSubmatch(normalizeNewlines(text))
|
||||||
|
if len(match) != 2 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return parseUpdatedValue(match[1])
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseProductTitle returns the first non-empty product line from cleaned text.
|
||||||
|
func ParseProductTitle(text string) string {
|
||||||
|
for _, line := range strings.Split(normalizeNewlines(text), "\n") {
|
||||||
|
line = strings.TrimSpace(line)
|
||||||
|
if line == "" || strings.HasPrefix(line, "Updated:") || productCodeRE.MatchString(line) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return line
|
||||||
|
}
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseHeadline returns the human-facing headline from cleaned text.
|
||||||
|
func ParseHeadline(text string) string {
|
||||||
|
return ParseProductTitle(text)
|
||||||
|
}
|
||||||
|
|
||||||
|
// ExtractSummary returns text under the ...SUMMARY... section through the next
|
||||||
|
// SPC section heading.
|
||||||
|
func ExtractSummary(text string) string {
|
||||||
|
lines := strings.Split(normalizeNewlines(text), "\n")
|
||||||
|
start := -1
|
||||||
|
for i, line := range lines {
|
||||||
|
if strings.EqualFold(strings.TrimSpace(line), "...SUMMARY...") {
|
||||||
|
start = i + 1
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if start < 0 {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
var out []string
|
||||||
|
for _, line := range lines[start:] {
|
||||||
|
if sectionRE.MatchString(line) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
out = append(out, line)
|
||||||
|
}
|
||||||
|
return trimBlankLines(strings.Join(out, "\n"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func cleanHTMLText(raw string) string {
|
||||||
|
raw = scriptBlockRE.ReplaceAllString(raw, "")
|
||||||
|
raw = tagRE.ReplaceAllString(raw, "")
|
||||||
|
raw = html.UnescapeString(raw)
|
||||||
|
raw = normalizeNewlines(raw)
|
||||||
|
return trimBlankLines(raw)
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeNewlines(text string) string {
|
||||||
|
text = strings.ReplaceAll(text, "\r\n", "\n")
|
||||||
|
text = strings.ReplaceAll(text, "\r", "\n")
|
||||||
|
return text
|
||||||
|
}
|
||||||
|
|
||||||
|
func trimBlankLines(text string) string {
|
||||||
|
lines := strings.Split(normalizeNewlines(text), "\n")
|
||||||
|
start := 0
|
||||||
|
for start < len(lines) && strings.TrimSpace(lines[start]) == "" {
|
||||||
|
start++
|
||||||
|
}
|
||||||
|
end := len(lines)
|
||||||
|
for end > start && strings.TrimSpace(lines[end-1]) == "" {
|
||||||
|
end--
|
||||||
|
}
|
||||||
|
return strings.Join(lines[start:end], "\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseUpdatedValue(value string) *time.Time {
|
||||||
|
value = strings.TrimSpace(value)
|
||||||
|
if value == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
if t := parseOptionalISOTimestamp(value); t != nil {
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
for _, layout := range []string{
|
||||||
|
"Mon Jan 2 15:04:05 UTC 2006",
|
||||||
|
"1504 UTC Mon Jan 2 2006",
|
||||||
|
"1504Z Mon Jan 2 2006",
|
||||||
|
"3:04 PM UTC Mon Jan 2 2006",
|
||||||
|
time.RFC1123,
|
||||||
|
time.RFC1123Z,
|
||||||
|
} {
|
||||||
|
t, err := time.Parse(layout, value)
|
||||||
|
if err == nil {
|
||||||
|
tt := t.UTC()
|
||||||
|
return &tt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
109
internal/providers/spc/discussion_test.go
Normal file
109
internal/providers/spc/discussion_test.go
Normal file
@@ -0,0 +1,109 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExtractProductTextCleansPreBlock(t *testing.T) {
|
||||||
|
raw := string(readTestFile(t, "day1_prt.html"))
|
||||||
|
|
||||||
|
got, err := ExtractProductText(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ExtractProductText() error = %v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(got, "<script") || strings.Contains(got, "<pre") {
|
||||||
|
t.Fatalf("ExtractProductText() retained HTML: %q", got)
|
||||||
|
}
|
||||||
|
if strings.Contains(got, "ignore me") {
|
||||||
|
t.Fatalf("ExtractProductText() retained script content: %q", got)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got, "Day 1 Convective Outlook") {
|
||||||
|
t.Fatalf("ExtractProductText() missing headline: %q", got)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(got, "SPC AC 111234") {
|
||||||
|
t.Fatalf("ExtractProductText() = %q, want product code prefix", got)
|
||||||
|
}
|
||||||
|
if strings.HasPrefix(got, "\n") || strings.HasSuffix(got, "\n") {
|
||||||
|
t.Fatalf("ExtractProductText() retained surrounding blank lines: %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseDiscussionHTMLExtractsHeadlineSummaryAndUpdated(t *testing.T) {
|
||||||
|
got, err := ParseDiscussionHTML(string(readTestFile(t, "day1_prt.html")))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseDiscussionHTML() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.ProductTitle != "Day 1 Convective Outlook" {
|
||||||
|
t.Fatalf("ProductTitle = %q", got.ProductTitle)
|
||||||
|
}
|
||||||
|
if got.Headline != "Day 1 Convective Outlook" {
|
||||||
|
t.Fatalf("Headline = %q", got.Headline)
|
||||||
|
}
|
||||||
|
wantSummary := "Severe thunderstorms are possible across parts of the central Plains\nand mid Mississippi Valley this afternoon and evening."
|
||||||
|
if got.Summary != wantSummary {
|
||||||
|
t.Fatalf("Summary = %q, want %q", got.Summary, wantSummary)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got.Discussion, "...DISCUSSION...") {
|
||||||
|
t.Fatalf("Discussion missing full text: %q", got.Discussion)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(got.Discussion, "SPC AC 111234") {
|
||||||
|
t.Fatalf("Discussion = %q, want product code prefix", got.Discussion)
|
||||||
|
}
|
||||||
|
wantUpdated := time.Date(2026, 6, 11, 12, 45, 0, 0, time.UTC)
|
||||||
|
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdated) {
|
||||||
|
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdated)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseProductTitleSkipsSPCProductCode(t *testing.T) {
|
||||||
|
got := ParseProductTitle("SPC AC 101959\nDay 1 Convective Outlook\nNWS Storm Prediction Center Norman OK")
|
||||||
|
if got != "Day 1 Convective Outlook" {
|
||||||
|
t.Fatalf("ParseProductTitle() = %q, want Day 1 Convective Outlook", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseDiscussionTextPreservesCorrectionMarker(t *testing.T) {
|
||||||
|
got, err := ParseDiscussionHTML(string(readTestFile(t, "day2_prt_corr.html")))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseDiscussionHTML() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(got.Headline, "CORR 1") {
|
||||||
|
t.Fatalf("Headline = %q, want correction marker", got.Headline)
|
||||||
|
}
|
||||||
|
if !strings.Contains(got.Discussion, "CORR 1") {
|
||||||
|
t.Fatalf("Discussion = %q, want correction marker", got.Discussion)
|
||||||
|
}
|
||||||
|
wantUpdated := time.Date(2026, 6, 11, 17, 30, 0, 0, time.UTC)
|
||||||
|
if got.UpdatedAt == nil || !got.UpdatedAt.Equal(wantUpdated) {
|
||||||
|
t.Fatalf("UpdatedAt = %v, want %s", got.UpdatedAt, wantUpdated)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseUpdatedTimestampReturnsNilWhenAbsent(t *testing.T) {
|
||||||
|
text, err := ExtractProductText(string(readTestFile(t, "day2_prt_corr.html")))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ExtractProductText() error = %v", err)
|
||||||
|
}
|
||||||
|
if got := ParseUpdatedTimestamp(text); got != nil {
|
||||||
|
t.Fatalf("ParseUpdatedTimestamp() = %v, want nil", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseUpdatedTimestampAcceptsSPCUTCFormat(t *testing.T) {
|
||||||
|
got := ParseUpdatedTimestamp("Updated: 1945 UTC Thu Jun 11 2026")
|
||||||
|
want := time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)
|
||||||
|
if got == nil || !got.Equal(want) {
|
||||||
|
t.Fatalf("ParseUpdatedTimestamp() = %v, want %s", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParsePageUpdatedTimestampAcceptsLiveSPCShape(t *testing.T) {
|
||||||
|
got := ParsePageUpdatedTimestamp(string(readTestFile(t, "day3_prt.html")))
|
||||||
|
want := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
|
||||||
|
if got == nil || !got.Equal(want) {
|
||||||
|
t.Fatalf("ParsePageUpdatedTimestamp() = %v, want %s", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
8
internal/providers/spc/doc.go
Normal file
8
internal/providers/spc/doc.go
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
// Package spc contains deterministic helper code for Storm Prediction Center
|
||||||
|
// products used by sources and normalizers.
|
||||||
|
//
|
||||||
|
// Rules:
|
||||||
|
// - No network I/O here.
|
||||||
|
// - Keep helpers deterministic and easy to unit test.
|
||||||
|
// - Preserve upstream payload fragments needed for canonical mapping.
|
||||||
|
package spc
|
||||||
17
internal/providers/spc/fixture_test.go
Normal file
17
internal/providers/spc/fixture_test.go
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func readTestFile(t *testing.T, name string) []byte {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join("testdata", name)
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read %s: %v", path, err)
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
}
|
||||||
120
internal/providers/spc/geojson.go
Normal file
120
internal/providers/spc/geojson.go
Normal file
@@ -0,0 +1,120 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
// GeoJSONFeatureCollection is the minimal SPC outlook FeatureCollection shape
|
||||||
|
// needed by weatherfeeder.
|
||||||
|
type GeoJSONFeatureCollection struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Features []GeoJSONFeature `json:"features"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// GeoJSONFeature preserves typed SPC properties and compact raw geometry.
|
||||||
|
type GeoJSONFeature struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Properties GeoJSONProperties `json:"properties"`
|
||||||
|
Geometry json.RawMessage `json:"geometry"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// GeoJSONProperties contains the SPC fields used by canonical mapping.
|
||||||
|
type GeoJSONProperties struct {
|
||||||
|
ValidISO string `json:"VALID_ISO"`
|
||||||
|
ExpireISO string `json:"EXPIRE_ISO"`
|
||||||
|
IssueISO string `json:"ISSUE_ISO"`
|
||||||
|
Forecaster string `json:"FORECASTER"`
|
||||||
|
Label string `json:"LABEL"`
|
||||||
|
Label2 string `json:"LABEL2"`
|
||||||
|
DN *int `json:"DN"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type geometryMetadata struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Geometries []json.RawMessage `json:"geometries"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// DecodeGeoJSON decodes an SPC GeoJSON outlook product and compacts feature
|
||||||
|
// geometry JSON for stable downstream storage.
|
||||||
|
func DecodeGeoJSON(raw []byte) (GeoJSONFeatureCollection, error) {
|
||||||
|
var collection GeoJSONFeatureCollection
|
||||||
|
if err := json.Unmarshal(raw, &collection); err != nil {
|
||||||
|
return GeoJSONFeatureCollection{}, fmt.Errorf("decode geojson: %w", err)
|
||||||
|
}
|
||||||
|
for i := range collection.Features {
|
||||||
|
geom, err := compactJSON(collection.Features[i].Geometry)
|
||||||
|
if err != nil {
|
||||||
|
return GeoJSONFeatureCollection{}, fmt.Errorf("features[%d].geometry: %w", i, err)
|
||||||
|
}
|
||||||
|
collection.Features[i].Geometry = geom
|
||||||
|
}
|
||||||
|
return collection, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsEmptyGeometryCollection reports whether raw is SPC's no-polygon placeholder
|
||||||
|
// geometry shape: a GeometryCollection with no child geometries.
|
||||||
|
func IsEmptyGeometryCollection(raw json.RawMessage) bool {
|
||||||
|
var meta geometryMetadata
|
||||||
|
if err := json.Unmarshal(raw, &meta); err != nil {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return meta.Type == "GeometryCollection" && len(meta.Geometries) == 0
|
||||||
|
}
|
||||||
|
|
||||||
|
func (p *GeoJSONProperties) UnmarshalJSON(raw []byte) error {
|
||||||
|
type alias GeoJSONProperties
|
||||||
|
var aux struct {
|
||||||
|
alias
|
||||||
|
DN any `json:"DN"`
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(raw, &aux); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
*p = GeoJSONProperties(aux.alias)
|
||||||
|
dn, err := parseSeverityRank(aux.DN)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
p.DN = dn
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseSeverityRank(value any) (*int, error) {
|
||||||
|
switch v := value.(type) {
|
||||||
|
case nil:
|
||||||
|
return nil, nil
|
||||||
|
case float64:
|
||||||
|
rank := int(v)
|
||||||
|
if float64(rank) != v {
|
||||||
|
return nil, fmt.Errorf("DN must be an integer, got %v", v)
|
||||||
|
}
|
||||||
|
return &rank, nil
|
||||||
|
case string:
|
||||||
|
v = strings.TrimSpace(v)
|
||||||
|
if v == "" {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
rank, err := strconv.Atoi(v)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("DN must be an integer, got %q", v)
|
||||||
|
}
|
||||||
|
return &rank, nil
|
||||||
|
default:
|
||||||
|
return nil, fmt.Errorf("DN must be an integer or string, got %T", value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func compactJSON(raw json.RawMessage) (json.RawMessage, error) {
|
||||||
|
if len(raw) == 0 {
|
||||||
|
return nil, fmt.Errorf("missing")
|
||||||
|
}
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := json.Compact(&buf, raw); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return json.RawMessage(buf.Bytes()), nil
|
||||||
|
}
|
||||||
115
internal/providers/spc/geojson_test.go
Normal file
115
internal/providers/spc/geojson_test.go
Normal file
@@ -0,0 +1,115 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDecodeGeoJSONExposesSPCPropertiesAndCompactGeometry(t *testing.T) {
|
||||||
|
raw := readTestFile(t, "day1_cat.geojson")
|
||||||
|
|
||||||
|
got, err := DecodeGeoJSON(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DecodeGeoJSON() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Type != "FeatureCollection" {
|
||||||
|
t.Fatalf("Type = %q, want FeatureCollection", got.Type)
|
||||||
|
}
|
||||||
|
if len(got.Features) != 1 {
|
||||||
|
t.Fatalf("Features length = %d, want 1", len(got.Features))
|
||||||
|
}
|
||||||
|
|
||||||
|
feature := got.Features[0]
|
||||||
|
props := feature.Properties
|
||||||
|
if props.ValidISO != "2026-06-11T13:00:00Z" {
|
||||||
|
t.Fatalf("VALID_ISO = %q", props.ValidISO)
|
||||||
|
}
|
||||||
|
if props.ExpireISO != "2026-06-12T12:00:00Z" {
|
||||||
|
t.Fatalf("EXPIRE_ISO = %q", props.ExpireISO)
|
||||||
|
}
|
||||||
|
if props.IssueISO != "2026-06-11T12:34:56Z" {
|
||||||
|
t.Fatalf("ISSUE_ISO = %q", props.IssueISO)
|
||||||
|
}
|
||||||
|
if props.Forecaster != "SMITH" {
|
||||||
|
t.Fatalf("FORECASTER = %q", props.Forecaster)
|
||||||
|
}
|
||||||
|
if props.Label != "SLGT" {
|
||||||
|
t.Fatalf("LABEL = %q", props.Label)
|
||||||
|
}
|
||||||
|
if props.Label2 != "Slight Risk" {
|
||||||
|
t.Fatalf("LABEL2 = %q", props.Label2)
|
||||||
|
}
|
||||||
|
if props.DN == nil || *props.DN != 3 {
|
||||||
|
t.Fatalf("DN = %v, want 3", props.DN)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantGeometry := `{"type":"Polygon","coordinates":[[[-91.0,38.0],[-90.0,38.0],[-90.0,39.0],[-91.0,39.0],[-91.0,38.0]]]}`
|
||||||
|
if string(feature.Geometry) != wantGeometry {
|
||||||
|
t.Fatalf("Geometry = %s, want %s", feature.Geometry, wantGeometry)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(feature.Geometry), "\n") || strings.Contains(string(feature.Geometry), " ") {
|
||||||
|
t.Fatalf("Geometry is not compact: %q", feature.Geometry)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDecodeGeoJSONParsesSeverityRankString(t *testing.T) {
|
||||||
|
raw := readTestFile(t, "day2_torn.geojson")
|
||||||
|
|
||||||
|
got, err := DecodeGeoJSON(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("DecodeGeoJSON() error = %v", err)
|
||||||
|
}
|
||||||
|
props := got.Features[0].Properties
|
||||||
|
if props.DN == nil || *props.DN != 5 {
|
||||||
|
t.Fatalf("DN = %v, want 5", props.DN)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIsEmptyGeometryCollection(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
raw string
|
||||||
|
want bool
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "empty geometry collection",
|
||||||
|
raw: `{"type":"GeometryCollection","geometries":[]}`,
|
||||||
|
want: true,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "non-empty geometry collection",
|
||||||
|
raw: `{"type":"GeometryCollection","geometries":[{"type":"Polygon","coordinates":[]} ]}`,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "polygon",
|
||||||
|
raw: `{"type":"Polygon","coordinates":[]}`,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid json",
|
||||||
|
raw: `{`,
|
||||||
|
want: false,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
if got := IsEmptyGeometryCollection([]byte(tt.raw)); got != tt.want {
|
||||||
|
t.Fatalf("IsEmptyGeometryCollection() = %v, want %v", got, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseISOTimestampTrimsAndReturnsUTC(t *testing.T) {
|
||||||
|
got, err := ParseISOTimestamp(" 2026-06-11T12:34:56Z ")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseISOTimestamp() error = %v", err)
|
||||||
|
}
|
||||||
|
want := time.Date(2026, 6, 11, 12, 34, 56, 0, time.UTC)
|
||||||
|
if !got.Equal(want) {
|
||||||
|
t.Fatalf("ParseISOTimestamp() = %s, want %s", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
86
internal/providers/spc/product.go
Normal file
86
internal/providers/spc/product.go
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import "fmt"
|
||||||
|
|
||||||
|
const (
|
||||||
|
OutlookTypeCategorical = "categorical"
|
||||||
|
OutlookTypeTornado = "tornado"
|
||||||
|
OutlookTypeHail = "hail"
|
||||||
|
OutlookTypeWind = "wind"
|
||||||
|
)
|
||||||
|
|
||||||
|
// GeoJSONProduct describes one required SPC convective outlook GeoJSON product.
|
||||||
|
type GeoJSONProduct struct {
|
||||||
|
Key string
|
||||||
|
Day int
|
||||||
|
OutlookType string
|
||||||
|
URL string
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiscussionProduct describes one required SPC convective outlook print page.
|
||||||
|
type DiscussionProduct struct {
|
||||||
|
Key string
|
||||||
|
Day int
|
||||||
|
URL string
|
||||||
|
}
|
||||||
|
|
||||||
|
var geoJSONProducts = []GeoJSONProduct{
|
||||||
|
{Key: "day1_categorical", Day: 1, OutlookType: OutlookTypeCategorical, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_cat.nolyr.geojson"},
|
||||||
|
{Key: "day1_tornado", Day: 1, OutlookType: OutlookTypeTornado, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_torn.nolyr.geojson"},
|
||||||
|
{Key: "day1_hail", Day: 1, OutlookType: OutlookTypeHail, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_hail.nolyr.geojson"},
|
||||||
|
{Key: "day1_wind", Day: 1, OutlookType: OutlookTypeWind, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_wind.nolyr.geojson"},
|
||||||
|
{Key: "day2_categorical", Day: 2, OutlookType: OutlookTypeCategorical, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_cat.nolyr.geojson"},
|
||||||
|
{Key: "day2_tornado", Day: 2, OutlookType: OutlookTypeTornado, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_torn.nolyr.geojson"},
|
||||||
|
{Key: "day2_hail", Day: 2, OutlookType: OutlookTypeHail, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_hail.nolyr.geojson"},
|
||||||
|
{Key: "day2_wind", Day: 2, OutlookType: OutlookTypeWind, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_wind.nolyr.geojson"},
|
||||||
|
{Key: "day3_categorical", Day: 3, OutlookType: OutlookTypeCategorical, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_cat.nolyr.geojson"},
|
||||||
|
}
|
||||||
|
|
||||||
|
var discussionProducts = []DiscussionProduct{
|
||||||
|
{Key: "day1", Day: 1, URL: "https://www.spc.noaa.gov/products/outlook/day1otlk_prt.html"},
|
||||||
|
{Key: "day2", Day: 2, URL: "https://www.spc.noaa.gov/products/outlook/day2otlk_prt.html"},
|
||||||
|
{Key: "day3", Day: 3, URL: "https://www.spc.noaa.gov/products/outlook/day3otlk_prt.html"},
|
||||||
|
}
|
||||||
|
|
||||||
|
// GeoJSONProducts returns the required SPC convective outlook GeoJSON products
|
||||||
|
// in stable day/type order.
|
||||||
|
func GeoJSONProducts() []GeoJSONProduct {
|
||||||
|
out := make([]GeoJSONProduct, len(geoJSONProducts))
|
||||||
|
copy(out, geoJSONProducts)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiscussionProducts returns the required SPC convective outlook print pages in
|
||||||
|
// stable day order.
|
||||||
|
func DiscussionProducts() []DiscussionProduct {
|
||||||
|
out := make([]DiscussionProduct, len(discussionProducts))
|
||||||
|
copy(out, discussionProducts)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
// GeoJSONProductByKey returns product metadata for a configured product key.
|
||||||
|
func GeoJSONProductByKey(key string) (GeoJSONProduct, bool) {
|
||||||
|
for _, product := range geoJSONProducts {
|
||||||
|
if product.Key == key {
|
||||||
|
return product, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return GeoJSONProduct{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
// DiscussionProductByKey returns discussion metadata for a configured day key.
|
||||||
|
func DiscussionProductByKey(key string) (DiscussionProduct, bool) {
|
||||||
|
for _, product := range discussionProducts {
|
||||||
|
if product.Key == key {
|
||||||
|
return product, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return DiscussionProduct{}, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateProductDay(day int) error {
|
||||||
|
if day < 1 || day > 3 {
|
||||||
|
return fmt.Errorf("day must be 1, 2, or 3, got %d", day)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
53
internal/providers/spc/product_test.go
Normal file
53
internal/providers/spc/product_test.go
Normal file
@@ -0,0 +1,53 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestGeoJSONProductsStableOrder(t *testing.T) {
|
||||||
|
got := GeoJSONProducts()
|
||||||
|
if len(got) != 9 {
|
||||||
|
t.Fatalf("GeoJSONProducts() length = %d, want 9", len(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
wantKeys := []string{
|
||||||
|
"day1_categorical",
|
||||||
|
"day1_tornado",
|
||||||
|
"day1_hail",
|
||||||
|
"day1_wind",
|
||||||
|
"day2_categorical",
|
||||||
|
"day2_tornado",
|
||||||
|
"day2_hail",
|
||||||
|
"day2_wind",
|
||||||
|
"day3_categorical",
|
||||||
|
}
|
||||||
|
for i, want := range wantKeys {
|
||||||
|
if got[i].Key != want {
|
||||||
|
t.Fatalf("GeoJSONProducts()[%d].Key = %q, want %q", i, got[i].Key, want)
|
||||||
|
}
|
||||||
|
if err := validateProductDay(got[i].Day); err != nil {
|
||||||
|
t.Fatalf("GeoJSONProducts()[%d].Day invalid: %v", i, err)
|
||||||
|
}
|
||||||
|
if got[i].URL == "" {
|
||||||
|
t.Fatalf("GeoJSONProducts()[%d].URL is empty", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDiscussionProductsStableOrder(t *testing.T) {
|
||||||
|
got := DiscussionProducts()
|
||||||
|
if len(got) != 3 {
|
||||||
|
t.Fatalf("DiscussionProducts() length = %d, want 3", len(got))
|
||||||
|
}
|
||||||
|
|
||||||
|
wantKeys := []string{"day1", "day2", "day3"}
|
||||||
|
for i, want := range wantKeys {
|
||||||
|
if got[i].Key != want {
|
||||||
|
t.Fatalf("DiscussionProducts()[%d].Key = %q, want %q", i, got[i].Key, want)
|
||||||
|
}
|
||||||
|
if got[i].Day != i+1 {
|
||||||
|
t.Fatalf("DiscussionProducts()[%d].Day = %d, want %d", i, got[i].Day, i+1)
|
||||||
|
}
|
||||||
|
if got[i].URL == "" {
|
||||||
|
t.Fatalf("DiscussionProducts()[%d].URL is empty", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
45
internal/providers/spc/raw.go
Normal file
45
internal/providers/spc/raw.go
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RawConvectiveOutlookBundle is the provider payload shape for SPC convective
|
||||||
|
// outlook fetch bundles.
|
||||||
|
type RawConvectiveOutlookBundle struct {
|
||||||
|
LocationID string `json:"locationId,omitempty"`
|
||||||
|
LocationName string `json:"locationName,omitempty"`
|
||||||
|
Latitude float64 `json:"latitude"`
|
||||||
|
Longitude float64 `json:"longitude"`
|
||||||
|
FetchedAt time.Time `json:"fetchedAt"`
|
||||||
|
Products []RawOutlookProduct `json:"products"`
|
||||||
|
Discussions []RawDiscussionPage `json:"discussions"`
|
||||||
|
RSS *RawRSSFeed `json:"rss,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RawOutlookProduct contains one fetched SPC GeoJSON product.
|
||||||
|
type RawOutlookProduct struct {
|
||||||
|
Key string `json:"key"`
|
||||||
|
Day int `json:"day"`
|
||||||
|
OutlookType string `json:"outlookType"`
|
||||||
|
URL string `json:"url"`
|
||||||
|
FetchedAt time.Time `json:"fetchedAt"`
|
||||||
|
Body json.RawMessage `json:"body"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RawDiscussionPage contains one fetched SPC print page.
|
||||||
|
type RawDiscussionPage struct {
|
||||||
|
Key string `json:"key"`
|
||||||
|
Day int `json:"day"`
|
||||||
|
URL string `json:"url"`
|
||||||
|
FetchedAt time.Time `json:"fetchedAt"`
|
||||||
|
Body string `json:"body"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// RawRSSFeed contains optional fetched SPC RSS metadata.
|
||||||
|
type RawRSSFeed struct {
|
||||||
|
URL string `json:"url"`
|
||||||
|
FetchedAt time.Time `json:"fetchedAt"`
|
||||||
|
Body string `json:"body"`
|
||||||
|
}
|
||||||
51
internal/providers/spc/raw_test.go
Normal file
51
internal/providers/spc/raw_test.go
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRawConvectiveOutlookBundleJSONShape(t *testing.T) {
|
||||||
|
fetchedAt := time.Date(2026, 6, 11, 20, 0, 0, 0, time.UTC)
|
||||||
|
bundle := RawConvectiveOutlookBundle{
|
||||||
|
LocationID: "stl",
|
||||||
|
LocationName: "St. Louis, MO",
|
||||||
|
Latitude: 38.6239,
|
||||||
|
Longitude: -90.3571,
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
Products: []RawOutlookProduct{{
|
||||||
|
Key: "day1_categorical",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: OutlookTypeCategorical,
|
||||||
|
URL: "https://example.invalid/day1.geojson",
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
Body: json.RawMessage(`{"type":"FeatureCollection","features":[]}`),
|
||||||
|
}},
|
||||||
|
Discussions: []RawDiscussionPage{{
|
||||||
|
Key: "day1",
|
||||||
|
Day: 1,
|
||||||
|
URL: "https://example.invalid/day1.html",
|
||||||
|
FetchedAt: fetchedAt,
|
||||||
|
Body: "Day 1 Convective Outlook",
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
|
||||||
|
raw, err := json.Marshal(bundle)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal() error = %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var got map[string]any
|
||||||
|
if err := json.Unmarshal(raw, &got); err != nil {
|
||||||
|
t.Fatalf("Unmarshal() error = %v", err)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"locationId", "locationName", "latitude", "longitude", "fetchedAt", "products", "discussions"} {
|
||||||
|
if _, ok := got[key]; !ok {
|
||||||
|
t.Fatalf("marshaled bundle missing key %q in %s", key, raw)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, ok := got["rss"]; ok {
|
||||||
|
t.Fatalf("marshaled bundle included empty rss: %s", raw)
|
||||||
|
}
|
||||||
|
}
|
||||||
81
internal/providers/spc/rss.go
Normal file
81
internal/providers/spc/rss.go
Normal file
@@ -0,0 +1,81 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/xml"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// RSSFeed is a minimal view of the optional SPC RSS feed.
|
||||||
|
type RSSFeed struct {
|
||||||
|
Title string
|
||||||
|
Link string
|
||||||
|
Description string
|
||||||
|
LastBuildDate *time.Time
|
||||||
|
Items []RSSItem
|
||||||
|
}
|
||||||
|
|
||||||
|
// RSSItem is a minimal view of one optional SPC RSS item.
|
||||||
|
type RSSItem struct {
|
||||||
|
Title string
|
||||||
|
Link string
|
||||||
|
Description string
|
||||||
|
PubDate string
|
||||||
|
GUID string
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseRSSFeed decodes supplemental SPC RSS metadata.
|
||||||
|
func ParseRSSFeed(raw string) (RSSFeed, error) {
|
||||||
|
var doc struct {
|
||||||
|
Channel struct {
|
||||||
|
Title string `xml:"title"`
|
||||||
|
Link string `xml:"link"`
|
||||||
|
Description string `xml:"description"`
|
||||||
|
LastBuildDate string `xml:"lastBuildDate"`
|
||||||
|
Items []struct {
|
||||||
|
Title string `xml:"title"`
|
||||||
|
Link string `xml:"link"`
|
||||||
|
Description string `xml:"description"`
|
||||||
|
PubDate string `xml:"pubDate"`
|
||||||
|
GUID string `xml:"guid"`
|
||||||
|
} `xml:"item"`
|
||||||
|
} `xml:"channel"`
|
||||||
|
}
|
||||||
|
if err := xml.Unmarshal([]byte(raw), &doc); err != nil {
|
||||||
|
return RSSFeed{}, fmt.Errorf("decode rss: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
feed := RSSFeed{
|
||||||
|
Title: strings.TrimSpace(doc.Channel.Title),
|
||||||
|
Link: strings.TrimSpace(doc.Channel.Link),
|
||||||
|
Description: strings.TrimSpace(doc.Channel.Description),
|
||||||
|
LastBuildDate: parseRSSDate(doc.Channel.LastBuildDate),
|
||||||
|
Items: make([]RSSItem, 0, len(doc.Channel.Items)),
|
||||||
|
}
|
||||||
|
for _, item := range doc.Channel.Items {
|
||||||
|
feed.Items = append(feed.Items, RSSItem{
|
||||||
|
Title: strings.TrimSpace(item.Title),
|
||||||
|
Link: strings.TrimSpace(item.Link),
|
||||||
|
Description: strings.TrimSpace(item.Description),
|
||||||
|
PubDate: strings.TrimSpace(item.PubDate),
|
||||||
|
GUID: strings.TrimSpace(item.GUID),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return feed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseRSSDate(value string) *time.Time {
|
||||||
|
value = strings.TrimSpace(value)
|
||||||
|
if value == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
for _, layout := range []string{time.RFC1123Z, time.RFC1123} {
|
||||||
|
t, err := time.Parse(layout, value)
|
||||||
|
if err == nil {
|
||||||
|
tt := t.UTC()
|
||||||
|
return &tt
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
43
internal/providers/spc/rss_test.go
Normal file
43
internal/providers/spc/rss_test.go
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseRSSFeed(t *testing.T) {
|
||||||
|
const raw = `<?xml version="1.0"?>
|
||||||
|
<rss version="2.0">
|
||||||
|
<channel>
|
||||||
|
<title>SPC AC RSS</title>
|
||||||
|
<link>https://www.spc.noaa.gov/products/</link>
|
||||||
|
<description>SPC products</description>
|
||||||
|
<lastBuildDate>Thu, 11 Jun 2026 19:00:00 +0000</lastBuildDate>
|
||||||
|
<item>
|
||||||
|
<title>Day 1 Convective Outlook</title>
|
||||||
|
<link>https://www.spc.noaa.gov/products/outlook/day1otlk.html</link>
|
||||||
|
<description>Outlook text</description>
|
||||||
|
<pubDate>Thu, 11 Jun 2026 18:55:00 +0000</pubDate>
|
||||||
|
<guid>day1</guid>
|
||||||
|
</item>
|
||||||
|
</channel>
|
||||||
|
</rss>`
|
||||||
|
|
||||||
|
got, err := ParseRSSFeed(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseRSSFeed() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Title != "SPC AC RSS" {
|
||||||
|
t.Fatalf("Title = %q", got.Title)
|
||||||
|
}
|
||||||
|
wantBuild := time.Date(2026, 6, 11, 19, 0, 0, 0, time.UTC)
|
||||||
|
if got.LastBuildDate == nil || !got.LastBuildDate.Equal(wantBuild) {
|
||||||
|
t.Fatalf("LastBuildDate = %v, want %s", got.LastBuildDate, wantBuild)
|
||||||
|
}
|
||||||
|
if len(got.Items) != 1 {
|
||||||
|
t.Fatalf("Items length = %d, want 1", len(got.Items))
|
||||||
|
}
|
||||||
|
if got.Items[0].GUID != "day1" {
|
||||||
|
t.Fatalf("Item GUID = %q", got.Items[0].GUID)
|
||||||
|
}
|
||||||
|
}
|
||||||
29
internal/providers/spc/testdata/day1_cat.geojson
vendored
Normal file
29
internal/providers/spc/testdata/day1_cat.geojson
vendored
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"type": "FeatureCollection",
|
||||||
|
"features": [
|
||||||
|
{
|
||||||
|
"type": "Feature",
|
||||||
|
"properties": {
|
||||||
|
"VALID_ISO": "2026-06-11T13:00:00Z",
|
||||||
|
"EXPIRE_ISO": "2026-06-12T12:00:00Z",
|
||||||
|
"ISSUE_ISO": "2026-06-11T12:34:56Z",
|
||||||
|
"FORECASTER": "SMITH",
|
||||||
|
"LABEL": "SLGT",
|
||||||
|
"LABEL2": "Slight Risk",
|
||||||
|
"DN": 3
|
||||||
|
},
|
||||||
|
"geometry": {
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [
|
||||||
|
[
|
||||||
|
[-91.0, 38.0],
|
||||||
|
[-90.0, 38.0],
|
||||||
|
[-90.0, 39.0],
|
||||||
|
[-91.0, 39.0],
|
||||||
|
[-91.0, 38.0]
|
||||||
|
]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
22
internal/providers/spc/testdata/day1_prt.html
vendored
Normal file
22
internal/providers/spc/testdata/day1_prt.html
vendored
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html>
|
||||||
|
<head><title>Day 1 Convective Outlook</title></head>
|
||||||
|
<body>
|
||||||
|
<table>
|
||||||
|
<tr><td align="center" class="rpttext" nowrap>Updated: Thu Jun 11 12:45:00 UTC 2026 (<a href="archive/day1-geojson.zip">geojson</a>)</td></tr>
|
||||||
|
</table>
|
||||||
|
<pre>
|
||||||
|
<script>window.bad = "<b>ignore me</b>";</script>
|
||||||
|
SPC AC 111234
|
||||||
|
Day 1 Convective Outlook
|
||||||
|
NWS Storm Prediction Center Norman OK
|
||||||
|
|
||||||
|
...SUMMARY...
|
||||||
|
Severe thunderstorms are possible across parts of the central Plains
|
||||||
|
and mid Mississippi Valley this afternoon and evening.
|
||||||
|
|
||||||
|
...DISCUSSION...
|
||||||
|
The primary threats will be damaging wind and large hail.
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
19
internal/providers/spc/testdata/day2_prt_corr.html
vendored
Normal file
19
internal/providers/spc/testdata/day2_prt_corr.html
vendored
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<table>
|
||||||
|
<tr><td class="rpttext">Updated: Thu Jun 11 17:30:00 UTC 2026 </td></tr>
|
||||||
|
</table>
|
||||||
|
<pre>
|
||||||
|
SPC AC 111730
|
||||||
|
Day 2 Convective Outlook CORR 1
|
||||||
|
NWS Storm Prediction Center Norman OK
|
||||||
|
|
||||||
|
...SUMMARY...
|
||||||
|
Scattered severe thunderstorms remain possible across the southern Plains.
|
||||||
|
|
||||||
|
...DISCUSSION...
|
||||||
|
Corrected outlook text remains otherwise unchanged.
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
29
internal/providers/spc/testdata/day2_torn.geojson
vendored
Normal file
29
internal/providers/spc/testdata/day2_torn.geojson
vendored
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"type": "FeatureCollection",
|
||||||
|
"features": [
|
||||||
|
{
|
||||||
|
"type": "Feature",
|
||||||
|
"properties": {
|
||||||
|
"VALID_ISO": "2026-06-12T12:00:00Z",
|
||||||
|
"EXPIRE_ISO": "2026-06-13T12:00:00Z",
|
||||||
|
"ISSUE_ISO": "2026-06-11T17:30:00Z",
|
||||||
|
"FORECASTER": "DOE",
|
||||||
|
"LABEL": "5",
|
||||||
|
"LABEL2": "5% Tornado",
|
||||||
|
"DN": "5"
|
||||||
|
},
|
||||||
|
"geometry": {
|
||||||
|
"type": "Polygon",
|
||||||
|
"coordinates": [
|
||||||
|
[
|
||||||
|
[-100.0, 35.0],
|
||||||
|
[-98.0, 35.0],
|
||||||
|
[-98.0, 37.0],
|
||||||
|
[-100.0, 37.0],
|
||||||
|
[-100.0, 35.0]
|
||||||
|
]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
31
internal/providers/spc/testdata/day3_cat.geojson
vendored
Normal file
31
internal/providers/spc/testdata/day3_cat.geojson
vendored
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
{
|
||||||
|
"type": "FeatureCollection",
|
||||||
|
"features": [
|
||||||
|
{
|
||||||
|
"type": "Feature",
|
||||||
|
"properties": {
|
||||||
|
"VALID_ISO": "2026-06-13T12:00:00Z",
|
||||||
|
"EXPIRE_ISO": "2026-06-14T12:00:00Z",
|
||||||
|
"ISSUE_ISO": "2026-06-11T19:45:00Z",
|
||||||
|
"FORECASTER": "LEE",
|
||||||
|
"LABEL": "MRGL",
|
||||||
|
"LABEL2": "Marginal Risk",
|
||||||
|
"DN": 2
|
||||||
|
},
|
||||||
|
"geometry": {
|
||||||
|
"type": "MultiPolygon",
|
||||||
|
"coordinates": [
|
||||||
|
[
|
||||||
|
[
|
||||||
|
[-103.0, 34.0],
|
||||||
|
[-101.0, 34.0],
|
||||||
|
[-101.0, 36.0],
|
||||||
|
[-103.0, 36.0],
|
||||||
|
[-103.0, 34.0]
|
||||||
|
]
|
||||||
|
]
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
19
internal/providers/spc/testdata/day3_prt.html
vendored
Normal file
19
internal/providers/spc/testdata/day3_prt.html
vendored
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html>
|
||||||
|
<body>
|
||||||
|
<table>
|
||||||
|
<tr><td class="rpttext">Updated: Thu Jun 11 20:00:00 UTC 2026 </td></tr>
|
||||||
|
</table>
|
||||||
|
<pre>
|
||||||
|
SPC AC 112000
|
||||||
|
Day 3 Convective Outlook
|
||||||
|
NWS Storm Prediction Center Norman OK
|
||||||
|
|
||||||
|
...SUMMARY...
|
||||||
|
A corridor of strong to severe storms may develop near a frontal zone.
|
||||||
|
|
||||||
|
...DISCUSSION...
|
||||||
|
Confidence remains moderate for organized storms.
|
||||||
|
</pre>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
24
internal/providers/spc/time.go
Normal file
24
internal/providers/spc/time.go
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
package spc
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ParseISOTimestamp parses SPC ISO timestamps from GeoJSON properties.
|
||||||
|
func ParseISOTimestamp(value string) (time.Time, error) {
|
||||||
|
return time.Parse(time.RFC3339, strings.TrimSpace(value))
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseOptionalISOTimestamp(value string) *time.Time {
|
||||||
|
value = strings.TrimSpace(value)
|
||||||
|
if value == "" {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
t, err := ParseISOTimestamp(value)
|
||||||
|
if err != nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
tt := t.UTC()
|
||||||
|
return &tt
|
||||||
|
}
|
||||||
@@ -8,29 +8,44 @@
|
|||||||
// Canonical input schemas:
|
// Canonical input schemas:
|
||||||
// - weather.observation.v1 -> model.WeatherObservation
|
// - weather.observation.v1 -> model.WeatherObservation
|
||||||
// - weather.forecast.v1 -> model.WeatherForecastRun
|
// - weather.forecast.v1 -> model.WeatherForecastRun
|
||||||
|
// - weather.forecast_discussion.v1 -> model.WeatherForecastDiscussion
|
||||||
|
// - weather.weather_story.v1 -> model.WeatherStoryRun
|
||||||
// - weather.alert.v1 -> model.WeatherAlertRun
|
// - weather.alert.v1 -> model.WeatherAlertRun
|
||||||
|
// - weather.outlook.v2 -> model.WeatherOutlookRun
|
||||||
//
|
//
|
||||||
// Parent/child relationships:
|
// Parent/child relationships:
|
||||||
// - observations.event_id -> observation_present_weather.event_id
|
// - observations.event_id -> observation_present_weather.event_id
|
||||||
// - forecasts.event_id -> forecast_periods.run_event_id
|
// - forecasts.event_id -> forecast_periods.run_event_id
|
||||||
|
// - forecast_discussions.event_id -> forecast_discussion_key_messages.run_event_id
|
||||||
|
// - weather_story_runs.event_id -> weather_stories.run_event_id
|
||||||
// - alert_runs.event_id -> alerts.run_event_id
|
// - alert_runs.event_id -> alerts.run_event_id
|
||||||
// - alerts.(run_event_id, alert_index) -> alert_references.(run_event_id, alert_index)
|
// - alerts.(run_event_id, alert_index) -> alert_references.(run_event_id, alert_index)
|
||||||
|
// - outlook_runs.event_id -> outlooks.run_event_id
|
||||||
|
// - outlook_runs.event_id -> outlook_discussions.run_event_id
|
||||||
//
|
//
|
||||||
// Dedupe and retention behavior:
|
// Dedupe and retention behavior:
|
||||||
// - Parent primary keys (event_id): observations, forecasts, alert_runs.
|
// - Parent primary keys (event_id): observations, forecasts, alert_runs, outlook_runs.
|
||||||
// - Child primary keys use positional indexes to preserve payload order.
|
// - Child primary keys use positional indexes to preserve payload order.
|
||||||
// - Prune columns:
|
// - Prune columns:
|
||||||
// - observations.observed_at
|
// - observations.observed_at
|
||||||
// - observation_present_weather.observed_at
|
// - observation_present_weather.observed_at
|
||||||
// - forecasts.issued_at
|
// - forecasts.issued_at
|
||||||
// - forecast_periods.issued_at
|
// - forecast_periods.issued_at
|
||||||
|
// - forecast_discussions.issued_at
|
||||||
|
// - forecast_discussion_key_messages.issued_at
|
||||||
|
// - weather_story_runs.as_of
|
||||||
|
// - weather_stories.as_of
|
||||||
// - alert_runs.as_of
|
// - alert_runs.as_of
|
||||||
// - alerts.as_of
|
// - alerts.as_of
|
||||||
// - alert_references.as_of
|
// - alert_references.as_of
|
||||||
|
// - outlook_runs.as_of
|
||||||
|
// - outlooks.as_of
|
||||||
|
// - outlook_discussions.as_of
|
||||||
//
|
//
|
||||||
// Envelope field mapping (shared parent columns)
|
// Envelope field mapping (shared parent columns)
|
||||||
//
|
//
|
||||||
// These columns exist on observations, forecasts, and alert_runs:
|
// These columns exist on parent tables such as observations, forecasts,
|
||||||
|
// forecast_discussions, weather_story_runs, alert_runs, and outlook_runs:
|
||||||
// - event_id TEXT -> event.id
|
// - event_id TEXT -> event.id
|
||||||
// - event_kind TEXT -> event.kind
|
// - event_kind TEXT -> event.kind
|
||||||
// - event_source TEXT -> event.source
|
// - event_source TEXT -> event.source
|
||||||
@@ -101,7 +116,7 @@
|
|||||||
// - end_time TIMESTAMPTZ -> payload.periods[i].endTime
|
// - end_time TIMESTAMPTZ -> payload.periods[i].endTime
|
||||||
// - name TEXT NULL -> payload.periods[i].name
|
// - name TEXT NULL -> payload.periods[i].name
|
||||||
// - is_day BOOLEAN NULL -> payload.periods[i].isDay
|
// - is_day BOOLEAN NULL -> payload.periods[i].isDay
|
||||||
// - condition_code INTEGER -> payload.periods[i].conditionCode
|
// - condition_code INTEGER NULL -> payload.periods[i].conditionCode
|
||||||
// - text_description TEXT NULL -> payload.periods[i].textDescription
|
// - text_description TEXT NULL -> payload.periods[i].textDescription
|
||||||
// - temperature_c DOUBLE PRECISION NULL -> payload.periods[i].temperatureC
|
// - temperature_c DOUBLE PRECISION NULL -> payload.periods[i].temperatureC
|
||||||
// - temperature_c_min DOUBLE PRECISION NULL -> payload.periods[i].temperatureCMin
|
// - temperature_c_min DOUBLE PRECISION NULL -> payload.periods[i].temperatureCMin
|
||||||
@@ -120,7 +135,35 @@
|
|||||||
// - snowfall_depth_mm DOUBLE PRECISION NULL -> payload.periods[i].snowfallDepthMm
|
// - snowfall_depth_mm DOUBLE PRECISION NULL -> payload.periods[i].snowfallDepthMm
|
||||||
// - uv_index DOUBLE PRECISION NULL -> payload.periods[i].uvIndex
|
// - uv_index DOUBLE PRECISION NULL -> payload.periods[i].uvIndex
|
||||||
//
|
//
|
||||||
// 5. alert_runs (PK: event_id)
|
// 5. weather_story_runs (PK: event_id)
|
||||||
|
//
|
||||||
|
// - event_id TEXT -> event.id
|
||||||
|
// - event_kind TEXT -> event.kind
|
||||||
|
// - event_source TEXT -> event.source
|
||||||
|
// - event_schema TEXT -> event.schema
|
||||||
|
// - event_emitted_at TIMESTAMPTZ -> event.emitted_at
|
||||||
|
// - event_effective_at TIMESTAMPTZ NULL -> event.effective_at
|
||||||
|
// - office_id TEXT NULL -> payload.officeId
|
||||||
|
// - as_of TIMESTAMPTZ -> payload.asOf
|
||||||
|
// - story_count INTEGER -> len(payload.stories)
|
||||||
|
//
|
||||||
|
// 6. weather_stories (PK: run_event_id, story_index)
|
||||||
|
//
|
||||||
|
// - run_event_id TEXT -> weather_story_runs.event_id / payload.stories[i]
|
||||||
|
// - story_index INTEGER -> i (array position in payload.stories)
|
||||||
|
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
|
||||||
|
// - office_id TEXT NULL -> payload.stories[i].officeId
|
||||||
|
// - start_time TIMESTAMPTZ -> payload.stories[i].startTime
|
||||||
|
// - end_time TIMESTAMPTZ -> payload.stories[i].endTime
|
||||||
|
// - updated_at TIMESTAMPTZ -> payload.stories[i].updatedAt
|
||||||
|
// - title TEXT NULL -> payload.stories[i].title
|
||||||
|
// - description TEXT NULL -> payload.stories[i].description
|
||||||
|
// - alt_text TEXT NULL -> payload.stories[i].altText
|
||||||
|
// - priority BOOLEAN -> payload.stories[i].priority
|
||||||
|
// - story_order INTEGER -> payload.stories[i].order
|
||||||
|
// - download_url TEXT NULL -> payload.stories[i].downloadUrl
|
||||||
|
//
|
||||||
|
// 7. alert_runs (PK: event_id)
|
||||||
//
|
//
|
||||||
// - event_id TEXT -> event.id
|
// - event_id TEXT -> event.id
|
||||||
// - event_kind TEXT -> event.kind
|
// - event_kind TEXT -> event.kind
|
||||||
@@ -135,7 +178,7 @@
|
|||||||
// - longitude DOUBLE PRECISION NULL -> payload.longitude
|
// - longitude DOUBLE PRECISION NULL -> payload.longitude
|
||||||
// - alert_count INTEGER -> len(payload.alerts)
|
// - alert_count INTEGER -> len(payload.alerts)
|
||||||
//
|
//
|
||||||
// 6. alerts (PK: run_event_id, alert_index)
|
// 8. alerts (PK: run_event_id, alert_index)
|
||||||
//
|
//
|
||||||
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i]
|
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i]
|
||||||
// - alert_index INTEGER -> i (array position in payload.alerts)
|
// - alert_index INTEGER -> i (array position in payload.alerts)
|
||||||
@@ -155,12 +198,13 @@
|
|||||||
// - sent TIMESTAMPTZ NULL -> payload.alerts[i].sent
|
// - sent TIMESTAMPTZ NULL -> payload.alerts[i].sent
|
||||||
// - effective TIMESTAMPTZ NULL -> payload.alerts[i].effective
|
// - effective TIMESTAMPTZ NULL -> payload.alerts[i].effective
|
||||||
// - onset TIMESTAMPTZ NULL -> payload.alerts[i].onset
|
// - onset TIMESTAMPTZ NULL -> payload.alerts[i].onset
|
||||||
|
// - ends TIMESTAMPTZ NULL -> payload.alerts[i].ends
|
||||||
// - expires TIMESTAMPTZ NULL -> payload.alerts[i].expires
|
// - expires TIMESTAMPTZ NULL -> payload.alerts[i].expires
|
||||||
// - area_description TEXT NULL -> payload.alerts[i].areaDescription
|
// - area_description TEXT NULL -> payload.alerts[i].areaDescription
|
||||||
// - sender_name TEXT NULL -> payload.alerts[i].senderName
|
// - sender_name TEXT NULL -> payload.alerts[i].senderName
|
||||||
// - reference_count INTEGER -> len(payload.alerts[i].references)
|
// - reference_count INTEGER -> len(payload.alerts[i].references)
|
||||||
//
|
//
|
||||||
// 7. alert_references (PK: run_event_id, alert_index, reference_index)
|
// 9. alert_references (PK: run_event_id, alert_index, reference_index)
|
||||||
//
|
//
|
||||||
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i].references[j]
|
// - run_event_id TEXT -> alert_runs.event_id / payload.alerts[i].references[j]
|
||||||
// - alert_index INTEGER -> i (array position in payload.alerts)
|
// - alert_index INTEGER -> i (array position in payload.alerts)
|
||||||
@@ -171,6 +215,57 @@
|
|||||||
// - sender TEXT NULL -> payload.alerts[i].references[j].sender
|
// - sender TEXT NULL -> payload.alerts[i].references[j].sender
|
||||||
// - sent TIMESTAMPTZ NULL -> payload.alerts[i].references[j].sent
|
// - sent TIMESTAMPTZ NULL -> payload.alerts[i].references[j].sent
|
||||||
//
|
//
|
||||||
|
// 10. outlook_runs (PK: event_id)
|
||||||
|
//
|
||||||
|
// - event_id TEXT -> event.id
|
||||||
|
// - event_kind TEXT -> event.kind
|
||||||
|
// - event_source TEXT -> event.source
|
||||||
|
// - event_schema TEXT -> event.schema
|
||||||
|
// - event_emitted_at TIMESTAMPTZ -> event.emitted_at
|
||||||
|
// - event_effective_at TIMESTAMPTZ NULL -> event.effective_at
|
||||||
|
// - location_id TEXT NULL -> payload.locationId
|
||||||
|
// - location_name TEXT NULL -> payload.locationName
|
||||||
|
// - latitude DOUBLE PRECISION NULL -> payload.latitude
|
||||||
|
// - longitude DOUBLE PRECISION NULL -> payload.longitude
|
||||||
|
// - as_of TIMESTAMPTZ -> payload.asOf
|
||||||
|
// - issued_at TIMESTAMPTZ NULL -> payload.issuedAt
|
||||||
|
// - outlook_count INTEGER -> len(payload.outlooks)
|
||||||
|
// - discussion_count INTEGER -> len(payload.discussions)
|
||||||
|
//
|
||||||
|
// 11. outlooks (PK: run_event_id, outlook_index)
|
||||||
|
//
|
||||||
|
// - run_event_id TEXT -> outlook_runs.event_id / payload.outlooks[i]
|
||||||
|
// - outlook_index INTEGER -> i (array position in payload.outlooks)
|
||||||
|
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
|
||||||
|
// - outlook_id TEXT -> payload.outlooks[i].id
|
||||||
|
// - provider TEXT -> payload.outlooks[i].provider
|
||||||
|
// - product TEXT -> payload.outlooks[i].product
|
||||||
|
// - day INTEGER -> payload.outlooks[i].day
|
||||||
|
// - outlook_type TEXT -> payload.outlooks[i].outlookType
|
||||||
|
// - label TEXT -> payload.outlooks[i].label
|
||||||
|
// - label_text TEXT NULL -> payload.outlooks[i].labelText
|
||||||
|
// - severity_rank INTEGER NULL -> payload.outlooks[i].severityRank
|
||||||
|
// - valid_from TIMESTAMPTZ -> payload.outlooks[i].validFrom
|
||||||
|
// - valid_to TIMESTAMPTZ -> payload.outlooks[i].validTo
|
||||||
|
// - issued_at TIMESTAMPTZ -> payload.outlooks[i].issuedAt
|
||||||
|
// - expires_at TIMESTAMPTZ -> payload.outlooks[i].expiresAt
|
||||||
|
// - forecaster TEXT NULL -> payload.outlooks[i].forecaster
|
||||||
|
// - source_url TEXT NULL -> payload.outlooks[i].sourceUrl
|
||||||
|
// - image_url TEXT NULL -> payload.outlooks[i].imageUrl
|
||||||
|
// - contains_location BOOLEAN -> payload.outlooks[i].containsLocation
|
||||||
|
// - geometry_json TEXT -> compact JSON payload.outlooks[i].geometry
|
||||||
|
//
|
||||||
|
// 12. outlook_discussions (PK: run_event_id, discussion_index)
|
||||||
|
//
|
||||||
|
// - run_event_id TEXT -> outlook_runs.event_id / payload.discussions[i]
|
||||||
|
// - discussion_index INTEGER -> i (array position in payload.discussions)
|
||||||
|
// - as_of TIMESTAMPTZ -> payload.asOf (copied from parent)
|
||||||
|
// - day INTEGER -> payload.discussions[i].day
|
||||||
|
// - headline TEXT NULL -> payload.discussions[i].headline
|
||||||
|
// - summary TEXT NULL -> payload.discussions[i].summary
|
||||||
|
// - discussion TEXT NULL -> payload.discussions[i].discussion
|
||||||
|
// - updated_at TIMESTAMPTZ NULL -> payload.discussions[i].updatedAt
|
||||||
|
//
|
||||||
// Reconstructing canonical JSON payloads
|
// Reconstructing canonical JSON payloads
|
||||||
//
|
//
|
||||||
// - WeatherObservation:
|
// - WeatherObservation:
|
||||||
@@ -181,8 +276,17 @@
|
|||||||
// read one row from forecasts, then join forecast_periods by run_event_id
|
// read one row from forecasts, then join forecast_periods by run_event_id
|
||||||
// ordered by period_index to rebuild periods.
|
// ordered by period_index to rebuild periods.
|
||||||
//
|
//
|
||||||
|
// - WeatherStoryRun:
|
||||||
|
// read one row from weather_story_runs, then join weather_stories by
|
||||||
|
// run_event_id ordered by story_index to rebuild stories.
|
||||||
|
//
|
||||||
// - WeatherAlertRun:
|
// - WeatherAlertRun:
|
||||||
// read one row from alert_runs, join alerts by run_event_id ordered by
|
// read one row from alert_runs, join alerts by run_event_id ordered by
|
||||||
// alert_index, then join alert_references by (run_event_id, alert_index)
|
// alert_index, then join alert_references by (run_event_id, alert_index)
|
||||||
// ordered by reference_index to rebuild references per alert.
|
// ordered by reference_index to rebuild references per alert.
|
||||||
|
//
|
||||||
|
// - WeatherOutlookRun:
|
||||||
|
// read one row from outlook_runs, join outlooks by run_event_id ordered by
|
||||||
|
// outlook_index to rebuild outlooks, then join outlook_discussions by
|
||||||
|
// run_event_id ordered by discussion_index to rebuild discussions.
|
||||||
package postgres
|
package postgres
|
||||||
|
|||||||
33
internal/sinks/postgres/docs_test.go
Normal file
33
internal/sinks/postgres/docs_test.go
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
package postgres
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDocumentedOutlookDiscussionStorage(t *testing.T) {
|
||||||
|
for _, path := range []string{
|
||||||
|
"../../../docs/integrations/postgres.md",
|
||||||
|
"../../../docs/internal/postgres-sink.md",
|
||||||
|
} {
|
||||||
|
t.Run(path, func(t *testing.T) {
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(%s) error = %v", path, err)
|
||||||
|
}
|
||||||
|
doc := string(raw)
|
||||||
|
|
||||||
|
for _, want := range []string{
|
||||||
|
tableOutlookDiscussions,
|
||||||
|
"discussion_count",
|
||||||
|
"discussion_index",
|
||||||
|
"weather.outlook.v2",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(doc, want) {
|
||||||
|
t.Fatalf("%s missing %q", path, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,6 +1,7 @@
|
|||||||
package postgres
|
package postgres
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"fmt"
|
"fmt"
|
||||||
@@ -22,8 +23,12 @@ func mapPostgresEvent(_ context.Context, e fkevent.Event) ([]fksinks.PostgresWri
|
|||||||
return mapForecastEvent(e)
|
return mapForecastEvent(e)
|
||||||
case standards.SchemaWeatherForecastDiscussionV1:
|
case standards.SchemaWeatherForecastDiscussionV1:
|
||||||
return mapForecastDiscussionEvent(e)
|
return mapForecastDiscussionEvent(e)
|
||||||
|
case standards.SchemaWeatherStoryV1:
|
||||||
|
return mapWeatherStoryEvent(e)
|
||||||
case standards.SchemaWeatherAlertV1:
|
case standards.SchemaWeatherAlertV1:
|
||||||
return mapAlertEvent(e)
|
return mapAlertEvent(e)
|
||||||
|
case standards.SchemaWeatherOutlookV2:
|
||||||
|
return mapOutlookEvent(e)
|
||||||
default:
|
default:
|
||||||
return nil, nil
|
return nil, nil
|
||||||
}
|
}
|
||||||
@@ -43,13 +48,7 @@ func mapObservationEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
|
|
||||||
writes = append(writes, fksinks.PostgresWrite{
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
Table: tableObservations,
|
Table: tableObservations,
|
||||||
Values: map[string]any{
|
Values: parentEventValues(e, map[string]any{
|
||||||
"event_id": e.ID,
|
|
||||||
"event_kind": string(e.Kind),
|
|
||||||
"event_source": e.Source,
|
|
||||||
"event_schema": e.Schema,
|
|
||||||
"event_emitted_at": e.EmittedAt.UTC(),
|
|
||||||
"event_effective_at": nullableTime(e.EffectiveAt),
|
|
||||||
"station_id": nullableString(obs.StationID),
|
"station_id": nullableString(obs.StationID),
|
||||||
"station_name": nullableString(obs.StationName),
|
"station_name": nullableString(obs.StationName),
|
||||||
"observed_at": observedAt,
|
"observed_at": observedAt,
|
||||||
@@ -65,7 +64,7 @@ func mapObservationEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
"visibility_meters": nullableFloat64(obs.VisibilityMeters),
|
"visibility_meters": nullableFloat64(obs.VisibilityMeters),
|
||||||
"relative_humidity_percent": nullableFloat64(obs.RelativeHumidityPercent),
|
"relative_humidity_percent": nullableFloat64(obs.RelativeHumidityPercent),
|
||||||
"apparent_temperature_c": nullableFloat64(obs.ApparentTemperatureC),
|
"apparent_temperature_c": nullableFloat64(obs.ApparentTemperatureC),
|
||||||
},
|
}),
|
||||||
})
|
})
|
||||||
|
|
||||||
for i, pw := range obs.PresentWeather {
|
for i, pw := range obs.PresentWeather {
|
||||||
@@ -104,23 +103,17 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
|
|
||||||
writes = append(writes, fksinks.PostgresWrite{
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
Table: tableForecasts,
|
Table: tableForecasts,
|
||||||
Values: map[string]any{
|
Values: parentEventValues(e, map[string]any{
|
||||||
"event_id": e.ID,
|
"location_id": nullableString(run.LocationID),
|
||||||
"event_kind": string(e.Kind),
|
"location_name": nullableString(run.LocationName),
|
||||||
"event_source": e.Source,
|
"issued_at": issuedAt,
|
||||||
"event_schema": e.Schema,
|
"updated_at": nullableTime(run.UpdatedAt),
|
||||||
"event_emitted_at": e.EmittedAt.UTC(),
|
"product": string(run.Product),
|
||||||
"event_effective_at": nullableTime(e.EffectiveAt),
|
"latitude": nullableFloat64(run.Latitude),
|
||||||
"location_id": nullableString(run.LocationID),
|
"longitude": nullableFloat64(run.Longitude),
|
||||||
"location_name": nullableString(run.LocationName),
|
"elevation_meters": nullableFloat64(run.ElevationMeters),
|
||||||
"issued_at": issuedAt,
|
"period_count": len(run.Periods),
|
||||||
"updated_at": nullableTime(run.UpdatedAt),
|
}),
|
||||||
"product": string(run.Product),
|
|
||||||
"latitude": nullableFloat64(run.Latitude),
|
|
||||||
"longitude": nullableFloat64(run.Longitude),
|
|
||||||
"elevation_meters": nullableFloat64(run.ElevationMeters),
|
|
||||||
"period_count": len(run.Periods),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
|
|
||||||
for i, p := range run.Periods {
|
for i, p := range run.Periods {
|
||||||
@@ -137,7 +130,7 @@ func mapForecastEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
"end_time": p.EndTime.UTC(),
|
"end_time": p.EndTime.UTC(),
|
||||||
"name": nullableString(p.Name),
|
"name": nullableString(p.Name),
|
||||||
"is_day": nullableBool(p.IsDay),
|
"is_day": nullableBool(p.IsDay),
|
||||||
"condition_code": int(p.ConditionCode),
|
"condition_code": nullableWMOCode(p.ConditionCode),
|
||||||
"text_description": nullableString(p.TextDescription),
|
"text_description": nullableString(p.TextDescription),
|
||||||
"temperature_c": nullableFloat64(p.TemperatureC),
|
"temperature_c": nullableFloat64(p.TemperatureC),
|
||||||
"temperature_c_min": nullableFloat64(p.TemperatureCMin),
|
"temperature_c_min": nullableFloat64(p.TemperatureCMin),
|
||||||
@@ -181,13 +174,7 @@ func mapForecastDiscussionEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error
|
|||||||
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.KeyMessages))
|
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.KeyMessages))
|
||||||
writes = append(writes, fksinks.PostgresWrite{
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
Table: tableForecastDiscussions,
|
Table: tableForecastDiscussions,
|
||||||
Values: map[string]any{
|
Values: parentEventValues(e, map[string]any{
|
||||||
"event_id": e.ID,
|
|
||||||
"event_kind": string(e.Kind),
|
|
||||||
"event_source": e.Source,
|
|
||||||
"event_schema": e.Schema,
|
|
||||||
"event_emitted_at": e.EmittedAt.UTC(),
|
|
||||||
"event_effective_at": nullableTime(e.EffectiveAt),
|
|
||||||
"office_id": nullableString(run.OfficeID),
|
"office_id": nullableString(run.OfficeID),
|
||||||
"office_name": nullableString(run.OfficeName),
|
"office_name": nullableString(run.OfficeName),
|
||||||
"issued_at": issuedAt,
|
"issued_at": issuedAt,
|
||||||
@@ -200,7 +187,7 @@ func mapForecastDiscussionEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error
|
|||||||
"long_term_issued_at": longTermIssuedAt,
|
"long_term_issued_at": longTermIssuedAt,
|
||||||
"long_term_text": longTermText,
|
"long_term_text": longTermText,
|
||||||
"key_message_count": len(run.KeyMessages),
|
"key_message_count": len(run.KeyMessages),
|
||||||
},
|
}),
|
||||||
})
|
})
|
||||||
|
|
||||||
for i, msg := range run.KeyMessages {
|
for i, msg := range run.KeyMessages {
|
||||||
@@ -218,6 +205,53 @@ func mapForecastDiscussionEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error
|
|||||||
return writes, nil
|
return writes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func mapWeatherStoryEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
||||||
|
run, err := decodePayload[model.WeatherStoryRun](e.Payload)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("decode weather story payload: %w", err)
|
||||||
|
}
|
||||||
|
if run.AsOf.IsZero() {
|
||||||
|
return nil, fmt.Errorf("decode weather story payload: asOf is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
asOf := run.AsOf.UTC()
|
||||||
|
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.Stories))
|
||||||
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
|
Table: tableWeatherStoryRuns,
|
||||||
|
Values: parentEventValues(e, map[string]any{
|
||||||
|
"office_id": nullableString(run.OfficeID),
|
||||||
|
"as_of": asOf,
|
||||||
|
"story_count": len(run.Stories),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
|
||||||
|
for i, story := range run.Stories {
|
||||||
|
if story.StartTime.IsZero() || story.EndTime.IsZero() || story.UpdatedAt.IsZero() {
|
||||||
|
return nil, fmt.Errorf("decode weather story payload: stories[%d] startTime/endTime/updatedAt are required", i)
|
||||||
|
}
|
||||||
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
|
Table: tableWeatherStories,
|
||||||
|
Values: map[string]any{
|
||||||
|
"run_event_id": e.ID,
|
||||||
|
"story_index": i,
|
||||||
|
"as_of": asOf,
|
||||||
|
"office_id": nullableString(story.OfficeID),
|
||||||
|
"start_time": story.StartTime.UTC(),
|
||||||
|
"end_time": story.EndTime.UTC(),
|
||||||
|
"updated_at": story.UpdatedAt.UTC(),
|
||||||
|
"title": nullableString(story.Title),
|
||||||
|
"description": nullableString(story.Description),
|
||||||
|
"alt_text": nullableString(story.AltText),
|
||||||
|
"priority": story.Priority,
|
||||||
|
"story_order": story.Order,
|
||||||
|
"download_url": nullableString(story.DownloadURL),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return writes, nil
|
||||||
|
}
|
||||||
|
|
||||||
func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
||||||
run, err := decodePayload[model.WeatherAlertRun](e.Payload)
|
run, err := decodePayload[model.WeatherAlertRun](e.Payload)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -232,20 +266,14 @@ func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
|
|
||||||
writes = append(writes, fksinks.PostgresWrite{
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
Table: tableAlertRuns,
|
Table: tableAlertRuns,
|
||||||
Values: map[string]any{
|
Values: parentEventValues(e, map[string]any{
|
||||||
"event_id": e.ID,
|
"location_id": nullableString(run.LocationID),
|
||||||
"event_kind": string(e.Kind),
|
"location_name": nullableString(run.LocationName),
|
||||||
"event_source": e.Source,
|
"as_of": asOf,
|
||||||
"event_schema": e.Schema,
|
"latitude": nullableFloat64(run.Latitude),
|
||||||
"event_emitted_at": e.EmittedAt.UTC(),
|
"longitude": nullableFloat64(run.Longitude),
|
||||||
"event_effective_at": nullableTime(e.EffectiveAt),
|
"alert_count": len(run.Alerts),
|
||||||
"location_id": nullableString(run.LocationID),
|
}),
|
||||||
"location_name": nullableString(run.LocationName),
|
|
||||||
"as_of": asOf,
|
|
||||||
"latitude": nullableFloat64(run.Latitude),
|
|
||||||
"longitude": nullableFloat64(run.Longitude),
|
|
||||||
"alert_count": len(run.Alerts),
|
|
||||||
},
|
|
||||||
})
|
})
|
||||||
|
|
||||||
for i, a := range run.Alerts {
|
for i, a := range run.Alerts {
|
||||||
@@ -274,6 +302,7 @@ func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
"sent": nullableTime(a.Sent),
|
"sent": nullableTime(a.Sent),
|
||||||
"effective": nullableTime(a.Effective),
|
"effective": nullableTime(a.Effective),
|
||||||
"onset": nullableTime(a.Onset),
|
"onset": nullableTime(a.Onset),
|
||||||
|
"ends": nullableTime(a.Ends),
|
||||||
"expires": nullableTime(a.Expires),
|
"expires": nullableTime(a.Expires),
|
||||||
"area_description": nullableString(a.AreaDescription),
|
"area_description": nullableString(a.AreaDescription),
|
||||||
"sender_name": nullableString(a.SenderName),
|
"sender_name": nullableString(a.SenderName),
|
||||||
@@ -301,6 +330,143 @@ func mapAlertEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
|||||||
return writes, nil
|
return writes, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func mapOutlookEvent(e fkevent.Event) ([]fksinks.PostgresWrite, error) {
|
||||||
|
run, err := decodePayload[model.WeatherOutlookRun](e.Payload)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("decode outlook payload: %w", err)
|
||||||
|
}
|
||||||
|
if run.AsOf.IsZero() {
|
||||||
|
return nil, fmt.Errorf("decode outlook payload: asOf is required")
|
||||||
|
}
|
||||||
|
|
||||||
|
asOf := run.AsOf.UTC()
|
||||||
|
if err := validateOutlookDiscussions(run.Discussions); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
writes := make([]fksinks.PostgresWrite, 0, 1+len(run.Outlooks)+len(run.Discussions))
|
||||||
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
|
Table: tableOutlookRuns,
|
||||||
|
Values: parentEventValues(e, map[string]any{
|
||||||
|
"location_id": nullableString(run.LocationID),
|
||||||
|
"location_name": nullableString(run.LocationName),
|
||||||
|
"latitude": nullableFloat64(run.Latitude),
|
||||||
|
"longitude": nullableFloat64(run.Longitude),
|
||||||
|
"as_of": asOf,
|
||||||
|
"issued_at": nullableTime(run.IssuedAt),
|
||||||
|
"outlook_count": len(run.Outlooks),
|
||||||
|
"discussion_count": len(run.Discussions),
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
|
||||||
|
for i, outlook := range run.Outlooks {
|
||||||
|
if err := validateOutlook(outlook, i); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
geometryJSON, err := requiredCompactJSONText(outlook.Geometry)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("decode outlook payload: outlooks[%d].geometry: %w", i, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
|
Table: tableOutlooks,
|
||||||
|
Values: map[string]any{
|
||||||
|
"run_event_id": e.ID,
|
||||||
|
"outlook_index": i,
|
||||||
|
"as_of": asOf,
|
||||||
|
"outlook_id": outlook.ID,
|
||||||
|
"provider": outlook.Provider,
|
||||||
|
"product": outlook.Product,
|
||||||
|
"day": outlook.Day,
|
||||||
|
"outlook_type": outlook.OutlookType,
|
||||||
|
"label": outlook.Label,
|
||||||
|
"label_text": nullableString(outlook.LabelText),
|
||||||
|
"severity_rank": nullableInt(outlook.SeverityRank),
|
||||||
|
"valid_from": outlook.ValidFrom.UTC(),
|
||||||
|
"valid_to": outlook.ValidTo.UTC(),
|
||||||
|
"issued_at": outlook.IssuedAt.UTC(),
|
||||||
|
"expires_at": outlook.ExpiresAt.UTC(),
|
||||||
|
"forecaster": nullableString(outlook.Forecaster),
|
||||||
|
"source_url": nullableString(outlook.SourceURL),
|
||||||
|
"image_url": nullableString(outlook.ImageURL),
|
||||||
|
"contains_location": outlook.ContainsLocation,
|
||||||
|
"geometry_json": geometryJSON,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
for i, discussion := range run.Discussions {
|
||||||
|
writes = append(writes, fksinks.PostgresWrite{
|
||||||
|
Table: tableOutlookDiscussions,
|
||||||
|
Values: map[string]any{
|
||||||
|
"run_event_id": e.ID,
|
||||||
|
"discussion_index": i,
|
||||||
|
"as_of": asOf,
|
||||||
|
"day": discussion.Day,
|
||||||
|
"headline": nullableString(discussion.Headline),
|
||||||
|
"summary": nullableString(discussion.Summary),
|
||||||
|
"discussion": nullableString(discussion.Discussion),
|
||||||
|
"updated_at": nullableTime(discussion.UpdatedAt),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return writes, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOutlook(outlook model.WeatherOutlook, index int) error {
|
||||||
|
if strings.TrimSpace(outlook.ID) == "" {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].id is required", index)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outlook.Provider) == "" {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].provider is required", index)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outlook.Product) == "" {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].product is required", index)
|
||||||
|
}
|
||||||
|
if outlook.Day == 0 {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].day is required", index)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outlook.OutlookType) == "" {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].outlookType is required", index)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(outlook.Label) == "" {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].label is required", index)
|
||||||
|
}
|
||||||
|
if outlook.ValidFrom.IsZero() || outlook.ValidTo.IsZero() {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d] validFrom/validTo are required", index)
|
||||||
|
}
|
||||||
|
if outlook.IssuedAt.IsZero() || outlook.ExpiresAt.IsZero() {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d] issuedAt/expiresAt are required", index)
|
||||||
|
}
|
||||||
|
if len(outlook.Geometry) == 0 {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].geometry is required", index)
|
||||||
|
}
|
||||||
|
if !outlook.ContainsLocation {
|
||||||
|
return fmt.Errorf("decode outlook payload: outlooks[%d].containsLocation must be true", index)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOutlookDiscussions(discussions []model.WeatherOutlookDiscussion) error {
|
||||||
|
seenDays := map[int]int{}
|
||||||
|
for i, discussion := range discussions {
|
||||||
|
if discussion.Day < 1 || discussion.Day > 3 {
|
||||||
|
return fmt.Errorf("decode outlook payload: discussions[%d].day must be 1, 2, or 3", i)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(discussion.Headline) == "" &&
|
||||||
|
strings.TrimSpace(discussion.Summary) == "" &&
|
||||||
|
strings.TrimSpace(discussion.Discussion) == "" {
|
||||||
|
return fmt.Errorf("decode outlook payload: discussions[%d] headline, summary, or discussion is required", i)
|
||||||
|
}
|
||||||
|
if first, ok := seenDays[discussion.Day]; ok {
|
||||||
|
return fmt.Errorf("decode outlook payload: discussions[%d].day duplicates discussions[%d].day %d", i, first, discussion.Day)
|
||||||
|
}
|
||||||
|
seenDays[discussion.Day] = i
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func decodePayload[T any](payload any) (T, error) {
|
func decodePayload[T any](payload any) (T, error) {
|
||||||
var out T
|
var out T
|
||||||
if payload == nil {
|
if payload == nil {
|
||||||
@@ -327,6 +493,21 @@ func decodePayload[T any](payload any) (T, error) {
|
|||||||
return out, nil
|
return out, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func parentEventValues(e fkevent.Event, values map[string]any) map[string]any {
|
||||||
|
out := map[string]any{
|
||||||
|
"event_id": e.ID,
|
||||||
|
"event_kind": string(e.Kind),
|
||||||
|
"event_source": e.Source,
|
||||||
|
"event_schema": e.Schema,
|
||||||
|
"event_emitted_at": e.EmittedAt.UTC(),
|
||||||
|
"event_effective_at": nullableTime(e.EffectiveAt),
|
||||||
|
}
|
||||||
|
for k, v := range values {
|
||||||
|
out[k] = v
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func nullableDiscussionSection(section *model.WeatherForecastDiscussionSection) (any, any, any) {
|
func nullableDiscussionSection(section *model.WeatherForecastDiscussionSection) (any, any, any) {
|
||||||
if section == nil {
|
if section == nil {
|
||||||
return nil, nil, nil
|
return nil, nil, nil
|
||||||
@@ -363,6 +544,13 @@ func nullableBool(v *bool) any {
|
|||||||
return *v
|
return *v
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func nullableInt(v *int) any {
|
||||||
|
if v == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return *v
|
||||||
|
}
|
||||||
|
|
||||||
func nullableTime(v *time.Time) any {
|
func nullableTime(v *time.Time) any {
|
||||||
if v == nil || v.IsZero() {
|
if v == nil || v.IsZero() {
|
||||||
return nil
|
return nil
|
||||||
@@ -370,6 +558,13 @@ func nullableTime(v *time.Time) any {
|
|||||||
return v.UTC()
|
return v.UTC()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func nullableWMOCode(v *model.WMOCode) any {
|
||||||
|
if v == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return int(*v)
|
||||||
|
}
|
||||||
|
|
||||||
func compactJSONText(v any) (any, error) {
|
func compactJSONText(v any) (any, error) {
|
||||||
if v == nil {
|
if v == nil {
|
||||||
return nil, nil
|
return nil, nil
|
||||||
@@ -383,3 +578,19 @@ func compactJSONText(v any) (any, error) {
|
|||||||
}
|
}
|
||||||
return string(b), nil
|
return string(b), nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func requiredCompactJSONText(v any) (string, error) {
|
||||||
|
compact, err := compactJSONText(v)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
s, ok := compact.(string)
|
||||||
|
if !ok || strings.TrimSpace(s) == "" || strings.TrimSpace(s) == "null" {
|
||||||
|
return "", fmt.Errorf("is required")
|
||||||
|
}
|
||||||
|
var buf bytes.Buffer
|
||||||
|
if err := json.Compact(&buf, []byte(s)); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return buf.String(), nil
|
||||||
|
}
|
||||||
|
|||||||
@@ -27,7 +27,7 @@ func TestMapPostgresEventObservationStructPayload(t *testing.T) {
|
|||||||
PresentWeather: []model.PresentWeather{{Raw: map[string]any{"a": 1, "b": "x"}}},
|
PresentWeather: []model.PresentWeather{{Raw: map[string]any{"a": 1, "b": "x"}}},
|
||||||
}
|
}
|
||||||
|
|
||||||
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherObservationV1, "observation", obs))
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherObservationV1, standards.KindObservation, obs))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("mapPostgresEvent() error = %v", err)
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -63,18 +63,18 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
|
|||||||
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
|
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
|
||||||
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
|
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
|
||||||
IsDay: &isDay,
|
IsDay: &isDay,
|
||||||
ConditionCode: model.WMOCode(2),
|
ConditionCode: wmoCodePtr(model.WMOCode(2)),
|
||||||
TemperatureC: &temp,
|
TemperatureC: &temp,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
StartTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
|
StartTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
|
||||||
EndTime: time.Date(2026, 3, 16, 21, 0, 0, 0, time.UTC),
|
EndTime: time.Date(2026, 3, 16, 21, 0, 0, 0, time.UTC),
|
||||||
ConditionCode: model.WMOCode(3),
|
ConditionCode: nil,
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastV1, "forecast", run))
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastV1, standards.KindForecast, run))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("mapPostgresEvent() error = %v", err)
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -94,12 +94,17 @@ func TestMapPostgresEventForecastStructPayload(t *testing.T) {
|
|||||||
if got := writes[1].Values["period_index"]; got != 0 {
|
if got := writes[1].Values["period_index"]; got != 0 {
|
||||||
t.Fatalf("first period index = %#v, want 0", got)
|
t.Fatalf("first period index = %#v, want 0", got)
|
||||||
}
|
}
|
||||||
|
if got := writes[2].Values["condition_code"]; got != nil {
|
||||||
|
t.Fatalf("second period condition_code = %#v, want nil", got)
|
||||||
|
}
|
||||||
|
|
||||||
assertAllWritesIncludeAllColumns(t, writes)
|
assertAllWritesIncludeAllColumns(t, writes)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMapPostgresEventAlertStructPayload(t *testing.T) {
|
func TestMapPostgresEventAlertStructPayload(t *testing.T) {
|
||||||
sent := time.Date(2026, 3, 16, 17, 0, 0, 0, time.UTC)
|
sent := time.Date(2026, 3, 16, 17, 0, 0, 0, time.UTC)
|
||||||
|
ends := time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC)
|
||||||
|
expires := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
|
||||||
run := model.WeatherAlertRun{
|
run := model.WeatherAlertRun{
|
||||||
AsOf: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
|
AsOf: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
|
||||||
Alerts: []model.WeatherAlert{
|
Alerts: []model.WeatherAlert{
|
||||||
@@ -107,6 +112,8 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
|
|||||||
ID: "urn:alert:1",
|
ID: "urn:alert:1",
|
||||||
Headline: "Winter Weather Advisory",
|
Headline: "Winter Weather Advisory",
|
||||||
Severity: "Moderate",
|
Severity: "Moderate",
|
||||||
|
Ends: &ends,
|
||||||
|
Expires: &expires,
|
||||||
References: []model.AlertReference{
|
References: []model.AlertReference{
|
||||||
{ID: "urn:ref:1", Sent: &sent},
|
{ID: "urn:ref:1", Sent: &sent},
|
||||||
{Identifier: "ref-two"},
|
{Identifier: "ref-two"},
|
||||||
@@ -119,7 +126,7 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherAlertV1, "alert", run))
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherAlertV1, standards.KindAlert, run))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("mapPostgresEvent() error = %v", err)
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -142,6 +149,20 @@ func TestMapPostgresEventAlertStructPayload(t *testing.T) {
|
|||||||
if got := firstAlert.Values["reference_count"]; got != 2 {
|
if got := firstAlert.Values["reference_count"]; got != 2 {
|
||||||
t.Fatalf("alerts reference_count = %#v, want 2", got)
|
t.Fatalf("alerts reference_count = %#v, want 2", got)
|
||||||
}
|
}
|
||||||
|
if got := firstAlert.Values["ends"]; got != ends {
|
||||||
|
t.Fatalf("alerts ends = %#v, want %#v", got, ends)
|
||||||
|
}
|
||||||
|
if got := firstAlert.Values["expires"]; got != expires {
|
||||||
|
t.Fatalf("alerts expires = %#v, want %#v", got, expires)
|
||||||
|
}
|
||||||
|
|
||||||
|
alertWrites := writesForTable(writes, tableAlerts)
|
||||||
|
if len(alertWrites) != 2 {
|
||||||
|
t.Fatalf("alert writes len = %d, want 2", len(alertWrites))
|
||||||
|
}
|
||||||
|
if got := alertWrites[1].Values["ends"]; got != nil {
|
||||||
|
t.Fatalf("second alert ends = %#v, want nil", got)
|
||||||
|
}
|
||||||
|
|
||||||
assertAllWritesIncludeAllColumns(t, writes)
|
assertAllWritesIncludeAllColumns(t, writes)
|
||||||
}
|
}
|
||||||
@@ -160,7 +181,7 @@ func TestMapPostgresEventForecastDiscussionStructPayload(t *testing.T) {
|
|||||||
LongTerm: &model.WeatherForecastDiscussionSection{Text: "Long term text"},
|
LongTerm: &model.WeatherForecastDiscussionSection{Text: "Long term text"},
|
||||||
}
|
}
|
||||||
|
|
||||||
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, "forecast_discussion", run))
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, standards.KindForecastDiscussion, run))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("mapPostgresEvent() error = %v", err)
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -190,6 +211,399 @@ func TestMapPostgresEventForecastDiscussionStructPayload(t *testing.T) {
|
|||||||
assertAllWritesIncludeAllColumns(t, writes)
|
assertAllWritesIncludeAllColumns(t, writes)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventWeatherStoryStructPayload(t *testing.T) {
|
||||||
|
run := model.WeatherStoryRun{
|
||||||
|
OfficeID: "LSX",
|
||||||
|
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
|
||||||
|
Stories: []model.WeatherStory{
|
||||||
|
{
|
||||||
|
OfficeID: "LSX",
|
||||||
|
StartTime: time.Date(2026, 5, 30, 8, 46, 0, 0, time.UTC),
|
||||||
|
EndTime: time.Date(2026, 5, 31, 11, 0, 0, 0, time.UTC),
|
||||||
|
UpdatedAt: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
|
||||||
|
Title: "Several Chances for Rain Through Monday",
|
||||||
|
Description: "Scattered showers and thunderstorms.",
|
||||||
|
AltText: "This slide shows the forecast.",
|
||||||
|
Priority: true,
|
||||||
|
Order: 1,
|
||||||
|
DownloadURL: "https://api.weather.gov/offices/LSX/weatherstories/download/story-1",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, standards.KindWeatherStory, run))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(writes) != 2 {
|
||||||
|
t.Fatalf("mapPostgresEvent() writes len = %d, want 2", len(writes))
|
||||||
|
}
|
||||||
|
if writes[0].Table != tableWeatherStoryRuns {
|
||||||
|
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableWeatherStoryRuns)
|
||||||
|
}
|
||||||
|
if got := writes[0].Values["story_count"]; got != 1 {
|
||||||
|
t.Fatalf("weather_story_runs story_count = %#v, want 1", got)
|
||||||
|
}
|
||||||
|
if writes[1].Table != tableWeatherStories {
|
||||||
|
t.Fatalf("writes[1].Table = %q, want %q", writes[1].Table, tableWeatherStories)
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["download_url"]; got != "https://api.weather.gov/offices/LSX/weatherstories/download/story-1" {
|
||||||
|
t.Fatalf("weather_stories download_url = %#v", got)
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["story_order"]; got != 1 {
|
||||||
|
t.Fatalf("weather_stories story_order = %#v, want 1", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertAllWritesIncludeAllColumns(t, writes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookStructPayload(t *testing.T) {
|
||||||
|
lat := 38.6239
|
||||||
|
lon := -90.3571
|
||||||
|
issuedAt := time.Date(2026, 6, 11, 19, 45, 0, 0, time.FixedZone("UTC-5", -5*60*60))
|
||||||
|
updatedAt := time.Date(2026, 6, 11, 21, 15, 0, 0, time.FixedZone("UTC-5", -5*60*60))
|
||||||
|
severity := 3
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
LocationID: "stl",
|
||||||
|
LocationName: "St. Louis, MO",
|
||||||
|
Latitude: &lat,
|
||||||
|
Longitude: &lon,
|
||||||
|
AsOf: time.Date(2026, 6, 12, 0, 45, 0, 0, time.UTC),
|
||||||
|
IssuedAt: &issuedAt,
|
||||||
|
Outlooks: []model.WeatherOutlook{
|
||||||
|
{
|
||||||
|
ID: "outlook-1",
|
||||||
|
Provider: "spc",
|
||||||
|
Product: "convective",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: "categorical",
|
||||||
|
Label: "SLGT",
|
||||||
|
LabelText: "Slight Risk",
|
||||||
|
SeverityRank: &severity,
|
||||||
|
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.FixedZone("UTC-5", -5*60*60)),
|
||||||
|
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
IssuedAt: issuedAt,
|
||||||
|
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
Forecaster: "SMITH",
|
||||||
|
SourceURL: "https://example.invalid/day1.geojson",
|
||||||
|
ContainsLocation: true,
|
||||||
|
Geometry: json.RawMessage(`{ "type" : "Polygon", "coordinates" : [ [ [ -91.0, 38.0 ], [ -90.0, 38.0 ], [ -90.0, 39.0 ], [ -91.0, 39.0 ], [ -91.0, 38.0 ] ] ] }`),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
ID: "outlook-2",
|
||||||
|
Provider: "spc",
|
||||||
|
Product: "convective",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: "wind",
|
||||||
|
Label: "15",
|
||||||
|
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
|
||||||
|
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
ContainsLocation: true,
|
||||||
|
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-100,35],[-98,35],[-98,37],[-100,37],[-100,35]]]}`),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
Discussions: []model.WeatherOutlookDiscussion{
|
||||||
|
{
|
||||||
|
Day: 1,
|
||||||
|
Headline: "Day 1 Convective Outlook",
|
||||||
|
Summary: "Severe thunderstorms are possible.",
|
||||||
|
Discussion: "Full discussion text.",
|
||||||
|
UpdatedAt: &updatedAt,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(writes) != 4 {
|
||||||
|
t.Fatalf("mapPostgresEvent() writes len = %d, want 4", len(writes))
|
||||||
|
}
|
||||||
|
if writes[0].Table != tableOutlookRuns {
|
||||||
|
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableOutlookRuns)
|
||||||
|
}
|
||||||
|
if got := writes[0].Values["outlook_count"]; got != 2 {
|
||||||
|
t.Fatalf("outlook_runs outlook_count = %#v, want 2", got)
|
||||||
|
}
|
||||||
|
if got := writes[0].Values["discussion_count"]; got != 1 {
|
||||||
|
t.Fatalf("outlook_runs discussion_count = %#v, want 1", got)
|
||||||
|
}
|
||||||
|
if got := writes[0].Values["issued_at"]; got != issuedAt.UTC() {
|
||||||
|
t.Fatalf("outlook_runs issued_at = %#v, want UTC %s", got, issuedAt.UTC())
|
||||||
|
}
|
||||||
|
if writes[1].Table != tableOutlooks || writes[2].Table != tableOutlooks {
|
||||||
|
t.Fatalf("outlook writes not in expected order")
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["outlook_index"]; got != 0 {
|
||||||
|
t.Fatalf("first outlook index = %#v, want 0", got)
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["outlook_id"]; got != "outlook-1" {
|
||||||
|
t.Fatalf("first outlook_id = %#v, want outlook-1", got)
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["provider"]; got != "spc" {
|
||||||
|
t.Fatalf("first provider = %#v, want spc", got)
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["valid_from"]; got != run.Outlooks[0].ValidFrom.UTC() {
|
||||||
|
t.Fatalf("first valid_from = %#v, want UTC %s", got, run.Outlooks[0].ValidFrom.UTC())
|
||||||
|
}
|
||||||
|
if got := writes[1].Values["geometry_json"]; got != `{"type":"Polygon","coordinates":[[[-91.0,38.0],[-90.0,38.0],[-90.0,39.0],[-91.0,39.0],[-91.0,38.0]]]}` {
|
||||||
|
t.Fatalf("first geometry_json = %#v", got)
|
||||||
|
}
|
||||||
|
if got := writes[2].Values["contains_location"]; got != true {
|
||||||
|
t.Fatalf("second contains_location = %#v, want true", got)
|
||||||
|
}
|
||||||
|
if writes[3].Table != tableOutlookDiscussions {
|
||||||
|
t.Fatalf("writes[3].Table = %q, want %q", writes[3].Table, tableOutlookDiscussions)
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["discussion_index"]; got != 0 {
|
||||||
|
t.Fatalf("discussion_index = %#v, want 0", got)
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["as_of"]; got != run.AsOf.UTC() {
|
||||||
|
t.Fatalf("discussion as_of = %#v, want %s", got, run.AsOf.UTC())
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["day"]; got != 1 {
|
||||||
|
t.Fatalf("discussion day = %#v, want 1", got)
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["headline"]; got != "Day 1 Convective Outlook" {
|
||||||
|
t.Fatalf("discussion headline = %#v", got)
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["summary"]; got != "Severe thunderstorms are possible." {
|
||||||
|
t.Fatalf("discussion summary = %#v", got)
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["discussion"]; got != "Full discussion text." {
|
||||||
|
t.Fatalf("discussion text = %#v", got)
|
||||||
|
}
|
||||||
|
if got := writes[3].Values["updated_at"]; got != updatedAt.UTC() {
|
||||||
|
t.Fatalf("discussion updated_at = %#v, want UTC %s", got, updatedAt.UTC())
|
||||||
|
}
|
||||||
|
|
||||||
|
assertAllWritesIncludeAllColumns(t, writes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookEmptyLocalRun(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
}
|
||||||
|
|
||||||
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(writes) != 1 {
|
||||||
|
t.Fatalf("mapPostgresEvent() writes len = %d, want 1", len(writes))
|
||||||
|
}
|
||||||
|
if writes[0].Table != tableOutlookRuns {
|
||||||
|
t.Fatalf("writes[0].Table = %q, want %q", writes[0].Table, tableOutlookRuns)
|
||||||
|
}
|
||||||
|
if got := writes[0].Values["outlook_count"]; got != 0 {
|
||||||
|
t.Fatalf("outlook_runs outlook_count = %#v, want 0", got)
|
||||||
|
}
|
||||||
|
if got := writes[0].Values["discussion_count"]; got != 0 {
|
||||||
|
t.Fatalf("outlook_runs discussion_count = %#v, want 0", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertAllWritesIncludeAllColumns(t, writes)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsMissingAsOf(t *testing.T) {
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, model.WeatherOutlookRun{}))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want missing asOf error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "asOf is required") {
|
||||||
|
t.Fatalf("error = %q, want asOf context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsMissingIDAndProvider(t *testing.T) {
|
||||||
|
base := model.WeatherOutlook{
|
||||||
|
ID: "outlook-1",
|
||||||
|
Provider: "spc",
|
||||||
|
Product: "convective",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: "categorical",
|
||||||
|
Label: "SLGT",
|
||||||
|
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
|
||||||
|
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
ContainsLocation: true,
|
||||||
|
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-91,38],[-90,38],[-90,39],[-91,39],[-91,38]]]}`),
|
||||||
|
}
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*model.WeatherOutlook)
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "missing id",
|
||||||
|
mutate: func(outlook *model.WeatherOutlook) { outlook.ID = "" },
|
||||||
|
wantErr: "outlooks[0].id is required",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing provider",
|
||||||
|
mutate: func(outlook *model.WeatherOutlook) { outlook.Provider = "" },
|
||||||
|
wantErr: "outlooks[0].provider is required",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
outlook := base
|
||||||
|
tt.mutate(&outlook)
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Outlooks: []model.WeatherOutlook{outlook},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want %q", tt.wantErr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||||
|
t.Fatalf("error = %q, want %q", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsMissingRequiredTimes(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Outlooks: []model.WeatherOutlook{{
|
||||||
|
ID: "outlook-1",
|
||||||
|
Provider: "spc",
|
||||||
|
Product: "convective",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: "categorical",
|
||||||
|
Label: "SLGT",
|
||||||
|
ContainsLocation: true,
|
||||||
|
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-91,38],[-90,38],[-90,39],[-91,39],[-91,38]]]}`),
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want missing time error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "outlooks[0] validFrom/validTo are required") {
|
||||||
|
t.Fatalf("error = %q, want outlook time context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsEmptyGeometry(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Outlooks: []model.WeatherOutlook{{
|
||||||
|
ID: "outlook-1",
|
||||||
|
Provider: "spc",
|
||||||
|
Product: "convective",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: "categorical",
|
||||||
|
Label: "SLGT",
|
||||||
|
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
|
||||||
|
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
ContainsLocation: true,
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want geometry error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "outlooks[0].geometry is required") {
|
||||||
|
t.Fatalf("error = %q, want geometry context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsDuplicateDiscussionDay(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Discussions: []model.WeatherOutlookDiscussion{
|
||||||
|
{Day: 1, Discussion: "First day one discussion."},
|
||||||
|
{Day: 1, Discussion: "Duplicate day one discussion."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want duplicate discussion day error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "discussions[1].day duplicates discussions[0].day 1") {
|
||||||
|
t.Fatalf("error = %q, want duplicate discussion day context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsInvalidDiscussionDay(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Discussions: []model.WeatherOutlookDiscussion{{Day: 4, Discussion: "Invalid day."}},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want invalid discussion day error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "discussions[0].day must be 1, 2, or 3") {
|
||||||
|
t.Fatalf("error = %q, want invalid discussion day context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsEmptyDiscussionContent(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Discussions: []model.WeatherOutlookDiscussion{{Day: 1}},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want empty discussion content error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "discussions[0] headline, summary, or discussion is required") {
|
||||||
|
t.Fatalf("error = %q, want empty discussion content context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventOutlookRejectsContainsLocationFalse(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{
|
||||||
|
AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
Outlooks: []model.WeatherOutlook{validTestOutlook()},
|
||||||
|
}
|
||||||
|
run.Outlooks[0].ContainsLocation = false
|
||||||
|
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV2, standards.KindOutlook, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want containsLocation error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "outlooks[0].containsLocation must be true") {
|
||||||
|
t.Fatalf("error = %q, want containsLocation context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventWeatherStoryRejectsMissingAsOf(t *testing.T) {
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, standards.KindWeatherStory, model.WeatherStoryRun{}))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want missing asOf error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "asOf is required") {
|
||||||
|
t.Fatalf("error = %q, want asOf context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventWeatherStoryRejectsMissingStoryTimes(t *testing.T) {
|
||||||
|
run := model.WeatherStoryRun{
|
||||||
|
AsOf: time.Date(2026, 5, 30, 9, 0, 34, 0, time.UTC),
|
||||||
|
Stories: []model.WeatherStory{{Title: "missing times"}},
|
||||||
|
}
|
||||||
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherStoryV1, standards.KindWeatherStory, run))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = nil, want missing story times error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "stories[0] startTime/endTime/updatedAt are required") {
|
||||||
|
t.Fatalf("error = %q, want story time context", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestMapPostgresEventMapPayload(t *testing.T) {
|
func TestMapPostgresEventMapPayload(t *testing.T) {
|
||||||
run := model.WeatherForecastRun{
|
run := model.WeatherForecastRun{
|
||||||
IssuedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
|
IssuedAt: time.Date(2026, 3, 16, 18, 0, 0, 0, time.UTC),
|
||||||
@@ -198,7 +612,7 @@ func TestMapPostgresEventMapPayload(t *testing.T) {
|
|||||||
{
|
{
|
||||||
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
|
StartTime: time.Date(2026, 3, 16, 19, 0, 0, 0, time.UTC),
|
||||||
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
|
EndTime: time.Date(2026, 3, 16, 20, 0, 0, 0, time.UTC),
|
||||||
ConditionCode: model.WMOCode(2),
|
ConditionCode: wmoCodePtr(model.WMOCode(2)),
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -211,7 +625,7 @@ func TestMapPostgresEventMapPayload(t *testing.T) {
|
|||||||
t.Fatalf("json.Unmarshal() error = %v", err)
|
t.Fatalf("json.Unmarshal() error = %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastV1, "forecast", payload))
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastV1, standards.KindForecast, payload))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("mapPostgresEvent() error = %v", err)
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -226,7 +640,18 @@ func TestMapPostgresEventMapPayload(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestMapPostgresEventUnknownSchemaNoOp(t *testing.T) {
|
func TestMapPostgresEventUnknownSchemaNoOp(t *testing.T) {
|
||||||
writes, err := mapPostgresEvent(context.Background(), testEvent("weather.unknown.v1", "observation", map[string]any{"x": 1}))
|
writes, err := mapPostgresEvent(context.Background(), testEvent("weather.unknown.v1", standards.KindObservation, map[string]any{"x": 1}))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(writes) != 0 {
|
||||||
|
t.Fatalf("mapPostgresEvent() writes len = %d, want 0", len(writes))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestMapPostgresEventLegacyOutlookSchemaNoOp(t *testing.T) {
|
||||||
|
run := model.WeatherOutlookRun{AsOf: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC)}
|
||||||
|
writes, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherOutlookV1, standards.KindOutlook, run))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("mapPostgresEvent() error = %v", err)
|
t.Fatalf("mapPostgresEvent() error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -236,7 +661,7 @@ func TestMapPostgresEventUnknownSchemaNoOp(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestMapPostgresEventMalformedPayload(t *testing.T) {
|
func TestMapPostgresEventMalformedPayload(t *testing.T) {
|
||||||
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastV1, "forecast", "bad"))
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastV1, standards.KindForecast, "bad"))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatalf("mapPostgresEvent() expected error for malformed payload")
|
t.Fatalf("mapPostgresEvent() expected error for malformed payload")
|
||||||
}
|
}
|
||||||
@@ -246,7 +671,7 @@ func TestMapPostgresEventMalformedPayload(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestMapPostgresEventForecastDiscussionMalformedPayload(t *testing.T) {
|
func TestMapPostgresEventForecastDiscussionMalformedPayload(t *testing.T) {
|
||||||
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, "forecast_discussion", "bad"))
|
_, err := mapPostgresEvent(context.Background(), testEvent(standards.SchemaWeatherForecastDiscussionV1, standards.KindForecastDiscussion, "bad"))
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatalf("mapPostgresEvent() expected error for malformed payload")
|
t.Fatalf("mapPostgresEvent() expected error for malformed payload")
|
||||||
}
|
}
|
||||||
@@ -255,6 +680,56 @@ func TestMapPostgresEventForecastDiscussionMalformedPayload(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestParentEventValuesAddsEnvelopeAndPreservesProductValues(t *testing.T) {
|
||||||
|
emittedAt := time.Date(2026, 3, 16, 13, 31, 0, 0, time.FixedZone("CDT", -5*60*60))
|
||||||
|
effectiveAt := time.Date(2026, 3, 16, 13, 30, 0, 0, time.FixedZone("CDT", -5*60*60))
|
||||||
|
event := fkevent.Event{
|
||||||
|
ID: "evt-envelope",
|
||||||
|
Kind: fkevent.Kind(standards.KindForecast),
|
||||||
|
Source: "test-source",
|
||||||
|
Schema: standards.SchemaWeatherForecastV1,
|
||||||
|
EmittedAt: emittedAt,
|
||||||
|
EffectiveAt: &effectiveAt,
|
||||||
|
}
|
||||||
|
|
||||||
|
got := parentEventValues(event, map[string]any{"product_col": "product-value"})
|
||||||
|
|
||||||
|
assertParentEnvelopeValues(t, got, event)
|
||||||
|
if got["product_col"] != "product-value" {
|
||||||
|
t.Fatalf("product_col = %#v, want product-value", got["product_col"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParentEventValuesNullEffectiveAt(t *testing.T) {
|
||||||
|
base := fkevent.Event{
|
||||||
|
ID: "evt-envelope",
|
||||||
|
Kind: fkevent.Kind(standards.KindObservation),
|
||||||
|
Source: "test-source",
|
||||||
|
Schema: standards.SchemaWeatherObservationV1,
|
||||||
|
EmittedAt: time.Date(2026, 3, 16, 18, 31, 0, 0, time.UTC),
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range []struct {
|
||||||
|
name string
|
||||||
|
mut func(*fkevent.Event)
|
||||||
|
}{
|
||||||
|
{name: "nil", mut: func(*fkevent.Event) {}},
|
||||||
|
{name: "zero", mut: func(event *fkevent.Event) {
|
||||||
|
zero := time.Time{}
|
||||||
|
event.EffectiveAt = &zero
|
||||||
|
}},
|
||||||
|
} {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
event := base
|
||||||
|
tt.mut(&event)
|
||||||
|
got := parentEventValues(event, nil)
|
||||||
|
if got["event_effective_at"] != nil {
|
||||||
|
t.Fatalf("event_effective_at = %#v, want nil", got["event_effective_at"])
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func testEvent(schema string, kind fkevent.Kind, payload any) fkevent.Event {
|
func testEvent(schema string, kind fkevent.Kind, payload any) fkevent.Event {
|
||||||
effectiveAt := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
|
effectiveAt := time.Date(2026, 3, 16, 18, 30, 0, 0, time.UTC)
|
||||||
return fkevent.Event{
|
return fkevent.Event{
|
||||||
@@ -268,6 +743,30 @@ func testEvent(schema string, kind fkevent.Kind, payload any) fkevent.Event {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func assertParentEnvelopeValues(t *testing.T, values map[string]any, event fkevent.Event) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if got := values["event_id"]; got != event.ID {
|
||||||
|
t.Fatalf("event_id = %#v, want %q", got, event.ID)
|
||||||
|
}
|
||||||
|
if got := values["event_kind"]; got != string(event.Kind) {
|
||||||
|
t.Fatalf("event_kind = %#v, want %q", got, event.Kind)
|
||||||
|
}
|
||||||
|
if got := values["event_source"]; got != event.Source {
|
||||||
|
t.Fatalf("event_source = %#v, want %q", got, event.Source)
|
||||||
|
}
|
||||||
|
if got := values["event_schema"]; got != event.Schema {
|
||||||
|
t.Fatalf("event_schema = %#v, want %q", got, event.Schema)
|
||||||
|
}
|
||||||
|
if got := values["event_emitted_at"]; got != event.EmittedAt.UTC() {
|
||||||
|
t.Fatalf("event_emitted_at = %#v, want %s", got, event.EmittedAt.UTC())
|
||||||
|
}
|
||||||
|
wantEffective := nullableTime(event.EffectiveAt)
|
||||||
|
if got := values["event_effective_at"]; got != wantEffective {
|
||||||
|
t.Fatalf("event_effective_at = %#v, want %#v", got, wantEffective)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func firstWriteForTable(writes []fksinks.PostgresWrite, table string) (fksinks.PostgresWrite, bool) {
|
func firstWriteForTable(writes []fksinks.PostgresWrite, table string) (fksinks.PostgresWrite, bool) {
|
||||||
for _, w := range writes {
|
for _, w := range writes {
|
||||||
if w.Table == table {
|
if w.Table == table {
|
||||||
@@ -277,6 +776,16 @@ func firstWriteForTable(writes []fksinks.PostgresWrite, table string) (fksinks.P
|
|||||||
return fksinks.PostgresWrite{}, false
|
return fksinks.PostgresWrite{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func writesForTable(writes []fksinks.PostgresWrite, table string) []fksinks.PostgresWrite {
|
||||||
|
out := make([]fksinks.PostgresWrite, 0)
|
||||||
|
for _, w := range writes {
|
||||||
|
if w.Table == table {
|
||||||
|
out = append(out, w)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
func assertAllWritesIncludeAllColumns(t *testing.T, writes []fksinks.PostgresWrite) {
|
func assertAllWritesIncludeAllColumns(t *testing.T, writes []fksinks.PostgresWrite) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
colCounts := tableColumnCounts()
|
colCounts := tableColumnCounts()
|
||||||
@@ -299,3 +808,25 @@ func tableColumnCounts() map[string]int {
|
|||||||
}
|
}
|
||||||
return m
|
return m
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validTestOutlook() model.WeatherOutlook {
|
||||||
|
return model.WeatherOutlook{
|
||||||
|
ID: "outlook-1",
|
||||||
|
Provider: "spc",
|
||||||
|
Product: "convective",
|
||||||
|
Day: 1,
|
||||||
|
OutlookType: "categorical",
|
||||||
|
Label: "SLGT",
|
||||||
|
ValidFrom: time.Date(2026, 6, 11, 13, 0, 0, 0, time.UTC),
|
||||||
|
ValidTo: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
IssuedAt: time.Date(2026, 6, 11, 19, 45, 0, 0, time.UTC),
|
||||||
|
ExpiresAt: time.Date(2026, 6, 12, 12, 0, 0, 0, time.UTC),
|
||||||
|
ContainsLocation: true,
|
||||||
|
Geometry: json.RawMessage(`{"type":"Polygon","coordinates":[[[-91,38],[-90,38],[-90,39],[-91,39],[-91,38]]]}`),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func wmoCodePtr(v model.WMOCode) *model.WMOCode {
|
||||||
|
out := v
|
||||||
|
return &out
|
||||||
|
}
|
||||||
|
|||||||
@@ -11,9 +11,14 @@ const (
|
|||||||
tableForecastPeriods = "forecast_periods"
|
tableForecastPeriods = "forecast_periods"
|
||||||
tableForecastDiscussions = "forecast_discussions"
|
tableForecastDiscussions = "forecast_discussions"
|
||||||
tableForecastDiscussionKeyMessages = "forecast_discussion_key_messages"
|
tableForecastDiscussionKeyMessages = "forecast_discussion_key_messages"
|
||||||
|
tableWeatherStoryRuns = "weather_story_runs"
|
||||||
|
tableWeatherStories = "weather_stories"
|
||||||
tableAlertRuns = "alert_runs"
|
tableAlertRuns = "alert_runs"
|
||||||
tableAlerts = "alerts"
|
tableAlerts = "alerts"
|
||||||
tableAlertReferences = "alert_references"
|
tableAlertReferences = "alert_references"
|
||||||
|
tableOutlookRuns = "outlook_runs"
|
||||||
|
tableOutlooks = "outlooks"
|
||||||
|
tableOutlookDiscussions = "outlook_discussions"
|
||||||
)
|
)
|
||||||
|
|
||||||
// PostgresSchema returns weatherfeeder's Postgres schema definition.
|
// PostgresSchema returns weatherfeeder's Postgres schema definition.
|
||||||
@@ -22,13 +27,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
Tables: []fksinks.PostgresTable{
|
Tables: []fksinks.PostgresTable{
|
||||||
{
|
{
|
||||||
Name: tableObservations,
|
Name: tableObservations,
|
||||||
Columns: []fksinks.PostgresColumn{
|
Columns: parentEnvelopeColumns([]fksinks.PostgresColumn{
|
||||||
{Name: "event_id", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_kind", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_source", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_schema", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
|
|
||||||
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
|
|
||||||
{Name: "station_id", Type: "TEXT", Nullable: true},
|
{Name: "station_id", Type: "TEXT", Nullable: true},
|
||||||
{Name: "station_name", Type: "TEXT", Nullable: true},
|
{Name: "station_name", Type: "TEXT", Nullable: true},
|
||||||
{Name: "observed_at", Type: "TIMESTAMPTZ", Nullable: false},
|
{Name: "observed_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
@@ -44,7 +43,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
{Name: "visibility_meters", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "visibility_meters", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "relative_humidity_percent", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "relative_humidity_percent", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "apparent_temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "apparent_temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
},
|
}...),
|
||||||
PrimaryKey: []string{"event_id"},
|
PrimaryKey: []string{"event_id"},
|
||||||
PruneColumn: "observed_at",
|
PruneColumn: "observed_at",
|
||||||
Indexes: []fksinks.PostgresIndex{
|
Indexes: []fksinks.PostgresIndex{
|
||||||
@@ -69,13 +68,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: tableForecasts,
|
Name: tableForecasts,
|
||||||
Columns: []fksinks.PostgresColumn{
|
Columns: parentEnvelopeColumns([]fksinks.PostgresColumn{
|
||||||
{Name: "event_id", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_kind", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_source", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_schema", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
|
|
||||||
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
|
|
||||||
{Name: "location_id", Type: "TEXT", Nullable: true},
|
{Name: "location_id", Type: "TEXT", Nullable: true},
|
||||||
{Name: "location_name", Type: "TEXT", Nullable: true},
|
{Name: "location_name", Type: "TEXT", Nullable: true},
|
||||||
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
|
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
@@ -85,7 +78,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
{Name: "longitude", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "longitude", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "elevation_meters", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "elevation_meters", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "period_count", Type: "INTEGER", Nullable: false},
|
{Name: "period_count", Type: "INTEGER", Nullable: false},
|
||||||
},
|
}...),
|
||||||
PrimaryKey: []string{"event_id"},
|
PrimaryKey: []string{"event_id"},
|
||||||
PruneColumn: "issued_at",
|
PruneColumn: "issued_at",
|
||||||
Indexes: []fksinks.PostgresIndex{
|
Indexes: []fksinks.PostgresIndex{
|
||||||
@@ -104,7 +97,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
|
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
{Name: "name", Type: "TEXT", Nullable: true},
|
{Name: "name", Type: "TEXT", Nullable: true},
|
||||||
{Name: "is_day", Type: "BOOLEAN", Nullable: true},
|
{Name: "is_day", Type: "BOOLEAN", Nullable: true},
|
||||||
{Name: "condition_code", Type: "INTEGER", Nullable: false},
|
{Name: "condition_code", Type: "INTEGER", Nullable: true},
|
||||||
{Name: "text_description", Type: "TEXT", Nullable: true},
|
{Name: "text_description", Type: "TEXT", Nullable: true},
|
||||||
{Name: "temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "temperature_c", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "temperature_c_min", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "temperature_c_min", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
@@ -133,13 +126,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: tableForecastDiscussions,
|
Name: tableForecastDiscussions,
|
||||||
Columns: []fksinks.PostgresColumn{
|
Columns: parentEnvelopeColumns([]fksinks.PostgresColumn{
|
||||||
{Name: "event_id", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_kind", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_source", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_schema", Type: "TEXT", Nullable: false},
|
|
||||||
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
|
|
||||||
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
|
|
||||||
{Name: "office_id", Type: "TEXT", Nullable: true},
|
{Name: "office_id", Type: "TEXT", Nullable: true},
|
||||||
{Name: "office_name", Type: "TEXT", Nullable: true},
|
{Name: "office_name", Type: "TEXT", Nullable: true},
|
||||||
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
|
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
@@ -152,7 +139,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
{Name: "long_term_issued_at", Type: "TIMESTAMPTZ", Nullable: true},
|
{Name: "long_term_issued_at", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
{Name: "long_term_text", Type: "TEXT", Nullable: true},
|
{Name: "long_term_text", Type: "TEXT", Nullable: true},
|
||||||
{Name: "key_message_count", Type: "INTEGER", Nullable: false},
|
{Name: "key_message_count", Type: "INTEGER", Nullable: false},
|
||||||
},
|
}...),
|
||||||
PrimaryKey: []string{"event_id"},
|
PrimaryKey: []string{"event_id"},
|
||||||
PruneColumn: "issued_at",
|
PruneColumn: "issued_at",
|
||||||
Indexes: []fksinks.PostgresIndex{
|
Indexes: []fksinks.PostgresIndex{
|
||||||
@@ -175,21 +162,54 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
},
|
},
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
Name: tableAlertRuns,
|
Name: tableWeatherStoryRuns,
|
||||||
|
Columns: parentEnvelopeColumns([]fksinks.PostgresColumn{
|
||||||
|
{Name: "office_id", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "story_count", Type: "INTEGER", Nullable: false},
|
||||||
|
}...),
|
||||||
|
PrimaryKey: []string{"event_id"},
|
||||||
|
PruneColumn: "as_of",
|
||||||
|
Indexes: []fksinks.PostgresIndex{
|
||||||
|
{Name: "idx_wf_story_run_office_as_of", Columns: []string{"office_id", "as_of"}},
|
||||||
|
{Name: "idx_wf_story_run_as_of", Columns: []string{"as_of"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: tableWeatherStories,
|
||||||
Columns: []fksinks.PostgresColumn{
|
Columns: []fksinks.PostgresColumn{
|
||||||
{Name: "event_id", Type: "TEXT", Nullable: false},
|
{Name: "run_event_id", Type: "TEXT REFERENCES weather_story_runs(event_id) ON DELETE CASCADE", Nullable: false},
|
||||||
{Name: "event_kind", Type: "TEXT", Nullable: false},
|
{Name: "story_index", Type: "INTEGER", Nullable: false},
|
||||||
{Name: "event_source", Type: "TEXT", Nullable: false},
|
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
{Name: "event_schema", Type: "TEXT", Nullable: false},
|
{Name: "office_id", Type: "TEXT", Nullable: true},
|
||||||
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
|
{Name: "start_time", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
|
{Name: "end_time", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "title", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "description", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "alt_text", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "priority", Type: "BOOLEAN", Nullable: false},
|
||||||
|
{Name: "story_order", Type: "INTEGER", Nullable: false},
|
||||||
|
{Name: "download_url", Type: "TEXT", Nullable: true},
|
||||||
|
},
|
||||||
|
PrimaryKey: []string{"run_event_id", "story_index"},
|
||||||
|
PruneColumn: "as_of",
|
||||||
|
Indexes: []fksinks.PostgresIndex{
|
||||||
|
{Name: "idx_wf_stories_start_time", Columns: []string{"start_time"}},
|
||||||
|
{Name: "idx_wf_stories_end_time", Columns: []string{"end_time"}},
|
||||||
|
{Name: "idx_wf_stories_updated_at", Columns: []string{"updated_at"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: tableAlertRuns,
|
||||||
|
Columns: parentEnvelopeColumns([]fksinks.PostgresColumn{
|
||||||
{Name: "location_id", Type: "TEXT", Nullable: true},
|
{Name: "location_id", Type: "TEXT", Nullable: true},
|
||||||
{Name: "location_name", Type: "TEXT", Nullable: true},
|
{Name: "location_name", Type: "TEXT", Nullable: true},
|
||||||
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
{Name: "latitude", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "latitude", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "longitude", Type: "DOUBLE PRECISION", Nullable: true},
|
{Name: "longitude", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
{Name: "alert_count", Type: "INTEGER", Nullable: false},
|
{Name: "alert_count", Type: "INTEGER", Nullable: false},
|
||||||
},
|
}...),
|
||||||
PrimaryKey: []string{"event_id"},
|
PrimaryKey: []string{"event_id"},
|
||||||
PruneColumn: "as_of",
|
PruneColumn: "as_of",
|
||||||
Indexes: []fksinks.PostgresIndex{
|
Indexes: []fksinks.PostgresIndex{
|
||||||
@@ -218,6 +238,7 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
{Name: "sent", Type: "TIMESTAMPTZ", Nullable: true},
|
{Name: "sent", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
{Name: "effective", Type: "TIMESTAMPTZ", Nullable: true},
|
{Name: "effective", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
{Name: "onset", Type: "TIMESTAMPTZ", Nullable: true},
|
{Name: "onset", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
|
{Name: "ends", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
{Name: "expires", Type: "TIMESTAMPTZ", Nullable: true},
|
{Name: "expires", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
{Name: "area_description", Type: "TEXT", Nullable: true},
|
{Name: "area_description", Type: "TEXT", Nullable: true},
|
||||||
{Name: "sender_name", Type: "TEXT", Nullable: true},
|
{Name: "sender_name", Type: "TEXT", Nullable: true},
|
||||||
@@ -250,7 +271,89 @@ func PostgresSchema() fksinks.PostgresSchema {
|
|||||||
{Name: "idx_wf_alert_refs_sent", Columns: []string{"sent"}},
|
{Name: "idx_wf_alert_refs_sent", Columns: []string{"sent"}},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
Name: tableOutlookRuns,
|
||||||
|
Columns: parentEnvelopeColumns([]fksinks.PostgresColumn{
|
||||||
|
{Name: "location_id", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "location_name", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "latitude", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
|
{Name: "longitude", Type: "DOUBLE PRECISION", Nullable: true},
|
||||||
|
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
|
{Name: "outlook_count", Type: "INTEGER", Nullable: false},
|
||||||
|
{Name: "discussion_count", Type: "INTEGER", Nullable: false},
|
||||||
|
}...),
|
||||||
|
PrimaryKey: []string{"event_id"},
|
||||||
|
PruneColumn: "as_of",
|
||||||
|
Indexes: []fksinks.PostgresIndex{
|
||||||
|
{Name: "idx_wf_outlook_run_location_as_of", Columns: []string{"location_id", "as_of"}},
|
||||||
|
{Name: "idx_wf_outlook_run_as_of", Columns: []string{"as_of"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: tableOutlooks,
|
||||||
|
Columns: []fksinks.PostgresColumn{
|
||||||
|
{Name: "run_event_id", Type: "TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE", Nullable: false},
|
||||||
|
{Name: "outlook_index", Type: "INTEGER", Nullable: false},
|
||||||
|
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "outlook_id", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "provider", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "product", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "day", Type: "INTEGER", Nullable: false},
|
||||||
|
{Name: "outlook_type", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "label", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "label_text", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "severity_rank", Type: "INTEGER", Nullable: true},
|
||||||
|
{Name: "valid_from", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "valid_to", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "issued_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "expires_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "forecaster", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "source_url", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "image_url", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "contains_location", Type: "BOOLEAN", Nullable: false},
|
||||||
|
{Name: "geometry_json", Type: "TEXT", Nullable: false},
|
||||||
|
},
|
||||||
|
PrimaryKey: []string{"run_event_id", "outlook_index"},
|
||||||
|
PruneColumn: "as_of",
|
||||||
|
Indexes: []fksinks.PostgresIndex{
|
||||||
|
{Name: "idx_wf_outlooks_contains_valid", Columns: []string{"contains_location", "valid_from", "valid_to"}},
|
||||||
|
{Name: "idx_wf_outlooks_day_type_label", Columns: []string{"day", "outlook_type", "label"}},
|
||||||
|
{Name: "idx_wf_outlooks_valid", Columns: []string{"valid_from", "valid_to"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: tableOutlookDiscussions,
|
||||||
|
Columns: []fksinks.PostgresColumn{
|
||||||
|
{Name: "run_event_id", Type: "TEXT REFERENCES outlook_runs(event_id) ON DELETE CASCADE", Nullable: false},
|
||||||
|
{Name: "discussion_index", Type: "INTEGER", Nullable: false},
|
||||||
|
{Name: "as_of", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "day", Type: "INTEGER", Nullable: false},
|
||||||
|
{Name: "headline", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "summary", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "discussion", Type: "TEXT", Nullable: true},
|
||||||
|
{Name: "updated_at", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
|
},
|
||||||
|
PrimaryKey: []string{"run_event_id", "discussion_index"},
|
||||||
|
PruneColumn: "as_of",
|
||||||
|
Indexes: []fksinks.PostgresIndex{
|
||||||
|
{Name: "idx_wf_outlook_discussions_day_as_of", Columns: []string{"day", "as_of"}},
|
||||||
|
{Name: "idx_wf_outlook_discussions_run_day", Columns: []string{"run_event_id", "day"}, Unique: true},
|
||||||
|
},
|
||||||
|
},
|
||||||
},
|
},
|
||||||
MapEvent: mapPostgresEvent,
|
MapEvent: mapPostgresEvent,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func parentEnvelopeColumns(extra ...fksinks.PostgresColumn) []fksinks.PostgresColumn {
|
||||||
|
columns := []fksinks.PostgresColumn{
|
||||||
|
{Name: "event_id", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "event_kind", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "event_source", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "event_schema", Type: "TEXT", Nullable: false},
|
||||||
|
{Name: "event_emitted_at", Type: "TIMESTAMPTZ", Nullable: false},
|
||||||
|
{Name: "event_effective_at", Type: "TIMESTAMPTZ", Nullable: true},
|
||||||
|
}
|
||||||
|
return append(columns, extra...)
|
||||||
|
}
|
||||||
|
|||||||
@@ -1,6 +1,12 @@
|
|||||||
package postgres
|
package postgres
|
||||||
|
|
||||||
import "testing"
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
fksinks "gitea.maximumdirect.net/ejr/feedkit/sinks"
|
||||||
|
)
|
||||||
|
|
||||||
func TestWeatherPostgresSchemaShape(t *testing.T) {
|
func TestWeatherPostgresSchemaShape(t *testing.T) {
|
||||||
s := PostgresSchema()
|
s := PostgresSchema()
|
||||||
@@ -15,9 +21,14 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
|
|||||||
tableForecastPeriods: true,
|
tableForecastPeriods: true,
|
||||||
tableForecastDiscussions: true,
|
tableForecastDiscussions: true,
|
||||||
tableForecastDiscussionKeyMessages: true,
|
tableForecastDiscussionKeyMessages: true,
|
||||||
|
tableWeatherStoryRuns: true,
|
||||||
|
tableWeatherStories: true,
|
||||||
tableAlertRuns: true,
|
tableAlertRuns: true,
|
||||||
tableAlerts: true,
|
tableAlerts: true,
|
||||||
tableAlertReferences: true,
|
tableAlertReferences: true,
|
||||||
|
tableOutlookRuns: true,
|
||||||
|
tableOutlooks: true,
|
||||||
|
tableOutlookDiscussions: true,
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(s.Tables) != len(wantTables) {
|
if len(s.Tables) != len(wantTables) {
|
||||||
@@ -40,3 +51,168 @@ func TestWeatherPostgresSchemaShape(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestWeatherPostgresSchemaIncludesOutlookTables(t *testing.T) {
|
||||||
|
runColumns := columnsForTable(t, tableOutlookRuns)
|
||||||
|
for _, col := range []string{"event_id", "event_kind", "event_source", "event_schema", "event_emitted_at", "event_effective_at", "location_id", "location_name", "latitude", "longitude", "as_of", "issued_at", "outlook_count", "discussion_count"} {
|
||||||
|
if !runColumns[col] {
|
||||||
|
t.Fatalf("%s missing %s column", tableOutlookRuns, col)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assertTablePrimaryKey(t, tableOutlookRuns, []string{"event_id"})
|
||||||
|
assertTableIndex(t, tableOutlookRuns, "idx_wf_outlook_run_location_as_of", []string{"location_id", "as_of"})
|
||||||
|
assertTableIndex(t, tableOutlookRuns, "idx_wf_outlook_run_as_of", []string{"as_of"})
|
||||||
|
|
||||||
|
outlookColumns := columnsForTable(t, tableOutlooks)
|
||||||
|
for _, col := range []string{"run_event_id", "outlook_index", "as_of", "outlook_id", "provider", "product", "day", "outlook_type", "label", "label_text", "severity_rank", "valid_from", "valid_to", "issued_at", "expires_at", "forecaster", "source_url", "image_url", "contains_location", "geometry_json"} {
|
||||||
|
if !outlookColumns[col] {
|
||||||
|
t.Fatalf("%s missing %s column", tableOutlooks, col)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, col := range []string{"headline", "summary", "discussion"} {
|
||||||
|
if outlookColumns[col] {
|
||||||
|
t.Fatalf("%s still includes legacy %s column", tableOutlooks, col)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assertTablePrimaryKey(t, tableOutlooks, []string{"run_event_id", "outlook_index"})
|
||||||
|
assertTableIndex(t, tableOutlooks, "idx_wf_outlooks_contains_valid", []string{"contains_location", "valid_from", "valid_to"})
|
||||||
|
assertTableIndex(t, tableOutlooks, "idx_wf_outlooks_day_type_label", []string{"day", "outlook_type", "label"})
|
||||||
|
assertTableIndex(t, tableOutlooks, "idx_wf_outlooks_valid", []string{"valid_from", "valid_to"})
|
||||||
|
|
||||||
|
discussionColumns := columnsForTable(t, tableOutlookDiscussions)
|
||||||
|
for _, col := range []string{"run_event_id", "discussion_index", "as_of", "day", "headline", "summary", "discussion", "updated_at"} {
|
||||||
|
if !discussionColumns[col] {
|
||||||
|
t.Fatalf("%s missing %s column", tableOutlookDiscussions, col)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assertTablePrimaryKey(t, tableOutlookDiscussions, []string{"run_event_id", "discussion_index"})
|
||||||
|
assertTablePruneColumn(t, tableOutlookDiscussions, "as_of")
|
||||||
|
assertTableIndex(t, tableOutlookDiscussions, "idx_wf_outlook_discussions_day_as_of", []string{"day", "as_of"})
|
||||||
|
assertTableUniqueIndex(t, tableOutlookDiscussions, "idx_wf_outlook_discussions_run_day", []string{"run_event_id", "day"})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWeatherPostgresSchemaIncludesAlertEndsColumn(t *testing.T) {
|
||||||
|
alertColumns := columnsForTable(t, tableAlerts)
|
||||||
|
for _, col := range []string{"run_event_id", "alert_index", "as_of", "alert_id", "onset", "ends", "expires"} {
|
||||||
|
if !alertColumns[col] {
|
||||||
|
t.Fatalf("%s missing %s column", tableAlerts, col)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWeatherPostgresSchemaIncludesWeatherStoryColumns(t *testing.T) {
|
||||||
|
runColumns := columnsForTable(t, tableWeatherStoryRuns)
|
||||||
|
if !runColumns["as_of"] {
|
||||||
|
t.Fatalf("%s missing as_of column", tableWeatherStoryRuns)
|
||||||
|
}
|
||||||
|
if !runColumns["story_count"] {
|
||||||
|
t.Fatalf("%s missing story_count column", tableWeatherStoryRuns)
|
||||||
|
}
|
||||||
|
|
||||||
|
storyColumns := columnsForTable(t, tableWeatherStories)
|
||||||
|
for _, col := range []string{"start_time", "end_time", "updated_at", "title", "description", "alt_text", "priority", "story_order", "download_url"} {
|
||||||
|
if !storyColumns[col] {
|
||||||
|
t.Fatalf("%s missing %s column", tableWeatherStories, col)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestWeatherPostgresSchemaParentTablesStartWithEnvelopeColumns(t *testing.T) {
|
||||||
|
for _, table := range []string{
|
||||||
|
tableObservations,
|
||||||
|
tableForecasts,
|
||||||
|
tableForecastDiscussions,
|
||||||
|
tableWeatherStoryRuns,
|
||||||
|
tableAlertRuns,
|
||||||
|
tableOutlookRuns,
|
||||||
|
} {
|
||||||
|
t.Run(table, func(t *testing.T) {
|
||||||
|
columns := orderedColumnsForTable(t, table)
|
||||||
|
want := parentEnvelopeColumns()
|
||||||
|
if len(columns) < len(want) {
|
||||||
|
t.Fatalf("%s has %d columns, want at least %d", table, len(columns), len(want))
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(columns[:len(want)], want) {
|
||||||
|
t.Fatalf("%s envelope prefix = %#v, want %#v", table, columns[:len(want)], want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTablePrimaryKey(t *testing.T, table string, want []string) {
|
||||||
|
t.Helper()
|
||||||
|
tbl := tableByName(t, table)
|
||||||
|
if strings.Join(tbl.PrimaryKey, ",") != strings.Join(want, ",") {
|
||||||
|
t.Fatalf("%s primary key = %#v, want %#v", table, tbl.PrimaryKey, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTablePruneColumn(t *testing.T, table string, want string) {
|
||||||
|
t.Helper()
|
||||||
|
tbl := tableByName(t, table)
|
||||||
|
if tbl.PruneColumn != want {
|
||||||
|
t.Fatalf("%s prune column = %q, want %q", table, tbl.PruneColumn, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTableIndex(t *testing.T, table string, name string, want []string) {
|
||||||
|
t.Helper()
|
||||||
|
assertTableIndexWithUnique(t, table, name, want, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTableUniqueIndex(t *testing.T, table string, name string, want []string) {
|
||||||
|
t.Helper()
|
||||||
|
assertTableIndexWithUnique(t, table, name, want, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertTableIndexWithUnique(t *testing.T, table string, name string, want []string, unique bool) {
|
||||||
|
t.Helper()
|
||||||
|
tbl := tableByName(t, table)
|
||||||
|
for _, idx := range tbl.Indexes {
|
||||||
|
if idx.Name == name {
|
||||||
|
if strings.Join(idx.Columns, ",") != strings.Join(want, ",") {
|
||||||
|
t.Fatalf("%s index %s columns = %#v, want %#v", table, name, idx.Columns, want)
|
||||||
|
}
|
||||||
|
if idx.Unique != unique {
|
||||||
|
t.Fatalf("%s index %s unique = %v, want %v", table, name, idx.Unique, unique)
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatalf("%s missing index %s", table, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
func tableByName(t *testing.T, table string) fksinks.PostgresTable {
|
||||||
|
t.Helper()
|
||||||
|
for _, tbl := range PostgresSchema().Tables {
|
||||||
|
if tbl.Name == table {
|
||||||
|
return tbl
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatalf("missing table %q", table)
|
||||||
|
return fksinks.PostgresTable{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func orderedColumnsForTable(t *testing.T, table string) []fksinks.PostgresColumn {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
schema := PostgresSchema()
|
||||||
|
for _, tbl := range schema.Tables {
|
||||||
|
if tbl.Name == table {
|
||||||
|
return tbl.Columns
|
||||||
|
}
|
||||||
|
}
|
||||||
|
t.Fatalf("missing table %q", table)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func columnsForTable(t *testing.T, table string) map[string]bool {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
ordered := orderedColumnsForTable(t, table)
|
||||||
|
cols := make(map[string]bool, len(ordered))
|
||||||
|
for _, col := range ordered {
|
||||||
|
cols[col.Name] = true
|
||||||
|
}
|
||||||
|
return cols
|
||||||
|
}
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/nws"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/nws"
|
||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openmeteo"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openmeteo"
|
||||||
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openweather"
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openweather"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/spc"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/ejr/feedkit/config"
|
"gitea.maximumdirect.net/ejr/feedkit/config"
|
||||||
fksource "gitea.maximumdirect.net/ejr/feedkit/sources"
|
fksource "gitea.maximumdirect.net/ejr/feedkit/sources"
|
||||||
@@ -15,18 +16,22 @@ type pollDriverRegistration struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var pollDriverRegistrations = []pollDriverRegistration{
|
var pollDriverRegistrations = []pollDriverRegistration{
|
||||||
{driver: "nws_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewObservationSource(cfg) }},
|
{driver: nws.DriverObservation, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewObservationSource(cfg) }},
|
||||||
{driver: "nws_alerts", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewAlertsSource(cfg) }},
|
{driver: nws.DriverAlerts, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewAlertsSource(cfg) }},
|
||||||
{driver: "nws_forecast_hourly", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewHourlyForecastSource(cfg) }},
|
{driver: nws.DriverForecastHourly, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewHourlyForecastSource(cfg) }},
|
||||||
{driver: "nws_forecast_narrative", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewNarrativeForecastSource(cfg) }},
|
{driver: nws.DriverForecastNarrative, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewNarrativeForecastSource(cfg) }},
|
||||||
{driver: "nws_forecast_discussion", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
|
{driver: nws.DriverForecastDiscussion, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
|
||||||
return nws.NewForecastDiscussionSource(cfg)
|
return nws.NewForecastDiscussionSource(cfg)
|
||||||
}},
|
}},
|
||||||
{driver: "openmeteo_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewObservationSource(cfg) }},
|
{driver: nws.DriverWeatherStories, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return nws.NewWeatherStoriesSource(cfg) }},
|
||||||
{driver: "openmeteo_forecast", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewForecastSource(cfg) }},
|
{driver: openmeteo.DriverObservation, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewObservationSource(cfg) }},
|
||||||
{driver: "openweather_observation", factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
|
{driver: openmeteo.DriverForecast, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) { return openmeteo.NewForecastSource(cfg) }},
|
||||||
|
{driver: openweather.DriverObservation, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
|
||||||
return openweather.NewObservationSource(cfg)
|
return openweather.NewObservationSource(cfg)
|
||||||
}},
|
}},
|
||||||
|
{driver: spc.DriverConvectiveOutlook, factory: func(cfg config.SourceConfig) (fksource.PollSource, error) {
|
||||||
|
return spc.NewConvectiveOutlookSource(cfg)
|
||||||
|
}},
|
||||||
}
|
}
|
||||||
|
|
||||||
// RegisterBuiltins registers the source drivers that ship with this binary.
|
// RegisterBuiltins registers the source drivers that ship with this binary.
|
||||||
|
|||||||
@@ -6,44 +6,29 @@ import (
|
|||||||
|
|
||||||
"gitea.maximumdirect.net/ejr/feedkit/config"
|
"gitea.maximumdirect.net/ejr/feedkit/config"
|
||||||
fksource "gitea.maximumdirect.net/ejr/feedkit/sources"
|
fksource "gitea.maximumdirect.net/ejr/feedkit/sources"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openweather"
|
||||||
|
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/spc"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestRegisterBuiltinsRegistersNWSHourlyForecastDriver(t *testing.T) {
|
func TestRegisterBuiltinsRegistersCurrentPollDrivers(t *testing.T) {
|
||||||
reg := fksource.NewRegistry()
|
reg := fksource.NewRegistry()
|
||||||
RegisterBuiltins(reg)
|
RegisterBuiltins(reg)
|
||||||
|
|
||||||
in, err := reg.BuildInput(sourceConfigForDriver("nws_forecast_hourly"))
|
if len(pollDriverRegistrations) == 0 {
|
||||||
if err != nil {
|
t.Fatalf("pollDriverRegistrations is empty")
|
||||||
t.Fatalf("BuildInput(nws_forecast_hourly) error = %v", err)
|
|
||||||
}
|
}
|
||||||
if _, ok := in.(fksource.PollSource); !ok {
|
|
||||||
t.Fatalf("BuildInput(nws_forecast_hourly) type = %T, want PollSource", in)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRegisterBuiltinsRegistersNWSNarrativeForecastDriver(t *testing.T) {
|
for _, tt := range pollDriverRegistrations {
|
||||||
reg := fksource.NewRegistry()
|
tt := tt
|
||||||
RegisterBuiltins(reg)
|
t.Run(tt.driver, func(t *testing.T) {
|
||||||
|
in, err := reg.BuildInput(sourceConfigForDriver(tt.driver))
|
||||||
in, err := reg.BuildInput(sourceConfigForDriver("nws_forecast_narrative"))
|
if err != nil {
|
||||||
if err != nil {
|
t.Fatalf("BuildInput(%q) error = %v", tt.driver, err)
|
||||||
t.Fatalf("BuildInput(nws_forecast_narrative) error = %v", err)
|
}
|
||||||
}
|
if _, ok := in.(fksource.PollSource); !ok {
|
||||||
if _, ok := in.(fksource.PollSource); !ok {
|
t.Fatalf("BuildInput(%q) type = %T, want PollSource", tt.driver, in)
|
||||||
t.Fatalf("BuildInput(nws_forecast_narrative) type = %T, want PollSource", in)
|
}
|
||||||
}
|
})
|
||||||
}
|
|
||||||
|
|
||||||
func TestRegisterBuiltinsRegistersNWSForecastDiscussionDriver(t *testing.T) {
|
|
||||||
reg := fksource.NewRegistry()
|
|
||||||
RegisterBuiltins(reg)
|
|
||||||
|
|
||||||
in, err := reg.BuildInput(sourceConfigForDriver("nws_forecast_discussion"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("BuildInput(nws_forecast_discussion) error = %v", err)
|
|
||||||
}
|
|
||||||
if _, ok := in.(fksource.PollSource); !ok {
|
|
||||||
t.Fatalf("BuildInput(nws_forecast_discussion) type = %T, want PollSource", in)
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -60,44 +45,23 @@ func TestRegisterBuiltinsDoesNotRegisterLegacyNWSForecastDriver(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRegisterBuiltinsRegistersAllCurrentDrivers(t *testing.T) {
|
|
||||||
reg := fksource.NewRegistry()
|
|
||||||
RegisterBuiltins(reg)
|
|
||||||
|
|
||||||
drivers := []string{
|
|
||||||
"nws_observation",
|
|
||||||
"nws_alerts",
|
|
||||||
"nws_forecast_hourly",
|
|
||||||
"nws_forecast_narrative",
|
|
||||||
"nws_forecast_discussion",
|
|
||||||
"openmeteo_observation",
|
|
||||||
"openmeteo_forecast",
|
|
||||||
"openweather_observation",
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, driver := range drivers {
|
|
||||||
in, err := reg.BuildInput(sourceConfigForDriver(driver))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("BuildInput(%s) error = %v", driver, err)
|
|
||||||
}
|
|
||||||
if _, ok := in.(fksource.PollSource); !ok {
|
|
||||||
t.Fatalf("BuildInput(%s) type = %T, want PollSource", driver, in)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func sourceConfigForDriver(driver string) config.SourceConfig {
|
func sourceConfigForDriver(driver string) config.SourceConfig {
|
||||||
url := "https://example.invalid"
|
url := "https://example.invalid"
|
||||||
if driver == "openweather_observation" {
|
if driver == openweather.DriverObservation {
|
||||||
url = "https://example.invalid?units=metric"
|
url = "https://example.invalid?units=metric"
|
||||||
}
|
}
|
||||||
|
params := map[string]any{
|
||||||
|
"url": url,
|
||||||
|
"user_agent": "test-agent",
|
||||||
|
}
|
||||||
|
if driver == spc.DriverConvectiveOutlook {
|
||||||
|
params["latitude"] = 38.6239
|
||||||
|
params["longitude"] = -90.3571
|
||||||
|
}
|
||||||
return config.SourceConfig{
|
return config.SourceConfig{
|
||||||
Name: "test-source",
|
Name: "test-source",
|
||||||
Driver: driver,
|
Driver: driver,
|
||||||
Mode: config.SourceModePoll,
|
Mode: config.SourceModePoll,
|
||||||
Params: map[string]any{
|
Params: params,
|
||||||
"url": url,
|
|
||||||
"user_agent": "test-agent",
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
32
internal/sources/docs_test.go
Normal file
32
internal/sources/docs_test.go
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
package sources
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestDocumentedRegisteredSourceDrivers(t *testing.T) {
|
||||||
|
docs := map[string]string{
|
||||||
|
"docs/config.md": readDoc(t, "../../docs/config.md"),
|
||||||
|
"docs/internal/sources.md": readDoc(t, "../../docs/internal/sources.md"),
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, reg := range pollDriverRegistrations {
|
||||||
|
for path, doc := range docs {
|
||||||
|
if !strings.Contains(doc, reg.driver) {
|
||||||
|
t.Fatalf("%s missing source driver %q", path, reg.driver)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func readDoc(t *testing.T, path string) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
raw, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(%s) error = %v", path, err)
|
||||||
|
}
|
||||||
|
return string(raw)
|
||||||
|
}
|
||||||
58
internal/sources/internal/httpconfig/config.go
Normal file
58
internal/sources/internal/httpconfig/config.go
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
package httpconfig
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/config"
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/transport"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Settings contains common HTTP client config for sources that fetch multiple documents.
|
||||||
|
type Settings struct {
|
||||||
|
Name string
|
||||||
|
UserAgent string
|
||||||
|
Timeout time.Duration
|
||||||
|
BodyLimitBytes int64
|
||||||
|
}
|
||||||
|
|
||||||
|
func Parse(driver string, cfg config.SourceConfig) (Settings, error) {
|
||||||
|
name := strings.TrimSpace(cfg.Name)
|
||||||
|
if name == "" {
|
||||||
|
return Settings{}, fmt.Errorf("%s: name is required", driver)
|
||||||
|
}
|
||||||
|
if cfg.Params == nil {
|
||||||
|
return Settings{}, fmt.Errorf("%s %q: params are required", driver, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
userAgent, ok := cfg.ParamString("user_agent", "userAgent")
|
||||||
|
if !ok {
|
||||||
|
return Settings{}, fmt.Errorf("%s %q: params.user_agent is required", driver, name)
|
||||||
|
}
|
||||||
|
|
||||||
|
timeout := transport.DefaultHTTPTimeout
|
||||||
|
if _, exists := cfg.Params["http_timeout"]; exists {
|
||||||
|
var ok bool
|
||||||
|
timeout, ok = cfg.ParamDuration("http_timeout")
|
||||||
|
if !ok || timeout <= 0 {
|
||||||
|
return Settings{}, fmt.Errorf("source %q: params.http_timeout must be a positive duration", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
bodyLimit := transport.DefaultHTTPResponseBodyLimitBytes
|
||||||
|
if _, exists := cfg.Params["http_response_body_limit_bytes"]; exists {
|
||||||
|
rawLimit, ok := cfg.ParamInt("http_response_body_limit_bytes")
|
||||||
|
if !ok || rawLimit <= 0 {
|
||||||
|
return Settings{}, fmt.Errorf("source %q: params.http_response_body_limit_bytes must be a positive integer", name)
|
||||||
|
}
|
||||||
|
bodyLimit = int64(rawLimit)
|
||||||
|
}
|
||||||
|
|
||||||
|
return Settings{
|
||||||
|
Name: name,
|
||||||
|
UserAgent: userAgent,
|
||||||
|
Timeout: timeout,
|
||||||
|
BodyLimitBytes: bodyLimit,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
115
internal/sources/internal/httpconfig/config_test.go
Normal file
115
internal/sources/internal/httpconfig/config_test.go
Normal file
@@ -0,0 +1,115 @@
|
|||||||
|
package httpconfig
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/config"
|
||||||
|
"gitea.maximumdirect.net/ejr/feedkit/transport"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseUsesRequiredValuesAndDefaults(t *testing.T) {
|
||||||
|
got, err := Parse("test_driver", config.SourceConfig{
|
||||||
|
Name: " test-source ",
|
||||||
|
Params: map[string]any{
|
||||||
|
"user_agent": "test-agent",
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.Name != "test-source" {
|
||||||
|
t.Fatalf("Name = %q, want test-source", got.Name)
|
||||||
|
}
|
||||||
|
if got.UserAgent != "test-agent" {
|
||||||
|
t.Fatalf("UserAgent = %q, want test-agent", got.UserAgent)
|
||||||
|
}
|
||||||
|
if got.Timeout != transport.DefaultHTTPTimeout {
|
||||||
|
t.Fatalf("Timeout = %s, want %s", got.Timeout, transport.DefaultHTTPTimeout)
|
||||||
|
}
|
||||||
|
if got.BodyLimitBytes != transport.DefaultHTTPResponseBodyLimitBytes {
|
||||||
|
t.Fatalf("BodyLimitBytes = %d, want %d", got.BodyLimitBytes, transport.DefaultHTTPResponseBodyLimitBytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseUsesAliasesAndOverrides(t *testing.T) {
|
||||||
|
got, err := Parse("test_driver", config.SourceConfig{
|
||||||
|
Name: "test-source",
|
||||||
|
Params: map[string]any{
|
||||||
|
"userAgent": "test-agent",
|
||||||
|
"http_timeout": "2s",
|
||||||
|
"http_response_body_limit_bytes": 2048,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v", err)
|
||||||
|
}
|
||||||
|
if got.UserAgent != "test-agent" {
|
||||||
|
t.Fatalf("UserAgent = %q, want test-agent", got.UserAgent)
|
||||||
|
}
|
||||||
|
if got.Timeout != 2*time.Second {
|
||||||
|
t.Fatalf("Timeout = %s, want 2s", got.Timeout)
|
||||||
|
}
|
||||||
|
if got.BodyLimitBytes != 2048 {
|
||||||
|
t.Fatalf("BodyLimitBytes = %d, want 2048", got.BodyLimitBytes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRejectsInvalidConfig(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
cfg config.SourceConfig
|
||||||
|
wantErr string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "missing name",
|
||||||
|
cfg: config.SourceConfig{Params: map[string]any{"user_agent": "test-agent"}},
|
||||||
|
wantErr: "test_driver: name is required",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing params",
|
||||||
|
cfg: config.SourceConfig{Name: "test-source"},
|
||||||
|
wantErr: `test_driver "test-source": params are required`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "missing user agent",
|
||||||
|
cfg: config.SourceConfig{Name: "test-source", Params: map[string]any{}},
|
||||||
|
wantErr: `test_driver "test-source": params.user_agent is required`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid timeout",
|
||||||
|
cfg: config.SourceConfig{
|
||||||
|
Name: "test-source",
|
||||||
|
Params: map[string]any{
|
||||||
|
"user_agent": "test-agent",
|
||||||
|
"http_timeout": "0s",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
wantErr: `source "test-source": params.http_timeout must be a positive duration`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid body limit",
|
||||||
|
cfg: config.SourceConfig{
|
||||||
|
Name: "test-source",
|
||||||
|
Params: map[string]any{
|
||||||
|
"user_agent": "test-agent",
|
||||||
|
"http_response_body_limit_bytes": 0,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
wantErr: `source "test-source": params.http_response_body_limit_bytes must be a positive integer`,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
_, err := Parse("test_driver", tt.cfg)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatalf("Parse() error = nil, want %q", tt.wantErr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), tt.wantErr) {
|
||||||
|
t.Fatalf("Parse() error = %q, want %q", err, tt.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user