38 Commits
v0.4.0 ... main

Author SHA1 Message Date
f464592c56 Updates to accommodate the new upstream version of feedkit (v0.5.0), which now supports both polling sources and streaming sources.
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-08 15:05:53 -06:00
123e8ff763 Moved the standards package out of internal/ so it can be imported by downstream consumers.
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-08 09:15:07 -06:00
5923592b53 Moved the weatherfeeder model out of internal/ so that downstream consumers can import it directly.
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-08 08:56:16 -06:00
c96a6bb78b Remove Go test files from .dockerignore; this will enable the CI system to run tests before building. 2026-02-08 08:55:35 -06:00
cff8c5f593 Merged SCHEMA.md into API.md and made some tweaks to ensure consistency with the underlying domain model.
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-07 19:36:50 -06:00
190420e908 Added a SCHEMA.md with espress documentation of the JSON output schemas emitted by weatherfeeder.
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-07 17:26:07 -06:00
9663fdfc43 Updated main.go to register the postgres and NATS sinks in addition to the stdout sink.
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-07 11:57:54 -06:00
ffd4dfa76d Updated feedkit dependency to v0.4.1.
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-02-07 11:37:53 -06:00
d858bc1f31 Updated Dockerfile and Woodpecker pipeline to use harbor.maximumdirect.net proxy cache for upstream images.
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
ci/woodpecker/push/build-image Pipeline was successful
2026-02-01 19:39:55 -06:00
2a88c3c5f3 Updated the Woodpecker pipeline to utilize caching.
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-02-01 19:04:22 -06:00
33c35b5f4a Update Dockerfile to streamline the weatherfeeder user creation.
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-02-01 18:17:12 -06:00
df3f42ef30 Updated the Dockerfile to reflect best practices; removed the dropreplace lines after updates to go.mod.
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-02-01 16:59:34 -06:00
e86d4a02e0 Added a go mod tidy step to the Dockerfile build.
All checks were successful
ci/woodpecker/manual/build-image Pipeline was successful
2026-02-01 10:15:50 -06:00
ba95371418 Added a second dropreplace step for ejr/feedkit.
Some checks failed
ci/woodpecker/manual/build-image Pipeline failed
2026-02-01 10:10:53 -06:00
5a92fc0401 Fixed a bug in the Dockerfile with respect to the location of the dropreplace for ejr/feedkit.
Some checks failed
ci/woodpecker/manual/build-image Pipeline failed
2026-02-01 10:01:02 -06:00
f522840fed Updated go.mod to require feedkit v0.4.0.
Some checks failed
ci/woodpecker/manual/build-image Pipeline failed
2026-02-01 09:57:10 -06:00
a46cbe17d1 Updated the Dockerfile to drop the ejr/feedkit replacement and to run go test before building.
Some checks failed
ci/woodpecker/manual/build-image Pipeline failed
2026-02-01 09:47:17 -06:00
7b8b3b98f2 Added a Dockerfile and completely refactored the Woodpecker pipeline.
Some checks failed
ci/woodpecker/manual/build-image Pipeline failed
2026-02-01 09:38:26 -06:00
4600ca053b woodpecker: Update pipeline to allow manual triggering. 2026-02-01 08:10:24 -06:00
88d2385576 woodpecker: allow manual pipeline triggering. 2026-02-01 07:33:01 -06:00
6cdbb29215 woodpecker: refactor pipeline to use kaniko. 2026-02-01 07:22:28 -06:00
62464f449e More woodpecker changes. 2026-01-31 13:56:12 -06:00
db8c66832f Removed privileged: true from the Woodpecker pipeline. 2026-01-31 13:53:27 -06:00
062b12c44f Add CI/CD pipeline configuration for building and publishing weatherfeeder. 2026-01-31 13:50:09 -06:00
da0231b20f Add day/night estimation for NWS observations based on solar elevation and update observation model. 2026-01-22 23:15:17 -06:00
c675045013 Updated documentation and added API.md to document the stable external wire format. 2026-01-17 17:39:18 -06:00
b4a67e208c Updated model and normalizers to map provider “feels like” fields into a single ApparentTemperatureC field. 2026-01-17 10:36:19 -06:00
c12cf91115 normalizers: implemented openmeteo forecast normalizer. 2026-01-17 10:16:50 -06:00
b47f1b2051 sources: added an OpenMeteo forecast source. 2026-01-17 08:00:23 -06:00
b8804d32d2 refactor(normalizers): deduplicate synthetic station ID generation
- Add common SynthStationID helpers for coordinate-based providers
- Use shared helper for Open-Meteo and OpenWeather station ID synthesis
- Require both lat/lon when generating synthetic IDs to avoid misleading defaults
- Remove unused Open-Meteo normalizer wrapper code

This reduces cross-provider duplication while keeping provider-specific
mapping logic explicit and readable.
2026-01-16 22:13:44 -06:00
00e811f8f7 normalizers/nws: add NWS alerts normalizer and canonical alert mapping
- Introduce AlertsNormalizer to convert Raw NWS Alerts (SchemaRawNWSAlertsV1)
  into canonical WeatherAlert runs (SchemaWeatherAlertV1)
- Add minimal NWS alerts response/types to support GeoJSON FeatureCollection parsing
- Map NWS alert properties (event, headline, severity, timing, area, references)
  into model.WeatherAlert with best-effort timestamp handling
- Establish clear AsOf / EffectiveAt policy for alert runs to support stable
  deduplication and snapshot semantics
- Register the new alerts normalizer alongside existing NWS observation and
  forecast normalizers
2026-01-16 21:40:20 -06:00
2eb2d4b90f feat(nws, normalizers): add NWS hourly forecast normalization and enforce canonical float rounding
- Implement full NWS hourly forecast normalizer (raw.nws.hourly.forecast.v1 → weather.forecast.v1)
- Add GeoJSON forecast types and helpers for NWS gridpoint hourly payloads
- Normalize temperatures, winds, humidity, PoP, and infer WMO condition codes from forecast text/icons
- Treat forecast IssuedAt as EffectiveAt for stable, dedupe-friendly event IDs

- Introduce project-wide float rounding at normalization finalization
  - Round all float values in canonical payloads to 2 decimal places
  - Apply consistently across pointers, slices, maps, and nested structs
  - Preserve opaque structs (e.g., time.Time) unchanged

- Add SchemaRawNWSHourlyForecastV1 and align schema matching/comments
- Clean up NWS helper organization and comments
- Update documentation to reflect numeric wire-format and normalization policies

This establishes a complete, deterministic hourly forecast pipeline for NWS
and improves JSON output stability across all canonical weather schemas.
2026-01-16 10:28:32 -06:00
0fcc536885 Updates in preparation for adding forecast sources. 2026-01-16 00:04:37 -06:00
e10ba804ca model: add explicit JSON tags and document canonical payload contract
Add lowerCamelCase JSON tags to canonical model types (observation, forecast,
alert) to stabilize the emitted wire format and make payload structure explicit
for downstream sinks.

Introduce internal/model/doc.go to document these structs as versioned,
schema-governed payloads and clarify compatibility expectations (additive
changes preferred; breaking changes require schema bumps).

No functional behavior changes; this formalizes the canonical output contract
ahead of additional sinks and consumers.
2026-01-15 22:39:37 -06:00
f13f43cf56 refactor(providers): centralize provider-specific parsing and invariants
- Introduce internal/providers/nws with shared timestamp parsing used by both
  NWS sources and normalizers
- Migrate NWS observation source + normalizer to use the shared provider helper
  for consistent RFC3339/RFC3339Nano handling
- Introduce internal/providers/openweather with a shared URL invariant helper
  enforcing units=metric
- Remove duplicated OpenWeather URL validation logic from the observation source
- Align provider layering: move provider “contract/quirk” logic out of
  normalizers and into internal/providers
- Update normalizer and standards documentation to clearly distinguish:
  provider helpers (internal/providers) vs canonical mapping logic
  (internal/normalizers)

This refactor reduces duplication between sources and normalizers, clarifies
layering boundaries, and establishes a scalable pattern for future forecast
and alert implementations.
2026-01-15 20:40:53 -06:00
a341aee5df normalizers: Updated error handling within the JSON helper function. 2026-01-15 20:17:46 -06:00
d8db58c004 sources: standardize Event.ID on Source:EffectiveAt; simplify raw event helper
- Adopt an opinionated Event.ID policy across sources:
  - use upstream-provided ID when available
  - otherwise derive a stable ID from Source:EffectiveAt (RFC3339Nano, UTC)
  - fall back to Source:EmittedAt when EffectiveAt is unavailable
- Add common/id helper to centralize ID selection logic and keep sources consistent
- Simplify common event construction by collapsing SingleRawEventAt/SingleRawEvent
  into a single explicit SingleRawEvent helper (emittedAt passed in)
- Update NWS/Open-Meteo/OpenWeather observation sources to:
  - compute EffectiveAt first
  - generate IDs via the shared helper
  - build envelopes via the unified SingleRawEvent helper
- Improve determinism and dedupe-friendliness without changing schemas or payloads
2026-01-15 19:38:15 -06:00
d9474b5a5b v0.x: add reusable HTTP source spine; fix routing; upstream HTTP transport helper
- fix dispatch route compilation so empty Kinds matches all (nil), not none
- introduce internal/sources/common/HTTPSource to centralize HTTP polling boilerplate:
  - standard cfg parsing (url + user_agent)
  - default HTTP client + Accept/User-Agent headers
  - consistent error wrapping
- refactor observation sources (nws/openmeteo/openweather) to use HTTPSource
- upstream generic HTTP fetch/limits/timeout helper from weatherfeeder to feedkit:
  - move internal/sources/common/http.go -> feedkit/transport/http.go
  - keep behavior: status checks, max-body limit, default timeout
2026-01-15 19:11:58 -06:00
59 changed files with 2991 additions and 699 deletions

7
.dockerignore Normal file
View File

@@ -0,0 +1,7 @@
.git
.gitignore
**/*.md
dist/
tmp/
.DS_Store

View File

@@ -0,0 +1,17 @@
when:
# Allow both normal runs (push) and UI-triggered runs (manual)
- event: [push, manual]
steps:
- name: build-and-push-image
image: harbor.maximumdirect.net/proxy-dockerhub/woodpeckerci/plugin-kaniko
settings:
registry: harbor.maximumdirect.net
repo: build/weatherfeeder
auto_tag: true
username:
from_secret: HARBOR_ROBOT_USER
password:
from_secret: HARBOR_ROBOT_TOKEN
cache: true
cache_repo: build-cache/weatherfeeder

319
API.md Normal file
View File

@@ -0,0 +1,319 @@
# weatherfeeder API (Wire Contract)
This document defines the stable, consumer-facing JSON contract emitted by weatherfeeder sinks.
weatherfeeder emits **events** encoded as JSON. Each event has:
- an **envelope** (metadata + schema identifier), and
- a **payload** whose shape is determined by `schema`.
Downstream consumers should:
1. parse the event envelope,
2. switch on `schema`, then
3. decode `payload` into the matching schema.
---
## Event envelope
All events are JSON objects with these fields:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | yes | Stable event identifier. Treat as opaque. |
| `schema` | string | yes | Schema identifier (e.g. `weather.observation.v1`). |
| `source` | string | yes | Provider/source identifier (stable within configuration). |
| `effectiveAt` | string (timestamp) | yes | RFC3339Nano timestamp indicating when this event is effective. |
| `payload` | object | yes | Schema-specific payload (see below). |
### Timestamp format
All timestamps are encoded as JSON strings using Gos `time.Time` JSON encoding (RFC3339Nano).
Examples:
- `"2026-01-17T14:27:00Z"`
- `"2026-01-17T08:27:00-06:00"`
---
## Canonical schemas
weatherfeeder emits three canonical domain schemas:
- `weather.observation.v1`
- `weather.forecast.v1`
- `weather.alert.v1`
Each payload is described below using the JSON field names as the contract.
---
## Shared Conventions
- Timestamps are JSON strings in RFC3339Nano format.
- Optional fields are omitted when unknown (`omitempty` behavior).
- Numeric measurements are normalized to metric units:
- `*C` = Celsius
- `*Kmh` = kilometers/hour
- `*Pa` = Pascals
- `*Meters` = meters
- `*Mm` = millimeters
- `*Percent` = percent (0-100)
- `conditionCode` is a WMO weather interpretation code (`int`).
- Unknown/unmappable is `-1`.
- Downstream consumers should treat unknown codes as “unknown conditions” rather than failing decoding.
- For readability and stability, weatherfeeder rounds floating-point values in canonical payloads to
**4 digits after the decimal** during normalization.
---
## Schema: `weather.observation.v1`
Payload type: `WeatherObservation`
A `WeatherObservation` represents a point-in-time observation for a station/location.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `stationId` | string | no | Provider station/location identifier |
| `stationName` | string | no | Human station name |
| `timestamp` | timestamp string | yes | Observation timestamp |
| `conditionCode` | int | yes | WMO code (`-1` unknown) |
| `conditionText` | string | no | Canonical short condition text |
| `isDay` | bool | no | Day/night hint |
| `providerRawDescription` | string | no | Provider-specific evidence text |
| `textDescription` | string | no | Legacy/transitional text description |
| `iconUrl` | string | no | Legacy/transitional icon URL |
| `temperatureC` | number | no | Celsius |
| `dewpointC` | number | no | Celsius |
| `windDirectionDegrees` | number | no | Degrees |
| `windSpeedKmh` | number | no | km/h |
| `windGustKmh` | number | no | km/h |
| `barometricPressurePa` | number | no | Pascals |
| `seaLevelPressurePa` | number | no | Pascals |
| `visibilityMeters` | number | no | Meters |
| `relativeHumidityPercent` | number | no | Percent |
| `apparentTemperatureC` | number | no | Celsius |
| `elevationMeters` | number | no | Meters |
| `rawMessage` | string | no | Provider raw message (for example METAR) |
| `presentWeather` | array | no | Provider-specific structured weather fragments |
| `cloudLayers` | array | no | Cloud layer details |
### Nested: `cloudLayers[]`
Each `cloudLayers[]` element:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `baseMeters` | number | no | Cloud base altitude in meters |
| `amount` | string | no | Provider string (e.g. FEW/SCT/BKN/OVC) |
### Nested: `presentWeather[]`
Each `presentWeather[]` element:
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `raw` | object | no | Provider-specific JSON object |
---
## Schema: `weather.forecast.v1`
Payload type: `WeatherForecastRun`
A `WeatherForecastRun` is a single issued forecast snapshot for a location and a specific product
(hourly / narrative / daily). The run contains an ordered list of forecast periods.
### `product` values
`product` is one of:
- `"hourly"`
- `"narrative"`
- `"daily"`
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `locationId` | string | no | Provider location identifier |
| `locationName` | string | no | Human name, if available |
| `issuedAt` | string (timestamp) | yes | When this run was generated/issued |
| `updatedAt` | string (timestamp) | no | Optional later update time |
| `product` | string | yes | One of `hourly`, `narrative`, `daily` |
| `latitude` | number | no | Degrees |
| `longitude` | number | no | Degrees |
| `elevationMeters` | number | no | meters |
| `periods` | array | yes | Chronological forecast periods |
### Nested: `periods[]` (`WeatherForecastPeriod`)
A `WeatherForecastPeriod` is valid for `[startTime, endTime)`.
| Field | Type | Required | Units / Notes |
|---|---:|:---:|---|
| `startTime` | string (timestamp) | yes | Period start |
| `endTime` | string (timestamp) | yes | Period end |
| `name` | string | no | Human label (often empty for hourly) |
| `isDay` | bool | no | Day/night hint |
| `conditionCode` | int | yes | WMO code (`-1` for unknown) |
| `conditionText` | string | no | Canonical short text |
| `providerRawDescription` | string | no | Provider-specific “evidence” text |
| `textDescription` | string | no | Human-facing short phrase |
| `detailedText` | string | no | Longer narrative |
| `iconUrl` | string | no | Legacy/transitional |
| `temperatureC` | number | no | °C |
| `temperatureCMin` | number | no | °C (aggregated products) |
| `temperatureCMax` | number | no | °C (aggregated products) |
| `dewpointC` | number | no | °C |
| `relativeHumidityPercent` | number | no | percent |
| `windDirectionDegrees` | number | no | degrees |
| `windSpeedKmh` | number | no | km/h |
| `windGustKmh` | number | no | km/h |
| `barometricPressurePa` | number | no | Pa |
| `visibilityMeters` | number | no | meters |
| `apparentTemperatureC` | number | no | °C |
| `cloudCoverPercent` | number | no | percent |
| `probabilityOfPrecipitationPercent` | number | no | percent |
| `precipitationAmountMm` | number | no | mm (liquid equivalent) |
| `snowfallDepthMm` | number | no | mm |
| `uvIndex` | number | no | unitless index |
---
## Schema: `weather.alert.v1`
Payload type: `WeatherAlertRun`
A `WeatherAlertRun` is a snapshot of *active* alerts for a location as-of a point in time.
A run may contain zero, one, or many alerts.
### Fields
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `locationId` | string | no | Provider location identifier |
| `locationName` | string | no | Human name, if available |
| `asOf` | string (timestamp) | yes | When the provider asserted this snapshot is current |
| `latitude` | number | no | Degrees |
| `longitude` | number | no | Degrees |
| `alerts` | array | yes | Active alerts (order provider-dependent) |
### Nested: `alerts[]` (`WeatherAlert`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | yes | Provider-stable identifier (often a URL/URI) |
| `event` | string | no | Classification/event label |
| `headline` | string | no | Alert headline |
| `severity` | string | no | Example: Extreme/Severe/Moderate/Minor/Unknown |
| `urgency` | string | no | Example: Immediate/Expected/Future/Past/Unknown |
| `certainty` | string | no | Example: Observed/Likely/Possible/Unlikely/Unknown |
| `status` | string | no | Example: Actual/Exercise/Test/System/Unknown |
| `messageType` | string | no | Example: Alert/Update/Cancel |
| `category` | string | no | Example: Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other |
| `response` | string | no | Example: Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None |
| `response` | string | no | e.g. Shelter/Evacuate/Prepare/... |
| `description` | string | no | Narrative |
| `instruction` | string | no | What to do |
| `sent` | string (timestamp) | no | Provider-dependent |
| `effective` | string (timestamp) | no | Provider-dependent |
| `onset` | string (timestamp) | no | Provider-dependent |
| `expires` | string (timestamp) | no | Provider-dependent |
| `areaDescription` | string | no | Often a provider string |
| `senderName` | string | no | Provenance |
| `references` | array | no | Related alert references |
### Nested: `references[]` (`AlertReference`)
| Field | Type | Required | Notes |
|---|---:|:---:|---|
| `id` | string | no | Provider reference ID/URI |
| `identifier` | string | no | Provider identifier string, if distinct |
| `sender` | string | no | Sender |
| `sent` | string (timestamp) | no | Timestamp |
---
## Compatibility rules
- Consumers **must** ignore unknown fields.
- Producers (weatherfeeder) prefer **additive changes** within a schema version.
- Renames/removals/semantic breaks require a **schema version bump** (`weather.*.v2`).
---
## Examples
### Observation event (`weather.observation.v1`)
```json
{
"id": "nws:KSTL:2026-01-17T14:00:00Z",
"schema": "weather.observation.v1",
"source": "nws_observation",
"effectiveAt": "2026-01-17T14:00:00Z",
"payload": {
"stationId": "KSTL",
"timestamp": "2026-01-17T14:00:00Z",
"conditionCode": 1,
"conditionText": "Mainly Sunny",
"temperatureC": 3.25,
"windSpeedKmh": 18.5
}
}
```
### Forecast event (`weather.forecast.v1`)
```json
{
"id": "openmeteo:38.63,-90.20:2026-01-17T13:00:00Z",
"schema": "weather.forecast.v1",
"source": "openmeteo_forecast",
"effectiveAt": "2026-01-17T13:00:00Z",
"payload": {
"locationName": "St. Louis, MO",
"issuedAt": "2026-01-17T13:00:00Z",
"product": "hourly",
"latitude": 38.63,
"longitude": -90.2,
"periods": [
{
"startTime": "2026-01-17T14:00:00Z",
"endTime": "2026-01-17T15:00:00Z",
"conditionCode": 2,
"conditionText": "Partly Cloudy",
"temperatureC": 3.5,
"probabilityOfPrecipitationPercent": 10
}
]
}
}
```
### Alert event (`weather.alert.v1`)
```json
{
"id": "nws:alerts:2026-01-17T14:10:00Z",
"schema": "weather.alert.v1",
"source": "nws_alerts",
"effectiveAt": "2026-01-17T14:10:00Z",
"payload": {
"asOf": "2026-01-17T14:05:00Z",
"alerts": [
{
"id": "https://api.weather.gov/alerts/abc123",
"event": "Winter Weather Advisory",
"headline": "Winter Weather Advisory issued January 17 at 8:05AM CST",
"severity": "Moderate",
"description": "Mixed precipitation expected...",
"expires": "2026-01-18T06:00:00Z"
}
]
}
}
```

78
Dockerfile Normal file
View File

@@ -0,0 +1,78 @@
# syntax=docker/dockerfile:1.6
ARG GO_VERSION=1.25
############################
# Build stage
############################
FROM harbor.maximumdirect.net/proxy-dockerhub/golang:${GO_VERSION}-bookworm AS build
WORKDIR /src
# Install baseline packages
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates tzdata git build-essential \
&& rm -rf /var/lib/apt/lists/*
# Cache dependencies first
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod \
go mod download
# Copy the rest of the source
COPY . .
# Ensure go.sum is complete after dropping the replace
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build
# Default to a static build (no CGO)
# If errors, can build with: --build-arg CGO_ENABLED=1
ARG CGO_ENABLED=0
ARG TARGETOS=linux
ARG TARGETARCH=amd64
ENV CGO_ENABLED=${CGO_ENABLED} \
GOOS=${TARGETOS} \
GOARCH=${TARGETARCH}
# Run tests before building the final binary
RUN --mount=type=cache,target=/go/pkg/mod \
--mount=type=cache,target=/root/.cache/go-build \
go test ./...
# Build the cmd entrypoint
RUN --mount=type=cache,target=/root/.cache/go-build \
go build \
-trimpath \
-ldflags="-s -w" \
-o /out/weatherfeeder \
./cmd/weatherfeeder
############################
# Runtime stage
############################
FROM harbor.maximumdirect.net/proxy-dockerhub/debian:bookworm-slim AS runtime
# Install runtime necessities
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates tzdata curl \
&& rm -rf /var/lib/apt/lists/*
# Define /weatherfeeder as the working directory
WORKDIR /weatherfeeder
# Create an unprivileged user
RUN useradd \
--uid 10001 \
--no-create-home \
--shell /usr/sbin/nologin \
weatherfeeder
# Copy the binary
COPY --chown=weatherfeeder:weatherfeeder --from=build /out/weatherfeeder /weatherfeeder/weatherfeeder
USER weatherfeeder
# The application expects config.yml in the same directory as the binary
ENTRYPOINT ["/weatherfeeder/weatherfeeder"]

View File

@@ -1,3 +1,34 @@
# weatherfeeder # weatherfeeder
A small daemon to poll weather observations, alerts, and forecasts from a variety of sources. weatherfeeder is a small daemon that polls weather observations, forecasts, and alerts from multiple upstream
providers, normalizes them into a provider-independent format, and emits them to a sink.
Today, the only implemented sink is `stdout`, which prints JSON-encoded events.
## What weatherfeeder emits
weatherfeeder emits **feed events** encoded as JSON. Each event includes a schema identifier and a payload.
Downstream consumers should key off the `schema` value and decode the `payload` accordingly.
Canonical domain schemas emitted after normalization:
- `weather.observation.v1``WeatherObservation`
- `weather.forecast.v1``WeatherForecastRun`
- `weather.alert.v1``WeatherAlertRun`
For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see:
- **API.md**
## Upstream providers (current MVP)
- NWS: observations, hourly forecasts, alerts
- Open-Meteo: observations, hourly forecasts
- OpenWeather: observations
## Versioning & compatibility
The JSON field names on canonical payload types are treated as part of the wire contract.
Additive changes are preferred. Renames/removals require a schema version bump.
See **API.md** for details.

View File

@@ -3,31 +3,31 @@ sources:
- name: NWSObservationKSTL - name: NWSObservationKSTL
kind: observation kind: observation
driver: nws_observation driver: nws_observation
every: 12m every: 10m
params: params:
url: "https://api.weather.gov/stations/KSTL/observations/latest" url: "https://api.weather.gov/stations/KSTL/observations/latest"
user_agent: "HomeOps (eric@maximumdirect.net)" user_agent: "HomeOps (eric@maximumdirect.net)"
- name: OpenMeteoObservation # - name: OpenMeteoObservation
kind: observation # kind: observation
driver: openmeteo_observation # driver: openmeteo_observation
every: 12m # every: 10m
params: # params:
url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m,wind_direction_10m,precipitation,surface_pressure,rain,showers,snowfall,cloud_cover,apparent_temperature,is_day,wind_gusts_10m,pressure_msl&forecast_days=1" # url: "https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&current=temperature_2m,relative_humidity_2m,weather_code,wind_speed_10m,wind_direction_10m,precipitation,surface_pressure,rain,showers,snowfall,cloud_cover,apparent_temperature,is_day,wind_gusts_10m,pressure_msl&forecast_days=1"
user_agent: "HomeOps (eric@maximumdirect.net)" # user_agent: "HomeOps (eric@maximumdirect.net)"
- name: OpenWeatherObservation # - name: OpenWeatherObservation
kind: observation # kind: observation
driver: openweather_observation # driver: openweather_observation
every: 12m # every: 10m
params: # params:
url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&appid=c954f2566cb7ccb56b43737b52e88fc6&units=metric" # url: "https://api.openweathermap.org/data/2.5/weather?lat=38.6239&lon=-90.3571&appid=c954f2566cb7ccb56b43737b52e88fc6&units=metric"
user_agent: "HomeOps (eric@maximumdirect.net)" # user_agent: "HomeOps (eric@maximumdirect.net)"
# - name: NWSObservationKSUS # - name: NWSObservationKSUS
# kind: observation # kind: observation
# driver: nws_observation # driver: nws_observation
# every: 18s # every: 10m
# params: # params:
# url: "https://api.weather.gov/stations/KSUS/observations/latest" # url: "https://api.weather.gov/stations/KSUS/observations/latest"
# user_agent: "HomeOps (eric@maximumdirect.net)" # user_agent: "HomeOps (eric@maximumdirect.net)"
@@ -35,24 +35,46 @@ sources:
# - name: NWSObservationKCPS # - name: NWSObservationKCPS
# kind: observation # kind: observation
# driver: nws_observation # driver: nws_observation
# every: 12m # every: 10m
# params: # params:
# url: "https://api.weather.gov/stations/KCPS/observations/latest" # url: "https://api.weather.gov/stations/KCPS/observations/latest"
# user_agent: "HomeOps (eric@maximumdirect.net)" # user_agent: "HomeOps (eric@maximumdirect.net)"
# - name: NWSAlertsSTL - name: NWSHourlyForecastSTL
# kind: alert kind: forecast
# driver: nws_alerts driver: nws_forecast
# every: 1m every: 45m
# params: params:
# url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=500" url: "https://api.weather.gov/gridpoints/LSX/90,74/forecast/hourly"
# user_agent: "HomeOps (eric@maximumdirect.net)" user_agent: "HomeOps (eric@maximumdirect.net)"
- name: OpenMeteoHourlyForecastSTL
kind: forecast
driver: openmeteo_forecast
every: 60m
params:
url: https://api.open-meteo.com/v1/forecast?latitude=38.6239&longitude=-90.3571&hourly=temperature_2m,relative_humidity_2m,dew_point_2m,apparent_temperature,precipitation_probability,precipitation,snowfall,weather_code,surface_pressure,wind_speed_10m,wind_direction_10m&forecast_days=3
user_agent: "HomeOps (eric@maximumdirect.net)"
- name: NWSAlertsSTL
kind: alert
driver: nws_alerts
every: 1m
params:
url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20"
user_agent: "HomeOps (eric@maximumdirect.net)"
sinks: sinks:
- name: stdout - name: stdout
driver: stdout driver: stdout
params: {} params: {}
- name: nats_weatherfeeder
driver: nats
params:
url: nats://nats:4222
exchange: weatherfeeder
# - name: logfile # - name: logfile
# driver: file # driver: file
# params: # params:
@@ -60,7 +82,10 @@ sinks:
routes: routes:
- sink: stdout - sink: stdout
kinds: ["observation"] kinds: ["observation", "forecast", "alert"]
- sink: nats_weatherfeeder
kinds: ["observation", "forecast", "alert"]
# - sink: logfile # - sink: logfile
# kinds: ["observation", "alert", "forecast"] # kinds: ["observation", "alert", "forecast"]

View File

@@ -40,16 +40,22 @@ func main() {
srcReg := fksources.NewRegistry() srcReg := fksources.NewRegistry()
wfsources.RegisterBuiltins(srcReg) wfsources.RegisterBuiltins(srcReg)
// Minimal sink set to compile: stdout only. // Compile stdout, Postgres, and NATS sinks for weatherfeeder. The former is useful for debugging and the latter are the main intended outputs.
sinkReg := fksinks.NewRegistry() sinkReg := fksinks.NewRegistry()
sinkReg.Register("stdout", func(cfg config.SinkConfig) (fksinks.Sink, error) { sinkReg.Register("stdout", func(cfg config.SinkConfig) (fksinks.Sink, error) {
return fksinks.NewStdoutSink(cfg.Name), nil return fksinks.NewStdoutSink(cfg.Name), nil
}) })
sinkReg.Register("postgres", func(cfg config.SinkConfig) (fksinks.Sink, error) {
return fksinks.NewPostgresSinkFromConfig(cfg)
})
sinkReg.Register("nats", func(cfg config.SinkConfig) (fksinks.Sink, error) {
return fksinks.NewNATSSinkFromConfig(cfg)
})
// --- Build sources into scheduler jobs --- // --- Build sources into scheduler jobs ---
var jobs []fkscheduler.Job var jobs []fkscheduler.Job
for i, sc := range cfg.Sources { for i, sc := range cfg.Sources {
src, err := srcReg.Build(sc) in, err := srcReg.BuildInput(sc) // may be polling or streaming
if err != nil { if err != nil {
log.Fatalf("build source failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err) log.Fatalf("build source failed (sources[%d] name=%q driver=%q): %v", i, sc.Name, sc.Driver, err)
} }
@@ -60,16 +66,25 @@ func main() {
if err != nil { if err != nil {
log.Fatalf("invalid kind in config (sources[%d] name=%q kind=%q): %v", i, sc.Name, sc.Kind, err) log.Fatalf("invalid kind in config (sources[%d] name=%q kind=%q): %v", i, sc.Name, sc.Kind, err)
} }
if src.Kind() != expectedKind { if in.Kind() != expectedKind {
log.Fatalf( log.Fatalf(
"source kind mismatch (sources[%d] name=%q driver=%q): config kind=%q but driver emits kind=%q", "source kind mismatch (sources[%d] name=%q driver=%q): config kind=%q but driver emits kind=%q",
i, sc.Name, sc.Driver, expectedKind, src.Kind(), i, sc.Name, sc.Driver, expectedKind, in.Kind(),
) )
} }
} }
// If this is a polling source, every is required.
if _, ok := in.(fksources.Source); ok && sc.Every.Duration <= 0 {
log.Fatalf(
"polling source missing/invalid interval (sources[%d] name=%q driver=%q): sources[].every must be > 0",
i, sc.Name, sc.Driver,
)
}
// For stream sources, Every is ignored; it is fine if omitted/zero.
jobs = append(jobs, fkscheduler.Job{ jobs = append(jobs, fkscheduler.Job{
Source: src, Source: in,
Every: sc.Every.Duration, Every: sc.Every.Duration,
}) })
} }

16
go.mod
View File

@@ -1,9 +1,15 @@
module gitea.maximumdirect.net/ejr/weatherfeeder module gitea.maximumdirect.net/ejr/weatherfeeder
go 1.22 go 1.25
require gitea.maximumdirect.net/ejr/feedkit v0.0.0 require gitea.maximumdirect.net/ejr/feedkit v0.5.0
require gopkg.in/yaml.v3 v3.0.1 // indirect require (
github.com/klauspost/compress v1.17.2 // indirect
replace gitea.maximumdirect.net/ejr/feedkit => ../feedkit github.com/nats-io/nats.go v1.34.0 // indirect
github.com/nats-io/nkeys v0.4.7 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/crypto v0.18.0 // indirect
golang.org/x/sys v0.16.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
)

14
go.sum
View File

@@ -1,3 +1,17 @@
gitea.maximumdirect.net/ejr/feedkit v0.5.0 h1:T4pRTo9Tj/o7TbZYUbp8UE7cQVLmIucUrYmD6G8E8ZQ=
gitea.maximumdirect.net/ejr/feedkit v0.5.0/go.mod h1:wYtA10GouvSe7L/8e1UEC+tqcp32HJofExIo1k+Wjls=
github.com/klauspost/compress v1.17.2 h1:RlWWUY/Dr4fL8qk9YG7DTZ7PDgME2V4csBXA8L/ixi4=
github.com/klauspost/compress v1.17.2/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
github.com/nats-io/nats.go v1.34.0 h1:fnxnPCNiwIG5w08rlMcEKTUw4AV/nKyGCOJE8TdhSPk=
github.com/nats-io/nats.go v1.34.0/go.mod h1:Ubdu4Nh9exXdSz0RVWRFBbRfrbSxOYd26oF0wkWclB8=
github.com/nats-io/nkeys v0.4.7 h1:RwNJbbIdYCoClSDNY7QVKZlyb/wfT6ugvFCiKy6vDvI=
github.com/nats-io/nkeys v0.4.7/go.mod h1:kqXRgRDPlGy7nGaEDMuYzmiJCIAAWDK0IMBtDmGD0nc=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.18.0 h1:PGVlW0xEltQnzFZ55hkuX5+KLyrMYhHld1YHO4AKcdc=
golang.org/x/crypto v0.18.0/go.mod h1:R0j02AL6hcrfOiy9T4ZYp/rcWeMxM3L6QYxlOuEG1mg=
golang.org/x/sys v0.16.0 h1:xWw16ngr6ZMtmxDyKyIgsE93KNKz5HKmMa3b8ALHidU=
golang.org/x/sys v0.16.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=

View File

@@ -1,23 +0,0 @@
package model
import "time"
// Placeholder for NWS alerts (GeoJSON feature properties are rich).
type WeatherAlert struct {
ID string
Event string
Headline string
Description string
Instruction string
Severity string
Urgency string
Certainty string
Sent *time.Time
Effective *time.Time
Expires *time.Time
Areas []string
}

View File

@@ -1,17 +0,0 @@
package model
import "time"
// WeatherForecast identity fields (as you described).
type WeatherForecast struct {
IssuedBy string // e.g. "NWS"
IssuedAt time.Time // when forecast product was issued
ForecastType string // e.g. "hourly", "daily"
ForecastStart time.Time // start of the applicable forecast period
// TODO: Youll likely want ForecastEnd too.
// TODO: Add meteorological fields you care about.
// Temperature, precip probability, wind, etc.
// Decide if you want a single "period" model or an array of periods.
}

View File

@@ -1,72 +0,0 @@
package model
import "time"
type WeatherObservation struct {
// Identity / metadata
StationID string
StationName string
Timestamp time.Time
// Canonical internal representation (provider-independent).
//
// ConditionCode should be populated by all sources. ConditionText should be the
// canonical human-readable string derived from the WMO code (not the provider's
// original wording).
//
// IsDay is optional; some providers supply a day/night flag (e.g., Open-Meteo),
// while others may not (e.g., NWS observations). When unknown, it can be nil.
ConditionCode WMOCode
ConditionText string
IsDay *bool
// Provider-specific “evidence” for troubleshooting mapping and drift.
//
// This is intentionally limited: it is not intended to be used downstream for
// business logic. Downstream logic should rely on ConditionCode / ConditionText.
ProviderRawDescription string
// Human-facing (legacy / transitional)
//
// TextDescription currently carries provider text in existing drivers.
// As we transition to WMO-based normalization, downstream presentation should
// switch to using ConditionText. After migration, this may be removed or repurposed.
TextDescription string
// Provider-specific (legacy / transitional)
//
// IconURL is not part of the canonical internal vocabulary. It's retained only
// because current sources populate it; it is not required for downstream systems.
IconURL string
// Core measurements (nullable)
TemperatureC *float64
DewpointC *float64
WindDirectionDegrees *float64
WindSpeedKmh *float64
WindGustKmh *float64
BarometricPressurePa *float64
SeaLevelPressurePa *float64
VisibilityMeters *float64
RelativeHumidityPercent *float64
WindChillC *float64
HeatIndexC *float64
ElevationMeters *float64
RawMessage string
PresentWeather []PresentWeather
CloudLayers []CloudLayer
}
type CloudLayer struct {
BaseMeters *float64
Amount string
}
type PresentWeather struct {
Raw map[string]any
}

View File

@@ -11,39 +11,9 @@
// - minimal abstractions (prefer straightforward functions) // - minimal abstractions (prefer straightforward functions)
// - easy to unit test // - easy to unit test
// //
// What belongs here // Numeric wire policy
// ----------------- // -------------------
// Put code in internal/normalizers/common when it is: // Canonical payloads are intended for sinks/serialization. To keep output stable and readable,
// // weatherfeeder rounds floating-point values in canonical payloads to a small, fixed precision
// - potentially reusable by more than one provider // at finalization time (see round.go).
// - provider-agnostic (no NWS/OpenWeather/Open-Meteo specific assumptions)
// - stable, small, and readable
//
// Typical examples:
// - unit conversion helpers (°F <-> °C, m/s <-> km/h, hPa <-> Pa, etc.)
// - json.RawMessage payload extraction helpers (with good error messages)
// - shared parsing helpers (timestamps, simple numeric coercions)
// - generic fallbacks (e.g., mapping a human text description into a coarse canonical code),
// so long as the logic truly applies across providers
//
// What does NOT belong here
// -------------------------
// Do NOT put the following in this package:
//
// - Normalizer implementations (types that satisfy feedkit/normalize.Normalizer)
// - provider-specific JSON structs or mapping logic (put those under
// internal/normalizers/<provider>/)
// - network or filesystem I/O (sources fetch; normalizers transform)
// - code that depends on event.Source naming, config fields, or driver-specific params
//
// Style and API guidelines
// ------------------------
// - Prefer small, single-purpose functions.
// - Keep function names explicit (avoid clever generic “DoThing” helpers).
// - Return typed errors with context (include schema/field names where helpful).
// - Keep dependencies minimal: standard library + weatherfeeder packages only.
// - Add unit tests for any non-trivial logic (especially parsing and fallbacks).
//
// Keeping this clean matters: common is shared by all providers, so complexity here
// multiplies across the project.
package common package common

View File

@@ -14,10 +14,13 @@ import (
// - ID/Kind/Source/EmittedAt are preserved by copying the input event. // - ID/Kind/Source/EmittedAt are preserved by copying the input event.
// - EffectiveAt is only overwritten when effectiveAt is non-zero. // - EffectiveAt is only overwritten when effectiveAt is non-zero.
// If effectiveAt is zero, any existing in.EffectiveAt is preserved. // If effectiveAt is zero, any existing in.EffectiveAt is preserved.
// - Payload floats are rounded to a stable wire-friendly precision (see round.go).
func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) { func Finalize(in event.Event, outSchema string, outPayload any, effectiveAt time.Time) (*event.Event, error) {
out := in out := in
out.Schema = outSchema out.Schema = outSchema
out.Payload = outPayload
// Enforce stable numeric presentation for sinks: round floats in the canonical payload.
out.Payload = RoundFloats(outPayload, DefaultFloatPrecision)
if !effectiveAt.IsZero() { if !effectiveAt.IsZero() {
t := effectiveAt.UTC() t := effectiveAt.UTC()

View File

@@ -0,0 +1,23 @@
// FILE: internal/normalizers/common/id.go
package common
import "fmt"
// SynthStationID formats a stable synthetic station identifier for providers that are
// coordinate-based rather than station-based.
//
// Example output:
//
// OPENMETEO(38.62700,-90.19940)
func SynthStationID(prefix string, lat, lon float64) string {
return fmt.Sprintf("%s(%.5f,%.5f)", prefix, lat, lon)
}
// SynthStationIDPtr is the pointer-friendly variant.
// If either coordinate is missing, it returns "" (unknown).
func SynthStationIDPtr(prefix string, lat, lon *float64) string {
if lat == nil || lon == nil {
return ""
}
return SynthStationID(prefix, *lat, *lon)
}

View File

@@ -42,6 +42,10 @@ func DecodeJSONPayload[T any](in event.Event) (T, error) {
// label should be short and specific, e.g. "openweather observation". // label should be short and specific, e.g. "openweather observation".
// outSchema should be the canonical schema constant. // outSchema should be the canonical schema constant.
// build should contain ONLY provider/domain mapping logic. // build should contain ONLY provider/domain mapping logic.
//
// Error policy:
// - NormalizeJSON wraps ALL failures with consistent context: "<label> normalize: <stage>: ..."
// - build() should return specific errors without repeating the label prefix.
func NormalizeJSON[T any, P any]( func NormalizeJSON[T any, P any](
in event.Event, in event.Event,
label string, label string,
@@ -55,8 +59,7 @@ func NormalizeJSON[T any, P any](
payload, effectiveAt, err := build(parsed) payload, effectiveAt, err := build(parsed)
if err != nil { if err != nil {
// build() should already include provider-specific context where appropriate. return nil, fmt.Errorf("%s normalize: build: %w", label, err)
return nil, err
} }
out, err := Finalize(in, outSchema, payload, effectiveAt) out, err := Finalize(in, outSchema, payload, effectiveAt)

View File

@@ -0,0 +1,212 @@
// FILE: ./internal/normalizers/common/round.go
package common
import (
"math"
"reflect"
)
// DefaultFloatPrecision is the project-wide wire-format policy for floating-point
// values in canonical payloads (weather.* schemas).
//
// Note: encoding/json will not necessarily print trailing zeros (e.g. 1.50 -> 1.5),
// but values will be *rounded* to this number of digits after the decimal point.
const DefaultFloatPrecision = 4
// RoundFloats returns a copy of v with all float32/float64 values (including pointers,
// slices, arrays, maps, and nested exported-struct fields) rounded to `decimals` digits
// after the decimal point.
//
// This is a best-effort helper meant for presentation stability. If reflection hits an
// unsupported/opaque type (e.g. structs with unexported fields like time.Time), that
// subtree is left unchanged.
func RoundFloats(v any, decimals int) any {
if v == nil || decimals < 0 {
return v
}
defer func() {
// Never let presentation formatting crash the pipeline.
_ = recover()
}()
rv := reflect.ValueOf(v)
out := roundValue(rv, decimals)
if !out.IsValid() {
return v
}
return out.Interface()
}
func roundValue(v reflect.Value, decimals int) reflect.Value {
if !v.IsValid() {
return v
}
// Unwrap interfaces.
if v.Kind() == reflect.Interface {
if v.IsNil() {
return v
}
elem := roundValue(v.Elem(), decimals)
// Re-wrap in the same interface type.
out := reflect.New(v.Type()).Elem()
if elem.IsValid() && elem.Type().AssignableTo(v.Type()) {
out.Set(elem)
return out
}
if elem.IsValid() && elem.Type().ConvertibleTo(v.Type()) {
out.Set(elem.Convert(v.Type()))
return out
}
// If we can't sensibly re-wrap, just keep the original.
return v
}
// Copy pointers (and round their targets).
if v.Kind() == reflect.Pointer {
if v.IsNil() {
return v
}
// If the pointed-to type is an opaque struct (e.g. time.Time), keep as-is.
if v.Elem().Kind() == reflect.Struct && isOpaqueStruct(v.Elem().Type()) {
return v
}
elem := roundValue(v.Elem(), decimals)
p := reflect.New(v.Type().Elem())
if elem.IsValid() && elem.Type().AssignableTo(v.Type().Elem()) {
p.Elem().Set(elem)
} else if elem.IsValid() && elem.Type().ConvertibleTo(v.Type().Elem()) {
p.Elem().Set(elem.Convert(v.Type().Elem()))
} else {
p.Elem().Set(v.Elem())
}
return p
}
switch v.Kind() {
case reflect.Float32, reflect.Float64:
f := v.Convert(reflect.TypeOf(float64(0))).Float()
r := roundFloat64(f, decimals)
return reflect.ValueOf(r).Convert(v.Type())
case reflect.Struct:
// Avoid reconstructing opaque structs (time.Time has unexported fields).
if isOpaqueStruct(v.Type()) {
return v
}
out := reflect.New(v.Type()).Elem()
out.Set(v) // start from a copy, then replace rounded fields
t := v.Type()
for i := 0; i < v.NumField(); i++ {
sf := t.Field(i)
// Only exported fields are safely settable across packages.
if sf.PkgPath != "" {
continue
}
fv := v.Field(i)
rf := roundValue(fv, decimals)
of := out.Field(i)
if !of.CanSet() {
continue
}
if rf.IsValid() && rf.Type().AssignableTo(of.Type()) {
of.Set(rf)
} else if rf.IsValid() && rf.Type().ConvertibleTo(of.Type()) {
of.Set(rf.Convert(of.Type()))
}
}
return out
case reflect.Slice:
if v.IsNil() {
return v
}
out := reflect.MakeSlice(v.Type(), v.Len(), v.Len())
for i := 0; i < v.Len(); i++ {
ev := v.Index(i)
re := roundValue(ev, decimals)
if re.IsValid() && re.Type().AssignableTo(out.Index(i).Type()) {
out.Index(i).Set(re)
} else if re.IsValid() && re.Type().ConvertibleTo(out.Index(i).Type()) {
out.Index(i).Set(re.Convert(out.Index(i).Type()))
} else {
out.Index(i).Set(ev)
}
}
return out
case reflect.Array:
out := reflect.New(v.Type()).Elem()
out.Set(v)
for i := 0; i < v.Len(); i++ {
ev := v.Index(i)
re := roundValue(ev, decimals)
if re.IsValid() && re.Type().AssignableTo(out.Index(i).Type()) {
out.Index(i).Set(re)
} else if re.IsValid() && re.Type().ConvertibleTo(out.Index(i).Type()) {
out.Index(i).Set(re.Convert(out.Index(i).Type()))
} else {
out.Index(i).Set(ev)
}
}
return out
case reflect.Map:
if v.IsNil() {
return v
}
out := reflect.MakeMapWithSize(v.Type(), v.Len())
iter := v.MapRange()
for iter.Next() {
k := iter.Key()
mv := iter.Value()
rv := roundValue(mv, decimals)
if rv.IsValid() && rv.Type().AssignableTo(v.Type().Elem()) {
out.SetMapIndex(k, rv)
} else if rv.IsValid() && rv.Type().ConvertibleTo(v.Type().Elem()) {
out.SetMapIndex(k, rv.Convert(v.Type().Elem()))
} else {
out.SetMapIndex(k, mv)
}
}
return out
default:
// ints, strings, bools, time.Time (handled as opaque), etc.
return v
}
}
func roundFloat64(f float64, decimals int) float64 {
if decimals <= 0 {
return math.Round(f)
}
pow := math.Pow10(decimals)
return math.Round(f*pow) / pow
}
// isOpaqueStruct returns true for structs that are unsafe/unhelpful to reconstruct via reflection.
// Any struct containing unexported fields (e.g. time.Time) is treated as opaque.
func isOpaqueStruct(t reflect.Type) bool {
if t.Kind() != reflect.Struct {
return false
}
for i := 0; i < t.NumField(); i++ {
if t.Field(i).PkgPath != "" {
return true
}
}
return false
}

View File

@@ -4,7 +4,7 @@ package common
import ( import (
"strings" "strings"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/model" "gitea.maximumdirect.net/ejr/weatherfeeder/model"
) )
// WMOFromTextDescription is a cross-provider fallback that tries to infer a WMO code // WMOFromTextDescription is a cross-provider fallback that tries to infer a WMO code

View File

@@ -34,14 +34,16 @@
// as they do not define additional Normalizer types. // as they do not define additional Normalizer types.
// //
// 2. Provider-level shared helpers live under the provider directory: // 2. Provider-level shared helpers live under the provider directory:
// internal/normalizers/<provider>/ // internal/providers/<provider>/
//
// Use this for provider-specific quirks that should be shared by BOTH sources
// and normalizers (time parsing, URL/unit invariants, ID normalization, etc.).
// Keep these helpers pure (no I/O) and easy to test.
// //
// You may use multiple helper files (recommended) when it improves clarity: // You may use multiple helper files (recommended) when it improves clarity:
// - types.go (provider JSON structs) // - types.go (provider JSON structs)
// - common.go (provider-shared helpers) // - common.go (provider-shared helpers)
// - mapping.go (provider mapping logic) // - mapping.go (provider mapping logic)
// Use common.go only when you truly have “shared across multiple normalizers
// within this provider” helpers.
// //
// 3. Cross-provider helpers live in: // 3. Cross-provider helpers live in:
// internal/normalizers/common/ // internal/normalizers/common/

View File

@@ -0,0 +1,238 @@
// FILE: internal/normalizers/nws/alerts.go
package nws
import (
"context"
"encoding/json"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// AlertsNormalizer converts:
//
// standards.SchemaRawNWSAlertsV1 -> standards.SchemaWeatherAlertV1
//
// It interprets NWS /alerts FeatureCollection payloads (GeoJSON-ish) and maps them into
// the canonical model.WeatherAlertRun representation.
//
// Caveats / policy:
// 1. Run.AsOf prefers the collection-level "updated" timestamp. If missing/unparseable,
// we fall back to the latest per-alert timestamp, and then to the input events
// EffectiveAt/EmittedAt.
// 2. Alert timing fields are best-effort parsed; invalid timestamps do not fail the
// entire normalization (they are left nil).
// 3. Some fields are intentionally passed through as strings (severity/urgency/etc.)
// since canonical vocabularies may evolve later.
type AlertsNormalizer struct{}
func (AlertsNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawNWSAlertsV1
}
func (AlertsNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
// If we can't derive AsOf from the payload, fall back to the existing event envelope.
fallbackAsOf := in.EmittedAt.UTC()
if in.EffectiveAt != nil && !in.EffectiveAt.IsZero() {
fallbackAsOf = in.EffectiveAt.UTC()
}
return normcommon.NormalizeJSON(
in,
"nws alerts",
standards.SchemaWeatherAlertV1,
func(parsed nwsAlertsResponse) (model.WeatherAlertRun, time.Time, error) {
return buildAlerts(parsed, fallbackAsOf)
},
)
}
// buildAlerts contains the domain mapping logic (provider -> canonical model).
func buildAlerts(parsed nwsAlertsResponse, fallbackAsOf time.Time) (model.WeatherAlertRun, time.Time, error) {
// 1) Determine AsOf (required by canonical model; also used as EffectiveAt).
asOf := nwscommon.ParseTimeBestEffort(parsed.Updated)
if asOf.IsZero() {
asOf = latestAlertTimestamp(parsed.Features)
}
if asOf.IsZero() {
asOf = fallbackAsOf.UTC()
}
run := model.WeatherAlertRun{
LocationID: "",
LocationName: strings.TrimSpace(parsed.Title),
AsOf: asOf,
Latitude: nil,
Longitude: nil,
Alerts: nil,
}
// 2) Map each feature into a canonical WeatherAlert.
alerts := make([]model.WeatherAlert, 0, len(parsed.Features))
for i, f := range parsed.Features {
p := f.Properties
id := firstNonEmpty(strings.TrimSpace(f.ID), strings.TrimSpace(p.ID), strings.TrimSpace(p.Identifier))
if id == "" {
// NWS usually supplies an ID, but be defensive. Prefer a stable-ish synth ID.
// Include the run as-of time to reduce collisions across snapshots.
id = fmt.Sprintf("nws:alert:%s:%d", asOf.UTC().Format(time.RFC3339Nano), i)
}
sent := nwscommon.ParseTimePtr(p.Sent)
effective := nwscommon.ParseTimePtr(p.Effective)
onset := nwscommon.ParseTimePtr(p.Onset)
// Expires: prefer "expires"; fall back to "ends" if present.
expires := nwscommon.ParseTimePtr(p.Expires)
if expires == nil {
expires = nwscommon.ParseTimePtr(p.Ends)
}
refs := parseNWSAlertReferences(p.References)
alert := model.WeatherAlert{
ID: id,
Event: strings.TrimSpace(p.Event),
Headline: strings.TrimSpace(p.Headline),
Severity: strings.TrimSpace(p.Severity),
Urgency: strings.TrimSpace(p.Urgency),
Certainty: strings.TrimSpace(p.Certainty),
Status: strings.TrimSpace(p.Status),
MessageType: strings.TrimSpace(p.MessageType),
Category: strings.TrimSpace(p.Category),
Response: strings.TrimSpace(p.Response),
Description: strings.TrimSpace(p.Description),
Instruction: strings.TrimSpace(p.Instruction),
Sent: sent,
Effective: effective,
Onset: onset,
Expires: expires,
AreaDescription: strings.TrimSpace(p.AreaDesc),
SenderName: firstNonEmpty(strings.TrimSpace(p.SenderName), strings.TrimSpace(p.Sender)),
References: refs,
}
// Headline fallback (NWS commonly provides it, but not guaranteed).
if alert.Headline == "" {
alert.Headline = alert.Event
}
alerts = append(alerts, alert)
}
run.Alerts = alerts
// EffectiveAt policy for alerts: treat AsOf as the effective time (dedupe-friendly).
return run, asOf, nil
}
// latestAlertTimestamp scans alert features for the most recent timestamp.
// It prefers Sent/Effective, and falls back to Expires/Ends when needed.
func latestAlertTimestamp(features []nwsAlertFeature) time.Time {
var latest time.Time
for _, f := range features {
p := f.Properties
candidates := []string{
p.Sent,
p.Effective,
p.Expires,
p.Ends,
p.Onset,
}
for _, s := range candidates {
t := nwscommon.ParseTimeBestEffort(s)
if t.IsZero() {
continue
}
if latest.IsZero() || t.After(latest) {
latest = t
}
}
}
return latest
}
func firstNonEmpty(vals ...string) string {
for _, v := range vals {
if strings.TrimSpace(v) != "" {
return strings.TrimSpace(v)
}
}
return ""
}
// parseNWSAlertReferences tries to interpret the NWS "references" field, which may
// vary by endpoint/version. We accept the common object-array form and a few
// degraded shapes (string array / single string).
func parseNWSAlertReferences(raw json.RawMessage) []model.AlertReference {
if len(raw) == 0 {
return nil
}
// Most common: array of objects.
var objs []nwsAlertReference
if err := json.Unmarshal(raw, &objs); err == nil && len(objs) > 0 {
out := make([]model.AlertReference, 0, len(objs))
for _, r := range objs {
ref := model.AlertReference{
ID: strings.TrimSpace(r.ID),
Identifier: strings.TrimSpace(r.Identifier),
Sender: strings.TrimSpace(r.Sender),
Sent: nwscommon.ParseTimePtr(r.Sent),
}
// If only Identifier is present, preserve it as ID too (useful downstream).
if ref.ID == "" && ref.Identifier != "" {
ref.ID = ref.Identifier
}
out = append(out, ref)
}
return out
}
// Sometimes: array of strings.
var strs []string
if err := json.Unmarshal(raw, &strs); err == nil && len(strs) > 0 {
out := make([]model.AlertReference, 0, len(strs))
for _, s := range strs {
id := strings.TrimSpace(s)
if id == "" {
continue
}
out = append(out, model.AlertReference{ID: id})
}
if len(out) > 0 {
return out
}
}
// Rare: single string.
var single string
if err := json.Unmarshal(raw, &single); err == nil {
id := strings.TrimSpace(single)
if id != "" {
return []model.AlertReference{{ID: id}}
}
}
return nil
}

View File

@@ -0,0 +1,74 @@
// FILE: internal/normalizers/nws/day_night.go
package nws
import (
"math"
"time"
)
const (
degToRad = math.Pi / 180.0
radToDeg = 180.0 / math.Pi
)
func observationLatLon(coords []float64) (lat *float64, lon *float64) {
if len(coords) < 2 {
return nil, nil
}
latVal := coords[1]
lonVal := coords[0]
return &latVal, &lonVal
}
// isDayFromLatLonTime estimates day/night from solar elevation.
// Uses a refraction-adjusted cutoff (-0.833 degrees).
func isDayFromLatLonTime(lat, lon float64, ts time.Time) *bool {
if ts.IsZero() || math.IsNaN(lat) || math.IsNaN(lon) || math.IsInf(lat, 0) || math.IsInf(lon, 0) {
return nil
}
if lat < -90.0 || lat > 90.0 || lon < -180.0 || lon > 180.0 {
return nil
}
t := ts.UTC()
day := float64(t.YearDay())
hour := float64(t.Hour())
min := float64(t.Minute())
sec := float64(t.Second()) + float64(t.Nanosecond())/1e9
utcHours := hour + min/60.0 + sec/3600.0
gamma := 2.0 * math.Pi / 365.0 * (day - 1.0 + (utcHours-12.0)/24.0)
eqtime := 229.18 * (0.000075 + 0.001868*math.Cos(gamma) - 0.032077*math.Sin(gamma) -
0.014615*math.Cos(2.0*gamma) - 0.040849*math.Sin(2.0*gamma))
decl := 0.006918 - 0.399912*math.Cos(gamma) + 0.070257*math.Sin(gamma) -
0.006758*math.Cos(2.0*gamma) + 0.000907*math.Sin(2.0*gamma) -
0.002697*math.Cos(3.0*gamma) + 0.00148*math.Sin(3.0*gamma)
timeOffset := eqtime + 4.0*lon
trueSolarMinutes := hour*60.0 + min + sec/60.0 + timeOffset
for trueSolarMinutes < 0 {
trueSolarMinutes += 1440.0
}
for trueSolarMinutes >= 1440.0 {
trueSolarMinutes -= 1440.0
}
hourAngleDeg := trueSolarMinutes/4.0 - 180.0
ha := hourAngleDeg * degToRad
latRad := lat * degToRad
cosZenith := math.Sin(latRad)*math.Sin(decl) + math.Cos(latRad)*math.Cos(decl)*math.Cos(ha)
if cosZenith > 1.0 {
cosZenith = 1.0
} else if cosZenith < -1.0 {
cosZenith = -1.0
}
zenith := math.Acos(cosZenith)
elevation := 90.0 - zenith*radToDeg
isDay := elevation > -0.833
return &isDay
}

View File

@@ -0,0 +1,159 @@
// FILE: internal/normalizers/nws/forecast.go
package nws
import (
"context"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// ForecastNormalizer converts:
//
// standards.SchemaRawNWSHourlyForecastV1 -> standards.SchemaWeatherForecastV1
//
// It interprets NWS GeoJSON gridpoint *hourly* forecast responses and maps them into
// the canonical model.WeatherForecastRun representation.
//
// Caveats / policy:
// 1. NWS forecast periods do not include METAR presentWeather phenomena, so ConditionCode
// is inferred from period.shortForecast (with a conservative icon-based fallback).
// 2. Temperature is converted to °C when NWS supplies °F.
// 3. WindSpeed is parsed from strings like "9 mph" / "10 to 15 mph" and converted to km/h.
type ForecastNormalizer struct{}
func (ForecastNormalizer) Match(e event.Event) bool {
s := strings.TrimSpace(e.Schema)
return s == standards.SchemaRawNWSHourlyForecastV1
}
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
return normcommon.NormalizeJSON(
in,
"nws hourly forecast",
standards.SchemaWeatherForecastV1,
buildForecast,
)
}
// buildForecast contains the domain mapping logic (provider -> canonical model).
func buildForecast(parsed nwsForecastResponse) (model.WeatherForecastRun, time.Time, error) {
// IssuedAt is required by the canonical model.
issuedStr := strings.TrimSpace(parsed.Properties.GeneratedAt)
if issuedStr == "" {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("missing properties.generatedAt")
}
issuedAt, err := nwscommon.ParseTime(issuedStr)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("invalid properties.generatedAt %q: %w", issuedStr, err)
}
issuedAt = issuedAt.UTC()
// UpdatedAt is optional.
var updatedAt *time.Time
if s := strings.TrimSpace(parsed.Properties.UpdateTime); s != "" {
if t, err := nwscommon.ParseTime(s); err == nil {
tt := t.UTC()
updatedAt = &tt
}
}
// Best-effort location centroid from the GeoJSON polygon (optional).
lat, lon := centroidLatLon(parsed.Geometry.Coordinates)
// Schema is explicitly hourly, so product is not a heuristic.
run := model.WeatherForecastRun{
LocationID: "",
LocationName: "",
IssuedAt: issuedAt,
UpdatedAt: updatedAt,
Product: model.ForecastProductHourly,
Latitude: lat,
Longitude: lon,
ElevationMeters: parsed.Properties.Elevation.Value,
Periods: nil,
}
periods := make([]model.WeatherForecastPeriod, 0, len(parsed.Properties.Periods))
for i, p := range parsed.Properties.Periods {
startStr := strings.TrimSpace(p.StartTime)
endStr := strings.TrimSpace(p.EndTime)
if startStr == "" || endStr == "" {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("periods[%d]: missing startTime/endTime", i)
}
start, err := nwscommon.ParseTime(startStr)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("periods[%d].startTime invalid %q: %w", i, startStr, err)
}
end, err := nwscommon.ParseTime(endStr)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("periods[%d].endTime invalid %q: %w", i, endStr, err)
}
start = start.UTC()
end = end.UTC()
// NWS hourly supplies isDaytime; make it a pointer to match the canonical model.
var isDay *bool
if p.IsDaytime != nil {
b := *p.IsDaytime
isDay = &b
}
tempC := tempCFromNWS(p.Temperature, p.TemperatureUnit)
// Infer WMO from shortForecast (and fall back to icon token).
providerDesc := strings.TrimSpace(p.ShortForecast)
wmo := wmoFromNWSForecast(providerDesc, p.Icon, tempC)
canonicalText := standards.WMOText(wmo, isDay)
period := model.WeatherForecastPeriod{
StartTime: start,
EndTime: end,
Name: strings.TrimSpace(p.Name),
IsDay: isDay,
ConditionCode: wmo,
ConditionText: canonicalText,
ProviderRawDescription: providerDesc,
// For forecasts, keep provider text as the human-facing description.
TextDescription: strings.TrimSpace(p.ShortForecast),
DetailedText: strings.TrimSpace(p.DetailedForecast),
IconURL: strings.TrimSpace(p.Icon),
TemperatureC: tempC,
DewpointC: p.Dewpoint.Value,
RelativeHumidityPercent: p.RelativeHumidity.Value,
WindDirectionDegrees: parseNWSWindDirectionDegrees(p.WindDirection),
WindSpeedKmh: parseNWSWindSpeedKmh(p.WindSpeed),
ProbabilityOfPrecipitationPercent: p.ProbabilityOfPrecipitation.Value,
}
periods = append(periods, period)
}
run.Periods = periods
// EffectiveAt policy for forecasts: treat IssuedAt as the effective time (dedupe-friendly).
return run, issuedAt, nil
}

View File

@@ -0,0 +1,235 @@
// FILE: internal/normalizers/nws/helpers.go
package nws
import (
"strconv"
"strings"
"unicode"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
)
// centroidLatLon returns a best-effort centroid (lat, lon) from a GeoJSON polygon.
// If geometry is missing or malformed, returns (nil, nil).
func centroidLatLon(coords [][][]float64) (lat *float64, lon *float64) {
if len(coords) == 0 || len(coords[0]) == 0 {
return nil, nil
}
var sumLon, sumLat float64
var n float64
for _, pt := range coords[0] {
if len(pt) < 2 {
continue
}
sumLon += pt[0]
sumLat += pt[1]
n++
}
if n == 0 {
return nil, nil
}
avgLon := sumLon / n
avgLat := sumLat / n
return &avgLat, &avgLon
}
func tempCFromNWS(v *float64, unit string) *float64 {
if v == nil {
return nil
}
u := strings.ToUpper(strings.TrimSpace(unit))
switch u {
case "F":
c := normcommon.TempCFromF(*v)
return &c
case "C":
c := *v
return &c
default:
// Unknown unit; be conservative.
return nil
}
}
// wmoFromNWSForecast infers a canonical WMO code for a forecast period.
//
// Strategy:
// 1. Try to infer from shortForecast using the cross-provider fallback.
// 2. Special-case mixed rain+snow using temperature when available (since our WMO table
// does not include a “mixed precip” code).
// 3. Fall back to an icon token (e.g., "rain", "snow", "ovc", "bkn", "sct", ...).
func wmoFromNWSForecast(shortForecast, iconURL string, tempC *float64) model.WMOCode {
sf := strings.TrimSpace(shortForecast)
s := strings.ToLower(sf)
// Mixed precip heuristic: choose rain vs snow based on temperature.
if strings.Contains(s, "rain") && strings.Contains(s, "snow") {
if tempC != nil && *tempC <= 0.0 {
return 73 // Snow
}
return 63 // Rain
}
if code := normcommon.WMOFromTextDescription(sf); code != model.WMOUnknown {
return code
}
// Icon fallback: token is usually the last path segment (before any comma/query).
if token := nwsIconToken(iconURL); token != "" {
// Try the general text fallback first (works for "rain", "snow", etc.).
if code := normcommon.WMOFromTextDescription(token); code != model.WMOUnknown {
return code
}
// Sky-condition icon tokens are common; map conservatively.
switch token {
case "ovc", "bkn", "cloudy", "ovcast":
return 3
case "sct", "bkn-sct":
return 2
case "few":
return 1
case "skc", "clr", "clear":
return 0
}
}
return model.WMOUnknown
}
func nwsIconToken(iconURL string) string {
u := strings.TrimSpace(iconURL)
if u == "" {
return ""
}
// Drop query string.
base := strings.SplitN(u, "?", 2)[0]
// Take last path segment.
parts := strings.Split(base, "/")
if len(parts) == 0 {
return ""
}
last := parts[len(parts)-1]
if last == "" && len(parts) > 1 {
last = parts[len(parts)-2]
}
// Some icons look like "rain,30" or "snow,20".
last = strings.SplitN(last, ",", 2)[0]
last = strings.ToLower(strings.TrimSpace(last))
return last
}
// parseNWSWindSpeedKmh parses NWS wind speed strings like:
// - "9 mph"
// - "10 to 15 mph"
//
// and converts to km/h.
//
// Policy: if a range is present, we use the midpoint (best effort).
func parseNWSWindSpeedKmh(s string) *float64 {
raw := strings.ToLower(strings.TrimSpace(s))
if raw == "" {
return nil
}
nums := extractFloats(raw)
if len(nums) == 0 {
return nil
}
val := nums[0]
if len(nums) >= 2 && (strings.Contains(raw, " to ") || strings.Contains(raw, "-")) {
val = (nums[0] + nums[1]) / 2.0
}
switch {
case strings.Contains(raw, "mph"):
k := normcommon.SpeedKmhFromMph(val)
return &k
case strings.Contains(raw, "km/h") || strings.Contains(raw, "kph"):
k := val
return &k
case strings.Contains(raw, "kt") || strings.Contains(raw, "kts") || strings.Contains(raw, "knot"):
// 1 knot = 1.852 km/h
k := val * 1.852
return &k
default:
// Unknown unit; be conservative.
return nil
}
}
// parseNWSWindDirectionDegrees maps compass directions to degrees.
// Returns nil if direction is empty/unknown.
func parseNWSWindDirectionDegrees(dir string) *float64 {
d := strings.ToUpper(strings.TrimSpace(dir))
if d == "" {
return nil
}
// 16-wind compass.
m := map[string]float64{
"N": 0,
"NNE": 22.5,
"NE": 45,
"ENE": 67.5,
"E": 90,
"ESE": 112.5,
"SE": 135,
"SSE": 157.5,
"S": 180,
"SSW": 202.5,
"SW": 225,
"WSW": 247.5,
"W": 270,
"WNW": 292.5,
"NW": 315,
"NNW": 337.5,
}
if deg, ok := m[d]; ok {
return &deg
}
return nil
}
func extractFloats(s string) []float64 {
var out []float64
var buf strings.Builder
flush := func() {
if buf.Len() == 0 {
return
}
v, err := strconv.ParseFloat(buf.String(), 64)
if err == nil {
out = append(out, v)
}
buf.Reset()
}
for _, r := range s {
if unicode.IsDigit(r) || r == '.' {
buf.WriteRune(r)
continue
}
flush()
}
flush()
return out
}

View File

@@ -8,9 +8,10 @@ import (
"time" "time"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/model"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common" normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/standards" nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ObservationNormalizer converts: // ObservationNormalizer converts:
@@ -46,11 +47,11 @@ func buildObservation(parsed nwsObservationResponse) (model.WeatherObservation,
// Timestamp (RFC3339) // Timestamp (RFC3339)
var ts time.Time var ts time.Time
if s := strings.TrimSpace(parsed.Properties.Timestamp); s != "" { if s := strings.TrimSpace(parsed.Properties.Timestamp); s != "" {
t, err := time.Parse(time.RFC3339, s) t, err := nwscommon.ParseTime(s)
if err != nil { if err != nil {
return model.WeatherObservation{}, time.Time{}, fmt.Errorf("nws observation normalize: invalid timestamp %q: %w", s, err) return model.WeatherObservation{}, time.Time{}, fmt.Errorf("invalid timestamp %q: %w", s, err)
} }
ts = t ts = t.UTC()
} }
cloudLayers := make([]model.CloudLayer, 0, len(parsed.Properties.CloudLayers)) cloudLayers := make([]model.CloudLayer, 0, len(parsed.Properties.CloudLayers))
@@ -75,9 +76,21 @@ func buildObservation(parsed nwsObservationResponse) (model.WeatherObservation,
// Determine canonical WMO condition code. // Determine canonical WMO condition code.
wmo := mapNWSToWMO(providerDesc, cloudLayers, phenomena) wmo := mapNWSToWMO(providerDesc, cloudLayers, phenomena)
var isDay *bool
if lat, lon := observationLatLon(parsed.Geometry.Coordinates); lat != nil && lon != nil {
isDay = isDayFromLatLonTime(*lat, *lon, ts)
}
// Canonical condition text comes from our WMO table. // Canonical condition text comes from our WMO table.
// NWS observation responses typically do not include a day/night flag -> nil. canonicalText := standards.WMOText(wmo, isDay)
canonicalText := standards.WMOText(wmo, nil)
// Apparent temperature: prefer wind chill when both are supplied.
var apparentC *float64
if parsed.Properties.WindChill.Value != nil {
apparentC = parsed.Properties.WindChill.Value
} else if parsed.Properties.HeatIndex.Value != nil {
apparentC = parsed.Properties.HeatIndex.Value
}
obs := model.WeatherObservation{ obs := model.WeatherObservation{
StationID: parsed.Properties.StationID, StationID: parsed.Properties.StationID,
@@ -86,7 +99,7 @@ func buildObservation(parsed nwsObservationResponse) (model.WeatherObservation,
ConditionCode: wmo, ConditionCode: wmo,
ConditionText: canonicalText, ConditionText: canonicalText,
IsDay: nil, IsDay: isDay,
ProviderRawDescription: providerDesc, ProviderRawDescription: providerDesc,
@@ -107,8 +120,7 @@ func buildObservation(parsed nwsObservationResponse) (model.WeatherObservation,
VisibilityMeters: parsed.Properties.Visibility.Value, VisibilityMeters: parsed.Properties.Visibility.Value,
RelativeHumidityPercent: parsed.Properties.RelativeHumidity.Value, RelativeHumidityPercent: parsed.Properties.RelativeHumidity.Value,
WindChillC: parsed.Properties.WindChill.Value, ApparentTemperatureC: apparentC,
HeatIndexC: parsed.Properties.HeatIndex.Value,
ElevationMeters: parsed.Properties.Elevation.Value, ElevationMeters: parsed.Properties.Elevation.Value,
RawMessage: parsed.Properties.RawMessage, RawMessage: parsed.Properties.RawMessage,

View File

@@ -13,4 +13,10 @@ func Register(reg *fknormalize.Registry) {
// Observations // Observations
reg.Register(ObservationNormalizer{}) reg.Register(ObservationNormalizer{})
// Forecasts
reg.Register(ForecastNormalizer{})
// Alerts
reg.Register(AlertsNormalizer{})
} }

View File

@@ -1,10 +1,18 @@
// FILE: ./internal/normalizers/nws/types.go // FILE: ./internal/normalizers/nws/types.go
package nws package nws
import (
"encoding/json"
)
// nwsObservationResponse is a minimal-but-sufficient representation of the NWS // nwsObservationResponse is a minimal-but-sufficient representation of the NWS
// station observation GeoJSON payload needed for mapping into model.WeatherObservation. // station observation GeoJSON payload needed for mapping into model.WeatherObservation.
type nwsObservationResponse struct { type nwsObservationResponse struct {
ID string `json:"id"` ID string `json:"id"`
Geometry struct {
Type string `json:"type"`
Coordinates []float64 `json:"coordinates"` // GeoJSON point: [lon, lat]
} `json:"geometry"`
Properties struct { Properties struct {
StationID string `json:"stationId"` StationID string `json:"stationId"`
StationName string `json:"stationName"` StationName string `json:"stationName"`
@@ -87,3 +95,127 @@ type nwsObservationResponse struct {
} `json:"cloudLayers"` } `json:"cloudLayers"`
} `json:"properties"` } `json:"properties"`
} }
// nwsForecastResponse is a minimal-but-sufficient representation of the NWS
// gridpoint forecast GeoJSON payload needed for mapping into model.WeatherForecastRun.
//
// This is currently designed to support the hourly forecast endpoint; revisions may be needed
// to accommodate other forecast endpoints in the future.
type nwsForecastResponse struct {
Geometry struct {
Type string `json:"type"`
Coordinates [][][]float64 `json:"coordinates"` // GeoJSON polygon: [ring][point][lon,lat]
} `json:"geometry"`
Properties struct {
Units string `json:"units"` // "us" or "si" (often "us" for hourly)
ForecastGenerator string `json:"forecastGenerator"` // e.g. "HourlyForecastGenerator"
GeneratedAt string `json:"generatedAt"` // RFC3339-ish
UpdateTime string `json:"updateTime"` // RFC3339-ish
ValidTimes string `json:"validTimes"`
Elevation struct {
UnitCode string `json:"unitCode"`
Value *float64 `json:"value"`
} `json:"elevation"`
Periods []nwsForecastPeriod `json:"periods"`
} `json:"properties"`
}
type nwsForecastPeriod struct {
Number int `json:"number"`
Name string `json:"name"`
StartTime string `json:"startTime"`
EndTime string `json:"endTime"`
IsDaytime *bool `json:"isDaytime"`
Temperature *float64 `json:"temperature"`
TemperatureUnit string `json:"temperatureUnit"` // "F" or "C"
TemperatureTrend any `json:"temperatureTrend"`
ProbabilityOfPrecipitation struct {
UnitCode string `json:"unitCode"`
Value *float64 `json:"value"`
} `json:"probabilityOfPrecipitation"`
Dewpoint struct {
UnitCode string `json:"unitCode"`
Value *float64 `json:"value"`
} `json:"dewpoint"`
RelativeHumidity struct {
UnitCode string `json:"unitCode"`
Value *float64 `json:"value"`
} `json:"relativeHumidity"`
WindSpeed string `json:"windSpeed"` // e.g. "9 mph", "10 to 15 mph"
WindDirection string `json:"windDirection"` // e.g. "W", "NW"
Icon string `json:"icon"`
ShortForecast string `json:"shortForecast"`
DetailedForecast string `json:"detailedForecast"`
}
// nwsAlertsResponse is a minimal-but-sufficient representation of the NWS /alerts
// FeatureCollection payload needed for mapping into model.WeatherAlertRun.
type nwsAlertsResponse struct {
Updated string `json:"updated"`
Title string `json:"title"`
Features []nwsAlertFeature `json:"features"`
}
type nwsAlertFeature struct {
ID string `json:"id"`
Properties nwsAlertProperties `json:"properties"`
}
type nwsAlertProperties struct {
// Identifiers.
ID string `json:"id"` // often a URL/URI
Identifier string `json:"identifier"` // CAP identifier string (sometimes distinct)
// Classification / headline.
Event string `json:"event"`
Headline string `json:"headline"`
Severity string `json:"severity"`
Urgency string `json:"urgency"`
Certainty string `json:"certainty"`
Status string `json:"status"`
MessageType string `json:"messageType"`
Category string `json:"category"`
Response string `json:"response"`
// Narrative.
Description string `json:"description"`
Instruction string `json:"instruction"`
// Timing.
Sent string `json:"sent"`
Effective string `json:"effective"`
Onset string `json:"onset"`
Expires string `json:"expires"`
Ends string `json:"ends"`
// Area / provenance.
AreaDesc string `json:"areaDesc"`
Sender string `json:"sender"`
SenderName string `json:"senderName"`
// Related alerts (updates/replacements).
// Keep flexible: some NWS payloads use an object array; others may degrade to strings.
References json.RawMessage `json:"references"`
}
type nwsAlertReference struct {
ID string `json:"id"`
Identifier string `json:"identifier"`
Sender string `json:"sender"`
Sent string `json:"sent"`
}

View File

@@ -4,8 +4,8 @@ package nws
import ( import (
"strings" "strings"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/model"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common" normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
) )
// mapNWSToWMO maps NWS signals into a canonical WMO code. // mapNWSToWMO maps NWS signals into a canonical WMO code.

View File

@@ -1,19 +0,0 @@
// FILE: ./internal/normalizers/openmeteo/common.go
package openmeteo
import (
"time"
openmeteo "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openmeteo"
)
// parseOpenMeteoTime parses Open-Meteo timestamps.
//
// The actual parsing logic lives in internal/providers/openmeteo so both the
// source (envelope EffectiveAt / event ID) and normalizer (canonical payload)
// can share identical timestamp behavior.
//
// We keep this thin wrapper to avoid churn in the normalizer package.
func parseOpenMeteoTime(s string, tz string, utcOffsetSeconds int) (time.Time, error) {
return openmeteo.ParseTime(s, tz, utcOffsetSeconds)
}

View File

@@ -0,0 +1,246 @@
package openmeteo
import (
"context"
"fmt"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/event"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
omcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// ForecastNormalizer converts:
//
// standards.SchemaRawOpenMeteoHourlyForecastV1 -> standards.SchemaWeatherForecastV1
//
// It interprets Open-Meteo hourly forecast JSON and maps it into the canonical
// model.WeatherForecastRun representation.
//
// Caveats / assumptions:
// - Open-Meteo does not provide a true "issued at" timestamp; IssuedAt uses
// Event.EmittedAt when present, otherwise the first hourly time.
// - Hourly payloads are array-oriented; missing fields are treated as nil per-period.
// - Snowfall is provided in centimeters and is converted to millimeters.
// - apparent_temperature is mapped to ApparentTemperatureC when present.
type ForecastNormalizer struct{}
func (ForecastNormalizer) Match(e event.Event) bool {
return strings.TrimSpace(e.Schema) == standards.SchemaRawOpenMeteoHourlyForecastV1
}
func (ForecastNormalizer) Normalize(ctx context.Context, in event.Event) (*event.Event, error) {
_ = ctx // normalization is pure/CPU; keep ctx for future expensive steps
// If present, prefer the existing event EmittedAt as IssuedAt.
var fallbackIssued time.Time
if !in.EmittedAt.IsZero() {
fallbackIssued = in.EmittedAt.UTC()
}
return normcommon.NormalizeJSON(
in,
"openmeteo hourly forecast",
standards.SchemaWeatherForecastV1,
func(parsed omForecastResponse) (model.WeatherForecastRun, time.Time, error) {
return buildForecast(parsed, fallbackIssued)
},
)
}
// buildForecast contains the domain mapping logic (provider -> canonical model).
func buildForecast(parsed omForecastResponse, fallbackIssued time.Time) (model.WeatherForecastRun, time.Time, error) {
times := parsed.Hourly.Time
if len(times) == 0 {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("missing hourly.time")
}
issuedAt := fallbackIssued.UTC()
run := model.WeatherForecastRun{
LocationID: normcommon.SynthStationIDPtr("OPENMETEO", parsed.Latitude, parsed.Longitude),
LocationName: "Open-Meteo",
IssuedAt: time.Time{},
UpdatedAt: nil,
Product: model.ForecastProductHourly,
Latitude: floatCopy(parsed.Latitude),
Longitude: floatCopy(parsed.Longitude),
ElevationMeters: floatCopy(parsed.Elevation),
Periods: nil,
}
periods := make([]model.WeatherForecastPeriod, 0, len(times))
var prevStart time.Time
for i := range times {
start, err := parseHourlyTime(parsed, i)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
if issuedAt.IsZero() && i == 0 {
issuedAt = start
}
end, err := parsePeriodEnd(parsed, i, start, prevStart)
if err != nil {
return model.WeatherForecastRun{}, time.Time{}, err
}
prevStart = start
var isDay *bool
if v := intAt(parsed.Hourly.IsDay, i); v != nil {
b := *v == 1
isDay = &b
}
wmo := wmoAt(parsed.Hourly.WeatherCode, i)
canonicalText := standards.WMOText(wmo, isDay)
period := model.WeatherForecastPeriod{
StartTime: start,
EndTime: end,
Name: "",
IsDay: isDay,
ConditionCode: wmo,
ConditionText: canonicalText,
ProviderRawDescription: "",
TextDescription: canonicalText,
DetailedText: "",
IconURL: "",
}
if v := floatAt(parsed.Hourly.Temperature2m, i); v != nil {
period.TemperatureC = v
}
if v := floatAt(parsed.Hourly.ApparentTemp, i); v != nil {
period.ApparentTemperatureC = v
}
if v := floatAt(parsed.Hourly.DewPoint2m, i); v != nil {
period.DewpointC = v
}
if v := floatAt(parsed.Hourly.RelativeHumidity2m, i); v != nil {
period.RelativeHumidityPercent = v
}
if v := floatAt(parsed.Hourly.WindDirection10m, i); v != nil {
period.WindDirectionDegrees = v
}
if v := floatAt(parsed.Hourly.WindSpeed10m, i); v != nil {
period.WindSpeedKmh = v
}
if v := floatAt(parsed.Hourly.WindGusts10m, i); v != nil {
period.WindGustKmh = v
}
if v := floatAt(parsed.Hourly.Visibility, i); v != nil {
period.VisibilityMeters = v
}
if v := floatAt(parsed.Hourly.CloudCover, i); v != nil {
period.CloudCoverPercent = v
}
if v := floatAt(parsed.Hourly.UVIndex, i); v != nil {
period.UVIndex = v
}
if v := floatAt(parsed.Hourly.PrecipProbability, i); v != nil {
period.ProbabilityOfPrecipitationPercent = v
}
if v := floatAt(parsed.Hourly.Precipitation, i); v != nil {
period.PrecipitationAmountMm = v
}
if v := floatAt(parsed.Hourly.Snowfall, i); v != nil {
mm := *v * 10.0
period.SnowfallDepthMM = &mm
}
if v := floatAt(parsed.Hourly.SurfacePressure, i); v != nil {
pa := normcommon.PressurePaFromHPa(*v)
period.BarometricPressurePa = &pa
} else if v := floatAt(parsed.Hourly.PressureMSL, i); v != nil {
pa := normcommon.PressurePaFromHPa(*v)
period.BarometricPressurePa = &pa
}
periods = append(periods, period)
}
run.Periods = periods
if issuedAt.IsZero() {
return model.WeatherForecastRun{}, time.Time{}, fmt.Errorf("missing issuedAt")
}
run.IssuedAt = issuedAt
// EffectiveAt policy for forecasts: treat IssuedAt as the effective time (dedupe-friendly).
return run, issuedAt, nil
}
func parseHourlyTime(parsed omForecastResponse, idx int) (time.Time, error) {
if idx < 0 || idx >= len(parsed.Hourly.Time) {
return time.Time{}, fmt.Errorf("hourly.time[%d] missing", idx)
}
raw := strings.TrimSpace(parsed.Hourly.Time[idx])
if raw == "" {
return time.Time{}, fmt.Errorf("hourly.time[%d] empty", idx)
}
t, err := omcommon.ParseTime(raw, parsed.Timezone, parsed.UTCOffsetSeconds)
if err != nil {
return time.Time{}, fmt.Errorf("hourly.time[%d] invalid %q: %w", idx, raw, err)
}
return t.UTC(), nil
}
func parsePeriodEnd(parsed omForecastResponse, idx int, start, prevStart time.Time) (time.Time, error) {
if idx+1 < len(parsed.Hourly.Time) {
return parseHourlyTime(parsed, idx+1)
}
step := time.Hour
if !prevStart.IsZero() {
if d := start.Sub(prevStart); d > 0 {
step = d
}
}
return start.Add(step), nil
}
func floatCopy(v *float64) *float64 {
if v == nil {
return nil
}
out := *v
return &out
}
func floatAt(vals []*float64, idx int) *float64 {
if idx < 0 || idx >= len(vals) {
return nil
}
return floatCopy(vals[idx])
}
func intAt(vals []*int, idx int) *int {
if idx < 0 || idx >= len(vals) {
return nil
}
if vals[idx] == nil {
return nil
}
out := *vals[idx]
return &out
}
func wmoAt(vals []*int, idx int) model.WMOCode {
if v := intAt(vals, idx); v != nil {
return model.WMOCode(*v)
}
return model.WMOUnknown
}

View File

@@ -8,9 +8,10 @@ import (
"time" "time"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/model"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common" normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/standards" omcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ObservationNormalizer converts: // ObservationNormalizer converts:
@@ -54,9 +55,9 @@ func buildObservation(parsed omResponse) (model.WeatherObservation, time.Time, e
// Parse current.time. // Parse current.time.
var ts time.Time var ts time.Time
if s := strings.TrimSpace(parsed.Current.Time); s != "" { if s := strings.TrimSpace(parsed.Current.Time); s != "" {
t, err := parseOpenMeteoTime(s, parsed.Timezone, parsed.UTCOffsetSeconds) t, err := omcommon.ParseTime(s, parsed.Timezone, parsed.UTCOffsetSeconds)
if err != nil { if err != nil {
return model.WeatherObservation{}, time.Time{}, fmt.Errorf("openmeteo observation normalize: parse time %q: %w", s, err) return model.WeatherObservation{}, time.Time{}, fmt.Errorf("parse time %q: %w", s, err)
} }
ts = t.UTC() ts = t.UTC()
} }
@@ -77,18 +78,8 @@ func buildObservation(parsed omResponse) (model.WeatherObservation, time.Time, e
canonicalText := standards.WMOText(wmo, isDay) canonicalText := standards.WMOText(wmo, isDay)
// Station identity: Open-Meteo is not a station feed; synthesize from coordinates. // Station identity: Open-Meteo is not a station feed; synthesize from coordinates.
stationID := "" // Require BOTH lat/lon to avoid misleading OPENMETEO(0.00000,...) IDs.
if parsed.Latitude != nil || parsed.Longitude != nil { stationID := normcommon.SynthStationIDPtr("OPENMETEO", parsed.Latitude, parsed.Longitude)
lat := 0.0
lon := 0.0
if parsed.Latitude != nil {
lat = *parsed.Latitude
}
if parsed.Longitude != nil {
lon = *parsed.Longitude
}
stationID = fmt.Sprintf("OPENMETEO(%.5f,%.5f)", lat, lon)
}
obs := model.WeatherObservation{ obs := model.WeatherObservation{
StationID: stationID, StationID: stationID,
@@ -116,6 +107,10 @@ func buildObservation(parsed omResponse) (model.WeatherObservation, time.Time, e
v := *parsed.Current.Temperature2m v := *parsed.Current.Temperature2m
obs.TemperatureC = &v obs.TemperatureC = &v
} }
if parsed.Current.ApparentTemperature != nil {
v := *parsed.Current.ApparentTemperature
obs.ApparentTemperatureC = &v
}
if parsed.Current.RelativeHumidity2m != nil { if parsed.Current.RelativeHumidity2m != nil {
v := *parsed.Current.RelativeHumidity2m v := *parsed.Current.RelativeHumidity2m

View File

@@ -13,4 +13,6 @@ func Register(reg *fknormalize.Registry) {
// Observations // Observations
reg.Register(ObservationNormalizer{}) reg.Register(ObservationNormalizer{})
// Forecasts
reg.Register(ForecastNormalizer{})
} }

View File

@@ -19,6 +19,7 @@ type omCurrent struct {
Time string `json:"time"` // e.g. "2026-01-10T12:30" (often no timezone suffix) Time string `json:"time"` // e.g. "2026-01-10T12:30" (often no timezone suffix)
Temperature2m *float64 `json:"temperature_2m"` Temperature2m *float64 `json:"temperature_2m"`
ApparentTemperature *float64 `json:"apparent_temperature"`
RelativeHumidity2m *float64 `json:"relative_humidity_2m"` RelativeHumidity2m *float64 `json:"relative_humidity_2m"`
WeatherCode *int `json:"weather_code"` WeatherCode *int `json:"weather_code"`
@@ -31,3 +32,38 @@ type omCurrent struct {
IsDay *int `json:"is_day"` // 0/1 IsDay *int `json:"is_day"` // 0/1
} }
// omForecastResponse is a minimal-but-sufficient representation of Open-Meteo
// hourly forecast payloads for mapping into model.WeatherForecastRun.
type omForecastResponse struct {
Latitude *float64 `json:"latitude"`
Longitude *float64 `json:"longitude"`
Timezone string `json:"timezone"`
UTCOffsetSeconds int `json:"utc_offset_seconds"`
Elevation *float64 `json:"elevation"`
Hourly omForecastHourly `json:"hourly"`
}
type omForecastHourly struct {
Time []string `json:"time"`
Temperature2m []*float64 `json:"temperature_2m"`
RelativeHumidity2m []*float64 `json:"relative_humidity_2m"`
DewPoint2m []*float64 `json:"dew_point_2m"`
ApparentTemp []*float64 `json:"apparent_temperature"`
PrecipProbability []*float64 `json:"precipitation_probability"`
Precipitation []*float64 `json:"precipitation"`
Snowfall []*float64 `json:"snowfall"`
WeatherCode []*int `json:"weather_code"`
SurfacePressure []*float64 `json:"surface_pressure"`
PressureMSL []*float64 `json:"pressure_msl"`
WindSpeed10m []*float64 `json:"wind_speed_10m"`
WindDirection10m []*float64 `json:"wind_direction_10m"`
WindGusts10m []*float64 `json:"wind_gusts_10m"`
IsDay []*int `json:"is_day"`
CloudCover []*float64 `json:"cloud_cover"`
Visibility []*float64 `json:"visibility"`
UVIndex []*float64 `json:"uv_index"`
}

View File

@@ -4,6 +4,8 @@ package openweather
import ( import (
"fmt" "fmt"
"strings" "strings"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
) )
// This file holds provider-specific helpers that are shared across multiple // This file holds provider-specific helpers that are shared across multiple
@@ -67,5 +69,5 @@ func openWeatherStationID(parsed owmResponse) string {
return fmt.Sprintf("OPENWEATHER(%d)", parsed.ID) return fmt.Sprintf("OPENWEATHER(%d)", parsed.ID)
} }
// Fallback: synthesize from coordinates. // Fallback: synthesize from coordinates.
return fmt.Sprintf("OPENWEATHER(%.5f,%.5f)", parsed.Coord.Lat, parsed.Coord.Lon) return normcommon.SynthStationID("OPENWEATHER", parsed.Coord.Lat, parsed.Coord.Lon)
} }

View File

@@ -7,9 +7,9 @@ import (
"time" "time"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/model"
normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common" normcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/normalizers/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/standards" "gitea.maximumdirect.net/ejr/weatherfeeder/model"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ObservationNormalizer converts: // ObservationNormalizer converts:
@@ -68,6 +68,11 @@ func buildObservation(parsed owmResponse) (model.WeatherObservation, time.Time,
// - wind speed is m/s -> km/h conversion // - wind speed is m/s -> km/h conversion
tempC := parsed.Main.Temp tempC := parsed.Main.Temp
rh := parsed.Main.Humidity rh := parsed.Main.Humidity
var apparentC *float64
if parsed.Main.FeelsLike != nil {
v := *parsed.Main.FeelsLike
apparentC = &v
}
surfacePa := normcommon.PressurePaFromHPa(parsed.Main.Pressure) surfacePa := normcommon.PressurePaFromHPa(parsed.Main.Pressure)
var seaLevelPa *float64 var seaLevelPa *float64
@@ -117,6 +122,7 @@ func buildObservation(parsed owmResponse) (model.WeatherObservation, time.Time,
IconURL: iconURL, IconURL: iconURL,
TemperatureC: &tempC, TemperatureC: &tempC,
ApparentTemperatureC: apparentC,
WindDirectionDegrees: parsed.Wind.Deg, WindDirectionDegrees: parsed.Wind.Deg,
WindSpeedKmh: &wsKmh, WindSpeedKmh: &wsKmh,

View File

@@ -16,6 +16,7 @@ type owmResponse struct {
Main struct { Main struct {
Temp float64 `json:"temp"` // °C when units=metric (enforced by source) Temp float64 `json:"temp"` // °C when units=metric (enforced by source)
FeelsLike *float64 `json:"feels_like"` // °C when units=metric (enforced by source)
Pressure float64 `json:"pressure"` // hPa Pressure float64 `json:"pressure"` // hPa
Humidity float64 `json:"humidity"` // % Humidity float64 `json:"humidity"` // %
SeaLevel *float64 `json:"sea_level"` SeaLevel *float64 `json:"sea_level"`

View File

@@ -1,7 +1,7 @@
// FILE: ./internal/normalizers/openweather/wmo_map.go // FILE: ./internal/normalizers/openweather/wmo_map.go
package openweather package openweather
import "gitea.maximumdirect.net/ejr/weatherfeeder/internal/model" import "gitea.maximumdirect.net/ejr/weatherfeeder/model"
// mapOpenWeatherToWMO maps OpenWeather weather condition IDs into weatherfeeder's // mapOpenWeatherToWMO maps OpenWeather weather condition IDs into weatherfeeder's
// canonical WMO code vocabulary. // canonical WMO code vocabulary.

View File

@@ -0,0 +1,8 @@
// Package nws contains provider-specific helper code for the National Weather Service
// used by both sources and normalizers.
//
// Rules:
// - No network I/O here (sources fetch; normalizers transform).
// - Keep helpers deterministic and easy to unit test.
// - Prefer putting provider quirks/parsing here when sources + normalizers both need it.
package nws

View File

@@ -0,0 +1,50 @@
package nws
import (
"fmt"
"strings"
"time"
)
// ParseTime parses NWS timestamps.
//
// NWS observation timestamps are typically RFC3339, sometimes with fractional seconds.
// We accept RFC3339Nano first, then RFC3339.
func ParseTime(s string) (time.Time, error) {
s = strings.TrimSpace(s)
if s == "" {
return time.Time{}, fmt.Errorf("empty time")
}
if t, err := time.Parse(time.RFC3339Nano, s); err == nil {
return t, nil
}
if t, err := time.Parse(time.RFC3339, s); err == nil {
return t, nil
}
return time.Time{}, fmt.Errorf("unsupported NWS timestamp format: %q", s)
}
// ParseTimeBestEffort parses an NWS timestamp and returns it in UTC.
//
// This is a convenience for normalizers that want "best effort" parsing:
// invalid/empty strings do not fail the entire normalization; they return zero time.
func ParseTimeBestEffort(s string) time.Time {
t, err := ParseTime(s)
if err != nil {
return time.Time{}
}
return t.UTC()
}
// ParseTimePtr parses an NWS timestamp and returns a UTC *time.Time.
//
// Empty/unparseable input returns nil. This is useful for optional CAP fields.
func ParseTimePtr(s string) *time.Time {
t := ParseTimeBestEffort(s)
if t.IsZero() {
return nil
}
return &t
}

View File

@@ -0,0 +1,8 @@
// Package openweather contains provider-specific helper code for OpenWeather used by
// both sources and normalizers.
//
// Rules:
// - No network I/O here.
// - Keep helpers deterministic and easy to unit test.
// - Put provider invariants here (e.g., units=metric enforcement).
package openweather

View File

@@ -0,0 +1,26 @@
package openweather
import (
"fmt"
"net/url"
"strings"
)
// RequireMetricUnits enforces weatherfeeder's OpenWeather invariant:
// the request URL must include units=metric (otherwise temperatures/winds/pressure differ).
func RequireMetricUnits(rawURL string) error {
u, err := url.Parse(strings.TrimSpace(rawURL))
if err != nil {
return fmt.Errorf("invalid url %q: %w", rawURL, err)
}
units := strings.ToLower(strings.TrimSpace(u.Query().Get("units")))
if units != "metric" {
if units == "" {
units = "(missing; defaults to standard)"
}
return fmt.Errorf("url must include units=metric (got units=%s)", units)
}
return nil
}

View File

@@ -1,8 +1,6 @@
package sources package sources
import ( import (
"fmt"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/nws" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openmeteo" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openweather" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/openweather"
@@ -29,28 +27,12 @@ func RegisterBuiltins(r *fksource.Registry) {
r.Register("openmeteo_observation", func(cfg config.SourceConfig) (fksource.Source, error) { r.Register("openmeteo_observation", func(cfg config.SourceConfig) (fksource.Source, error) {
return openmeteo.NewObservationSource(cfg) return openmeteo.NewObservationSource(cfg)
}) })
r.Register("openmeteo_forecast", func(cfg config.SourceConfig) (fksource.Source, error) {
return openmeteo.NewForecastSource(cfg)
})
// OpenWeatherMap drivers // OpenWeatherMap drivers
r.Register("openweather_observation", func(cfg config.SourceConfig) (fksource.Source, error) { r.Register("openweather_observation", func(cfg config.SourceConfig) (fksource.Source, error) {
return openweather.NewObservationSource(cfg) return openweather.NewObservationSource(cfg)
}) })
} }
// Optional: centralize some common config checks used by multiple drivers.
//
// NOTE: feedkit/config.SourceConfig intentionally keeps driver-specific options
// inside cfg.Params, so drivers can evolve independently without feedkit
// importing domain config packages.
func RequireURL(cfg config.SourceConfig) error {
if cfg.Params == nil {
return fmt.Errorf("source %q: params.url is required", cfg.Name)
}
// Canonical key is "url". We also accept "URL" as a convenience.
url, ok := cfg.ParamString("url", "URL")
if !ok {
return fmt.Errorf("source %q: params.url is required", cfg.Name)
}
_ = url // (optional) return it if you want this helper to provide the value
return nil
}

View File

@@ -9,17 +9,34 @@ import (
// SingleRawEvent constructs, validates, and returns a slice containing exactly one event. // SingleRawEvent constructs, validates, and returns a slice containing exactly one event.
// //
// This removes the repetitive "event envelope ceremony" from individual sources. // This removes repetitive "event envelope ceremony" from individual sources.
// Sources remain responsible for: // Sources remain responsible for:
// - fetching bytes (raw payload) // - fetching bytes (raw payload)
// - choosing Schema (raw schema identifier) // - choosing Schema (raw schema identifier)
// - computing a stable Event.ID and (optional) EffectiveAt // - computing Event.ID and (optional) EffectiveAt
func SingleRawEvent(kind event.Kind, sourceName string, schema string, id string, effectiveAt *time.Time, payload any) ([]event.Event, error) { //
// emittedAt is explicit so callers can compute IDs using the same timestamp (or
// so tests can provide a stable value).
func SingleRawEvent(
kind event.Kind,
sourceName string,
schema string,
id string,
emittedAt time.Time,
effectiveAt *time.Time,
payload any,
) ([]event.Event, error) {
if emittedAt.IsZero() {
emittedAt = time.Now().UTC()
} else {
emittedAt = emittedAt.UTC()
}
e := event.Event{ e := event.Event{
ID: id, ID: id,
Kind: kind, Kind: kind,
Source: sourceName, Source: sourceName,
EmittedAt: time.Now().UTC(), EmittedAt: emittedAt,
EffectiveAt: effectiveAt, EffectiveAt: effectiveAt,
// RAW schema (normalizer matches on this). // RAW schema (normalizer matches on this).

View File

@@ -1,70 +1,76 @@
// FILE: ./internal/sources/common/http.go // FILE: ./internal/sources/common/http_source.go
package common package common
import ( import (
"context" "context"
"encoding/json"
"fmt" "fmt"
"io"
"net/http" "net/http"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/transport"
) )
// maxResponseBodyBytes is a hard safety limit on HTTP response bodies. // HTTPSource is a tiny, reusable "HTTP polling spine" for weatherfeeder sources.
// API responses should be small, so this protects us from accidental //
// or malicious large responses. // It centralizes the boring parts:
const maxResponseBodyBytes = 2 << 21 // 4 MiB // - standard config shape (url + user_agent) via RequireHTTPSourceConfig
// - a default http.Client with timeout
// DefaultHTTPTimeout is the standard timeout used by weatherfeeder HTTP sources. // - FetchBody / headers / max-body safety limit
// Individual drivers may override this if they have a specific need. // - consistent error wrapping (driver + source name)
const DefaultHTTPTimeout = 10 * time.Second //
// Individual drivers remain responsible for:
// NewHTTPClient returns a simple http.Client configured with a timeout. // - decoding minimal metadata (for Event.ID / EffectiveAt)
// If timeout <= 0, DefaultHTTPTimeout is used. // - constructing the event envelope (kind/schema/payload)
func NewHTTPClient(timeout time.Duration) *http.Client { type HTTPSource struct {
if timeout <= 0 { Driver string
timeout = DefaultHTTPTimeout Name string
} URL string
return &http.Client{Timeout: timeout} UserAgent string
Accept string
Client *http.Client
} }
func FetchBody(ctx context.Context, client *http.Client, url, userAgent, accept string) ([]byte, error) { // NewHTTPSource builds an HTTPSource using weatherfeeder's standard HTTP source
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil) // config (params.url + params.user_agent) and a default HTTP client.
func NewHTTPSource(driver string, cfg config.SourceConfig, accept string) (*HTTPSource, error) {
c, err := RequireHTTPSourceConfig(driver, cfg)
if err != nil { if err != nil {
return nil, err return nil, err
} }
if userAgent != "" { return &HTTPSource{
req.Header.Set("User-Agent", userAgent) Driver: driver,
} Name: c.Name,
if accept != "" { URL: c.URL,
req.Header.Set("Accept", accept) UserAgent: c.UserAgent,
Accept: accept,
Client: transport.NewHTTPClient(transport.DefaultHTTPTimeout),
}, nil
}
// FetchBytes fetches the URL and returns the raw response body bytes.
func (s *HTTPSource) FetchBytes(ctx context.Context) ([]byte, error) {
client := s.Client
if client == nil {
// Defensive: allow tests or callers to nil out Client; keep behavior sane.
client = transport.NewHTTPClient(transport.DefaultHTTPTimeout)
} }
res, err := client.Do(req) b, err := transport.FetchBody(ctx, client, s.URL, s.UserAgent, s.Accept)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("%s %q: %w", s.Driver, s.Name, err)
} }
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
return nil, fmt.Errorf("HTTP %s", res.Status)
}
// Read at most maxResponseBodyBytes + 1 so we can detect overflow.
limited := io.LimitReader(res.Body, maxResponseBodyBytes+1)
b, err := io.ReadAll(limited)
if err != nil {
return nil, err
}
if len(b) == 0 {
return nil, fmt.Errorf("empty response body")
}
if len(b) > maxResponseBodyBytes {
return nil, fmt.Errorf("response body too large (>%d bytes)", maxResponseBodyBytes)
}
return b, nil return b, nil
} }
// FetchJSON fetches the URL and returns the raw body as json.RawMessage.
// json.Unmarshal accepts json.RawMessage directly, so callers can decode minimal
// metadata without keeping both []byte and RawMessage in their own structs.
func (s *HTTPSource) FetchJSON(ctx context.Context) (json.RawMessage, error) {
b, err := s.FetchBytes(ctx)
if err != nil {
return nil, err
}
return json.RawMessage(b), nil
}

View File

@@ -0,0 +1,39 @@
// FILE: ./internal/sources/common/id.go
package common
import (
"fmt"
"strings"
"time"
)
// ChooseEventID applies weatherfeeder's opinionated Event.ID policy:
//
// - If upstream provides an ID, use it (trimmed).
// - Otherwise, ID is "<Source>:<EffectiveAt>" when available.
// - If EffectiveAt is unavailable, fall back to "<Source>:<EmittedAt>".
//
// Timestamps are encoded as RFC3339Nano in UTC.
func ChooseEventID(upstreamID, sourceName string, effectiveAt *time.Time, emittedAt time.Time) string {
if id := strings.TrimSpace(upstreamID); id != "" {
return id
}
src := strings.TrimSpace(sourceName)
if src == "" {
src = "UNKNOWN_SOURCE"
}
// Prefer EffectiveAt for dedupe friendliness.
if effectiveAt != nil && !effectiveAt.IsZero() {
return fmt.Sprintf("%s:%s", src, effectiveAt.UTC().Format(time.RFC3339Nano))
}
// Fall back to EmittedAt (still stable within a poll invocation).
t := emittedAt.UTC()
if t.IsZero() {
t = time.Now().UTC()
}
return fmt.Sprintf("%s:%s", src, t.Format(time.RFC3339Nano))
}

View File

@@ -1,54 +1,147 @@
// FILE: internal/sources/nws/alerts.go
package nws package nws
import ( import (
"context" "context"
"fmt" "encoding/json"
"strings" "strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// AlertsSource polls an NWS alerts endpoint and emits a RAW alerts Event.
//
// It intentionally emits the *entire* upstream payload as json.RawMessage and only decodes
// minimal metadata for Event.EffectiveAt and Event.ID.
//
// Output schema:
// - standards.SchemaRawNWSAlertsV1
type AlertsSource struct { type AlertsSource struct {
name string http *common.HTTPSource
url string
userAgent string
} }
func NewAlertsSource(cfg config.SourceConfig) (*AlertsSource, error) { func NewAlertsSource(cfg config.SourceConfig) (*AlertsSource, error) {
if strings.TrimSpace(cfg.Name) == "" { const driver = "nws_alerts"
return nil, fmt.Errorf("nws_alerts: name is required")
} // NWS alerts responses are GeoJSON-ish; allow fallback to plain JSON as well.
if cfg.Params == nil { hs, err := common.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
return nil, fmt.Errorf("nws_alerts %q: params are required (need params.url and params.user_agent)", cfg.Name) if err != nil {
return nil, err
} }
// Driver-specific options live in cfg.Params to keep feedkit domain-agnostic. return &AlertsSource{http: hs}, nil
// Use the typed accessor so callers cant accidentally pass non-strings to TrimSpace.
url, ok := cfg.ParamString("url", "URL")
if !ok {
return nil, fmt.Errorf("nws_alerts %q: params.url is required", cfg.Name)
}
ua, ok := cfg.ParamString("user_agent", "userAgent")
if !ok {
return nil, fmt.Errorf("nws_alerts %q: params.user_agent is required", cfg.Name)
}
return &AlertsSource{
name: cfg.Name,
url: url,
userAgent: ua,
}, nil
} }
func (s *AlertsSource) Name() string { return s.name } func (s *AlertsSource) Name() string { return s.http.Name }
// Kind is used for routing/policy. // Kind is used for routing/policy.
// The envelope type is event.Event; payload will eventually be something like model.WeatherAlert.
func (s *AlertsSource) Kind() event.Kind { return event.Kind("alert") } func (s *AlertsSource) Kind() event.Kind { return event.Kind("alert") }
func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *AlertsSource) Poll(ctx context.Context) ([]event.Event, error) {
_ = ctx raw, meta, err := s.fetchRaw(ctx)
return nil, fmt.Errorf("nws.AlertsSource.Poll: TODO implement (url=%s)", s.url) if err != nil {
return nil, err
}
// EffectiveAt policy for alerts:
// Prefer the collection-level "updated" timestamp (best dedupe signal).
// If missing, fall back to the most recent per-alert timestamp we can parse.
var effectiveAt *time.Time
switch {
case !meta.ParsedUpdated.IsZero():
t := meta.ParsedUpdated.UTC()
effectiveAt = &t
case !meta.ParsedLatestFeatureTime.IsZero():
t := meta.ParsedLatestFeatureTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
// NWS alerts collections do not provide a stable per-snapshot ID.
// Use Source:EffectiveAt (or Source:EmittedAt fallback) for dedupe friendliness.
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawNWSAlertsV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
// alertsMeta is a minimal view of the NWS /alerts FeatureCollection.
// We only decode fields used to set EffectiveAt deterministically.
type alertsMeta struct {
Updated string `json:"updated"`
Features []struct {
Properties struct {
Sent string `json:"sent"`
Effective string `json:"effective"`
Expires string `json:"expires"`
Ends string `json:"ends"`
} `json:"properties"`
} `json:"features"`
ParsedUpdated time.Time `json:"-"`
ParsedLatestFeatureTime time.Time `json:"-"`
}
func (s *AlertsSource) fetchRaw(ctx context.Context) (json.RawMessage, alertsMeta, error) {
raw, err := s.http.FetchJSON(ctx)
if err != nil {
return nil, alertsMeta{}, err
}
var meta alertsMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, alertsMeta{}, nil
}
// Top-level updated (preferred).
if us := strings.TrimSpace(meta.Updated); us != "" {
if t, err := nwscommon.ParseTime(us); err == nil {
meta.ParsedUpdated = t.UTC()
}
}
// If Updated is missing/unparseable, compute a best-effort "latest" feature timestamp.
// We prefer Sent/Effective, and fall back to Expires/Ends if needed.
var latest time.Time
for _, f := range meta.Features {
candidates := []string{
f.Properties.Sent,
f.Properties.Effective,
f.Properties.Expires,
f.Properties.Ends,
}
for _, s := range candidates {
ts := strings.TrimSpace(s)
if ts == "" {
continue
}
t, err := nwscommon.ParseTime(ts)
if err != nil {
continue
}
t = t.UTC()
if latest.IsZero() || t.After(latest) {
latest = t
}
}
}
meta.ParsedLatestFeatureTime = latest
return raw, meta, nil
} }

View File

@@ -1,51 +1,129 @@
// FILE: internal/sources/nws/forecast.go
package nws package nws
import ( import (
"context" "context"
"fmt" "encoding/json"
"strings" "strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ForecastSource polls an NWS forecast endpoint (narrative or hourly) and emits a RAW forecast Event.
//
// It intentionally emits the *entire* upstream payload as json.RawMessage and only decodes
// minimal metadata for Event.EffectiveAt and Event.ID.
//
// Output schema (current implementation):
// - standards.SchemaRawNWSHourlyForecastV1
type ForecastSource struct { type ForecastSource struct {
name string http *common.HTTPSource
url string
userAgent string
} }
func NewForecastSource(cfg config.SourceConfig) (*ForecastSource, error) { func NewForecastSource(cfg config.SourceConfig) (*ForecastSource, error) {
if strings.TrimSpace(cfg.Name) == "" { const driver = "nws_forecast"
return nil, fmt.Errorf("nws_forecast: name is required")
} // NWS forecast endpoints are GeoJSON (and sometimes also advertise json-ld/json).
if cfg.Params == nil { hs, err := common.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
return nil, fmt.Errorf("nws_forecast %q: params are required (need params.url and params.user_agent)", cfg.Name) if err != nil {
return nil, err
} }
url, ok := cfg.ParamString("url", "URL") return &ForecastSource{http: hs}, nil
if !ok {
return nil, fmt.Errorf("nws_forecast %q: params.url is required", cfg.Name)
}
ua, ok := cfg.ParamString("user_agent", "userAgent")
if !ok {
return nil, fmt.Errorf("nws_forecast %q: params.user_agent is required", cfg.Name)
}
return &ForecastSource{
name: cfg.Name,
url: url,
userAgent: ua,
}, nil
} }
func (s *ForecastSource) Name() string { return s.name } func (s *ForecastSource) Name() string { return s.http.Name }
// Kind is used for routing/policy. // Kind is used for routing/policy.
func (s *ForecastSource) Kind() event.Kind { return event.Kind("forecast") } func (s *ForecastSource) Kind() event.Kind { return event.Kind("forecast") }
func (s *ForecastSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
_ = ctx raw, meta, err := s.fetchRaw(ctx)
return nil, fmt.Errorf("nws.ForecastSource.Poll: TODO implement (url=%s)", s.url) if err != nil {
return nil, err
}
// EffectiveAt is optional; for forecasts its most naturally the run “issued” time.
// NWS gridpoint forecasts expose generatedAt (preferred) and updateTime/updated.
var effectiveAt *time.Time
switch {
case !meta.ParsedGeneratedAt.IsZero():
t := meta.ParsedGeneratedAt.UTC()
effectiveAt = &t
case !meta.ParsedUpdateTime.IsZero():
t := meta.ParsedUpdateTime.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
// NWS gridpoint forecast GeoJSON commonly has a stable "id" equal to the endpoint URL.
// That is *not* unique per issued run, so we intentionally do not use it for Event.ID.
// Instead we rely on Source:EffectiveAt (or Source:EmittedAt fallback).
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawNWSHourlyForecastV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
type forecastMeta struct {
// Present for GeoJSON Feature responses, but often stable (endpoint URL).
ID string `json:"id"`
Properties struct {
GeneratedAt string `json:"generatedAt"` // preferred “issued/run generated” time
UpdateTime string `json:"updateTime"` // last update time of underlying data
Updated string `json:"updated"` // deprecated alias for updateTime
} `json:"properties"`
ParsedGeneratedAt time.Time `json:"-"`
ParsedUpdateTime time.Time `json:"-"`
}
func (s *ForecastSource) fetchRaw(ctx context.Context) (json.RawMessage, forecastMeta, error) {
raw, err := s.http.FetchJSON(ctx)
if err != nil {
return nil, forecastMeta{}, err
}
var meta forecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, forecastMeta{}, nil
}
// generatedAt (preferred)
genStr := strings.TrimSpace(meta.Properties.GeneratedAt)
if genStr != "" {
if t, err := nwscommon.ParseTime(genStr); err == nil {
meta.ParsedGeneratedAt = t.UTC()
}
}
// updateTime, with fallback to deprecated "updated"
updStr := strings.TrimSpace(meta.Properties.UpdateTime)
if updStr == "" {
updStr = strings.TrimSpace(meta.Properties.Updated)
}
if updStr != "" {
if t, err := nwscommon.ParseTime(updStr); err == nil {
meta.ParsedUpdateTime = t.UTC()
}
}
return raw, meta, nil
} }

View File

@@ -4,74 +4,42 @@ package nws
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"net/http"
"strings" "strings"
"time" "time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
nwscommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/nws"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/standards" "gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ObservationSource polls an NWS station observation endpoint and emits a RAW observation Event. // ObservationSource polls an NWS station observation endpoint and emits a RAW observation Event.
//
// This corresponds to URLs like:
//
// https://api.weather.gov/stations/KSTL/observations/latest
type ObservationSource struct { type ObservationSource struct {
name string http *common.HTTPSource
url string
userAgent string
client *http.Client
} }
func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) { func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
const driver = "nws_observation" const driver = "nws_observation"
c, err := common.RequireHTTPSourceConfig(driver, cfg) hs, err := common.NewHTTPSource(driver, cfg, "application/geo+json, application/json")
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &ObservationSource{ return &ObservationSource{http: hs}, nil
name: c.Name,
url: c.URL,
userAgent: c.UserAgent,
client: common.NewHTTPClient(common.DefaultHTTPTimeout),
}, nil
} }
func (s *ObservationSource) Name() string { return s.name } func (s *ObservationSource) Name() string { return s.http.Name }
// Kind is used for routing/policy.
// We keep Kind canonical (observation) even for raw events; Schema differentiates raw vs canonical.
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") } func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") }
// Poll fetches NWS "latest observation" and emits exactly one RAW Event.
// The RAW payload is json.RawMessage and Schema is standards.SchemaRawNWSObservationV1.
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, err := s.fetchRaw(ctx) raw, meta, err := s.fetchRaw(ctx)
if err != nil { if err != nil {
return nil, err return nil, err
} }
// Event.ID must be set BEFORE normalization (feedkit requires it).
// Prefer NWS-provided "id" (stable URL). Fallback to a stable computed key.
eventID := strings.TrimSpace(meta.ID)
if eventID == "" {
ts := meta.ParsedTimestamp
if ts.IsZero() {
ts = time.Now().UTC()
}
station := strings.TrimSpace(meta.StationID)
if station == "" {
station = "UNKNOWN"
}
eventID = fmt.Sprintf("nws:observation:%s:%s:%s", s.name, station, ts.UTC().Format(time.RFC3339Nano))
}
// EffectiveAt is optional; for observations its naturally the observation timestamp. // EffectiveAt is optional; for observations its naturally the observation timestamp.
var effectiveAt *time.Time var effectiveAt *time.Time
if !meta.ParsedTimestamp.IsZero() { if !meta.ParsedTimestamp.IsZero() {
@@ -79,11 +47,15 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
effectiveAt = &t effectiveAt = &t
} }
emittedAt := time.Now().UTC()
eventID := common.ChooseEventID(meta.ID, s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent( return common.SingleRawEvent(
s.Kind(), s.Kind(),
s.name, s.http.Name,
standards.SchemaRawNWSObservationV1, standards.SchemaRawNWSObservationV1,
eventID, eventID,
emittedAt,
effectiveAt, effectiveAt,
raw, raw,
) )
@@ -91,40 +63,31 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
// ---- RAW fetch + minimal metadata decode ---- // ---- RAW fetch + minimal metadata decode ----
// observationMeta is a *minimal* decode of the NWS payload used only to build
// a stable Event.ID and a useful EffectiveAt for the envelope.
type observationMeta struct { type observationMeta struct {
ID string `json:"id"` ID string `json:"id"`
Properties struct { Properties struct {
StationID string `json:"stationId"`
Timestamp string `json:"timestamp"` Timestamp string `json:"timestamp"`
} `json:"properties"` } `json:"properties"`
// Convenience fields populated after decode.
ParsedTimestamp time.Time `json:"-"` ParsedTimestamp time.Time `json:"-"`
StationID string `json:"-"`
} }
func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, observationMeta, error) { func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, observationMeta, error) {
b, err := common.FetchBody(ctx, s.client, s.url, s.userAgent, "application/geo+json, application/json") raw, err := s.http.FetchJSON(ctx)
if err != nil { if err != nil {
return nil, observationMeta{}, fmt.Errorf("nws_observation %q: %w", s.name, err) return nil, observationMeta{}, err
} }
raw := json.RawMessage(b)
var meta observationMeta var meta observationMeta
if err := json.Unmarshal(b, &meta); err != nil { if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; envelope will fall back to computed ID. // If metadata decode fails, still return raw; envelope will fall back to Source:EffectiveAt.
return raw, observationMeta{}, nil return raw, observationMeta{}, nil
} }
meta.StationID = strings.TrimSpace(meta.Properties.StationID)
tsStr := strings.TrimSpace(meta.Properties.Timestamp) tsStr := strings.TrimSpace(meta.Properties.Timestamp)
if tsStr != "" { if tsStr != "" {
if t, err := time.Parse(time.RFC3339, tsStr); err == nil { if t, err := nwscommon.ParseTime(tsStr); err == nil {
meta.ParsedTimestamp = t meta.ParsedTimestamp = t.UTC()
} }
} }

View File

@@ -0,0 +1,110 @@
package openmeteo
import (
"context"
"encoding/json"
"strings"
"time"
"gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/standards"
)
// ForecastSource polls an Open-Meteo hourly forecast endpoint and emits one RAW Forecast Event.
type ForecastSource struct {
http *common.HTTPSource
}
func NewForecastSource(cfg config.SourceConfig) (*ForecastSource, error) {
const driver = "openmeteo_forecast"
hs, err := common.NewHTTPSource(driver, cfg, "application/json")
if err != nil {
return nil, err
}
return &ForecastSource{http: hs}, nil
}
func (s *ForecastSource) Name() string { return s.http.Name }
func (s *ForecastSource) Kind() event.Kind { return event.Kind("forecast") }
func (s *ForecastSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, err := s.fetchRaw(ctx)
if err != nil {
return nil, err
}
// Open-Meteo does not expose a true "issued at" timestamp for forecast runs.
// We use current.time when present; otherwise we fall back to the first hourly time
// as a proxy for the start of the forecast horizon.
var effectiveAt *time.Time
if !meta.ParsedTimestamp.IsZero() {
t := meta.ParsedTimestamp.UTC()
effectiveAt = &t
}
emittedAt := time.Now().UTC()
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent(
s.Kind(),
s.http.Name,
standards.SchemaRawOpenMeteoHourlyForecastV1,
eventID,
emittedAt,
effectiveAt,
raw,
)
}
// ---- RAW fetch + minimal metadata decode ----
type forecastMeta struct {
Timezone string `json:"timezone"`
UTCOffsetSeconds int `json:"utc_offset_seconds"`
Current struct {
Time string `json:"time"`
} `json:"current"`
Hourly struct {
Time []string `json:"time"`
} `json:"hourly"`
ParsedTimestamp time.Time `json:"-"`
}
func (s *ForecastSource) fetchRaw(ctx context.Context) (json.RawMessage, forecastMeta, error) {
raw, err := s.http.FetchJSON(ctx)
if err != nil {
return nil, forecastMeta{}, err
}
var meta forecastMeta
if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; Poll will fall back to Source:EmittedAt.
return raw, forecastMeta{}, nil
}
ts := strings.TrimSpace(meta.Current.Time)
if ts == "" {
for _, v := range meta.Hourly.Time {
if ts = strings.TrimSpace(v); ts != "" {
break
}
}
}
if ts != "" {
if t, err := openmeteo.ParseTime(ts, meta.Timezone, meta.UTCOffsetSeconds); err == nil {
meta.ParsedTimestamp = t.UTC()
}
}
return raw, meta, nil
}

View File

@@ -4,72 +4,56 @@ package openmeteo
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"fmt"
"net/http"
"strings"
"time" "time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openmeteo" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openmeteo"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/standards" "gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ObservationSource polls an Open-Meteo endpoint and emits one RAW Observation Event. // ObservationSource polls an Open-Meteo endpoint and emits one RAW Observation Event.
type ObservationSource struct { type ObservationSource struct {
name string http *common.HTTPSource
url string
userAgent string
client *http.Client
} }
func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) { func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
const driver = "openmeteo_observation" const driver = "openmeteo_observation"
// We require params.user_agent for uniformity across sources (even though Open-Meteo hs, err := common.NewHTTPSource(driver, cfg, "application/json")
// itself does not strictly require a special User-Agent).
c, err := common.RequireHTTPSourceConfig(driver, cfg)
if err != nil { if err != nil {
return nil, err return nil, err
} }
return &ObservationSource{ return &ObservationSource{http: hs}, nil
name: c.Name,
url: c.URL,
userAgent: c.UserAgent,
client: common.NewHTTPClient(common.DefaultHTTPTimeout),
}, nil
} }
func (s *ObservationSource) Name() string { return s.name } func (s *ObservationSource) Name() string { return s.http.Name }
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") } func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") }
// Poll fetches Open-Meteo "current" and emits exactly one RAW Event.
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
raw, meta, err := s.fetchRaw(ctx) raw, meta, err := s.fetchRaw(ctx)
if err != nil { if err != nil {
return nil, err return nil, err
} }
eventID := buildEventID(s.name, meta)
if strings.TrimSpace(eventID) == "" {
// Extremely defensive fallback: keep the envelope valid no matter what.
eventID = fmt.Sprintf("openmeteo:current:%s:%s", s.name, time.Now().UTC().Format(time.RFC3339Nano))
}
var effectiveAt *time.Time var effectiveAt *time.Time
if !meta.ParsedTimestamp.IsZero() { if !meta.ParsedTimestamp.IsZero() {
t := meta.ParsedTimestamp.UTC() t := meta.ParsedTimestamp.UTC()
effectiveAt = &t effectiveAt = &t
} }
emittedAt := time.Now().UTC()
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent( return common.SingleRawEvent(
s.Kind(), s.Kind(),
s.name, s.http.Name,
standards.SchemaRawOpenMeteoCurrentV1, standards.SchemaRawOpenMeteoCurrentV1,
eventID, eventID,
emittedAt,
effectiveAt, effectiveAt,
raw, raw,
) )
@@ -78,8 +62,6 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
// ---- RAW fetch + minimal metadata decode ---- // ---- RAW fetch + minimal metadata decode ----
type openMeteoMeta struct { type openMeteoMeta struct {
Latitude float64 `json:"latitude"`
Longitude float64 `json:"longitude"`
Timezone string `json:"timezone"` Timezone string `json:"timezone"`
UTCOffsetSeconds int `json:"utc_offset_seconds"` UTCOffsetSeconds int `json:"utc_offset_seconds"`
@@ -91,40 +73,20 @@ type openMeteoMeta struct {
} }
func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, openMeteoMeta, error) { func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, openMeteoMeta, error) {
b, err := common.FetchBody(ctx, s.client, s.url, s.userAgent, "application/json") raw, err := s.http.FetchJSON(ctx)
if err != nil { if err != nil {
return nil, openMeteoMeta{}, fmt.Errorf("openmeteo_observation %q: %w", s.name, err) return nil, openMeteoMeta{}, err
} }
raw := json.RawMessage(b)
var meta openMeteoMeta var meta openMeteoMeta
if err := json.Unmarshal(b, &meta); err != nil { if err := json.Unmarshal(raw, &meta); err != nil {
// If metadata decode fails, still return raw; envelope will fall back to computed ID without EffectiveAt. // If metadata decode fails, still return raw; envelope will omit EffectiveAt.
return raw, openMeteoMeta{}, nil return raw, openMeteoMeta{}, nil
} }
// Best effort: compute a stable EffectiveAt + event ID component.
// If parsing fails, we simply omit EffectiveAt and fall back to time.Now() in buildEventID.
if t, err := openmeteo.ParseTime(meta.Current.Time, meta.Timezone, meta.UTCOffsetSeconds); err == nil { if t, err := openmeteo.ParseTime(meta.Current.Time, meta.Timezone, meta.UTCOffsetSeconds); err == nil {
meta.ParsedTimestamp = t.UTC() meta.ParsedTimestamp = t.UTC()
} }
return raw, meta, nil return raw, meta, nil
} }
func buildEventID(sourceName string, meta openMeteoMeta) string {
locKey := ""
if meta.Latitude != 0 || meta.Longitude != 0 {
locKey = fmt.Sprintf("coord:%.5f,%.5f", meta.Latitude, meta.Longitude)
} else {
locKey = "loc:unknown"
}
ts := meta.ParsedTimestamp
if ts.IsZero() {
ts = time.Now().UTC()
}
return fmt.Sprintf("openmeteo:current:%s:%s:%s", sourceName, locKey, ts.Format(time.RFC3339Nano))
}

View File

@@ -5,58 +5,41 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"fmt" "fmt"
"net/http"
"net/url"
"strings"
"time" "time"
"gitea.maximumdirect.net/ejr/feedkit/config" "gitea.maximumdirect.net/ejr/feedkit/config"
"gitea.maximumdirect.net/ejr/feedkit/event" "gitea.maximumdirect.net/ejr/feedkit/event"
owcommon "gitea.maximumdirect.net/ejr/weatherfeeder/internal/providers/openweather"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common" "gitea.maximumdirect.net/ejr/weatherfeeder/internal/sources/common"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/standards" "gitea.maximumdirect.net/ejr/weatherfeeder/standards"
) )
// ObservationSource polls the OpenWeatherMap "Current weather" endpoint and emits a RAW observation Event.
//
// IMPORTANT UNIT POLICY (weatherfeeder convention):
// OpenWeather changes units based on the `units` query parameter but does NOT include the unit
// system in the response body. To keep normalization deterministic, this driver *requires*
// `units=metric`. If absent (or non-metric), the driver returns an error.
type ObservationSource struct { type ObservationSource struct {
name string http *common.HTTPSource
url string
userAgent string
client *http.Client
} }
func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) { func NewObservationSource(cfg config.SourceConfig) (*ObservationSource, error) {
const driver = "openweather_observation" const driver = "openweather_observation"
c, err := common.RequireHTTPSourceConfig(driver, cfg) hs, err := common.NewHTTPSource(driver, cfg, "application/json")
if err != nil { if err != nil {
return nil, err return nil, err
} }
if err := requireMetricUnits(c.URL); err != nil { if err := owcommon.RequireMetricUnits(hs.URL); err != nil {
return nil, fmt.Errorf("openweather_observation %q: %w", c.Name, err) return nil, fmt.Errorf("%s %q: %w", hs.Driver, hs.Name, err)
} }
return &ObservationSource{ return &ObservationSource{http: hs}, nil
name: c.Name,
url: c.URL,
userAgent: c.UserAgent,
client: common.NewHTTPClient(common.DefaultHTTPTimeout),
}, nil
} }
func (s *ObservationSource) Name() string { return s.name } func (s *ObservationSource) Name() string { return s.http.Name }
func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") } func (s *ObservationSource) Kind() event.Kind { return event.Kind("observation") }
func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) { func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
// Re-check policy defensively (in case the URL is mutated after construction). if err := owcommon.RequireMetricUnits(s.http.URL); err != nil {
if err := requireMetricUnits(s.url); err != nil { return nil, fmt.Errorf("%s %q: %w", s.http.Driver, s.http.Name, err)
return nil, fmt.Errorf("openweather_observation %q: %w", s.name, err)
} }
raw, meta, err := s.fetchRaw(ctx) raw, meta, err := s.fetchRaw(ctx)
@@ -64,22 +47,21 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
return nil, err return nil, err
} }
eventID := buildEventID(s.name, meta)
if strings.TrimSpace(eventID) == "" {
eventID = fmt.Sprintf("openweather:current:%s:%s", s.name, time.Now().UTC().Format(time.RFC3339Nano))
}
var effectiveAt *time.Time var effectiveAt *time.Time
if !meta.ParsedTimestamp.IsZero() { if !meta.ParsedTimestamp.IsZero() {
t := meta.ParsedTimestamp.UTC() t := meta.ParsedTimestamp.UTC()
effectiveAt = &t effectiveAt = &t
} }
emittedAt := time.Now().UTC()
eventID := common.ChooseEventID("", s.http.Name, effectiveAt, emittedAt)
return common.SingleRawEvent( return common.SingleRawEvent(
s.Kind(), s.Kind(),
s.name, s.http.Name,
standards.SchemaRawOpenWeatherCurrentV1, standards.SchemaRawOpenWeatherCurrentV1,
eventID, eventID,
emittedAt,
effectiveAt, effectiveAt,
raw, raw,
) )
@@ -90,27 +72,17 @@ func (s *ObservationSource) Poll(ctx context.Context) ([]event.Event, error) {
type openWeatherMeta struct { type openWeatherMeta struct {
Dt int64 `json:"dt"` // unix seconds, UTC Dt int64 `json:"dt"` // unix seconds, UTC
ID int64 `json:"id"`
Name string `json:"name"`
Coord struct {
Lon float64 `json:"lon"`
Lat float64 `json:"lat"`
} `json:"coord"`
ParsedTimestamp time.Time `json:"-"` ParsedTimestamp time.Time `json:"-"`
} }
func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, openWeatherMeta, error) { func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, openWeatherMeta, error) {
b, err := common.FetchBody(ctx, s.client, s.url, s.userAgent, "application/json") raw, err := s.http.FetchJSON(ctx)
if err != nil { if err != nil {
return nil, openWeatherMeta{}, fmt.Errorf("openweather_observation %q: %w", s.name, err) return nil, openWeatherMeta{}, err
} }
raw := json.RawMessage(b)
var meta openWeatherMeta var meta openWeatherMeta
if err := json.Unmarshal(b, &meta); err != nil { if err := json.Unmarshal(raw, &meta); err != nil {
return raw, openWeatherMeta{}, nil return raw, openWeatherMeta{}, nil
} }
@@ -120,38 +92,3 @@ func (s *ObservationSource) fetchRaw(ctx context.Context) (json.RawMessage, open
return raw, meta, nil return raw, meta, nil
} }
func buildEventID(sourceName string, meta openWeatherMeta) string {
locKey := ""
if meta.ID != 0 {
locKey = fmt.Sprintf("city:%d", meta.ID)
} else if meta.Coord.Lat != 0 || meta.Coord.Lon != 0 {
locKey = fmt.Sprintf("coord:%.5f,%.5f", meta.Coord.Lat, meta.Coord.Lon)
} else {
locKey = "loc:unknown"
}
ts := meta.ParsedTimestamp
if ts.IsZero() {
ts = time.Now().UTC()
}
return fmt.Sprintf("openweather:current:%s:%s:%s", sourceName, locKey, ts.Format(time.RFC3339Nano))
}
func requireMetricUnits(rawURL string) error {
u, err := url.Parse(strings.TrimSpace(rawURL))
if err != nil {
return fmt.Errorf("invalid url %q: %w", rawURL, err)
}
units := strings.ToLower(strings.TrimSpace(u.Query().Get("units")))
if units != "metric" {
if units == "" {
units = "(missing; defaults to standard)"
}
return fmt.Errorf("url must include units=metric (got units=%s)", units)
}
return nil
}

View File

@@ -1,127 +0,0 @@
package standards
import "gitea.maximumdirect.net/ejr/weatherfeeder/internal/model"
// This file provides small, shared helper functions for reasoning about WMO codes.
// These are intentionally "coarse" categories that are useful for business logic,
// dashboards, and alerting decisions.
//
// Example uses:
// - jogging suitability: precipitation? thunderstorm? freezing precip?
// - quick glance: "is it cloudy?" "is there any precip?"
// - downstream normalizers / aggregators
func IsThunderstorm(code model.WMOCode) bool {
switch code {
case 95, 96, 99:
return true
default:
return false
}
}
func IsHail(code model.WMOCode) bool {
switch code {
case 96, 99:
return true
default:
return false
}
}
func IsFog(code model.WMOCode) bool {
switch code {
case 45, 48:
return true
default:
return false
}
}
// IsPrecipitation returns true if the code represents any precipitation
// (drizzle, rain, snow, showers, etc.).
func IsPrecipitation(code model.WMOCode) bool {
switch code {
// Drizzle
case 51, 53, 55, 56, 57:
return true
// Rain
case 61, 63, 65, 66, 67:
return true
// Snow
case 71, 73, 75, 77:
return true
// Showers
case 80, 81, 82, 85, 86:
return true
// Thunderstorm (often includes rain/hail)
case 95, 96, 99:
return true
default:
return false
}
}
func IsRainFamily(code model.WMOCode) bool {
switch code {
// Drizzle + freezing drizzle
case 51, 53, 55, 56, 57:
return true
// Rain + freezing rain
case 61, 63, 65, 66, 67:
return true
// Rain showers
case 80, 81, 82:
return true
// Thunderstorm often implies rain
case 95, 96, 99:
return true
default:
return false
}
}
func IsSnowFamily(code model.WMOCode) bool {
switch code {
// Snow and related
case 71, 73, 75, 77:
return true
// Snow showers
case 85, 86:
return true
default:
return false
}
}
// IsFreezingPrecip returns true if the code represents freezing drizzle/rain.
func IsFreezingPrecip(code model.WMOCode) bool {
switch code {
case 56, 57, 66, 67:
return true
default:
return false
}
}
// IsSkyOnly returns true for codes that represent "sky condition only"
// (clear/mostly/partly/cloudy) rather than fog/precip/etc.
func IsSkyOnly(code model.WMOCode) bool {
switch code {
case 0, 1, 2, 3:
return true
default:
return false
}
}

78
model/alert.go Normal file
View File

@@ -0,0 +1,78 @@
// FILE: internal/model/alert.go
package model
import "time"
// WeatherAlertRun is a snapshot of *active* alerts for a location as-of a point in time.
//
// This mirrors WeatherForecastRun's "one issued snapshot -> many contained items" shape:
//
// - A single run may contain zero, one, or many alerts.
// - Runs are intended to be immutable snapshots (“provider asserted X at AsOf”).
//
// Normalizers should prefer to set AsOf from a provider-supplied “updated/generated” timestamp.
// If unavailable, AsOf may be set to the poll/emit time as a fallback.
type WeatherAlertRun struct {
// Optional location metadata (provider-dependent).
LocationID string `json:"locationId,omitempty"`
LocationName string `json:"locationName,omitempty"`
// AsOf is when the provider asserted this alert snapshot is current (required).
AsOf time.Time `json:"asOf"`
// Optional spatial context.
Latitude *float64 `json:"latitude,omitempty"`
Longitude *float64 `json:"longitude,omitempty"`
// Active alerts contained in this snapshot (order is provider-dependent).
Alerts []WeatherAlert `json:"alerts"`
}
// WeatherAlert is the canonical representation of a single alert.
//
// This is intentionally a “useful subset” of rich provider payloads.
// Normalizers may populate ProviderExtras for structured provider-specific fields
// that dont cleanly fit the canonical shape.
type WeatherAlert struct {
// Provider-stable identifier (often a URL/URI).
ID string `json:"id"`
// Classification / headline fields.
Event string `json:"event,omitempty"`
Headline string `json:"headline,omitempty"`
Severity string `json:"severity,omitempty"` // e.g. Extreme/Severe/Moderate/Minor/Unknown
Urgency string `json:"urgency,omitempty"` // e.g. Immediate/Expected/Future/Past/Unknown
Certainty string `json:"certainty,omitempty"` // e.g. Observed/Likely/Possible/Unlikely/Unknown
Status string `json:"status,omitempty"` // e.g. Actual/Exercise/Test/System/Unknown
MessageType string `json:"messageType,omitempty"` // e.g. Alert/Update/Cancel
Category string `json:"category,omitempty"` // e.g. Met/Geo/Safety/Rescue/Fire/Health/Env/Transport/Infra/CBRNE/Other
Response string `json:"response,omitempty"` // e.g. Shelter/Evacuate/Prepare/Execute/Avoid/Monitor/Assess/AllClear/None
// Narrative.
Description string `json:"description,omitempty"`
Instruction string `json:"instruction,omitempty"`
// Timing (all optional; provider-dependent).
Sent *time.Time `json:"sent,omitempty"`
Effective *time.Time `json:"effective,omitempty"`
Onset *time.Time `json:"onset,omitempty"`
Expires *time.Time `json:"expires,omitempty"`
// Scope / affected area.
AreaDescription string `json:"areaDescription,omitempty"` // often a provider string
// Provenance.
SenderName string `json:"senderName,omitempty"`
References []AlertReference `json:"references,omitempty"`
}
// AlertReference is a reference to a related alert (updates, replacements, etc.).
type AlertReference struct {
ID string `json:"id,omitempty"` // provider reference ID/URI
Identifier string `json:"identifier,omitempty"` // provider identifier string, if distinct
Sender string `json:"sender,omitempty"`
Sent *time.Time `json:"sent,omitempty"`
}

10
model/doc.go Normal file
View File

@@ -0,0 +1,10 @@
// FILE: internal/model/doc.go
// Package model defines weatherfeeder's canonical domain payload types.
//
// These structs are emitted as the Payload of canonical events (schemas "weather.*.vN").
// JSON tags are treated as part of the wire contract for sinks (stdout today; others later).
//
// Compatibility guidance:
// - Prefer additive changes.
// - Avoid renaming/removing fields without a schema version bump.
package model

118
model/forecast.go Normal file
View File

@@ -0,0 +1,118 @@
// FILE: internal/model/forecast.go
package model
import "time"
// ForecastProduct distinguishes *what kind* of forecast a provider is offering.
// This is intentionally canonical and not provider-nomenclature.
type ForecastProduct string
const (
// ForecastProductHourly is a sub-daily forecast where each period is typically ~1 hour.
ForecastProductHourly ForecastProduct = "hourly"
// ForecastProductNarrative is a human-oriented sequence of periods like
// "Tonight", "Friday", "Friday Night" with variable duration.
//
// (NWS "/forecast" looks like this; it is not strictly “daily”.)
ForecastProductNarrative ForecastProduct = "narrative"
// ForecastProductDaily is a calendar-day (or day-bucketed) forecast where a period
// commonly carries min/max values (Open-Meteo daily, many others).
ForecastProductDaily ForecastProduct = "daily"
)
// WeatherForecastRun is a single issued forecast snapshot for a location and product.
//
// Design goals:
// - Immutable snapshot semantics: “provider asserted X at IssuedAt”.
// - Provider-independent schema: normalize many upstreams into one shape.
// - Retrieval-friendly: periods are inside the run, but can be stored/indexed separately.
type WeatherForecastRun struct {
// Identity / metadata (aligned with WeatherObservations StationID/StationName/Timestamp).
LocationID string `json:"locationId,omitempty"`
LocationName string `json:"locationName,omitempty"`
IssuedAt time.Time `json:"issuedAt"` // required: when this run was generated/issued
// Some providers include both a generated time and a later update time.
// Keep UpdatedAt optional; many providers wont supply it.
UpdatedAt *time.Time `json:"updatedAt,omitempty"`
// What sort of forecast this run represents (hourly vs narrative vs daily).
Product ForecastProduct `json:"product"`
// Optional spatial context. Many providers are fundamentally lat/lon-based.
Latitude *float64 `json:"latitude,omitempty"`
Longitude *float64 `json:"longitude,omitempty"`
// Kept to align with WeatherObservation and because elevation is sometimes important
// for interpreting temps/precip, even if not always supplied by the provider.
ElevationMeters *float64 `json:"elevationMeters,omitempty"`
// The forecast periods contained in this issued run. Order should be chronological.
Periods []WeatherForecastPeriod `json:"periods"`
}
// WeatherForecastPeriod is a forecast for a specific valid interval [StartTime, EndTime).
//
// Conceptually, it mirrors WeatherObservation (condition + measurements), but:
// / - It has a time *range* (start/end) instead of a single timestamp.
// / - It adds forecast-specific fields like probability/amount of precip.
// / - It supports min/max for “daily” products (and other aggregated periods).
type WeatherForecastPeriod struct {
// Identity / validity window (required)
StartTime time.Time `json:"startTime"`
EndTime time.Time `json:"endTime"`
// Human-facing period label (e.g., "Tonight", "Friday"). Often empty for hourly.
Name string `json:"name,omitempty"`
// Canonical day/night hint (aligned with WeatherObservation.IsDay).
// Providers vary in whether they explicitly include this.
IsDay *bool `json:"isDay,omitempty"`
// Canonical internal representation (provider-independent).
// Like WeatherObservation, this is required; use an “unknown” WMOCode if unmappable.
ConditionCode WMOCode `json:"conditionCode"`
// Provider-independent short text describing the conditions (normalized, if possible).
ConditionText string `json:"conditionText,omitempty"`
// Provider-specific “evidence” for troubleshooting mapping and drift.
ProviderRawDescription string `json:"providerRawDescription,omitempty"`
// Human-facing narrative. Not all providers supply rich text (Open-Meteo often wont).
TextDescription string `json:"textDescription,omitempty"` // short phrase / summary
DetailedText string `json:"detailedText,omitempty"` // longer narrative, if available
// Provider-specific (legacy / transitional)
IconURL string `json:"iconUrl,omitempty"`
// Core predicted measurements (nullable; units align with WeatherObservation)
TemperatureC *float64 `json:"temperatureC,omitempty"`
// For aggregated products (notably “daily”), providers may supply min/max.
TemperatureCMin *float64 `json:"temperatureCMin,omitempty"`
TemperatureCMax *float64 `json:"temperatureCMax,omitempty"`
DewpointC *float64 `json:"dewpointC,omitempty"`
RelativeHumidityPercent *float64 `json:"relativeHumidityPercent,omitempty"`
WindDirectionDegrees *float64 `json:"windDirectionDegrees,omitempty"`
WindSpeedKmh *float64 `json:"windSpeedKmh,omitempty"`
WindGustKmh *float64 `json:"windGustKmh,omitempty"`
BarometricPressurePa *float64 `json:"barometricPressurePa,omitempty"`
VisibilityMeters *float64 `json:"visibilityMeters,omitempty"`
ApparentTemperatureC *float64 `json:"apparentTemperatureC,omitempty"`
CloudCoverPercent *float64 `json:"cloudCoverPercent,omitempty"`
// Precipitation (forecast-specific). Keep these generic and provider-independent.
ProbabilityOfPrecipitationPercent *float64 `json:"probabilityOfPrecipitationPercent,omitempty"`
// Quantitative precip is not universally available, but OpenWeather/Open-Meteo often supply it.
// Use liquid-equivalent mm for interoperability.
PrecipitationAmountMm *float64 `json:"precipitationAmountMm,omitempty"`
SnowfallDepthMM *float64 `json:"snowfallDepthMm,omitempty"`
// Optional extras that some providers supply and downstream might care about.
UVIndex *float64 `json:"uvIndex,omitempty"`
}

55
model/observation.go Normal file
View File

@@ -0,0 +1,55 @@
// FILE: internal/model/observation.go
package model
import "time"
type WeatherObservation struct {
// Identity / metadata
StationID string `json:"stationId,omitempty"`
StationName string `json:"stationName,omitempty"`
Timestamp time.Time `json:"timestamp"`
// Canonical internal representation (provider-independent).
ConditionCode WMOCode `json:"conditionCode"`
ConditionText string `json:"conditionText,omitempty"`
IsDay *bool `json:"isDay,omitempty"`
// Provider-specific “evidence” for troubleshooting mapping and drift.
ProviderRawDescription string `json:"providerRawDescription,omitempty"`
// Human-facing (legacy / transitional)
TextDescription string `json:"textDescription,omitempty"`
// Provider-specific (legacy / transitional)
IconURL string `json:"iconUrl,omitempty"`
// Core measurements (nullable)
TemperatureC *float64 `json:"temperatureC,omitempty"`
DewpointC *float64 `json:"dewpointC,omitempty"`
WindDirectionDegrees *float64 `json:"windDirectionDegrees,omitempty"`
WindSpeedKmh *float64 `json:"windSpeedKmh,omitempty"`
WindGustKmh *float64 `json:"windGustKmh,omitempty"`
BarometricPressurePa *float64 `json:"barometricPressurePa,omitempty"`
SeaLevelPressurePa *float64 `json:"seaLevelPressurePa,omitempty"`
VisibilityMeters *float64 `json:"visibilityMeters,omitempty"`
RelativeHumidityPercent *float64 `json:"relativeHumidityPercent,omitempty"`
ApparentTemperatureC *float64 `json:"apparentTemperatureC,omitempty"`
ElevationMeters *float64 `json:"elevationMeters,omitempty"`
RawMessage string `json:"rawMessage,omitempty"`
PresentWeather []PresentWeather `json:"presentWeather,omitempty"`
CloudLayers []CloudLayer `json:"cloudLayers,omitempty"`
}
type CloudLayer struct {
BaseMeters *float64 `json:"baseMeters,omitempty"`
Amount string `json:"amount,omitempty"`
}
type PresentWeather struct {
Raw map[string]any `json:"raw,omitempty"`
}

View File

@@ -5,10 +5,19 @@
// - Schema identifiers and versioning conventions (see schema.go). // - Schema identifiers and versioning conventions (see schema.go).
// - Canonical interpretations / cross-provider mappings that are not specific to a // - Canonical interpretations / cross-provider mappings that are not specific to a
// single upstream API (e.g., shared code tables, text heuristics, unit policy). // single upstream API (e.g., shared code tables, text heuristics, unit policy).
// - Wire-format conventions for canonical payloads.
// //
// Standards are used by both sources and normalizers. Keep this package free of // Standards are used by both sources and normalizers. Keep this package free of
// provider-specific logic and free of dependencies on internal/sources/* or // provider-specific logic and free of dependencies on internal/sources/* or
// internal/normalizers/* to avoid import cycles. // internal/normalizers/* to avoid import cycles.
// //
// Provider-specific decoding and mapping lives in internal/normalizers/<provider>. // Wire-format conventions
// -----------------------
// For readability and stability, canonical payloads (weather.* schemas) should not emit
// noisy floating-point representations. weatherfeeder enforces this by rounding float
// values in canonical payloads to 4 digits after the decimal point at normalization
// finalization time.
//
// Provider-specific decoding helpers and quirks live in internal/providers/<provider>.
// Normalizer implementations and canonical mapping logic live in internal/normalizers/<provider>.
package standards package standards

View File

@@ -15,6 +15,14 @@ const (
SchemaRawOpenMeteoCurrentV1 = "raw.openmeteo.current.v1" SchemaRawOpenMeteoCurrentV1 = "raw.openmeteo.current.v1"
SchemaRawOpenWeatherCurrentV1 = "raw.openweather.current.v1" SchemaRawOpenWeatherCurrentV1 = "raw.openweather.current.v1"
SchemaRawNWSHourlyForecastV1 = "raw.nws.hourly.forecast.v1"
SchemaRawOpenMeteoHourlyForecastV1 = "raw.openmeteo.hourly.forecast.v1"
SchemaRawOpenWeatherHourlyForecastV1 = "raw.openweather.hourly.forecast.v1"
SchemaRawNWSAlertsV1 = "raw.nws.alerts.v1"
// Canonical domain schemas (emitted after normalization). // Canonical domain schemas (emitted after normalization).
SchemaWeatherObservationV1 = "weather.observation.v1" SchemaWeatherObservationV1 = "weather.observation.v1"
SchemaWeatherForecastV1 = "weather.forecast.v1"
SchemaWeatherAlertV1 = "weather.alert.v1"
) )

View File

@@ -1,9 +1,18 @@
package standards package standards
// This file provides small, shared helper functions for reasoning about WMO codes.
// These are intentionally "coarse" categories that are useful for business logic,
// dashboards, and alerting decisions.
//
// Example uses:
// - jogging suitability: precipitation? thunderstorm? freezing precip?
// - quick glance: "is it cloudy?" "is there any precip?"
// - downstream normalizers / aggregators
import ( import (
"fmt" "fmt"
"gitea.maximumdirect.net/ejr/weatherfeeder/internal/model" "gitea.maximumdirect.net/ejr/weatherfeeder/model"
) )
type WMODescription struct { type WMODescription struct {
@@ -100,3 +109,118 @@ func IsKnownWMO(code model.WMOCode) bool {
_, ok := WMODescriptions[code] _, ok := WMODescriptions[code]
return ok return ok
} }
func IsThunderstorm(code model.WMOCode) bool {
switch code {
case 95, 96, 99:
return true
default:
return false
}
}
func IsHail(code model.WMOCode) bool {
switch code {
case 96, 99:
return true
default:
return false
}
}
func IsFog(code model.WMOCode) bool {
switch code {
case 45, 48:
return true
default:
return false
}
}
// IsPrecipitation returns true if the code represents any precipitation
// (drizzle, rain, snow, showers, etc.).
func IsPrecipitation(code model.WMOCode) bool {
switch code {
// Drizzle
case 51, 53, 55, 56, 57:
return true
// Rain
case 61, 63, 65, 66, 67:
return true
// Snow
case 71, 73, 75, 77:
return true
// Showers
case 80, 81, 82, 85, 86:
return true
// Thunderstorm (often includes rain/hail)
case 95, 96, 99:
return true
default:
return false
}
}
func IsRainFamily(code model.WMOCode) bool {
switch code {
// Drizzle + freezing drizzle
case 51, 53, 55, 56, 57:
return true
// Rain + freezing rain
case 61, 63, 65, 66, 67:
return true
// Rain showers
case 80, 81, 82:
return true
// Thunderstorm often implies rain
case 95, 96, 99:
return true
default:
return false
}
}
func IsSnowFamily(code model.WMOCode) bool {
switch code {
// Snow and related
case 71, 73, 75, 77:
return true
// Snow showers
case 85, 86:
return true
default:
return false
}
}
// IsFreezingPrecip returns true if the code represents freezing drizzle/rain.
func IsFreezingPrecip(code model.WMOCode) bool {
switch code {
case 56, 57, 66, 67:
return true
default:
return false
}
}
// IsSkyOnly returns true for codes that represent "sky condition only"
// (clear/mostly/partly/cloudy) rather than fog/precip/etc.
func IsSkyOnly(code model.WMOCode) bool {
switch code {
case 0, 1, 2, 3:
return true
default:
return false
}
}