diff --git a/README.md b/README.md index 71a13ce..9e87768 100644 --- a/README.md +++ b/README.md @@ -1,36 +1,30 @@ # weatherfeeder -weatherfeeder is a small daemon that polls weather observations, forecasts, and alerts from multiple upstream -providers, normalizes them into a provider-independent format, and emits them to a sink. +`weatherfeeder` is a config-driven daemon that polls weather providers, normalizes +provider-specific responses into canonical weather events, and routes those +events to configured sinks. -Today, the only implemented sink is `stdout`, which prints JSON-encoded events. +It currently supports NWS observations, alerts, hourly forecasts, narrative +forecasts, forecast discussions, and weather stories; Open-Meteo observations +and hourly forecasts; and OpenWeather observations. Implemented sinks are +stdout, NATS, and Postgres. -## What weatherfeeder emits +## Quickstart -weatherfeeder emits **feed events** encoded as JSON. Each event includes a schema identifier and a payload. -Downstream consumers should key off the `schema` value and decode the `payload` accordingly. +Run the checked-in sample config: -Canonical domain schemas emitted after normalization: +```sh +cd cmd/weatherfeeder +go run . +``` -- `weather.observation.v1` → `WeatherObservation` -- `weather.forecast.v1` → `WeatherForecastRun` -- `weather.forecast_discussion.v1` → `WeatherForecastDiscussion` -- `weather.weather_story.v1` → `WeatherStoryRun` -- `weather.alert.v1` → `WeatherAlertRun` +The sample config at `cmd/weatherfeeder/config.yml` is load-tested and can be +used as a starting point. The executable always reads `config.yml` from its +current working directory. -For the complete wire contract (event envelope + payload schemas, fields, units, and compatibility rules), see: +## Documentation -- **API.md** - -## Upstream providers (current MVP) - -- NWS: observations, hourly forecasts, narrative forecasts, forecast discussions, weather stories, alerts -- Open-Meteo: observations, hourly forecasts -- OpenWeather: observations - -## Versioning & compatibility - -The JSON field names on canonical payload types are treated as part of the wire contract. -Additive changes are preferred. Renames/removals require a schema version bump. - -See **API.md** for details. +- [CLI reference](docs/cli.md) +- [Configuration reference](docs/config.md) +- [Architecture policy](docs/policy/architecture.md) +- [Documentation policy](docs/policy/documentation.md) diff --git a/docs/cli.md b/docs/cli.md new file mode 100644 index 0000000..8597d48 --- /dev/null +++ b/docs/cli.md @@ -0,0 +1,70 @@ +# CLI Reference + +## Shortest Useful Command + +Run `weatherfeeder` from a directory containing `config.yml`: + +```sh +cd cmd/weatherfeeder +go run . +``` + +When using a built binary: + +```sh +./weatherfeeder +``` + +## Command Overview + +`weatherfeeder` starts a long-running polling daemon. On startup it: + +1. reads `config.yml` from the current working directory; +2. builds configured sources, sinks, and routes; +3. starts polling sources on their configured intervals; +4. normalizes and deduplicates events; +5. dispatches matching events to configured sinks. + +The command logs startup, runtime, and shutdown messages to stderr using the Go +standard logger. + +## Flags + +There are currently no CLI flags, subcommands, or environment-variable based +configuration controls. + +The config path is fixed at `config.yml` relative to the process current working +directory. To run with a different config, change the working directory or place +the desired file at that path. + +## Common Workflows + +Run the checked-in sample config: + +```sh +cd cmd/weatherfeeder +go run . +``` + +Build and run a local binary: + +```sh +go build -o weatherfeeder ./cmd/weatherfeeder +cp cmd/weatherfeeder/config.yml . +./weatherfeeder +``` + +Run in the project container image with a mounted config: + +```sh +docker run --rm -v "$PWD/config.yml:/weatherfeeder/config.yml:ro" weatherfeeder +``` + +The Docker image sets `/weatherfeeder` as the working directory, so the mounted +file must appear at `/weatherfeeder/config.yml`. + +## Shutdown + +Stop the daemon with `Ctrl-C` or `SIGTERM`. The process uses context-aware +shutdown for scheduler, dispatcher, processors, sources, and sinks, then logs +`shutdown complete`. diff --git a/docs/config.md b/docs/config.md new file mode 100644 index 0000000..4bfb897 --- /dev/null +++ b/docs/config.md @@ -0,0 +1,214 @@ +# Configuration Reference + +## Config File + +`weatherfeeder` reads exactly one YAML file named `config.yml` from the current +working directory. There is no config path flag and no search path. + +YAML decoding is strict for config struct fields: misspelled fields such as +`sources[].drviver` fail startup. Driver-specific `params` maps are validated by +the source or sink constructor that consumes them. + +The top-level file contains: + +```yaml +sources: + - name: NWSObservationKSTL + mode: poll + driver: nws_observation + every: 10m + kinds: ["observation"] + params: + url: "https://api.weather.gov/stations/KSTL/observations/latest" + user_agent: "Example weatherfeeder operator (ops@example.com)" + +sinks: + - name: stdout + driver: stdout + params: {} + +routes: + - sink: stdout + kinds: ["observation"] +``` + +`sources` and `sinks` must each contain at least one entry. `routes` is optional. +When `routes` is omitted, every configured sink receives every event kind. + +## Production-Oriented Shape + +A typical deployment uses multiple polling sources and sends the same canonical +event stream to a broker or database: + +```yaml +sources: + - name: NWSAlertsLocal + mode: poll + driver: nws_alerts + every: 1m + kinds: ["alert"] + params: + url: "https://api.weather.gov/alerts?point=38.6239,-90.3571&limit=20" + user_agent: "Example weatherfeeder operator (ops@example.com)" + +sinks: + - name: nats_weather + driver: nats + params: + url: nats://nats:4222 + subject: weatherfeeder + + - name: pg_weather + driver: postgres + params: + uri: postgres://weatherdb:5432/weatherdb?sslmode=disable + username: weatherdb + password: "" + prune: 3d + +routes: + - sink: nats_weather + kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] + + - sink: pg_weather + kinds: ["observation", "forecast", "forecast_discussion", "weather_story", "alert"] +``` + +Do not commit real API keys, database passwords, or personal contact addresses in +copyable configs. + +## Top-Level Fields + +| Field | Required | Description | +|---|:---:|---| +| `sources` | yes | List of configured input sources. | +| `sinks` | yes | List of configured output sinks. | +| `routes` | no | List of sink routing rules. If omitted, all sinks receive all kinds. | + +## Source Fields + +| Field | Required | Description | +|---|:---:|---| +| `name` | yes | Unique source name. Used as the event source identifier. | +| `driver` | yes | Source driver name. | +| `mode` | no | `poll`, `stream`, or omitted for auto. Current weatherfeeder drivers are polling drivers. | +| `every` | yes | Poll interval for current weatherfeeder source drivers. | +| `kinds` | no | Expected event kinds. If present, startup verifies they match the source driver. | +| `params` | driver-specific | Driver parameters. Current source drivers require HTTP params. | + +Current event kinds are `observation`, `forecast`, `forecast_discussion`, +`weather_story`, and `alert`. + +## Source Drivers + +| Driver | Kind | Upstream product | +|---|---|---| +| `nws_observation` | `observation` | NWS station latest observation. | +| `nws_alerts` | `alert` | NWS alerts collection. | +| `nws_forecast_hourly` | `forecast` | NWS hourly gridpoint forecast. | +| `nws_forecast_narrative` | `forecast` | NWS narrative gridpoint forecast. | +| `nws_forecast_discussion` | `forecast_discussion` | NWS forecast discussion HTML product. | +| `nws_weatherstories` | `weather_story` | NWS office weather stories. | +| `openmeteo_observation` | `observation` | Open-Meteo current conditions. | +| `openmeteo_forecast` | `forecast` | Open-Meteo hourly forecast. | +| `openweather_observation` | `observation` | OpenWeather current weather. | + +## HTTP Source Params + +All current source drivers use the shared HTTP polling helper. + +| Param | Required | Description | +|---|:---:|---| +| `url` | yes | Full upstream request URL. `URL` is also accepted by the helper. | +| `user_agent` | yes | User-Agent sent to the upstream provider. `userAgent` is also accepted by the helper. | +| `conditional` | no | Boolean. Defaults to `true`; enables ETag and Last-Modified conditional requests. | +| `http_timeout` | no | Positive duration for the HTTP client timeout. | +| `http_response_body_limit_bytes` | no | Positive integer response body limit in bytes. | + +When `conditional` is enabled and the upstream returns `304 Not Modified`, the +source emits no events for that poll. + +OpenWeather observation URLs must include `units=metric`. Startup fails if the +URL omits it or sets another unit system. + +## Sink Fields + +| Field | Required | Description | +|---|:---:|---| +| `name` | yes | Unique sink name. Routes refer to this value. | +| `driver` | yes | Sink driver name. | +| `params` | driver-specific | Sink parameters. | + +## Sink Drivers + +### `stdout` + +Prints each event as JSON to stdout. + +```yaml +sinks: + - name: stdout + driver: stdout + params: {} +``` + +### `nats` + +Publishes each event as JSON to a NATS subject. + +| Param | Required | Description | +|---|:---:|---| +| `url` | yes | NATS server URL, such as `nats://localhost:4222`. | +| `subject` | yes | Subject to publish events to. | + +### `postgres` + +Writes supported canonical weather events to Postgres using weatherfeeder's +registered schema mapping. + +| Param | Required | Description | +|---|:---:|---| +| `uri` | yes | PostgreSQL connection URI. | +| `username` | yes | Database username. | +| `password` | yes | Database password. | +| `prune` | no | Retention window. If set, rows older than the window are pruned on each write transaction. | + +`prune` accepts Go duration strings such as `72h`, plus day and week suffixes +such as `3d` and `2w`. + +## Routes + +Routes connect event kinds to sinks: + +```yaml +routes: + - sink: stdout + kinds: ["observation", "alert"] +``` + +| Field | Required | Description | +|---|:---:|---| +| `sink` | yes | Name of a configured sink. | +| `kinds` | no | Event kinds to send to that sink. Omit or use an empty list to match all kinds. | + +Route `kinds` values are trimmed and lowercased by the dispatcher. Blank entries +are rejected. + +## Duration Formats + +Top-level source `every` accepts: + +- Go duration strings such as `30s`, `10m`, or `1h`; +- integer values, interpreted as minutes; +- numeric strings such as `"15"`, also interpreted as minutes. + +HTTP param durations such as `http_timeout` accept Go duration strings. Numeric +values and numeric strings are interpreted as seconds. + +Postgres `prune` must be a string duration. + +## Secrets + +The config file is read directly from disk and has no built-in secret expansion. +Keep real credentials out of repository-tracked configs. Use deployment tooling +to render `config.yml` with the needed secret values before starting the daemon.