diff --git a/README.md b/README.md index a7009f6..40e7603 100644 --- a/README.md +++ b/README.md @@ -26,6 +26,8 @@ current working directory. - [CLI reference](docs/cli.md) - [Configuration reference](docs/config.md) +- [Operations guide](docs/operations.md) +- [Troubleshooting guide](docs/troubleshooting.md) - [Event wire contract](docs/integrations/events.md) - [Postgres table contract](docs/integrations/postgres.md) - [Architecture policy](docs/policy/architecture.md) diff --git a/docs/operations.md b/docs/operations.md new file mode 100644 index 0000000..ab639a7 --- /dev/null +++ b/docs/operations.md @@ -0,0 +1,156 @@ +# Operations + +This document describes how to run and observe the `weatherfeeder` daemon in its +current form. For configuration syntax, see [configuration](config.md). For the +CLI surface, see [CLI reference](cli.md). + +## Normal Workflow + +1. Prepare `config.yml` in the process working directory. +2. Start the daemon with `./weatherfeeder` or `go run .` from + `cmd/weatherfeeder`. +3. Watch stderr logs for startup or runtime errors. +4. Consume events from the configured sinks. +5. Stop the process with `Ctrl-C` or `SIGTERM`. + +The daemon has no admin subcommands and no runtime reload command. Change the +config file and restart the process to apply configuration changes. + +## Runtime Lifecycle + +On startup, `weatherfeeder`: + +1. loads `config.yml` from the current working directory; +2. registers built-in source drivers; +3. registers stdout, NATS, and weatherfeeder Postgres sink drivers; +4. builds sources and validates configured `kinds` against source metadata; +5. builds sinks and compiles routes; +6. starts the scheduler and dispatcher; +7. processes events through normalization, then in-memory dedupe; +8. routes processed events to configured sinks. + +Startup errors are fatal and terminate the process. Runtime poll, pipeline, and +sink errors are logged and the process continues unless the scheduler or +dispatcher returns a fatal error. + +## Logs + +The process uses the Go standard logger with date, time, and microseconds. Logs +go to stderr. + +Common log prefixes: + +| Prefix | Meaning | +|---|---| +| `config load failed` | `config.yml` could not be read, parsed, or validated. | +| `build source failed` | A source driver or its params are invalid. | +| `source expected kinds validation failed` | Configured source `kinds` do not match the source driver. | +| `build sink failed` | A sink driver or its params are invalid, or a sink could not initialize. | +| `compile routes failed` | Routes reference invalid sinks or kinds. | +| `scheduler: poll failed` | A source poll failed; the source will be polled again on its next interval. | +| `dispatcher: pipeline error` | Normalization or dedupe failed for one event. | +| `dispatch: sink ... failed consuming event` | A sink failed to consume one event. | +| `shutdown complete` | Scheduler and dispatcher have exited. | + +## Scheduling And Polling + +Current weatherfeeder sources are polling sources. Each source uses its +configured `every` interval. The scheduler applies jitter before the first poll +and before each interval tick. If no jitter is configured in code, feedkit uses +`min(every/10, 30s)`, capped at half the interval. + +Poll failures are logged and do not stop the daemon. A failed poll emits no +events for that source until a later poll succeeds. + +## Conditional HTTP Fetches + +All current sources use feedkit's HTTP polling helper. By default, +`params.conditional` is `true`, so the helper keeps ETag and Last-Modified +validators in memory for each source instance. + +If the upstream returns `304 Not Modified`, the source emits no events for that +poll. Validator state is in memory only; restarting the process starts with no +cached validators. + +## Processing And Dedupe + +Every event passes through normalization first and dedupe second. + +Normalizers match raw source schemas and produce canonical `weather.*.v1` +payloads. If an event has no matching normalizer, the normalize processor passes +it through unchanged. + +Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs. +Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts +with an empty dedupe set. + +## Routing And Sink Fanout + +Routes choose sinks by event kind. If `routes` is omitted, every sink receives +every event kind. If a route omits `kinds`, that route also matches all kinds. + +The dispatcher creates one queue and one worker goroutine per sink. The default +per-sink queue size is 64. `weatherfeeder` does not currently expose config +fields for sink queue size, enqueue timeout, or consume timeout. + +Sink errors are logged per event. A sink failure does not stop other sinks from +receiving the same event. + +## Sink Behavior + +### stdout + +The stdout sink validates each event and prints one JSON object per line to +stdout. This is useful for local inspection and log forwarding. + +### NATS + +The NATS sink connects lazily on the first event, reuses the connection while it +is open, and publishes each event as JSON to the configured subject. Connection, +marshal, and publish failures are logged by the dispatch worker. + +### Postgres + +The Postgres sink opens the database during startup. It creates missing tables +and indexes with `CREATE TABLE IF NOT EXISTS` and `CREATE INDEX IF NOT EXISTS`. +It does not modify existing table definitions. + +Each mapped canonical event is written in one transaction. If `params.prune` is +set, the sink deletes rows older than `now - prune` from every weatherfeeder +table in that same transaction. See the +[Postgres table contract](integrations/postgres.md). + +## State And Recovery + +`weatherfeeder` keeps only runtime state in process memory: + +- scheduler goroutines and timers; +- HTTP conditional request validators; +- event channel buffers; +- per-sink fanout queues; +- the dedupe ID set. + +Durable state is external sink state: NATS broker state outside this process and +Postgres tables managed by the configured database. + +There is no internal checkpoint, replay log, or resume marker. To recover from a +process failure, fix the underlying issue and restart the daemon from a working +directory containing the desired `config.yml`. + +## Shutdown + +`weatherfeeder` listens for `os.Interrupt` and `SIGTERM`. On shutdown, the +shared context is canceled. Scheduler jobs stop polling, dispatch workers stop, +and the process logs `shutdown complete`. + +Queued sink work may be dropped when shutdown context cancellation reaches the +fanout workers. Use external sink durability, such as Postgres or broker +retention, for durable downstream state. + +## Caveats + +- There is no health-check endpoint. +- There is no runtime config reload. +- There are no built-in metrics. +- Source conditional request state and dedupe state are reset by restart. +- Existing Postgres schemas are not migrated automatically. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..1b283ce --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,236 @@ +# Troubleshooting + +Use this guide with [configuration](config.md) and [operations](operations.md). +Messages are emitted through the standard logger on stderr. + +## `config load failed: ... read "config.yml"` + +Symptom: startup exits before building sources or sinks. + +Likely cause: the process working directory does not contain `config.yml`, or +the runtime user cannot read it. + +Diagnostic: run `pwd` in the same working directory used by the process, then +check `ls -l config.yml`. + +Safe fix: place the intended config at `./config.yml`, change the working +directory, or mount the file at `/weatherfeeder/config.yml` when using the +provided container image. + +## `config load failed: ... parse YAML` + +Symptom: startup exits with a YAML parse error or an unknown field error. + +Likely cause: invalid YAML syntax, multiple YAML documents, or a misspelled +config struct field. + +Diagnostic: inspect the line and field in the error. Feedkit uses strict YAML +field decoding for config struct fields. + +Safe fix: correct the YAML and compare the shape with +[configuration](config.md). Driver-specific `params` keys are validated later by +their source or sink constructors. + +## `config validation failed` + +Symptom: startup exits and prints one or more validation messages. + +Likely cause: missing `sources` or `sinks`, blank names, duplicate source or sink +names, invalid `mode`, missing `every` for a polling source with `mode: poll`, or +a route that references an unknown sink. + +Diagnostic: read every bullet under `config validation failed`; the loader sorts +these messages so multiple issues can be fixed in one edit. + +Safe fix: update the top-level config fields as documented in +[configuration](config.md). + +## `unknown source driver` + +Symptom: startup exits with `build source failed`. + +Likely cause: `sources[].driver` does not match a registered weatherfeeder +source driver. + +Diagnostic: compare the configured driver with the source driver table in +[configuration](config.md#source-drivers). + +Safe fix: correct the driver name. Current drivers include `nws_observation`, +`nws_alerts`, `nws_forecast_hourly`, `nws_forecast_narrative`, +`nws_forecast_discussion`, `nws_weatherstories`, `openmeteo_observation`, +`openmeteo_forecast`, and `openweather_observation`. + +## `unknown sink driver` + +Symptom: startup exits with `build sink failed`. + +Likely cause: `sinks[].driver` is not registered. + +Diagnostic: compare the configured driver with the sink driver table in +[configuration](config.md#sink-drivers). + +Safe fix: use `stdout`, `nats`, or `postgres`. + +## `source expected kinds validation failed` + +Symptom: startup exits after building a source. + +Likely cause: `sources[].kinds` declares a kind the source does not emit. + +Diagnostic: compare the configured `kinds` list with the source driver kind in +[configuration](config.md#source-drivers). + +Safe fix: remove `kinds` or set it to the kind emitted by that driver. + +## `params.url is required` Or `params.user_agent is required` + +Symptom: startup exits with `build source failed`. + +Likely cause: a source is missing required HTTP params, or the values are blank +or not strings. + +Diagnostic: inspect the named source in the error and check its `params`. + +Safe fix: add non-empty `url` and `user_agent` values. See +[HTTP source params](config.md#http-source-params). + +## `url must include units=metric` + +Symptom: startup exits for an `openweather_observation` source. + +Likely cause: the OpenWeather URL omits `units=metric` or sets another unit +system. + +Diagnostic: inspect the query string in `params.url`. + +Safe fix: add `units=metric` to the OpenWeather current-weather URL. Keep API +keys out of committed configs. + +## `source ... sources[].every must be > 0 for polling sources` + +Symptom: startup exits while building scheduler jobs. + +Likely cause: a current weatherfeeder polling source has no usable `every` +interval. + +Diagnostic: inspect the named `sources[]` entry and check `every`. + +Safe fix: set a positive duration such as `1m`, `10m`, or `1h`. + +## `build sink failed ... params.url is required` + +Symptom: startup exits while building a NATS sink. + +Likely cause: the NATS sink is missing `params.url`, or the value is blank or +not a string. + +Diagnostic: inspect the named sink in the error and check its `params`. + +Safe fix: set a NATS URL such as `nats://localhost:4222`. + +## `build sink failed ... params.subject is required` + +Symptom: startup exits while building a NATS sink. + +Likely cause: the NATS sink is missing `params.subject`, or the value is blank +or not a string. + +Diagnostic: inspect the named sink in the error and check its `params`. + +Safe fix: set a non-empty subject such as `weatherfeeder`. + +## `dispatch: sink ... failed consuming event ... NATS sink: connect` + +Symptom: the daemon starts, but NATS events are not published. + +Likely cause: the NATS server URL is unreachable, the server is not accepting +connections, or the configured URL is wrong for the runtime network. + +Diagnostic: from the same runtime environment, check that the host and port in +`sinks[].params.url` are reachable. + +Safe fix: correct the NATS URL or restore broker connectivity. Other configured +sinks continue receiving events. + +## `postgres sink ... open db` + +Symptom: startup exits while building a Postgres sink. + +Likely cause: the database URI, username, password, network path, or database +availability is wrong. + +Diagnostic: inspect `sinks[].params.uri`, `username`, and `password`; verify +that the same runtime environment can reach the database. + +Safe fix: correct the credentials or URI, restore database connectivity, then +restart the daemon. + +## `postgres sink ... ensure table` Or `ensure index` + +Symptom: startup exits during Postgres initialization. + +Likely cause: the database user cannot create required tables or indexes, an +existing object conflicts with weatherfeeder's expected table contract, or the +database is unavailable during initialization. + +Diagnostic: inspect the named table or index in the error and compare existing +database objects with the [Postgres table contract](integrations/postgres.md). + +Safe fix: grant the needed database privileges, create a compatible schema, or +perform an operator-managed migration before restarting. + +## `postgres sink: insert into ...` + +Symptom: the daemon starts, but Postgres writes for some events fail. + +Likely cause: a duplicate primary key, incompatible existing table definition, +database constraint error, or connection failure during a write transaction. + +Diagnostic: inspect the table name and database error in the log. Compare the +table with [Postgres integration](integrations/postgres.md). + +Safe fix: repair the database schema or address the duplicate/connection issue. +Other configured sinks continue receiving events. + +## No Events Appear On A Sink + +Symptom: the daemon is running but the expected sink receives no events. + +Likely cause: the route does not match the event kind, the source has not +emitted changed content, or the sink is failing per event. + +Diagnostic: check `routes`, source `kinds`, and logs for `scheduler: poll +failed`, `dispatcher: pipeline error`, or `dispatch: sink ... failed consuming +event`. + +Safe fix: correct the route or source configuration. If the source uses +conditional HTTP and the upstream has not changed, no event is emitted for a +`304 Not Modified` response; wait for changed upstream content or temporarily +set `params.conditional: false` for diagnosis. + +## `scheduler: poll failed` + +Symptom: one source logs poll failures while the daemon keeps running. + +Likely cause: upstream HTTP error, bad URL, timeout, response body limit, or +provider response shape that the source cannot parse. + +Diagnostic: inspect the source name in the log and review its HTTP params. + +Safe fix: correct the URL, user agent, timeout, or body limit. The next +scheduled poll will retry. + +## `dispatcher: pipeline error` + +Symptom: source polling succeeds, but one event is dropped before sinks. + +Likely cause: a normalizer could not decode or map the raw payload, or dedupe +received an invalid event ID. + +Diagnostic: inspect the error text and the source/schema that produced the +event. Review the [event wire contract](integrations/events.md) for expected +canonical fields. + +Safe fix: correct source configuration if it points to the wrong upstream +product. If the upstream payload changed shape, update the relevant normalizer +and tests.