weatherfeeder/docs/operations.md

Operations

This document describes how to run and observe the weatherfeeder daemon in its current form. For configuration syntax, see configuration. For the CLI surface, see CLI reference.

Normal Workflow

1. Prepare config.yml in the process working directory.
2. Start the daemon with ./weatherfeeder or go run . from cmd/weatherfeeder.
3. Watch stderr logs for startup or runtime errors.
4. Consume events from the configured sinks.
5. Stop the process with Ctrl-C or SIGTERM.

The daemon has no admin subcommands and no runtime reload command. Change the config file and restart the process to apply configuration changes.

Maintained copyable configs are available under examples/.

Runtime Lifecycle

On startup, weatherfeeder:

1. loads config.yml from the current working directory;
2. registers built-in source drivers;
3. registers stdout, NATS, and weatherfeeder Postgres sink drivers;
4. builds sources and validates configured kinds against source metadata;
5. builds sinks and compiles routes;
6. starts the scheduler and dispatcher;
7. processes events through normalization, then in-memory dedupe;
8. routes processed events to configured sinks.

Startup errors are fatal and terminate the process. Runtime poll, pipeline, and sink errors are logged and the process continues unless the scheduler or dispatcher returns a fatal error.

Logs

The process uses the Go standard logger with date, time, and microseconds. Logs go to stderr.

Common log prefixes:

Prefix	Meaning
config load failed	config.yml could not be read, parsed, or validated.
build source failed	A source driver or its params are invalid.
source expected kinds validation failed	Configured source kinds do not match the source driver.
build sink failed	A sink driver or its params are invalid, or a sink could not initialize.
compile routes failed	Routes reference invalid sinks or kinds.
scheduler: poll failed	A source poll failed; the source will be polled again on its next interval.
dispatcher: pipeline error	Normalization or dedupe failed for one event.
dispatch: sink ... failed consuming event	A sink failed to consume one event.
shutdown complete	Scheduler and dispatcher have exited.

Scheduling And Polling

Current weatherfeeder sources are polling sources. Each source uses its configured every interval. The scheduler applies jitter before the first poll and before each interval tick. If no jitter is configured in code, feedkit uses min(every/10, 30s), capped at half the interval.

Poll failures are logged and do not stop the daemon. A failed poll emits no events for that source until a subsequent poll succeeds.

Unchanged Upstream Content

Most sources use feedkit's single-document HTTP polling helper. By default, params.conditional is true for those sources, so the helper keeps ETag and Last-Modified validators in memory for each source instance.

If the upstream returns 304 Not Modified, the source emits no events for that poll. Validator state is in memory only; restarting the process starts with no cached validators.

The SPC convective outlook source polls multiple required documents as one bundle. It emits no event when a later complete bundle has the same fetched document bodies as the previous successful poll.

Processing And Dedupe

Every event passes through normalization first and dedupe second.

Normalizers match raw source schemas and produce canonical weather.*.v1 payloads. If an event has no matching normalizer, the normalize processor passes it through unchanged.

Dedupe keys by event ID and stores a bounded in-memory set of 2048 recent IDs. Duplicate IDs are dropped. Dedupe state is not persisted, so a restart starts with an empty dedupe set.

Routing And Sink Fanout

Routes choose sinks by event kind. If routes is omitted, every sink receives every event kind. If a route omits kinds, that route also matches all kinds.

The dispatcher creates one queue and one worker goroutine per sink. The default per-sink queue size is 64. weatherfeeder does not currently expose config fields for sink queue size, enqueue timeout, or consume timeout.

Sink errors are logged per event. A sink failure does not stop other sinks from receiving the same event.

Sink Behavior

stdout

The stdout sink validates each event and prints one JSON object per line to stdout. This is useful for local inspection and log forwarding.

NATS

The NATS sink connects lazily on the first event, reuses the connection while it is open, and publishes each event as JSON to the configured subject. Connection, marshal, and publish failures are logged by the dispatch worker.

Postgres

The Postgres sink opens the database during startup. It creates missing tables and indexes with CREATE TABLE IF NOT EXISTS and CREATE INDEX IF NOT EXISTS. It does not modify existing table definitions.

Each mapped canonical event is written in one transaction. If params.prune is set, the sink deletes rows older than now - prune from every weatherfeeder table in that same transaction. See the Postgres table contract.

State And Recovery

weatherfeeder keeps only runtime state in process memory:

• scheduler goroutines and timers;
• HTTP conditional request validators;
• source-local unchanged-content state;
• event channel buffers;
• per-sink fanout queues;
• the dedupe ID set.

Durable state is external sink state: NATS broker state outside this process and Postgres tables managed by the configured database.

There is no internal checkpoint, replay log, or resume marker. To recover from a process failure, fix the underlying issue and restart the daemon from a working directory containing the desired config.yml.

Shutdown

weatherfeeder listens for os.Interrupt and SIGTERM. On shutdown, the shared context is canceled. Scheduler jobs stop polling, dispatch workers stop, and the process logs shutdown complete.

Queued sink work may be dropped when shutdown context cancellation reaches the fanout workers. Use external sink durability, such as Postgres or broker retention, for durable downstream state.

Caveats

• There is no health-check endpoint.
• There is no runtime config reload.
• There are no built-in metrics.
• Source conditional request state and dedupe state are reset by restart.
• Existing Postgres schemas are not migrated automatically.