weatherfeeder/docs/integrations/feedkit.md

Feedkit Integration

Purpose

This document describes the feedkit runtime behavior that weatherfeeder currently depends on. It is for maintainers and LLM coding agents changing runtime wiring, config behavior, source construction, processing, routing, or sink behavior.

Weather-domain behavior belongs in weatherfeeder. Generic daemon mechanics belong to feedkit.

Current Dependency

weatherfeeder imports feedkit as its daemon framework dependency. The exact module version is declared in go.mod.

Feedkit provides:

• YAML config loading and validation.
• Source, processor, and sink registries.
• HTTP source helper behavior.
• Scheduler polling.
• Normalize and dedupe processors.
• Route compilation and sink dispatch.
• Built-in stdout, NATS, and Postgres sink mechanics.

Config Contract

cmd/weatherfeeder calls feedkit config loading for config.yml in the current working directory.

Implemented behavior relied on by weatherfeeder docs and tests:

• Top-level config contains sources, sinks, and optional routes.
• Config struct fields are decoded strictly, so misspelled struct fields fail startup.
• Driver-specific params maps are decoded generically and validated by the source or sink constructor that consumes them.
• Source kinds can be validated against a source's advertised Kinds().

Source And HTTP Contract

Most weatherfeeder sources use feedkit's single-document HTTP source helper for:

• request construction;
• User-Agent and Accept headers;
• optional conditional GET validators;
• response body size limits;
• context-aware HTTP work;
• unchanged 304 Not Modified responses that emit no events.

The SPC convective outlook source fetches multiple documents itself, but it uses feedkit transport helpers for HTTP clients and response body limits.

Scheduler And Processing Contract

Weatherfeeder builds feedkit scheduler jobs from source configs. Current source drivers are polling drivers and use the configured every interval.

Events flow through a feedkit pipeline in this order:

1. normalize processor;
2. dedupe processor.

The normalize processor is configured with RequireMatch=false, so unmatched schemas pass through unchanged. Weatherfeeder registers its built-in normalizers and owns the provider-to-canonical mapping.

The dedupe processor stores a bounded in-memory set of recent event IDs. The bound is configured in cmd/weatherfeeder.

Dispatch And Sink Contract

Feedkit compiles routes from config and dispatches processed events to matching sinks. If routes is omitted, every configured sink receives every event kind.

Feedkit owns sink fanout mechanics, per-sink workers, queueing, context-aware shutdown, and sink error logging. Weatherfeeder owns the event kinds and schemas that make routes meaningful.

Built-in feedkit sinks used by weatherfeeder:

• stdout: validates and writes JSON events to stdout.
• nats: publishes JSON events to a configured subject.
• generic postgres sink factory: opens the database, ensures tables/indexes, runs transactions, inserts mapped rows, and prunes when configured.

Weatherfeeder supplies its Postgres table schema and event mapper to feedkit's Postgres sink factory. The table contract is documented in postgres.md.

Boundaries

Do not move weather-domain policy into feedkit. Weatherfeeder owns:

• provider source drivers;
• raw and canonical schema constants;
• event kind meaning;
• canonical payload structs;
• normalizers;
• Postgres table shape and row mapping.

Do not duplicate generic feedkit mechanics in weatherfeeder unless there is a narrow weather-specific reason. Runtime composition details are documented in ../internal/runtime.md.