Eric Rakestraw
8041f99782
All checks were successful
ci/woodpecker/push/build-image Pipeline was successful
107 lines
3.7 KiB
Markdown
107 lines
3.7 KiB
Markdown
# Feedkit Integration
|
|
|
|
## Purpose
|
|
|
|
This document describes the feedkit runtime behavior that `weatherfeeder`
|
|
currently depends on. It is for maintainers and LLM coding agents changing
|
|
runtime wiring, config behavior, source construction, processing, routing, or
|
|
sink behavior.
|
|
|
|
Weather-domain behavior belongs in `weatherfeeder`. Generic daemon mechanics
|
|
belong to feedkit.
|
|
|
|
## Current Dependency
|
|
|
|
`weatherfeeder` imports feedkit as its daemon framework dependency. The exact
|
|
module version is declared in `go.mod`.
|
|
|
|
Feedkit provides:
|
|
|
|
- YAML config loading and validation.
|
|
- Source, processor, and sink registries.
|
|
- HTTP source helper behavior.
|
|
- Scheduler polling.
|
|
- Normalize and dedupe processors.
|
|
- Route compilation and sink dispatch.
|
|
- Built-in stdout, NATS, and Postgres sink mechanics.
|
|
|
|
## Config Contract
|
|
|
|
`cmd/weatherfeeder` calls feedkit config loading for `config.yml` in the current
|
|
working directory.
|
|
|
|
Implemented behavior relied on by weatherfeeder docs and tests:
|
|
|
|
- Top-level config contains `sources`, `sinks`, and optional `routes`.
|
|
- Config struct fields are decoded strictly, so misspelled struct fields fail
|
|
startup.
|
|
- Driver-specific `params` maps are decoded generically and validated by the
|
|
source or sink constructor that consumes them.
|
|
- Source `kinds` can be validated against a source's advertised `Kinds()`.
|
|
|
|
## Source And HTTP Contract
|
|
|
|
Most weatherfeeder sources use feedkit's single-document HTTP source helper for:
|
|
|
|
- request construction;
|
|
- `User-Agent` and `Accept` headers;
|
|
- optional conditional GET validators;
|
|
- response body size limits;
|
|
- context-aware HTTP work;
|
|
- unchanged `304 Not Modified` responses that emit no events.
|
|
|
|
The SPC convective outlook source fetches multiple documents itself, but it uses
|
|
feedkit transport helpers for HTTP clients and response body limits.
|
|
|
|
## Scheduler And Processing Contract
|
|
|
|
Weatherfeeder builds feedkit scheduler jobs from source configs. Current source
|
|
drivers are polling drivers and use the configured `every` interval.
|
|
|
|
Events flow through a feedkit pipeline in this order:
|
|
|
|
1. normalize processor;
|
|
2. dedupe processor.
|
|
|
|
The normalize processor is configured with `RequireMatch=false`, so unmatched
|
|
schemas pass through unchanged. Weatherfeeder registers its built-in normalizers
|
|
and owns the provider-to-canonical mapping.
|
|
|
|
The dedupe processor stores a bounded in-memory set of recent event IDs. The
|
|
bound is configured in `cmd/weatherfeeder`.
|
|
|
|
## Dispatch And Sink Contract
|
|
|
|
Feedkit compiles routes from config and dispatches processed events to matching
|
|
sinks. If `routes` is omitted, every configured sink receives every event kind.
|
|
|
|
Feedkit owns sink fanout mechanics, per-sink workers, queueing, context-aware
|
|
shutdown, and sink error logging. Weatherfeeder owns the event kinds and schemas
|
|
that make routes meaningful.
|
|
|
|
Built-in feedkit sinks used by weatherfeeder:
|
|
|
|
- `stdout`: validates and writes JSON events to stdout.
|
|
- `nats`: publishes JSON events to a configured subject.
|
|
- generic `postgres` sink factory: opens the database, ensures tables/indexes,
|
|
runs transactions, inserts mapped rows, and prunes when configured.
|
|
|
|
Weatherfeeder supplies its Postgres table schema and event mapper to feedkit's
|
|
Postgres sink factory. The table contract is documented in
|
|
[`postgres.md`](postgres.md).
|
|
|
|
## Boundaries
|
|
|
|
Do not move weather-domain policy into feedkit. Weatherfeeder owns:
|
|
|
|
- provider source drivers;
|
|
- raw and canonical schema constants;
|
|
- event kind meaning;
|
|
- canonical payload structs;
|
|
- normalizers;
|
|
- Postgres table shape and row mapping.
|
|
|
|
Do not duplicate generic feedkit mechanics in weatherfeeder unless there is a
|
|
narrow weather-specific reason. Runtime composition details are documented in
|
|
[`../internal/runtime.md`](../internal/runtime.md).
|