feedkit/README.md

feedkit

feedkit provides domain-agnostic plumbing for feed-processing daemons.

A daemon built on feedkit typically:

• ingests upstream input (polling APIs or consuming streams)
• emits domain-agnostic event.Event values
• applies optional processing (normalization, dedupe, policy)
• routes events to sinks (stdout, NATS, files, databases, etc.)

Philosophy

feedkit is not a framework. It provides small composable packages and leaves lifecycle, domain schemas, and domain-specific validation in your daemon.

Conceptual pipeline

Collect -> Normalize (optional) -> Policy -> Route -> Emit

Stage	Package(s)
Collect	sources, scheduler
Normalize	normalize (optional in pipeline)
Policy	pipeline
Route	dispatch
Emit	sinks
Configure	config

Core packages

config

Loads YAML config with strict decoding and domain-agnostic validation.

SourceConfig supports both source modes:

• mode: poll requires every
• mode: stream forbids every
• omitted mode means auto (inferred from the registered driver type)

It also supports optional expected source kinds:

• kinds: ["observation", "alert"] (preferred)
• kind: "observation" (legacy fallback)

event

Defines the domain-agnostic event envelope (event.Event) used across the system.

sources

Defines source interfaces and driver registry:

type Input interface {
    Name() string
}

type PollSource interface {
    Input
    Poll(ctx context.Context) ([]event.Event, error)
}

type StreamSource interface {
    Input
    Run(ctx context.Context, out chan<- event.Event) error
}

Notes:

• a poll can emit 0..N events
• stream sources emit events continuously
• a single source may emit multiple event kinds
• driver implementations live in downstream daemons and are registered via sources.Registry

scheduler

Runs one goroutine per source job:

• poll sources: cadence driven (every + jitter)
• stream sources: continuous run loop

pipeline

Optional processing chain between collection and dispatch. Processors can transform, drop, or reject events.

normalize

Optional normalization package (already implemented). Typical use: sources emit raw payload events, then normalize to canonical schemas in a pipeline stage.

dispatch

Compiles routes and fans out events to sinks with per-sink queue/worker isolation.

sinks

Defines sink interface and sink registry. Built-ins include stdout and nats, with additional sink implementations at varying maturity.

Typical wiring

1. Load config.
2. Register/build sources from cfg.Sources.
3. Register/build sinks from cfg.Sinks.
4. Compile routes.
5. Start scheduler (sources -> bus).
6. Start dispatcher (bus -> pipeline -> sinks).

Non-goals

feedkit intentionally does not:

• define domain payload schemas
• enforce domain-specific event kinds
• own application lifecycle
• prescribe observability stack choices