feedkit/README.md

feedkit

feedkit provides domain-agnostic plumbing for feed-processing daemons.

A daemon built on feedkit typically:

• ingests upstream input (polling APIs or consuming streams)
• emits domain-agnostic event.Event values
• applies optional processing (normalization, dedupe, policy)
• routes events to sinks (stdout, NATS, files, databases, etc.)

Philosophy

feedkit is not a framework. It provides small composable packages and leaves lifecycle, domain schemas, and domain-specific validation in your daemon.

Conceptual pipeline

Collect -> Process (optional stages, including dedupe + normalize) -> Route -> Emit

Stage	Package(s)
Collect	sources, scheduler
Process	pipeline, processors, processors/dedupe, processors/normalize (optional stages)
Route	dispatch
Emit	sinks
Configure	config

Core packages

config

Loads YAML config with strict decoding and domain-agnostic validation.

SourceConfig supports both source modes:

• mode: poll requires every
• mode: stream forbids every
• omitted mode means auto (inferred from the registered driver type)

It also supports optional expected source kinds:

• kinds: ["observation", "alert"] (preferred)
• kind: "observation" (legacy fallback)

event

Defines the domain-agnostic event envelope (event.Event) used across the system.

sources

Defines source interfaces and driver registry:

type Input interface {
    Name() string
}

type PollSource interface {
    Input
    Poll(ctx context.Context) ([]event.Event, error)
}

type StreamSource interface {
    Input
    Run(ctx context.Context, out chan<- event.Event) error
}

Notes:

• a poll can emit 0..N events
• stream sources emit events continuously
• a single source may emit multiple event kinds
• driver implementations live in downstream daemons and are registered via sources.Registry

scheduler

Runs one goroutine per source job:

• poll sources: cadence driven (every + jitter)
• stream sources: continuous run loop

pipeline

Optional processing chain between collection and dispatch. Processors can transform, drop, or reject events.

processors

Defines the generic processor interface and a named-driver registry used by daemons to build ordered processor chains.

processors/dedupe

Built-in in-memory LRU dedupe processor that drops repeated events by Event.ID.

processors/normalize

Concrete normalization processor implementation. Typical use: sources emit raw payload events, then a normalize stage maps them to canonical schemas.

dispatch

Compiles routes and fans out events to sinks with per-sink queue/worker isolation.

sinks

Defines sink interface and sink registry. Built-ins include:

• stdout
• nats
• postgres

Detailed Postgres configuration and wiring examples live in package docs: sinks/doc.go.

Typical wiring

1. Load config.
2. Register/build sources from cfg.Sources.
3. Register/build sinks from cfg.Sinks.
4. Compile routes.
5. Start scheduler (sources -> bus).
6. Start dispatcher (bus -> pipeline -> sinks).

Non-goals

feedkit intentionally does not:

• define domain payload schemas
• enforce domain-specific event kinds
• own application lifecycle
• prescribe observability stack choices