feedkit/README.md

feedkit

feedkit is a small Go toolkit for building feed-processing daemons.

It gives you the reusable plumbing around collection, processing, routing, and emission, while leaving domain concepts, schemas, and application wiring in your daemon. The intended shape is a family of sibling applications such as weatherfeeder, newsfeeder, or earthquakefeeder that all share the same infrastructure patterns without sharing domain logic.

What It Does

A daemon built on feedkit typically:

• ingests upstream input by polling HTTP APIs or consuming streams
• emits domain-agnostic event.Event values
• optionally processes those events with stages like dedupe or normalization
• routes events to one or more sinks such as stdout, NATS, or Postgres

Conceptually, the pipeline is:

Collect -> Process -> Route -> Emit

Philosophy

feedkit is intentionally not a framework.

It does not try to own:

• your domain payload schemas
• your domain event kinds
• your daemon lifecycle or main.go
• your observability stack or deployment model

Instead, it provides small composable packages that are easy to wire together in different daemons.

When To Use It

feedkit is a good fit when you want:

• multiple small ingestion daemons with shared infrastructure patterns
• clear separation between raw upstream payloads and normalized canonical models
• reusable routing and sink behavior across domains
• strong config and event-envelope conventions without centralizing domain rules

It is a poor fit if you want a monolithic framework that dictates application structure end-to-end.

Built-In Capabilities

feedkit currently includes:

• strict YAML config loading and validation
• polling and streaming source abstractions
• scheduler orchestration for configured sources
• optional pipeline processors
• built-in dedupe and normalization processors
• route compilation and sink fanout
• built-in sinks for stdout, nats, and postgres

The Postgres sink is intentionally split between feedkit-owned infrastructure and daemon-owned schema mapping. feedkit manages connection setup, DDL, writes, and pruning; downstream applications define the schema and event mapper.

Typical Wiring

At a high level, a daemon built on feedkit does this:

1. Load config.
2. Register domain-specific source drivers.
3. Register built-in and/or custom sinks.
4. Build sources, sinks, and optional processor chain from config.
5. Compile routes.
6. Start the scheduler and dispatcher.

The package docs are the better source of truth for code-level details. In particular, each subpackage doc.go describes its external API surface and any optional helper APIs in helpers.go.

Package Layout

The major packages are:

• config: config loading and validation
• event: the domain-agnostic event envelope
• sources: source interfaces and reusable source helpers
• scheduler: source execution and cadence management
• processors: processor interfaces and registry
• processors/dedupe: built-in in-memory dedupe processor
• processors/normalize: built-in normalization processor and helpers
• pipeline: optional processor chain
• dispatch: route compilation and fanout
• sinks: sink interfaces, built-ins, and explicit Postgres factory helpers

The root package docs in doc.go provide a concise package-by-package map for Go documentation consumers.