Eric Rakestraw
eb9a7cb349
Remove global Postgres schema registration in favor of explicit schema-aware sink factory wiring, and update weatherfeeder to register the Postgres sink explicitly. Add optional per-source HTTP timeout and response body limit overrides while keeping feedkit defaults. Remove remaining legacy source/config compatibility surfaces, including singular kind support and old source registry/type aliases, and migrate weatherfeeder sources to plural `Kinds()` metadata. Clean up related docs, tests, and sample config to match the new Postgres, HTTP, and NATS configuration model.
93 lines
3.3 KiB
Markdown
93 lines
3.3 KiB
Markdown
# feedkit
|
|
|
|
`feedkit` is a small Go toolkit for building feed-processing daemons.
|
|
|
|
It gives you the reusable plumbing around collection, processing, routing, and
|
|
emission, while leaving domain concepts, schemas, and application wiring in
|
|
your daemon. The intended shape is a family of sibling applications such as
|
|
`weatherfeeder`, `newsfeeder`, or `earthquakefeeder` that all share the same
|
|
infrastructure patterns without sharing domain logic.
|
|
|
|
## What It Does
|
|
|
|
A daemon built on `feedkit` typically:
|
|
- ingests upstream input by polling HTTP APIs or consuming streams
|
|
- emits domain-agnostic `event.Event` values
|
|
- optionally processes those events with stages like dedupe or normalization
|
|
- routes events to one or more sinks such as stdout, NATS, or Postgres
|
|
|
|
Conceptually, the pipeline is:
|
|
|
|
`Collect -> Process -> Route -> Emit`
|
|
|
|
## Philosophy
|
|
|
|
`feedkit` is intentionally not a framework.
|
|
|
|
It does not try to own:
|
|
- your domain payload schemas
|
|
- your domain event kinds
|
|
- your daemon lifecycle or `main.go`
|
|
- your observability stack or deployment model
|
|
|
|
Instead, it provides small composable packages that are easy to wire together in
|
|
different daemons.
|
|
|
|
## When To Use It
|
|
|
|
`feedkit` is a good fit when you want:
|
|
- multiple small ingestion daemons with shared infrastructure patterns
|
|
- clear separation between raw upstream payloads and normalized canonical models
|
|
- reusable routing and sink behavior across domains
|
|
- strong config and event-envelope conventions without centralizing domain rules
|
|
|
|
It is a poor fit if you want a monolithic framework that dictates application
|
|
structure end-to-end.
|
|
|
|
## Built-In Capabilities
|
|
|
|
`feedkit` currently includes:
|
|
- strict YAML config loading and validation
|
|
- polling and streaming source abstractions
|
|
- scheduler orchestration for configured sources
|
|
- optional pipeline processors
|
|
- built-in dedupe and normalization processors
|
|
- route compilation and sink fanout
|
|
- built-in sinks for `stdout`, `nats`, and `postgres`
|
|
|
|
The Postgres sink is intentionally split between feedkit-owned infrastructure
|
|
and daemon-owned schema mapping. `feedkit` manages connection setup, DDL,
|
|
writes, and pruning; downstream applications define the schema and event mapper.
|
|
|
|
## Typical Wiring
|
|
|
|
At a high level, a daemon built on `feedkit` does this:
|
|
|
|
1. Load config.
|
|
2. Register domain-specific source drivers.
|
|
3. Register built-in and/or custom sinks.
|
|
4. Build sources, sinks, and optional processor chain from config.
|
|
5. Compile routes.
|
|
6. Start the scheduler and dispatcher.
|
|
|
|
The package docs are the better source of truth for code-level details. In
|
|
particular, each subpackage `doc.go` describes its external API surface and any
|
|
optional helper APIs in `helpers.go`.
|
|
|
|
## Package Layout
|
|
|
|
The major packages are:
|
|
- `config`: config loading and validation
|
|
- `event`: the domain-agnostic event envelope
|
|
- `sources`: source interfaces and reusable source helpers
|
|
- `scheduler`: source execution and cadence management
|
|
- `processors`: processor interfaces and registry
|
|
- `processors/dedupe`: built-in in-memory dedupe processor
|
|
- `processors/normalize`: built-in normalization processor and helpers
|
|
- `pipeline`: optional processor chain
|
|
- `dispatch`: route compilation and fanout
|
|
- `sinks`: sink interfaces, built-ins, and explicit Postgres factory helpers
|
|
|
|
The root package docs in `doc.go` provide a concise package-by-package map for
|
|
Go documentation consumers.
|