128 lines
3.3 KiB
Markdown
128 lines
3.3 KiB
Markdown
# feedkit
|
|
|
|
`feedkit` provides domain-agnostic plumbing for feed-processing daemons.
|
|
|
|
A daemon built on feedkit typically:
|
|
- ingests upstream input (polling APIs or consuming streams)
|
|
- emits domain-agnostic `event.Event` values
|
|
- applies optional processing (normalization, dedupe, policy)
|
|
- routes events to sinks (stdout, NATS, files, databases, etc.)
|
|
|
|
## Philosophy
|
|
|
|
feedkit is not a framework. It provides small composable packages and leaves
|
|
lifecycle, domain schemas, and domain-specific validation in your daemon.
|
|
|
|
## Conceptual pipeline
|
|
|
|
Collect -> Process (optional stages, including dedupe + normalize) -> Route -> Emit
|
|
|
|
| Stage | Package(s) |
|
|
|---|---|
|
|
| Collect | `sources`, `scheduler` |
|
|
| Process | `pipeline`, `processors`, `processors/dedupe`, `processors/normalize` (optional stages) |
|
|
| Route | `dispatch` |
|
|
| Emit | `sinks` |
|
|
| Configure | `config` |
|
|
|
|
## Core packages
|
|
|
|
### `config`
|
|
|
|
Loads YAML config with strict decoding and domain-agnostic validation.
|
|
|
|
`SourceConfig` supports both source modes:
|
|
- `mode: poll` requires `every`
|
|
- `mode: stream` forbids `every`
|
|
- omitted `mode` means auto (inferred from the registered driver type)
|
|
|
|
It also supports optional expected source kinds:
|
|
- `kinds: ["observation", "alert"]` (preferred)
|
|
- `kind: "observation"` (legacy fallback)
|
|
|
|
### `event`
|
|
|
|
Defines the domain-agnostic event envelope (`event.Event`) used across the system.
|
|
|
|
### `sources`
|
|
|
|
Defines source interfaces and driver registry:
|
|
|
|
```go
|
|
type Input interface {
|
|
Name() string
|
|
}
|
|
|
|
type PollSource interface {
|
|
Input
|
|
Poll(ctx context.Context) ([]event.Event, error)
|
|
}
|
|
|
|
type StreamSource interface {
|
|
Input
|
|
Run(ctx context.Context, out chan<- event.Event) error
|
|
}
|
|
```
|
|
|
|
Notes:
|
|
- a poll can emit `0..N` events
|
|
- stream sources emit events continuously
|
|
- a single source may emit multiple event kinds
|
|
- driver implementations live in downstream daemons and are registered via `sources.Registry`
|
|
|
|
### `scheduler`
|
|
|
|
Runs one goroutine per source job:
|
|
- poll sources: cadence driven (`every` + jitter)
|
|
- stream sources: continuous run loop
|
|
|
|
### `pipeline`
|
|
|
|
Optional processing chain between collection and dispatch.
|
|
Processors can transform, drop, or reject events.
|
|
|
|
### `processors`
|
|
|
|
Defines the generic processor interface and a named-driver registry used by
|
|
daemons to build ordered processor chains.
|
|
|
|
### `processors/dedupe`
|
|
|
|
Built-in in-memory LRU dedupe processor that drops repeated events by `Event.ID`.
|
|
|
|
### `processors/normalize`
|
|
|
|
Concrete normalization processor implementation. Typical use: sources emit raw
|
|
payload events, then a normalize stage maps them to canonical schemas.
|
|
|
|
### `dispatch`
|
|
|
|
Compiles routes and fans out events to sinks with per-sink queue/worker isolation.
|
|
|
|
### `sinks`
|
|
|
|
Defines sink interface and sink registry. Built-ins include:
|
|
- `stdout`
|
|
- `nats`
|
|
- `postgres`
|
|
|
|
Detailed Postgres configuration and wiring examples live in package docs:
|
|
`sinks/doc.go`.
|
|
|
|
## Typical wiring
|
|
|
|
1. Load config.
|
|
2. Register/build sources from `cfg.Sources`.
|
|
3. Register/build sinks from `cfg.Sinks`.
|
|
4. Compile routes.
|
|
5. Start scheduler (`sources -> bus`).
|
|
6. Start dispatcher (`bus -> pipeline -> sinks`).
|
|
|
|
## Non-goals
|
|
|
|
feedkit intentionally does not:
|
|
- define domain payload schemas
|
|
- enforce domain-specific event kinds
|
|
- own application lifecycle
|
|
- prescribe observability stack choices
|