feat(sources)!: split source contracts into PollSource/StreamSource and add mode-aware source config
- Introduce explicit source interfaces: sources.PollSource and sources.StreamSource, with shared sources.Input (Name() only). - Remove mandatory Kind() from the base source contract to support sources that emit multiple kinds. - Add config.SourceMode (poll, stream, or omitted/auto) and SourceConfig.Kinds (plural expected kinds), while keeping legacy SourceConfig.Kind for compatibility. - Enforce mode semantics in config validation (poll requires every, stream forbids every) and detect mode/driver mismatches in sources.Registry. - Update docs and tests for the new source model and config behavior.
This commit is contained in:
283
README.md
283
README.md
@@ -1,255 +1,114 @@
|
||||
# feedkit
|
||||
|
||||
**feedkit** provides the domain-agnostic core plumbing for *feed-processing daemons*.
|
||||
`feedkit` provides domain-agnostic plumbing for feed-processing daemons.
|
||||
|
||||
A feed daemon is a long-running process that:
|
||||
- polls one or more upstream providers (HTTP APIs, RSS feeds, etc.)
|
||||
- normalizes upstream data into a consistent internal representation
|
||||
- applies lightweight policy (dedupe, rate-limit, filtering)
|
||||
- emits events to one or more sinks (stdout, files, databases, brokers)
|
||||
|
||||
feedkit is designed to be reused by many concrete daemons (e.g. `weatherfeeder`,
|
||||
`newsfeeder`, `rssfeeder`) without embedding *any* domain-specific logic.
|
||||
|
||||
---
|
||||
A daemon built on feedkit typically:
|
||||
- ingests upstream input (polling APIs or consuming streams)
|
||||
- emits domain-agnostic `event.Event` values
|
||||
- applies optional processing (normalization, dedupe, policy)
|
||||
- routes events to sinks (stdout, NATS, files, databases, etc.)
|
||||
|
||||
## Philosophy
|
||||
|
||||
feedkit is **not a framework**.
|
||||
|
||||
It does **not**:
|
||||
- define domain schemas
|
||||
- enforce allowed event kinds
|
||||
- hide control flow behind inversion-of-control magic
|
||||
- own your application lifecycle
|
||||
|
||||
Instead, it provides **small, composable primitives** that concrete daemons wire
|
||||
together explicitly. The goal is clarity, predictability, and long-term
|
||||
maintainability.
|
||||
|
||||
---
|
||||
feedkit is not a framework. It provides small composable packages and leaves
|
||||
lifecycle, domain schemas, and domain-specific validation in your daemon.
|
||||
|
||||
## Conceptual pipeline
|
||||
|
||||
Collect → Normalize → Filter / Policy → Route → Persist / Emit
|
||||
Collect -> Normalize (optional) -> Policy -> Route -> Emit
|
||||
|
||||
In feedkit terms:
|
||||
| Stage | Package(s) |
|
||||
|---|---|
|
||||
| Collect | `sources`, `scheduler` |
|
||||
| Normalize | `normalize` (optional in `pipeline`) |
|
||||
| Policy | `pipeline` |
|
||||
| Route | `dispatch` |
|
||||
| Emit | `sinks` |
|
||||
| Configure | `config` |
|
||||
|
||||
| Stage | Package(s) |
|
||||
|------------|--------------------------------------|
|
||||
| Collect | `sources`, `scheduler` |
|
||||
| Normalize | *(today: domain code; planned: pipeline processor)* |
|
||||
| Policy | `pipeline` |
|
||||
| Route | `dispatch` |
|
||||
| Emit | `sinks` |
|
||||
| Configure | `config` |
|
||||
## Core packages
|
||||
|
||||
---
|
||||
### `config`
|
||||
|
||||
## Public API overview
|
||||
Loads YAML config with strict decoding and domain-agnostic validation.
|
||||
|
||||
### `config` — Configuration loading & validation
|
||||
**Status:** 🟢 Stable
|
||||
`SourceConfig` supports both source modes:
|
||||
- `mode: poll` requires `every`
|
||||
- `mode: stream` forbids `every`
|
||||
- omitted `mode` means auto (inferred from the registered driver type)
|
||||
|
||||
- Loads YAML configuration
|
||||
- Strict decoding (`KnownFields(true)`)
|
||||
- Domain-agnostic validation only (shape, required fields, references)
|
||||
- Flexible `Params map[string]any` with typed helpers
|
||||
It also supports optional expected source kinds:
|
||||
- `kinds: ["observation", "alert"]` (preferred)
|
||||
- `kind: "observation"` (legacy fallback)
|
||||
|
||||
Key types:
|
||||
- `config.Config`
|
||||
- `config.SourceConfig`
|
||||
- `config.SinkConfig`
|
||||
- `config.Load(path)`
|
||||
### `event`
|
||||
|
||||
---
|
||||
Defines the domain-agnostic event envelope (`event.Event`) used across the system.
|
||||
|
||||
### `event` — Domain-agnostic event envelope
|
||||
**Status:** 🟢 Stable
|
||||
### `sources`
|
||||
|
||||
Defines the canonical event structure that moves through feedkit.
|
||||
|
||||
Includes:
|
||||
- Stable ID
|
||||
- Kind (stringly-typed, domain-defined)
|
||||
- Source name
|
||||
- Timestamps (`EmittedAt`, optional `EffectiveAt`)
|
||||
- Optional `Schema` for payload versioning
|
||||
- Opaque `Payload`
|
||||
|
||||
Key types:
|
||||
- `event.Event`
|
||||
- `event.Kind`
|
||||
- `event.ParseKind`
|
||||
- `event.Event.Validate`
|
||||
|
||||
feedkit infrastructure never inspects `Payload`.
|
||||
|
||||
---
|
||||
|
||||
### `sources` — Polling abstraction
|
||||
**Status:** 🟢 Stable (interface); 🔵 evolving patterns
|
||||
|
||||
Defines the contract implemented by domain-specific polling jobs.
|
||||
Defines source interfaces and driver registry:
|
||||
|
||||
```go
|
||||
type Source interface {
|
||||
type Input interface {
|
||||
Name() string
|
||||
Kind() event.Kind
|
||||
}
|
||||
|
||||
type PollSource interface {
|
||||
Input
|
||||
Poll(ctx context.Context) ([]event.Event, error)
|
||||
}
|
||||
```
|
||||
Includes a registry (sources.Registry) so daemons can register drivers
|
||||
(e.g. openweather_observation, rss_feed) without switch statements.
|
||||
|
||||
Note: Today, most sources both fetch and normalize. A dedicated
|
||||
normalization hook is planned (see below).
|
||||
|
||||
### `scheduler` — Time-based polling
|
||||
|
||||
**Status:** 🟢 Stable
|
||||
|
||||
Runs one goroutine per source on a configured interval with jitter.
|
||||
|
||||
Features:
|
||||
- Per-source interval
|
||||
- Deterministic jitter (avoids thundering herd)
|
||||
- Immediate poll at startup
|
||||
- Context-aware shutdown
|
||||
|
||||
Key types:
|
||||
- `scheduler.Scheduler`
|
||||
- `scheduler.Job`
|
||||
|
||||
### `pipeline` — Event processing chain
|
||||
**Status:** 🟡 Partial (API stable, processors evolving)
|
||||
|
||||
Allows events to be transformed, dropped, or rejected between collection
|
||||
and dispatch.
|
||||
|
||||
```go
|
||||
type Processor interface {
|
||||
Process(ctx context.Context, in event.Event) (*event.Event, error)
|
||||
type StreamSource interface {
|
||||
Input
|
||||
Run(ctx context.Context, out chan<- event.Event) error
|
||||
}
|
||||
```
|
||||
|
||||
Current state:
|
||||
- `pipeline.Pipeline` is fully implemented
|
||||
Notes:
|
||||
- a poll can emit `0..N` events
|
||||
- stream sources emit events continuously
|
||||
- a single source may emit multiple event kinds
|
||||
- driver implementations live in downstream daemons and are registered via `sources.Registry`
|
||||
|
||||
Placeholder files exist for:
|
||||
- `dedupe` (planned)
|
||||
- `ratelimit` (planned)
|
||||
### `scheduler`
|
||||
|
||||
This is the intended home for:
|
||||
- normalization
|
||||
- deduplication
|
||||
- rate limiting
|
||||
- lightweight policy enforcement
|
||||
Runs one goroutine per source job:
|
||||
- poll sources: cadence driven (`every` + jitter)
|
||||
- stream sources: continuous run loop
|
||||
|
||||
### `dispatch` — Routing & fan-out
|
||||
**Status:** 🟢 Stable
|
||||
### `pipeline`
|
||||
|
||||
Routes events to sinks based on kind and isolates slow sinks.
|
||||
Optional processing chain between collection and dispatch.
|
||||
Processors can transform, drop, or reject events.
|
||||
|
||||
Features:
|
||||
- Compiled routing rules
|
||||
- Per-sink buffered queues
|
||||
- Bounded enqueue timeouts
|
||||
- Per-consume timeouts
|
||||
- Sink panic isolation
|
||||
- Context-aware shutdown
|
||||
### `normalize`
|
||||
|
||||
Key types:
|
||||
- `dispatch.Dispatcher`
|
||||
- `dispatch.Route`
|
||||
- `dispatch.Fanout`
|
||||
Optional normalization package (already implemented). Typical use: sources emit raw
|
||||
payload events, then normalize to canonical schemas in a pipeline stage.
|
||||
|
||||
### `sinks` — Output adapters
|
||||
***Status:*** 🟡 Mixed
|
||||
### `dispatch`
|
||||
|
||||
Defines where events go after processing.
|
||||
Compiles routes and fans out events to sinks with per-sink queue/worker isolation.
|
||||
|
||||
```go
|
||||
type Sink interface {
|
||||
Name() string
|
||||
Consume(ctx context.Context, e event.Event) error
|
||||
}
|
||||
```
|
||||
### `sinks`
|
||||
|
||||
Registry-based construction allows daemons to opt into any sink drivers.
|
||||
Defines sink interface and sink registry. Built-ins include `stdout` and `nats`, with
|
||||
additional sink implementations at varying maturity.
|
||||
|
||||
Sink Status
|
||||
stdout 🟢 Implemented
|
||||
nats 🟢 Implemented
|
||||
file 🔴 Stub
|
||||
postgres 🔴 Stub
|
||||
## Typical wiring
|
||||
|
||||
All sinks are required to respect context cancellation.
|
||||
|
||||
### Normalization (planned)
|
||||
**Status:** 🔵 Planned (API design in progress)
|
||||
|
||||
Currently, most domain implementations normalize upstream data inside
|
||||
`sources.Source.Poll`, which leads to:
|
||||
- very large source files
|
||||
- mixed responsibilities (HTTP + mapping)
|
||||
- duplicated helper code
|
||||
|
||||
The intended evolution is:
|
||||
- Sources emit raw events (e.g. `json.RawMessage`)
|
||||
- A dedicated normalization processor runs in the pipeline
|
||||
- Normalizers are selected by `Event.Schema`, `Kind`, or `Source`
|
||||
|
||||
This keeps:
|
||||
- `feedkit` domain-agnostic
|
||||
- `sources` small and focused
|
||||
- normalization logic centralized and testable
|
||||
|
||||
### Runner helper (planned)
|
||||
**Status:** 🔵 Planned (optional convenience)
|
||||
|
||||
Most daemons wire together the same steps:
|
||||
- load config
|
||||
- build sources
|
||||
- build sinks
|
||||
- compile routes
|
||||
- start scheduler
|
||||
- start dispatcher
|
||||
|
||||
A small, opt-in `Runner` helper may be added to reduce boilerplate while keeping the system explicit and debuggable.
|
||||
|
||||
This is not intended to become a framework.
|
||||
|
||||
## Stability summary
|
||||
Area Status
|
||||
Event model 🟢 Stable
|
||||
Config API 🟢 Stable
|
||||
Scheduler 🟢 Stable
|
||||
Dispatcher 🟢 Stable
|
||||
Source interface 🟢 Stable
|
||||
Pipeline core 🟡 Partial
|
||||
Normalization 🔵 Planned
|
||||
Dedupe/Ratelimit 🔵 Planned
|
||||
Non-stdout sinks 🔴 Stub
|
||||
|
||||
Legend:
|
||||
🟢 Stable — API considered solid
|
||||
🟡 Partial — usable, but incomplete
|
||||
🔵 Planned — design direction agreed, not yet implemented
|
||||
🔴 Stub — placeholder only
|
||||
1. Load config.
|
||||
2. Register/build sources from `cfg.Sources`.
|
||||
3. Register/build sinks from `cfg.Sinks`.
|
||||
4. Compile routes.
|
||||
5. Start scheduler (`sources -> bus`).
|
||||
6. Start dispatcher (`bus -> pipeline -> sinks`).
|
||||
|
||||
## Non-goals
|
||||
`feedkit` intentionally does not:
|
||||
|
||||
feedkit intentionally does not:
|
||||
- define domain payload schemas
|
||||
- enforce domain-specific validation
|
||||
- manage persistence semantics beyond sink adapters
|
||||
- own observability, metrics, or tracing (left to daemons)
|
||||
|
||||
Those concerns belong in concrete implementations.
|
||||
|
||||
## See also
|
||||
- NAMING.md — repository and daemon naming conventions
|
||||
- event/doc.go — detailed event semantics
|
||||
- **Concrete example:** weatherfeeder (reference implementation)
|
||||
|
||||
---
|
||||
- enforce domain-specific event kinds
|
||||
- own application lifecycle
|
||||
- prescribe observability stack choices
|
||||
|
||||
Reference in New Issue
Block a user