feedkit: ergonomics pass (shared logger, route compiler, param helpers)

- Add logging.Logf as the canonical printf-style logger type used across feedkit. - Update scheduler and dispatch to alias their Logger types to logging.Logf. - Eliminates type-mismatch friction when wiring one log function through the system. - Add dispatch.CompileRoutes(*config.Config) ([]dispatch.Route, error) - Compiles config routes into dispatch routes with event.ParseKind normalization. - If routes: is omitted, defaults to “all sinks receive all kinds”. - Expand config param helpers for both SourceConfig and SinkConfig - Add ParamBool/ParamInt/ParamDuration/ParamStringSlice (+ Default variants). - Supports common YAML-decoded types (bool/int/float/string, []any, etc.) - Keeps driver code cleaner and reduces repeated type assertions. - Fix Postgres sink validation error prefix ("postgres sink", not "rabbitmq sink").
2026-01-13 14:40:29 -06:00
parent 0cc2862170
commit 09bc65e947
9 changed files with 896 additions and 44 deletions
--- a/README.md
+++ b/README.md
@@ -1,3 +1,255 @@
 # feedkit

-Feedkit provides core interfaces and plumbing for applications that ingest and process feeds.
+**feedkit** provides the domain-agnostic core plumbing for *feed-processing daemons*.
+
+A feed daemon is a long-running process that:
+- polls one or more upstream providers (HTTP APIs, RSS feeds, etc.)
+- normalizes upstream data into a consistent internal representation
+- applies lightweight policy (dedupe, rate-limit, filtering)
+- emits events to one or more sinks (stdout, files, databases, brokers)
+
+feedkit is designed to be reused by many concrete daemons (e.g. `weatherfeeder`,
+`newsfeeder`, `rssfeeder`) without embedding *any* domain-specific logic.
+
+---
+
+## Philosophy
+
+feedkit is **not a framework**.
+
+It does **not**:
+- define domain schemas
+- enforce allowed event kinds
+- hide control flow behind inversion-of-control magic
+- own your application lifecycle
+
+Instead, it provides **small, composable primitives** that concrete daemons wire
+together explicitly. The goal is clarity, predictability, and long-term
+maintainability.
+
+---
+
+## Conceptual pipeline
+
+Collect → Normalize → Filter / Policy → Route → Persist / Emit
+
+In feedkit terms:
+
+| Stage      | Package(s)                           |
+|------------|--------------------------------------|
+| Collect    | `sources`, `scheduler`               |
+| Normalize  | *(today: domain code; planned: pipeline processor)* |
+| Policy     | `pipeline`                           |
+| Route      | `dispatch`                           |
+| Emit       | `sinks`                              |
+| Configure  | `config`                             |
+
+---
+
+## Public API overview
+
+### `config` — Configuration loading & validation  
+**Status:** 🟢 Stable
+
+- Loads YAML configuration
+- Strict decoding (`KnownFields(true)`)
+- Domain-agnostic validation only (shape, required fields, references)
+- Flexible `Params map[string]any` with typed helpers
+
+Key types:
+- `config.Config`
+- `config.SourceConfig`
+- `config.SinkConfig`
+- `config.Load(path)`
+
+---
+
+### `event` — Domain-agnostic event envelope  
+**Status:** 🟢 Stable
+
+Defines the canonical event structure that moves through feedkit.
+
+Includes:
+- Stable ID
+- Kind (stringly-typed, domain-defined)
+- Source name
+- Timestamps (`EmittedAt`, optional `EffectiveAt`)
+- Optional `Schema` for payload versioning
+- Opaque `Payload`
+
+Key types:
+- `event.Event`
+- `event.Kind`
+- `event.ParseKind`
+- `event.Event.Validate`
+
+feedkit infrastructure never inspects `Payload`.
+
+---
+
+### `sources` — Polling abstraction  
+**Status:** 🟢 Stable (interface); 🔵 evolving patterns
+
+Defines the contract implemented by domain-specific polling jobs.
+
+```go
+type Source interface {
+    Name() string
+    Kind() event.Kind
+    Poll(ctx context.Context) ([]event.Event, error)
+}
+```
+Includes a registry (sources.Registry) so daemons can register drivers
+(e.g. openweather_observation, rss_feed) without switch statements.
+
+Note: Today, most sources both fetch and normalize. A dedicated
+normalization hook is planned (see below).
+
+### `scheduler` — Time-based polling
+
+**Status:** 🟢 Stable
+
+Runs one goroutine per source on a configured interval with jitter.
+
+Features:
+- Per-source interval
+- Deterministic jitter (avoids thundering herd)
+- Immediate poll at startup
+- Context-aware shutdown
+
+Key types:
+- `scheduler.Scheduler`
+- `scheduler.Job`
+
+### `pipeline` — Event processing chain
+**Status:** 🟡 Partial (API stable, processors evolving)
+
+Allows events to be transformed, dropped, or rejected between collection
+and dispatch.
+
+```go
+type Processor interface {
+    Process(ctx context.Context, in event.Event) (*event.Event, error)
+}
+```
+
+Current state:
+- `pipeline.Pipeline` is fully implemented
+
+Placeholder files exist for:
+- `dedupe` (planned)
+- `ratelimit` (planned)
+
+This is the intended home for:
+- normalization
+- deduplication
+- rate limiting
+- lightweight policy enforcement
+
+### `dispatch` — Routing & fan-out
+**Status:** 🟢 Stable
+
+Routes events to sinks based on kind and isolates slow sinks.
+
+Features:
+- Compiled routing rules
+- Per-sink buffered queues
+- Bounded enqueue timeouts
+- Per-consume timeouts
+- Sink panic isolation
+- Context-aware shutdown
+
+Key types:
+- `dispatch.Dispatcher`
+- `dispatch.Route`
+- `dispatch.Fanout`
+
+### `sinks` — Output adapters
+***Status:*** 🟡 Mixed
+
+Defines where events go after processing.
+
+```go
+type Sink interface {
+    Name() string
+    Consume(ctx context.Context, e event.Event) error
+}
+```
+
+Registry-based construction allows daemons to opt into any sink drivers.
+
+Sink	Status
+stdout	🟢 Implemented
+file	🔴 Stub
+postgres	🔴 Stub
+rabbitmq	🔴 Stub
+
+All sinks are required to respect context cancellation.
+
+### Normalization (planned)
+**Status:** 🔵 Planned (API design in progress)
+
+Currently, most domain implementations normalize upstream data inside
+`sources.Source.Poll`, which leads to:
+- very large source files
+- mixed responsibilities (HTTP + mapping)
+- duplicated helper code
+
+The intended evolution is:
+- Sources emit raw events (e.g. `json.RawMessage`)
+- A dedicated normalization processor runs in the pipeline
+- Normalizers are selected by `Event.Schema`, `Kind`, or `Source`
+
+This keeps:
+- `feedkit` domain-agnostic
+- `sources` small and focused
+- normalization logic centralized and testable
+
+### Runner helper (planned)
+**Status:** 🔵 Planned (optional convenience)
+
+Most daemons wire together the same steps:
+- load config
+- build sources
+- build sinks
+- compile routes
+- start scheduler
+- start dispatcher
+
+A small, opt-in `Runner` helper may be added to reduce boilerplate while keeping the system explicit and debuggable.
+
+This is not intended to become a framework.
+
+## Stability summary
+Area	Status
+Event model	🟢 Stable
+Config API	🟢 Stable
+Scheduler	🟢 Stable
+Dispatcher	🟢 Stable
+Source interface	🟢 Stable
+Pipeline core	🟡 Partial
+Normalization	🔵 Planned
+Dedupe/Ratelimit	🔵 Planned
+Non-stdout sinks	🔴 Stub
+
+Legend:
+🟢 Stable — API considered solid
+🟡 Partial — usable, but incomplete
+🔵 Planned — design direction agreed, not yet implemented
+🔴 Stub — placeholder only
+
+## Non-goals
+`feedkit` intentionally does not:
+- define domain payload schemas
+- enforce domain-specific validation
+- manage persistence semantics beyond sink adapters
+- own observability, metrics, or tracing (left to daemons)
+
+Those concerns belong in concrete implementations.
+
+## See also
+- NAMING.md — repository and daemon naming conventions
+- event/doc.go — detailed event semantics
+- **Concrete example:** weatherfeeder (reference implementation)
+
+---