Added a dedupe processor, and moved processor packages under processors/*

This commit is contained in:
2026-03-16 18:17:53 -05:00
parent 4572c53580
commit 215afe1acf
11 changed files with 297 additions and 13 deletions

View File

@@ -15,12 +15,12 @@ lifecycle, domain schemas, and domain-specific validation in your daemon.
## Conceptual pipeline
Collect -> Process (optional stages, including normalize) -> Route -> Emit
Collect -> Process (optional stages, including dedupe + normalize) -> Route -> Emit
| Stage | Package(s) |
|---|---|
| Collect | `sources`, `scheduler` |
| Process | `pipeline`, `processors`, `normalize` (optional stage) |
| Process | `pipeline`, `processors`, `processors/dedupe`, `processors/normalize` (optional stages) |
| Route | `dispatch` |
| Emit | `sinks` |
| Configure | `config` |
@@ -86,7 +86,11 @@ Processors can transform, drop, or reject events.
Defines the generic processor interface and a named-driver registry used by
daemons to build ordered processor chains.
### `normalize`
### `processors/dedupe`
Built-in in-memory LRU dedupe processor that drops repeated events by `Event.ID`.
### `processors/normalize`
Concrete normalization processor implementation. Typical use: sources emit raw
payload events, then a normalize stage maps them to canonical schemas.
@@ -100,7 +104,7 @@ Compiles routes and fans out events to sinks with per-sink queue/worker isolatio
Defines sink interface and sink registry. Built-ins include:
- `stdout`
- `nats`
- `postgres` (downstream registers table schema + event mapper; feedkit handles create-if-missing DDL, transactional inserts, and optional prune APIs)
- `postgres`
Detailed Postgres configuration and wiring examples live in package docs:
`sinks/doc.go`.