Added a dedupe processor, and moved processor packages under processors/*
This commit is contained in:
12
README.md
12
README.md
@@ -15,12 +15,12 @@ lifecycle, domain schemas, and domain-specific validation in your daemon.
|
||||
|
||||
## Conceptual pipeline
|
||||
|
||||
Collect -> Process (optional stages, including normalize) -> Route -> Emit
|
||||
Collect -> Process (optional stages, including dedupe + normalize) -> Route -> Emit
|
||||
|
||||
| Stage | Package(s) |
|
||||
|---|---|
|
||||
| Collect | `sources`, `scheduler` |
|
||||
| Process | `pipeline`, `processors`, `normalize` (optional stage) |
|
||||
| Process | `pipeline`, `processors`, `processors/dedupe`, `processors/normalize` (optional stages) |
|
||||
| Route | `dispatch` |
|
||||
| Emit | `sinks` |
|
||||
| Configure | `config` |
|
||||
@@ -86,7 +86,11 @@ Processors can transform, drop, or reject events.
|
||||
Defines the generic processor interface and a named-driver registry used by
|
||||
daemons to build ordered processor chains.
|
||||
|
||||
### `normalize`
|
||||
### `processors/dedupe`
|
||||
|
||||
Built-in in-memory LRU dedupe processor that drops repeated events by `Event.ID`.
|
||||
|
||||
### `processors/normalize`
|
||||
|
||||
Concrete normalization processor implementation. Typical use: sources emit raw
|
||||
payload events, then a normalize stage maps them to canonical schemas.
|
||||
@@ -100,7 +104,7 @@ Compiles routes and fans out events to sinks with per-sink queue/worker isolatio
|
||||
Defines sink interface and sink registry. Built-ins include:
|
||||
- `stdout`
|
||||
- `nats`
|
||||
- `postgres` (downstream registers table schema + event mapper; feedkit handles create-if-missing DDL, transactional inserts, and optional prune APIs)
|
||||
- `postgres`
|
||||
|
||||
Detailed Postgres configuration and wiring examples live in package docs:
|
||||
`sinks/doc.go`.
|
||||
|
||||
Reference in New Issue
Block a user