ejr/feedkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Added a dedupe processor, and moved processor packages under processors/*

Browse Source
This commit is contained in:
Eric Rakestraw
2026-03-16 18:17:53 -05:00
parent 4572c53580
commit 215afe1acf
11 changed files with 297 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
doc.go
View File

@@ -5,12 +5,12 @@
//
// Conceptual flow:
//
// Collect -> Process (optional stages, including normalize) -> Route -> Emit
// Collect -> Process (optional stages, including dedupe + normalize) -> Route -> Emit
//
// In feedkit this maps to:
//
// Collect: sources + scheduler
// Process: pipeline + processors + normalize (optional stage)
// Process: pipeline + processors + processors/dedupe + processors/normalize (optional stages)
// Route: dispatch
// Emit: sinks
// Config: config
@@ -64,7 +64,10 @@
// - processors
// Generic processor interface and named factory registry for wiring chains.
//
// - normalize
// - processors/dedupe
// Built-in in-memory LRU dedupe processor keyed by Event.ID.
//
// - processors/normalize
// Concrete pipeline processor for raw->canonical mapping.
// If no normalizer matches, the event passes through unchanged by default.
//