Canonicalize chunk units before extraction
This commit is contained in:
@@ -49,6 +49,10 @@ The `generic` chunker splits source units into ordered chunks. It validates the
|
||||
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
|
||||
and records chunk metadata for start unit, end unit, and unit count.
|
||||
|
||||
The pipeline runner canonicalizes chunk units from the source document by ID
|
||||
before extractors and mergers run. Chunker-owned context should stay in
|
||||
`SourceChunk.Metadata`.
|
||||
|
||||
Options:
|
||||
|
||||
- `max_units`: positive integer, default `50`;
|
||||
|
||||
Reference in New Issue
Block a user