Canonicalize chunk units before extraction

This commit is contained in:
2026-07-05 13:28:09 +00:00
parent e700df82d8
commit 95218218e2
5 changed files with 173 additions and 13 deletions

View File

@@ -49,6 +49,10 @@ The `generic` chunker splits source units into ordered chunks. It validates the
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
and records chunk metadata for start unit, end unit, and unit count.
The pipeline runner canonicalizes chunk units from the source document by ID
before extractors and mergers run. Chunker-owned context should stay in
`SourceChunk.Metadata`.
Options:
- `max_units`: positive integer, default `50`;