Canonicalize chunk units before extraction
This commit is contained in:
@@ -49,6 +49,10 @@ The `generic` chunker splits source units into ordered chunks. It validates the
|
||||
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
|
||||
and records chunk metadata for start unit, end unit, and unit count.
|
||||
|
||||
The pipeline runner canonicalizes chunk units from the source document by ID
|
||||
before extractors and mergers run. Chunker-owned context should stay in
|
||||
`SourceChunk.Metadata`.
|
||||
|
||||
Options:
|
||||
|
||||
- `max_units`: positive integer, default `50`;
|
||||
|
||||
@@ -97,6 +97,11 @@ invariants before running extractors:
|
||||
- every chunk source unit must exist in the source document;
|
||||
- source units inside each chunk must appear in source-document order.
|
||||
|
||||
After validation, the runner rebuilds each chunk from source-document units by
|
||||
ID, preserving the chunk boundary order and cloning chunk metadata. Extractors
|
||||
and downstream stages therefore see canonical source units, while
|
||||
`SourceChunk.Metadata` remains the supported place for chunker-owned context.
|
||||
|
||||
The framework does not require complete source-unit coverage and does not reject
|
||||
overlap between different chunks. Stricter policies, such as full coverage or
|
||||
non-overlap, belong to individual chunk modules when they are part of that
|
||||
|
||||
Reference in New Issue
Block a user