eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Canonicalize chunk units before extraction

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-05 13:28:09 +00:00
parent e700df82d8
commit 95218218e2
5 changed files with 173 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/internal/pipeline.md
View File

@@ -97,6 +97,11 @@ invariants before running extractors:
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.
After validation, the runner rebuilds each chunk from source-document units by
ID, preserving the chunk boundary order and cloning chunk metadata. Extractors
and downstream stages therefore see canonical source units, while
`SourceChunk.Metadata` remains the supported place for chunker-owned context.
The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that