Move chunks into the canonical source model

This commit is contained in:
2026-07-17 05:32:11 +00:00
parent 40709e4ad8
commit 075888c97f
33 changed files with 548 additions and 325 deletions

View File

@@ -77,7 +77,8 @@ normalize requests retain access to the original source material.
Source validation requires every unit to carry a canonical self-reference to
its containing document and its own unit ID. Explicit clone, checkpoint, and
debug boundaries retain that reference, and the canonical source digest covers
it deterministically.
it deterministically. Chunks use the same source model and carry one canonical
reference spanning the first selected unit through the last.
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
rejected results, warnings, checkpoint events, and logical files returned by the
@@ -115,13 +116,15 @@ whose results are accepted and used.
## Chunk Canonicalization
Before lane execution, generic validation requires unique chunk IDs, matching
source identity, indexes matching returned order, valid ordered boundaries,
source identity, indexes matching returned order, a valid canonical reference,
non-empty content and media type, and at least one valid source unit per chunk.
Units may not repeat inside a chunk and must preserve source-document order.
Units may not repeat inside a chunk and must form a contiguous range in
source-document order. The chunk reference must exactly match the source and
the first and last unit references.
The runner then rebuilds each chunk's unit slice from the source document by
unit ID. It preserves the module-owned boundaries, content, media type, and
cloned metadata. The framework permits gaps and overlap between separate
unit ID. It preserves the canonical reference, content, media type, and cloned
metadata. The framework permits gaps and overlap between separate
chunks; stricter coverage policy belongs to the chunk implementation.
## Validation And Retries