Move chunks into the canonical source model
This commit is contained in:
@@ -77,7 +77,8 @@ normalize requests retain access to the original source material.
|
||||
Source validation requires every unit to carry a canonical self-reference to
|
||||
its containing document and its own unit ID. Explicit clone, checkpoint, and
|
||||
debug boundaries retain that reference, and the canonical source digest covers
|
||||
it deterministically.
|
||||
it deterministically. Chunks use the same source model and carry one canonical
|
||||
reference spanning the first selected unit through the last.
|
||||
|
||||
`pipeline.RunOutput` carries the run manifest, accepted normalized results,
|
||||
rejected results, warnings, checkpoint events, and logical files returned by the
|
||||
@@ -115,13 +116,15 @@ whose results are accepted and used.
|
||||
## Chunk Canonicalization
|
||||
|
||||
Before lane execution, generic validation requires unique chunk IDs, matching
|
||||
source identity, indexes matching returned order, valid ordered boundaries,
|
||||
source identity, indexes matching returned order, a valid canonical reference,
|
||||
non-empty content and media type, and at least one valid source unit per chunk.
|
||||
Units may not repeat inside a chunk and must preserve source-document order.
|
||||
Units may not repeat inside a chunk and must form a contiguous range in
|
||||
source-document order. The chunk reference must exactly match the source and
|
||||
the first and last unit references.
|
||||
|
||||
The runner then rebuilds each chunk's unit slice from the source document by
|
||||
unit ID. It preserves the module-owned boundaries, content, media type, and
|
||||
cloned metadata. The framework permits gaps and overlap between separate
|
||||
unit ID. It preserves the canonical reference, content, media type, and cloned
|
||||
metadata. The framework permits gaps and overlap between separate
|
||||
chunks; stricter coverage policy belongs to the chunk implementation.
|
||||
|
||||
## Validation And Retries
|
||||
|
||||
Reference in New Issue
Block a user