eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make chunk validation a framework contract

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-04 12:52:16 +00:00
parent b95af4f87d
commit e5eb0ba5c8
8 changed files with 249 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/internal/modules.md
View File

@@ -20,6 +20,11 @@ A production module package should provide:
Module specs should describe capabilities accurately. Resolution uses specs to
reject incompatible pipelines before execution.
Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
when they need model-backed chunking. The pipeline runner validates generic
chunk result invariants before extraction; module-owned policies may be stricter
but must stay within the module package.
## `seriatim` Input
Package: `internal/modules/input/seriatim`

30
docs/internal/pipeline.md
View File

@@ -73,8 +73,34 @@ The runner:
2. builds the input adapter and parses the raw input into a source document;
3. validates the source document;
4. builds the chunker and produces source chunks;
5. runs each selected artifact lane in sorted resolved order;
6. builds the output encoder and validates logical output file names.
5. validates source chunks against framework invariants;
6. runs each selected artifact lane in sorted resolved order;
7. builds the output encoder and validates logical output file names.
## Chunk Results
Chunkers implement `contracts.Chunker` and receive a `contracts.ChunkRequest`
with the validated source document, the structured LLM client, the configured
LLM profile, module options, and run metadata. Deterministic and LLM-backed
chunkers use the same contract; provider construction stays outside chunk
modules.
After `Chunk` returns, the runner appends chunker warnings before returning any
chunker error. When chunking succeeds, the runner validates generic chunk
invariants before running extractors:
- chunk IDs must be non-empty and unique in the chunk result;
- each chunk `SourceID` must match the source document ID;
- each chunk `Index` must match its zero-based returned order;
- each chunk must contain at least one source unit;
- a chunk must not repeat a source unit;
- every chunk source unit must exist in the source document;
- source units inside each chunk must appear in source-document order.
The framework does not require complete source-unit coverage and does not reject
overlap between different chunks. Stricter policies, such as full coverage or
non-overlap, belong to individual chunk modules when they are part of that
module's contract.
Within an artifact lane, the runner: