eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Move chunks into the canonical source model

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 05:32:11 +00:00
parent 40709e4ad8
commit 075888c97f
33 changed files with 548 additions and 325 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
docs/internal/modules.md
View File

@@ -57,7 +57,8 @@ rules are defined in the
The generic chunker validates the source document, walks units in configured
windows, clones each selected unit, and emits deterministic ordered chunk IDs.
Overlap changes the next window start but never reorders units. It records the
first and last unit and unit count in chunk metadata.
first and last unit and unit count in chunk metadata, and derives the chunk's
canonical source reference from those unit references.
The accepted options and defaults are defined in
[Configuration](../config.md#implemented-production-modules). Generic
@@ -68,7 +69,7 @@ framework validation canonicalizes the returned unit slices before extraction.
The scene chunker prepares a structured Scriptorium request from the full
transcript, session, and optional D&D reference inputs. It validates the model's
scene boundaries against source-unit IDs and converts them into deterministic
chunks.
chunks with canonical source references spanning each scene's units.
Scene validation requires sequential, contiguous, non-overlapping coverage from
the first source unit through the last. Each chunk contains JSON scene content