eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Implement integer source units and chunk payloads

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-07 18:34:23 +00:00
parent 4f057b99ac
commit 9e3f8809b3
45 changed files with 618 additions and 451 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
docs/internal/modules.md
View File

@@ -84,9 +84,10 @@ The `generic` chunker splits source units into ordered chunks. It validates the
source document, clones source units, assigns chunk IDs such as `chunk-000001`,
and records chunk metadata for start unit, end unit, and unit count.
The pipeline runner canonicalizes chunk units from the source document by ID
before extractors and mergers run. Chunker-owned context should stay in
`SourceChunk.Metadata`.
The pipeline runner canonicalizes chunk units from the source document by
integer ID before extractors and mergers run. Chunkers also populate chunk
start and end unit IDs, content bytes, and media type. Chunker-owned context
should stay in `SourceChunk.Metadata`.
Options:
@@ -126,12 +127,11 @@ Options: none. Non-empty options are rejected.
The chunker enforces full source-unit coverage from the first source unit to the
last, sequential contiguous scenes, and no overlap. Its LLM-facing schema uses
integer `start_unit_id` and `end_unit_id` values as 1-based source-unit numbers;
the module canonicalizes valid integer references to source-unit IDs before
producing chunks. It assigns chunk IDs such as `scene-000001` and stores scene
metadata including title, primary mode, participants, summary, boundary note,
confidence, boundary unit IDs, and unit count. Boundary caveats become warnings
with reason code
integer `start_unit_id` and `end_unit_id` values matching source-unit IDs. It
assigns chunk IDs such as `scene-000001`, emits JSON chunk content, and stores
scene metadata including title, primary mode, participants, summary, boundary
note, confidence, boundary unit IDs, and unit count. Boundary caveats become
warnings with reason code
`scene_boundary_caveat`. Whitespace-only caveats are treated as malformed
structured output rather than silently dropped.
@@ -150,8 +150,7 @@ input materials, response schema, and session ID to the runtime; converts
spell-cast responses into artifact candidates; and supplies deterministic
validators.
Its LLM-facing source-reference schema uses integer `start_unit_id` and
`end_unit_id` values as 1-based source-unit numbers; the module canonicalizes
valid integer references to source-unit IDs before validation and output.
`end_unit_id` values matching source-unit IDs.
Its prompt definition lives under `assets/prompts` and its schema under
`assets/schemas`. Shared reusable D&D prompt fragments are provided by