notarius/docs/roadmap/chunk.md

Chunk Module Roadmap

Current Notarius behavior is documented in the canonical README, CLI, configuration, operations, internal, and integration docs. This roadmap records future chunk-module behavior only.

Goal

Chunk modules should be a clear module-author boundary, and LLM-backed chunking should be a first-class capability.

The immediate target is a D&D-specific scene chunker that divides transcript source units into coherent scenes before extraction. The broader target is that any chunk module can be implemented as a black box when it satisfies the framework chunk contract.

Target Chunk Contract

The framework chunk contract should support deterministic and LLM-backed chunkers through the same module interface.

Chunkers should receive runtime dependencies from the runner, including the structured LLM client when a chunker needs model calls. Chunkers should not construct provider clients internally.

The framework should validate these result invariants for every chunk module:

• chunk IDs are non-empty and unique within a run;
• each chunk references the input source document ID;
• chunk indexes are deterministic and sequential in returned order;
• each chunk contains at least one source unit;
• each chunk source unit comes from the source document;
• source units within each chunk appear in source-document order.

The framework should not require complete source-unit coverage and should not forbid overlap between chunks. Individual chunk modules may enforce stricter policies, such as full coverage or non-overlap, when those policies are part of the module's own contract.

Chunk metadata should remain flexible and module-owned. Framework code should preserve chunk metadata and pass it to downstream modules, but it should not adopt transcript-specific or D&D-specific metadata fields.

D&D Scene Chunker Target

The D&D scene chunker should live under internal/modules/chunk/dnd/scenes and use the module key dnd/scenes.

It should require transcript source capabilities and provide the generic chunks capability plus a scene-specific chunk capability. D&D scene-boundary prompt logic, response-schema interpretation, and stricter scene policies belong inside the module.

The module should use the framework structured LLM client for scene-boundary detection. The model response should describe source-unit boundaries and useful scene metadata; the Go module should validate the response and convert it into contracts.SourceChunk values.

For dnd/scenes, the module-owned policy should be:

• cover the full source document from first source unit to last source unit;
• return sequential, contiguous, non-overlapping scenes;
• use exact source-unit IDs for boundaries;
• fail with actionable errors for malformed model output rather than silently falling back to a generic chunker.

Scene chunk IDs and indexes should be assigned by the module, not trusted from model output. Useful scene information should be stored in chunk metadata, such as title, primary mode, participants, summary, boundary note, and boundary confidence. Overall boundary caveats should be surfaced as chunker warnings.

Draft Asset Target

Initial D&D scene chunker prompt and schema drafts exist under internal/modules/chunk/dnd/scenes/assets. They should be revised before the module is implemented.

The response schema should be versioned and named consistently with existing module-owned response schemas, such as dnd_scenes.v1.json, with a schema key, schema ID, schema version, and OpenAI-compatible response schema name.

Boundary fields should use source-unit ID strings, not integer segment IDs. The schema should focus on boundary and metadata decisions rather than final framework chunk fields. Prompt terminology and schema terminology should match exactly, including primary mode enum values and boundary field names.

The user prompt should be a Go template that includes source document ID, chunking scope, ordered source units, and selected metadata such as speaker and timestamps when available. It may contain D&D-specific scene guidance, but it should not imply that the framework itself is transcript-specific.

Documentation Target

Current-behavior docs should be updated only after the corresponding behavior is implemented.

Internal module-author documentation should eventually define the chunk module API, including Chunker, ChunkRequest, ChunkResult, SourceChunk, validation invariants, warning semantics, LLM-backed chunker expectations, module specs, capability guidance, and option parsing expectations.

When dnd/scenes becomes production behavior, configuration, CLI, internal module, and troubleshooting docs should describe the implemented module and its failure modes.