105 lines
4.7 KiB
Markdown
105 lines
4.7 KiB
Markdown
# Chunk Module Roadmap
|
|
|
|
Current Notarius behavior is documented in the canonical README, CLI,
|
|
configuration, operations, internal, and integration docs. This roadmap records
|
|
future chunk-module behavior only.
|
|
|
|
## Goal
|
|
|
|
Chunk modules should be a clear module-author boundary, and LLM-backed chunking
|
|
should be a first-class capability.
|
|
|
|
The immediate target is a D&D-specific scene chunker that divides transcript
|
|
source units into coherent scenes before extraction. The broader target is that
|
|
any chunk module can be implemented as a black box when it satisfies the
|
|
framework chunk contract.
|
|
|
|
## Target Chunk Contract
|
|
|
|
The framework chunk contract should support deterministic and LLM-backed
|
|
chunkers through the same module interface.
|
|
|
|
Chunkers should receive runtime dependencies from the runner, including the
|
|
structured LLM client when a chunker needs model calls. Chunkers should not
|
|
construct provider clients internally.
|
|
|
|
The framework should validate these result invariants for every chunk module:
|
|
|
|
- chunk IDs are non-empty and unique within a run;
|
|
- each chunk references the input source document ID;
|
|
- chunk indexes are deterministic and sequential in returned order;
|
|
- each chunk contains at least one source unit;
|
|
- each chunk source unit comes from the source document;
|
|
- source units within each chunk appear in source-document order.
|
|
|
|
The framework should not require complete source-unit coverage and should not
|
|
forbid overlap between chunks. Individual chunk modules may enforce stricter
|
|
policies, such as full coverage or non-overlap, when those policies are part of
|
|
the module's own contract.
|
|
|
|
Chunk metadata should remain flexible and module-owned. Framework code should
|
|
preserve chunk metadata and pass it to downstream modules, but it should not
|
|
adopt transcript-specific or D&D-specific metadata fields.
|
|
|
|
## D&D Scene Chunker Target
|
|
|
|
The D&D scene chunker should live under
|
|
`internal/modules/chunk/dnd/scenes` and use the module key `dnd/scenes`.
|
|
|
|
It should require transcript source capabilities and provide the generic
|
|
`chunks` capability plus a scene-specific chunk capability. D&D scene-boundary
|
|
prompt logic, response-schema interpretation, and stricter scene policies belong
|
|
inside the module.
|
|
|
|
The module should use the framework structured LLM client for scene-boundary
|
|
detection. The model response should describe source-unit boundaries and useful
|
|
scene metadata; the Go module should validate the response and convert it into
|
|
`contracts.SourceChunk` values.
|
|
|
|
For `dnd/scenes`, the module-owned policy should be:
|
|
|
|
- cover the full source document from first source unit to last source unit;
|
|
- return sequential, contiguous, non-overlapping scenes;
|
|
- use exact source-unit IDs for boundaries;
|
|
- fail with actionable errors for malformed model output rather than silently
|
|
falling back to a generic chunker.
|
|
|
|
Scene chunk IDs and indexes should be assigned by the module, not trusted from
|
|
model output. Useful scene information should be stored in chunk metadata, such
|
|
as title, primary mode, participants, summary, boundary note, and boundary
|
|
confidence. Overall boundary caveats should be surfaced as chunker warnings.
|
|
|
|
## Draft Asset Target
|
|
|
|
Initial D&D scene chunker prompt and schema drafts exist under
|
|
`internal/modules/chunk/dnd/scenes/assets`. They should be revised before the
|
|
module is implemented.
|
|
|
|
The response schema should be versioned and named consistently with existing
|
|
module-owned response schemas, such as `dnd_scenes.v1.json`, with a schema key,
|
|
schema ID, schema version, and OpenAI-compatible response schema name.
|
|
|
|
Boundary fields should use source-unit ID strings, not integer segment IDs.
|
|
The schema should focus on boundary and metadata decisions rather than final
|
|
framework chunk fields. Prompt terminology and schema terminology should match
|
|
exactly, including primary mode enum values and boundary field names.
|
|
|
|
The user prompt should be a Go template that includes source document ID,
|
|
chunking scope, ordered source units, and selected metadata such as speaker and
|
|
timestamps when available. It may contain D&D-specific scene guidance, but it
|
|
should not imply that the framework itself is transcript-specific.
|
|
|
|
## Documentation Target
|
|
|
|
Current-behavior docs should be updated only after the corresponding behavior is
|
|
implemented.
|
|
|
|
Internal module-author documentation should eventually define the chunk module
|
|
API, including `Chunker`, `ChunkRequest`, `ChunkResult`, `SourceChunk`,
|
|
validation invariants, warning semantics, LLM-backed chunker expectations,
|
|
module specs, capability guidance, and option parsing expectations.
|
|
|
|
When `dnd/scenes` becomes production behavior, configuration, CLI, internal
|
|
module, and troubleshooting docs should describe the implemented module and its
|
|
failure modes.
|