Add a roadmap to implement a D&D-specific chunk module
This commit is contained in:
104
docs/roadmap/chunk.md
Normal file
104
docs/roadmap/chunk.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Chunk Module Roadmap
|
||||
|
||||
Current Notarius behavior is documented in the canonical README, CLI,
|
||||
configuration, operations, internal, and integration docs. This roadmap records
|
||||
future chunk-module behavior only.
|
||||
|
||||
## Goal
|
||||
|
||||
Chunk modules should be a clear module-author boundary, and LLM-backed chunking
|
||||
should be a first-class capability.
|
||||
|
||||
The immediate target is a D&D-specific scene chunker that divides transcript
|
||||
source units into coherent scenes before extraction. The broader target is that
|
||||
any chunk module can be implemented as a black box when it satisfies the
|
||||
framework chunk contract.
|
||||
|
||||
## Target Chunk Contract
|
||||
|
||||
The framework chunk contract should support deterministic and LLM-backed
|
||||
chunkers through the same module interface.
|
||||
|
||||
Chunkers should receive runtime dependencies from the runner, including the
|
||||
structured LLM client when a chunker needs model calls. Chunkers should not
|
||||
construct provider clients internally.
|
||||
|
||||
The framework should validate these result invariants for every chunk module:
|
||||
|
||||
- chunk IDs are non-empty and unique within a run;
|
||||
- each chunk references the input source document ID;
|
||||
- chunk indexes are deterministic and sequential in returned order;
|
||||
- each chunk contains at least one source unit;
|
||||
- each chunk source unit comes from the source document;
|
||||
- source units within each chunk appear in source-document order.
|
||||
|
||||
The framework should not require complete source-unit coverage and should not
|
||||
forbid overlap between chunks. Individual chunk modules may enforce stricter
|
||||
policies, such as full coverage or non-overlap, when those policies are part of
|
||||
the module's own contract.
|
||||
|
||||
Chunk metadata should remain flexible and module-owned. Framework code should
|
||||
preserve chunk metadata and pass it to downstream modules, but it should not
|
||||
adopt transcript-specific or D&D-specific metadata fields.
|
||||
|
||||
## D&D Scene Chunker Target
|
||||
|
||||
The D&D scene chunker should live under
|
||||
`internal/modules/chunk/dnd/scenes` and use the module key `dnd/scenes`.
|
||||
|
||||
It should require transcript source capabilities and provide the generic
|
||||
`chunks` capability plus a scene-specific chunk capability. D&D scene-boundary
|
||||
prompt logic, response-schema interpretation, and stricter scene policies belong
|
||||
inside the module.
|
||||
|
||||
The module should use the framework structured LLM client for scene-boundary
|
||||
detection. The model response should describe source-unit boundaries and useful
|
||||
scene metadata; the Go module should validate the response and convert it into
|
||||
`contracts.SourceChunk` values.
|
||||
|
||||
For `dnd/scenes`, the module-owned policy should be:
|
||||
|
||||
- cover the full source document from first source unit to last source unit;
|
||||
- return sequential, contiguous, non-overlapping scenes;
|
||||
- use exact source-unit IDs for boundaries;
|
||||
- fail with actionable errors for malformed model output rather than silently
|
||||
falling back to a generic chunker.
|
||||
|
||||
Scene chunk IDs and indexes should be assigned by the module, not trusted from
|
||||
model output. Useful scene information should be stored in chunk metadata, such
|
||||
as title, primary mode, participants, summary, boundary note, and boundary
|
||||
confidence. Overall boundary caveats should be surfaced as chunker warnings.
|
||||
|
||||
## Draft Asset Target
|
||||
|
||||
Initial D&D scene chunker prompt and schema drafts exist under
|
||||
`internal/modules/chunk/dnd/scenes/assets`. They should be revised before the
|
||||
module is implemented.
|
||||
|
||||
The response schema should be versioned and named consistently with existing
|
||||
module-owned response schemas, such as `dnd_scenes.v1.json`, with a schema key,
|
||||
schema ID, schema version, and OpenAI-compatible response schema name.
|
||||
|
||||
Boundary fields should use source-unit ID strings, not integer segment IDs.
|
||||
The schema should focus on boundary and metadata decisions rather than final
|
||||
framework chunk fields. Prompt terminology and schema terminology should match
|
||||
exactly, including primary mode enum values and boundary field names.
|
||||
|
||||
The user prompt should be a Go template that includes source document ID,
|
||||
chunking scope, ordered source units, and selected metadata such as speaker and
|
||||
timestamps when available. It may contain D&D-specific scene guidance, but it
|
||||
should not imply that the framework itself is transcript-specific.
|
||||
|
||||
## Documentation Target
|
||||
|
||||
Current-behavior docs should be updated only after the corresponding behavior is
|
||||
implemented.
|
||||
|
||||
Internal module-author documentation should eventually define the chunk module
|
||||
API, including `Chunker`, `ChunkRequest`, `ChunkResult`, `SourceChunk`,
|
||||
validation invariants, warning semantics, LLM-backed chunker expectations,
|
||||
module specs, capability guidance, and option parsing expectations.
|
||||
|
||||
When `dnd/scenes` becomes production behavior, configuration, CLI, internal
|
||||
module, and troubleshooting docs should describe the implemented module and its
|
||||
failure modes.
|
||||
Reference in New Issue
Block a user