Add a roadmap to implement a D&D-specific chunk module

2026-07-04 07:48:36 -05:00
parent 11073b613c
commit b95af4f87d
5 changed files with 628 additions and 0 deletions
--- a/docs/roadmap/chunk.md
+++ b/docs/roadmap/chunk.md
@@ -0,0 +1,104 @@
+# Chunk Module Roadmap
+
+Current Notarius behavior is documented in the canonical README, CLI,
+configuration, operations, internal, and integration docs. This roadmap records
+future chunk-module behavior only.
+
+## Goal
+
+Chunk modules should be a clear module-author boundary, and LLM-backed chunking
+should be a first-class capability.
+
+The immediate target is a D&D-specific scene chunker that divides transcript
+source units into coherent scenes before extraction. The broader target is that
+any chunk module can be implemented as a black box when it satisfies the
+framework chunk contract.
+
+## Target Chunk Contract
+
+The framework chunk contract should support deterministic and LLM-backed
+chunkers through the same module interface.
+
+Chunkers should receive runtime dependencies from the runner, including the
+structured LLM client when a chunker needs model calls. Chunkers should not
+construct provider clients internally.
+
+The framework should validate these result invariants for every chunk module:
+
+- chunk IDs are non-empty and unique within a run;
+- each chunk references the input source document ID;
+- chunk indexes are deterministic and sequential in returned order;
+- each chunk contains at least one source unit;
+- each chunk source unit comes from the source document;
+- source units within each chunk appear in source-document order.
+
+The framework should not require complete source-unit coverage and should not
+forbid overlap between chunks. Individual chunk modules may enforce stricter
+policies, such as full coverage or non-overlap, when those policies are part of
+the module's own contract.
+
+Chunk metadata should remain flexible and module-owned. Framework code should
+preserve chunk metadata and pass it to downstream modules, but it should not
+adopt transcript-specific or D&D-specific metadata fields.
+
+## D&D Scene Chunker Target
+
+The D&D scene chunker should live under
+`internal/modules/chunk/dnd/scenes` and use the module key `dnd/scenes`.
+
+It should require transcript source capabilities and provide the generic
+`chunks` capability plus a scene-specific chunk capability. D&D scene-boundary
+prompt logic, response-schema interpretation, and stricter scene policies belong
+inside the module.
+
+The module should use the framework structured LLM client for scene-boundary
+detection. The model response should describe source-unit boundaries and useful
+scene metadata; the Go module should validate the response and convert it into
+`contracts.SourceChunk` values.
+
+For `dnd/scenes`, the module-owned policy should be:
+
+- cover the full source document from first source unit to last source unit;
+- return sequential, contiguous, non-overlapping scenes;
+- use exact source-unit IDs for boundaries;
+- fail with actionable errors for malformed model output rather than silently
+  falling back to a generic chunker.
+
+Scene chunk IDs and indexes should be assigned by the module, not trusted from
+model output. Useful scene information should be stored in chunk metadata, such
+as title, primary mode, participants, summary, boundary note, and boundary
+confidence. Overall boundary caveats should be surfaced as chunker warnings.
+
+## Draft Asset Target
+
+Initial D&D scene chunker prompt and schema drafts exist under
+`internal/modules/chunk/dnd/scenes/assets`. They should be revised before the
+module is implemented.
+
+The response schema should be versioned and named consistently with existing
+module-owned response schemas, such as `dnd_scenes.v1.json`, with a schema key,
+schema ID, schema version, and OpenAI-compatible response schema name.
+
+Boundary fields should use source-unit ID strings, not integer segment IDs.
+The schema should focus on boundary and metadata decisions rather than final
+framework chunk fields. Prompt terminology and schema terminology should match
+exactly, including primary mode enum values and boundary field names.
+
+The user prompt should be a Go template that includes source document ID,
+chunking scope, ordered source units, and selected metadata such as speaker and
+timestamps when available. It may contain D&D-specific scene guidance, but it
+should not imply that the framework itself is transcript-specific.
+
+## Documentation Target
+
+Current-behavior docs should be updated only after the corresponding behavior is
+implemented.
+
+Internal module-author documentation should eventually define the chunk module
+API, including `Chunker`, `ChunkRequest`, `ChunkResult`, `SourceChunk`,
+validation invariants, warning semantics, LLM-backed chunker expectations,
+module specs, capability guidance, and option parsing expectations.
+
+When `dnd/scenes` becomes production behavior, configuration, CLI, internal
+module, and troubleshooting docs should describe the implemented module and its
+failure modes.
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,312 @@
+# Chunk Module Implementation Plan
+
+This plan implements the accepted target state in
+[Chunk Module Roadmap](chunk.md). It is written for an LLM coding agent that
+will implement each stage in order.
+
+Before beginning any stage, review:
+
+- `docs/policy/architecture.md`
+- `docs/policy/development.md`
+- `docs/policy/documentation.md`
+- `docs/roadmap/chunk.md`
+
+Do not move planned behavior into non-roadmap docs until the corresponding code
+is implemented. Do not revert unrelated user changes.
+
+## Stage 1: Framework Chunk Contract
+
+Goal: make LLM-backed chunking first-class and enforce generic chunk result
+invariants without adding any D&D-specific framework behavior.
+
+Code changes:
+
+- Add `LLMClient contracts.StructuredLLMClient` to `contracts.ChunkRequest` in
+  `internal/framework/contracts/contracts.go`.
+- Update `internal/framework/pipeline/runner.go` so the runner passes
+  `input.LLMClient` to the chunker in `contracts.ChunkRequest`.
+- Add framework-level chunk result validation after `chunker.Chunk` returns and
+  before lanes execute.
+- Keep validation source-generic. The validator should reject:
+  - empty chunk ID;
+  - duplicate chunk ID;
+  - chunk `SourceID` that does not match the source document ID;
+  - chunk `Index` that does not match returned order;
+  - empty chunk units;
+  - repeated source unit inside one chunk;
+  - source unit not found in the source document;
+  - chunk units that do not appear in source-document order.
+- The validator must not require complete coverage and must not reject overlap
+  between different chunks.
+- Preserve existing warning behavior: append chunker warnings before returning
+  chunk errors, as the runner does today.
+
+Documentation changes:
+
+- Update implemented internal docs under `docs/internal/` to define the chunk
+  module API and validation invariants once the code exists.
+- Keep examples and user docs unchanged in this stage unless an existing doc
+  becomes inaccurate.
+
+Tests:
+
+- Update contract tests for the new `ChunkRequest.LLMClient` field where useful.
+- Add focused pipeline runner tests for each invalid chunk result case listed
+  above.
+- Add runner tests proving partial coverage and overlapping chunks remain
+  accepted.
+- Run:
+
+```sh
+go test ./internal/framework/contracts ./internal/framework/pipeline
+```
+
+Stage completion criteria:
+
+- Existing generic chunking still works.
+- A fake chunker can receive the structured LLM client through `ChunkRequest`.
+- Framework tests prove the accepted generic chunk invariants.
+
+## Stage 2: D&D Scene Assets And Module Skeleton
+
+Goal: revise the draft D&D scene prompt and schema into module-owned assets and
+add load/render plumbing without registering production behavior.
+
+Asset decisions:
+
+- Rename `internal/modules/chunk/dnd/scenes/assets/schemas/scene_map.schema.json`
+  to `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`.
+- Use these schema constants unless a code-local naming conflict requires a
+  mechanical adjustment:
+  - prompt ID: `dnd.scenes`
+  - response schema key: `dnd_scenes`
+  - response schema ID: `notarius.dnd.scenes`
+  - response schema version: `v1`
+  - response schema name: `notarius_dnd_scenes_v1`
+- Use this structured response shape:
+
+```json
+{
+  "scenes": [
+    {
+      "start_unit_id": "seg-001",
+      "end_unit_id": "seg-010",
+      "short_title": "Ambush at the gate",
+      "primary_mode": "Combat",
+      "main_participants": ["Aria", "Bandit mage"],
+      "summary": "The party fights the bandit mage at the gate.",
+      "boundary_note": "The scene begins when combat starts and ends when the immediate threat is resolved.",
+      "boundary_confidence": "High"
+    }
+  ],
+  "boundary_caveats": []
+}
+```
+
+- Required top-level fields: `scenes`, `boundary_caveats`.
+- Required scene fields: `start_unit_id`, `end_unit_id`, `short_title`,
+  `primary_mode`, `main_participants`, `summary`, `boundary_note`,
+  `boundary_confidence`.
+- Boundary fields are source-unit ID strings, not integers.
+- `primary_mode` enum: `Recap`, `Discussion`, `Combat`, `Narrative`.
+- `boundary_confidence` enum: `High`, `Medium`, `Low`.
+- Keep `additionalProperties: false` throughout the schema.
+- Do not include model-authored final chunk IDs or chunk indexes in the schema.
+  The Go module assigns deterministic chunk IDs and indexes.
+
+Prompt decisions:
+
+- Keep D&D-specific scene guidance in the D&D scene module.
+- Make the user prompt a Go template similar to the spell extractor prompt.
+- Include source document ID and ordered source units.
+- Include selected source-unit metadata when present: `speaker`, `start`, and
+  `end`.
+- Align prompt terms exactly with schema field names and enum values.
+- Keep `dnd/scenes` module policy explicit in the prompt: full coverage,
+  sequential scenes, no gaps, no overlap, exact source-unit IDs.
+
+Code changes:
+
+- Add `assets.go` with an `embed.FS` for prompts and schemas.
+- Add `schema.go` with the constants and a `loadResponseSchema` function using
+  `llm.LoadResponseSchema`, following the pattern in
+  `internal/modules/extract/dnd/spells/schema.go`.
+- Add prompt rendering code using `framework/prompt.Bundle`, following the
+  pattern in `internal/modules/extract/dnd/spells/prompt.go`.
+- Add internal response structs for the schema shape.
+- Do not register the module in `internal/cli/catalog.go` in this stage.
+
+Tests:
+
+- Add tests that the schema loads, is valid JSON, has the expected metadata, and
+  rejects the old integer-boundary assumption through Go-side type expectations.
+- Add prompt rendering tests that source unit IDs and selected metadata appear
+  in the rendered user prompt.
+- Run:
+
+```sh
+go test ./internal/modules/chunk/dnd/scenes
+```
+
+Stage completion criteria:
+
+- The scene schema and prompts are loadable embedded assets.
+- The prompt/schema terminology is internally consistent.
+- No production catalog behavior changes yet.
+
+## Stage 3: D&D Scene Chunker Implementation
+
+Goal: implement `dnd/scenes` as a contract-compliant chunk module with strict
+module-owned validation.
+
+Module decisions:
+
+- Package path: `internal/modules/chunk/dnd/scenes`.
+- Package name: `scenes`.
+- Module key: `dnd/scenes`.
+- `ModuleSpec`:
+  - `Stage`: `pipeline.StageChunk`
+  - `Requires`: `source.transcript`
+  - `Provides`: `chunks`, `chunks.scenes`
+- Constructor: `New() *Chunker`.
+- Registration function: `Register(registry *pipeline.ChunkerRegistry) error`.
+- No module options initially. Reject non-empty options with an actionable
+  module-prefixed error unless a clear option is implemented in the same stage.
+
+Chunking behavior:
+
+- Validate `context.Context`, source document, non-empty source units, and
+  non-nil `LLMClient`.
+- Render the scene prompt over the full source document.
+- Call `LLMClient.CompleteStructured` with:
+  - `StageName`: `dnd/scenes`
+  - response schema name and schema JSON from the module schema loader.
+- Validate the decoded response before producing chunks:
+  - `scenes` must be present and non-empty;
+  - every boundary ID must exist in the source document;
+  - each scene start must be at or before its end;
+  - the first scene starts at the first source unit;
+  - the final scene ends at the final source unit;
+  - scenes are contiguous in source order;
+  - scenes do not overlap;
+  - required metadata fields are non-empty after trimming;
+  - `main_participants` entries are trimmed and empty entries rejected.
+- Assign deterministic chunk fields:
+  - `ID`: `scene-000001`, `scene-000002`, and so on;
+  - `SourceID`: source document ID;
+  - `Index`: zero-based returned order;
+  - `Units`: defensive copies of the source units in the scene range.
+- Store per-scene metadata on each chunk:
+  - `scene_title`
+  - `primary_mode`
+  - `main_participants`
+  - `summary`
+  - `boundary_note`
+  - `boundary_confidence`
+  - `start_unit_id`
+  - `end_unit_id`
+  - `unit_count`
+- Convert each `boundary_caveats` entry into a `contracts.Warning` with:
+  - `Scope`: `dnd/scenes`
+  - `ReasonCode`: `scene_boundary_caveat`
+  - `Message`: the caveat text.
+- Fail explicitly for malformed model output. Do not fall back to `generic`.
+- Implement `contracts.ManifestMetadataProvider` and include prompt and
+  response-schema provenance without raw prompts, raw schemas, source text, or
+  secrets.
+
+Tests:
+
+- Registration and `ModuleSpec`.
+- Successful chunking from a fake LLM response.
+- Prompt request uses the expected schema name and schema JSON.
+- Caveats become warnings.
+- Defensive copy behavior for source units and metadata.
+- Errors for nil context, nil source, invalid source, nil LLM client, empty
+  model scenes, unknown boundary ID, out-of-order boundaries, gaps, overlap,
+  incomplete coverage, empty metadata fields, and non-empty unsupported options.
+- Manifest metadata contains prompt/schema provenance.
+- Run:
+
+```sh
+go test ./internal/modules/chunk/dnd/scenes
+go test ./internal/framework/pipeline
+```
+
+Stage completion criteria:
+
+- `dnd/scenes` works in focused tests with fake LLM clients.
+- It is still not production-registered unless Stage 4 is completed.
+
+## Stage 4: Production Registration And Implemented Docs
+
+Goal: make `dnd/scenes` available in production configuration and document only
+the behavior that now exists.
+
+Code changes:
+
+- Register `dnd/scenes` in `internal/cli/catalog.go`.
+- Add or update catalog/default module tests so the production catalog exposes
+  the new chunk module.
+- Add CLI/config validation tests proving a pipeline can select
+  `chunk: dnd/scenes`.
+- Do not change the existing maintained example config unless the related CLI
+  fixture tests are updated to keep it loadable and useful.
+
+Documentation changes:
+
+- Update `docs/config.md` implemented production module tables and chunk module
+  notes.
+- Update `docs/cli.md` implemented production module list.
+- Update `docs/internal/modules.md` with `dnd/scenes` behavior, capabilities,
+  metadata, and failure policy.
+- Update or add internal chunk-module documentation if Stage 1 did not already
+  create a clear API reference.
+- Update `docs/troubleshooting.md` for common scene chunker failures:
+  malformed model output, invalid boundaries, incomplete coverage, and provider
+  failures during chunking.
+- Keep roadmap docs for any deferred options or future prompt tuning.
+
+Tests:
+
+```sh
+go test ./internal/cli
+go test ./internal/core/config
+go test ./internal/framework/pipeline
+go test ./internal/modules/chunk/dnd/scenes
+```
+
+Stage completion criteria:
+
+- Config resolution can bind `dnd/scenes`.
+- User and internal docs describe the implemented module accurately.
+- Existing examples and CLI docs remain truthful.
+
+## Stage 5: Full Verification
+
+Goal: verify the complete feature across contracts, production wiring, docs, and
+the command entry point.
+
+Run:
+
+```sh
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
+
+Inspect diagnostics-sensitive output manually in tests or fixtures where
+relevant:
+
+- no raw prompts, source text, provider payloads, API keys, or secrets in
+  manifest metadata;
+- errors name the module and operation;
+- warnings are preserved in `RunOutput.Warnings`;
+- run manifests record the `dnd/scenes` chunker when selected.
+
+Stage completion criteria:
+
+- Full validation commands pass.
+- The feature is documented as implemented only where code supports it.
+- `docs/roadmap/chunk.md` retains target-state context and does not duplicate
+  current-behavior reference material.