notarius/docs/roadmap/implementation.md

Chunk Module Implementation Plan

This plan implements the accepted target state in Chunk Module Roadmap. It is written for an LLM coding agent that will implement each stage in order.

Before beginning any stage, review:

• docs/policy/architecture.md
• docs/policy/development.md
• docs/policy/documentation.md
• docs/roadmap/chunk.md

Do not move planned behavior into non-roadmap docs until the corresponding code is implemented. Do not revert unrelated user changes.

Stage 1: Framework Chunk Contract

Goal: make LLM-backed chunking first-class and enforce generic chunk result invariants without adding any D&D-specific framework behavior.

Code changes:

• Add LLMClient contracts.StructuredLLMClient to contracts.ChunkRequest in internal/framework/contracts/contracts.go.
• Update internal/framework/pipeline/runner.go so the runner passes input.LLMClient to the chunker in contracts.ChunkRequest.
• Add framework-level chunk result validation after chunker.Chunk returns and before lanes execute.
• Keep validation source-generic. The validator should reject:
  • empty chunk ID;
  • duplicate chunk ID;
  • chunk SourceID that does not match the source document ID;
  • chunk Index that does not match returned order;
  • empty chunk units;
  • repeated source unit inside one chunk;
  • source unit not found in the source document;
  • chunk units that do not appear in source-document order.
• The validator must not require complete coverage and must not reject overlap between different chunks.
• Preserve existing warning behavior: append chunker warnings before returning chunk errors, as the runner does today.

Documentation changes:

• Update implemented internal docs under docs/internal/ to define the chunk module API and validation invariants once the code exists.
• Keep examples and user docs unchanged in this stage unless an existing doc becomes inaccurate.

Tests:

• Update contract tests for the new ChunkRequest.LLMClient field where useful.
• Add focused pipeline runner tests for each invalid chunk result case listed above.
• Add runner tests proving partial coverage and overlapping chunks remain accepted.
• Run:

go test ./internal/framework/contracts ./internal/framework/pipeline

Stage completion criteria:

• Existing generic chunking still works.
• A fake chunker can receive the structured LLM client through ChunkRequest.
• Framework tests prove the accepted generic chunk invariants.

Stage 2: D&D Scene Assets And Module Skeleton

Goal: revise the draft D&D scene prompt and schema into module-owned assets and add load/render plumbing without registering production behavior.

Asset decisions:

• Rename internal/modules/chunk/dnd/scenes/assets/schemas/scene_map.schema.json to internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json.
• Use these schema constants unless a code-local naming conflict requires a mechanical adjustment:
  • prompt ID: dnd.scenes
  • response schema key: dnd_scenes
  • response schema ID: notarius.dnd.scenes
  • response schema version: v1
  • response schema name: notarius_dnd_scenes_v1
• Use this structured response shape:

{
  "scenes": [
    {
      "start_unit_id": "seg-001",
      "end_unit_id": "seg-010",
      "short_title": "Ambush at the gate",
      "primary_mode": "Combat",
      "main_participants": ["Aria", "Bandit mage"],
      "summary": "The party fights the bandit mage at the gate.",
      "boundary_note": "The scene begins when combat starts and ends when the immediate threat is resolved.",
      "boundary_confidence": "High"
    }
  ],
  "boundary_caveats": []
}

• Required top-level fields: scenes, boundary_caveats.
• Required scene fields: start_unit_id, end_unit_id, short_title, primary_mode, main_participants, summary, boundary_note, boundary_confidence.
• Boundary fields are source-unit ID strings, not integers.
• primary_mode enum: Recap, Discussion, Combat, Narrative.
• boundary_confidence enum: High, Medium, Low.
• Keep additionalProperties: false throughout the schema.
• Do not include model-authored final chunk IDs or chunk indexes in the schema. The Go module assigns deterministic chunk IDs and indexes.

Prompt decisions:

• Keep D&D-specific scene guidance in the D&D scene module.
• Make the user prompt a Go template similar to the spell extractor prompt.
• Include source document ID and ordered source units.
• Include selected source-unit metadata when present: speaker, start, and end.
• Align prompt terms exactly with schema field names and enum values.
• Keep dnd/scenes module policy explicit in the prompt: full coverage, sequential scenes, no gaps, no overlap, exact source-unit IDs.

Code changes:

• Add assets.go with an embed.FS for prompts and schemas.
• Add schema.go with the constants and a loadResponseSchema function using llm.LoadResponseSchema, following the pattern in internal/modules/extract/dnd/spells/schema.go.
• Add prompt rendering code using framework/prompt.Bundle, following the pattern in internal/modules/extract/dnd/spells/prompt.go.
• Add internal response structs for the schema shape.
• Do not register the module in internal/cli/catalog.go in this stage.

Tests:

• Add tests that the schema loads, is valid JSON, has the expected metadata, and rejects the old integer-boundary assumption through Go-side type expectations.
• Add prompt rendering tests that source unit IDs and selected metadata appear in the rendered user prompt.
• Run:

go test ./internal/modules/chunk/dnd/scenes

Stage completion criteria:

• The scene schema and prompts are loadable embedded assets.
• The prompt/schema terminology is internally consistent.
• No production catalog behavior changes yet.

Stage 3: D&D Scene Chunker Implementation

Goal: implement dnd/scenes as a contract-compliant chunk module with strict module-owned validation.

Module decisions:

• Package path: internal/modules/chunk/dnd/scenes.
• Package name: scenes.
• Module key: dnd/scenes.
• ModuleSpec:
  • Stage: pipeline.StageChunk
  • Requires: source.transcript
  • Provides: chunks, chunks.scenes
• Constructor: New() *Chunker.
• Registration function: Register(registry *pipeline.ChunkerRegistry) error.
• No module options initially. Reject non-empty options with an actionable module-prefixed error unless a clear option is implemented in the same stage.

Chunking behavior:

• Validate context.Context, source document, non-empty source units, and non-nil LLMClient.
• Render the scene prompt over the full source document.
• Call LLMClient.CompleteStructured with:
  • StageName: dnd/scenes
  • response schema name and schema JSON from the module schema loader.
• Validate the decoded response before producing chunks:
  • scenes must be present and non-empty;
  • every boundary ID must exist in the source document;
  • each scene start must be at or before its end;
  • the first scene starts at the first source unit;
  • the final scene ends at the final source unit;
  • scenes are contiguous in source order;
  • scenes do not overlap;
  • required metadata fields are non-empty after trimming;
  • main_participants entries are trimmed and empty entries rejected.
• Assign deterministic chunk fields:
  • ID: scene-000001, scene-000002, and so on;
  • SourceID: source document ID;
  • Index: zero-based returned order;
  • Units: defensive copies of the source units in the scene range.
• Store per-scene metadata on each chunk:
  • scene_title
  • primary_mode
  • main_participants
  • summary
  • boundary_note
  • boundary_confidence
  • start_unit_id
  • end_unit_id
  • unit_count
• Convert each boundary_caveats entry into a contracts.Warning with:
  • Scope: dnd/scenes
  • ReasonCode: scene_boundary_caveat
  • Message: the caveat text.
• Fail explicitly for malformed model output. Do not fall back to generic.
• Implement contracts.ManifestMetadataProvider and include prompt and response-schema provenance without raw prompts, raw schemas, source text, or secrets.

Tests:

• Registration and ModuleSpec.
• Successful chunking from a fake LLM response.
• Prompt request uses the expected schema name and schema JSON.
• Caveats become warnings.
• Defensive copy behavior for source units and metadata.
• Errors for nil context, nil source, invalid source, nil LLM client, empty model scenes, unknown boundary ID, out-of-order boundaries, gaps, overlap, incomplete coverage, empty metadata fields, and non-empty unsupported options.
• Manifest metadata contains prompt/schema provenance.
• Run:

go test ./internal/modules/chunk/dnd/scenes
go test ./internal/framework/pipeline

Stage completion criteria:

• dnd/scenes works in focused tests with fake LLM clients.
• It is still not production-registered unless Stage 4 is completed.

Stage 4: Production Registration And Implemented Docs

Goal: make dnd/scenes available in production configuration and document only the behavior that now exists.

Code changes:

• Register dnd/scenes in internal/cli/catalog.go.
• Add or update catalog/default module tests so the production catalog exposes the new chunk module.
• Add CLI/config validation tests proving a pipeline can select chunk: dnd/scenes.
• Do not change the existing maintained example config unless the related CLI fixture tests are updated to keep it loadable and useful.

Documentation changes:

• Update docs/config.md implemented production module tables and chunk module notes.
• Update docs/cli.md implemented production module list.
• Update docs/internal/modules.md with dnd/scenes behavior, capabilities, metadata, and failure policy.
• Update or add internal chunk-module documentation if Stage 1 did not already create a clear API reference.
• Update docs/troubleshooting.md for common scene chunker failures: malformed model output, invalid boundaries, incomplete coverage, and provider failures during chunking.
• Keep roadmap docs for any deferred options or future prompt tuning.

Tests:

go test ./internal/cli
go test ./internal/core/config
go test ./internal/framework/pipeline
go test ./internal/modules/chunk/dnd/scenes

Stage completion criteria:

• Config resolution can bind dnd/scenes.
• User and internal docs describe the implemented module accurately.
• Existing examples and CLI docs remain truthful.

Stage 5: Full Verification

Goal: verify the complete feature across contracts, production wiring, docs, and the command entry point.

Run:

go test ./...
go vet ./...
go build ./cmd/notarius

Inspect diagnostics-sensitive output manually in tests or fixtures where relevant:

• no raw prompts, source text, provider payloads, API keys, or secrets in manifest metadata;
• errors name the module and operation;
• warnings are preserved in RunOutput.Warnings;
• run manifests record the dnd/scenes chunker when selected.

Stage completion criteria:

• Full validation commands pass.
• The feature is documented as implemented only where code supports it.
• docs/roadmap/chunk.md retains target-state context and does not duplicate current-behavior reference material.