notarius/docs/roadmap/minimal-dnd-scene-chunking.md

# Minimal D&D Scene Chunking

Status: Implemented

## Purpose

Narrow the `dnd/scenes` chunk module to one responsibility: divide a transcript
into coherent, contiguous Dungeons & Dragons scenes. The current model response
also asks for titles, modes, participants, summaries, boundary explanations,
confidence labels, and plan-wide caveats. Those fields increase prompt and
response complexity, overlap with dedicated extraction artifacts, and are not
needed to materialize or validate chunks.

This change applies the same minimal-contract policy used by the D&D extraction
lanes. The model should propose only facts that require model judgment and that
the chunking operation consumes. Notarius should derive or validate everything
else deterministically.

## Desired End State

The `dnd/scenes` chunker asks the model for an ordered, non-empty list of
inclusive source-unit ranges. Each range represents one coherent scene and
contains exactly:

- `start_unit_id`; and
- `end_unit_id`.

The response contains no scene title, summary, kind, participant list,
boundary note, confidence label, or plan-wide caveat list. The accepted
`source.ChunkPlan` contains the validated ranges and source digest, with no D&D
scene-specific plan or range annotations.

The module key, prompt identity, configured model profile, registration, and
the generic accepted chunk-map contract remain unchanged. The private prompt
and response-schema content changes in place; this pre-release application does
not need a compatibility layer for prior private responses.

Existing canonical chunk plans may contain the annotations removed by this
feature. Invalidate the pre-change chunk-plan cache format once so those records
are regenerated. This is a storage compatibility transition, not a change to
the source-addressed lookup policy: after the transition, canonical plans
remain keyed only by source digest and continue to be reused independently of
prompt, schema, profile, or module configuration. Prompt and schema hashes
remain provenance rather than cache-key inputs.

## Scene Boundary Policy

A scene is a coherent unit of play. A new scene is appropriate when the
transcript establishes a meaningful change in location, objective, threat,
activity, encounter, or mode of play. Examples include:

- moving to a materially different location;
- beginning or ending combat;
- entering a substantially different phase of an encounter;
- changing between combat, exploration, social interaction, planning, travel,
  rest, or downtime;
- shifting the central NPC, faction, threat, or immediate objective; or
- a sustained table-level interruption that materially changes the activity.

A scene should not begin solely because a speaker changes, a combat round
changes, a routine turn occurs, or the table briefly digresses. The chunker
should prefer a smaller number of coherent scenes over speculative or
fine-grained boundaries.

The prompt owns this semantic boundary guidance. Deterministic code owns all
structural invariants.

## Model And Prompt Boundary

Keep the existing shared D&D system, transcript, and reference assets and the
established prompt ordering. Simplify only the scene-specific task and
instructions needed to request boundary ranges. Do not move scene-specific
wording into shared assets unless another module needs exactly identical
content.

The private structured-output schema must:

- require one top-level `scenes` array;
- require at least one scene;
- permit only integer `start_unit_id` and `end_unit_id` fields on each scene;
- require positive unit IDs; and
- reject unknown fields at every object boundary.

Schema validation is intentionally structural. The model is not responsible
for chunk IDs, chunk indexes, source identity, digests, annotations, or
diagnostics.

The prompt must require full transcript coverage in source order without gaps
or overlaps. These instructions guide the model, but deterministic validation
remains authoritative.

## Deterministic Planning And Validation

Convert an accepted model response directly into a `source.ChunkPlan`. Preserve
the existing guarantees that:

- the scene list is present and non-empty;
- every endpoint identifies a unit in the current source document;
- each start appears at or before its corresponding end in document order;
- the first range starts at the first source unit;
- adjacent ranges are contiguous and do not overlap;
- ranges preserve source-document order; and
- the final range ends at the final source unit.

Do not assume that source-unit IDs are numerically contiguous or that numeric
ID order is document order. Continue to validate positions against the source
document's ordered unit collection.

Materialization remains responsible for deterministic chunk IDs, indexes, unit
membership, and generic plan validation. The chunker should not reproduce
framework-owned checks except where rejecting the private model response is
necessary to construct a valid plan.

Because the minimal response has no advisory fields, the chunker emits no
model-derived boundary warnings. Transport, schema, endpoint, coverage, order,
gap, and overlap failures remain ordinary chunk-planning errors. Do not replace
removed caveats or confidence labels with free-form diagnostics or hidden
annotations.

## Ownership Of Removed Semantics

Removed fields do not move into another chunker contract:

- `kind`, `title`, and `summary` belong to the accepted
  `dnd/scene-description-list` artifact produced by the
  `dnd/scene-descriptions` extraction lane.
- NPC occurrence evidence belongs to `dnd/npc-interactions`; a scene
  participant view may later be derived deterministically by intersecting
  artifact evidence ranges with scene ranges.
- Combat participation and turns belong to their dedicated artifacts.
- Boundary confidence, boundary notes, and plan-wide caveats are omitted until
  a concrete validator or operator workflow demonstrates a durable need for
  them.

The generic `source/chunk-map` artifact continues to represent the exact
accepted materialized chunks. For plans produced by `dnd/scenes`, its
`plan_annotations` and each chunk's `annotations` map are empty. Do not add
D&D-specific fields to the generic chunk-map schema.

## Quality Policy

Protect the minimal response contract and deterministic range invariants at
their stable behavioral boundaries. Do not add tests that freeze incidental
prompt wording, message length, or exact prompt hashes. Cache compatibility
tests may assert the chunk-plan storage version because it is a deliberate wire
compatibility boundary.

Evaluate boundary quality on a small human-reviewed transcript set containing
combat transitions, location changes, planning, social interaction, brief
digressions, and ambiguous gradual transitions. Review whether the simpler
contract improves full-coverage success and boundary usefulness on the smaller
models the application is intended to support. Treat this evaluation as a
human development aid, not a deterministic correctness oracle or an offline
test-suite requirement.

## Non-Goals

This scope does not:

- change how source documents or chunks are represented;
- change generic chunk validation or materialization;
- change the durable `source/chunk-map` schema or output option;
- add scene descriptions, classifications, summaries, or participants to the
  chunker;
- change the `dnd/scene-descriptions` artifact;
- make combat extraction depend on scene descriptions or skip non-combat
  chunks;
- introduce ordered pipeline dependencies, a DAG, or cross-lane
  reconciliation;
- add a new LLM validator, repair pass, or semantic post-processing step; or
- preserve compatibility with prior private scene-chunker responses or cached
  plans.