Minimal D&D Scene Chunking

Status: Implemented

Purpose

Narrow the dnd/scenes chunk module to one responsibility: divide a transcript into coherent, contiguous Dungeons & Dragons scenes. The current model response also asks for titles, modes, participants, summaries, boundary explanations, confidence labels, and plan-wide caveats. Those fields increase prompt and response complexity, overlap with dedicated extraction artifacts, and are not needed to materialize or validate chunks.

This change applies the same minimal-contract policy used by the D&D extraction lanes. The model should propose only facts that require model judgment and that the chunking operation consumes. Notarius should derive or validate everything else deterministically.

Desired End State

The dnd/scenes chunker asks the model for an ordered, non-empty list of inclusive source-unit ranges. Each range represents one coherent scene and contains exactly:

start_unit_id; and
end_unit_id.

The response contains no scene title, summary, kind, participant list, boundary note, confidence label, or plan-wide caveat list. The accepted source.ChunkPlan contains the validated ranges and source digest, with no D&D scene-specific plan or range annotations.

The module key, prompt identity, configured model profile, registration, and the generic accepted chunk-map contract remain unchanged. The private prompt and response-schema content changes in place; this pre-release application does not need a compatibility layer for prior private responses.

Existing canonical chunk plans may contain the annotations removed by this feature. Invalidate the pre-change chunk-plan cache format once so those records are regenerated. This is a storage compatibility transition, not a change to the source-addressed lookup policy: after the transition, canonical plans remain keyed only by source digest and continue to be reused independently of prompt, schema, profile, or module configuration. Prompt and schema hashes remain provenance rather than cache-key inputs.

Scene Boundary Policy

A scene is a coherent unit of play. A new scene is appropriate when the transcript establishes a meaningful change in location, objective, threat, activity, encounter, or mode of play. Examples include:

moving to a materially different location;
beginning or ending combat;
entering a substantially different phase of an encounter;
changing between combat, exploration, social interaction, planning, travel, rest, or downtime;
shifting the central NPC, faction, threat, or immediate objective; or
a sustained table-level interruption that materially changes the activity.

A scene should not begin solely because a speaker changes, a combat round changes, a routine turn occurs, or the table briefly digresses. The chunker should prefer a smaller number of coherent scenes over speculative or fine-grained boundaries.

The prompt owns this semantic boundary guidance. Deterministic code owns all structural invariants.

Model And Prompt Boundary

Keep the existing shared D&D system, transcript, and reference assets and the established prompt ordering. Simplify only the scene-specific task and instructions needed to request boundary ranges. Do not move scene-specific wording into shared assets unless another module needs exactly identical content.

The private structured-output schema must:

require one top-level scenes array;
require at least one scene;
permit only integer start_unit_id and end_unit_id fields on each scene;
require positive unit IDs; and
reject unknown fields at every object boundary.

Schema validation is intentionally structural. The model is not responsible for chunk IDs, chunk indexes, source identity, digests, annotations, or diagnostics.

The prompt must require full transcript coverage in source order without gaps or overlaps. These instructions guide the model, but deterministic validation remains authoritative.

Deterministic Planning And Validation

Convert an accepted model response directly into a source.ChunkPlan. Preserve the existing guarantees that:

the scene list is present and non-empty;
every endpoint identifies a unit in the current source document;
each start appears at or before its corresponding end in document order;
the first range starts at the first source unit;
adjacent ranges are contiguous and do not overlap;
ranges preserve source-document order; and
the final range ends at the final source unit.

Do not assume that source-unit IDs are numerically contiguous or that numeric ID order is document order. Continue to validate positions against the source document's ordered unit collection.

Materialization remains responsible for deterministic chunk IDs, indexes, unit membership, and generic plan validation. The chunker should not reproduce framework-owned checks except where rejecting the private model response is necessary to construct a valid plan.

Because the minimal response has no advisory fields, the chunker emits no model-derived boundary warnings. Transport, schema, endpoint, coverage, order, gap, and overlap failures remain ordinary chunk-planning errors. Do not replace removed caveats or confidence labels with free-form diagnostics or hidden annotations.

Ownership Of Removed Semantics

Removed fields do not move into another chunker contract:

kind, title, and summary belong to the accepted dnd/scene-description-list artifact produced by the dnd/scene-descriptions extraction lane.
NPC occurrence evidence belongs to dnd/npc-interactions; a scene participant view may later be derived deterministically by intersecting artifact evidence ranges with scene ranges.
Combat participation and turns belong to their dedicated artifacts.
Boundary confidence, boundary notes, and plan-wide caveats are omitted until a concrete validator or operator workflow demonstrates a durable need for them.

The generic source/chunk-map artifact continues to represent the exact accepted materialized chunks. For plans produced by dnd/scenes, its plan_annotations and each chunk's annotations map are empty. Do not add D&D-specific fields to the generic chunk-map schema.

Quality Policy

Protect the minimal response contract and deterministic range invariants at their stable behavioral boundaries. Do not add tests that freeze incidental prompt wording, message length, or exact prompt hashes. Cache compatibility tests may assert the chunk-plan storage version because it is a deliberate wire compatibility boundary.

Evaluate boundary quality on a small human-reviewed transcript set containing combat transitions, location changes, planning, social interaction, brief digressions, and ambiguous gradual transitions. Review whether the simpler contract improves full-coverage success and boundary usefulness on the smaller models the application is intended to support. Treat this evaluation as a human development aid, not a deterministic correctness oracle or an offline test-suite requirement.

Non-Goals

This scope does not:

change how source documents or chunks are represented;
change generic chunk validation or materialization;
change the durable source/chunk-map schema or output option;
add scene descriptions, classifications, summaries, or participants to the chunker;
change the dnd/scene-descriptions artifact;
make combat extraction depend on scene descriptions or skip non-combat chunks;
introduce ordered pipeline dependencies, a DAG, or cross-lane reconciliation;
add a new LLM validator, repair pass, or semantic post-processing step; or
preserve compatibility with prior private scene-chunker responses or cached plans.

7.6 KiB Raw Blame History