notarius/docs/roadmap/source-reference-canonicalization.md

D&D Source-Reference Endpoint Canonicalization

Purpose

Make model-supplied D&D evidence ranges resilient to reversed endpoints without spending producer retries on a losslessly repairable representation error.

Problem

The shared extraction-evidence prompt requires start_unit_id and end_unit_id, but does not state that start means the earlier unit in transcript order. The private LLM schemas cannot express that cross-field relationship. Current extraction adapters sort and de-duplicate ranges but preserve reversed endpoints, so deterministic source-reference validators reject otherwise useful candidates and may exhaust the producer retry budget.

Target State

• The shared D&D extraction-evidence prompt states that start_unit_id is the earlier endpoint and end_unit_id is the same or a later endpoint according to transcript order.
• The existing LLM-facing and durable start_unit_id/end_unit_id shapes stay unchanged. No unordered-pair schema or compatibility migration is introduced.
• One helper owned by internal/modules/dnd/shared orders the endpoints of a source reference using source.DocumentIndex positions. It never assumes that numerically smaller unit IDs occur earlier.
• The helper swaps endpoints only when the reference belongs to the current source and both endpoint IDs resolve in that source. Forward and single-unit ranges remain unchanged. Wrong-source, missing, non-positive, or otherwise unresolved endpoints remain unchanged for deterministic validators to reject.
• Every D&D extraction adapter that consumes model-supplied evidence ranges applies this endpoint operation before its existing range sorting, de-duplication, earliest-evidence calculation, and validation. The covered artifact families are spells, NPC registry, NPC occurrences, item registry, item occurrences, location registry, location occurrences, combat turns, and enemy events.
• Successful endpoint swapping is ordinary deterministic canonicalization. It does not emit a warning, quality diagnostic, or retry request.
• The exact raw model response remains available through existing model-candidate and debug provenance. Only the typed candidate presented to validators and later stages receives the canonical endpoint order.
• Source-reference validators remain strict and continue rejecting reversed ranges that reach them from another producer or indicate an application bug. Normalizers do not become a fallback repair boundary for invalid extraction candidates.

Required Work

1. Clarify endpoint ordering in assets/dnd/shared/prompts/common-dnd-extraction-evidence.md, preserving one byte-identical shared instruction for every consuming prompt.
2. Extend the shared source-reference ordering utility with a mutation-safe endpoint-ordering operation that uses document position and preserves unresolvable references.
3. Integrate the shared operation into all nine model-response-to-candidate mappings listed above. Do not duplicate endpoint comparison logic in the individual artifact packages.
4. Update docs/internal/dnd.md to distinguish lossless endpoint ordering from malformed-reference validation. External artifact documentation and schemas require no change because their contract already requires ordered ranges.
5. Add lean regression coverage for forward, reversed, single-unit, non-monotonic-ID, wrong-source, and unresolved endpoint cases. Add enough adapter-level coverage to prove the shared operation is wired into every affected extraction family without duplicating the helper's full case matrix in each package.
6. Update prompt-asset and fingerprint expectations affected by the shared prompt revision, while avoiding exact token-, byte-length-, or prose-snapshot change-detector tests.

Non-Goals

• Changing durable source-reference schemas or replacing named endpoints with an unordered LLM response shape.
• Weakening source-reference validation or silently repairing missing, out-of-source, or out-of-chunk evidence.
• Applying this policy to scene chunk-plan ranges, which are not extraction evidence references and have a separate ownership and validation boundary.
• Adding warnings or durable normalization observations for routine endpoint ordering.

Acceptance Criteria

• A D&D extraction candidate whose two resolvable evidence endpoints are reversed reaches validation as the equivalent forward range without a producer retry.
• Ordering follows document position even when unit IDs are not numerically increasing.
• Every affected D&D extractor uses the shared implementation; no module-local endpoint-swap implementation remains.
• References that cannot be safely ordered remain unchanged and are rejected by the existing deterministic validators where applicable.
• Raw model-candidate provenance remains byte-faithful to the provider response.
• LLM-facing and durable schemas, validator strictness, module keys, default chains, and output contracts remain unchanged.
• Focused package tests, repository-wide tests, go vet ./..., and go build ./cmd/notarius pass.