Repair reversed D&D evidence ranges
This commit is contained in:
@@ -1,6 +1,8 @@
|
||||
Transcript units are the only evidence for extracted events and factual claims.
|
||||
Every reported factual claim must be supported by cited transcript units. Use
|
||||
integer `start_unit_id` and `end_unit_id` values from the transcript.
|
||||
Within each range, `start_unit_id` must identify the earlier transcript unit and
|
||||
`end_unit_id` the same or a later unit according to transcript order.
|
||||
|
||||
When supporting evidence is non-contiguous, use multiple narrow ranges rather
|
||||
than a broad range that bridges unrelated conversation.
|
||||
|
||||
@@ -120,11 +120,13 @@ manifest and prompt declaration.
|
||||
## Evidence, Candidates, And Normalization
|
||||
|
||||
The current transcript is the only durable evidence source. Extractors assign
|
||||
the current source identity, preserve candidate evidence ranges for validators,
|
||||
and canonically order or remove exact duplicate ranges without asking the
|
||||
model to repair semantic errors. Campaign context and generated artifacts may
|
||||
ground names or control routing, but they never establish evidence for a D&D
|
||||
result.
|
||||
the current source identity and losslessly order any reversed range whose two
|
||||
endpoints resolve in that source, using transcript position rather than numeric
|
||||
unit-ID order. They then canonically order ranges and remove exact duplicates.
|
||||
This routine canonicalization does not request a retry or emit a warning.
|
||||
Unresolvable or wrong-source ranges remain unchanged for validators to reject.
|
||||
Campaign context and generated artifacts may ground names or control routing,
|
||||
but they never establish evidence for a D&D result.
|
||||
|
||||
Default chains keep responsibilities separate: structural validators assess the
|
||||
candidate, source-reference validators resolve cited ranges against the current
|
||||
|
||||
@@ -14,8 +14,8 @@ eligibility, and the separation of actionable process warnings from quality
|
||||
diagnostics. The remaining near-term work applies those completed foundations
|
||||
to domain review and empirical evaluation.
|
||||
|
||||
The active first production application is defined by
|
||||
[D&D Combat Scene Semantic Validation](combat-scene-validation.md).
|
||||
The active D&D reliability work is defined by
|
||||
[D&D Source-Reference Endpoint Canonicalization](source-reference-canonicalization.md).
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
|
||||
96
docs/roadmap/source-reference-canonicalization.md
Normal file
96
docs/roadmap/source-reference-canonicalization.md
Normal file
@@ -0,0 +1,96 @@
|
||||
# D&D Source-Reference Endpoint Canonicalization
|
||||
|
||||
## Purpose
|
||||
|
||||
Make model-supplied D&D evidence ranges resilient to reversed endpoints without
|
||||
spending producer retries on a losslessly repairable representation error.
|
||||
|
||||
## Problem
|
||||
|
||||
The shared extraction-evidence prompt requires `start_unit_id` and
|
||||
`end_unit_id`, but does not state that start means the earlier unit in
|
||||
transcript order. The private LLM schemas cannot express that cross-field
|
||||
relationship. Current extraction adapters sort and de-duplicate ranges but
|
||||
preserve reversed endpoints, so deterministic source-reference validators
|
||||
reject otherwise useful candidates and may exhaust the producer retry budget.
|
||||
|
||||
## Target State
|
||||
|
||||
- The shared D&D extraction-evidence prompt states that `start_unit_id` is the
|
||||
earlier endpoint and `end_unit_id` is the same or a later endpoint according
|
||||
to transcript order.
|
||||
- The existing LLM-facing and durable `start_unit_id`/`end_unit_id` shapes stay
|
||||
unchanged. No unordered-pair schema or compatibility migration is introduced.
|
||||
- One helper owned by `internal/modules/dnd/shared` orders the endpoints of a
|
||||
source reference using `source.DocumentIndex` positions. It never assumes
|
||||
that numerically smaller unit IDs occur earlier.
|
||||
- The helper swaps endpoints only when the reference belongs to the current
|
||||
source and both endpoint IDs resolve in that source. Forward and single-unit
|
||||
ranges remain unchanged. Wrong-source, missing, non-positive, or otherwise
|
||||
unresolved endpoints remain unchanged for deterministic validators to reject.
|
||||
- Every D&D extraction adapter that consumes model-supplied evidence ranges
|
||||
applies this endpoint operation before its existing range sorting,
|
||||
de-duplication, earliest-evidence calculation, and validation. The covered
|
||||
artifact families are spells, NPC registry, NPC occurrences, item registry,
|
||||
item occurrences, location registry, location occurrences, combat turns, and
|
||||
enemy events.
|
||||
- Successful endpoint swapping is ordinary deterministic canonicalization. It
|
||||
does not emit a warning, quality diagnostic, or retry request.
|
||||
- The exact raw model response remains available through existing model-candidate
|
||||
and debug provenance. Only the typed candidate presented to validators and
|
||||
later stages receives the canonical endpoint order.
|
||||
- Source-reference validators remain strict and continue rejecting reversed
|
||||
ranges that reach them from another producer or indicate an application bug.
|
||||
Normalizers do not become a fallback repair boundary for invalid extraction
|
||||
candidates.
|
||||
|
||||
## Required Work
|
||||
|
||||
1. Clarify endpoint ordering in
|
||||
`assets/dnd/shared/prompts/common-dnd-extraction-evidence.md`, preserving one
|
||||
byte-identical shared instruction for every consuming prompt.
|
||||
2. Extend the shared source-reference ordering utility with a mutation-safe
|
||||
endpoint-ordering operation that uses document position and preserves
|
||||
unresolvable references.
|
||||
3. Integrate the shared operation into all nine model-response-to-candidate
|
||||
mappings listed above. Do not duplicate endpoint comparison logic in the
|
||||
individual artifact packages.
|
||||
4. Update `docs/internal/dnd.md` to distinguish lossless endpoint ordering from
|
||||
malformed-reference validation. External artifact documentation and schemas
|
||||
require no change because their contract already requires ordered ranges.
|
||||
5. Add lean regression coverage for forward, reversed, single-unit,
|
||||
non-monotonic-ID, wrong-source, and unresolved endpoint cases. Add enough
|
||||
adapter-level coverage to prove the shared operation is wired into every
|
||||
affected extraction family without duplicating the helper's full case
|
||||
matrix in each package.
|
||||
6. Update prompt-asset and fingerprint expectations affected by the shared
|
||||
prompt revision, while avoiding exact token-, byte-length-, or prose-snapshot
|
||||
change-detector tests.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Changing durable source-reference schemas or replacing named endpoints with
|
||||
an unordered LLM response shape.
|
||||
- Weakening source-reference validation or silently repairing missing,
|
||||
out-of-source, or out-of-chunk evidence.
|
||||
- Applying this policy to scene chunk-plan ranges, which are not extraction
|
||||
evidence references and have a separate ownership and validation boundary.
|
||||
- Adding warnings or durable normalization observations for routine endpoint
|
||||
ordering.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A D&D extraction candidate whose two resolvable evidence endpoints are
|
||||
reversed reaches validation as the equivalent forward range without a
|
||||
producer retry.
|
||||
- Ordering follows document position even when unit IDs are not numerically
|
||||
increasing.
|
||||
- Every affected D&D extractor uses the shared implementation; no module-local
|
||||
endpoint-swap implementation remains.
|
||||
- References that cannot be safely ordered remain unchanged and are rejected by
|
||||
the existing deterministic validators where applicable.
|
||||
- Raw model-candidate provenance remains byte-faithful to the provider response.
|
||||
- LLM-facing and durable schemas, validator strictness, module keys, default
|
||||
chains, and output contracts remain unchanged.
|
||||
- Focused package tests, repository-wide tests, `go vet ./...`, and
|
||||
`go build ./cmd/notarius` pass.
|
||||
@@ -264,7 +264,7 @@ func TestChangedSemanticSpellCatalogFingerprintCannotResumeRecordedCheckpoint(t
|
||||
if _, decision := changedLoader.Normalize("spells", spellnormalize.Key, normalizeDependencies); decision.Reused {
|
||||
t.Fatalf("changed normalize fingerprint decision = %#v, want normalize checkpoint cold miss", decision)
|
||||
}
|
||||
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.v3")
|
||||
changedMapping := replaceCheckpointFingerprintValue(t, fingerprints, extractSpellMappingFingerprintName(), "dnd.spells.extract_mapping.changed")
|
||||
assertOnlyCheckpointFingerprintChanged(t, fingerprints, changedMapping, extractSpellMappingFingerprintName())
|
||||
_, mappingLoader, err := checkpointHandlersForRun(settings, Options{}, materialized, changedMapping, llmFingerprints, []byte("same input"), nil, nil, "", "", LLMRuntimeOverrides{}, true)
|
||||
if err != nil {
|
||||
|
||||
@@ -45,7 +45,7 @@ func canonicalizeCombatTurn(turn *combatTurnResponse, order shared.SourceRefOrde
|
||||
if turn == nil {
|
||||
return 0, false
|
||||
}
|
||||
refs := order.Canonicalize(canonicalSourceRefs(turn.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(canonicalSourceRefs(turn.SourceRefs, sourceID)))
|
||||
turn.SourceRefs = combatResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/combat-turns"
|
||||
mappingPolicy = "dnd.combat_turns.extract_mapping.v2"
|
||||
mappingPolicy = "dnd.combat_turns.extract_mapping.v3"
|
||||
sceneGatePolicy = "dnd.combat_turns.scene_gate.v1"
|
||||
)
|
||||
|
||||
|
||||
@@ -25,8 +25,8 @@ func TestExtractMapsAndOrdersCombatTurnsBySourcePosition(t *testing.T) {
|
||||
},
|
||||
{
|
||||
Actor: "Aria", TurnKind: "reaction", SourceRefs: []combatSourceRefResponse{
|
||||
{StartUnitID: 10, EndUnitID: 10},
|
||||
{StartUnitID: 10, EndUnitID: 10},
|
||||
{StartUnitID: 2, EndUnitID: 10},
|
||||
{StartUnitID: 2, EndUnitID: 10},
|
||||
},
|
||||
},
|
||||
{
|
||||
@@ -42,7 +42,7 @@ func TestExtractMapsAndOrdersCombatTurnsBySourcePosition(t *testing.T) {
|
||||
if got := []string{result.Value.CombatTurns[0].Actor, result.Value.CombatTurns[1].Actor, result.Value.CombatTurns[2].Actor}; !reflect.DeepEqual(got, []string{"Aria", "Borin", "Unknown"}) {
|
||||
t.Fatalf("actor order = %#v, want source-position order with invalid evidence last", got)
|
||||
}
|
||||
wantRefs := []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 10, EndUnitID: 10}}
|
||||
wantRefs := []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 10, EndUnitID: 2}}
|
||||
if !reflect.DeepEqual(result.Value.CombatTurns[0].SourceRefs, wantRefs) {
|
||||
t.Fatalf("canonical refs = %#v, want %#v", result.Value.CombatTurns[0].SourceRefs, wantRefs)
|
||||
}
|
||||
|
||||
@@ -13,7 +13,7 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/enemy-events"
|
||||
mappingPolicy = "dnd.enemy_events.extract_mapping.v1"
|
||||
mappingPolicy = "dnd.enemy_events.extract_mapping.v2"
|
||||
sceneGatePolicy = "dnd.enemy_events.scene_gate.v1"
|
||||
)
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
|
||||
func TestExtractMapsEnemyEventsInSourceOrder(t *testing.T) {
|
||||
client := &fakeEnemyEventsLLMClient{response: extractionResponse{Events: []enemyEventResponse{
|
||||
{Name: "Ashfang", Kind: "killed", SourceRefs: []enemySourceRefResponse{{StartUnitID: 4, EndUnitID: 4}}},
|
||||
{Name: "Ashfang", Kind: "killed", SourceRefs: []enemySourceRefResponse{{StartUnitID: 4, EndUnitID: 3}}},
|
||||
{Name: "Ashfang", Kind: "engaged", SourceRefs: []enemySourceRefResponse{{StartUnitID: 1, EndUnitID: 1}, {StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "Orcs", Kind: "fled", SourceRefs: []enemySourceRefResponse{{StartUnitID: 3, EndUnitID: 3}}},
|
||||
{Name: "One orc", Kind: "captured", SourceRefs: []enemySourceRefResponse{{StartUnitID: 2, EndUnitID: 2}}},
|
||||
@@ -31,14 +31,17 @@ func TestExtractMapsEnemyEventsInSourceOrder(t *testing.T) {
|
||||
dnd.EnemyEventKindEngaged,
|
||||
dnd.EnemyEventKindCaptured,
|
||||
dnd.EnemyEventKindIncapacitated,
|
||||
dnd.EnemyEventKindFled,
|
||||
dnd.EnemyEventKindKilled,
|
||||
dnd.EnemyEventKindFled,
|
||||
}) {
|
||||
t.Fatalf("event order = %#v", got)
|
||||
}
|
||||
if refs := result.Value.Events[0].SourceRefs; !reflect.DeepEqual(refs, []source.SourceRef{{SourceID: "combat-session", StartUnitID: 1, EndUnitID: 1}}) {
|
||||
t.Fatalf("canonical source refs = %#v", refs)
|
||||
}
|
||||
if refs := result.Value.Events[3].SourceRefs; !reflect.DeepEqual(refs, []source.SourceRef{{SourceID: "combat-session", StartUnitID: 3, EndUnitID: 4}}) {
|
||||
t.Fatalf("reversed source refs = %#v", refs)
|
||||
}
|
||||
if len(client.requests) != 1 {
|
||||
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
||||
}
|
||||
|
||||
@@ -20,7 +20,7 @@ func canonicalEnemyEventList(response extractionResponse, order shared.SourceRef
|
||||
}
|
||||
ordered := make([]orderedEnemyEvent, len(response.Events))
|
||||
for index, event := range response.Events {
|
||||
refs := order.Canonicalize(sourceRefs(event.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(sourceRefs(event.SourceRefs, sourceID)))
|
||||
earliest, hasEvidence := order.EarliestValid(refs)
|
||||
ordered[index] = orderedEnemyEvent{
|
||||
value: dnd.EnemyEvent{
|
||||
|
||||
@@ -26,7 +26,7 @@ func canonicalItemOccurrenceList(response extractionResponse, order shared.Sourc
|
||||
if !found {
|
||||
return dnd.ItemOccurrenceList{}, fmt.Errorf("occurrences[%d].name is not in the item registry", index)
|
||||
}
|
||||
refs := order.Canonicalize(itemOccurrenceSourceRefs(occurrence.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(itemOccurrenceSourceRefs(occurrence.SourceRefs, sourceID)))
|
||||
earliest, hasEvidence := order.EarliestValid(refs)
|
||||
ordered[index] = orderedItemOccurrence{
|
||||
value: dnd.ItemOccurrence{
|
||||
|
||||
@@ -20,7 +20,7 @@ const (
|
||||
ItemRegistryMaxBytes = itemregistry.MaxBytes
|
||||
)
|
||||
|
||||
const mappingPolicy = "dnd.item_occurrences.extract_mapping.v2"
|
||||
const mappingPolicy = "dnd.item_occurrences.extract_mapping.v3"
|
||||
|
||||
var requiredCapabilities = []string{
|
||||
"chunks",
|
||||
|
||||
@@ -11,7 +11,7 @@ import (
|
||||
|
||||
func TestExtractGroundsOccurrencesInRequiredRegistry(t *testing.T) {
|
||||
id := itemidentity.DeriveID("Torch")
|
||||
rawResponse := []byte(`{"occurrences":[{"name":"Torch","kind":"lost","from":"party","source_refs":[{"start_unit_id":1,"end_unit_id":1}]}]}`)
|
||||
rawResponse := []byte(`{"occurrences":[{"name":"Torch","kind":"lost","from":"party","source_refs":[{"start_unit_id":2,"end_unit_id":1}]}]}`)
|
||||
client := &fakeItemOccurrencesLLMClient{content: append([]byte(nil), rawResponse...)}
|
||||
req := extractionRequest()
|
||||
req.References = itemRegistryReferences(t)
|
||||
@@ -27,7 +27,7 @@ func TestExtractGroundsOccurrencesInRequiredRegistry(t *testing.T) {
|
||||
if len(result.Value.Occurrences) != 1 || result.Value.Occurrences[0].ItemID != id || result.Value.Occurrences[0].Name != "Torch" {
|
||||
t.Fatalf("occurrences = %#v", result.Value.Occurrences)
|
||||
}
|
||||
if refs := result.Value.Occurrences[0].SourceRefs; len(refs) != 1 || refs[0].SourceID != req.Source.ID || refs[0].StartUnitID != 1 || refs[0].EndUnitID != 1 {
|
||||
if refs := result.Value.Occurrences[0].SourceRefs; len(refs) != 1 || refs[0].SourceID != req.Source.ID || refs[0].StartUnitID != 1 || refs[0].EndUnitID != 2 {
|
||||
t.Fatalf("occurrence evidence = %#v, want current-source unit range", refs)
|
||||
}
|
||||
request := client.requests[0]
|
||||
|
||||
@@ -43,7 +43,7 @@ func canonicalizeItem(item *itemResponse, order shared.SourceRefOrder, sourceID
|
||||
return 0, false
|
||||
}
|
||||
item.Name = identity.NormalizeDisplay(item.Name)
|
||||
refs := order.Canonicalize(canonicalSourceRefs(item.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(canonicalSourceRefs(item.SourceRefs, sourceID)))
|
||||
item.SourceRefs = itemResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/item-registry"
|
||||
mappingPolicy = "dnd.item_registry.extract_mapping.v1"
|
||||
mappingPolicy = "dnd.item_registry.extract_mapping.v2"
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"chunks", "source.transcript"}
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
|
||||
func TestExtractMapsItemsWithOwnedEvidenceAndDeterministicOrder(t *testing.T) {
|
||||
client := &fakeItemsLLMClient{response: extractionResponse{Items: []itemResponse{
|
||||
{Name: " Gold Pieces ", SourceRefs: responseSourceRefs(3, 3)},
|
||||
{Name: " Gold Pieces ", SourceRefs: responseSourceRefs(3, 2)},
|
||||
{Name: "Star Compass", SourceRefs: []itemSourceRefResponse{{StartUnitID: 2, EndUnitID: 2}, {StartUnitID: 1, EndUnitID: 1}, {StartUnitID: 1, EndUnitID: 1}}},
|
||||
}}}
|
||||
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||
@@ -25,7 +25,7 @@ func TestExtractMapsItemsWithOwnedEvidenceAndDeterministicOrder(t *testing.T) {
|
||||
refs := []source.SourceRef{{SourceID: "session-items", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session-items", StartUnitID: 2, EndUnitID: 2}}
|
||||
want := dnd.ItemRegistry{Items: []dnd.Item{
|
||||
{ID: identity.DeriveID("Star Compass"), Name: "Star Compass", SourceRefs: refs},
|
||||
{ID: identity.DeriveID("Gold Pieces"), Name: "Gold Pieces", SourceRefs: []source.SourceRef{{SourceID: "session-items", StartUnitID: 3, EndUnitID: 3}}},
|
||||
{ID: identity.DeriveID("Gold Pieces"), Name: "Gold Pieces", SourceRefs: []source.SourceRef{{SourceID: "session-items", StartUnitID: 2, EndUnitID: 3}}},
|
||||
}}
|
||||
if !reflect.DeepEqual(result.Value, want) {
|
||||
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
||||
|
||||
@@ -27,7 +27,7 @@ func canonicalOccurrenceList(response extractionResponse, order shared.SourceRef
|
||||
if !ok {
|
||||
return dnd.LocationOccurrenceList{}, fmt.Errorf("occurrence %d does not match a supplied location selector", index)
|
||||
}
|
||||
refs := order.Canonicalize(canonicalSourceRefs(occurrence.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(canonicalSourceRefs(occurrence.SourceRefs, sourceID)))
|
||||
earliest, hasEvidence := order.EarliestValid(refs)
|
||||
ordered[index] = orderedOccurrence{value: dnd.LocationOccurrence{
|
||||
LocationID: location.ID,
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/location-occurrences"
|
||||
mappingPolicy = "dnd.location_occurrences.extract_mapping.v2"
|
||||
mappingPolicy = "dnd.location_occurrences.extract_mapping.v3"
|
||||
)
|
||||
|
||||
const (
|
||||
|
||||
@@ -23,7 +23,7 @@ func TestExtractMapsKindsOrdersOccurrencesAndPreservesIndependentFacts(t *testin
|
||||
client := &fakeOccurrencesLLMClient{response: extractionResponse{Occurrences: []occurrenceResponse{
|
||||
{Name: second.Name, RegistryRefs: registryRefs(second), Kind: "mentioned", SourceRefs: occurrenceRefs(30, 30)},
|
||||
{Name: first.Name, RegistryRefs: registryRefs(first), Kind: "mentioned", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{Name: first.Name, RegistryRefs: registryRefs(first), Kind: "recalled", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{Name: first.Name, RegistryRefs: registryRefs(first), Kind: "recalled", SourceRefs: occurrenceRefs(20, 10)},
|
||||
{Name: first.Name, RegistryRefs: registryRefs(first), Kind: "planned", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{Name: first.Name, RegistryRefs: registryRefs(first), Kind: "visited", SourceRefs: append(occurrenceRefs(10, 10), occurrenceRefs(10, 10)...)},
|
||||
{Name: first.Name, RegistryRefs: registryRefs(first), Kind: "visited", SourceRefs: occurrenceRefs(20, 20)},
|
||||
@@ -49,6 +49,9 @@ func TestExtractMapsKindsOrdersOccurrencesAndPreservesIndependentFacts(t *testin
|
||||
if !reflect.DeepEqual(got[0].SourceRefs, []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 10, EndUnitID: 10}}) {
|
||||
t.Fatalf("canonical evidence = %#v", got[0].SourceRefs)
|
||||
}
|
||||
if !reflect.DeepEqual(got[2].SourceRefs, []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 10, EndUnitID: 20}}) {
|
||||
t.Fatalf("reversed evidence = %#v", got[2].SourceRefs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractResolvesContextualSelectorsAndUsesCurrentTranscriptEvidenceOnly(t *testing.T) {
|
||||
|
||||
@@ -42,7 +42,7 @@ func canonicalizeLocation(location *locationResponse, order shared.SourceRefOrde
|
||||
if location == nil {
|
||||
return 0, false
|
||||
}
|
||||
refs := order.Canonicalize(canonicalSourceRefs(location.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(canonicalSourceRefs(location.SourceRefs, sourceID)))
|
||||
location.SourceRefs = locationResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/location-registry"
|
||||
mappingPolicy = "dnd.location_registry.extract_mapping.v1"
|
||||
mappingPolicy = "dnd.location_registry.extract_mapping.v2"
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"chunks", "source.transcript"}
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
|
||||
func TestExtractMapsLocationsWithOwnedEvidenceAndDeterministicOrder(t *testing.T) {
|
||||
client := &fakeLocationsLLMClient{response: extractionResponse{Locations: []locationResponse{
|
||||
{Name: "The Tavern", SourceRefs: responseSourceRefs(3, 3)},
|
||||
{Name: "The Tavern", SourceRefs: responseSourceRefs(3, 2)},
|
||||
{Name: "Old Mill", SourceRefs: []locationSourceRefResponse{{StartUnitID: 2, EndUnitID: 2}, {StartUnitID: 1, EndUnitID: 1}, {StartUnitID: 1, EndUnitID: 1}}},
|
||||
}}}
|
||||
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||
@@ -25,7 +25,7 @@ func TestExtractMapsLocationsWithOwnedEvidenceAndDeterministicOrder(t *testing.T
|
||||
refs := []source.SourceRef{{SourceID: "session-locations", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session-locations", StartUnitID: 2, EndUnitID: 2}}
|
||||
want := dnd.LocationRegistry{Locations: []dnd.Location{
|
||||
{ID: identity.DeriveID("Old Mill", refs), Name: "Old Mill", SourceRefs: refs},
|
||||
{ID: identity.DeriveID("The Tavern", []source.SourceRef{{SourceID: "session-locations", StartUnitID: 3, EndUnitID: 3}}), Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: "session-locations", StartUnitID: 3, EndUnitID: 3}}},
|
||||
{ID: identity.DeriveID("The Tavern", []source.SourceRef{{SourceID: "session-locations", StartUnitID: 2, EndUnitID: 3}}), Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: "session-locations", StartUnitID: 2, EndUnitID: 3}}},
|
||||
}}
|
||||
if !reflect.DeepEqual(result.Value, want) {
|
||||
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
||||
|
||||
@@ -47,7 +47,7 @@ func canonicalizeOccurrence(occurrence *occurrenceResponse, order shared.SourceR
|
||||
if occurrence == nil {
|
||||
return 0, false
|
||||
}
|
||||
refs := order.Canonicalize(canonicalSourceRefs(occurrence.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(canonicalSourceRefs(occurrence.SourceRefs, sourceID)))
|
||||
occurrence.SourceRefs = occurrenceResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/npc-occurrences"
|
||||
mappingPolicy = "dnd.npc_occurrences.extract_mapping.v3"
|
||||
mappingPolicy = "dnd.npc_occurrences.extract_mapping.v4"
|
||||
)
|
||||
|
||||
const (
|
||||
|
||||
@@ -22,7 +22,7 @@ func TestExtractMapsEveryKindAndOrdersBySourcePosition(t *testing.T) {
|
||||
{Name: "Opponent", Kind: "combat_opponent", SourceRefs: occurrenceRefs(20, 20)},
|
||||
{Name: "Ally", Kind: "combat_ally", SourceRefs: occurrenceRefs(5, 5)},
|
||||
{Name: "Speaker", Kind: "dialogue", SourceRefs: append(occurrenceRefs(2, 2), occurrenceRefs(2, 2)...)},
|
||||
{Name: "Present", Kind: "noncombat_presence", SourceRefs: occurrenceRefs(7, 7)},
|
||||
{Name: "Present", Kind: "noncombat_presence", SourceRefs: occurrenceRefs(7, 2)},
|
||||
{Name: "Mentioned", Kind: "mentioned", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{Name: "Invalid", Kind: "unsupported", SourceRefs: occurrenceRefs(0, 0)},
|
||||
}}}
|
||||
@@ -51,6 +51,9 @@ func TestExtractMapsEveryKindAndOrdersBySourcePosition(t *testing.T) {
|
||||
if refs := result.Value.Occurrences[1].SourceRefs; !reflect.DeepEqual(refs, []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}) {
|
||||
t.Fatalf("canonical source refs = %#v", refs)
|
||||
}
|
||||
if refs := result.Value.Occurrences[2].SourceRefs; !reflect.DeepEqual(refs, []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 7}}) {
|
||||
t.Fatalf("reversed source refs = %#v", refs)
|
||||
}
|
||||
if id := result.Value.Occurrences[0].NPCID; id != identity.DeriveID("Mentioned") {
|
||||
t.Fatalf("durable NPC ID = %q, want registry identity", id)
|
||||
}
|
||||
|
||||
@@ -46,7 +46,7 @@ func canonicalizeNPC(npc *npcResponse, order shared.SourceRefOrder, sourceID str
|
||||
if npc == nil {
|
||||
return 0, false
|
||||
}
|
||||
refs := order.Canonicalize(canonicalSourceRefs(npc.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(canonicalSourceRefs(npc.SourceRefs, sourceID)))
|
||||
npc.SourceRefs = npcResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
@@ -13,7 +13,7 @@ import (
|
||||
|
||||
const Key = "dnd/npc-registry"
|
||||
|
||||
const mappingPolicy = "dnd.npc_registry.extract_mapping.v2"
|
||||
const mappingPolicy = "dnd.npc_registry.extract_mapping.v3"
|
||||
|
||||
var requiredCapabilities = []string{
|
||||
"chunks",
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
func TestExtractReturnsCanonicalNPCRegistryFromPrivateResponse(t *testing.T) {
|
||||
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{
|
||||
{
|
||||
Name: "Captain Vale", SourceRefs: responseSourceRefs(3, 3),
|
||||
Name: "Captain Vale", SourceRefs: responseSourceRefs(3, 2),
|
||||
},
|
||||
{
|
||||
Name: "Mira Thorn",
|
||||
@@ -34,7 +34,7 @@ func TestExtractReturnsCanonicalNPCRegistryFromPrivateResponse(t *testing.T) {
|
||||
}
|
||||
want := dnd.NPCRegistry{NPCs: []dnd.NPC{
|
||||
{ID: identity.DeriveID("Mira Thorn"), Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}, {SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}},
|
||||
{ID: identity.DeriveID("Captain Vale"), Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 3}}},
|
||||
{ID: identity.DeriveID("Captain Vale"), Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 3}}},
|
||||
}}
|
||||
if !reflect.DeepEqual(result.Value, want) {
|
||||
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
||||
|
||||
@@ -45,7 +45,7 @@ func canonicalizeSpellCast(spell *spellCastResponse, order shared.SourceRefOrder
|
||||
if spell == nil {
|
||||
return 0, false
|
||||
}
|
||||
refs := order.Canonicalize(spellSourceRefs(spell.SourceRefs, sourceID))
|
||||
refs := order.Canonicalize(order.OrderEndpoints(spellSourceRefs(spell.SourceRefs, sourceID)))
|
||||
spell.SourceRefs = spellResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
@@ -16,7 +16,7 @@ import (
|
||||
const Key = "dnd/spells"
|
||||
const SchemaVersion = "v1"
|
||||
|
||||
const mappingPolicy = "dnd.spells.extract_mapping.v2"
|
||||
const mappingPolicy = "dnd.spells.extract_mapping.v3"
|
||||
|
||||
const (
|
||||
NPCRegistryReferenceSlot = npcregistry.ReferenceSlot
|
||||
|
||||
@@ -21,7 +21,7 @@ func TestExtractReturnsCanonicalSpellListFromPrivateResponse(t *testing.T) {
|
||||
{
|
||||
Caster: " Aria ",
|
||||
Spell: " Cure Wounds ",
|
||||
SourceRefs: responseSourceRefs(1, 2),
|
||||
SourceRefs: responseSourceRefs(2, 1),
|
||||
},
|
||||
}}}
|
||||
req := extractionRequest()
|
||||
|
||||
@@ -23,14 +23,15 @@ import (
|
||||
|
||||
func TestExtractionPromptComposition(t *testing.T) {
|
||||
const (
|
||||
transcriptSentinel = "shared-transcript-sentinel"
|
||||
playersSentinel = "shared-players-sentinel"
|
||||
partySentinel = "shared-party-sentinel"
|
||||
glossarySentinel = "shared-glossary-sentinel"
|
||||
npcSentinel = "npc-registry-sentinel"
|
||||
catalogSentinel = "spell-catalog-sentinel"
|
||||
evidenceSentinel = "Transcript units are the only evidence"
|
||||
combatPolicySentinel = "substantive active combat materially organizes the"
|
||||
transcriptSentinel = "shared-transcript-sentinel"
|
||||
playersSentinel = "shared-players-sentinel"
|
||||
partySentinel = "shared-party-sentinel"
|
||||
glossarySentinel = "shared-glossary-sentinel"
|
||||
npcSentinel = "npc-registry-sentinel"
|
||||
catalogSentinel = "spell-catalog-sentinel"
|
||||
evidenceSentinel = "Transcript units are the only evidence"
|
||||
endpointOrderSentinel = "according to transcript order"
|
||||
combatPolicySentinel = "substantive active combat materially organizes the"
|
||||
)
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := registerPromptAssets(registry); err != nil {
|
||||
@@ -51,12 +52,12 @@ func TestExtractionPromptComposition(t *testing.T) {
|
||||
suffixGroups [][]string
|
||||
middleSentinel string
|
||||
}{
|
||||
{name: "npcs", promptID: npcextract.PromptID, promptVersion: npcextract.SchemaVersion, inputs: commonInputs, suffixGroups: [][]string{{evidenceSentinel}}},
|
||||
{name: "locations", promptID: locationextract.PromptID, promptVersion: locationextract.SchemaVersion, inputs: commonInputs, suffixGroups: [][]string{{evidenceSentinel}}},
|
||||
{name: "npcs", promptID: npcextract.PromptID, promptVersion: npcextract.SchemaVersion, inputs: commonInputs, suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}}},
|
||||
{name: "locations", promptID: locationextract.PromptID, promptVersion: locationextract.SchemaVersion, inputs: commonInputs, suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}}},
|
||||
{name: "item occurrences", promptID: itemoccurrenceextract.PromptID, promptVersion: itemoccurrenceextract.SchemaVersion, inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||
"item_registry": promptkit.Inline(`{"items":[{"id":"item-registry-sentinel","name":"Torch"}]}`),
|
||||
}), suffixGroups: [][]string{{evidenceSentinel}, {"item-registry-sentinel"}}},
|
||||
{name: "item registry", promptID: itemregistryextract.PromptID, promptVersion: itemregistryextract.SchemaVersion, inputs: commonInputs, suffixGroups: [][]string{{evidenceSentinel}}},
|
||||
}), suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}, {"item-registry-sentinel"}}},
|
||||
{name: "item registry", promptID: itemregistryextract.PromptID, promptVersion: itemregistryextract.SchemaVersion, inputs: commonInputs, suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}}},
|
||||
{name: "scene descriptions", promptID: scenedescriptionextract.PromptID, promptVersion: scenedescriptionextract.SchemaVersion, inputs: commonInputs, middleSentinel: combatPolicySentinel},
|
||||
{
|
||||
name: "combat turns",
|
||||
@@ -65,7 +66,7 @@ func TestExtractionPromptComposition(t *testing.T) {
|
||||
inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||
"npc_registry": promptkit.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
||||
}),
|
||||
suffixGroups: [][]string{{evidenceSentinel}, {npcSentinel}},
|
||||
suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}, {npcSentinel}},
|
||||
},
|
||||
{
|
||||
name: "enemy events",
|
||||
@@ -77,7 +78,7 @@ func TestExtractionPromptComposition(t *testing.T) {
|
||||
"npc_occurrences": promptkit.Inline(`{"sentinel":"npc-occurrences-sentinel"}`),
|
||||
}),
|
||||
suffixGroups: [][]string{
|
||||
{evidenceSentinel},
|
||||
{evidenceSentinel, endpointOrderSentinel},
|
||||
{npcSentinel},
|
||||
{"combat-turns-sentinel", "npc-occurrences-sentinel"},
|
||||
},
|
||||
@@ -89,7 +90,7 @@ func TestExtractionPromptComposition(t *testing.T) {
|
||||
inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||
"npc_registry": promptkit.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
||||
}),
|
||||
suffixGroups: [][]string{{evidenceSentinel}, {npcSentinel}},
|
||||
suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}, {npcSentinel}},
|
||||
},
|
||||
{
|
||||
name: "location occurrences",
|
||||
@@ -98,7 +99,7 @@ func TestExtractionPromptComposition(t *testing.T) {
|
||||
inputs: withPromptInputs(commonInputs, map[string]promptkit.ArtifactRef{
|
||||
"location_registry": promptkit.Inline(`{"sentinel":"location-registry-sentinel"}`),
|
||||
}),
|
||||
suffixGroups: [][]string{{evidenceSentinel}, {"location-registry-sentinel"}},
|
||||
suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}, {"location-registry-sentinel"}},
|
||||
},
|
||||
{
|
||||
name: "spells",
|
||||
@@ -108,7 +109,7 @@ func TestExtractionPromptComposition(t *testing.T) {
|
||||
"npc_registry": promptkit.Inline(`{"sentinel":"` + npcSentinel + `"}`),
|
||||
"spell_catalog": promptkit.Inline(`{"sentinel":"` + catalogSentinel + `"}`),
|
||||
}),
|
||||
suffixGroups: [][]string{{evidenceSentinel}, {npcSentinel}, {catalogSentinel}},
|
||||
suffixGroups: [][]string{{evidenceSentinel, endpointOrderSentinel}, {npcSentinel}, {catalogSentinel}},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
@@ -63,6 +63,29 @@ func (o SourceRefOrder) EarliestValid(refs []source.SourceRef) (int, bool) {
|
||||
return earliest, found
|
||||
}
|
||||
|
||||
// OrderEndpoints returns an owned copy of refs with each resolvable range
|
||||
// ordered by document position. References outside the indexed source, or
|
||||
// whose endpoints cannot both be resolved, remain unchanged so validators can
|
||||
// diagnose them.
|
||||
func (o SourceRefOrder) OrderEndpoints(refs []source.SourceRef) []source.SourceRef {
|
||||
if refs == nil {
|
||||
return nil
|
||||
}
|
||||
ordered := append([]source.SourceRef{}, refs...)
|
||||
for index := range ordered {
|
||||
ref := &ordered[index]
|
||||
if ref.SourceID != o.sourceID || ref.StartUnitID <= 0 || ref.EndUnitID <= 0 {
|
||||
continue
|
||||
}
|
||||
start, startOK := o.index.Position(ref.StartUnitID)
|
||||
end, endOK := o.index.Position(ref.EndUnitID)
|
||||
if startOK && endOK && start > end {
|
||||
ref.StartUnitID, ref.EndUnitID = ref.EndUnitID, ref.StartUnitID
|
||||
}
|
||||
}
|
||||
return ordered
|
||||
}
|
||||
|
||||
// Canonicalize returns an owned, stable-sorted, exactly de-duplicated copy of
|
||||
// refs. It deliberately preserves invalid references for diagnostics.
|
||||
func (o SourceRefOrder) Canonicalize(refs []source.SourceRef) []source.SourceRef {
|
||||
|
||||
@@ -58,6 +58,38 @@ func TestSourceRefOrderCanonicalizePreservesRepresentationAndOwnership(t *testin
|
||||
}
|
||||
}
|
||||
|
||||
func TestSourceRefOrderOrdersOnlyResolvableCurrentSourceEndpoints(t *testing.T) {
|
||||
doc := unitRefSourceDocument(30, 10, 20)
|
||||
order := NewSourceRefOrder(doc)
|
||||
input := []source.SourceRef{
|
||||
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 10},
|
||||
{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10},
|
||||
{SourceID: "other", StartUnitID: 20, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 999, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 0},
|
||||
}
|
||||
want := []source.SourceRef{
|
||||
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 10},
|
||||
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 20},
|
||||
{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10},
|
||||
{SourceID: "other", StartUnitID: 20, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 999, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 0},
|
||||
}
|
||||
got := order.OrderEndpoints(input)
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("OrderEndpoints() = %#v, want %#v", got, want)
|
||||
}
|
||||
got[0].StartUnitID = 999
|
||||
if input[0].StartUnitID != 30 {
|
||||
t.Fatal("OrderEndpoints() output aliases input")
|
||||
}
|
||||
if got := order.OrderEndpoints(nil); got != nil {
|
||||
t.Fatalf("OrderEndpoints(nil) = %#v, want nil", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSourceRefOrderSnapshotAndEarliestValid(t *testing.T) {
|
||||
doc := unitRefSourceDocument(30, 10, 20)
|
||||
sourceID := doc.ID
|
||||
|
||||
@@ -51,11 +51,11 @@ func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
for name, value := range map[string]string{
|
||||
"extract:npc_registry:dnd/npc-registry:mapping_policy": "dnd.npc_registry.extract_mapping.v2",
|
||||
"extract:npc_registry:dnd/npc-registry:mapping_policy": "dnd.npc_registry.extract_mapping.v3",
|
||||
"normalize:npc_registry:dnd/npc-registry:identity_policy": "dnd.npc_registry.identity.v1",
|
||||
"normalize:npc_registry:dnd/npc-registry:normalization_policy": npcnormalize.NormalizationPolicy,
|
||||
"normalize:npc_registry:dnd/npc-registry:semantic_reconciliation_policy": semanticreconcile.Policy,
|
||||
"extract:spells:dnd/spells:mapping_policy": "dnd.spells.extract_mapping.v2",
|
||||
"extract:spells:dnd/spells:mapping_policy": "dnd.spells.extract_mapping.v3",
|
||||
"extract:combat:dnd/combat-turns:scene_gate_policy": "dnd.combat_turns.scene_gate.v1",
|
||||
} {
|
||||
assertFingerprintValue(t, prepared.CheckpointFingerprints(), name, value)
|
||||
|
||||
Reference in New Issue
Block a user