notarius/docs/roadmap/contextual-entity-grounding.md

# Contextual Entity Grounding

## Purpose

Notarius should use an LLM for semantic interpretation of source evidence, not
for referential-integrity work that deterministic code can perform more
reliably. D&D prompts must therefore stop requiring models to reproduce opaque
machine identifiers such as hash-derived entity IDs. Models should identify
entities through human-readable, evidence-grounded context, after which
Notarius resolves the selection and attaches the canonical durable identity.

This roadmap defines the policy, affected D&D prompt families, and intended
end state. The ordered work needed to reach that state is maintained in
[Implementation Plan](implementation.md).

## User Intent

The change has two goals:

- prevent otherwise useful model responses from failing because a long,
  non-semantic string was copied incorrectly; and
- avoid spending prompt space and model effort on exact-copy work that provides
  no semantic value.

The policy is not a ban on identifiers. Durable artifacts may continue to use
application-owned IDs, and prompts may continue to request source-unit ranges
that locate evidence. The policy governs which identity work is assigned to
the model.

## Policy

An LLM-facing prompt input or private response schema must not require a model
to reproduce an opaque machine identifier when Notarius can establish the same
association deterministically.

Opaque machine identifiers include cryptographic hashes, UUIDs, digests,
database keys, durable entity IDs, and other tokens whose characters do not
carry source-grounded meaning for the model. These values may remain in
application state, provenance, diagnostics, checkpoints, and durable artifact
contracts, but should be omitted from model-visible material when they do not
help the model make a semantic decision.

The intended responsibility boundary is:

- the model decides which contextual entity is supported by the supplied
  evidence and returns the bounded semantic facts requested by the module;
- the calling module validates that the contextual selection resolves to
  exactly one supplied candidate;
- deterministic code supplies the canonical display value and durable entity
  ID; and
- existing validators continue to enforce referential integrity at later
  artifact boundaries.

Transcript `start_unit_id` and `end_unit_id` values are permitted. They are
contextual source coordinates and form part of the evidence contract rather
than arbitrary identity tokens. Prompt IDs, schema IDs, fingerprints, session
IDs, and digests may also remain in runtime metadata that the model is not
asked to reproduce.

Short request-local labels are a narrowly permitted fallback only when a
contextual selector cannot uniquely represent the available choices without
unreasonable prompt cost. Such a label must be compact, scoped to one request,
validated against the supplied candidate set, and never reused as a durable
identity. Current D&D occurrence and reconciliation prompts should be designed
without this exception; adopting it later requires a concrete demonstrated
need and documented rationale.

## Current State

The initial NPC, item, and location registry extractors already follow the
desired pattern: the model returns contextual names and evidence, and Notarius
derives durable IDs afterward. Spells, combat turns, and enemy events use
contextual actor names rather than requiring hash-derived NPC IDs.

Two current prompt families diverge from that pattern:

1. `dnd/npc-occurrences`, `dnd/item-occurrences`, and
   `dnd/location-occurrences` place durable registry IDs in model-visible
   projections and require the private LLM response to repeat those IDs.
2. NPC-, item-, and location-registry normalization use the shared entity
   reconciliation prompt, which labels candidates with opaque
   `candidate-000001`-style keys and requires the model to copy those keys into
   duplicate-group proposals.

The durable occurrence artifacts correctly retain canonical entity IDs. The
problem is the private model transport contract, not the published artifact
contract.

## Target Architecture

### Model proposals and durable artifacts

Private LLM response types must express contextual semantic proposals rather
than reuse the durable artifact type when that type contains an opaque entity
ID. The extractor maps a validated private response into the existing durable
artifact only after identity resolution succeeds.

No affected durable artifact kind, media type, schema ID, schema version, or
JSON field changes as part of this work. NPC, item, and location occurrence
artifacts continue to publish their exact canonical ID/name pair. Registry
artifacts likewise retain their IDs and evidence.

The private schemas and prompt declarations may remain at their current `v1`
identities because Notarius is pre-release and these are not external
contracts. Their content hashes, mapping-policy fingerprints, and affected
prompt fingerprints must change so incompatible checkpoints are not reused.

### NPC occurrence grounding

The NPC occurrence prompt receives an ordered names-only projection of the
normalized NPC registry. Its private response contains the canonical NPC name,
occurrence kind, and current-transcript source ranges, but no `npc_id`.

The extractor resolves the returned name under the existing NPC comparison
policy. Resolution must produce exactly one registry entry. It then writes that
entry's canonical display name and durable ID into the `dnd.NPCOccurrence`.
An unknown or ambiguous selection invalidates the extraction operation; the
extractor must not guess, use fuzzy matching, silently omit the record, or
accept a partial response.

### Item occurrence grounding

The item occurrence prompt receives an ordered names-only projection of the
normalized item registry. Its private response contains the canonical item
name, occurrence kind, kind-specific fields, and current-transcript source
ranges, but no `item_id`.

The extractor resolves the returned name under the existing item comparison
and identity policies. Resolution must produce exactly one registry entry,
whose canonical name and durable ID are attached deterministically. Unknown or
ambiguous selections invalidate the complete extraction operation rather than
being guessed, repaired by similarity, or dropped.

### Location occurrence grounding

Location identity cannot always be resolved from a display name alone: the
current registry intentionally permits same-name locations with distinct
source anchors. The location occurrence prompt must therefore receive a
contextual registry descriptor that contains the canonical display name plus
the minimum source-grounded registry evidence needed to distinguish same-name
records. It must not contain the durable `location:sha256:...` value.

The private response uses two required selector fields: `name` and
`registry_refs`. For a comparison-unique canonical name, `registry_refs` is an
empty array and Notarius resolves the name under the location comparison
policy. For a name shared by multiple registry records, `registry_refs`
contains that record's complete canonically ordered registry ranges as
`start_unit_id` and `end_unit_id` pairs, without `source_id`.

Every model-facing registry entry uses one fixed shape with required `name`,
`registry_refs`, and `context` fields. `context` is an array of strict objects
containing only `unit_id` and `text`. Comparison-unique entries use empty
`registry_refs` and `context` arrays. Same-name entries use the complete
registry-range selector and the bounded context described below. The model
returns only `name` and `registry_refs`; it does not reproduce `context`.

For same-name groups, the projection also supplies bounded transcript units
covered by each record's registry ranges so the model receives meaningful
identity context rather than coordinates alone. Those ranges must resolve
against the current source document, and the resulting contextual selectors
must be unique. An invalid range or selector collision prevents the LLM call
and fails the operation. Unique-name entries do not repeat registry ranges or
context in the selector, preserving compatibility with a valid registry from
another source when the name alone is unambiguous.

The private response separately supplies current-transcript `source_refs` that
prove the occurrence. Registry identity evidence and occurrence evidence must
remain different fields and must never be merged. The model should omit an
occurrence when the transcript does not support choosing among same-name
locations. If a returned selector does not resolve to exactly one supplied
registry record, the extractor invalidates the complete operation rather than
guessing.

### Registry reconciliation

The shared entity-reconciliation input replaces opaque candidate keys with
contextual candidate descriptors. At minimum, a descriptor contains the
candidate's display name and its canonically ordered source-reference ranges;
the existing transcript windows remain available for semantic judgment.

Duplicate-group members and the canonical member in the private response use
the same contextual descriptor shape. The shared reconciliation helper maps
each descriptor back to exactly one internal candidate before assessing the
proposal. Exact deterministic duplicates should already be removed before the
LLM call; any remaining descriptor collision makes the affected candidate
ineligible for model-assisted reconciliation rather than authorizing an
arbitrary choice.

Existing safety behavior remains in force: groups must contain at least two
supplied candidates, the canonical candidate must be a member, groups must not
overlap, and domain-specific eligibility rules remain authoritative. Invalid,
ambiguous, or unsafe groups are discarded through the existing bounded
fallback and diagnostic behavior. The model never directly mutates the
durable registry.

The shared private reconciliation schema and helper must remain domain-neutral
within the D&D family. NPC-, item-, and location-specific duplicate policy
continues to live in the owning normalizer.

## Prompt And Asset Changes

The following LLM-facing assets are in scope:

- the prompt instructions, registry input fragments, and private response
  schemas for NPC, item, and location occurrences;
- the prompt manifests where input shape or selected fragments change;
- the shared D&D entity-reconciliation fragment and private response schema;
  and
- the NPC-, item-, and location-registry normalization prompt inputs that use
  the shared reconciliation contract.

Affected projections must exclude durable entity IDs rather than merely stop
mentioning them in prose. Prompt instructions should describe the contextual
selection rule once at the narrowest owning asset and must preserve the current
distinction between registry grounding and transcript evidence.

Prompt ordering and cache controls should remain unchanged unless the new
contextual input requires an intentional manifest change. Unrelated shared
prompt bytes should not be edited. Prompt and schema fingerprints should
invalidate only the operations whose selected assets or mapping semantics
changed.

## Code And Validation Changes

The occurrence extractors need private response types and deterministic
registry-resolution paths appropriate to their domain. Shared code is
appropriate only for demonstrated mechanics that have identical semantics;
NPC, item, and location ambiguity policies must not be forced behind a generic
resolver merely to reduce line count.

Registry projections should expose explicit model-facing methods whose names
describe whether they are names-only or contextual identity projections. The
existing ID/name projections may remain only for deterministic consumers that
genuinely require them; they must no longer be wired to an LLM input.

Mapping-policy and normalization-policy identifiers must be reviewed and
advanced wherever their semantics change. Checkpoint fingerprints must cover
the new projection content, private schema, prompt assets, and mapping policy,
while continuing to exclude irrelevant internal implementation details.

Durable occurrence normalizers and registry validators remain defense in
depth. They continue to validate exact ID/name pairs on artifacts entering
through checkpoints, codecs, or other boundaries even though the LLM no longer
produces the ID directly.

## Testing And Evaluation

Tests should protect the behavioral boundary rather than prompt prose or
private helper structure. The completed work should demonstrate that:

- affected model-facing registry projections do not contain durable entity
  IDs;
- private occurrence schemas reject opaque ID fields and accept the intended
  contextual shape;
- valid contextual selections map to the exact canonical durable ID/name pair;
- unknown, mismatched, and ambiguous selections fail without fuzzy matching,
  partial acceptance, or arbitrary reassignment;
- same-name locations remain distinguishable through contextual evidence;
- reconciliation preserves equal-name candidates, resolves valid contextual
  groups, and discards ambiguous or unsafe proposals;
- registry evidence never becomes occurrence evidence;
- durable codec, normalization, and validator behavior remains compatible; and
- representative assembled D&D pipelines still prepare and execute with fake
  structured-LLM responses.

Do not add repository-wide prompt-prose snapshots, exact-message-count tests,
or a change-detector test that merely scans for today's field names. Focused
projection, schema, mapping, fallback, and integration tests are the stable
owners of these risks. Model-quality evaluation with representative
transcripts remains a manual development aid rather than an offline test gate.

## Documentation And Architectural Record

This policy is durable and applies to future modules, so it warrants
`docs/adr/0012-resolve-opaque-entity-identifiers-deterministically.md`, which
records:

- the semantic-proposal versus referential-integrity boundary;
- why durable opaque IDs are excluded from model response contracts;
- why contextual evidence coordinates remain permitted;
- the narrowly scoped request-local-label exception;
- alternatives including durable IDs, names-only matching, and short opaque
  handles; and
- the consequences for private schemas, deterministic resolution, debugging,
  and ambiguous identities.

`docs/policy/architecture.md` states the general LLM boundary invariant and
links to the ADR. `docs/internal/dnd.md` describes the concrete occurrence
projections, contextual reconciliation selectors, resolution and failure
behavior, and the continued separation of registry grounding from occurrence
evidence. `docs/internal/llm.md` contains only a short clarification that
caller-owned modules, not PromptKit or the transport adapter, resolve
contextual model selections into application identities.

The NPC, item, and location occurrence and registry integration documents must
continue to own their durable wire contracts, while removing current claims
that the model-facing consumer projection contains `{id,name}` or that the raw
LLM response supplies the durable ID. They should instead explain that
Notarius resolves contextual model output and publishes the same exact durable
ID/name pair. No public schema examples need to remove those IDs.

The generic LLM-assisted deduplication entry in `docs/roadmap/future.md` must be
reconciled with this policy: stable IDs may exist inside deterministic state,
but a future model-facing proposal should use contextual selectors or a
documented request-local-label exception rather than durable IDs.

## Compatibility And Operational Effects

This work intentionally changes private prompt inputs, private structured
responses, and mapping semantics. It will invalidate affected checkpoints
through existing prompt, schema, projection, and policy fingerprints. No
manual checkpoint migration is required.

Durable D&D artifacts and generated-reference compatibility remain unchanged.
Operators do not receive new configuration fields or CLI controls. The feature
does not change PromptKit, provider routing, profile selection, retries,
concurrency, or public output placement.

## Non-Goals

This work does not:

- remove canonical IDs from durable registries or occurrence artifacts;
- change occurrence categories, evidence rules, or registry identity policy;
- add fuzzy, probabilistic, or embedding-based entity resolution;
- allow registry provenance to substitute for occurrence evidence;
- introduce a general entity graph or cross-artifact identity framework;
- redesign unrelated D&D prompts or their schemas;
- implement the future generic deduplication normalizer; or
- add provider-specific prompt behavior.

## Acceptance Criteria

The target state is complete when:

- no maintained D&D prompt requires a model to reproduce a durable opaque
  entity ID;
- current D&D reconciliation prompts no longer require opaque candidate keys;
- NPC, item, and location occurrence LLM outputs are resolved
  deterministically into their unchanged durable artifacts;
- same-name location and reconciliation cases remain safe and unambiguous;
- invalid contextual selections preserve the existing extraction-failure or
  normalization-fallback semantics appropriate to their stage;
- affected checkpoint identities change without altering public schema
  versions;
- focused and repository-wide tests pass offline;
- the ADR, architecture invariant, D&D internal guide, LLM internal guide,
  relevant integration contracts, and future roadmap accurately describe
  their canonical portions of the implemented policy; and
- no unrelated code, prompt behavior, or public contract changes are included.