notarius/docs/roadmap/entity-registries.md

# Canonical D&D Entity Registries And Occurrences

## Purpose

This roadmap defines the target state for D&D entity extraction in Notarius.
NPCs, locations, and items should follow one legible architecture: a registry
lane first identifies canonical entities, and a later occurrence lane consumes
that registry to extract independently evidenced facts about those entities.

The work is a clean pre-release contract redesign. It should remove the current
mixture of bare plural, interaction, occurrence, and event terminology; add the
missing item registry; tighten entity eligibility; and make generated-reference
relationships visible in module, artifact, schema, prompt, validation, example,
and documentation names.

This roadmap owns the desired behavior and policy. The
[implementation plan](implementation.md) owns sequencing and task breakdowns.

## Background

Notarius already implements most of the required platform mechanics:

- ordered pipeline steps create a barrier between registry production and
  occurrence extraction;
- generated artifact references are validated for topology, media type,
  artifact kind, schema, and codec compatibility before execution;
- NPC and location registries have deterministic identities and bounded
  LLM-assisted reconciliation;
- NPC interactions and location occurrences consume normalized registry
  artifacts as source-free grounding; and
- validators keep registry provenance separate from occurrence evidence.

The remaining D&D domain contracts are inconsistent:

- `dnd/npcs` produces a registry, but its name does not communicate that role;
- `dnd/npc-interactions` is an occurrence lane under a narrower name;
- `dnd/locations` produces a registry while `dnd/location-occurrences` already
  uses the desired downstream terminology;
- `dnd/item-events` extracts occurrences directly and has no canonical item
  registry; and
- current location eligibility permits generic labels that are not suitable
  canonical nouns for reuse by downstream consumers.

[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md)
already establishes that changing encounter-level facts belong in occurrence
artifacts rather than registry attributes. [ADR-0008](../adr/0008-ordered-pipeline-steps.md)
already establishes explicit ordered artifact handoffs. This feature applies
those decisions consistently; it does not change the platform architecture.

## Goals

The completed feature should:

- make registry producers and occurrence consumers immediately recognizable by
  their names;
- provide registry and occurrence lanes for NPCs, locations, and items;
- require every entity occurrence to resolve to one canonical registry ID and
  display name;
- tighten registry eligibility so entries are stable nouns or designations
  that a downstream consumer can reuse;
- preserve minimal, evidence-grounded durable artifacts;
- keep registry provenance distinct from occurrence evidence;
- use ordered generated references rather than inferred dependencies;
- preserve domain-specific occurrence categories and identity rules where the
  entity types genuinely differ;
- use a clean pre-release migration with no aliases or compatibility shims; and
- leave the codebase with three concrete registry implementations from which a
  later generic deduplication design can be evaluated.

## Non-Goals

This work does not introduce:

- a general workflow DAG or implicit dependency inference;
- a generic framework-level LLM deduplication module;
- campaign-wide or cross-run registries;
- an inventory, currency balance, possession ledger, or current-state model;
- item-instance tracking for interchangeable objects;
- narrative descriptions, biographies, relationships, ownership summaries, or
  other enrichment on registry entries;
- occurrence-derived attributes copied back onto registry entries;
- NPC IDs on spell, combat-turn, or enemy-event artifacts merely because those
  lanes consume NPC grounding;
- migration tooling for old checkpoints or durable artifacts; or
- compatibility aliases for retired module keys, artifact kinds, prompt IDs,
  reference slots, schemas, or validator keys.

## Canonical Naming

### Modules

Registry extractors and their matching normalizers use
`dnd/<singular-entity>-registry`. Occurrence extractors and their matching
normalizers use `dnd/<singular-entity>-occurrences`.

| Role | Current module key | Target module key |
| --- | --- | --- |
| NPC registry | `dnd/npcs` | `dnd/npc-registry` |
| NPC occurrences | `dnd/npc-interactions` | `dnd/npc-occurrences` |
| Location registry | `dnd/locations` | `dnd/location-registry` |
| Location occurrences | `dnd/location-occurrences` | `dnd/location-occurrences` |
| Item registry | none | `dnd/item-registry` |
| Item occurrences | `dnd/item-events` | `dnd/item-occurrences` |

The same key is used for a lane's extract and normalize modules, as it is for
other current D&D lanes. Stage identity continues to distinguish those module
registrations.

### Durable artifacts

Registry artifact kinds name the registry directly. Occurrence artifact kinds
retain the `-list` suffix because their durable value is an ordered collection
of independently evidenced observations.

| Role | Target artifact kind | Target schema ID | Root collection |
| --- | --- | --- | --- |
| NPC registry | `dnd/npc-registry` | `notarius.dnd.npc_registry` | `npcs` |
| NPC occurrences | `dnd/npc-occurrence-list` | `notarius.dnd.npc_occurrences` | `occurrences` |
| Location registry | `dnd/location-registry` | `notarius.dnd.location_registry` | `locations` |
| Location occurrences | `dnd/location-occurrence-list` | `notarius.dnd.location_occurrences` | `occurrences` |
| Item registry | `dnd/item-registry` | `notarius.dnd.item_registry` | `items` |
| Item occurrences | `dnd/item-occurrence-list` | `notarius.dnd.item_occurrences` | `occurrences` |

All redesigned durable schemas remain version `v1`. Notarius is pre-release,
and these names replace rather than version or alias the retired contracts.
Schema names should follow the corresponding underscore form, such as
`notarius_dnd_item_registry_v1`.

Canonical Go types, codec packages, validator packages, prompt IDs, private
response-schema identities, capability names, and asset subtrees should use the
same registry/occurrence vocabulary. Prompt IDs should be
`dnd.<entity>_registry`, `dnd.<entity>_registry.normalize`, and
`dnd.<entity>_occurrences` as applicable.

The corresponding durable integration contracts should be:

- `docs/integrations/dnd-npc-registry-artifacts.md`;
- `docs/integrations/dnd-npc-occurrence-artifacts.md`;
- `docs/integrations/dnd-location-registry-artifacts.md`;
- `docs/integrations/dnd-location-occurrence-artifacts.md`;
- `docs/integrations/dnd-item-registry-artifacts.md`; and
- `docs/integrations/dnd-item-occurrence-artifacts.md`.

Those documents will become canonical only when the contracts are implemented;
until then, this roadmap is the sole owner of the proposed names and behavior.

### References

Registry-consuming modules expose explicit reference slots named:

- `npc_registry`;
- `location_registry`; and
- `item_registry`.

Generated-reference aliases in maintained examples should use those same names.
The reference slot, not a matching lane label, owns dependency resolution. Each
slot accepts only the corresponding registry artifact kind and media type and
retains an explicit size bound.

Existing NPC-grounded consumers such as spells, combat turns, enemy events, and
NPC occurrences should use `npc_registry`. Location occurrences should use
`location_registry`, and item occurrences should require `item_registry` at
both extraction and normalization.

## Registry Contract

Every registry answers one narrow question: which reusable canonical entities
of this type are established by the current transcript?

Each registry record contains only:

- a deterministic application-owned `id`;
- a canonical display `name`; and
- one or more transcript `source_refs` establishing the entity identity.

Registries do not contain encounter roles, occurrence categories, descriptions,
summaries, aliases, relationships, inventories, holder state, or inferred
attributes. Their source references establish registry provenance only.

Registry extraction remains chunk-scoped. Merge and normalization combine
candidate records across chunks, canonicalize evidence in source-document
order, apply deterministic identity rules, and may use a bounded structured-LLM
proposal to reconcile semantic duplicates. The model proposes duplicate groups;
deterministic code validates keys, membership, overlap, canonical selection,
and application. Invalid or uncertain proposals preserve the safe deterministic
result and produce bounded diagnostics.

The existing shared D&D entity-reconciliation substrate should support all
three registries. Entity-specific candidate construction, identity judgment,
canonical-name preference, and consolidation remain in the owning D&D module.
This feature must not move D&D policy into generic framework packages.

## Entity Eligibility And Identity

Eligibility is based on whether the transcript establishes a stable reusable
referent, not on capitalization alone. Transcript capitalization is unreliable,
and a capitalized generic phrase is not automatically a canonical entity.

### NPC registry

Include an NPC when the transcript establishes an individually identifiable
non-player character through a proper name or a stable unique designation. A
title or descriptive designation qualifies only when the transcript uses it as
a persistent identity, such as `the Masked Envoy`; a transient generic role
such as `a guard`, `the bartender`, or `one of the bandits` does not qualify.

Exclude player characters, speakers known only by player identity, anonymous
groups, creature categories, generic roles, speculative identities, and labels
invented merely to make an occurrence referable.

NPC IDs use the `npc:sha256:` prefix and identity policy
`dnd.npc_registry.identity.v1`. The digest is the lowercase SHA-256 of compact
JSON for `[policy, comparison_name]`, where the comparison name uses the
existing Unicode, apostrophe, whitespace, and case normalization. Semantic
reconciliation may collapse supported aliases but may not merge distinct people
merely because their names or roles are similar.

### Location registry

Include a location only when the transcript establishes a stable proper name or
unique in-world designation, such as `Waterdeep`, `The Yawning Portal`, or a
uniquely named chamber. Exclude generic, temporary, relative, or purely
descriptive references such as `the room`, `the bar`, `the hallway`, `outside`,
or `upstairs`.

A later occurrence may resolve a generic phrase such as `the bar` to an existing
named registry location when the current transcript context supports that
coreference. The generic phrase must not create a new registry entry.

Location IDs use the `location:sha256:` prefix and identity policy
`dnd.location_registry.identity.v1`. The digest is the lowercase SHA-256 of
compact JSON for `[policy, comparison_name, source_id, start_unit_id,
end_unit_id]`, using the earliest canonical evidence anchor. Source anchoring
keeps distinct places with the same display name distinguishable. Nested places
and same-name places remain distinct unless a validated semantic proposal
establishes that candidates identify the same physical place.

### Item registry

Items do not need proper names. Include a reusable, transcript-established item
name at the narrowest useful type or unique-designation level, including:

- named unique objects;
- concrete item types such as `Healing Potion` when the transcript establishes
  that type;
- stable unique designations used by the session; and
- individual currency denominations such as `Silver Pieces`.

Exclude vague or non-reusable descriptions such as `loot`, `treasure`, `some
gear`, `a weapon`, or `something valuable` unless the transcript supplies a
more stable name or designation. Do not infer mechanical subtype, magic
properties, denomination, quantity, or uniqueness.

An item-registry record identifies a canonical item kind or designation, not a
physical instance. Interchangeable objects with the same canonical name share
one registry identity and are distinguished in occurrences by evidence,
quantity, and holder transitions. A genuinely named unique object naturally
has its own canonical identity.

Item IDs use the `item:sha256:` prefix and identity policy
`dnd.item_registry.identity.v1`. The digest is the lowercase SHA-256 of compact
JSON for `[policy, comparison_name]`, using the same text normalization family
as other entity registries. Item normalization may reconcile transcript-
supported aliases and abbreviation variants but may not collapse distinct
denominations or materially different item types.

## Occurrence Contract

Every occurrence answers a separate question: what source-grounded event or
relationship to a known registry entity occurred in this transcript passage?

Each occurrence record contains:

- the exact entity ID from its required registry;
- the exact canonical display name associated with that ID;
- one bounded occurrence kind; and
- one or more source references from the current transcript.

Entity-specific fields are allowed only when required by that occurrence
contract. Registry IDs and names provide grounding; registry source references
never become occurrence evidence. Unknown IDs and mismatched ID/name pairs are
rejected rather than guessed, reassigned, or repaired from display-name
similarity.

Occurrence extractors may resolve an alias, pronoun, generic reference, or
abbreviation to an existing registry entry only when the current chunk supports
that coreference. They may not add entities to the registry. Omit an occurrence
when entity resolution or the occurrence itself is not adequately supported.

Occurrence normalization is deterministic. It validates registry membership,
canonicalizes the display name from the registry, orders and deduplicates source
references, preserves distinct observations, orders records by source-document
chronology and stable domain tie-breakers, and collapses only exact duplicates.

### NPC occurrences

The durable record contains required `npc_id`, `name`, `kind`, and
`source_refs`. It retains the current interaction categories:

- `mentioned`;
- `noncombat_presence`;
- `dialogue`;
- `combat_ally`;
- `combat_opponent`; and
- `other`.

The precedence and splitting rules of the current NPC-interaction contract
remain applicable. Renaming the artifact does not turn it into a biography,
relationship model, sentiment analysis, or persistent NPC state.

Enemy-event grounding consumes `combat_opponent` NPC occurrences under the new
contract. Other NPC-grounded lanes continue to consume the registry directly.

### Location occurrences

The durable record continues to contain required `location_id`, `name`, `kind`,
and `source_refs`. It retains the current categories and precedence:

- `visited`;
- `planned`;
- `recalled`; and
- `mentioned`.

Generic current-chunk language may resolve to a named registry location, but
the occurrence must cite the current transcript passage that supports both the
resolution and the occurrence category.

### Item occurrences

The durable record contains required `item_id`, `name`, `kind`, and
`source_refs`, plus the current semantically conditional `quantity`, `from`,
and `to` fields. It retains the current categories:

- `discovered`;
- `acquired`;
- `lost`;
- `consumed`; and
- `transferred`.

Holder and quantity rules remain unchanged: discovery has no holder; acquisition
requires a gaining holder; loss and consumption require a losing holder; and
transfer requires distinct losing and gaining holders. Currency remains an
ordinary item occurrence with an explicit denomination and a quantity only
when the transcript establishes one.

Item occurrences do not compute inventory, merge successive state changes,
convert denominations, infer unspoken holders, or distinguish physical
instances that the transcript does not identify.

Private LLM response schemas should remain strict and compatible with providers
that require every declared property to be required. Nullable or explicit
absence representations may be used privately for semantically conditional
item fields; deterministic mapping owns omission from the durable artifact when
the public contract calls for an absent field.

## Pipeline End State

A complete D&D pipeline produces all three registries in an earlier step and
binds them explicitly into later consumers. Conceptually:

```text
registry step
  npc-registry
  location-registry
  item-registry
  scene-descriptions

occurrence and action step
  npc-occurrences      <- npc-registry
  location-occurrences <- location-registry
  item-occurrences     <- item-registry
  spells               <- optional npc-registry
  combat-turns         <- optional npc-registry + required scene descriptions

derived enemy step
  enemy-events <- npc-registry + npc-occurrences + combat turns + scene descriptions
```

Steps remain barriers with the existing whole-run failure behavior. A required
registry producer that fails, rejects, or does not publish an accepted
normalized artifact prevents its dependent later step and fails the run under
the existing ordered-pipeline contract. Configuration with a same-step or
forward generated reference remains invalid.

The maintained complete D&D example should use generated references for these
same-run handoffs. Registry slots continue to accept compatible external path
references under the existing configuration contract; this feature does not
remove the ability to supply a previously produced or independently maintained
registry artifact.

The shared scene plan remains pipeline-wide. This feature changes eligible
lanes and reference bindings, not chunking or scheduling semantics.

## Validation And Normalization

Each registry receives production defaults for:

- JSON and durable-schema validity;
- registry shape and required evidence;
- deterministic identity and uniqueness;
- source-reference validity;
- source relatedness; and
- safe bounded reconciliation during normalization.

Each occurrence family receives production defaults for:

- JSON and durable-schema validity;
- occurrence shape and kind-specific invariants;
- exact registry ID/name membership;
- source-reference validity;
- source relatedness; and
- deterministic ordering and exact-duplicate removal.

Validator and capability keys should adopt the new module vocabulary. Registry
membership validators consume the same immutable registry projection used by
the extractor and normalizer. Validators do not use registry provenance as a
substitute for occurrence evidence and do not mutate supplied artifacts or
references.

Warnings remain bounded, content-safe, and deterministic. Prompt, schema,
registry, and identity fingerprints remain represented by hashes or stable
identities rather than raw content in manifests and diagnostics.

## Package And Asset Organization

Implementation packages should make the two roles visible, using Go-compatible
names such as `npcregistry`, `npcoccurrences`, `locationregistry`, and
`itemoccurrences` beneath the existing extract, normalize, codec, and validate
families. Entity-specific identity and immutable registry-resolution helpers
remain inside the D&D domain.

LLM assets should use corresponding subtrees beneath `assets/dnd/`, for example
`assets/dnd/npc-registry/{extract,normalize}` and
`assets/dnd/item-occurrences`. The root `assets` package remains a single-file,
content-only filesystem boundary with no business logic, registration, or
PromptKit dependency.

All extraction prompts retain the shared D&D prompt prefix and chunk scope.
Registry prompts own only entity eligibility and canonical-name policy;
occurrence prompts own only occurrence semantics and entity-specific fields.
Registry input fragments own the meaning of their source-free projections.

## Migration Policy

The redesign is intentionally breaking and immediate:

- remove retired module and validator keys;
- remove retired artifact kinds, schema IDs, schema names, Go types, prompt IDs,
  private schema identities, reference slots, capability names, asset paths,
  and generated-reference aliases;
- do not register aliases or decode old shapes;
- do not retain duplicate integration documents for old contracts; and
- accept invalidation of old checkpoints, prompt fingerprints, provider caches,
  example output, and other reconstructible pre-release state.

New contracts use version `v1`; `v2` does not communicate useful compatibility
information while there is no supported release to preserve. Current-behavior
documentation, maintained examples, module catalogs, and integration contracts
must change atomically with implementation.

## Evaluation Expectations

Implementation should be assessed against representative transcript cases that
cover:

- named NPCs, stable NPC designations, transient roles, aliases, and anonymous
  groups;
- named locations, same-name locations, nested locations, generic spatial
  phrases, and supported generic coreferences to named places;
- named items, ordinary item types, ambiguous descriptions, aliases, quantities,
  transfers, consumption, and each currency denomination;
- registry candidates mentioned across multiple chunks;
- unsupported occurrence links and mismatched registry ID/name pairs; and
- output from cost-effective smaller models as well as the default production
  profile.

Deterministic tests should protect schemas, identity rules, registry resolution,
reference compatibility, ordering, validation, normalization, and assembled
multi-step configuration. Live model quality remains an opt-in human evaluation
activity rather than a default automated gate.

## Target End State

The feature is complete when:

- the six canonical entity modules are the only selectable NPC, location, and
  item registry/occurrence modules;
- all six durable artifacts use the target kinds, schema identities, root
  collections, minimal record shapes, and `v1` contracts defined here;
- every occurrence contains and validates an exact registry ID/name pair;
- NPC, location, and item registry eligibility follows the policies above;
- the maintained complete pipeline uses explicit generated registry references
  across ordered steps, while compatible external registry references remain
  supported;
- item occurrences no longer run independently of an accepted item registry;
- spell, combat, enemy, and occurrence consumers use the renamed registry slots
  without treating registry provenance as evidence;
- current module catalogs, default validator chains, prompts, assets, examples,
  integration contracts, internal documentation, and tests use only the new
  vocabulary; and
- repository searches find no production compatibility aliases or stale current-
  behavior claims for `dnd/npcs`, `dnd/npc-interactions`, `dnd/locations`, or
  `dnd/item-events`.