Plan canonical D&D entity registries and occurrences

2026-08-05 18:16:37 +00:00
parent 299c110267
commit 916a32195b
4 changed files with 1248 additions and 996 deletions
--- a/docs/roadmap/entity-registries.md
+++ b/docs/roadmap/entity-registries.md
@@ -0,0 +1,503 @@
+# Canonical D&D Entity Registries And Occurrences
+
+## Purpose
+
+This roadmap defines the target state for D&D entity extraction in Notarius.
+NPCs, locations, and items should follow one legible architecture: a registry
+lane first identifies canonical entities, and a later occurrence lane consumes
+that registry to extract independently evidenced facts about those entities.
+
+The work is a clean pre-release contract redesign. It should remove the current
+mixture of bare plural, interaction, occurrence, and event terminology; add the
+missing item registry; tighten entity eligibility; and make generated-reference
+relationships visible in module, artifact, schema, prompt, validation, example,
+and documentation names.
+
+This roadmap owns the desired behavior and policy. The
+[implementation plan](implementation.md) owns sequencing and task breakdowns.
+
+## Background
+
+Notarius already implements most of the required platform mechanics:
+
+- ordered pipeline steps create a barrier between registry production and
+  occurrence extraction;
+- generated artifact references are validated for topology, media type,
+  artifact kind, schema, and codec compatibility before execution;
+- NPC and location registries have deterministic identities and bounded
+  LLM-assisted reconciliation;
+- NPC interactions and location occurrences consume normalized registry
+  artifacts as source-free grounding; and
+- validators keep registry provenance separate from occurrence evidence.
+
+The remaining D&D domain contracts are inconsistent:
+
+- `dnd/npcs` produces a registry, but its name does not communicate that role;
+- `dnd/npc-interactions` is an occurrence lane under a narrower name;
+- `dnd/locations` produces a registry while `dnd/location-occurrences` already
+  uses the desired downstream terminology;
+- `dnd/item-events` extracts occurrences directly and has no canonical item
+  registry; and
+- current location eligibility permits generic labels that are not suitable
+  canonical nouns for reuse by downstream consumers.
+
+[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md)
+already establishes that changing encounter-level facts belong in occurrence
+artifacts rather than registry attributes. [ADR-0008](../adr/0008-ordered-pipeline-steps.md)
+already establishes explicit ordered artifact handoffs. This feature applies
+those decisions consistently; it does not change the platform architecture.
+
+## Goals
+
+The completed feature should:
+
+- make registry producers and occurrence consumers immediately recognizable by
+  their names;
+- provide registry and occurrence lanes for NPCs, locations, and items;
+- require every entity occurrence to resolve to one canonical registry ID and
+  display name;
+- tighten registry eligibility so entries are stable nouns or designations
+  that a downstream consumer can reuse;
+- preserve minimal, evidence-grounded durable artifacts;
+- keep registry provenance distinct from occurrence evidence;
+- use ordered generated references rather than inferred dependencies;
+- preserve domain-specific occurrence categories and identity rules where the
+  entity types genuinely differ;
+- use a clean pre-release migration with no aliases or compatibility shims; and
+- leave the codebase with three concrete registry implementations from which a
+  later generic deduplication design can be evaluated.
+
+## Non-Goals
+
+This work does not introduce:
+
+- a general workflow DAG or implicit dependency inference;
+- a generic framework-level LLM deduplication module;
+- campaign-wide or cross-run registries;
+- an inventory, currency balance, possession ledger, or current-state model;
+- item-instance tracking for interchangeable objects;
+- narrative descriptions, biographies, relationships, ownership summaries, or
+  other enrichment on registry entries;
+- occurrence-derived attributes copied back onto registry entries;
+- NPC IDs on spell, combat-turn, or enemy-event artifacts merely because those
+  lanes consume NPC grounding;
+- migration tooling for old checkpoints or durable artifacts; or
+- compatibility aliases for retired module keys, artifact kinds, prompt IDs,
+  reference slots, schemas, or validator keys.
+
+## Canonical Naming
+
+### Modules
+
+Registry extractors and their matching normalizers use
+`dnd/<singular-entity>-registry`. Occurrence extractors and their matching
+normalizers use `dnd/<singular-entity>-occurrences`.
+
+| Role | Current module key | Target module key |
+| --- | --- | --- |
+| NPC registry | `dnd/npcs` | `dnd/npc-registry` |
+| NPC occurrences | `dnd/npc-interactions` | `dnd/npc-occurrences` |
+| Location registry | `dnd/locations` | `dnd/location-registry` |
+| Location occurrences | `dnd/location-occurrences` | `dnd/location-occurrences` |
+| Item registry | none | `dnd/item-registry` |
+| Item occurrences | `dnd/item-events` | `dnd/item-occurrences` |
+
+The same key is used for a lane's extract and normalize modules, as it is for
+other current D&D lanes. Stage identity continues to distinguish those module
+registrations.
+
+### Durable artifacts
+
+Registry artifact kinds name the registry directly. Occurrence artifact kinds
+retain the `-list` suffix because their durable value is an ordered collection
+of independently evidenced observations.
+
+| Role | Target artifact kind | Target schema ID | Root collection |
+| --- | --- | --- | --- |
+| NPC registry | `dnd/npc-registry` | `notarius.dnd.npc_registry` | `npcs` |
+| NPC occurrences | `dnd/npc-occurrence-list` | `notarius.dnd.npc_occurrences` | `occurrences` |
+| Location registry | `dnd/location-registry` | `notarius.dnd.location_registry` | `locations` |
+| Location occurrences | `dnd/location-occurrence-list` | `notarius.dnd.location_occurrences` | `occurrences` |
+| Item registry | `dnd/item-registry` | `notarius.dnd.item_registry` | `items` |
+| Item occurrences | `dnd/item-occurrence-list` | `notarius.dnd.item_occurrences` | `occurrences` |
+
+All redesigned durable schemas remain version `v1`. Notarius is pre-release,
+and these names replace rather than version or alias the retired contracts.
+Schema names should follow the corresponding underscore form, such as
+`notarius_dnd_item_registry_v1`.
+
+Canonical Go types, codec packages, validator packages, prompt IDs, private
+response-schema identities, capability names, and asset subtrees should use the
+same registry/occurrence vocabulary. Prompt IDs should be
+`dnd.<entity>_registry`, `dnd.<entity>_registry.normalize`, and
+`dnd.<entity>_occurrences` as applicable.
+
+The corresponding durable integration contracts should be:
+
+- `docs/integrations/dnd-npc-registry-artifacts.md`;
+- `docs/integrations/dnd-npc-occurrence-artifacts.md`;
+- `docs/integrations/dnd-location-registry-artifacts.md`;
+- `docs/integrations/dnd-location-occurrence-artifacts.md`;
+- `docs/integrations/dnd-item-registry-artifacts.md`; and
+- `docs/integrations/dnd-item-occurrence-artifacts.md`.
+
+Those documents will become canonical only when the contracts are implemented;
+until then, this roadmap is the sole owner of the proposed names and behavior.
+
+### References
+
+Registry-consuming modules expose explicit reference slots named:
+
+- `npc_registry`;
+- `location_registry`; and
+- `item_registry`.
+
+Generated-reference aliases in maintained examples should use those same names.
+The reference slot, not a matching lane label, owns dependency resolution. Each
+slot accepts only the corresponding registry artifact kind and media type and
+retains an explicit size bound.
+
+Existing NPC-grounded consumers such as spells, combat turns, enemy events, and
+NPC occurrences should use `npc_registry`. Location occurrences should use
+`location_registry`, and item occurrences should require `item_registry` at
+both extraction and normalization.
+
+## Registry Contract
+
+Every registry answers one narrow question: which reusable canonical entities
+of this type are established by the current transcript?
+
+Each registry record contains only:
+
+- a deterministic application-owned `id`;
+- a canonical display `name`; and
+- one or more transcript `source_refs` establishing the entity identity.
+
+Registries do not contain encounter roles, occurrence categories, descriptions,
+summaries, aliases, relationships, inventories, holder state, or inferred
+attributes. Their source references establish registry provenance only.
+
+Registry extraction remains chunk-scoped. Merge and normalization combine
+candidate records across chunks, canonicalize evidence in source-document
+order, apply deterministic identity rules, and may use a bounded structured-LLM
+proposal to reconcile semantic duplicates. The model proposes duplicate groups;
+deterministic code validates keys, membership, overlap, canonical selection,
+and application. Invalid or uncertain proposals preserve the safe deterministic
+result and produce bounded diagnostics.
+
+The existing shared D&D entity-reconciliation substrate should support all
+three registries. Entity-specific candidate construction, identity judgment,
+canonical-name preference, and consolidation remain in the owning D&D module.
+This feature must not move D&D policy into generic framework packages.
+
+## Entity Eligibility And Identity
+
+Eligibility is based on whether the transcript establishes a stable reusable
+referent, not on capitalization alone. Transcript capitalization is unreliable,
+and a capitalized generic phrase is not automatically a canonical entity.
+
+### NPC registry
+
+Include an NPC when the transcript establishes an individually identifiable
+non-player character through a proper name or a stable unique designation. A
+title or descriptive designation qualifies only when the transcript uses it as
+a persistent identity, such as `the Masked Envoy`; a transient generic role
+such as `a guard`, `the bartender`, or `one of the bandits` does not qualify.
+
+Exclude player characters, speakers known only by player identity, anonymous
+groups, creature categories, generic roles, speculative identities, and labels
+invented merely to make an occurrence referable.
+
+NPC IDs use the `npc:sha256:` prefix and identity policy
+`dnd.npc_registry.identity.v1`. The digest is the lowercase SHA-256 of compact
+JSON for `[policy, comparison_name]`, where the comparison name uses the
+existing Unicode, apostrophe, whitespace, and case normalization. Semantic
+reconciliation may collapse supported aliases but may not merge distinct people
+merely because their names or roles are similar.
+
+### Location registry
+
+Include a location only when the transcript establishes a stable proper name or
+unique in-world designation, such as `Waterdeep`, `The Yawning Portal`, or a
+uniquely named chamber. Exclude generic, temporary, relative, or purely
+descriptive references such as `the room`, `the bar`, `the hallway`, `outside`,
+or `upstairs`.
+
+A later occurrence may resolve a generic phrase such as `the bar` to an existing
+named registry location when the current transcript context supports that
+coreference. The generic phrase must not create a new registry entry.
+
+Location IDs use the `location:sha256:` prefix and identity policy
+`dnd.location_registry.identity.v1`. The digest is the lowercase SHA-256 of
+compact JSON for `[policy, comparison_name, source_id, start_unit_id,
+end_unit_id]`, using the earliest canonical evidence anchor. Source anchoring
+keeps distinct places with the same display name distinguishable. Nested places
+and same-name places remain distinct unless a validated semantic proposal
+establishes that candidates identify the same physical place.
+
+### Item registry
+
+Items do not need proper names. Include a reusable, transcript-established item
+name at the narrowest useful type or unique-designation level, including:
+
+- named unique objects;
+- concrete item types such as `Healing Potion` when the transcript establishes
+  that type;
+- stable unique designations used by the session; and
+- individual currency denominations such as `Silver Pieces`.
+
+Exclude vague or non-reusable descriptions such as `loot`, `treasure`, `some
+gear`, `a weapon`, or `something valuable` unless the transcript supplies a
+more stable name or designation. Do not infer mechanical subtype, magic
+properties, denomination, quantity, or uniqueness.
+
+An item-registry record identifies a canonical item kind or designation, not a
+physical instance. Interchangeable objects with the same canonical name share
+one registry identity and are distinguished in occurrences by evidence,
+quantity, and holder transitions. A genuinely named unique object naturally
+has its own canonical identity.
+
+Item IDs use the `item:sha256:` prefix and identity policy
+`dnd.item_registry.identity.v1`. The digest is the lowercase SHA-256 of compact
+JSON for `[policy, comparison_name]`, using the same text normalization family
+as other entity registries. Item normalization may reconcile transcript-
+supported aliases and abbreviation variants but may not collapse distinct
+denominations or materially different item types.
+
+## Occurrence Contract
+
+Every occurrence answers a separate question: what source-grounded event or
+relationship to a known registry entity occurred in this transcript passage?
+
+Each occurrence record contains:
+
+- the exact entity ID from its required registry;
+- the exact canonical display name associated with that ID;
+- one bounded occurrence kind; and
+- one or more source references from the current transcript.
+
+Entity-specific fields are allowed only when required by that occurrence
+contract. Registry IDs and names provide grounding; registry source references
+never become occurrence evidence. Unknown IDs and mismatched ID/name pairs are
+rejected rather than guessed, reassigned, or repaired from display-name
+similarity.
+
+Occurrence extractors may resolve an alias, pronoun, generic reference, or
+abbreviation to an existing registry entry only when the current chunk supports
+that coreference. They may not add entities to the registry. Omit an occurrence
+when entity resolution or the occurrence itself is not adequately supported.
+
+Occurrence normalization is deterministic. It validates registry membership,
+canonicalizes the display name from the registry, orders and deduplicates source
+references, preserves distinct observations, orders records by source-document
+chronology and stable domain tie-breakers, and collapses only exact duplicates.
+
+### NPC occurrences
+
+The durable record contains required `npc_id`, `name`, `kind`, and
+`source_refs`. It retains the current interaction categories:
+
+- `mentioned`;
+- `noncombat_presence`;
+- `dialogue`;
+- `combat_ally`;
+- `combat_opponent`; and
+- `other`.
+
+The precedence and splitting rules of the current NPC-interaction contract
+remain applicable. Renaming the artifact does not turn it into a biography,
+relationship model, sentiment analysis, or persistent NPC state.
+
+Enemy-event grounding consumes `combat_opponent` NPC occurrences under the new
+contract. Other NPC-grounded lanes continue to consume the registry directly.
+
+### Location occurrences
+
+The durable record continues to contain required `location_id`, `name`, `kind`,
+and `source_refs`. It retains the current categories and precedence:
+
+- `visited`;
+- `planned`;
+- `recalled`; and
+- `mentioned`.
+
+Generic current-chunk language may resolve to a named registry location, but
+the occurrence must cite the current transcript passage that supports both the
+resolution and the occurrence category.
+
+### Item occurrences
+
+The durable record contains required `item_id`, `name`, `kind`, and
+`source_refs`, plus the current semantically conditional `quantity`, `from`,
+and `to` fields. It retains the current categories:
+
+- `discovered`;
+- `acquired`;
+- `lost`;
+- `consumed`; and
+- `transferred`.
+
+Holder and quantity rules remain unchanged: discovery has no holder; acquisition
+requires a gaining holder; loss and consumption require a losing holder; and
+transfer requires distinct losing and gaining holders. Currency remains an
+ordinary item occurrence with an explicit denomination and a quantity only
+when the transcript establishes one.
+
+Item occurrences do not compute inventory, merge successive state changes,
+convert denominations, infer unspoken holders, or distinguish physical
+instances that the transcript does not identify.
+
+Private LLM response schemas should remain strict and compatible with providers
+that require every declared property to be required. Nullable or explicit
+absence representations may be used privately for semantically conditional
+item fields; deterministic mapping owns omission from the durable artifact when
+the public contract calls for an absent field.
+
+## Pipeline End State
+
+A complete D&D pipeline produces all three registries in an earlier step and
+binds them explicitly into later consumers. Conceptually:
+
+```text
+registry step
+  npc-registry
+  location-registry
+  item-registry
+  scene-descriptions
+
+occurrence and action step
+  npc-occurrences      <- npc-registry
+  location-occurrences <- location-registry
+  item-occurrences     <- item-registry
+  spells               <- optional npc-registry
+  combat-turns         <- optional npc-registry + required scene descriptions
+
+derived enemy step
+  enemy-events <- npc-registry + npc-occurrences + combat turns + scene descriptions
+```
+
+Steps remain barriers with the existing whole-run failure behavior. A required
+registry producer that fails, rejects, or does not publish an accepted
+normalized artifact prevents its dependent later step and fails the run under
+the existing ordered-pipeline contract. Configuration with a same-step or
+forward generated reference remains invalid.
+
+The maintained complete D&D example should use generated references for these
+same-run handoffs. Registry slots continue to accept compatible external path
+references under the existing configuration contract; this feature does not
+remove the ability to supply a previously produced or independently maintained
+registry artifact.
+
+The shared scene plan remains pipeline-wide. This feature changes eligible
+lanes and reference bindings, not chunking or scheduling semantics.
+
+## Validation And Normalization
+
+Each registry receives production defaults for:
+
+- JSON and durable-schema validity;
+- registry shape and required evidence;
+- deterministic identity and uniqueness;
+- source-reference validity;
+- source relatedness; and
+- safe bounded reconciliation during normalization.
+
+Each occurrence family receives production defaults for:
+
+- JSON and durable-schema validity;
+- occurrence shape and kind-specific invariants;
+- exact registry ID/name membership;
+- source-reference validity;
+- source relatedness; and
+- deterministic ordering and exact-duplicate removal.
+
+Validator and capability keys should adopt the new module vocabulary. Registry
+membership validators consume the same immutable registry projection used by
+the extractor and normalizer. Validators do not use registry provenance as a
+substitute for occurrence evidence and do not mutate supplied artifacts or
+references.
+
+Warnings remain bounded, content-safe, and deterministic. Prompt, schema,
+registry, and identity fingerprints remain represented by hashes or stable
+identities rather than raw content in manifests and diagnostics.
+
+## Package And Asset Organization
+
+Implementation packages should make the two roles visible, using Go-compatible
+names such as `npcregistry`, `npcoccurrences`, `locationregistry`, and
+`itemoccurrences` beneath the existing extract, normalize, codec, and validate
+families. Entity-specific identity and immutable registry-resolution helpers
+remain inside the D&D domain.
+
+LLM assets should use corresponding subtrees beneath `assets/dnd/`, for example
+`assets/dnd/npc-registry/{extract,normalize}` and
+`assets/dnd/item-occurrences`. The root `assets` package remains a single-file,
+content-only filesystem boundary with no business logic, registration, or
+PromptKit dependency.
+
+All extraction prompts retain the shared D&D prompt prefix and chunk scope.
+Registry prompts own only entity eligibility and canonical-name policy;
+occurrence prompts own only occurrence semantics and entity-specific fields.
+Registry input fragments own the meaning of their source-free projections.
+
+## Migration Policy
+
+The redesign is intentionally breaking and immediate:
+
+- remove retired module and validator keys;
+- remove retired artifact kinds, schema IDs, schema names, Go types, prompt IDs,
+  private schema identities, reference slots, capability names, asset paths,
+  and generated-reference aliases;
+- do not register aliases or decode old shapes;
+- do not retain duplicate integration documents for old contracts; and
+- accept invalidation of old checkpoints, prompt fingerprints, provider caches,
+  example output, and other reconstructible pre-release state.
+
+New contracts use version `v1`; `v2` does not communicate useful compatibility
+information while there is no supported release to preserve. Current-behavior
+documentation, maintained examples, module catalogs, and integration contracts
+must change atomically with implementation.
+
+## Evaluation Expectations
+
+Implementation should be assessed against representative transcript cases that
+cover:
+
+- named NPCs, stable NPC designations, transient roles, aliases, and anonymous
+  groups;
+- named locations, same-name locations, nested locations, generic spatial
+  phrases, and supported generic coreferences to named places;
+- named items, ordinary item types, ambiguous descriptions, aliases, quantities,
+  transfers, consumption, and each currency denomination;
+- registry candidates mentioned across multiple chunks;
+- unsupported occurrence links and mismatched registry ID/name pairs; and
+- output from cost-effective smaller models as well as the default production
+  profile.
+
+Deterministic tests should protect schemas, identity rules, registry resolution,
+reference compatibility, ordering, validation, normalization, and assembled
+multi-step configuration. Live model quality remains an opt-in human evaluation
+activity rather than a default automated gate.
+
+## Target End State
+
+The feature is complete when:
+
+- the six canonical entity modules are the only selectable NPC, location, and
+  item registry/occurrence modules;
+- all six durable artifacts use the target kinds, schema identities, root
+  collections, minimal record shapes, and `v1` contracts defined here;
+- every occurrence contains and validates an exact registry ID/name pair;
+- NPC, location, and item registry eligibility follows the policies above;
+- the maintained complete pipeline uses explicit generated registry references
+  across ordered steps, while compatible external registry references remain
+  supported;
+- item occurrences no longer run independently of an accepted item registry;
+- spell, combat, enemy, and occurrence consumers use the renamed registry slots
+  without treating registry provenance as evidence;
+- current module catalogs, default validator chains, prompts, assets, examples,
+  integration contracts, internal documentation, and tests use only the new
+  vocabulary; and
+- repository searches find no production compatibility aliases or stale current-
+  behavior claims for `dnd/npcs`, `dnd/npc-interactions`, `dnd/locations`, or
+  `dnd/item-events`.