notarius/docs/roadmap/dnd-npc-extraction.md

D&D NPC Extraction And Registry

Status: Accepted.

Purpose

Add a production D&D NPC pipeline that turns a session transcript into a canonical, evidence-backed NPC registry. The registry should be useful on its own and should serve as reference material for later sequential spell and combat-turn pipelines.

This is the next recommended increment because participant identity is already a demonstrated source of extraction error, while combat-turn extraction will depend on consistent identities across many more events. Establishing the NPC artifact first gives later pipelines a durable vocabulary without introducing a DAG or cross-lane orchestration.

Target Outcome

An operator can:

1. run an NPC pipeline against a D&D transcript;
2. receive a validated, normalized JSON NPC registry;
3. provide that registry as an explicit reference to a later spell pipeline; and
4. receive spell artifacts that prefer the registry's canonical NPC identities when resolving casters, targets, and other named participants.

The pipelines remain independent CLI invocations over the same input. Notarius does not automatically schedule them, discover prior output, or reconcile their results concurrently.

NPC Artifact Contract

The new typed artifact kind represents a list of NPC records. Each record has:

• a stable, deterministic ID derived by Notarius from the normalized NPC identity rather than authored freely by the model;
• one canonical in-world name or, for an unnamed but individually distinguishable NPC, one stable descriptive label;
• zero or more observed aliases;
• a concise transcript-grounded description;
• zero or more explicitly supported relationships to PCs, NPCs, groups, or locations; and
• one or more source references that collectively support the record's identity and every reported description, alias, and relationship.

The external contract defines the exact JSON shape, ID syntax, ordering, normalization, and compatibility policy. IDs are stable for equivalent normalized identities under the same policy; they do not promise permanence when human review or new evidence changes the canonical identity.

Descriptions report only facts established by the session transcript. General D&D knowledge, campaign reference material, and model inference do not supply biography, statistics, alignment, motivations, relationships, or outcomes. Reference material may disambiguate an identity but is not source evidence.

Inclusion Policy

Include an in-world non-PC participant when the transcript establishes that it appears, acts, speaks, or is materially discussed and either:

• gives it a proper name;
• gives it a stable alias or title; or
• distinguishes it individually in a way useful to later extraction, such as a titled attendant or a distinct item-bearing guard.

Exclude:

• human players, transcript speakers, and the GM as out-of-world people;
• player characters identified by the party or player references;
• incidental name drops, hypothetical examples, corrected transcription mistakes, and characters mentioned only by reference material;
• interchangeable unnamed crowds or groups whose members cannot be distinguished; and
• temporary summoned creatures or spell effects unless the transcript gives one an individual, persistent identity relevant beyond the summoning event.

An NPC may be friendly, hostile, neutral, allied with the party, controlled by a player temporarily, or absent from the current combat. Classification rests on whether the entity is an in-world non-PC participant, not on disposition or who rolls its dice.

Extraction Semantics

The extractor uses the existing shared D&D transcript, player, party, and glossary prompt inputs. It returns canonical in-world identities rather than speaker names and cites only transcript units.

Each record is intentionally concise. It captures identity and stable facts useful for recognizing the NPC elsewhere; it does not summarize every action, follow the NPC through the entire session, or duplicate combat-turn and narrative artifacts. Relationships must be stated or directly demonstrated by the cited transcript rather than inferred from D&D lore.

The model may propose canonical names, aliases, descriptions, relationships, and evidence. Deterministic code owns artifact IDs, structural integrity, ordering, safe consolidation, and validation of all source ranges.

Validation And Normalization

The production default validator chain remains deterministic. It must reject:

• malformed or empty required fields;
• missing, invalid, or foreign source references;
• duplicate or colliding IDs;
• an alias that resolves to more than one retained NPC;
• unsupported structural combinations defined by the artifact contract.

Relatedness checks may emit warnings when a proposed canonical name or alias is absent from its cited source text. A name legitimately disambiguated through opaque player, party, or glossary reference material may still produce that warning; the warning does not turn the reference into event evidence. Exclusion of PCs is prompt policy and a human-evaluation dimension until a future structured roster contract makes deterministic membership checks reliable.

Normalization is deterministic and conservative. It:

• normalizes whitespace, case-insensitive comparison keys, apostrophe variants, and exact source-reference order;
• consolidates records whose normalized canonical names match or whose canonical-name and alias sets establish one unambiguous identity;
• unions unique aliases, relationships, and source references in stable order;
• retains one supported description without synthesizing new prose; and
• emits scoped warnings for every canonicalization or collapsed record.

Ambiguous identities remain separate or cause a bounded validation rejection; the normalizer does not guess. Broader semantic reconciliation belongs to the future generic LLM-assisted deduplication work after real NPC output shows that deterministic identity overlap is insufficient.

Sequential Reference Use

The normalized NPC registry is accepted as an optional structured JSON reference by the D&D spell extractor. The spell prompt uses it to select exact canonical NPC names and aliases, just as party and player references ground PC identities. The registry does not establish that an NPC cast a spell and never becomes spell source evidence.

Reference binding stays explicit in configuration or through the existing CLI reference override. The maintained examples and operator documentation should demonstrate separate NPC and spell invocations over the same transcript, with the first output bound to the second invocation.

The reference contract should be designed for reuse by the future combat-turn extractor without adding combat-specific fields to the NPC artifact.

Provenance, Checkpoints, And Diagnostics

The NPC extractor and normalizer report their prompt, private schema, and normalization-policy identities through the existing manifest and prepared checkpoint-fingerprint contracts as applicable. Changes to any semantic input that can change NPC identity or consolidation intentionally produce cold checkpoint misses.

Raw transcript, reference, prompt, schema, and NPC content remain outside manifest metadata and checkpoint fingerprints. Existing debug and rejected- attempt behavior applies without a domain-specific filesystem surface.

Evaluation

Evaluation is human-reviewed and model-aware rather than a deterministic golden output gate. The initial corpus should include the existing D&D sessions that exposed player-versus-character attribution and repeated NPC appearances.

Review should score separately:

• NPC detection precision and recall;
• exclusion of PCs and out-of-world people;
• canonical identity and alias quality;
• duplicate consolidation;
• description and relationship fidelity;
• source-reference completeness; and
• usefulness as grounding for a subsequent spell extraction run.

Small-model development runs may expose more semantic errors than production frontier models. Deterministic contracts and validators should protect structure and provenance without attempting to turn subjective extraction quality into brittle fixture equality.

Out Of Scope

• Combat turns, initiative, actions, damage, conditions, or encounter state.
• General narrative or scene summaries.
• PC extraction or replacement of the existing party and player references.
• Automatic multi-pipeline scheduling, prior-output discovery, or DAG execution.
• An LLM-backed NPC validator or generic semantic deduplication normalizer.
• Campaign-wide identity persistence, a database, cross-session entity merges, or manual identity-editing UI.
• Retrieval, embeddings, reference summarization, or token-budget management.

Acceptance Criteria

• A configured D&D NPC lane produces a typed, durable JSON registry with the identity, evidence, and inclusion semantics above.
• Production shape, source-reference, identity-collision, and relatedness validators are registered with a deterministic default chain.
• The NPC normalizer assigns stable IDs and safely consolidates only unambiguous identities while preserving provenance and warnings.
• The registry can be bound directly to the spell extractor as an optional NPC reference, and spell extraction prefers its canonical NPC names without treating it as source evidence.
• Prompt, schema, reference, and normalization-policy changes participate in checkpoint identity where they can affect output.
• External, configuration, internal, operational, and maintained-example documentation is updated in its canonical location when the feature lands.
• Human review on representative transcripts demonstrates useful NPC grounding for a subsequent spell run without requiring exact model-output fixtures.