notarius/docs/roadmap/dnd-npc-extraction.md

# D&D NPC Extraction And Registry

Status: Accepted.

## Purpose

Add a production D&D NPC pipeline that turns a session transcript into a
canonical, evidence-backed NPC registry. The registry should be useful on its
own and should serve as reference material for later sequential spell and
combat-turn pipelines.

This is the next recommended increment because participant identity is already
a demonstrated source of extraction error, while combat-turn extraction will
depend on consistent identities across many more events. Establishing the NPC
artifact first gives later pipelines a durable vocabulary without introducing
a DAG or cross-lane orchestration.

## Target Outcome

An operator can:

1. run an NPC pipeline against a D&D transcript;
2. receive a validated, normalized JSON NPC registry;
3. provide that registry as an explicit reference to a later spell pipeline;
   and
4. receive spell artifacts that prefer the registry's canonical NPC identities
   when resolving casters, targets, and other named participants.

The pipelines remain independent CLI invocations over the same input. Notarius
does not automatically schedule them, discover prior output, or reconcile
their results concurrently.

## NPC Artifact Contract

The new typed artifact kind represents a list of NPC records. Each record has:

- a stable, deterministic ID derived by Notarius from the normalized NPC
  identity rather than authored freely by the model;
- one canonical in-world name or, for an unnamed but individually
  distinguishable NPC, one stable descriptive label;
- zero or more observed aliases;
- a concise transcript-grounded description;
- zero or more explicitly supported relationships to PCs, NPCs, groups, or
  locations; and
- one or more source references that collectively support the record's
  identity and every reported description, alias, and relationship.

The external contract defines the exact JSON shape, ID syntax, ordering,
normalization, and compatibility policy. IDs are stable for equivalent
normalized identities under the same policy; they do not promise permanence
when human review or new evidence changes the canonical identity.

Descriptions report only facts established by the session transcript. General
D&D knowledge, campaign reference material, and model inference do not supply
biography, statistics, alignment, motivations, relationships, or outcomes.
Reference material may disambiguate an identity but is not source evidence.

## Inclusion Policy

Include an in-world non-PC participant when the transcript establishes that it
appears, acts, speaks, or is materially discussed and either:

- gives it a proper name;
- gives it a stable alias or title; or
- distinguishes it individually in a way useful to later extraction, such as
  a titled attendant or a distinct item-bearing guard.

Exclude:

- human players, transcript speakers, and the GM as out-of-world people;
- player characters identified by the party or player references;
- incidental name drops, hypothetical examples, corrected transcription
  mistakes, and characters mentioned only by reference material;
- interchangeable unnamed crowds or groups whose members cannot be
  distinguished; and
- temporary summoned creatures or spell effects unless the transcript gives
  one an individual, persistent identity relevant beyond the summoning event.

An NPC may be friendly, hostile, neutral, allied with the party, controlled by
a player temporarily, or absent from the current combat. Classification rests
on whether the entity is an in-world non-PC participant, not on disposition or
who rolls its dice.

## Extraction Semantics

The extractor uses the existing shared D&D transcript, player, party, and
glossary prompt inputs. It returns canonical in-world identities rather than
speaker names and cites only transcript units.

Each record is intentionally concise. It captures identity and stable facts
useful for recognizing the NPC elsewhere; it does not summarize every action,
follow the NPC through the entire session, or duplicate combat-turn and
narrative artifacts. Relationships must be stated or directly demonstrated by
the cited transcript rather than inferred from D&D lore.

The model may propose canonical names, aliases, descriptions, relationships,
and evidence. Deterministic code owns artifact IDs, structural integrity,
ordering, safe consolidation, and validation of all source ranges.

## Validation And Normalization

The production default validator chain remains deterministic. It must reject:

- malformed or empty required fields;
- missing, invalid, or foreign source references;
- duplicate or colliding IDs;
- an alias that resolves to more than one retained NPC;
- unsupported structural combinations defined by the artifact contract.

Relatedness checks may emit warnings when a proposed canonical name or alias is
absent from its cited source text. A name legitimately disambiguated through
opaque player, party, or glossary reference material may still produce that
warning; the warning does not turn the reference into event evidence. Exclusion
of PCs is prompt policy and a human-evaluation dimension until a future
structured roster contract makes deterministic membership checks reliable.

Normalization is deterministic and conservative. It:

- normalizes whitespace, case-insensitive comparison keys, apostrophe variants,
  and exact source-reference order;
- consolidates records whose normalized canonical names match or whose
  canonical-name and alias sets establish one unambiguous identity;
- unions unique aliases, relationships, and source references in stable order;
- retains one supported description without synthesizing new prose; and
- emits scoped warnings for every canonicalization or collapsed record.

Ambiguous identities remain separate or cause a bounded validation rejection;
the normalizer does not guess. Broader semantic reconciliation belongs to the
future generic LLM-assisted deduplication work after real NPC output shows that
deterministic identity overlap is insufficient.

## Sequential Reference Use

The normalized NPC registry is accepted as an optional structured JSON
reference by the D&D spell extractor. The spell prompt uses it to select exact
canonical NPC names and aliases, just as party and player references ground PC
identities. The registry does not establish that an NPC cast a spell and never
becomes spell source evidence.

Reference binding stays explicit in configuration or through the existing CLI
reference override. The maintained examples and operator documentation should
demonstrate separate NPC and spell invocations over the same transcript, with
the first output bound to the second invocation.

The reference contract should be designed for reuse by the future combat-turn
extractor without adding combat-specific fields to the NPC artifact.

## Provenance, Checkpoints, And Diagnostics

The NPC extractor and normalizer report their prompt, private schema, and
normalization-policy identities through the existing manifest and prepared
checkpoint-fingerprint contracts as applicable. Changes to any semantic input
that can change NPC identity or consolidation intentionally produce cold
checkpoint misses.

Raw transcript, reference, prompt, schema, and NPC content remain outside
manifest metadata and checkpoint fingerprints. Existing debug and rejected-
attempt behavior applies without a domain-specific filesystem surface.

## Evaluation

Evaluation is human-reviewed and model-aware rather than a deterministic golden
output gate. The initial corpus should include the existing D&D sessions that
exposed player-versus-character attribution and repeated NPC appearances.

Review should score separately:

- NPC detection precision and recall;
- exclusion of PCs and out-of-world people;
- canonical identity and alias quality;
- duplicate consolidation;
- description and relationship fidelity;
- source-reference completeness; and
- usefulness as grounding for a subsequent spell extraction run.

Small-model development runs may expose more semantic errors than production
frontier models. Deterministic contracts and validators should protect
structure and provenance without attempting to turn subjective extraction
quality into brittle fixture equality.

## Out Of Scope

- Combat turns, initiative, actions, damage, conditions, or encounter state.
- General narrative or scene summaries.
- PC extraction or replacement of the existing party and player references.
- Automatic multi-pipeline scheduling, prior-output discovery, or DAG
  execution.
- An LLM-backed NPC validator or generic semantic deduplication normalizer.
- Campaign-wide identity persistence, a database, cross-session entity merges,
  or manual identity-editing UI.
- Retrieval, embeddings, reference summarization, or token-budget management.

## Acceptance Criteria

- A configured D&D NPC lane produces a typed, durable JSON registry with the
  identity, evidence, and inclusion semantics above.
- Production shape, source-reference, identity-collision, and relatedness
  validators are registered with a deterministic default chain.
- The NPC normalizer assigns stable IDs and safely consolidates only
  unambiguous identities while preserving provenance and warnings.
- The registry can be bound directly to the spell extractor as an optional NPC
  reference, and spell extraction prefers its canonical NPC names without
  treating it as source evidence.
- Prompt, schema, reference, and normalization-policy changes participate in
  checkpoint identity where they can affect output.
- External, configuration, internal, operational, and maintained-example
  documentation is updated in its canonical location when the feature lands.
- Human review on representative transcripts demonstrates useful NPC grounding
  for a subsequent spell run without requiring exact model-output fixtures.