Add feature roadmap and implementation plan for a D&D NPC extraction module

docs/roadmap/dnd-npc-extraction.md

@@ -0,0 +1,209 @@
+# D&D NPC Extraction And Registry
+
+Status: Accepted.
+
+## Purpose
+
+Add a production D&D NPC pipeline that turns a session transcript into a
+canonical, evidence-backed NPC registry. The registry should be useful on its
+own and should serve as reference material for later sequential spell and
+combat-turn pipelines.
+
+This is the next recommended increment because participant identity is already
+a demonstrated source of extraction error, while combat-turn extraction will
+depend on consistent identities across many more events. Establishing the NPC
+artifact first gives later pipelines a durable vocabulary without introducing
+a DAG or cross-lane orchestration.
+
+## Target Outcome
+
+An operator can:
+
+1. run an NPC pipeline against a D&D transcript;
+2. receive a validated, normalized JSON NPC registry;
+3. provide that registry as an explicit reference to a later spell pipeline;
+   and
+4. receive spell artifacts that prefer the registry's canonical NPC identities
+   when resolving casters, targets, and other named participants.
+
+The pipelines remain independent CLI invocations over the same input. Notarius
+does not automatically schedule them, discover prior output, or reconcile
+their results concurrently.
+
+## NPC Artifact Contract
+
+The new typed artifact kind represents a list of NPC records. Each record has:
+
+- a stable, deterministic ID derived by Notarius from the normalized NPC
+  identity rather than authored freely by the model;
+- one canonical in-world name or, for an unnamed but individually
+  distinguishable NPC, one stable descriptive label;
+- zero or more observed aliases;
+- a concise transcript-grounded description;
+- zero or more explicitly supported relationships to PCs, NPCs, groups, or
+  locations; and
+- one or more source references that collectively support the record's
+  identity and every reported description, alias, and relationship.
+
+The external contract defines the exact JSON shape, ID syntax, ordering,
+normalization, and compatibility policy. IDs are stable for equivalent
+normalized identities under the same policy; they do not promise permanence
+when human review or new evidence changes the canonical identity.
+
+Descriptions report only facts established by the session transcript. General
+D&D knowledge, campaign reference material, and model inference do not supply
+biography, statistics, alignment, motivations, relationships, or outcomes.
+Reference material may disambiguate an identity but is not source evidence.
+
+## Inclusion Policy
+
+Include an in-world non-PC participant when the transcript establishes that it
+appears, acts, speaks, or is materially discussed and either:
+
+- gives it a proper name;
+- gives it a stable alias or title; or
+- distinguishes it individually in a way useful to later extraction, such as
+  a titled attendant or a distinct item-bearing guard.
+
+Exclude:
+
+- human players, transcript speakers, and the GM as out-of-world people;
+- player characters identified by the party or player references;
+- incidental name drops, hypothetical examples, corrected transcription
+  mistakes, and characters mentioned only by reference material;
+- interchangeable unnamed crowds or groups whose members cannot be
+  distinguished; and
+- temporary summoned creatures or spell effects unless the transcript gives
+  one an individual, persistent identity relevant beyond the summoning event.
+
+An NPC may be friendly, hostile, neutral, allied with the party, controlled by
+a player temporarily, or absent from the current combat. Classification rests
+on whether the entity is an in-world non-PC participant, not on disposition or
+who rolls its dice.
+
+## Extraction Semantics
+
+The extractor uses the existing shared D&D transcript, player, party, and
+glossary prompt inputs. It returns canonical in-world identities rather than
+speaker names and cites only transcript units.
+
+Each record is intentionally concise. It captures identity and stable facts
+useful for recognizing the NPC elsewhere; it does not summarize every action,
+follow the NPC through the entire session, or duplicate combat-turn and
+narrative artifacts. Relationships must be stated or directly demonstrated by
+the cited transcript rather than inferred from D&D lore.
+
+The model may propose canonical names, aliases, descriptions, relationships,
+and evidence. Deterministic code owns artifact IDs, structural integrity,
+ordering, safe consolidation, and validation of all source ranges.
+
+## Validation And Normalization
+
+The production default validator chain remains deterministic. It must reject:
+
+- malformed or empty required fields;
+- missing, invalid, or foreign source references;
+- duplicate or colliding IDs;
+- an alias that resolves to more than one retained NPC;
+- unsupported structural combinations defined by the artifact contract.
+
+Relatedness checks may emit warnings when a proposed canonical name or alias is
+absent from its cited source text. A name legitimately disambiguated through
+opaque player, party, or glossary reference material may still produce that
+warning; the warning does not turn the reference into event evidence. Exclusion
+of PCs is prompt policy and a human-evaluation dimension until a future
+structured roster contract makes deterministic membership checks reliable.
+
+Normalization is deterministic and conservative. It:
+
+- normalizes whitespace, case-insensitive comparison keys, apostrophe variants,
+  and exact source-reference order;
+- consolidates records whose normalized canonical names match or whose
+  canonical-name and alias sets establish one unambiguous identity;
+- unions unique aliases, relationships, and source references in stable order;
+- retains one supported description without synthesizing new prose; and
+- emits scoped warnings for every canonicalization or collapsed record.
+
+Ambiguous identities remain separate or cause a bounded validation rejection;
+the normalizer does not guess. Broader semantic reconciliation belongs to the
+future generic LLM-assisted deduplication work after real NPC output shows that
+deterministic identity overlap is insufficient.
+
+## Sequential Reference Use
+
+The normalized NPC registry is accepted as an optional structured JSON
+reference by the D&D spell extractor. The spell prompt uses it to select exact
+canonical NPC names and aliases, just as party and player references ground PC
+identities. The registry does not establish that an NPC cast a spell and never
+becomes spell source evidence.
+
+Reference binding stays explicit in configuration or through the existing CLI
+reference override. The maintained examples and operator documentation should
+demonstrate separate NPC and spell invocations over the same transcript, with
+the first output bound to the second invocation.
+
+The reference contract should be designed for reuse by the future combat-turn
+extractor without adding combat-specific fields to the NPC artifact.
+
+## Provenance, Checkpoints, And Diagnostics
+
+The NPC extractor and normalizer report their prompt, private schema, and
+normalization-policy identities through the existing manifest and prepared
+checkpoint-fingerprint contracts as applicable. Changes to any semantic input
+that can change NPC identity or consolidation intentionally produce cold
+checkpoint misses.
+
+Raw transcript, reference, prompt, schema, and NPC content remain outside
+manifest metadata and checkpoint fingerprints. Existing debug and rejected-
+attempt behavior applies without a domain-specific filesystem surface.
+
+## Evaluation
+
+Evaluation is human-reviewed and model-aware rather than a deterministic golden
+output gate. The initial corpus should include the existing D&D sessions that
+exposed player-versus-character attribution and repeated NPC appearances.
+
+Review should score separately:
+
+- NPC detection precision and recall;
+- exclusion of PCs and out-of-world people;
+- canonical identity and alias quality;
+- duplicate consolidation;
+- description and relationship fidelity;
+- source-reference completeness; and
+- usefulness as grounding for a subsequent spell extraction run.
+
+Small-model development runs may expose more semantic errors than production
+frontier models. Deterministic contracts and validators should protect
+structure and provenance without attempting to turn subjective extraction
+quality into brittle fixture equality.
+
+## Out Of Scope
+
+- Combat turns, initiative, actions, damage, conditions, or encounter state.
+- General narrative or scene summaries.
+- PC extraction or replacement of the existing party and player references.
+- Automatic multi-pipeline scheduling, prior-output discovery, or DAG
+  execution.
+- An LLM-backed NPC validator or generic semantic deduplication normalizer.
+- Campaign-wide identity persistence, a database, cross-session entity merges,
+  or manual identity-editing UI.
+- Retrieval, embeddings, reference summarization, or token-budget management.
+
+## Acceptance Criteria
+
+- A configured D&D NPC lane produces a typed, durable JSON registry with the
+  identity, evidence, and inclusion semantics above.
+- Production shape, source-reference, identity-collision, and relatedness
+  validators are registered with a deterministic default chain.
+- The NPC normalizer assigns stable IDs and safely consolidates only
+  unambiguous identities while preserving provenance and warnings.
+- The registry can be bound directly to the spell extractor as an optional NPC
+  reference, and spell extraction prefers its canonical NPC names without
+  treating it as source evidence.
+- Prompt, schema, reference, and normalization-policy changes participate in
+  checkpoint identity where they can affect output.
+- External, configuration, internal, operational, and maintained-example
+  documentation is updated in its canonical location when the feature lands.
+- Human review on representative transcripts demonstrates useful NPC grounding
+  for a subsequent spell run without requiring exact model-output fixtures.