Plan simpler D&D extraction contracts

docs/roadmap/minimal-dnd-extraction-contracts.md

@@ -0,0 +1,220 @@
+# Minimal D&D Extraction Contracts
+
+**Status:** Proposed
+
+## Intent
+
+Redesign the D&D spell, NPC, and combat-turn artifacts around the principle in
+[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md):
+each extractor should answer one narrow question with the smallest useful set
+of source-grounded fields.
+
+The redesign favors extraction precision, evidence quality, valid-output rate,
+smaller-model reliability, and lower prompt and response cost over descriptive
+richness. It removes synthesis responsibilities rather than preserving obsolete
+fields as optional, nullable, empty, or application-generated placeholders.
+
+## Goals
+
+- Make every model-produced field necessary to the artifact's core question.
+- Require direct transcript evidence for every extracted record.
+- Remove overlapping prose, inferred enrichment, and nested structures without
+  a demonstrated consumer.
+- Keep catalog and identity references as disambiguation aids rather than
+  evidence.
+- Preserve deterministic canonicalization, evidence ordering, exact
+  deduplication, identity assignment, and bounded domain diagnostics where
+  those responsibilities still apply.
+- Reduce downstream prompt material to the fields a consumer actually needs.
+- Keep the unpublished v1 identities while replacing their pre-release shapes
+  in place.
+
+## Non-Goals
+
+- Generating session narrative, rules analysis, biographies, relationship
+  graphs, encounter summaries, or prose descriptions.
+- Preserving removed fields for source compatibility through empty strings,
+  nullable values, or synthetic defaults.
+- Adding fuzzy entity resolution, LLM-assisted enrichment, or a general schema
+  migration framework.
+- Treating campaign references, catalogs, or earlier artifacts as evidence that
+  an event occurred in the current transcript.
+- Combining the three D&D artifact families into one model call.
+
+## Shared Contract Policy
+
+All three artifacts remain ordered lists. Each record contains at least one
+source reference, and the complete reference collection supports every
+model-produced field in that record. Source IDs continue to be assigned by the
+application from the current input; the model returns only source-unit ranges.
+
+Private LLM schemas remain strict about their transport envelope: required
+fields, JSON types, object and array shape, nullability where applicable, and
+unknown-field rejection. Deterministic validators continue to own semantic
+rules such as nonblank identities, catalog membership, enum membership,
+positive and resolvable source units, and canonical normalized invariants.
+
+Prompts retain the shared D&D evidence, identity, reference, NPC-grounding, and
+transcript assets that remain relevant. Module-specific task and instruction
+assets must delete duties associated with removed fields. The existing
+cache-friendly ordering keeps stable shared and reference material before the
+chunk-variable transcript.
+
+## Spell Cast
+
+The spell extractor answers:
+
+> Which spell was cast, by which in-world caster, and where is that event
+> established in the source?
+
+The artifact kind remains `dnd/spell-list`. Its durable schema remains v1.
+Each spell-cast record contains exactly:
+
+- `caster`: required nonblank in-world display identity;
+- `spell`: required nonblank canonical or catalog-resolvable spell name; and
+- `source_refs`: one or more current-source evidence ranges.
+
+The model-facing response contains the same fields except for application-owned
+`source_id` values within references. The current `effect` and
+`narrative_description` fields are removed from the private response, public Go
+type, durable schema, codec, validators, normalizer, fixtures, and integration
+contract.
+
+The inclusion boundary remains an actual casting event or an unambiguously
+declared casting attempt, not a spell mention, hypothetical plan, rules
+discussion, or catalog match. The spell catalog helps recognize and canonicalize
+the name but never establishes that a cast occurred.
+
+Normalization continues to canonicalize spell names, canonicalize evidence,
+and collapse exact duplicate events using caster, canonical spell name, and
+complete valid evidence. It performs no prose selection or merging.
+
+## NPC Registry
+
+The NPC extractor answers:
+
+> Which individually identifiable non-player characters are established in the
+> source, and where is each identity established?
+
+The artifact kind remains `dnd/npc-list`. Its durable schema remains v1.
+Each durable NPC record contains exactly:
+
+- `id`: deterministic application-assigned identity derived under the NPC
+  identity policy;
+- `name`: required nonblank source-supported display identity; and
+- `source_refs`: one or more evidence ranges supporting that identity.
+
+The private model response omits `id` and reference `source_id` values. A
+`name` may be a proper name or a stable, individually distinguishing title or
+alias supported by the transcript. The extractor does not invent descriptive
+labels for anonymous creatures, crowds, or generic roles.
+
+The current `aliases`, `description`, and `relationships` fields are removed
+from the private response, public Go type, durable schema, codec, validators,
+normalizer, registry, fixtures, and integration contract. Normalization
+consolidates only identities supported by the retained name policy and unions
+exact evidence; it does not infer alias equivalence or relationships.
+
+Spell and combat consumers receive a prompt projection containing only the
+canonical NPC names needed for identity grounding. Application-owned NPC IDs
+remain available to deterministic registry and normalization code but are not
+sent to a model that cannot return or otherwise consume them. NPC source
+references remain provenance in the durable registry and are not included as
+current-transcript evidence or copied into downstream event artifacts.
+
+Encounter context is deliberately not a scalar NPC registry field. Dialogue,
+combat alignment, presence, or third-party mention can vary across occurrences.
+If a demonstrated consumer needs that information, add a separate ordered
+NPC-occurrence artifact whose records contain `name`, a small mutually
+exclusive context enum, and `source_refs`. A candidate starting vocabulary is
+`dialogue`, `combat_ally`, `combat_opponent`, `noncombat_presence`, `mentioned`,
+and `other`; its exact semantics require a separate feature decision.
+
+## Combat Event
+
+The combat extractor answers:
+
+> Which in-world participant took a turn or discrete interrupting combat
+> event, what kind of event was it, and where is it established in the source?
+
+The existing `dnd/combat-turn-list` artifact kind and v1 durable-schema identity
+remain. Each record contains exactly:
+
+- `actor`: required nonblank in-world display identity;
+- `turn_kind`: one of `turn`, `reaction`, `legendary_action`, `lair_action`, or
+  `other`; and
+- `source_refs`: one or more current-source evidence ranges.
+
+The current `round`, `actions`, and `summary` fields, including nested action
+categories, declarations, targets, and resolutions, are removed from the
+private response, public Go types, durable schema, codec, validators,
+normalizer, fixtures, and integration contract.
+
+Normalization continues to display-normalize and registry-canonicalize actors,
+canonicalize evidence, order events by source position, and collapse exact
+duplicates using actor, turn kind, and complete valid evidence. It no longer
+normalizes targets, declarations, summaries, or resolutions.
+
+If action-level facts later have a demonstrated consumer, they belong in a
+separate combat-action artifact rather than restoring a nested synthesis
+contract to combat-turn detection. Spell casts and future item events remain
+owned by their dedicated artifact lanes.
+
+## Pre-Release Schema Policy
+
+Notarius and these contracts are pre-release. The existing v1 artifacts and
+private model-response schemas have not been published as compatibility
+contracts, so their shapes change in place. The implementation does not retain
+the rich pre-redesign shape, add v2 assets, migrate old output, support multiple
+versions, or preserve old fixtures solely for compatibility testing.
+
+Artifact kinds, schema keys and IDs, schema names and versions, prompt IDs and
+versions, module keys, capabilities, and media types all remain unchanged.
+Changing prompt and schema content invalidates the relevant content-addressed
+development state; any remaining pre-redesign local output or cache is
+disposable and may be regenerated.
+
+| Lane | Durable v1 schema | Private model-response v1 schema | Prompt ID |
+| --- | --- | --- | --- |
+| Spell cast | ID `notarius.dnd.spells`, name `notarius_dnd_spells_v1` | key `dnd_spells`, ID `notarius.dnd.spells`, name `notarius_dnd_spells_v1` | `dnd.spells` |
+| NPC registry | ID `notarius.dnd.npcs`, name `notarius_dnd_npcs_v1` | key `dnd_npcs_llm`, ID `notarius.dnd.npcs.llm`, name `notarius_dnd_npcs_llm_v1` | `dnd.npcs` |
+| Combat event | ID `notarius.dnd.combat_turns`, name `notarius_dnd_combat_turns_v1` | key `dnd_combat_turns_llm`, ID `notarius.dnd.combat_turns.llm`, name `notarius_dnd_combat_turns_llm_v1` | `dnd.combat_turns` |
+
+## Quality And Evaluation
+
+The implemented contract should protect the remaining meaningful risks:
+
+- private schemas accept only the new structural envelopes;
+- durable codecs strictly round-trip the redesigned contracts;
+- domain validators own blank identities, enum and catalog membership, and
+  invalid evidence;
+- normalizers preserve source-grounded values while applying only their stated
+  deterministic transformations;
+- generated NPC references expose the minimal identity projection and never
+  become event evidence; and
+- representative assembled pipelines attribute retries and rejections to the
+  owning domain boundary.
+
+Post-cutover model evaluation should compare the rich-schema baseline and the
+minimal-schema result on a small human-reviewed transcript set using:
+
+- event/entity precision and recall;
+- caster, NPC, and combat-actor attribution accuracy;
+- source-range validity and evidence sufficiency;
+- catalog and enum accuracy;
+- structurally valid completion rate and exhausted-retry rate;
+- unsupported-claim rate; and
+- input/output tokens, latency, and model cost.
+
+The evaluation exists to identify prompt or model-quality follow-up work, not
+to gate the approved minimal contract on live-provider behavior. Human review
+and live model calls are evaluation aids rather than deterministic CI gates.
+
+## Documentation Ownership
+
+The spell, NPC, and combat integration documents are the canonical owners of
+their redesigned durable schemas once implemented. Internal LLM and module
+documents own the corresponding current prompt, DTO, validator, normalizer, and
+NPC prompt-projection behavior. This roadmap owns the desired end state and
+policy; [the implementation plan](implementation.md) owns sequencing and
+completion criteria until the cutover is complete.