Document D&D spell normalization behavior

docs/config.md

@@ -285,6 +285,7 @@ production validators do not call the LLM and must not set `llm_profile`.
 | extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
 | merge | `appendorder` | Combines typed artifacts in chunk order. |
 | normalize | `noop` | Passes merged typed artifacts through unchanged. |
+| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
 | output | `json` | Produces JSON output files for normalized `application/json` lanes. |
 
 ## Implemented Production Validators
@@ -300,7 +301,8 @@ production validators do not call the LLM and must not set `llm_profile`.
 | `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
 | `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
 
-The production default chain for the `dnd/spells` extractor is:
+The production default chain for `dnd/spells` is used for both its extract and
+normalize stages:
 
 ```yaml
 validators:
@@ -342,6 +344,13 @@ not allow multiple files. Its format is defined in the
 The extractor uses campaign references only as supporting disambiguation
 material; spell casts still must be present in the source transcript.
 
+The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
+When an overlay is used, bind it independently under
+`artifacts.<lane>.normalize.references.spell_catalog`; normalize-stage
+references are local to that stage and are not inherited from extraction. The
+normalizer uses the embedded SRD catalog when no normalize-stage overlay is
+bound.
+
 ## State Surfaces
 
 The `output`, `cache`, and `debug` top-level fields select independent physical

docs/integrations/dnd-spell-artifacts.md

@@ -68,6 +68,39 @@ Reference slot keys and accepted file types are defined in
 supporting disambiguation material, not source evidence, and are not
 addressable through `source_refs`.
 
+## Normalization Behavior
+
+When the `dnd/spells` normalizer is selected, each recognized spell name is
+rewritten to the effective catalog's canonical display name. Lookup uses the
+catalog's case-insensitive, whitespace-normalizing, apostrophe-normalizing, and
+alias rules. Unknown names are preserved exactly for the normalize validators;
+the normalizer does not guess or apply fuzzy matching.
+
+Each cast's `source_refs` is copied, sorted by exact `source_id`,
+`start_unit_id`, and `end_unit_id`, and stripped of exact structural
+duplicates. Adjacent or overlapping ranges are not merged, and the normalizer
+does not synthesize references or change their boundaries.
+
+After those per-cast changes, duplicate identity requires the same canonical
+spell name, the same caster after case folding and whitespace normalization,
+and the same complete, non-empty set of source references valid for the source
+document. Only the first occurrence is retained, in stable order. Its caster,
+effect, narrative description, and canonical references are preserved without
+prose merging or source union. Unknown names, empty or invalid evidence, and
+casts with different evidence remain separate.
+
+Mutation and duplicate decisions are returned through the normal warnings
+surface. Warning scopes use the merged input index, such as `spell_casts[0]`,
+so they remain meaningful even when a later duplicate is removed. The
+normalizer uses these reason codes:
+
+| Reason code | Meaning |
+| --- | --- |
+| `spell_name_canonicalized` | A catalog lookup replaced an input name with its canonical display name. |
+| `spell_name_unresolved` | A name was not found in the effective catalog and was retained unchanged. |
+| `source_references_normalized` | Reference order changed or exact duplicate references were removed. |
+| `duplicate_spell_cast_collapsed` | A later cast matched the retained cast's complete duplicate identity. |
+
 ## Manifest Metadata
 
 The extractor adds prompt and response-schema provenance under the artifact lane
@@ -88,6 +121,11 @@ manifest metadata:
       "catalog_base_id": "dnd-5e-2014-srd-spells",
       "catalog_digest": "sha256:...",
       "catalog_overlay_ids": ["campaign.example"]
+    },
+    "normalizer": {
+      "catalog_base_id": "dnd-5e-2014-srd-spells",
+      "catalog_digest": "sha256:...",
+      "catalog_overlay_ids": ["campaign.example"]
     }
   }
 }
@@ -96,6 +134,7 @@ manifest metadata:
 `catalog_digest` identifies the effective semantic catalog, while
 `catalog_overlay_ids` is sorted and empty for a base-only configuration. Raw
 prompt, schema, catalog, alias, and local overlay-file content are not
-included in manifest metadata. Overlay origin, media type, byte size, and raw
-digest are recorded separately in the manifest's reference provenance; see
-the [JSON output contract](json-output.md#manifestjson).
+included in manifest metadata. The `normalizer` metadata uses the same catalog
+identity fields when that module is selected. Overlay origin, media type, byte
+size, and raw digest are recorded separately in the manifest's reference
+provenance; see the [JSON output contract](json-output.md#manifestjson).

docs/internal/modules.md

@@ -186,6 +186,30 @@ function.
 The normalizer returns the merged domain value unchanged and is reusable for
 any registered artifact type.
 
+### `internal/modules/dnd/normalize/spells`
+
+The typed spell normalizer resolves the optional `spell_catalog` reference into
+the same immutable SRD-plus-overlay effective catalog used by spell extraction
+and catalog validation. It performs no LLM calls. For each spell cast it
+canonicalizes recognized names using the catalog's case, whitespace,
+apostrophe, and alias rules; sorts source references by source identity and
+unit boundaries; removes only exact reference duplicates; and emits bounded,
+scoped warnings for each mutation or unresolved name.
+
+After those per-cast changes, it collapses only casts with the same canonical
+spell, case-folded and whitespace-normalized caster, and complete non-empty
+valid source-reference set. It retains the first occurrence and its caster,
+effect, narrative description, and stable order. Unknown names, empty or
+invalid evidence, and adjacent or overlapping but different ranges remain
+unchanged for validation.
+
+The normalizer exposes the effective catalog digest as its independently scoped
+`effective_catalog` checkpoint fingerprint and reports catalog base ID, digest,
+and overlay IDs as manifest metadata. Catalog contents, reference paths, and
+raw overlay bytes are not included in either surface. The normalize-stage
+reference is stage-local, so an overlay-capable pipeline binds the catalog
+independently for extraction and normalization.
+
 ## Output Encoder
 
 ### `internal/modules/generic/output/json`
@@ -240,7 +264,7 @@ complete framework registry set and one LLM asset registry. It invokes
 `internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
 that order, then exposes the matching catalog for resolution. The generic and
 Seriatim registrars own their production leaf registrations. The D&D registrar
-owns D&D leaf registrations, the spell default-validator chain, and D&D
+owns D&D leaf registrations, the typed spell default-validator chains, and D&D
 prompt/schema asset collection.
 
 Concrete implementation packages do not import generic implementation

docs/internal/overview.md

@@ -65,8 +65,8 @@ run-local construction closures. Preparation injects shared dependencies and
 constructs input, chunk, validators, ordered lanes, and output before source
 parsing. Production modules use strict construction-time option decoding, and
 LLM-backed modules retain the injected shared client. The D&D family registers
-the canonical `dnd/spell-list` codec, typed spell extractor and validators, and
-kind-specific generic merge and normalize strategies; generic JSON validators
+the canonical `dnd/spell-list` codec, typed spell extractor, normalizer, and
+validators, plus kind-specific generic merge strategies; generic JSON validators
 use the serialized-validation contract. The runner executes lanes through
 private exact-type-checked closures, coordinates extract results independently
 of completion timing, and serializes artifacts only through their codec at
@@ -90,6 +90,7 @@ Configuration. The implemented module packages are:
 | `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
 | `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
 | `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
+| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
 | `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
 
 `internal/modules/dnd/shared` owns reusable D&D prompt fragments,

docs/roadmap/dnd-spell-normalization.md

@@ -2,14 +2,8 @@
 
 ## Status
 
-<<<<<<< HEAD
-Proposed as the next D&D pipeline milestone. This feature completes the first
-domain-specific normalize stage for the spell pipeline before work begins on
-NPC and combat-turn artifacts.
-=======
-Accepted target state. Implementation details are maintained separately in
-`docs/roadmap/implementation.md`.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
+Implemented deterministic feature. Implementation details are maintained
+separately in `docs/roadmap/implementation.md`.
 
 ## Objective
 
@@ -18,13 +12,8 @@ that emits canonical catalog names, removes only clearly identical duplicate
 casts, preserves source provenance, and makes every mutation visible through
 scoped warnings.
 
-<<<<<<< HEAD
-The milestone should improve the consistency of durable spell output without
-adding another LLM call or introducing fuzzy repair policy.
-=======
 The normalizer improves the consistency of durable spell output without adding
 another LLM call or introducing fuzzy repair policy.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ## Target Behavior
 
@@ -43,24 +32,16 @@ depend on extractor-private state.
 The normalizer contributes its effective catalog digest through
 `pipeline.CheckpointFingerprintProvider`. Changing the embedded catalog,
 composition policy, or normalize-stage overlay therefore invalidates reusable
-<<<<<<< HEAD
-normalize checkpoints.
-=======
 normalize checkpoints. It also records catalog base ID, digest, and overlay IDs
 as manifest metadata. Neither identity surface includes catalog contents or
 reference paths.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ### Canonical spell names
 
 For every spell cast, look up the extracted name using the effective catalog's
 existing case, whitespace, apostrophe, and alias rules. Replace a recognized
 value with its canonical display name. This is the only spell-name repair in
-<<<<<<< HEAD
-the initial feature.
-=======
 this feature.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 Do not use edit distance, phonetic matching, model judgment, or another fuzzy
 heuristic. If a value does not resolve, retain it unchanged and emit a scoped
@@ -69,12 +50,8 @@ acceptance or rejection.
 
 Emit a warning for each changed spell name. Diagnostics should identify the
 artifact index and the original and canonical values without modifying other
-<<<<<<< HEAD
-fields.
-=======
 fields. Diagnostics are deterministic and bounded: user-controlled spell names
 are truncated to 128 Unicode code points before display.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ### Source-reference normalization
 
@@ -84,12 +61,8 @@ overlapping ranges, because doing so could broaden the evidence attributed to
 an event.
 
 The normalizer must not synthesize source references, alter source-unit
-<<<<<<< HEAD
-boundaries, or use auxiliary references as evidence.
-=======
 boundaries, or use auxiliary references as evidence. Reordering references or
 removing exact duplicates emits a scoped warning for the affected input cast.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ### Conservative duplicate collapse
 
@@ -98,12 +71,8 @@ event only when all of the following match:
 
 - canonical spell name;
 - caster after case folding and whitespace normalization; and
-<<<<<<< HEAD
-- the complete canonical source-reference set.
-=======
 - the complete canonical source-reference set, which must be non-empty and
   valid against the source document.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 Collapse each such group into its first occurrence, preserving stable pipeline
 order. Retain the first cast's caster, effect, and narrative description. Do
@@ -116,14 +85,11 @@ the removed indices. Casts with different evidence remain distinct even when
 their spell and caster match. In particular, adjacency at a chunk or scene
 boundary is not sufficient evidence of duplication.
 
-<<<<<<< HEAD
-=======
 Unknown spell names, empty evidence sets, and invalid source references cannot
 establish duplicate identity. They remain in the artifact for the configured
 normalize validators to accept or reject. A duplicate warning displays at most
 20 removed input indices and reports the exact omitted count.
 
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 ### Production composition and validation
 
 Register the typed spell normalizer in the D&D family and make it the default
@@ -141,40 +107,34 @@ without a catalog reference.
 
 ## Acceptance Criteria
 
-- Base and overlay aliases are written using canonical catalog display names.
-- Unknown names are never guessed, deleted, or silently changed.
-- Source references are deterministically sorted and exactly deduplicated.
-- Only casts with matching canonical spell, normalized caster, and identical
-<<<<<<< HEAD
-  evidence sets collapse; near or adjacent events remain separate.
-- Normalization preserves the first cast's non-name fields and stable order.
-- Every name change and duplicate collapse emits a scoped warning.
-=======
+- [x] Base and overlay aliases are written using canonical catalog display names.
+- [x] Unknown names are never guessed, deleted, or silently changed.
+- [x] Source references are deterministically sorted and exactly deduplicated.
+- [x] Only casts with matching canonical spell, normalized caster, and identical
   non-empty valid evidence sets collapse; near, invalid, unknown, or adjacent
   events remain separate.
-- Normalization preserves the first cast's non-name fields and stable order.
-- Every name change, unresolved name, source-reference mutation, and duplicate
+- [x] Normalization preserves the first cast's non-name fields and stable order.
+- [x] Every name change, unresolved name, source-reference mutation, and duplicate
   collapse emits a scoped warning.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
-- Extract, normalize, and catalog-validation catalog identities agree for the
+- [x] Extract, normalize, and catalog-validation catalog identities agree for the
   same bound references, and catalog changes invalidate normalize checkpoints.
-- The maintained production pipeline and current-behavior documentation use
+- [x] The maintained production pipeline and current-behavior documentation use
   the D&D normalizer without changing the durable artifact schema.
 
 ## Evaluation
 
-Maintain a small human-reviewed set of representative spell outputs covering
-canonical names, aliases, repeated casts, duplicate model output, and scene
-boundaries. Use it to review normalization behavior and warnings, not as a
-claim that LLM extraction is deterministically correct.
+Status: pending qualitative review. No approved human-reviewed transcript
+corpus is available locally under repository policy, so no sensitive transcript
+content is committed and no claims about real transcript quality are made.
+
+A compact deterministic fixture set covers canonical names, exact source
+reference normalization, repeated casts, duplicate output, and distinct
+evidence. It exercises the normalizer contract without requiring paid or
+network LLM calls.
 
 Record cases where likely duplicates remain because their evidence differs.
 Those examples should inform later LLM-assisted deduplication work rather than
-<<<<<<< HEAD
-causing this deterministic milestone to adopt broader heuristics.
-=======
 causing this deterministic feature to adopt broader heuristics.
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ## Deferred Work
 
@@ -186,10 +146,6 @@ causing this deterministic feature to adopt broader heuristics.
 - Repair-aware extraction retries or LLM-backed semantic validation.
 - Changes to the spell artifact schema, including stable cast IDs.
 
-<<<<<<< HEAD
-An LLM-backed normalizer is explicitly outside this milestone. If later
-=======
 An LLM-backed normalizer is explicitly outside this feature. If later
->>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 evaluation justifies one, shared spell-catalog prompt material should be
 designed at that time around the actual normalization or repair request.

docs/roadmap/future.md

@@ -7,11 +7,8 @@ not as committed release dates.
 
 ## Near-Term D&D Pipeline
 
-### Solidify Spell Extraction
+### Evaluate Spell Extraction And Normalization
 
-- Implement the deterministic catalog-aware normalizer defined in
-  [D&D Spell Normalization](dnd-spell-normalization.md), including conservative
-  exact-evidence duplicate collapse.
 - Evaluate ordinary extraction retries and the completed normalization path
   against a human-reviewed transcript set before adding repair-aware retries or
   an LLM-backed semantic validator.

examples/dnd-spells-production.config.yml

@@ -29,9 +29,11 @@ pipelines:
         extract:
           module: dnd/spells
           retries: 2
+          # Overlay behavior binds the same catalog independently at each stage.
           references:
             spell_catalog: ./dnd-spells-catalog.json
         normalize:
           module: dnd/spells
+          # Normalize-stage references are local and must be bound explicitly.
           references:
             spell_catalog: ./dnd-spells-catalog.json

examples/dnd-spells.config.yml

@@ -15,4 +15,5 @@ pipelines:
     artifacts:
       spells:
         extract: dnd/spells
+        # Base-only behavior: normalization uses the embedded SRD catalog.
         normalize: dnd/spells

internal/modules/dnd/normalize/spells/fixture_test.go

@@ -0,0 +1,56 @@
+package spells
+
+import (
+	"context"
+	"encoding/json"
+	"os"
+	"reflect"
+	"testing"
+
+	"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
+)
+
+type normalizerFixtureSet struct {
+	Cases []normalizerFixtureCase `json:"cases"`
+}
+
+type normalizerFixtureCase struct {
+	Name               string        `json:"name"`
+	Input              dnd.SpellList `json:"input"`
+	Output             dnd.SpellList `json:"output"`
+	WarningReasonCodes []string      `json:"warning_reason_codes"`
+}
+
+func TestNormalizeAcceptedFixtures(t *testing.T) {
+	data, err := os.ReadFile("testdata/normalizer_accepted_cases.json")
+	if err != nil {
+		t.Fatalf("read fixture: %v", err)
+	}
+	var fixtures normalizerFixtureSet
+	if err := json.Unmarshal(data, &fixtures); err != nil {
+		t.Fatalf("decode fixture: %v", err)
+	}
+	if len(fixtures.Cases) == 0 {
+		t.Fatal("fixture set is empty")
+	}
+
+	for _, fixture := range fixtures.Cases {
+		t.Run(fixture.Name, func(t *testing.T) {
+			result, err := newNormalizer(t).Normalize(context.Background(), normalizeRequestWithSource(fixture.Input, sourceDocument(2)))
+			if err != nil {
+				t.Fatalf("Normalize() error = %v, want nil", err)
+			}
+			if !reflect.DeepEqual(result.Value, fixture.Output) {
+				t.Fatalf("normalized value = %#v, want %#v", result.Value, fixture.Output)
+			}
+
+			gotReasonCodes := make([]string, 0, len(result.Warnings))
+			for _, warning := range result.Warnings {
+				gotReasonCodes = append(gotReasonCodes, warning.ReasonCode)
+			}
+			if !reflect.DeepEqual(gotReasonCodes, fixture.WarningReasonCodes) {
+				t.Fatalf("warning reason codes = %#v, want %#v", gotReasonCodes, fixture.WarningReasonCodes)
+			}
+		})
+	}
+}

internal/modules/dnd/normalize/spells/testdata/normalizer_accepted_cases.json

@@ -0,0 +1,99 @@
+{
+  "cases": [
+    {
+      "name": "canonicalizes references and collapses exact duplicate",
+      "input": {
+        "spell_casts": [
+          {
+            "caster": "  Aria ",
+            "spell": " cure   wounds ",
+            "effect": "first effect",
+            "narrative_description": "first narrative",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 2, "end_unit_id": 2},
+              {"source_id": "source", "start_unit_id": 1, "end_unit_id": 1},
+              {"source_id": "source", "start_unit_id": 1, "end_unit_id": 1}
+            ]
+          },
+          {
+            "caster": "aria",
+            "spell": "Cure Wounds",
+            "effect": "duplicate effect",
+            "narrative_description": "duplicate narrative",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 1, "end_unit_id": 1},
+              {"source_id": "source", "start_unit_id": 2, "end_unit_id": 2}
+            ]
+          },
+          {
+            "caster": "aria",
+            "spell": "Cure Wounds",
+            "effect": "distinct effect",
+            "narrative_description": "distinct narrative",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 2, "end_unit_id": 2}
+            ]
+          }
+        ]
+      },
+      "output": {
+        "spell_casts": [
+          {
+            "caster": "  Aria ",
+            "spell": "Cure Wounds",
+            "effect": "first effect",
+            "narrative_description": "first narrative",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 1, "end_unit_id": 1},
+              {"source_id": "source", "start_unit_id": 2, "end_unit_id": 2}
+            ]
+          },
+          {
+            "caster": "aria",
+            "spell": "Cure Wounds",
+            "effect": "distinct effect",
+            "narrative_description": "distinct narrative",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 2, "end_unit_id": 2}
+            ]
+          }
+        ]
+      },
+      "warning_reason_codes": [
+        "spell_name_canonicalized",
+        "source_references_normalized",
+        "duplicate_spell_cast_collapsed"
+      ]
+    },
+    {
+      "name": "leaves already normalized output unchanged",
+      "input": {
+        "spell_casts": [
+          {
+            "caster": "Aria",
+            "spell": "Cure Wounds",
+            "effect": "heals an ally",
+            "narrative_description": "Aria restores an ally's wounds.",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 1, "end_unit_id": 1}
+            ]
+          }
+        ]
+      },
+      "output": {
+        "spell_casts": [
+          {
+            "caster": "Aria",
+            "spell": "Cure Wounds",
+            "effect": "heals an ally",
+            "narrative_description": "Aria restores an ally's wounds.",
+            "source_refs": [
+              {"source_id": "source", "start_unit_id": 1, "end_unit_id": 1}
+            ]
+          }
+        ]
+      },
+      "warning_reason_codes": []
+    }
+  ]
+}