D&D Spell Normalization

Status

Proposed as the next D&D pipeline milestone. This feature completes the first domain-specific normalize stage for the spell pipeline before work begins on NPC and combat-turn artifacts.

Objective

Replace the spell lane's no-op normalizer with a deterministic D&D normalizer that emits canonical catalog names, removes only clearly identical duplicate casts, preserves source provenance, and makes every mutation visible through scoped warnings.

The milestone should improve the consistency of durable spell output without adding another LLM call or introducing fuzzy repair policy.

Target Behavior

Effective catalog construction

The normalizer constructs the same immutable SRD-plus-overlay effective catalog used by spell extraction and catalog validation. It declares the optional spell_catalog normalize-stage reference slot and resolves it during pipeline preparation, before source parsing or module execution.

An overlay-capable pipeline binds the same catalog file independently at the extract and normalize stages. This repetition is intentional: references are stage-local, and the normalizer must not reach into a constructed extractor or depend on extractor-private state.

The normalizer contributes its effective catalog digest through pipeline.CheckpointFingerprintProvider. Changing the embedded catalog, composition policy, or normalize-stage overlay therefore invalidates reusable normalize checkpoints.

Canonical spell names

For every spell cast, look up the extracted name using the effective catalog's existing case, whitespace, apostrophe, and alias rules. Replace a recognized value with its canonical display name. This is the only spell-name repair in the initial feature.

Do not use edit distance, phonetic matching, model judgment, or another fuzzy heuristic. If a value does not resolve, retain it unchanged and emit a scoped warning; the configured normalize validator chain remains responsible for acceptance or rejection.

Emit a warning for each changed spell name. Diagnostics should identify the artifact index and the original and canonical values without modifying other fields.

Source-reference normalization

Sort each cast's source references by source identity, start unit, and end unit, then remove exact duplicate references. Do not merge adjacent or merely overlapping ranges, because doing so could broaden the evidence attributed to an event.

The normalizer must not synthesize source references, alter source-unit boundaries, or use auxiliary references as evidence.

Conservative duplicate collapse

After name and source-reference canonicalization, treat two casts as the same event only when all of the following match:

canonical spell name;
caster after case folding and whitespace normalization; and
the complete canonical source-reference set.

Collapse each such group into its first occurrence, preserving stable pipeline order. Retain the first cast's caster, effect, and narrative description. Do not combine prose fields or select a winner based on length, confidence, or model-like semantic judgment. The retained cast receives the group's already canonical source-reference set.

Emit one scoped warning per collapsed group, including the retained index and the removed indices. Casts with different evidence remain distinct even when their spell and caster match. In particular, adjacency at a chunk or scene boundary is not sufficient evidence of duplication.

Production composition and validation

Register the typed spell normalizer in the D&D family and make it the default normalizer for the maintained production spell pipeline. Keep the artifact kind and durable spell-list schema unchanged.

Add a normalize-stage production validator chain using the existing generic JSON and JSON Schema validators followed by the existing spell shape, catalog, source-reference, and source-relatedness validators in their current order. Explicit validator overrides remain authoritative.

The maintained overlay-capable example should bind spell_catalog at both the extract and normalize stages. The base-only example should continue to work without a catalog reference.

Acceptance Criteria

Base and overlay aliases are written using canonical catalog display names.
Unknown names are never guessed, deleted, or silently changed.
Source references are deterministically sorted and exactly deduplicated.
Only casts with matching canonical spell, normalized caster, and identical evidence sets collapse; near or adjacent events remain separate.
Normalization preserves the first cast's non-name fields and stable order.
Every name change and duplicate collapse emits a scoped warning.
Extract, normalize, and catalog-validation catalog identities agree for the same bound references, and catalog changes invalidate normalize checkpoints.
The maintained production pipeline and current-behavior documentation use the D&D normalizer without changing the durable artifact schema.

Evaluation

Maintain a small human-reviewed set of representative spell outputs covering canonical names, aliases, repeated casts, duplicate model output, and scene boundaries. Use it to review normalization behavior and warnings, not as a claim that LLM extraction is deterministically correct.

Record cases where likely duplicates remain because their evidence differs. Those examples should inform later LLM-assisted deduplication work rather than causing this deterministic milestone to adopt broader heuristics.

Deferred Work

Fuzzy correction of unknown spell names.
Collapsing casts based only on overlapping, adjacent, or semantically similar evidence.
LLM-assisted duplicate proposals or prose reconciliation.
A generic deduplication contract based on stable artifact-element IDs.
Repair-aware extraction retries or LLM-backed semantic validation.
Changes to the spell artifact schema, including stable cast IDs.

An LLM-backed normalizer is explicitly outside this milestone. If later evaluation justifies one, shared spell-catalog prompt material should be designed at that time around the actual normalization or repair request.

6.1 KiB Raw Blame History