notarius/docs/roadmap/dnd-spell-normalization.md

D&D Spell Normalization

Status

<<<<<<< HEAD Proposed as the next D&D pipeline milestone. This feature completes the first domain-specific normalize stage for the spell pipeline before work begins on NPC and combat-turn artifacts.

Accepted target state. Implementation details are maintained separately in docs/roadmap/implementation.md.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Objective

Replace the spell lane's no-op normalizer with a deterministic D&D normalizer that emits canonical catalog names, removes only clearly identical duplicate casts, preserves source provenance, and makes every mutation visible through scoped warnings.

<<<<<<< HEAD The milestone should improve the consistency of durable spell output without adding another LLM call or introducing fuzzy repair policy.

The normalizer improves the consistency of durable spell output without adding another LLM call or introducing fuzzy repair policy.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Target Behavior

Effective catalog construction

The normalizer constructs the same immutable SRD-plus-overlay effective catalog used by spell extraction and catalog validation. It declares the optional spell_catalog normalize-stage reference slot and resolves it during pipeline preparation, before source parsing or module execution.

An overlay-capable pipeline binds the same catalog file independently at the extract and normalize stages. This repetition is intentional: references are stage-local, and the normalizer must not reach into a constructed extractor or depend on extractor-private state.

The normalizer contributes its effective catalog digest through pipeline.CheckpointFingerprintProvider. Changing the embedded catalog, composition policy, or normalize-stage overlay therefore invalidates reusable <<<<<<< HEAD normalize checkpoints.

normalize checkpoints. It also records catalog base ID, digest, and overlay IDs as manifest metadata. Neither identity surface includes catalog contents or reference paths.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Canonical spell names

For every spell cast, look up the extracted name using the effective catalog's existing case, whitespace, apostrophe, and alias rules. Replace a recognized value with its canonical display name. This is the only spell-name repair in <<<<<<< HEAD the initial feature.

this feature.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Do not use edit distance, phonetic matching, model judgment, or another fuzzy heuristic. If a value does not resolve, retain it unchanged and emit a scoped warning; the configured normalize validator chain remains responsible for acceptance or rejection.

Emit a warning for each changed spell name. Diagnostics should identify the artifact index and the original and canonical values without modifying other <<<<<<< HEAD fields.

fields. Diagnostics are deterministic and bounded: user-controlled spell names are truncated to 128 Unicode code points before display.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Source-reference normalization

Sort each cast's source references by source identity, start unit, and end unit, then remove exact duplicate references. Do not merge adjacent or merely overlapping ranges, because doing so could broaden the evidence attributed to an event.

The normalizer must not synthesize source references, alter source-unit <<<<<<< HEAD boundaries, or use auxiliary references as evidence.

boundaries, or use auxiliary references as evidence. Reordering references or removing exact duplicates emits a scoped warning for the affected input cast.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Conservative duplicate collapse

After name and source-reference canonicalization, treat two casts as the same event only when all of the following match:

• canonical spell name;
• caster after case folding and whitespace normalization; and <<<<<<< HEAD
• the complete canonical source-reference set. =======
• the complete canonical source-reference set, which must be non-empty and valid against the source document.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Collapse each such group into its first occurrence, preserving stable pipeline order. Retain the first cast's caster, effect, and narrative description. Do not combine prose fields or select a winner based on length, confidence, or model-like semantic judgment. The retained cast receives the group's already canonical source-reference set.

Emit one scoped warning per collapsed group, including the retained index and the removed indices. Casts with different evidence remain distinct even when their spell and caster match. In particular, adjacency at a chunk or scene boundary is not sufficient evidence of duplication.

<<<<<<< HEAD

Unknown spell names, empty evidence sets, and invalid source references cannot establish duplicate identity. They remain in the artifact for the configured normalize validators to accept or reject. A duplicate warning displays at most 20 removed input indices and reports the exact omitted count.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Production composition and validation

Register the typed spell normalizer in the D&D family and make it the default normalizer for the maintained production spell pipeline. Keep the artifact kind and durable spell-list schema unchanged.

Add a normalize-stage production validator chain using the existing generic JSON and JSON Schema validators followed by the existing spell shape, catalog, source-reference, and source-relatedness validators in their current order. Explicit validator overrides remain authoritative.

The maintained overlay-capable example should bind spell_catalog at both the extract and normalize stages. The base-only example should continue to work without a catalog reference.

Acceptance Criteria

• Base and overlay aliases are written using canonical catalog display names.
• Unknown names are never guessed, deleted, or silently changed.
• Source references are deterministically sorted and exactly deduplicated.
• Only casts with matching canonical spell, normalized caster, and identical <<<<<<< HEAD evidence sets collapse; near or adjacent events remain separate.
• Normalization preserves the first cast's non-name fields and stable order.
• Every name change and duplicate collapse emits a scoped warning. ======= non-empty valid evidence sets collapse; near, invalid, unknown, or adjacent events remain separate.
• Normalization preserves the first cast's non-name fields and stable order.
• Every name change, unresolved name, source-reference mutation, and duplicate collapse emits a scoped warning.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

• Extract, normalize, and catalog-validation catalog identities agree for the same bound references, and catalog changes invalidate normalize checkpoints.
• The maintained production pipeline and current-behavior documentation use the D&D normalizer without changing the durable artifact schema.

Evaluation

Maintain a small human-reviewed set of representative spell outputs covering canonical names, aliases, repeated casts, duplicate model output, and scene boundaries. Use it to review normalization behavior and warnings, not as a claim that LLM extraction is deterministically correct.

Record cases where likely duplicates remain because their evidence differs. Those examples should inform later LLM-assisted deduplication work rather than <<<<<<< HEAD causing this deterministic milestone to adopt broader heuristics.

causing this deterministic feature to adopt broader heuristics.

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)

Deferred Work

• Fuzzy correction of unknown spell names.
• Collapsing casts based only on overlapping, adjacent, or semantically similar evidence.
• LLM-assisted duplicate proposals or prose reconciliation.
• A generic deduplication contract based on stable artifact-element IDs.
• Repair-aware extraction retries or LLM-backed semantic validation.
• Changes to the spell artifact schema, including stable cast IDs.

<<<<<<< HEAD An LLM-backed normalizer is explicitly outside this milestone. If later

An LLM-backed normalizer is explicitly outside this feature. If later

6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module) evaluation justifies one, shared spell-catalog prompt material should be designed at that time around the actual normalization or repair request.