notarius/docs/roadmap/implementation.md

D&D Spell Normalization Implementation Plan

Status

Ready for implementation.

This plan implements the accepted target state in dnd-spell-normalization.md. That document owns feature intent and policy; this document owns implementation order and concrete engineering decisions. Implement each stage in sequence and keep every stage buildable and testable before proceeding.

Scope and invariants

Add a deterministic dnd/spells normalize-stage module. It materializes the same effective embedded-base-plus-overlay spell catalog used by extraction, canonicalizes catalog-backed spell names, canonicalizes exact source-reference sets, and conservatively collapses duplicate spell casts. It does not call an LLM and does not add a prompt asset.

Preserve these boundaries throughout the work:

• Keep domain behavior under internal/modules/dnd; do not add D&D concepts to the pipeline framework or generic contracts.
• Keep the fixed pipeline architecture unchanged. This work implements an existing normalize stage; it does not introduce branches or DAG behavior.
• Keep dnd.SpellList and its JSON schema unchanged. Do not add artifact IDs, schema constraints, or fields used only for normalization.
• Keep existing extractor, merger, validator, retry, and checkpoint formats unchanged unless this plan explicitly identifies an integration adjustment.
• Do not add fuzzy spell matching, caster entity resolution, source-range merging, prose reconciliation, generic duplicate normalization, or LLM-backed validation.
• Do not mutate request values or slices that may share backing storage with an upstream result. Construct an independent normalized result.
• Preserve nil and empty spell/reference collections rather than repairing malformed shape. Validators own shape and source-validity rejection.
• Keep warnings stable, deterministic, bounded, and free of catalog content, reference paths, or other sensitive material.

The normalize-stage public selector is dnd/spells. Stage-specific registries make this unambiguous despite the extractor using the same selector.

Stage 1: Typed normalizer and catalog contract

Create internal/modules/dnd/normalize/spells as a typed normalize module. Follow the construction and registration patterns used by the existing D&D spell extractor and deterministic normalizers.

Module construction

• Define an empty Options type and use the repository's strict option decoder so unknown configuration fields are rejected.
• Expose a constructor that accepts decoded options and the prepared contracts.ReferenceSet.
• Resolve the effective catalog during module construction with the existing internal/modules/dnd/spells/catalog resolver. Do not reimplement catalog parsing, alias handling, or digest calculation.
• Declare one optional reference slot named spell_catalog, matching the extractor's media type and size limit. Do not declare campaign references the normalizer does not consume.
• Make malformed, duplicated, oversized, or otherwise invalid catalog references fail during pipeline preparation, before run-state or checkpoint composition.
• Register the module as pipeline.StageNormalize, artifact kind dnd.SpellListKind, under selector dnd/spells.
• Do not require or consume an LLM profile. The normalizer remains fully deterministic and offline.

Prepared identity and audit metadata

Implement pipeline.CheckpointFingerprintProvider. Contribute exactly one local fingerprint:

• name: effective_catalog
• value: the effective catalog's semantic digest

Preparation scopes this fingerprint by the normalize-stage component identity. Return defensive data according to the provider contract.

Also implement contracts.ManifestMetadataProvider with the same non-secret audit fields used by catalog-backed extraction components:

• catalog_base_id
• catalog_digest
• catalog_overlay_ids

Return defensive copies of collection values. Do not include catalog entries, raw reference content, or local paths in either identity or metadata.

Name canonicalization

For each input spell cast, call the effective catalog's existing Lookup operation. This intentionally inherits its case, whitespace, apostrophe, and explicit-alias policy.

• If lookup succeeds, write the catalog's canonical display name to the output.
• If the input is already byte-for-byte equal to the canonical name, emit no name warning.
• If lookup succeeds and changes the value, emit reason code spell_name_canonicalized at scope spell_casts[N], where N is the merged input index. The message identifies the input index, bounded original value, and canonical replacement.
• If lookup fails, preserve the original value exactly and emit reason code spell_name_unresolved at scope spell_casts[N]. Do not guess, drop, or rewrite it.

Before quoting either an extracted name or an overlay-provided canonical name in a warning, limit it to 128 Unicode code points by retaining the first 127 and appending a visible ellipsis. Never truncate by byte position or produce invalid UTF-8. Use normal Go quoting so control characters cannot corrupt diagnostic structure.

Source-reference canonicalization

For every cast, copy and sort references by this exact tuple:

1. SourceID, compared as an exact string;
2. StartUnitID;
3. EndUnitID.

Then remove exact structural duplicates. Do not trim or rewrite SourceID, alter boundaries, merge adjacent or overlapping ranges, or consult auxiliary reference text. Preserve nil as nil and an empty non-nil slice as empty.

If sorting changes order or exact duplicates are removed, emit one warning with reason code source_references_normalized at spell_casts[N]. Its message reports the input index, original count, final count, whether order changed, and the duplicate-removal count. Do not emit it when the canonical result is structurally identical to the input.

Stage 1 tests

Add focused package tests that establish:

• strict option decoding and the exact module/reference-slot contract;
• successful embedded-base and overlay catalog construction;
• preparation-time failure for invalid catalog references;
• checkpoint fingerprint and manifest metadata values and defensive copying;
• canonical, case/whitespace/apostrophe, and overlay-alias lookups;
• canonicalization and unresolved-name warning semantics;
• safe truncation of long Unicode names and quoting of control characters;
• reference tuple ordering and exact de-duplication;
• no merging of adjacent or overlapping ranges;
• source-normalization warning semantics;
• preservation of nil versus empty collections and input immutability; and
• the repository's standard nil receiver, nil context, and already-canceled context behavior for module calls.

Do not assert entire warning prose when reason code, scope, and semantic message fragments are the durable contract.

Stage 2: Conservative duplicate collapse

Perform duplicate detection after name and source-reference canonicalization. Use original merged input positions as stable indices for warnings and retained entry selection.

Duplicate identity

A cast is eligible for duplicate grouping only when all of these hold:

• its spell name resolved successfully through the effective catalog;
• its canonical source-reference set is non-empty; and
• every canonical source reference passes source.ValidateRef against the normalize request's source document.

Eligible casts are duplicates only when all of these keys are exactly equal:

• canonical catalog display name;
• normalized caster key; and
• complete canonical source-reference set.

Build the caster key by applying strings.Fields, joining with one ASCII space, and then applying golang.org/x/text/cases.Fold. This implements Unicode case folding rather than locale-specific lowercasing. Make golang.org/x/text a direct module dependency when importing it; it is already present indirectly. Leave output caster text untouched. Do not remove punctuation, apply aliases, or attempt entity resolution.

Compare source-reference sets only after the deterministic sorting and exact de-duplication from Stage 1. Exact equality includes source identity and both boundaries. Unknown names, empty evidence, or any invalid source reference make a cast ineligible for duplicate grouping; the cast remains in output so the configured normalize validators can decide its validity.

Collapse behavior

• Retain the first input occurrence of each duplicate group and preserve stable output order.
• The retained artifact is its normalized Stage 1 copy. Preserve that first occurrence's caster, effect, and narrative verbatim.
• Do not union sources, combine prose, select a “better” occurrence, or use adjacency/overlap as duplicate evidence.
• Preserve all casts that do not meet the complete duplicate identity.

Emit one duplicate_spell_cast_collapsed warning per collapsed group. Scope it to the retained input cast (spell_casts[R]) and identify the retained input index and removed input indices. Order group warnings by retained input index. Display at most 20 removed indices and report the exact omitted count when more exist.

Warning order for the complete normalizer is:

1. Walk input casts in input order. For each cast, emit its name warning, if any, followed by its source-reference warning, if any.
2. Emit duplicate-group warnings ordered by retained input index.

Per-cast warnings remain present even when that input cast is later removed by duplicate collapse; warning scopes deliberately refer to merged input indices.

Stage 2 tests

Add table-driven and focused behavioral tests for:

• two- and three-member duplicate groups;
• caster case and whitespace normalization without output caster rewriting;
• first-occurrence retention, stable output ordering, and preservation of the first occurrence's non-identity fields;
• distinct canonical spells, casters, or evidence sets remaining separate;
• exact evidence equality across differently ordered or duplicated input refs;
• unknown spells, empty evidence, and invalid refs never collapsing;
• no collapse based only on adjacent or overlapping evidence;
• no source union or prose combination;
• deterministic warning ordering and input-index semantics;
• the 20-index warning display bound and exact omitted count; and
• output/input slice independence under mutation checks.

Include an idempotence test: normalizing an already normalized successful result does not change artifacts or emit new mutation warnings.

Stage 3: D&D defaults and maintained configurations

Update the D&D registrar without changing global framework defaults.

Registration and validator composition

• Add the spell normalizer package to the D&D registrar and register its normalize-stage variant.
• Register this default validator chain for stage normalize, module dnd/spells, in this exact order: json, schema, extract/dnd/spells/shape, extract/dnd/spells/catalog, extract/dnd/spells/source_refs, and extract/dnd/spells/source_relatedness.
• Reuse the existing validator implementations and selectors. Do not rename them merely because they are now also used after normalization.
• Leave the extractor's default validator chain unchanged.
• Preserve explicit pipeline validator overrides as authoritative; defaults are used only when configuration does not supply an override.
• Leave the global/default noop normalizer behavior unchanged for all other pipelines and lanes.

The relatedness validator may conservatively warn when a transcript uses an alias while the artifact contains the canonical spelling. That warning is non-fatal and does not justify changing validator behavior in this scope.

Maintained examples

Update maintained D&D spell configurations as follows:

• The embedded-base example explicitly selects normalize: dnd/spells and requires no normalize-stage catalog reference.
• The overlay example independently binds the same catalog document under pipelines.<pipeline>.artifacts.<lane>.normalize.references.spell_catalog as it does under extraction. Do not rely on implicit cross-stage reference sharing.

Ensure both maintained examples pass strict configuration loading and effective pipeline materialization. Do not add an LLM profile to the deterministic normalizer.

Stage 3 tests

Add or update registration and configuration tests to prove:

• dnd/spells resolves as the typed spell normalizer;
• its exact normalize default validator order is stable;
• existing extract defaults remain unchanged;
• explicit normalize validator overrides remain authoritative;
• the base example works without spell_catalog at normalize; and
• the overlay example binds its catalog independently to both extraction and normalization.

Prefer asserting semantic scope and values over global literal counts that become brittle whenever another prepared component contributes catalog identity.

Stage 4: Assembled execution, provenance, and checkpoint identity

Add an offline assembled-pipeline test using deterministic or fake upstream output. The fixture must exercise records crossing chunk/merge boundaries without an external LLM call. Verify that the normalizer:

• receives merged spell casts;
• canonicalizes a catalog alias or spelling variant;
• normalizes reference order;
• collapses only an exact supported duplicate;
• leaves different evidence as a distinct event;
• returns warnings through the existing run result and manifest path; and
• produces output accepted by the default normalize validator chain.

Add a separate assertion that an explicit normalize validator override remains in force in an assembled pipeline.

Extend checkpoint and provenance coverage to show:

• the normalizer contributes its independently scoped effective_catalog fingerprint;
• catalog-backed extractor and validators continue to contribute their own independently scoped identities;
• all components materialized from the same reference set report the same semantic digest;
• changing effective catalog semantics changes run/checkpoint identity and prevents normalize-checkpoint reuse;
• semantically equivalent catalog material retains the semantic digest under the existing catalog rules;
• raw overlay reference provenance remains present independently of the semantic catalog fingerprint; and
• the manifest contains the normalizer's base ID, digest, and overlay IDs.

Do not change checkpoint schema, layout, compatibility, or the component fingerprint framework. If an existing test assumes an exact total fingerprint count, replace that brittle assertion with scoped name/value assertions while retaining checks for LLM-profile and reference provenance.

Stage 5: Documentation and evaluation fixtures

Update current-state documentation only as behavior becomes implemented:

• docs/config.md: document dnd/spells as a normalize selector, its optional stage-local spell_catalog reference, and its default validator chain.
• docs/internal/modules.md and docs/internal/overview.md: add the typed normalizer, deterministic behavior, catalog dependency, metadata, and checkpoint identity where each document's current-behavior scope requires it.
• docs/integrations/dnd-spell-artifacts.md: document canonical-name behavior, exact source-reference normalization, conservative duplicate identity, preservation rules, and warning reason codes.
• Maintained examples: ensure their comments explain base-only versus independently bound overlay behavior.
• docs/roadmap/dnd-spell-normalization.md: mark implemented acceptance items complete while leaving any unevaluated human-review claims explicitly pending.
• docs/roadmap/future.md: remove or revise only entries made obsolete by this completed feature. Leave deferred LLM, fuzzy, generic-normalizer, and richer reconciliation work in the future roadmap.

Do not create internal/modules/dnd/shared/assets/prompts/common-dnd-spells.md in this implementation. No component in scope makes an LLM call, so adding the prompt would create an unused contract and would not improve backend cache reuse.

Add a compact deterministic fixture set covering the normalizer's accepted input/output behavior. Do not fabricate claims about real transcript quality or make paid or network LLM calls. If an approved human-reviewed transcript corpus is available locally and repository policy permits its use, record aggregate observations in the feature roadmap without committing sensitive transcript content. Otherwise leave the qualitative evaluation item pending and state why.

Verification gate

At the end of every stage, run the narrowest directly affected package tests. Before declaring the implementation complete, run:

git diff --check
go test ./...
go vet ./...
go build ./cmd/notarius

Run targeted race detection for the affected execution surfaces:

go test -race ./internal/modules/dnd/... ./internal/framework/pipeline ./internal/cli ./internal/modules/integration

All tests must remain offline and deterministic. This targeted command supplements, rather than changes, the repository-wide validation contract in docs/development.md.

Review the final diff for accidental framework expansion, schema changes, prompt assets, generated output, local paths, or unrelated worktree edits.

Open Questions

None. The feature roadmap and the decisions above are sufficient to implement the work without further product or architecture choices.