Add feature roadmap and implementation plan for D&D spell normalization module

docs/roadmap/dnd-spell-normalization.md

@@ -2,9 +2,14 @@
 
 ## Status
 
+<<<<<<< HEAD
 Proposed as the next D&D pipeline milestone. This feature completes the first
 domain-specific normalize stage for the spell pipeline before work begins on
 NPC and combat-turn artifacts.
+=======
+Accepted target state. Implementation details are maintained separately in
+`docs/roadmap/implementation.md`.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ## Objective
 
@@ -13,8 +18,13 @@ that emits canonical catalog names, removes only clearly identical duplicate
 casts, preserves source provenance, and makes every mutation visible through
 scoped warnings.
 
+<<<<<<< HEAD
 The milestone should improve the consistency of durable spell output without
 adding another LLM call or introducing fuzzy repair policy.
+=======
+The normalizer improves the consistency of durable spell output without adding
+another LLM call or introducing fuzzy repair policy.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ## Target Behavior
 
@@ -33,14 +43,24 @@ depend on extractor-private state.
 The normalizer contributes its effective catalog digest through
 `pipeline.CheckpointFingerprintProvider`. Changing the embedded catalog,
 composition policy, or normalize-stage overlay therefore invalidates reusable
+<<<<<<< HEAD
 normalize checkpoints.
+=======
+normalize checkpoints. It also records catalog base ID, digest, and overlay IDs
+as manifest metadata. Neither identity surface includes catalog contents or
+reference paths.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ### Canonical spell names
 
 For every spell cast, look up the extracted name using the effective catalog's
 existing case, whitespace, apostrophe, and alias rules. Replace a recognized
 value with its canonical display name. This is the only spell-name repair in
+<<<<<<< HEAD
 the initial feature.
+=======
+this feature.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 Do not use edit distance, phonetic matching, model judgment, or another fuzzy
 heuristic. If a value does not resolve, retain it unchanged and emit a scoped
@@ -49,7 +69,12 @@ acceptance or rejection.
 
 Emit a warning for each changed spell name. Diagnostics should identify the
 artifact index and the original and canonical values without modifying other
+<<<<<<< HEAD
 fields.
+=======
+fields. Diagnostics are deterministic and bounded: user-controlled spell names
+are truncated to 128 Unicode code points before display.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ### Source-reference normalization
 
@@ -59,7 +84,12 @@ overlapping ranges, because doing so could broaden the evidence attributed to
 an event.
 
 The normalizer must not synthesize source references, alter source-unit
+<<<<<<< HEAD
 boundaries, or use auxiliary references as evidence.
+=======
+boundaries, or use auxiliary references as evidence. Reordering references or
+removing exact duplicates emits a scoped warning for the affected input cast.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ### Conservative duplicate collapse
 
@@ -68,7 +98,12 @@ event only when all of the following match:
 
 - canonical spell name;
 - caster after case folding and whitespace normalization; and
+<<<<<<< HEAD
 - the complete canonical source-reference set.
+=======
+- the complete canonical source-reference set, which must be non-empty and
+  valid against the source document.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 Collapse each such group into its first occurrence, preserving stable pipeline
 order. Retain the first cast's caster, effect, and narrative description. Do
@@ -81,6 +116,14 @@ the removed indices. Casts with different evidence remain distinct even when
 their spell and caster match. In particular, adjacency at a chunk or scene
 boundary is not sufficient evidence of duplication.
 
+<<<<<<< HEAD
+=======
+Unknown spell names, empty evidence sets, and invalid source references cannot
+establish duplicate identity. They remain in the artifact for the configured
+normalize validators to accept or reject. A duplicate warning displays at most
+20 removed input indices and reports the exact omitted count.
+
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 ### Production composition and validation
 
 Register the typed spell normalizer in the D&D family and make it the default
@@ -102,9 +145,17 @@ without a catalog reference.
 - Unknown names are never guessed, deleted, or silently changed.
 - Source references are deterministically sorted and exactly deduplicated.
 - Only casts with matching canonical spell, normalized caster, and identical
+<<<<<<< HEAD
   evidence sets collapse; near or adjacent events remain separate.
 - Normalization preserves the first cast's non-name fields and stable order.
 - Every name change and duplicate collapse emits a scoped warning.
+=======
+  non-empty valid evidence sets collapse; near, invalid, unknown, or adjacent
+  events remain separate.
+- Normalization preserves the first cast's non-name fields and stable order.
+- Every name change, unresolved name, source-reference mutation, and duplicate
+  collapse emits a scoped warning.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 - Extract, normalize, and catalog-validation catalog identities agree for the
   same bound references, and catalog changes invalidate normalize checkpoints.
 - The maintained production pipeline and current-behavior documentation use
@@ -119,7 +170,11 @@ claim that LLM extraction is deterministically correct.
 
 Record cases where likely duplicates remain because their evidence differs.
 Those examples should inform later LLM-assisted deduplication work rather than
+<<<<<<< HEAD
 causing this deterministic milestone to adopt broader heuristics.
+=======
+causing this deterministic feature to adopt broader heuristics.
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 
 ## Deferred Work
 
@@ -131,6 +186,10 @@ causing this deterministic milestone to adopt broader heuristics.
 - Repair-aware extraction retries or LLM-backed semantic validation.
 - Changes to the spell artifact schema, including stable cast IDs.
 
+<<<<<<< HEAD
 An LLM-backed normalizer is explicitly outside this milestone. If later
+=======
+An LLM-backed normalizer is explicitly outside this feature. If later
+>>>>>>> 6fbdf6b (Add feature roadmap and implementation plan for D&D spell normalization module)
 evaluation justifies one, shared spell-catalog prompt material should be
 designed at that time around the actual normalization or repair request.

docs/roadmap/implementation.md

@@ -0,0 +1,388 @@
+# D&D Spell Normalization Implementation Plan
+
+## Status
+
+Ready for implementation.
+
+This plan implements the accepted target state in
+[`dnd-spell-normalization.md`](dnd-spell-normalization.md). That document owns
+feature intent and policy; this document owns implementation order and concrete
+engineering decisions. Implement each stage in sequence and keep every stage
+buildable and testable before proceeding.
+
+## Scope and invariants
+
+Add a deterministic `dnd/spells` normalize-stage module. It materializes the
+same effective embedded-base-plus-overlay spell catalog used by extraction,
+canonicalizes catalog-backed spell names, canonicalizes exact source-reference
+sets, and conservatively collapses duplicate spell casts. It does not call an
+LLM and does not add a prompt asset.
+
+Preserve these boundaries throughout the work:
+
+- Keep domain behavior under `internal/modules/dnd`; do not add D&D concepts to
+  the pipeline framework or generic contracts.
+- Keep the fixed pipeline architecture unchanged. This work implements an
+  existing normalize stage; it does not introduce branches or DAG behavior.
+- Keep `dnd.SpellList` and its JSON schema unchanged. Do not add artifact IDs,
+  schema constraints, or fields used only for normalization.
+- Keep existing extractor, merger, validator, retry, and checkpoint formats
+  unchanged unless this plan explicitly identifies an integration adjustment.
+- Do not add fuzzy spell matching, caster entity resolution, source-range
+  merging, prose reconciliation, generic duplicate normalization, or LLM-backed
+  validation.
+- Do not mutate request values or slices that may share backing storage with an
+  upstream result. Construct an independent normalized result.
+- Preserve nil and empty spell/reference collections rather than repairing
+  malformed shape. Validators own shape and source-validity rejection.
+- Keep warnings stable, deterministic, bounded, and free of catalog content,
+  reference paths, or other sensitive material.
+
+The normalize-stage public selector is `dnd/spells`. Stage-specific registries
+make this unambiguous despite the extractor using the same selector.
+
+## Stage 1: Typed normalizer and catalog contract
+
+Create `internal/modules/dnd/normalize/spells` as a typed normalize module.
+Follow the construction and registration patterns used by the existing D&D
+spell extractor and deterministic normalizers.
+
+### Module construction
+
+- Define an empty `Options` type and use the repository's strict option decoder
+  so unknown configuration fields are rejected.
+- Expose a constructor that accepts decoded options and the prepared
+  `contracts.ReferenceSet`.
+- Resolve the effective catalog during module construction with the existing
+  `internal/modules/dnd/spells/catalog` resolver. Do not reimplement catalog
+  parsing, alias handling, or digest calculation.
+- Declare one optional reference slot named `spell_catalog`, matching the
+  extractor's media type and size limit. Do not declare campaign references the
+  normalizer does not consume.
+- Make malformed, duplicated, oversized, or otherwise invalid catalog
+  references fail during pipeline preparation, before run-state or checkpoint
+  composition.
+- Register the module as `pipeline.StageNormalize`, artifact kind
+  `dnd.SpellListKind`, under selector `dnd/spells`.
+- Do not require or consume an LLM profile. The normalizer remains fully
+  deterministic and offline.
+
+### Prepared identity and audit metadata
+
+Implement `pipeline.CheckpointFingerprintProvider`. Contribute exactly one
+local fingerprint:
+
+- name: `effective_catalog`
+- value: the effective catalog's semantic digest
+
+Preparation scopes this fingerprint by the normalize-stage component identity.
+Return defensive data according to the provider contract.
+
+Also implement `contracts.ManifestMetadataProvider` with the same non-secret
+audit fields used by catalog-backed extraction components:
+
+- `catalog_base_id`
+- `catalog_digest`
+- `catalog_overlay_ids`
+
+Return defensive copies of collection values. Do not include catalog entries,
+raw reference content, or local paths in either identity or metadata.
+
+### Name canonicalization
+
+For each input spell cast, call the effective catalog's existing `Lookup`
+operation. This intentionally inherits its case, whitespace, apostrophe, and
+explicit-alias policy.
+
+- If lookup succeeds, write the catalog's canonical display name to the output.
+- If the input is already byte-for-byte equal to the canonical name, emit no
+  name warning.
+- If lookup succeeds and changes the value, emit reason code
+  `spell_name_canonicalized` at scope `spell_casts[N]`, where `N` is the merged
+  input index. The message identifies the input index, bounded original value,
+  and canonical replacement.
+- If lookup fails, preserve the original value exactly and emit reason code
+  `spell_name_unresolved` at scope `spell_casts[N]`. Do not guess, drop, or
+  rewrite it.
+
+Before quoting either an extracted name or an overlay-provided canonical name
+in a warning, limit it to 128 Unicode code points by retaining the first 127 and
+appending a visible ellipsis. Never truncate by byte position or produce invalid
+UTF-8. Use normal Go quoting so control characters cannot corrupt diagnostic
+structure.
+
+### Source-reference canonicalization
+
+For every cast, copy and sort references by this exact tuple:
+
+1. `SourceID`, compared as an exact string;
+2. `StartUnitID`;
+3. `EndUnitID`.
+
+Then remove exact structural duplicates. Do not trim or rewrite `SourceID`,
+alter boundaries, merge adjacent or overlapping ranges, or consult auxiliary
+reference text. Preserve nil as nil and an empty non-nil slice as empty.
+
+If sorting changes order or exact duplicates are removed, emit one warning with
+reason code `source_references_normalized` at `spell_casts[N]`. Its message
+reports the input index, original count, final count, whether order changed, and
+the duplicate-removal count. Do not emit it when the canonical result is
+structurally identical to the input.
+
+### Stage 1 tests
+
+Add focused package tests that establish:
+
+- strict option decoding and the exact module/reference-slot contract;
+- successful embedded-base and overlay catalog construction;
+- preparation-time failure for invalid catalog references;
+- checkpoint fingerprint and manifest metadata values and defensive copying;
+- canonical, case/whitespace/apostrophe, and overlay-alias lookups;
+- canonicalization and unresolved-name warning semantics;
+- safe truncation of long Unicode names and quoting of control characters;
+- reference tuple ordering and exact de-duplication;
+- no merging of adjacent or overlapping ranges;
+- source-normalization warning semantics;
+- preservation of nil versus empty collections and input immutability; and
+- the repository's standard nil receiver, nil context, and already-canceled
+  context behavior for module calls.
+
+Do not assert entire warning prose when reason code, scope, and semantic message
+fragments are the durable contract.
+
+## Stage 2: Conservative duplicate collapse
+
+Perform duplicate detection after name and source-reference canonicalization.
+Use original merged input positions as stable indices for warnings and retained
+entry selection.
+
+### Duplicate identity
+
+A cast is eligible for duplicate grouping only when all of these hold:
+
+- its spell name resolved successfully through the effective catalog;
+- its canonical source-reference set is non-empty; and
+- every canonical source reference passes `source.ValidateRef` against the
+  normalize request's source document.
+
+Eligible casts are duplicates only when all of these keys are exactly equal:
+
+- canonical catalog display name;
+- normalized caster key; and
+- complete canonical source-reference set.
+
+Build the caster key by applying `strings.Fields`, joining with one ASCII space,
+and then applying `golang.org/x/text/cases.Fold`. This implements Unicode case
+folding rather than locale-specific lowercasing. Make `golang.org/x/text` a
+direct module dependency when importing it; it is already present indirectly.
+Leave output caster text untouched. Do not remove punctuation, apply aliases,
+or attempt entity resolution.
+
+Compare source-reference sets only after the deterministic sorting and exact
+de-duplication from Stage 1. Exact equality includes source identity and both
+boundaries. Unknown names, empty evidence, or any invalid source reference make
+a cast ineligible for duplicate grouping; the cast remains in output so the
+configured normalize validators can decide its validity.
+
+### Collapse behavior
+
+- Retain the first input occurrence of each duplicate group and preserve stable
+  output order.
+- The retained artifact is its normalized Stage 1 copy. Preserve that first
+  occurrence's caster, effect, and narrative verbatim.
+- Do not union sources, combine prose, select a “better” occurrence, or use
+  adjacency/overlap as duplicate evidence.
+- Preserve all casts that do not meet the complete duplicate identity.
+
+Emit one `duplicate_spell_cast_collapsed` warning per collapsed group. Scope it
+to the retained input cast (`spell_casts[R]`) and identify the retained input
+index and removed input indices. Order group warnings by retained input index.
+Display at most 20 removed indices and report the exact omitted count when more
+exist.
+
+Warning order for the complete normalizer is:
+
+1. Walk input casts in input order. For each cast, emit its name warning, if
+   any, followed by its source-reference warning, if any.
+2. Emit duplicate-group warnings ordered by retained input index.
+
+Per-cast warnings remain present even when that input cast is later removed by
+duplicate collapse; warning scopes deliberately refer to merged input indices.
+
+### Stage 2 tests
+
+Add table-driven and focused behavioral tests for:
+
+- two- and three-member duplicate groups;
+- caster case and whitespace normalization without output caster rewriting;
+- first-occurrence retention, stable output ordering, and preservation of the
+  first occurrence's non-identity fields;
+- distinct canonical spells, casters, or evidence sets remaining separate;
+- exact evidence equality across differently ordered or duplicated input refs;
+- unknown spells, empty evidence, and invalid refs never collapsing;
+- no collapse based only on adjacent or overlapping evidence;
+- no source union or prose combination;
+- deterministic warning ordering and input-index semantics;
+- the 20-index warning display bound and exact omitted count; and
+- output/input slice independence under mutation checks.
+
+Include an idempotence test: normalizing an already normalized successful result
+does not change artifacts or emit new mutation warnings.
+
+## Stage 3: D&D defaults and maintained configurations
+
+Update the D&D registrar without changing global framework defaults.
+
+### Registration and validator composition
+
+- Add the spell normalizer package to the D&D registrar and register its
+  normalize-stage variant.
+- Register this default validator chain for stage `normalize`, module
+  `dnd/spells`, in this exact order: `json`, `schema`,
+  `extract/dnd/spells/shape`, `extract/dnd/spells/catalog`,
+  `extract/dnd/spells/source_refs`, and
+  `extract/dnd/spells/source_relatedness`.
+- Reuse the existing validator implementations and selectors. Do not rename
+  them merely because they are now also used after normalization.
+- Leave the extractor's default validator chain unchanged.
+- Preserve explicit pipeline validator overrides as authoritative; defaults are
+  used only when configuration does not supply an override.
+- Leave the global/default `noop` normalizer behavior unchanged for all other
+  pipelines and lanes.
+
+The relatedness validator may conservatively warn when a transcript uses an
+alias while the artifact contains the canonical spelling. That warning is
+non-fatal and does not justify changing validator behavior in this scope.
+
+### Maintained examples
+
+Update maintained D&D spell configurations as follows:
+
+- The embedded-base example explicitly selects `normalize: dnd/spells` and
+  requires no normalize-stage catalog reference.
+- The overlay example independently binds the same catalog document under
+  `pipelines.<pipeline>.artifacts.<lane>.normalize.references.spell_catalog` as
+  it does under extraction. Do not rely on implicit cross-stage reference
+  sharing.
+
+Ensure both maintained examples pass strict configuration loading and effective
+pipeline materialization. Do not add an LLM profile to the deterministic
+normalizer.
+
+### Stage 3 tests
+
+Add or update registration and configuration tests to prove:
+
+- `dnd/spells` resolves as the typed spell normalizer;
+- its exact normalize default validator order is stable;
+- existing extract defaults remain unchanged;
+- explicit normalize validator overrides remain authoritative;
+- the base example works without `spell_catalog` at normalize; and
+- the overlay example binds its catalog independently to both extraction and
+  normalization.
+
+Prefer asserting semantic scope and values over global literal counts that
+become brittle whenever another prepared component contributes catalog
+identity.
+
+## Stage 4: Assembled execution, provenance, and checkpoint identity
+
+Add an offline assembled-pipeline test using deterministic or fake upstream
+output. The fixture must exercise records crossing chunk/merge boundaries
+without an external LLM call. Verify that the normalizer:
+
+- receives merged spell casts;
+- canonicalizes a catalog alias or spelling variant;
+- normalizes reference order;
+- collapses only an exact supported duplicate;
+- leaves different evidence as a distinct event;
+- returns warnings through the existing run result and manifest path; and
+- produces output accepted by the default normalize validator chain.
+
+Add a separate assertion that an explicit normalize validator override remains
+in force in an assembled pipeline.
+
+Extend checkpoint and provenance coverage to show:
+
+- the normalizer contributes its independently scoped `effective_catalog`
+  fingerprint;
+- catalog-backed extractor and validators continue to contribute their own
+  independently scoped identities;
+- all components materialized from the same reference set report the same
+  semantic digest;
+- changing effective catalog semantics changes run/checkpoint identity and
+  prevents normalize-checkpoint reuse;
+- semantically equivalent catalog material retains the semantic digest under
+  the existing catalog rules;
+- raw overlay reference provenance remains present independently of the
+  semantic catalog fingerprint; and
+- the manifest contains the normalizer's base ID, digest, and overlay IDs.
+
+Do not change checkpoint schema, layout, compatibility, or the component
+fingerprint framework. If an existing test assumes an exact total fingerprint
+count, replace that brittle assertion with scoped name/value assertions while
+retaining checks for LLM-profile and reference provenance.
+
+## Stage 5: Documentation and evaluation fixtures
+
+Update current-state documentation only as behavior becomes implemented:
+
+- `docs/config.md`: document `dnd/spells` as a normalize selector, its optional
+  stage-local `spell_catalog` reference, and its default validator chain.
+- `docs/internal/modules.md` and `docs/internal/overview.md`: add the typed
+  normalizer, deterministic behavior, catalog dependency, metadata, and
+  checkpoint identity where each document's current-behavior scope requires it.
+- `docs/integrations/dnd-spell-artifacts.md`: document canonical-name behavior,
+  exact source-reference normalization, conservative duplicate identity,
+  preservation rules, and warning reason codes.
+- Maintained examples: ensure their comments explain base-only versus
+  independently bound overlay behavior.
+- `docs/roadmap/dnd-spell-normalization.md`: mark implemented acceptance items
+  complete while leaving any unevaluated human-review claims explicitly
+  pending.
+- `docs/roadmap/future.md`: remove or revise only entries made obsolete by this
+  completed feature. Leave deferred LLM, fuzzy, generic-normalizer, and richer
+  reconciliation work in the future roadmap.
+
+Do not create
+`internal/modules/dnd/shared/assets/prompts/common-dnd-spells.md` in this
+implementation. No component in scope makes an LLM call, so adding the prompt
+would create an unused contract and would not improve backend cache reuse.
+
+Add a compact deterministic fixture set covering the normalizer's accepted
+input/output behavior. Do not fabricate claims about real transcript quality or
+make paid or network LLM calls. If an approved human-reviewed transcript corpus
+is available locally and repository policy permits its use, record aggregate
+observations in the feature roadmap without committing sensitive transcript
+content. Otherwise leave the qualitative evaluation item pending and state
+why.
+
+## Verification gate
+
+At the end of every stage, run the narrowest directly affected package tests.
+Before declaring the implementation complete, run:
+
+```text
+git diff --check
+go test ./...
+go vet ./...
+go build ./cmd/notarius
+```
+
+Run targeted race detection for the affected execution surfaces:
+
+```text
+go test -race ./internal/modules/dnd/... ./internal/framework/pipeline ./internal/cli ./internal/modules/integration
+```
+
+All tests must remain offline and deterministic. This targeted command
+supplements, rather than changes, the repository-wide validation contract in
+`docs/development.md`.
+
+Review the final diff for accidental framework expansion, schema changes,
+prompt assets, generated output, local paths, or unrelated worktree edits.
+
+## Open Questions
+
+None. The feature roadmap and the decisions above are sufficient to implement
+the work without further product or architecture choices.