Add feature roadmap and implementation plan for D&D spell extraction and validation upgrades

docs/roadmap/dnd-spell-extraction.md

@@ -0,0 +1,155 @@
+# D&D Spell Extraction Quality
+
+## Status
+
+The immediate baseline feature is accepted and pending implementation.
+Feedback-aware repair and semantic LLM validation are deferred until the
+baseline has been evaluated.
+
+## Objective
+
+Improve the precision and consistency of `dnd/spells` extraction by first
+grounding the LLM in an effective spell catalog composed from the embedded SRD
+5.1 catalog and configured overlays, rejecting invalid catalog names
+deterministically, and measuring the existing retry mechanism. Preserve a clear
+separation between response shape, domain policy, retry orchestration, and later
+semantic review.
+
+## Accepted Baseline Scope
+
+### Ground extraction with a catalog input
+
+Provide the spell catalog to the extraction prompt as a separate input rather
+than embedding all spell names as an enum in the private LLM response schema.
+The response schema should continue to define transport shape, while the
+catalog remains the single source of truth for recognized spell names.
+
+Build the prompt input from the effective catalog. Include canonical names
+only; aliases remain available to deterministic lookup but do not belong in the
+prompt. Omit levels and class memberships unless evaluation demonstrates that
+they improve extraction. Instruct the model to return canonical catalog names
+and to use reference material only for disambiguation, never as source
+evidence.
+
+The effective catalog must support configured overlays from the first
+implementation so non-SRD, sourcebook, and homebrew spells can participate in
+the same grounding and validation path. Overlay composition must validate
+canonical names, aliases, duplicates, conflicts, and provenance
+deterministically. The prompt and validator must consume the same resolved
+catalog; configuration and catalog identity must participate in effective
+pipeline and checkpoint identity wherever they can change accepted output.
+
+This design avoids duplicating the catalog in a schema asset and allows
+deterministic validation to produce a specific diagnostic for an unknown name.
+
+### Add deterministic catalog validation
+
+Add a typed deterministic validator for extracted spell names and include it in
+the production default chain after shape validation. It should use the catalog
+lookup rules so case, repeated whitespace, and supported apostrophe variants do
+not cause false rejections.
+
+The validator should validate only; canonical display-name replacement belongs
+in the future D&D spell normalizer. It must not silently discard an individual
+unknown cast from an otherwise valid extraction result.
+
+The initial target chain is:
+
+```yaml
+validators:
+  - generic/valid_json
+  - generic/valid_json_schema
+  - extract/dnd/spells/shape
+  - extract/dnd/spells/catalog
+  - extract/dnd/spells/source_refs
+  - extract/dnd/spells/source_relatedness
+```
+
+A name absent from the effective base-plus-overlay catalog should reject the
+extraction result. Overlay support must not weaken this policy implicitly.
+
+### Measure ordinary extraction retries
+
+Configure the spell extraction binding with `retries: 2`. This means one
+initial extraction attempt plus at most two additional attempts for a module
+error, validator error, or validator rejection. The retry wraps extraction and
+the complete extraction validator chain.
+
+Keep the framework-wide retry default at zero. The two-retry policy is specific
+to the spell extraction workflow and should not silently apply to deterministic
+stages or future LLM modules with different cost and failure characteristics.
+
+The baseline deliberately retains current retry behavior: each retry repeats
+the same extraction request without the rejected candidate or validator
+diagnostic. Measure this behavior before introducing corrective prompts so the
+effects of catalog grounding, deterministic validation, and later repair-aware
+retry can be distinguished.
+
+### Baseline evaluation policy
+
+Exercise the grounded extractor against a small human-reviewed transcript set.
+Record false positives, false negatives, unknown-name rejections, retry
+outcomes, acceptance by attempt, and model-call cost. Preserve representative
+cases that can compare ordinary retry with a future repair-aware strategy.
+
+Do not add an LLM validator during this milestone. Finite catalog membership is
+deterministic, and semantic LLM validation needs evidence of a specific failure
+mode before its extra cost and nondeterminism are justified.
+
+## Target-State Acceptance Criteria
+
+- The extraction prompt receives canonical names from the effective catalog
+  without maintaining a second handwritten spell list or schema enum; aliases
+  are excluded from the prompt.
+- Configured overlays compose deterministically with the embedded catalog, and
+  the extractor and validator consume the same effective catalog.
+- Every accepted spell name resolves through the effective catalog, including
+  aliases recognized only by deterministic lookup.
+- Unknown spell names produce a scoped deterministic rejection and can consume
+  the configured extraction retry budget.
+- Two retries result in no more than three extraction calls for a rejected
+  chunk, excluding any later separately configured LLM validator calls.
+- Discarded-attempt warnings are not promoted, and an exhausted rejection
+  remains a rejected pipeline outcome under existing runner semantics.
+- The default production validator chain, maintained examples, and applicable
+  current-behavior documentation are updated alongside implementation.
+
+## Deferred Retry And Validation Work
+
+Reassess the following only after the baseline evaluation is available:
+
+1. Add bounded, field-specific structured validation issues rather than relying
+   only on a reason code and free-form message. Validators should report all
+   related issues they can safely identify in one pass.
+2. Classify retryable outcomes. Provider failures and actionable validation
+   rejections may retry; cancellation must stop; configuration, schema-loading,
+   and internal invariant failures should fail without spending more LLM calls.
+3. Add an optional typed repair capability to the extractor contract. The
+   framework should transport the rejected candidate and diagnostics without
+   constructing a domain prompt; each module should own its repair prompt.
+4. Evaluate a hybrid two-retry policy: first repair the rejected candidate with
+   structured feedback, then use a fresh extraction without the candidate if
+   repair is still rejected. Compare it with repeated repair and the ordinary
+   retry baseline.
+5. Consider a narrowly scoped LLM validator only if human review demonstrates
+   semantic failures that deterministic checks cannot resolve, such as
+   distinguishing a true cast from discussion, intent, table chatter, or an
+   effect inferred from general D&D knowledge.
+6. Implement catalog-aware name canonicalization in the D&D normalizer after
+   catalog-validator behavior is stable; this is separate from retry repair.
+7. Define the measurements and thresholds that would justify feedback-aware
+   repair or semantic LLM validation. The immediate baseline records evidence
+   but does not need to establish those gates.
+
+Repair responses should remain complete extraction replacements rather than
+patches. Diagnostics must be treated as bounded data, particularly if a future
+LLM-backed validator can contribute their text.
+
+## Resolved Scope Decisions
+
+- Catalog overlays are part of the immediate implementation, not a later
+  extension.
+- Prompt grounding includes canonical spell names only. Aliases participate
+  only in deterministic lookup and validation.
+- Measurements and decision thresholds for feedback-aware repair and semantic
+  LLM validation are deferred future work and do not block the baseline.