notarius/docs/roadmap/dnd-spell-extraction.md

D&D Spell Extraction Quality

Status

The immediate baseline feature is accepted and pending implementation. Feedback-aware repair and semantic LLM validation are deferred until the baseline has been evaluated.

Objective

Improve the precision and consistency of dnd/spells extraction by first grounding the LLM in an effective spell catalog composed from the embedded SRD 5.1 catalog and configured overlays, rejecting invalid catalog names deterministically, and measuring the existing retry mechanism. Preserve a clear separation between response shape, domain policy, retry orchestration, and later semantic review.

Accepted Baseline Scope

Ground extraction with a catalog input

Provide the spell catalog to the extraction prompt as a separate input rather than embedding all spell names as an enum in the private LLM response schema. The response schema should continue to define transport shape, while the catalog remains the single source of truth for recognized spell names.

Build the prompt input from the effective catalog. Include canonical names only; aliases remain available to deterministic lookup but do not belong in the prompt. Omit levels and class memberships unless evaluation demonstrates that they improve extraction. Instruct the model to return canonical catalog names and to use reference material only for disambiguation, never as source evidence.

The effective catalog must support configured overlays from the first implementation so non-SRD, sourcebook, and homebrew spells can participate in the same grounding and validation path. Overlay composition must validate canonical names, aliases, duplicates, conflicts, and provenance deterministically. The prompt and validator must consume the same resolved catalog; configuration and catalog identity must participate in effective pipeline and checkpoint identity wherever they can change accepted output.

This design avoids duplicating the catalog in a schema asset and allows deterministic validation to produce a specific diagnostic for an unknown name.

Add deterministic catalog validation

Add a typed deterministic validator for extracted spell names and include it in the production default chain after shape validation. It should use the catalog lookup rules so case, repeated whitespace, and supported apostrophe variants do not cause false rejections.

The validator should validate only; canonical display-name replacement belongs in the future D&D spell normalizer. It must not silently discard an individual unknown cast from an otherwise valid extraction result.

The initial target chain is:

validators:
  - generic/valid_json
  - generic/valid_json_schema
  - extract/dnd/spells/shape
  - extract/dnd/spells/catalog
  - extract/dnd/spells/source_refs
  - extract/dnd/spells/source_relatedness

A name absent from the effective base-plus-overlay catalog should reject the extraction result. Overlay support must not weaken this policy implicitly.

Measure ordinary extraction retries

Configure the spell extraction binding with retries: 2. This means one initial extraction attempt plus at most two additional attempts for a module error, validator error, or validator rejection. The retry wraps extraction and the complete extraction validator chain.

Keep the framework-wide retry default at zero. The two-retry policy is specific to the spell extraction workflow and should not silently apply to deterministic stages or future LLM modules with different cost and failure characteristics.

The baseline deliberately retains current retry behavior: each retry repeats the same extraction request without the rejected candidate or validator diagnostic. Measure this behavior before introducing corrective prompts so the effects of catalog grounding, deterministic validation, and later repair-aware retry can be distinguished.

Baseline evaluation policy

Exercise the grounded extractor against a small human-reviewed transcript set. Record false positives, false negatives, unknown-name rejections, retry outcomes, acceptance by attempt, and model-call cost. Preserve representative cases that can compare ordinary retry with a future repair-aware strategy.

Do not add an LLM validator during this milestone. Finite catalog membership is deterministic, and semantic LLM validation needs evidence of a specific failure mode before its extra cost and nondeterminism are justified.

Target-State Acceptance Criteria

• The extraction prompt receives canonical names from the effective catalog without maintaining a second handwritten spell list or schema enum; aliases are excluded from the prompt.
• Configured overlays compose deterministically with the embedded catalog, and the extractor and validator consume the same effective catalog.
• Every accepted spell name resolves through the effective catalog, including aliases recognized only by deterministic lookup.
• Unknown spell names produce a scoped deterministic rejection and can consume the configured extraction retry budget.
• Two retries result in no more than three extraction calls for a rejected chunk, excluding any later separately configured LLM validator calls.
• Discarded-attempt warnings are not promoted, and an exhausted rejection remains a rejected pipeline outcome under existing runner semantics.
• The default production validator chain, maintained examples, and applicable current-behavior documentation are updated alongside implementation.

Deferred Retry And Validation Work

Reassess the following only after the baseline evaluation is available:

1. Add bounded, field-specific structured validation issues rather than relying only on a reason code and free-form message. Validators should report all related issues they can safely identify in one pass.
2. Classify retryable outcomes. Provider failures and actionable validation rejections may retry; cancellation must stop; configuration, schema-loading, and internal invariant failures should fail without spending more LLM calls.
3. Add an optional typed repair capability to the extractor contract. The framework should transport the rejected candidate and diagnostics without constructing a domain prompt; each module should own its repair prompt.
4. Evaluate a hybrid two-retry policy: first repair the rejected candidate with structured feedback, then use a fresh extraction without the candidate if repair is still rejected. Compare it with repeated repair and the ordinary retry baseline.
5. Consider a narrowly scoped LLM validator only if human review demonstrates semantic failures that deterministic checks cannot resolve, such as distinguishing a true cast from discussion, intent, table chatter, or an effect inferred from general D&D knowledge.
6. Implement catalog-aware name canonicalization in the D&D normalizer after catalog-validator behavior is stable; this is separate from retry repair.
7. Define the measurements and thresholds that would justify feedback-aware repair or semantic LLM validation. The immediate baseline records evidence but does not need to establish those gates.

Repair responses should remain complete extraction replacements rather than patches. Diagnostics must be treated as bounded data, particularly if a future LLM-backed validator can contribute their text.

Resolved Scope Decisions

• Catalog overlays are part of the immediate implementation, not a later extension.
• Prompt grounding includes canonical spell names only. Aliases participate only in deterministic lookup and validation.
• Measurements and decision thresholds for feedback-aware repair and semantic LLM validation are deferred future work and do not block the baseline.