156 lines
7.6 KiB
Markdown
156 lines
7.6 KiB
Markdown
# D&D Spell Extraction Quality
|
|
|
|
## Status
|
|
|
|
The immediate baseline feature is accepted and pending implementation.
|
|
Feedback-aware repair and semantic LLM validation are deferred until the
|
|
baseline has been evaluated.
|
|
|
|
## Objective
|
|
|
|
Improve the precision and consistency of `dnd/spells` extraction by first
|
|
grounding the LLM in an effective spell catalog composed from the embedded SRD
|
|
5.1 catalog and configured overlays, rejecting invalid catalog names
|
|
deterministically, and measuring the existing retry mechanism. Preserve a clear
|
|
separation between response shape, domain policy, retry orchestration, and later
|
|
semantic review.
|
|
|
|
## Accepted Baseline Scope
|
|
|
|
### Ground extraction with a catalog input
|
|
|
|
Provide the spell catalog to the extraction prompt as a separate input rather
|
|
than embedding all spell names as an enum in the private LLM response schema.
|
|
The response schema should continue to define transport shape, while the
|
|
catalog remains the single source of truth for recognized spell names.
|
|
|
|
Build the prompt input from the effective catalog. Include canonical names
|
|
only; aliases remain available to deterministic lookup but do not belong in the
|
|
prompt. Omit levels and class memberships unless evaluation demonstrates that
|
|
they improve extraction. Instruct the model to return canonical catalog names
|
|
and to use reference material only for disambiguation, never as source
|
|
evidence.
|
|
|
|
The effective catalog must support configured overlays from the first
|
|
implementation so non-SRD, sourcebook, and homebrew spells can participate in
|
|
the same grounding and validation path. Overlay composition must validate
|
|
canonical names, aliases, duplicates, conflicts, and provenance
|
|
deterministically. The prompt and validator must consume the same resolved
|
|
catalog; configuration and catalog identity must participate in effective
|
|
pipeline and checkpoint identity wherever they can change accepted output.
|
|
|
|
This design avoids duplicating the catalog in a schema asset and allows
|
|
deterministic validation to produce a specific diagnostic for an unknown name.
|
|
|
|
### Add deterministic catalog validation
|
|
|
|
Add a typed deterministic validator for extracted spell names and include it in
|
|
the production default chain after shape validation. It should use the catalog
|
|
lookup rules so case, repeated whitespace, and supported apostrophe variants do
|
|
not cause false rejections.
|
|
|
|
The validator should validate only; canonical display-name replacement belongs
|
|
in the future D&D spell normalizer. It must not silently discard an individual
|
|
unknown cast from an otherwise valid extraction result.
|
|
|
|
The initial target chain is:
|
|
|
|
```yaml
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/spells/shape
|
|
- extract/dnd/spells/catalog
|
|
- extract/dnd/spells/source_refs
|
|
- extract/dnd/spells/source_relatedness
|
|
```
|
|
|
|
A name absent from the effective base-plus-overlay catalog should reject the
|
|
extraction result. Overlay support must not weaken this policy implicitly.
|
|
|
|
### Measure ordinary extraction retries
|
|
|
|
Configure the spell extraction binding with `retries: 2`. This means one
|
|
initial extraction attempt plus at most two additional attempts for a module
|
|
error, validator error, or validator rejection. The retry wraps extraction and
|
|
the complete extraction validator chain.
|
|
|
|
Keep the framework-wide retry default at zero. The two-retry policy is specific
|
|
to the spell extraction workflow and should not silently apply to deterministic
|
|
stages or future LLM modules with different cost and failure characteristics.
|
|
|
|
The baseline deliberately retains current retry behavior: each retry repeats
|
|
the same extraction request without the rejected candidate or validator
|
|
diagnostic. Measure this behavior before introducing corrective prompts so the
|
|
effects of catalog grounding, deterministic validation, and later repair-aware
|
|
retry can be distinguished.
|
|
|
|
### Baseline evaluation policy
|
|
|
|
Exercise the grounded extractor against a small human-reviewed transcript set.
|
|
Record false positives, false negatives, unknown-name rejections, retry
|
|
outcomes, acceptance by attempt, and model-call cost. Preserve representative
|
|
cases that can compare ordinary retry with a future repair-aware strategy.
|
|
|
|
Do not add an LLM validator during this milestone. Finite catalog membership is
|
|
deterministic, and semantic LLM validation needs evidence of a specific failure
|
|
mode before its extra cost and nondeterminism are justified.
|
|
|
|
## Target-State Acceptance Criteria
|
|
|
|
- The extraction prompt receives canonical names from the effective catalog
|
|
without maintaining a second handwritten spell list or schema enum; aliases
|
|
are excluded from the prompt.
|
|
- Configured overlays compose deterministically with the embedded catalog, and
|
|
the extractor and validator consume the same effective catalog.
|
|
- Every accepted spell name resolves through the effective catalog, including
|
|
aliases recognized only by deterministic lookup.
|
|
- Unknown spell names produce a scoped deterministic rejection and can consume
|
|
the configured extraction retry budget.
|
|
- Two retries result in no more than three extraction calls for a rejected
|
|
chunk, excluding any later separately configured LLM validator calls.
|
|
- Discarded-attempt warnings are not promoted, and an exhausted rejection
|
|
remains a rejected pipeline outcome under existing runner semantics.
|
|
- The default production validator chain, maintained examples, and applicable
|
|
current-behavior documentation are updated alongside implementation.
|
|
|
|
## Deferred Retry And Validation Work
|
|
|
|
Reassess the following only after the baseline evaluation is available:
|
|
|
|
1. Add bounded, field-specific structured validation issues rather than relying
|
|
only on a reason code and free-form message. Validators should report all
|
|
related issues they can safely identify in one pass.
|
|
2. Classify retryable outcomes. Provider failures and actionable validation
|
|
rejections may retry; cancellation must stop; configuration, schema-loading,
|
|
and internal invariant failures should fail without spending more LLM calls.
|
|
3. Add an optional typed repair capability to the extractor contract. The
|
|
framework should transport the rejected candidate and diagnostics without
|
|
constructing a domain prompt; each module should own its repair prompt.
|
|
4. Evaluate a hybrid two-retry policy: first repair the rejected candidate with
|
|
structured feedback, then use a fresh extraction without the candidate if
|
|
repair is still rejected. Compare it with repeated repair and the ordinary
|
|
retry baseline.
|
|
5. Consider a narrowly scoped LLM validator only if human review demonstrates
|
|
semantic failures that deterministic checks cannot resolve, such as
|
|
distinguishing a true cast from discussion, intent, table chatter, or an
|
|
effect inferred from general D&D knowledge.
|
|
6. Implement catalog-aware name canonicalization in the D&D normalizer after
|
|
catalog-validator behavior is stable; this is separate from retry repair.
|
|
7. Define the measurements and thresholds that would justify feedback-aware
|
|
repair or semantic LLM validation. The immediate baseline records evidence
|
|
but does not need to establish those gates.
|
|
|
|
Repair responses should remain complete extraction replacements rather than
|
|
patches. Diagnostics must be treated as bounded data, particularly if a future
|
|
LLM-backed validator can contribute their text.
|
|
|
|
## Resolved Scope Decisions
|
|
|
|
- Catalog overlays are part of the immediate implementation, not a later
|
|
extension.
|
|
- Prompt grounding includes canonical spell names only. Aliases participate
|
|
only in deterministic lookup and validation.
|
|
- Measurements and decision thresholds for feedback-aware repair and semantic
|
|
LLM validation are deferred future work and do not block the baseline.
|