181 lines
8.8 KiB
Markdown
181 lines
8.8 KiB
Markdown
# D&D Spell Extraction Quality
|
|
|
|
## Status
|
|
|
|
The accepted baseline implementation is complete. It provides effective
|
|
SRD-plus-overlay catalog composition, canonical-name prompt grounding,
|
|
deterministic catalog validation, the maintained `retries: 2` policy, and
|
|
provenance and checkpoint identity coverage through the assembled offline
|
|
workflow. Checkpoint identity includes independently scoped extractor and
|
|
validator fingerprints of the effective catalog, closing the earlier gap in
|
|
which raw overlay changes invalidated reuse but embedded catalog changes did
|
|
not.
|
|
|
|
External quality evaluation is pending. This repository contains the
|
|
maintained example and offline fake-LLM coverage, but no approved
|
|
human-reviewed transcript corpus or authorized live-model evaluation was
|
|
available for this implementation run. The maintained example can be run
|
|
from the repository root with:
|
|
|
|
```sh
|
|
go run ./cmd/notarius run dnd-session \
|
|
--config examples/dnd-spells-production.config.yml \
|
|
--input examples/seriatim-minimal-transcript.json \
|
|
--chunk_cache bypass
|
|
```
|
|
|
|
That command requires the configured Scriptorium profile credential. When an
|
|
approved profile and reviewed corpus are available, record anonymized
|
|
aggregate false positives, false negatives, unknown-name rejections,
|
|
acceptance by attempt, and model-call cost here without committing transcripts
|
|
or raw prompts. Feedback-aware repair and semantic LLM validation remain
|
|
deferred until that baseline evaluation is available.
|
|
|
|
## Objective
|
|
|
|
Improve the precision and consistency of `dnd/spells` extraction by first
|
|
grounding the LLM in an effective spell catalog composed from the embedded SRD
|
|
5.1 catalog and configured overlays, rejecting invalid catalog names
|
|
deterministically, and measuring the existing retry mechanism. Preserve a clear
|
|
separation between response shape, domain policy, retry orchestration, and later
|
|
semantic review.
|
|
|
|
## Accepted Baseline Scope
|
|
|
|
### Ground extraction with a catalog input
|
|
|
|
Provide the spell catalog to the extraction prompt as a separate input rather
|
|
than embedding all spell names as an enum in the private LLM response schema.
|
|
The response schema should continue to define transport shape, while the
|
|
catalog remains the single source of truth for recognized spell names.
|
|
|
|
Build the prompt input from the effective catalog. Include canonical names
|
|
only; aliases remain available to deterministic lookup but do not belong in the
|
|
prompt. Omit levels and class memberships unless evaluation demonstrates that
|
|
they improve extraction. Instruct the model to return canonical catalog names
|
|
and to use reference material only for disambiguation, never as source
|
|
evidence.
|
|
|
|
The effective catalog must support configured overlays from the first
|
|
implementation so non-SRD, sourcebook, and homebrew spells can participate in
|
|
the same grounding and validation path. Overlay composition must validate
|
|
canonical names, aliases, duplicates, conflicts, and provenance
|
|
deterministically. The prompt and validator must consume the same resolved
|
|
catalog; configuration and catalog identity must participate in effective
|
|
pipeline and checkpoint identity wherever they can change accepted output.
|
|
|
|
This design avoids duplicating the catalog in a schema asset and allows
|
|
deterministic validation to produce a specific diagnostic for an unknown name.
|
|
|
|
### Add deterministic catalog validation
|
|
|
|
Add a typed deterministic validator for extracted spell names and include it in
|
|
the production default chain after shape validation. It should use the catalog
|
|
lookup rules so case, repeated whitespace, and supported apostrophe variants do
|
|
not cause false rejections.
|
|
|
|
The validator should validate only; canonical display-name replacement belongs
|
|
in the future D&D spell normalizer. It must not silently discard an individual
|
|
unknown cast from an otherwise valid extraction result.
|
|
|
|
The initial target chain is:
|
|
|
|
```yaml
|
|
validators:
|
|
- generic/valid_json
|
|
- generic/valid_json_schema
|
|
- extract/dnd/spells/shape
|
|
- extract/dnd/spells/catalog
|
|
- extract/dnd/spells/source_refs
|
|
- extract/dnd/spells/source_relatedness
|
|
```
|
|
|
|
A name absent from the effective base-plus-overlay catalog should reject the
|
|
extraction result. Overlay support must not weaken this policy implicitly.
|
|
|
|
### Measure ordinary extraction retries
|
|
|
|
Configure the spell extraction binding with `retries: 2`. This means one
|
|
initial extraction attempt plus at most two additional attempts for a module
|
|
error, validator error, or validator rejection. The retry wraps extraction and
|
|
the complete extraction validator chain.
|
|
|
|
Keep the framework-wide retry default at zero. The two-retry policy is specific
|
|
to the spell extraction workflow and should not silently apply to deterministic
|
|
stages or future LLM modules with different cost and failure characteristics.
|
|
|
|
The baseline deliberately retains current retry behavior: each retry repeats
|
|
the same extraction request without the rejected candidate or validator
|
|
diagnostic. Measure this behavior before introducing corrective prompts so the
|
|
effects of catalog grounding, deterministic validation, and later repair-aware
|
|
retry can be distinguished.
|
|
|
|
### Baseline evaluation policy
|
|
|
|
Exercise the grounded extractor against a small human-reviewed transcript set.
|
|
Record false positives, false negatives, unknown-name rejections, retry
|
|
outcomes, acceptance by attempt, and model-call cost. Preserve representative
|
|
cases that can compare ordinary retry with a future repair-aware strategy.
|
|
|
|
Do not add an LLM validator during this milestone. Finite catalog membership is
|
|
deterministic, and semantic LLM validation needs evidence of a specific failure
|
|
mode before its extra cost and nondeterminism are justified.
|
|
|
|
## Target-State Acceptance Criteria
|
|
|
|
- The extraction prompt receives canonical names from the effective catalog
|
|
without maintaining a second handwritten spell list or schema enum; aliases
|
|
are excluded from the prompt.
|
|
- Configured overlays compose deterministically with the embedded catalog, and
|
|
the extractor and validator consume the same effective catalog.
|
|
- Every accepted spell name resolves through the effective catalog, including
|
|
aliases recognized only by deterministic lookup.
|
|
- Unknown spell names produce a scoped deterministic rejection and can consume
|
|
the configured extraction retry budget.
|
|
- Two retries result in no more than three extraction calls for a rejected
|
|
chunk, excluding any later separately configured LLM validator calls.
|
|
- Discarded-attempt warnings are not promoted, and an exhausted rejection
|
|
remains a rejected pipeline outcome under existing runner semantics.
|
|
- The default production validator chain, maintained examples, and applicable
|
|
current-behavior documentation are updated alongside implementation.
|
|
|
|
## Deferred Retry And Validation Work
|
|
|
|
Reassess the following only after the baseline evaluation is available:
|
|
|
|
1. Add bounded, field-specific structured validation issues rather than relying
|
|
only on a reason code and free-form message. Validators should report all
|
|
related issues they can safely identify in one pass.
|
|
2. Classify retryable outcomes. Provider failures and actionable validation
|
|
rejections may retry; cancellation must stop; configuration, schema-loading,
|
|
and internal invariant failures should fail without spending more LLM calls.
|
|
3. Add an optional typed repair capability to the extractor contract. The
|
|
framework should transport the rejected candidate and diagnostics without
|
|
constructing a domain prompt; each module should own its repair prompt.
|
|
4. Evaluate a hybrid two-retry policy: first repair the rejected candidate with
|
|
structured feedback, then use a fresh extraction without the candidate if
|
|
repair is still rejected. Compare it with repeated repair and the ordinary
|
|
retry baseline.
|
|
5. Consider a narrowly scoped LLM validator only if human review demonstrates
|
|
semantic failures that deterministic checks cannot resolve, such as
|
|
distinguishing a true cast from discussion, intent, table chatter, or an
|
|
effect inferred from general D&D knowledge.
|
|
6. Implement catalog-aware name canonicalization in the D&D normalizer after
|
|
catalog-validator behavior is stable; this is separate from retry repair.
|
|
7. Define the measurements and thresholds that would justify feedback-aware
|
|
repair or semantic LLM validation. The immediate baseline records evidence
|
|
but does not need to establish those gates.
|
|
|
|
Repair responses should remain complete extraction replacements rather than
|
|
patches. Diagnostics must be treated as bounded data, particularly if a future
|
|
LLM-backed validator can contribute their text.
|
|
|
|
## Resolved Scope Decisions
|
|
|
|
- Catalog overlays are part of the immediate implementation, not a later
|
|
extension.
|
|
- Prompt grounding includes canonical spell names only. Aliases participate
|
|
only in deterministic lookup and validation.
|
|
- Measurements and decision thresholds for feedback-aware repair and semantic
|
|
LLM validation are deferred future work and do not block the baseline.
|