Document combat semantics validator evaluation
This commit is contained in:
@@ -504,7 +504,24 @@ Available validator keys are:
|
||||
| Item occurrences | **extract/dnd/item-occurrences/shape**, **extract/dnd/item-occurrences/registry**, **extract/dnd/item-occurrences/source_refs**, **extract/dnd/item-occurrences/source_relatedness**, **normalize/dnd/item-occurrences/invariants** |
|
||||
| Item registry | **extract/dnd/item-registry/shape**, **extract/dnd/item-registry/source_refs**, **extract/dnd/item-registry/source_relatedness**, **normalize/dnd/item-registry/identity** |
|
||||
| NPC occurrences | **extract/dnd/npc-occurrences/shape**, **extract/dnd/npc-occurrences/registry**, **extract/dnd/npc-occurrences/source_refs**, **extract/dnd/npc-occurrences/source_relatedness**, **normalize/dnd/npc-occurrences/invariants** |
|
||||
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **normalize/dnd/scene-descriptions/invariants** |
|
||||
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **extract/dnd/scene-descriptions/combat_semantics** (LLM-backed, opt-in), **normalize/dnd/scene-descriptions/invariants** |
|
||||
|
||||
`extract/dnd/scene-descriptions/combat_semantics` is not in a production default chain. To opt in, replace the scene extractor validator chain with the current ordered chain plus the semantic validator last, and set a positive producer retry budget if a rejection should request a corrected scene:
|
||||
|
||||
~~~yaml
|
||||
extract:
|
||||
module: dnd/scene-descriptions
|
||||
retries: 1
|
||||
validators:
|
||||
- generic/valid_json
|
||||
- extract/dnd/scene-descriptions/shape
|
||||
- extract/dnd/scene-descriptions/source_refs
|
||||
- generic/valid_json_schema
|
||||
- extract/dnd/scene-descriptions/source_relatedness
|
||||
- extract/dnd/scene-descriptions/combat_semantics
|
||||
~~~
|
||||
|
||||
An override replaces, rather than extends, the default chain. See [Module Bindings And Validators](#module-bindings-and-validators) for binding, profile, repair, retry, and failure-policy rules.
|
||||
| Enemy events | **extract/dnd/enemy-events/shape**, **extract/dnd/enemy-events/engagements**, **extract/dnd/enemy-events/source_refs**, **extract/dnd/enemy-events/source_relatedness**, **normalize/dnd/enemy-events/invariants** |
|
||||
| Location registry | **extract/dnd/location-registry/shape**, **extract/dnd/location-registry/source_refs**, **extract/dnd/location-registry/source_relatedness**, **normalize/dnd/location-registry/identity** |
|
||||
| Location occurrences | **extract/dnd/location-occurrences/shape**, **extract/dnd/location-occurrences/registry**, **extract/dnd/location-occurrences/source_refs**, **extract/dnd/location-occurrences/source_relatedness**, **normalize/dnd/location-occurrences/invariants** |
|
||||
|
||||
@@ -134,6 +134,19 @@ relatedness validators report advisory evidence concerns. The configured order
|
||||
is documented in
|
||||
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
||||
|
||||
The optional `extract/dnd/scene-descriptions/combat_semantics` validator is the
|
||||
D&D family's LLM-backed review of only combat versus non-combat classification.
|
||||
It selects the shared combat-policy prompt fragment, receives the proposed kind
|
||||
and current chunk, and maps its verdict deterministically into producer
|
||||
guidance. It does not assess titles, summaries, non-combat subtype, or scene
|
||||
boundaries; deferred boundary-coherence review remains separate. It is opt-in;
|
||||
[Configuration](../config.md) owns selection and retry/failure behavior.
|
||||
|
||||
Before proposing default-chain inclusion, run a deliberate provider evaluation
|
||||
against the synthetic corpus and record false acceptance, false rejection,
|
||||
producer-correction success, added calls, latency, and token use. Default tests
|
||||
remain offline and do not measure provider quality.
|
||||
|
||||
Every D&D rejection describes the correction in transcript-grounded domain
|
||||
terms, using contextual names, artifact fields, and source segment ranges when
|
||||
useful. The guidance must not ask the model to reproduce durable entity IDs,
|
||||
|
||||
@@ -374,6 +374,8 @@ This stage is small enough for one implementation prompt.
|
||||
|
||||
## Stage 6: Add Evaluation Material And Documentation
|
||||
|
||||
✅ Complete
|
||||
|
||||
### Goal
|
||||
|
||||
Provide accurate opt-in configuration guidance and a bounded human-reviewed
|
||||
|
||||
Reference in New Issue
Block a user