Document combat semantics validator evaluation

This commit is contained in:
2026-08-28 00:23:50 +00:00
parent 9cb7462800
commit 22e6caa2a0
5 changed files with 92 additions and 1 deletions

View File

@@ -504,7 +504,24 @@ Available validator keys are:
| Item occurrences | **extract/dnd/item-occurrences/shape**, **extract/dnd/item-occurrences/registry**, **extract/dnd/item-occurrences/source_refs**, **extract/dnd/item-occurrences/source_relatedness**, **normalize/dnd/item-occurrences/invariants** |
| Item registry | **extract/dnd/item-registry/shape**, **extract/dnd/item-registry/source_refs**, **extract/dnd/item-registry/source_relatedness**, **normalize/dnd/item-registry/identity** |
| NPC occurrences | **extract/dnd/npc-occurrences/shape**, **extract/dnd/npc-occurrences/registry**, **extract/dnd/npc-occurrences/source_refs**, **extract/dnd/npc-occurrences/source_relatedness**, **normalize/dnd/npc-occurrences/invariants** |
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **normalize/dnd/scene-descriptions/invariants** |
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **extract/dnd/scene-descriptions/combat_semantics** (LLM-backed, opt-in), **normalize/dnd/scene-descriptions/invariants** |
`extract/dnd/scene-descriptions/combat_semantics` is not in a production default chain. To opt in, replace the scene extractor validator chain with the current ordered chain plus the semantic validator last, and set a positive producer retry budget if a rejection should request a corrected scene:
~~~yaml
extract:
module: dnd/scene-descriptions
retries: 1
validators:
- generic/valid_json
- extract/dnd/scene-descriptions/shape
- extract/dnd/scene-descriptions/source_refs
- generic/valid_json_schema
- extract/dnd/scene-descriptions/source_relatedness
- extract/dnd/scene-descriptions/combat_semantics
~~~
An override replaces, rather than extends, the default chain. See [Module Bindings And Validators](#module-bindings-and-validators) for binding, profile, repair, retry, and failure-policy rules.
| Enemy events | **extract/dnd/enemy-events/shape**, **extract/dnd/enemy-events/engagements**, **extract/dnd/enemy-events/source_refs**, **extract/dnd/enemy-events/source_relatedness**, **normalize/dnd/enemy-events/invariants** |
| Location registry | **extract/dnd/location-registry/shape**, **extract/dnd/location-registry/source_refs**, **extract/dnd/location-registry/source_relatedness**, **normalize/dnd/location-registry/identity** |
| Location occurrences | **extract/dnd/location-occurrences/shape**, **extract/dnd/location-occurrences/registry**, **extract/dnd/location-occurrences/source_refs**, **extract/dnd/location-occurrences/source_relatedness**, **normalize/dnd/location-occurrences/invariants** |

View File

@@ -134,6 +134,19 @@ relatedness validators report advisory evidence concerns. The configured order
is documented in
[Configuration](../config.md#production-validator-keys-and-default-chains).
The optional `extract/dnd/scene-descriptions/combat_semantics` validator is the
D&D family's LLM-backed review of only combat versus non-combat classification.
It selects the shared combat-policy prompt fragment, receives the proposed kind
and current chunk, and maps its verdict deterministically into producer
guidance. It does not assess titles, summaries, non-combat subtype, or scene
boundaries; deferred boundary-coherence review remains separate. It is opt-in;
[Configuration](../config.md) owns selection and retry/failure behavior.
Before proposing default-chain inclusion, run a deliberate provider evaluation
against the synthetic corpus and record false acceptance, false rejection,
producer-correction success, added calls, latency, and token use. Default tests
remain offline and do not measure provider quality.
Every D&D rejection describes the correction in transcript-grounded domain
terms, using contextual names, artifact fields, and source segment ranges when
useful. The guidance must not ask the model to reproduce durable entity IDs,

View File

@@ -374,6 +374,8 @@ This stage is small enough for one implementation prompt.
## Stage 6: Add Evaluation Material And Documentation
✅ Complete
### Goal
Provide accurate opt-in configuration guidance and a bounded human-reviewed