|
|
|
|
@@ -153,28 +153,32 @@ is documented in
|
|
|
|
|
|
|
|
|
|
The optional `extract/dnd/scene-descriptions/combat_semantics` validator is the
|
|
|
|
|
D&D family's LLM-backed review of only combat versus non-combat classification.
|
|
|
|
|
It selects the shared combat-policy prompt fragment, receives the proposed kind
|
|
|
|
|
and current chunk, and maps its verdict deterministically into producer
|
|
|
|
|
guidance. It does not assess titles, summaries, non-combat subtype, or scene
|
|
|
|
|
boundaries; deferred boundary-coherence review remains separate. It is opt-in;
|
|
|
|
|
It selects the shared combat-policy prompt fragment and asks the model to
|
|
|
|
|
classify the current chunk independently as `combat` or `non_combat` without
|
|
|
|
|
receiving the proposed scene kind. Deterministic code compares that
|
|
|
|
|
classification with the proposed kind and either approves it or produces the
|
|
|
|
|
appropriate correction guidance. This keeps every schema-valid classification
|
|
|
|
|
interpretable and avoids anchoring the reviewer on the producer's answer. It
|
|
|
|
|
does not assess titles, summaries, non-combat subtype, or scene boundaries;
|
|
|
|
|
deferred boundary-coherence review remains separate. It is opt-in;
|
|
|
|
|
[Configuration](../config.md) owns selection and retry/failure behavior.
|
|
|
|
|
|
|
|
|
|
### Combat-semantics provider evaluation
|
|
|
|
|
|
|
|
|
|
The human-reviewed corpus at
|
|
|
|
|
`internal/modules/dnd/validate/scenedescriptions/combat_semantics/testdata/evaluation_cases.json`
|
|
|
|
|
owns the proposed kind, expected verdict, and reviewer rationale for each
|
|
|
|
|
synthetic case. Its package test validates the fixture contract only. Provider
|
|
|
|
|
evaluation remains an explicit maintainer operation and must not be added to
|
|
|
|
|
the default offline test suite.
|
|
|
|
|
owns the proposed kind, expected combat classification, and reviewer rationale
|
|
|
|
|
for each synthetic case. Its package test validates the fixture contract only.
|
|
|
|
|
Provider evaluation remains an explicit maintainer operation and must not be
|
|
|
|
|
added to the default offline test suite.
|
|
|
|
|
|
|
|
|
|
Use the following protocol before proposing default-chain inclusion:
|
|
|
|
|
|
|
|
|
|
1. Record the Notarius commit, prompt and schema fingerprints, provider, model,
|
|
|
|
|
profile settings, reasoning effort, structural-repair setting, number of
|
|
|
|
|
repetitions, and evaluation date before collecting results. Do not revise
|
|
|
|
|
expected verdicts merely to agree with provider output; a substantive corpus
|
|
|
|
|
correction requires independent human review.
|
|
|
|
|
expected classifications merely to agree with provider output; a substantive
|
|
|
|
|
corpus correction requires independent human review.
|
|
|
|
|
2. Exercise the production validator construction and prompt assets from an
|
|
|
|
|
explicitly invoked, disposable evaluation driver or test in the validator
|
|
|
|
|
package. For each corpus case, construct transcript source units from the
|
|
|
|
|
@@ -186,12 +190,14 @@ Use the following protocol before proposing default-chain inclusion:
|
|
|
|
|
validator at extract stage through the production registry and scheduled LLM
|
|
|
|
|
client. Do not commit provider credentials, generated source material, or an
|
|
|
|
|
always-on live test.
|
|
|
|
|
3. Compare the resulting approval or reason code with `expected_verdict`.
|
|
|
|
|
3. Compare the model classification with `expected_classification`, then verify
|
|
|
|
|
that its deterministic comparison with `proposed_kind` yields the expected
|
|
|
|
|
approval or rejection direction.
|
|
|
|
|
Record an unexpected approval of an expected rejection as a false
|
|
|
|
|
acceptance, an unexpected rejection of an expected approval as a false
|
|
|
|
|
rejection, the opposite rejection direction separately, and any validator
|
|
|
|
|
execution failure separately from semantic accuracy. Retain per-case results
|
|
|
|
|
so repeated trials and systematic failure modes remain visible.
|
|
|
|
|
rejection, and any validator execution failure separately from semantic
|
|
|
|
|
accuracy. Retain per-case results so repeated trials and systematic failure
|
|
|
|
|
modes remain visible.
|
|
|
|
|
4. Evaluate producer correction separately with representative complete
|
|
|
|
|
scene-description runs configured as shown in
|
|
|
|
|
[Configuration](../config.md#production-validator-keys-and-default-chains).
|
|
|
|
|
@@ -204,10 +210,10 @@ Use the following protocol before proposing default-chain inclusion:
|
|
|
|
|
from the debug attempt records. Compare these values with an otherwise
|
|
|
|
|
identical run whose scene-description chain omits the semantic validator.
|
|
|
|
|
|
|
|
|
|
The default-chain review must consider false acceptance, false rejection,
|
|
|
|
|
wrong-direction rejection, execution failure, producer-correction success,
|
|
|
|
|
added calls, latency, and token use together. A structurally successful
|
|
|
|
|
provider run alone is not evidence that the validator should become a default.
|
|
|
|
|
The default-chain review must consider classification error, false acceptance,
|
|
|
|
|
false rejection, execution failure, producer-correction success, added calls,
|
|
|
|
|
latency, and token use together. A structurally successful provider run alone
|
|
|
|
|
is not evidence that the validator should become a default.
|
|
|
|
|
|
|
|
|
|
Every producer-correctable D&D rejection describes all currently detectable
|
|
|
|
|
corrections in transcript-grounded domain terms, using contextual names,
|
|
|
|
|
|