notarius/docs/roadmap/combat-scene-validation.md

# D&D Combat Scene Semantic Validation

## Purpose

Add an optional LLM-backed validator for `dnd/scene-descriptions` that checks
whether the proposed scene kind correctly represents substantive active combat
in the current transcript chunk. The validator will improve the reliability of
the existing downstream rule that combat turns and enemy events run only for
scenes classified as `combat`, without moving scene classification into the
chunker or broadening the durable scene-description contract.

This roadmap defines the intended final state and the code and documentation
changes required to reach it.

## Current State

- `dnd/scenes` produces a complete, gap-free scene plan containing source
  ranges. It does not classify those ranges.
- `dnd/scene-descriptions` processes one accepted chunk at a time and produces
  exactly one scene with a deterministic chunk ID and source range plus the
  model-supplied `kind`, `title`, and `summary`.
- The supported scene kinds are `combat`, `narrative`, `recap`, and `meta`.
- The production extract chain validates shape, source identity, durable JSON
  schema, and advisory lexical relatedness. None of those checks can decide
  whether active combat was classified correctly.
- Combat-turn and enemy-event extraction already require an exact matching
  scene description whose kind is `combat`. A missing, ambiguous, or
  non-combat classification does not run those LLM extractors.
- The validation framework already supports typed LLM-backed validators,
  PromptKit structured-output repair, bounded validator execution retries,
  aggregated semantic correction guidance, and configurable terminal handling
  of validator rejection or execution failure.
- There are currently no production LLM-backed validators. The new module will
  establish the first concrete D&D implementation of that existing framework
  capability.

## User Intent And Policy

The validator exists to catch both consequential classification errors:

1. a scene containing substantive active combat is assigned a non-combat kind;
   and
2. a scene without substantive active combat is assigned `combat`.

“Substantive active combat” means that an in-session encounter is materially
organized around participants taking or resolving hostile actions, such as
initiative or turn exchanges, attacks, combat spells, damage, saves, movement,
or similarly sustained conflict. The following do not establish active combat
by themselves:

- planning or preparing for a possible fight;
- threats, hostile dialogue, or a tense confrontation;
- immediate aftermath, looting, healing, or discussion of a completed fight;
- a recap or in-world recollection of earlier combat; or
- out-of-character rules discussion without active encounter play.

A chunk with substantive active combat remains combat when it also contains
brief setup, rules clarification, interruption, phase transition, or immediate
aftermath. The validator judges whether the supplied chunk's proposed combat
status is supported; it does not redesign the scene boundary or review the
quality of the title and summary.

The model must receive only semantically useful material: the transcript chunk,
the proposed scene kind, and the combat decision policy. Transcript unit IDs
may remain visible as part of the ordinary source presentation, but the model
must not be asked to reproduce them or the chunk ID, source range, hashes,
validator keys, reason codes, or any other opaque application identifier.
Application code owns all mapping, provenance, and correction metadata
deterministically.

## Target End State

### Validator Module

Add a typed validator package at
`internal/modules/dnd/validate/scenedescriptions/combat_semantics` with the
registered key
`extract/dnd/scene-descriptions/combat_semantics`. The module will:

- register for `dnd.SceneDescriptionListKind`;
- declare `contracts.ExecutionClassLLMBacked`;
- accept no module-specific options and reject unknown options;
- require an extract-stage request containing one valid scene and the current
  transcript chunk;
- call the run-scoped scheduled `StructuredLLMClient` supplied through
  `pipeline.BuildRequest`;
- forward the resolved validator `llm_profile`, session ID, and structured
  output repair override without inventing a separate concurrency or provider
  path;
- expose prompt/schema/policy fingerprints through the existing manifest and
  checkpoint metadata conventions; and
- return an ordinary `contracts.ValidationResult` whose semantic meaning is
  fully owned by the module while terminal disposition remains framework and
  pipeline policy.

The module is intentionally extract-stage-specific. If it is configured for a
merged or normalized value, receives no current chunk, receives anything other
than exactly one scene, or cannot establish the required input invariants, it
must return a contextual execution error rather than make a semantic judgment
against incomplete or ambiguous material.

### LLM Decision Contract

The private LLM response will contain exactly two required fields:

- `verdict`, one of `approved`, `combat_should_be_added`, or
  `combat_should_be_removed`; and
- `explanation`, a concise, transcript-grounded explanation of the decision.

The JSON schema must reject unknown fields and require every declared field.
It must contain no optional properties and no `uniqueItems` keyword. The prompt
must force a best judgment and must not request a confidence score or an
“uncertain” verdict; model-estimated uncertainty is a quality signal, not a
process warning.

The adapter will validate verdict consistency deterministically:

- `approved` accepts either a supported combat classification or a supported
  non-combat classification;
- `combat_should_be_added` is valid only when the proposed kind is not
  `combat`; and
- `combat_should_be_removed` is valid only when the proposed kind is
  `combat`.

An inconsistent verdict, blank or over-limit explanation, malformed final
response, missing prompt input, or completion error is a validator execution
failure. PromptKit may repair structurally invalid output within its configured
budget, and the framework may re-execute the validator within its distinct
validator retry budget. The validator will not create a recursive semantic
validation or feedback loop for its own response.

On semantic mismatch, application code will assign stable internal reason
codes and build bounded, actionable correction guidance. The guidance will
state whether the corrected scene must be `combat` or must use the appropriate
non-combat kind and will include the model's safe, bounded evidence explanation.
It will not expose validator keys or reason codes to the producer. The existing
producer retry mechanism will append that guidance to the exact defective
scene-description response and request one complete replacement.

### Prompt Assets

Store the validator's LLM-facing assets under
`assets/dnd/scene-descriptions/validate/combat-semantics/`, using the existing
content-only root assets package and module-owned prompt registration pattern.
The asset set will include a consistently named `prompt.yaml`, one concise
module instruction file, and one private v1 response schema.

The prompt declaration will use `dnd.scene_descriptions.validate_combat` as
its prompt ID and `dnd-extraction` as its default profile. Stable shared and
module policy messages will precede variable inputs so PromptKit backends can
reuse identical prefixes. The prompt will reuse the shared D&D system and
chunk-transcript assets rather than copying their contents. A small local input
asset will present only the proposed scene kind; the complete scene artifact,
its deterministic ID, and its source reference will not be rendered.

Factor the combat/non-combat decision policy currently embedded in the
scene-description extractor instructions into one `common-dnd-` shared asset
selected by both the extractor and validator manifests. Keep the non-combat
`narrative`, `recap`, and `meta` selection rules local to the extractor. The
shared combat policy must be byte-identical for both consumers; do not create
two near-duplicate definitions. The manifests, asset hashes, and prompt-cache
tests must make the resulting ownership and message ordering explicit without
using brittle exact-length assertions.

### Registration And Configuration

Register the validator builder and its prompt/schema assets through the D&D
registrar. Catalog inspection must report its typed artifact kind and
`llm_backed` execution class.

The validator will be selectable in an extract validator override, but it will
not be added to the production default chain by this feature. An operator who
opts in must place it after the existing deterministic shape, source-reference,
and durable-schema checks so malformed candidates are rejected before a paid
LLM call. It should be last in the chain after advisory relatedness unless
evaluation demonstrates a concrete reason to change that order.

The validator uses the ordinary profile precedence and validator binding
fields already documented by Notarius. It introduces no new configuration
field, retry budget, failure policy, scheduler, or environment variable.
Validator rejection and execution failure continue to follow the configured
stage validation policy. Under the application defaults, exhausted semantic
rejection fails the run, while exhausted validator execution failure may
advance with an actionable process warning and incomplete-validation
provenance.

### Default-Chain Promotion Gate

The final state for this roadmap is a production-quality, documented,
operator-selectable validator that remains opt-in. It must not enter the
registered production default chain until provider-backed evaluation shows
that its accuracy, retry value, latency, and token cost justify enabling an
additional model call for every scene-description chunk.

Promotion, if later selected, requires a separate deliberate change to the
default chain and maintained configuration expectations. It does not require a
new architecture decision so long as the validator remains at the
scene-description extract stage.

## Required Code Changes

The implementation must make the following coherent changes:

- add the new typed validator, strict option decoder, LLM response model,
  response interpretation, error wrapping, fingerprints, and registration;
- add and register its embedded prompt, instruction, input, and response-schema
  assets;
- factor the combat classification policy into a shared prompt fragment if
  doing so is necessary to keep the extractor and validator language exactly
  identical;
- update D&D registrar tests, prompt-asset registration tests, prompt-cache
  ordering tests, and catalog/spec assertions for the new selectable key;
- add focused validator tests with a fake structured LLM client for approval,
  both rejection directions, inconsistent verdicts, malformed or failed
  completions, input preconditions, profile/session/repair forwarding, bounded
  explanations, correction guidance, and immutable candidate handling;
- add an integration test proving that a semantic rejection from this
  validator supplies useful correction guidance to a retry-capable
  `dnd/scene-descriptions` producer and that the corrected candidate is
  revalidated before acceptance;
- retain and extend downstream contract coverage proving that combat-turn and
  enemy-event extraction still runs only for an exact accepted `combat` scene;
  and
- update current configuration and D&D internal documentation to list the key,
  describe its opt-in status and recommended chain position, explain its
  process-failure behavior, and distinguish it from a future chunk-boundary
  validator.

No durable artifact schema or integration contract needs a version change.
The validator produces validation provenance and correction behavior, not a
new published artifact.

## Evaluation Requirements

Maintain a small human-reviewed evaluation set that exercises at least:

- ordinary active combat and initiative-like exchanges;
- combat preceded by setup or followed by immediate aftermath;
- multi-phase encounters and brief rules interruptions;
- planning, threats, hostile dialogue, and tense confrontations without active
  combat;
- aftermath, looting, and healing after combat has ended;
- prior-session recap and in-world recollection of combat; and
- sustained out-of-character combat rules discussion.

Default automated tests must remain deterministic, offline, and provider-free.
They should test prompt inputs, response interpretation, retry integration, and
failure policy with fakes rather than asserting exact natural-language output.
Provider-backed evaluation is an explicit manual or opt-in task. Record false
acceptance, false rejection, successful producer correction, added calls,
latency, and token use before proposing default-chain promotion.

## Documentation And Architecture Record

Update `docs/config.md` as the canonical owner of the selectable validator key,
binding behavior, and chain placement. Update `docs/internal/dnd.md` as the
canonical owner of the D&D-specific decision policy, prompt ownership, and
module boundary. Link rather than duplicate the generic retry, diagnostic, and
failure-policy contracts owned by their existing architecture and operator
documentation.

No new ADR is required for this scope. It applies the accepted generic
validation and feedback-aware retry architecture without changing stage
ownership or a durable contract. Create or supersede an ADR only if later work
moves classification into `dnd/scenes`, adds annotations to the chunk plan, or
otherwise transfers ownership across pipeline stages.

## Out Of Scope

- changing scene boundaries or deciding whether one combat was fragmented
  across multiple chunks;
- adding an LLM-backed chunk-plan validator;
- moving scene kind into chunk-plan annotations;
- changing the scene-description, combat-turn, or enemy-event durable schemas;
- replacing the deterministic combat-only downstream extraction gate;
- validating title or summary quality with this module;
- adding confidence scores or surfacing model uncertainty as warnings;
- adding a validator-specific concurrency or retry subsystem; and
- enabling the validator in the production default chain before evaluation.

## Acceptance Criteria

- The new LLM-backed validator is registered for scene-description artifacts
  and can be selected only through the existing validator binding mechanism.
- It reviews exactly one extract-stage scene against the corresponding
  transcript chunk and detects both missing and unsupported `combat` tags.
- Its response contract and candidate presentation do not ask the model to
  reproduce transcript unit IDs, chunk IDs, source ranges, hashes, validator
  keys, or reason codes.
- Its strict structured schema has only required fields, rejects unknown
  fields, and avoids unsupported JSON Schema keywords.
- Rejections produce bounded, transcript-grounded corrective guidance that the
  existing scene-description producer retry can use.
- Validator contract or execution failures remain distinct from semantic
  rejection and follow existing configured policy.
- The existing deterministic validator chain and downstream exact-combat gate
  remain intact.
- Prompt definitions share identical combat policy content rather than
  maintaining near-duplicate instructions.
- Automated coverage is deterministic and provider-free; an explicit
  human-reviewed evaluation protocol exists for later default-chain review.
- Current documentation describes implemented selection and behavior without
  claiming that the validator is enabled by default.