Improve D&D validation reliability

docs/roadmap/dnd-validation-reliability.md

@@ -0,0 +1,240 @@
+# D&D Validation Reliability Hardening
+
+## Purpose
+
+Improve the correctness and efficiency of D&D validation retries by giving the
+producer complete, contextual, semantically useful correction guidance; moving
+simple closed-value checks into compatible LLM-facing schemas; and removing
+duplicated source-range containment logic. The work should make one retry more
+likely to repair every detectable defect without weakening deterministic
+validation or exposing internal identifiers to a model.
+
+## Current State And Findings
+
+The validation architecture is coherent and the major safety boundaries are in
+place. Validators run through the framework-managed retry protocol, model-facing
+guidance is distinct from reason codes and operator diagnostics, schemas use
+provider-compatible required fields, and D&D extraction adapters canonically
+order safely resolvable source-reference endpoints before validation. The
+review nevertheless found four opportunities to make that architecture more
+consistent and effective.
+
+### Generic correction guidance
+
+Most rejecting D&D validators produce detailed operator-facing messages but
+pair them with a fixed, general correction sentence. The framework correctly
+constructs retry text from `CorrectionGuidance` alone, so the model does not see
+the internal diagnostic—and therefore often does not learn which records were
+wrong or how each record must change. Item-occurrence shape validation and the
+combat-semantics validator already demonstrate the intended contextual pattern.
+
+Passing the operator message through unchanged is not an acceptable fix.
+Operator messages may contain Go paths, array indexes that do not reliably map
+back to canonicalized model output, reason codes, validator terminology, or
+opaque application identities. Those details are useful for diagnosis but are
+not appropriate model context.
+
+### Missing closed-value schema constraints
+
+Four private LLM response schemas describe categorical fields as unconstrained
+strings even though their prompts, durable artifact schemas, and deterministic
+validators define closed value sets:
+
+- `combat-turns`: `turn_kind`;
+- `enemy-events`: `kind`;
+- `npc-occurrences`: `kind`; and
+- `scene-descriptions`: `kind`.
+
+This defers an inexpensive structural check until the outer semantic-validation
+loop. Item and location occurrences already use the preferable private-schema
+enum pattern.
+
+### First-defect validation
+
+Spell shape validation returns after the first invalid cast field, and enemy
+engagement validation returns after the first duplicated engagement subject.
+When multiple defects exist, each retry can therefore reveal only one of them.
+That needlessly consumes the bounded producer-attempt budget and makes
+correction less reliable for smaller models.
+
+### Duplicated source-range containment
+
+Nine extraction source-reference validators independently determine whether a
+citation belongs to the current chunk. Five validate every source unit spanned
+by a reference, while four check only the endpoints. Those behaviors are
+equivalent for today's contiguous materialized chunks, but the duplicated
+implementations and different apparent contracts create a drift risk.
+Scene-description validation has a separate exact-one-scene range contract and
+should remain specialized.
+
+## Target End State
+
+- Every producer-correctable D&D rejection supplies bounded contextual guidance
+  that identifies all affected records using transcript-grounded names, source
+  ranges, and response fields as appropriate, explains each semantic defect,
+  and requests one complete corrected replacement.
+- Model-facing guidance never contains reason codes, validator keys, Go field
+  paths, opaque or hash-derived entity IDs, or raw internal error text.
+  Operator diagnostics remain detailed and separately available.
+- Every detectable issue in a candidate is collected in one validator pass,
+  subject to the repository's established diagnostic bounds. A retry is not
+  spent merely to reveal the next defect.
+- Closed categorical fields are constrained by compatible enums in the private
+  LLM schemas. Deterministic typed validators retain the same rules as
+  defense-in-depth and as protection for non-LLM producers and later stages.
+- One D&D-owned helper implements full-span chunk containment for extraction
+  evidence. All applicable source-reference validators use it and no
+  module-local endpoint-only or coverage implementation remains.
+- Validator results remain immutable, deterministic, bounded, and suitable for
+  the existing feedback-aware replacement-request protocol. Retry budgets,
+  warning policy, durable artifact schemas, and accepted output semantics do
+  not change.
+
+## Required Work
+
+### Contextual correction guidance
+
+Audit each rejecting D&D validator according to the stage that can actually
+correct its result:
+
+- Extraction validators should build model-facing issue descriptions from the
+  candidate and its transcript context. Shape, registry-membership, identity,
+  catalog, engagement, and source-reference validators should name the
+  contextual artifact and cited range where useful, state the invalid value or
+  relationship in plain language, and state the required replacement shape.
+- LLM-backed NPC, item, and location registry reconciliation should continue to
+  translate its typed proposal issues through the shared category-to-prose
+  renderer, including any domain-specific supplement.
+- Normalization-only deterministic invariant validators should retain useful
+  operator diagnostics. They should not claim that a producer can directly
+  repair deterministically derived ordering, identity, or normalization state.
+  If such a rejection can reach a feedback-capable producer, guidance must be
+  expressed only in terms of the source candidate that producer controls.
+- LLM-backed semantic validators may continue using their verdict explanation
+  when it is bounded and semantically meaningful.
+
+Introduce a small D&D-shared diagnostic utility only for demonstrated common
+mechanics such as stable grouping, de-duplication, contextual source-range
+rendering, and bounded correction aggregation. Domain validators must continue
+to own the meaning and prose of their rules. The generic pipeline must not gain
+D&D knowledge, and operator `Message` values must never be mechanically copied
+into `CorrectionGuidance`.
+
+Update validator policy fingerprints wherever correction behavior changes so
+checkpoints created under generic feedback are not reused as though the policy
+were identical.
+
+### Private-schema enum guardrails
+
+Add the existing supported value sets to the four private LLM schemas:
+
+- `assets/dnd/combat-turns/schemas/dnd_combat_turns_llm.v1.json`;
+- `assets/dnd/enemy-events/schemas/dnd_enemy_events_llm.v1.json`;
+- `assets/dnd/npc-occurrences/schemas/dnd_npc_occurrences_llm.v1.json`; and
+- `assets/dnd/scene-descriptions/schemas/dnd_scene_descriptions_llm.v1.json`.
+
+Use the exact values owned by the corresponding durable contract and typed
+domain constants. Keep all object properties required, retain current nullable
+types where present, and do not introduce `uniqueItems`, optional properties,
+or conditional schema logic. Revise schema tests that currently accept unknown
+values, and rely on computed asset fingerprints to invalidate incompatible
+LLM-output checkpoints. Keep deterministic enum validation in place.
+
+### Complete per-attempt issue collection
+
+Refactor spell shape and enemy engagement validation to inspect the complete
+candidate and collect every detectable violation before returning. Build the
+operator diagnostic and contextual correction request from the same evaluated
+issue set while preserving their different audiences. De-duplicate repeated
+semantic instructions, retain enough contextual identification for every
+affected record, and use the established bounded diagnostic behavior rather
+than an unbounded error string.
+
+Review the surrounding D&D validators while applying the contextual-guidance
+change. Remove any additional accidental early exits that prevent independent
+candidate defects from being reported together, but retain immediate returns
+for request-level prerequisites whose absence makes further inspection unsafe
+or meaningless.
+
+### Shared full-span chunk containment
+
+Add one helper under `internal/modules/dnd/shared` that determines whether a
+source reference's complete document-ordered span is contained by a chunk. The
+helper must:
+
+- validate source identity;
+- resolve endpoints through `source.DocumentIndex` rather than numeric-ID
+  assumptions;
+- require every unit in the inclusive span to be present in the chunk;
+- handle nil or unresolved inputs without panic;
+- avoid mutating the source, chunk, or reference; and
+- leave source-reference validity and error wording to the consuming validator.
+
+Use the helper from the spell, NPC-registry, NPC-occurrence, item-registry,
+item-occurrence, location-registry, location-occurrence, combat-turn, and
+enemy-event source-reference validators. Keep the scene-description exact-range
+validator separate because it enforces a materially different contract.
+
+## Testing And Documentation
+
+Add lean offline behavioral coverage at the narrowest stable boundary:
+
+- correction tests should prove that multiple contextual defects produce one
+  actionable, bounded request and that internal identifiers and diagnostic
+  syntax are absent; they should not snapshot exact prose or message length;
+- schema tests should prove rejection of representative unsupported categorical
+  values and acceptance of the supported sets without duplicating every
+  provider behavior;
+- spell-shape and engagement tests should prove that independent defects are
+  reported together;
+- the shared containment helper should own the full case matrix, including
+  partial spans, non-monotonic unit IDs, wrong sources, unresolved endpoints,
+  and nil inputs; consuming validators need only enough coverage to prove they
+  use the common policy; and
+- existing validator tests should be simplified when the shared helper makes
+  module-local cases redundant.
+
+Update `docs/internal/dnd.md` to document the durable conventions for
+contextual correction guidance, complete issue collection, private-schema enum
+guardrails, and shared full-span extraction containment. Update
+`docs/internal/pipeline.md` only if framework behavior changes; the intended
+work applies its existing contract and should normally require only a link or
+no change. No new ADR is required because ADR-0014 and the architecture policy
+already decide the separation between semantic guidance and internal
+diagnostics. Create an ADR only if implementation requires changing that
+framework-level decision.
+
+## Non-Goals
+
+- Passing operator diagnostics or internal errors directly to an LLM.
+- Exposing durable IDs, hashes, reason codes, validator keys, or raw provider
+  responses in correction text.
+- Changing stage retry counts, PromptKit structural-repair budgets, validator
+  failure policy, or warning classification.
+- Silently repairing domain-semantic defects or weakening deterministic
+  validators after adding schema guardrails.
+- Adding provider-sensitive schema constructs beyond simple enums.
+- Moving D&D-specific behavior into the generic pipeline framework.
+- Generalizing the scene-description exact-range contract into the shared
+  extraction containment helper.
+
+## Acceptance Criteria
+
+- Every producer-correctable D&D rejection reviewed in this work gives the next
+  attempt all currently detectable, actionable corrections in contextual prose.
+- No model-facing correction request contains an opaque entity ID, hash,
+  validator key, reason code, Go-style field path, or unfiltered operator error.
+- Spell shape and enemy engagement validation aggregate independent defects in
+  one pass and keep outputs immutable.
+- The four private LLM schemas reject unsupported categorical values, retain
+  provider-compatible required-only shapes, and leave durable v1 contracts
+  unchanged.
+- All nine applicable extraction source-reference validators use one D&D-shared
+  full-span containment policy; scene descriptions retain their specialized
+  exact-range check.
+- Validator and asset fingerprints change wherever their effective policy
+  changes, preventing reuse of stale checkpoints.
+- Canonical internal documentation records the conventions future D&D
+  validators must follow.
+- Focused tests, `go test ./...`, `go vet ./...`, and
+  `go build ./cmd/notarius` pass in a supported development environment.