From 0f2b23dce14b4d5ac807a4034d2df226193a9732 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 27 Aug 2026 23:26:59 +0000 Subject: [PATCH] Plan combat scene validation --- docs/roadmap/combat-scene-validation.md | 303 +++++++++++++++ docs/roadmap/future.md | 43 +-- docs/roadmap/implementation.md | 488 ++++++++++++++++++++++++ 3 files changed, 793 insertions(+), 41 deletions(-) create mode 100644 docs/roadmap/combat-scene-validation.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/combat-scene-validation.md b/docs/roadmap/combat-scene-validation.md new file mode 100644 index 00000000..7c616adf --- /dev/null +++ b/docs/roadmap/combat-scene-validation.md @@ -0,0 +1,303 @@ +# D&D Combat Scene Semantic Validation + +## Purpose + +Add an optional LLM-backed validator for `dnd/scene-descriptions` that checks +whether the proposed scene kind correctly represents substantive active combat +in the current transcript chunk. The validator will improve the reliability of +the existing downstream rule that combat turns and enemy events run only for +scenes classified as `combat`, without moving scene classification into the +chunker or broadening the durable scene-description contract. + +This roadmap defines the intended final state and the code and documentation +changes required to reach it. + +## Current State + +- `dnd/scenes` produces a complete, gap-free scene plan containing source + ranges. It does not classify those ranges. +- `dnd/scene-descriptions` processes one accepted chunk at a time and produces + exactly one scene with a deterministic chunk ID and source range plus the + model-supplied `kind`, `title`, and `summary`. +- The supported scene kinds are `combat`, `narrative`, `recap`, and `meta`. +- The production extract chain validates shape, source identity, durable JSON + schema, and advisory lexical relatedness. None of those checks can decide + whether active combat was classified correctly. +- Combat-turn and enemy-event extraction already require an exact matching + scene description whose kind is `combat`. A missing, ambiguous, or + non-combat classification does not run those LLM extractors. +- The validation framework already supports typed LLM-backed validators, + PromptKit structured-output repair, bounded validator execution retries, + aggregated semantic correction guidance, and configurable terminal handling + of validator rejection or execution failure. +- There are currently no production LLM-backed validators. The new module will + establish the first concrete D&D implementation of that existing framework + capability. + +## User Intent And Policy + +The validator exists to catch both consequential classification errors: + +1. a scene containing substantive active combat is assigned a non-combat kind; + and +2. a scene without substantive active combat is assigned `combat`. + +“Substantive active combat” means that an in-session encounter is materially +organized around participants taking or resolving hostile actions, such as +initiative or turn exchanges, attacks, combat spells, damage, saves, movement, +or similarly sustained conflict. The following do not establish active combat +by themselves: + +- planning or preparing for a possible fight; +- threats, hostile dialogue, or a tense confrontation; +- immediate aftermath, looting, healing, or discussion of a completed fight; +- a recap or in-world recollection of earlier combat; or +- out-of-character rules discussion without active encounter play. + +A chunk with substantive active combat remains combat when it also contains +brief setup, rules clarification, interruption, phase transition, or immediate +aftermath. The validator judges whether the supplied chunk's proposed combat +status is supported; it does not redesign the scene boundary or review the +quality of the title and summary. + +The model must receive only semantically useful material: the transcript chunk, +the proposed scene kind, and the combat decision policy. Transcript unit IDs +may remain visible as part of the ordinary source presentation, but the model +must not be asked to reproduce them or the chunk ID, source range, hashes, +validator keys, reason codes, or any other opaque application identifier. +Application code owns all mapping, provenance, and correction metadata +deterministically. + +## Target End State + +### Validator Module + +Add a typed validator package at +`internal/modules/dnd/validate/scenedescriptions/combat_semantics` with the +registered key +`extract/dnd/scene-descriptions/combat_semantics`. The module will: + +- register for `dnd.SceneDescriptionListKind`; +- declare `contracts.ExecutionClassLLMBacked`; +- accept no module-specific options and reject unknown options; +- require an extract-stage request containing one valid scene and the current + transcript chunk; +- call the run-scoped scheduled `StructuredLLMClient` supplied through + `pipeline.BuildRequest`; +- forward the resolved validator `llm_profile`, session ID, and structured + output repair override without inventing a separate concurrency or provider + path; +- expose prompt/schema/policy fingerprints through the existing manifest and + checkpoint metadata conventions; and +- return an ordinary `contracts.ValidationResult` whose semantic meaning is + fully owned by the module while terminal disposition remains framework and + pipeline policy. + +The module is intentionally extract-stage-specific. If it is configured for a +merged or normalized value, receives no current chunk, receives anything other +than exactly one scene, or cannot establish the required input invariants, it +must return a contextual execution error rather than make a semantic judgment +against incomplete or ambiguous material. + +### LLM Decision Contract + +The private LLM response will contain exactly two required fields: + +- `verdict`, one of `approved`, `combat_should_be_added`, or + `combat_should_be_removed`; and +- `explanation`, a concise, transcript-grounded explanation of the decision. + +The JSON schema must reject unknown fields and require every declared field. +It must contain no optional properties and no `uniqueItems` keyword. The prompt +must force a best judgment and must not request a confidence score or an +“uncertain” verdict; model-estimated uncertainty is a quality signal, not a +process warning. + +The adapter will validate verdict consistency deterministically: + +- `approved` accepts either a supported combat classification or a supported + non-combat classification; +- `combat_should_be_added` is valid only when the proposed kind is not + `combat`; and +- `combat_should_be_removed` is valid only when the proposed kind is + `combat`. + +An inconsistent verdict, blank or over-limit explanation, malformed final +response, missing prompt input, or completion error is a validator execution +failure. PromptKit may repair structurally invalid output within its configured +budget, and the framework may re-execute the validator within its distinct +validator retry budget. The validator will not create a recursive semantic +validation or feedback loop for its own response. + +On semantic mismatch, application code will assign stable internal reason +codes and build bounded, actionable correction guidance. The guidance will +state whether the corrected scene must be `combat` or must use the appropriate +non-combat kind and will include the model's safe, bounded evidence explanation. +It will not expose validator keys or reason codes to the producer. The existing +producer retry mechanism will append that guidance to the exact defective +scene-description response and request one complete replacement. + +### Prompt Assets + +Store the validator's LLM-facing assets under +`assets/dnd/scene-descriptions/validate/combat-semantics/`, using the existing +content-only root assets package and module-owned prompt registration pattern. +The asset set will include a consistently named `prompt.yaml`, one concise +module instruction file, and one private v1 response schema. + +The prompt declaration will use `dnd.scene_descriptions.validate_combat` as +its prompt ID and `dnd-extraction` as its default profile. Stable shared and +module policy messages will precede variable inputs so PromptKit backends can +reuse identical prefixes. The prompt will reuse the shared D&D system and +chunk-transcript assets rather than copying their contents. A small local input +asset will present only the proposed scene kind; the complete scene artifact, +its deterministic ID, and its source reference will not be rendered. + +Factor the combat/non-combat decision policy currently embedded in the +scene-description extractor instructions into one `common-dnd-` shared asset +selected by both the extractor and validator manifests. Keep the non-combat +`narrative`, `recap`, and `meta` selection rules local to the extractor. The +shared combat policy must be byte-identical for both consumers; do not create +two near-duplicate definitions. The manifests, asset hashes, and prompt-cache +tests must make the resulting ownership and message ordering explicit without +using brittle exact-length assertions. + +### Registration And Configuration + +Register the validator builder and its prompt/schema assets through the D&D +registrar. Catalog inspection must report its typed artifact kind and +`llm_backed` execution class. + +The validator will be selectable in an extract validator override, but it will +not be added to the production default chain by this feature. An operator who +opts in must place it after the existing deterministic shape, source-reference, +and durable-schema checks so malformed candidates are rejected before a paid +LLM call. It should be last in the chain after advisory relatedness unless +evaluation demonstrates a concrete reason to change that order. + +The validator uses the ordinary profile precedence and validator binding +fields already documented by Notarius. It introduces no new configuration +field, retry budget, failure policy, scheduler, or environment variable. +Validator rejection and execution failure continue to follow the configured +stage validation policy. Under the application defaults, exhausted semantic +rejection fails the run, while exhausted validator execution failure may +advance with an actionable process warning and incomplete-validation +provenance. + +### Default-Chain Promotion Gate + +The final state for this roadmap is a production-quality, documented, +operator-selectable validator that remains opt-in. It must not enter the +registered production default chain until provider-backed evaluation shows +that its accuracy, retry value, latency, and token cost justify enabling an +additional model call for every scene-description chunk. + +Promotion, if later selected, requires a separate deliberate change to the +default chain and maintained configuration expectations. It does not require a +new architecture decision so long as the validator remains at the +scene-description extract stage. + +## Required Code Changes + +The implementation must make the following coherent changes: + +- add the new typed validator, strict option decoder, LLM response model, + response interpretation, error wrapping, fingerprints, and registration; +- add and register its embedded prompt, instruction, input, and response-schema + assets; +- factor the combat classification policy into a shared prompt fragment if + doing so is necessary to keep the extractor and validator language exactly + identical; +- update D&D registrar tests, prompt-asset registration tests, prompt-cache + ordering tests, and catalog/spec assertions for the new selectable key; +- add focused validator tests with a fake structured LLM client for approval, + both rejection directions, inconsistent verdicts, malformed or failed + completions, input preconditions, profile/session/repair forwarding, bounded + explanations, correction guidance, and immutable candidate handling; +- add an integration test proving that a semantic rejection from this + validator supplies useful correction guidance to a retry-capable + `dnd/scene-descriptions` producer and that the corrected candidate is + revalidated before acceptance; +- retain and extend downstream contract coverage proving that combat-turn and + enemy-event extraction still runs only for an exact accepted `combat` scene; + and +- update current configuration and D&D internal documentation to list the key, + describe its opt-in status and recommended chain position, explain its + process-failure behavior, and distinguish it from a future chunk-boundary + validator. + +No durable artifact schema or integration contract needs a version change. +The validator produces validation provenance and correction behavior, not a +new published artifact. + +## Evaluation Requirements + +Maintain a small human-reviewed evaluation set that exercises at least: + +- ordinary active combat and initiative-like exchanges; +- combat preceded by setup or followed by immediate aftermath; +- multi-phase encounters and brief rules interruptions; +- planning, threats, hostile dialogue, and tense confrontations without active + combat; +- aftermath, looting, and healing after combat has ended; +- prior-session recap and in-world recollection of combat; and +- sustained out-of-character combat rules discussion. + +Default automated tests must remain deterministic, offline, and provider-free. +They should test prompt inputs, response interpretation, retry integration, and +failure policy with fakes rather than asserting exact natural-language output. +Provider-backed evaluation is an explicit manual or opt-in task. Record false +acceptance, false rejection, successful producer correction, added calls, +latency, and token use before proposing default-chain promotion. + +## Documentation And Architecture Record + +Update `docs/config.md` as the canonical owner of the selectable validator key, +binding behavior, and chain placement. Update `docs/internal/dnd.md` as the +canonical owner of the D&D-specific decision policy, prompt ownership, and +module boundary. Link rather than duplicate the generic retry, diagnostic, and +failure-policy contracts owned by their existing architecture and operator +documentation. + +No new ADR is required for this scope. It applies the accepted generic +validation and feedback-aware retry architecture without changing stage +ownership or a durable contract. Create or supersede an ADR only if later work +moves classification into `dnd/scenes`, adds annotations to the chunk plan, or +otherwise transfers ownership across pipeline stages. + +## Out Of Scope + +- changing scene boundaries or deciding whether one combat was fragmented + across multiple chunks; +- adding an LLM-backed chunk-plan validator; +- moving scene kind into chunk-plan annotations; +- changing the scene-description, combat-turn, or enemy-event durable schemas; +- replacing the deterministic combat-only downstream extraction gate; +- validating title or summary quality with this module; +- adding confidence scores or surfacing model uncertainty as warnings; +- adding a validator-specific concurrency or retry subsystem; and +- enabling the validator in the production default chain before evaluation. + +## Acceptance Criteria + +- The new LLM-backed validator is registered for scene-description artifacts + and can be selected only through the existing validator binding mechanism. +- It reviews exactly one extract-stage scene against the corresponding + transcript chunk and detects both missing and unsupported `combat` tags. +- Its response contract and candidate presentation do not ask the model to + reproduce transcript unit IDs, chunk IDs, source ranges, hashes, validator + keys, or reason codes. +- Its strict structured schema has only required fields, rejects unknown + fields, and avoids unsupported JSON Schema keywords. +- Rejections produce bounded, transcript-grounded corrective guidance that the + existing scene-description producer retry can use. +- Validator contract or execution failures remain distinct from semantic + rejection and follow existing configured policy. +- The existing deterministic validator chain and downstream exact-combat gate + remain intact. +- Prompt definitions share identical combat policy content rather than + maintaining near-duplicate instructions. +- Automated coverage is deterministic and provider-free; an explicit + human-reviewed evaluation protocol exists for later default-chain review. +- Current documentation describes implemented selection and behavior without + claiming that the validator is enabled by default. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index f987b450..4176c47b 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -14,47 +14,8 @@ eligibility, and the separation of actionable process warnings from quality diagnostics. The remaining near-term work applies those completed foundations to domain review and empirical evaluation. -### D&D Combat Scene Semantic Validation - -- Add an optional production LLM-backed D&D validator that determines whether - proposed scene boundaries and classifications represent substantive active - combat correctly. Its central quality goal is that active combat is kept in - coherent scenes classified as `combat`, rather than split incorrectly or - hidden inside scenes classified as `narrative`, `recap`, or `meta`. -- Resolve the validator's exact target before implementation. The current - `dnd/scenes` chunker owns only complete, gap-free source ranges, while the - per-chunk `dnd/scene-descriptions` extractor owns the `combat`, `narrative`, - `recap`, and `meta` classification. The preferred initial placement is - therefore an extract-stage validator for `dnd/scene-descriptions`, where it - can compare one proposed kind with the corresponding transcript chunk. -- Consider a chunk-stage LLM validator only for a distinct boundary-coherence - question that can be answered from the complete transcript and proposed - range map, such as whether one continuous combat was fragmented across - inappropriate scene boundaries. Do not duplicate the same classification - judgment at both stages. Moving classification into chunk-plan annotations - would change the deliberately minimal, annotation-free chunk contract and - requires an explicit architecture review before it is selected. -- Validate both false negatives and false positives: a non-combat kind must not - omit substantive active combat, and a combat kind must be supported by such - combat. Keep the existing deterministic downstream rule that combat-turn - extraction runs only for an exact `combat` scene classification; semantic - review improves the upstream classification but does not replace that gate. -- Run the semantic validator through PromptKit, use a minimal required-field - structured response schema, and let PromptKit repair structural validator - output within its bounded budget. A contract-invalid final validator response - is a validator execution failure, not a semantic rejection and not a reason - to recursively validate the validator. -- Evaluate the prompt and decision policy against a small human-reviewed set - containing combat setup, active turns, interruptions, multi-phase encounters, - brief rules discussion, aftermath, recalled combat, and false-positive - hostile dialogue. Measure false acceptance, false rejection, retry success, - added calls, latency, and token cost before placing it in the production - default chain. -- An ADR is not required if classification remains owned by - `dnd/scene-descriptions` and the validator follows the generic validation ADR. - Create or supersede an ADR if the work transfers scene classification into - the chunker or otherwise changes stage ownership or the durable chunk-plan - contract. +The active first production application is defined by +[D&D Combat Scene Semantic Validation](combat-scene-validation.md). ## Near-Term D&D Pipeline diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 00000000..7e2116be --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,488 @@ +# Implementation Plan + +## Objective + +Implement [D&D Combat Scene Semantic Validation](combat-scene-validation.md): +add the first production D&D LLM-backed validator as an optional extract-stage +validator for `dnd/scene-descriptions`, while preserving the existing chunk +plan, durable artifacts, downstream combat gate, maintained examples, and +production default validator chains. + +This plan is ordered and decision-complete. Each numbered stage is scoped for +one gpt-5.6-terra implementation prompt. Complete and validate each stage +before beginning the next one. + +## Decisions Shared By All Stages + +- Scene classification remains owned by the per-chunk + `dnd/scene-descriptions` extractor. Do not annotate `dnd/scenes` plans or add + a chunk-stage validator. +- Add the typed validator package + `internal/modules/dnd/validate/scenedescriptions/combat_semantics` with key + `extract/dnd/scene-descriptions/combat_semantics`, registered for + `dnd.SceneDescriptionListKind` with execution class `llm_backed`. +- The validator is selectable through the existing validator override but is + absent from all production default chains. It reviews only combat versus + non-combat, not title, summary, non-combat subtype, or boundary quality. +- Reuse the injected scheduled `StructuredLLMClient`, existing profile + precedence, PromptKit structural-repair budget, validator execution retries, + producer semantic retries, and validation failure policy. Add no parallel + provider, scheduler, retry, or configuration mechanism. +- Use prompt ID `dnd.scene_descriptions.validate_combat`, default profile + `dnd-extraction`, and private prompt/schema version `v1`. +- The LLM response has exactly two required fields: `verdict` and + `explanation`. Verdict is one of `approved`, `combat_should_be_added`, or + `combat_should_be_removed`. Reject unknown fields, optional properties, and + `uniqueItems`. Require a trimmed, nonblank explanation of at most 512 Unicode + code points in both schema and deterministic interpretation. +- Map rejections to application-owned reason codes + `scene_active_combat_not_classified` and + `scene_combat_classification_unsupported`. Codes, validator keys, opaque IDs, + and source coordinates are never model instructions. +- Transcript unit IDs may remain in ordinary source presentation, but the + model must not reproduce them. Candidate presentation contains only the + proposed kind; application code owns IDs, ranges, mapping, and provenance. +- PromptKit may repair structural validator output. A final malformed, + inconsistent, blank, or oversized validator response is an execution + failure, not a semantic rejection. Do not recursively validate or provide + semantic feedback to the validator's own LLM call. +- Rejection guidance tells the producer either to return `kind: combat` or to + choose the appropriate `narrative`, `recap`, or `meta` kind, and includes the + bounded transcript-grounded explanation. It contains no internal code. +- Keep durable schemas at their current versions and retain the exact `combat` + gates in combat-turn and enemy-event extraction. +- Default tests are deterministic, offline, and credential-free. Use fakes at + the structured LLM boundary. Do not add exact prompt-length, token-count, or + complete prose snapshot tests. +- No ADR is required. This feature applies the architecture and ADRs 0012, + 0014, and 0015 without changing stage ownership or a durable contract. + +## Stage 1: Extract The Shared Combat Policy + +### Goal + +Give the existing extractor and future validator one byte-identical owner for +the combat decision policy without changing the extractor contract or the +common extraction prompt prefix. + +### Work + +1. Add `assets/dnd/shared/prompts/common-dnd-scene-combat-policy.md` and move + into it the current combat definition, mixed-chunk precedence, and explicit + non-examples: planning, threats, hostile dialogue, aftermath, recollection, + and rules discussion without active encounter play. +2. Remove those rules from the local scene-description instructions. Retain + the exactly-one-scene task, all four supported kind names, the + `narrative`/`recap`/`meta` definitions and precedence, and title/summary + policy. The rendered prompt must remain complete and non-repetitive. +3. Add the shared file to the scene extractor's `PromptAssetManifest` and + `prompt.yaml`. Preserve the extraction-wide prefix through the cached + transcript. Place the combat-policy message after the transcript and before + local instructions so other extraction lanes retain an identical prefix. +4. Update prompt asset and composition tests to prove the shared policy appears + once in the correct relative position. Use semantic sentinels rather than a + full prompt snapshot or exact size assertion. +5. Do not change the prompt ID/version, response schema, extractor logic, + artifact, or validator chains. + +### Inspect + +- `assets/dnd/shared/prompts/` +- `assets/dnd/scene-descriptions/prompts/` +- `internal/modules/dnd/extract/scenedescriptions/prompt_assets.go` +- `internal/modules/dnd/extract/scenedescriptions/prompt_assets_test.go` +- `internal/modules/dnd/register/prompt_cache_test.go` + +### Acceptance And Validation + +- One embedded shared asset owns the combat policy. +- The scene prompt still prepares with complete classification instructions. +- All extraction prompts retain the same prefix through the transcript. +- No brittle change-detector test is added. + +Run: + +```sh +go test ./internal/modules/dnd/extract/scenedescriptions +go test ./internal/modules/dnd/register +git diff --check +``` + +This stage is small enough for one implementation prompt. + +## Stage 2: Add Validator Prompt And Schema Assets + +### Goal + +Create and locally verify the private PromptKit contract before implementing +semantic logic or global registration. + +### Work + +1. Create + `assets/dnd/scene-descriptions/validate/combat-semantics/` with: + + - `prompts/prompt.yaml`; + - `prompts/instructions.md`; + - `prompts/proposed-kind.md`; and + - `schemas/dnd_scene_combat_semantics_llm.v1.json`. + +2. Define required inputs `transcript` (`application/json`) and + `proposed_kind` (`text/plain`). Order messages as system, shared combat + policy, stable local validator instructions, variable proposed kind, and + final variable shared chunk transcript. Mark only the two variable messages + ephemeral. Do not select campaign references or the identity fragment. +3. Define the verdict semantics precisely. `approved` approves only the + proposed combat status, not the whole scene description. +4. Define a strict object schema with `additionalProperties: false`, both + fields in `required`, the three-value verdict enum, and explanation + `minLength: 1` and `maxLength: 512`. Include no confidence, ID, range, + diagnostics, or correction field. +5. Add package-local asset/schema plumbing in `combat_semantics`, following + existing D&D manifests. Use: + + - schema key `dnd_scene_combat_semantics_llm`; + - schema ID `notarius.dnd.scene_descriptions.combat_semantics.llm`; and + - schema name `notarius_dnd_scene_combat_semantics_llm_v1`. + + Expose `RegisterPromptAssets`, cached asset metadata, and mutation-safe + schema loading. Do not register them at the D&D composition root yet. +6. Add focused tests that prepare the embedded prompt, require both inputs, + accept all valid verdicts, and reject missing, extra, mistyped, blank, and + oversized fields. Metadata tests must prove hashes are available without + exposing raw assets. + +### Inspect + +- existing scene-description extractor asset and schema code +- analogous D&D semantic-reconciliation prompt assets +- `internal/framework/llm` +- `internal/framework/promptfs` + +### Acceptance And Validation + +- The package-local embedded prompt and schema prepare offline. +- All schema fields are required and no `uniqueItems` appears. +- The prompt contains only the policy, proposed kind, and transcript needed for + this decision and selects rather than copies the shared policy. +- No global registration changes occur yet. + +Run: + +```sh +go test ./internal/modules/dnd/validate/scenedescriptions/combat_semantics +go test ./internal/modules/dnd/extract/scenedescriptions +git diff --check +``` + +This stage is small enough for one implementation prompt. + +## Stage 3: Implement The Typed Validator + +### Goal + +Implement package-local validator construction, preconditions, structured +completion, verdict interpretation, corrections, fingerprints, and behavioral +tests. + +### Work + +1. Define the module key, two reason codes, zero-field strict `Options`, private + response type, and validator holding only the injected client and immutable + prompt/schema/policy metadata. Construction fails contextually for a nil LLM + dependency. +2. Register through a typed builder consuming `request.Dependencies.LLM`. + Retain no unrelated references or mutable request maps. +3. Before calling the model, require extract stage, non-nil source and chunk, + valid chunk-scoped source input, exactly one scene satisfying the existing + extract shape contract, and scene ID/range equal to the current chunk. + Violations are execution errors, not approval or skip. +4. Call `CompleteStructured` once per execution with validator key as stage + name; the selected prompt/version; request profile, session ID, and repair + pointer; cloned chunk source input as `transcript`; and only the scene kind + string as `proposed_kind`. Do not forward references, scene prose, + identifiers, ranges, reason codes, or semantic correction. +5. Interpret output deterministically: + + - trim and enforce the 512-code-point explanation bound; + - approve `approved` for either proposed combat status; + - accept `combat_should_be_added` only for a non-combat proposal; + - accept `combat_should_be_removed` only for a combat proposal; and + - treat every unknown or inconsistent result as execution failure. + +6. Build bounded `ValidationResult` messages and semantic correction guidance. + Approved results carry no rejection message or quality diagnostic. +7. Implement `Name`, `ExecutionClass`, `Spec`, manifest metadata, checkpoint + fingerprints, and strict option validation. A validator spec does not + declare a producer correction protocol. +8. Add consolidated table-driven tests with a recording fake LLM for approvals, + both rejection directions, inconsistent results, explanation bounds, input + preconditions, completion error/cancellation, request-field forwarding, + model-visible input exclusions, option rejection, and metadata ownership. + +### Inspect + +- `internal/framework/contracts/{contracts,typed_pipeline}.go` +- `internal/framework/pipeline/{construction,validator_registry}.go` +- `internal/modules/dnd/validate/scenedescriptions/{shape,source_refs}` +- existing LLM-backed D&D extractor and normalizer packages + +### Acceptance And Validation + +- The package is usable without provider-specific types or global state. +- It cannot judge a wrong-stage, aggregate, malformed, or wrong-chunk value. +- Semantic rejection and execution failure remain distinct. +- Guidance is meaningful and contains no opaque identifiers or internal codes. +- Consequential contracts are tested offline through the package boundary. + +Run: + +```sh +go test ./internal/modules/dnd/validate/scenedescriptions/... +go vet ./internal/modules/dnd/validate/scenedescriptions/... +git diff --check +``` + +This stage is small enough for one implementation prompt. + +## Stage 4: Register And Resolve The Optional Validator + +### Goal + +Expose the module through production D&D composition and prove explicit +configuration works without changing defaults. + +### Work + +1. Register the validator builder in + `internal/modules/dnd/register/validators.go` and its assets in + `internal/modules/dnd/register/modules.go`, preserving deterministic + registration and contextual errors. +2. Extend registrar/catalog tests to prove the typed kind, execution class, + production prompt/schema availability, and absence from both scene default + chains. +3. Add one focused resolution/preparation case whose scene extract override + repeats the current default validators in order and appends the semantic + validator last. Prove validator `llm_profile`, + `structured_output_repair_attempts`, and `retries` reach the prepared + LLM-backed validator through existing machinery. +4. Add a negative case for selecting this typed validator against an + incompatible artifact kind if existing typed-registry tests do not already + exercise the concrete production key. Rely on generic tests for all other + validator-binding rules. +5. Do not alter the minimal or complete D&D example configurations. + +### Inspect + +- `internal/modules/dnd/register/{validators,modules,chains}.go` +- `internal/modules/dnd/register/register_test.go` +- `internal/modules/dnd/register/prompt_cache_test.go` +- typed validator resolution/preparation tests under + `internal/framework/pipeline` + +### Acceptance And Validation + +- Production registration exposes the validator and prompt. +- An explicit compatible override resolves before execution. +- Default chain membership/order and maintained examples are unchanged. +- Incompatible selection still fails during resolution or preparation. + +Run: + +```sh +go test ./internal/modules/dnd/register +go test ./internal/framework/pipeline +go test ./internal/cli +git diff --check +``` + +This stage is small enough for one implementation prompt. + +## Stage 5: Prove Feedback-Aware Producer Correction + +### Goal + +Prove the assembled D&D path can reject a well-formed wrong scene kind, guide +the scene producer, and accept a corrected replacement without leaking +internal identifiers. + +### Work + +1. Add one focused test under `internal/modules/integration` (or extend the + narrowest assembled scene test) with a stateful fake structured client that + distinguishes scene producer and semantic validator prompt IDs. +2. Configure the current deterministic scene extract chain followed by the + semantic validator, with a positive producer retry budget and no unnecessary + validator execution retries. +3. Use a chunk containing clear active combat. Return a valid non-combat scene + first, `combat_should_be_added` from the validator, a corrected combat scene + after feedback, and approval on revalidation. +4. Assert outcomes rather than private choreography: + + - the accepted normalized scene is `combat`; + - superseded attempts leave no rejection or process warning; + - the producer receives the exact latest defective model candidate and one + semantic correction request; + - guidance directs `kind: combat` and includes the bounded explanation; + - guidance excludes validator key, reason code, chunk ID, and source range; + and + - the corrected candidate traverses the complete validator chain again. + +5. Keep unsupported-combat verdict mapping owned by Stage 3 package tests + unless adding it to this integration test is materially cheaper than a + duplicate assembled setup. +6. Rely on existing framework tests for aggregate guidance, exhaustion, + `warn_continue`, validator retries, and checkpoint ineligibility. Change + generic framework code only if this test exposes a real defect, and report + the scope expansion explicitly. + +### Inspect + +- `internal/modules/integration/` +- framework extract-correction and semantic-correction tests +- `internal/cli/assembled_spell_pipeline_contract_test.go` +- `internal/modules/dnd/extract/scenedescriptions/` + +### Acceptance And Validation + +- One deterministic assembled test proves D&D semantic correction and + revalidation. +- Only the latest defective response and semantic guidance reach the producer. +- No internal code or deterministic identifier reaches correction text. +- Existing framework budgets and policies remain the only orchestration path. + +Run: + +```sh +go test ./internal/modules/integration +go test ./internal/framework/pipeline +go test ./internal/modules/dnd/extract/scenedescriptions +git diff --check +``` + +This stage is small enough for one implementation prompt. + +## Stage 6: Add Evaluation Material And Documentation + +### Goal + +Provide accurate opt-in configuration guidance and a bounded human-reviewed +corpus for evaluation before any future default-chain proposal. + +### Work + +1. Add a synthetic, non-sensitive corpus at + `internal/modules/dnd/validate/scenedescriptions/combat_semantics/testdata/evaluation_cases.json`. + Each case has a human-readable unique name, transcript units, proposed kind, + expected verdict, and reviewer rationale. Cover active combat, setup and + aftermath around combat, multi-phase combat, rules interruption, planning, + threats, hostile dialogue, aftermath without combat, recap/recollection, + and sustained out-of-character discussion. +2. Add one cheap fixture-contract test for strict decoding, unique/nonblank + names, nonblank input, supported proposed kinds/verdicts, and representation + of all verdict classes. Do not assert the exact fixture count or treat fake + responses as a measurement of model quality. +3. Document a manual or explicitly opt-in provider evaluation procedure in + `docs/internal/dnd.md`. Require recording false acceptance, false rejection, + producer correction success, added calls, latency, and token use before + default-chain promotion. Do not add live calls to the default suite. +4. Update `docs/config.md`, the canonical configuration owner, to list the key, + state its LLM-backed opt-in status, and show the smallest illustrative + override that repeats the existing scene extract chain and appends the + semantic validator. Emphasize that an override replaces the chain and that + a positive producer retry budget is required if rejection should trigger a + corrected scene-description attempt. Link to the existing binding table + instead of repeating profile, structural-repair, producer-retry, or + validator-retry rules. +5. Update `docs/internal/dnd.md` to describe the module boundary, prompt-policy + ownership, verdict mapping, deterministic correction mapping, and difference + from deferred boundary-coherence review. Link to configuration and generic + retry/failure owners rather than duplicating volatile contracts. +6. Update `docs/internal/overview.md` or `docs/internal/modules.md` only if its + implemented inventory would otherwise become false. Do not add an ADR, a + third example, or a durable integration contract change. + +### Inspect + +- `docs/{config,internal/dnd,internal/modules,internal/overview}.md` +- all files under `docs/policy/` +- the two maintained D&D example configurations + +### Acceptance And Validation + +- The corpus is bounded, synthetic, strict, and covers the intended cases + without masquerading as a provider quality test. +- Configuration documents one correct opt-in path and preserves replacement + semantics. +- Canonical documentation owners describe only behavior implemented in Stages + 1–5, with no volatile duplication or default-enabled claim. +- All new relative Markdown links resolve manually; the repository currently + has no automated documentation link checker. + +Run: + +```sh +go test ./internal/modules/dnd/validate/scenedescriptions/combat_semantics +go test ./internal/modules/dnd/register ./internal/framework/pipeline ./internal/cli +git diff --check +``` + +This stage is small enough for one implementation prompt. + +## Stage 7: Final Verification And Scope Audit + +### Goal + +Verify the feature as one coherent change and leave it ready for a separate +post-implementation review. + +### Work + +1. Compare the complete diff with the feature roadmap and every stage above. + Implement or report every unmet acceptance criterion; do not silently defer + required work. +2. Confirm by inspection that the validator is registered but absent from + default chains/examples; both consumers select the shared combat policy; no + duplicate policy remains; its schema has all fields required and no + `uniqueItems`; model-facing content requests no opaque IDs or internal + codes; it uses the scheduled client and existing budgets; and downstream + exact-combat gates are unchanged. +3. Format changed Go files. Remove only redundant tests, stale comments, unused + helpers, or roadmap-inconsistent documentation found during this review; do + not perform unrelated refactoring. +4. Run the full validation suite. If an environmental limitation blocks a + command, run unaffected checks and record the exact limitation rather than + weakening tests. +5. Leave both roadmap documents in place for the post-implementation audit. Do + not promote the validator or retire the roadmaps. + +### Acceptance And Validation + +- Focused and repository-wide checks pass. +- No durable schema, chunk plan, default chain, example, or downstream gate + changed outside the roadmap. +- The worktree contains no binary, credential, private transcript, ignored + debris, or unrelated modification. +- Documentation paths and links resolve, and the repository is ready for + completeness and code-quality review. + +Run: + +```sh +gofmt -w +go test ./... +go test -race ./... +go vet ./... +go build ./cmd/notarius +git diff --check +git status --short --untracked-files=all +``` + +This stage is small enough for one implementation prompt. + +## Open Questions + +None. The feature roadmap and this plan fix the stage, scope, schema, prompt +ownership, reason-code mapping, configuration status, failure semantics, +evaluation gate, and documentation ownership. Default-chain promotion and +chunk-boundary semantic review are explicitly outside this plan.