From b8b3f3abfa2c932bb2be0f5af6a38506b9770f75 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Thu, 23 Jul 2026 18:59:34 +0000 Subject: [PATCH] Plan D&D scene description extraction --- docs/roadmap/dnd-scene-descriptions.md | 259 +++++++++++++ docs/roadmap/future.md | 25 +- docs/roadmap/implementation.md | 485 +++++++++++++++++++++++++ 3 files changed, 748 insertions(+), 21 deletions(-) create mode 100644 docs/roadmap/dnd-scene-descriptions.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/dnd-scene-descriptions.md b/docs/roadmap/dnd-scene-descriptions.md new file mode 100644 index 0000000..ad009b0 --- /dev/null +++ b/docs/roadmap/dnd-scene-descriptions.md @@ -0,0 +1,259 @@ +# D&D Scene Descriptions + +Status: Accepted + +## Purpose + +Add a focused D&D artifact that describes each accepted scene with a bounded +classification, a concise title, and a concise summary. This is the explicit +home for the small amount of source-bounded synthesis useful to downstream +applications; scene chunking should not remain the long-term owner of those +descriptive fields. + +The feature follows +[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md). +It is intentionally richer than a fact-only occurrence extractor, but every +model-produced field has a specific scene-description purpose and is grounded +by the exact accepted chunk range. + +## Desired End State + +A D&D pipeline that uses accepted scene chunks can add a +`dnd/scene-descriptions` artifact lane. The extractor runs once per accepted +chunk and returns exactly one description for that chunk. The merged, +normalized artifact is an ordered list with one independently identified and +evidenced scene record for every successful chunk extraction. + +Use these production identities: + +| Concern | Identity | +| --- | --- | +| Extractor key | `dnd/scene-descriptions` | +| Normalizer key | `dnd/scene-descriptions` | +| Artifact kind | `dnd/scene-description-list` | +| Durable schema ID | `notarius.dnd.scene_descriptions` | +| Durable schema name | `notarius_dnd_scene_descriptions_v1` | +| Durable schema version | `v1` | +| Durable media type | `application/json` | +| Private prompt ID | `dnd.scene_descriptions` | +| Private response schema ID | `notarius.dnd.scene_descriptions.llm` | +| Private response schema version | `v1` | + +The lane uses the existing typed artifact, append-order merge, codec, +registration, validator-chain, prompt-asset, retry, and output conventions. It +does not introduce a new pipeline stage or a dependency on the durable +chunk-map export. + +## Durable Artifact Contract + +The durable payload is a strict object containing a `scenes` array. The array +is ordered by accepted source position. Each scene contains exactly: + +- `id`: the accepted chunk ID, copied deterministically; +- `source_ref`: the accepted chunk's exact inclusive current-source range; +- `kind`: one value from the closed scene-kind vocabulary; +- `title`: a concise source-grounded label for the scene; and +- `summary`: a concise source-grounded account of the scene's main activity. + +The accepted chunk ID is the scene ID. Do not add a second `chunk_id` field or +generate an unrelated scene identifier: this artifact has exactly one record +per accepted scene chunk, so the chunk identity is already the stable identity +needed for joining with chunk maps and other range-grounded artifacts. + +Every fixed object rejects unknown fields. `id` and `source_ref` are +application-owned evidence and identity fields. The model must not emit or +reproduce chunk IDs, source IDs, unit IDs, or ranges. + +The extractor maps one successful private response into a single-element +durable `scenes` array before extraction validation. It attaches the current +chunk ID and range at that boundary rather than relying on normalization to +manufacture required provenance. + +## Model-Owned Fields + +The private model response contains exactly: + +- `kind`; +- `title`; and +- `summary`. + +It represents one scene object, not a list. An accepted chunk is already the +scene boundary, so the extractor must neither split the chunk nor return +multiple descriptions. It must not return an empty result; inability to +classify or describe the chunk is an extraction failure subject to the +configured retry and rejection policy. + +`title` should be a short, distinguishing phrase rather than a sentence, +chapter number, or generic label such as “Scene.” It may use names and places +established by the transcript or optional campaign references, but must not +invent a proper noun to make the title more colorful. + +`summary` should briefly state the main activity and material transition or +outcome established within the chunk. It must not add analysis, inferred +motives, hidden state, future consequences, relationship claims, or facts from +outside the chunk. Do not require a structured participant list, encounter +outcome, location, mood, or confidence field. + +## Scene-Kind Vocabulary + +Use this closed vocabulary: + +| Kind | Meaning | +| --- | --- | +| `combat` | Active combat materially organizes the scene, including initiative-like exchanges or sustained hostile action. Planning a fight or discussing a completed fight is not combat by itself. | +| `narrative` | Current-session in-world play that is not principally active combat, a prior-session recap, or sustained out-of-character session discussion. This includes exploration, travel, dialogue, investigation, planning in character, and aftermath. | +| `recap` | The scene's organizing purpose is to recount events from a previous session for the table. An in-world character recounting history during current play remains narrative. | +| `meta` | The scene's organizing purpose is sustained out-of-character discussion about the game or session rather than advancing current in-world play. | + +`narrative` is the default category for actual current-session gameplay that +does not meet another definition. The model must choose exactly one kind and +must not emit an uncertainty or mixed category. + +## Residual Mixed-Scene Policy + +Scene chunking should normally separate sustained transitions between kinds. +When an accepted chunk nevertheless contains more than one kind, classify the +activity that best describes the chunk's organizing purpose and main +current-session progression: + +1. use `combat` when active combat is a substantive central activity, even if + the chunk includes brief setup, rules clarification, or immediate aftermath; +2. use `recap` when recounting a previous session is the chunk's primary table + purpose, not merely a brief reminder before current play; +3. use `meta` when sustained out-of-character session discussion is primary and + the chunk contains no more than incidental in-world progression; and +4. use `narrative` for all remaining current-session in-world play. + +Brief table talk, dice resolution, rules clarification, jokes, or administrative +comments are incidental to the enclosing gameplay scene and do not make it +`meta`. A short recollection used to orient current action does not make a scene +`recap`. + +This precedence resolves residual ambiguity for description. It does not alter +the accepted chunk boundary or authorize the extractor to conceal a +demonstrably poor scene plan. + +## Evidence, References, And Prompt Boundary + +The exact accepted chunk range is the evidence for all three model-owned +fields. Do not ask the model for finer-grained source ranges, and do not use +campaign references as evidence. + +The lane requires no generated artifact reference. Existing optional D&D +campaign references may be exposed through the established shared slots when +they help disambiguate names or setting terms in titles and summaries. They +must not introduce events, participants, or claims absent from the current +chunk. Do not require an NPC registry: structured participants remain owned by +NPC and NPC-interaction artifacts. + +Follow the established D&D prompt ordering and prompt-cache boundary: + +1. shared D&D system and identity policy; +2. stable campaign references, including explicit empty placeholders; +3. lane task and classification instructions; and +4. the variable current-chunk transcript last. + +Factor wording genuinely shared with other D&D modules into existing shared +prompt assets. Keep scene-kind definitions and mixed-scene policy in +scene-description-owned assets rather than broadening unrelated extractors. +Do not render the existing shared extraction-evidence asset in this lane: that +asset instructs the model to emit source ranges, while scene identity and the +entire accepted chunk range are attached deterministically by the application. +This is a deliberate exception to the shared extraction prompt sequence, not +permission to duplicate compatible shared wording in lane-owned assets. + +## Merge And Normalization + +Use the existing append-order merge behavior. Normalization remains +deterministic and must: + +- validate every record against the current source document; +- require an exact closed-vocabulary kind without case-folding or repairing an + invalid model value; +- trim surrounding title and summary whitespace without otherwise rewriting + their prose; +- order scenes by source-document position, with accepted chunk ID as the + deterministic tie-breaker; +- remove only completely identical duplicate records; and +- reject conflicting records that reuse one scene ID or exact source range with + different model-owned content. + +Normalization must not combine adjacent scenes, regenerate titles or summaries, +change classifications through heuristics, or infer missing records. Exact +deduplication is defensive cleanup, not semantic reconciliation. + +## Validation And Quality + +Provide the same layered production validation used by other typed D&D +artifacts: + +- valid JSON and strict private/durable shape; +- exactly one private description and one mapped durable scene per chunk; +- closed kind vocabulary; +- non-empty, trimmed title and summary; +- non-empty accepted chunk ID; +- a current-source range exactly equal to the extraction chunk range; +- source-relatedness checks for title and summary; and +- strict durable JSON Schema validation after normalization. + +Shape, identity, enum, and source-range violations are deterministic +rejections. Source-relatedness should initially remain warning-only unless +evaluation establishes a reliable rejection rule. Diagnostics must be bounded +and must not copy transcript or campaign-reference content. + +Evaluate the lane on a small human-reviewed transcript set covering all four +kinds, incidental table talk, brief and sustained recap, combat setup and +aftermath, mixed residual chunks, sparse scenes, proper-noun ambiguity, and +unsupported synthesis. Review classification agreement, title usefulness, +summary fidelity, evidence attachment, and performance on the cost-effective +models Notarius is intended to support. Human evaluation is a development aid, +not an exact-text CI oracle. + +Tests should protect structural contracts, deterministic mapping, +normalization, validation, prompt assembly, registration, and one assembled +production workflow. Do not assert exact generated prose or use brittle prompt +length and shared-prefix change detectors. + +## Relationship To Scene Chunking + +This artifact becomes the intended owner of scene kind, title, and summary. +The existing `dnd/scenes` chunker remains unchanged in this scope. Downstream +consumers should prefer the normalized scene-description artifact over chunk +annotations for kind, title, and summary. + +Removing descriptive fields from the chunker is a separate follow-on scope. It +requires its own review of chunker prompt quality, warning/validator consumers, +cache behavior, and the compatibility implications of changing accepted plan +annotations. Do not couple that migration to the initial artifact lane. + +Likewise, do not make combat extraction conditional on scene kind in this +scope. A later ordered-pipeline feature may consume the accepted normalized +scene-description artifact and no-op combat extraction for non-combat scenes +without making generic chunk materialization depend on D&D classification. + +## Documentation Ownership + +Once implemented, the durable contract belongs in `docs/integrations/`; +selectable keys, optional campaign-reference bindings, and default validator +chains belong in configuration documentation; and prompt, module, validator, +and pipeline mechanics belong in current internal documentation. A maintained +D&D example should demonstrate scene chunking and scene-description extraction +without presenting chunk annotations as the durable description contract. + +## Non-Goals + +This scope does not: + +- change scene boundaries or run a second chunker; +- ask the model for IDs, source ranges, segment numbers, or evidence selection; +- add structured participants, locations, encounters, outcomes, moods, + confidence, or relationship fields; +- duplicate NPC or NPC-interaction extraction; +- require an NPC registry or another generated artifact; +- enrich descriptions from campaign references when the chunk lacks support; +- semantically merge or reconcile adjacent scene descriptions; +- minimize the existing scene chunker in the same change; +- gate combat extraction by scene kind; +- make scene descriptions generated references for later steps; +- change chunk-map export, chunk caching, or plan validation; or +- introduce a DAG, a new pipeline stage, or a new reference mechanism. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 43044d5..d996af2 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -18,27 +18,10 @@ not as committed release dates. ### Extract D&D Scene Descriptions -- Add a `dnd/scene-descriptions` extractor that runs once for each accepted - scene chunk and explicitly owns the small amount of scene synthesis useful to - downstream applications. -- Keep the private model response to exactly `kind`, `title`, and `summary`. - Use the enum `combat`, `narrative`, `recap`, and `meta`: `narrative` means - current-session in-world gameplay that is not combat, recap, or sustained - out-of-character discussion. -- Treat brief table talk or rules clarification as incidental to the enclosing - gameplay scene. A sustained transition between kinds should normally create - a chunk boundary; define a primary-kind rule for residual mixed chunks before - implementation. -- Deterministically attach the accepted chunk ID and its exact source range - while mapping the private response into the durable artifact. Do not ask the - model to reproduce IDs or segment boundaries, and do not defer required - identity or evidence until normalization. -- Keep normalization limited to stable ordering, exact deduplication, and - canonical invariant enforcement. Titles and summaries are explicit, - source-bounded synthesis owned by this artifact rather than by the chunker. -- Do not add `participants`. Derive participant-oriented views by joining scene - ranges with NPC evidence or, preferably, NPC-interaction occurrences. An NPC - registry reference proves identity, not exhaustive presence in every scene. +Add one minimal, source-bounded description for each accepted D&D scene chunk. +The proposed durable contract, scene-kind policy, evidence boundary, and +relationship to later chunker minimization are defined in +[D&D Scene Descriptions](dnd-scene-descriptions.md). ### Minimize And Use D&D Scene Chunking diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..b40d954 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,485 @@ +# D&D Scene Descriptions Implementation Plan + +Status: Ready for implementation + +## Objective And Authority + +Implement the accepted target state in +[D&D Scene Descriptions](dnd-scene-descriptions.md): one strict, typed +`dnd/scene-description-list` artifact record for every successfully extracted +accepted scene chunk. The model owns only `kind`, `title`, and `summary`; +Notarius owns the scene ID and exact current-source range. + +This plan is the execution document. The feature roadmap owns intent, durable +policy, classification semantics, and non-goals. If wording here appears to +conflict with that roadmap, preserve the roadmap's contract and update this +plan before proceeding. + +Follow: + +- [Architecture Policy](../policy/architecture.md), especially typed domain + codecs, source-reference ownership, central registration, and the fixed + input/chunk/extract/merge/normalize/output flow; +- [Testing Policy](../policy/testing.md), especially behavioral tests at the + stable owner, offline fake-LLM coverage, and the prohibition on prompt-length + or shared-prefix change-detector tests; and +- [Documentation Policy](../policy/documentation.md), especially one canonical + owner for each contract and updating current-state docs only when behavior is + implemented. + +The repository is pre-release. Use the `v1` identities fixed below; do not add +compatibility aliases, dual schemas, migration adapters, or legacy decoding. + +## Fixed Implementation Decisions + +### Public identities + +Use these values exactly: + +| Concern | Value | +| --- | --- | +| Extractor key | `dnd/scene-descriptions` | +| Normalizer key | `dnd/scene-descriptions` | +| Artifact kind | `dnd/scene-description-list` | +| Durable schema ID | `notarius.dnd.scene_descriptions` | +| Durable schema name | `notarius_dnd_scene_descriptions_v1` | +| Durable schema version | `v1` | +| Durable media type | `application/json` | +| Prompt ID | `dnd.scene_descriptions` | +| Private response-schema key | `dnd_scene_descriptions_llm` | +| Private response-schema ID | `notarius.dnd.scene_descriptions.llm` | +| Private response-schema name | `notarius_dnd_scene_descriptions_llm_v1` | +| Private response-schema version | `v1` | +| Extractor mapping policy | `dnd.scene_descriptions.mapping.v1` | +| Normalizer policy | `dnd.scene_descriptions.normalizer.v1` | + +### Typed and JSON shapes + +Add the following canonical D&D types in `internal/modules/dnd/types.go`: + +```go +type SceneKind string + +const ( + SceneKindCombat SceneKind = "combat" + SceneKindNarrative SceneKind = "narrative" + SceneKindRecap SceneKind = "recap" + SceneKindMeta SceneKind = "meta" +) + +type SceneDescriptionList struct { + Scenes []SceneDescription `json:"scenes"` +} + +type SceneDescription struct { + ID string `json:"id"` + SourceRef source.SourceRef `json:"source_ref"` + Kind SceneKind `json:"kind"` + Title string `json:"title"` + Summary string `json:"summary"` +} +``` + +Also add +`SceneDescriptionListKind contracts.ArtifactKind = +"dnd/scene-description-list"`. + +The durable JSON object contains exactly `scenes`; every scene object contains +exactly `id`, `source_ref`, `kind`, `title`, and `summary`. The private model +response is one object containing exactly `kind`, `title`, and `summary`. It is +not wrapped in `scenes`, cannot be an array, and cannot represent “no result.” +Both schemas reject unknown fields. Both schemas enumerate all four kind +values. Durable strings and private `title` and `summary` use `minLength: 1`; +do not impose arbitrary maximum lengths in schema or deterministic validation. + +### Application-owned mapping + +For each extraction request: + +- obtain transcript material with `shared.ChunkPromptMaterial`; +- prepare optional `players`, `party`, and `glossary` inputs, including the + established explicit empty placeholders, with `shared.PromptInputs`; +- call the private structured prompt once; +- copy `req.Chunk.ID` to the durable `id`; +- copy `req.Chunk.Ref` to the durable `source_ref`; +- copy `kind` without trimming, case-folding, aliasing, or repair; and +- trim surrounding whitespace from `title` and `summary`, then return a + single-element durable `scenes` list. + +An invalid or absent model object is an extraction failure handled by existing +structured-output and retry policy. The extractor must never synthesize a +fallback classification or empty scene list. + +Include the prompt digest, response-schema digest, and mapping policy in +extractor manifest metadata and checkpoint fingerprints, following the combat +turn extractor pattern. The optional campaign references affect the prompt +input identity through existing framework behavior; do not add a generated +artifact reference or an NPC registry slot. + +### Prompt assets and caching + +Create package-owned prompt assets under +`internal/modules/dnd/extract/scenedescriptions/assets/`. Use this exact message +order: + +1. shared `common-dnd-system.md` as the system message; +2. shared `common-dnd-identity.md` as a user message and cache boundary; +3. shared `common-dnd-references.md` as a user message and cache boundary; +4. package-owned `task.md`; +5. package-owned `instructions.md` as the final stable cache boundary; and +6. shared `common-dnd-transcript.md` as the final user message, without cache + control. + +Declare `transcript`, `players`, `party`, and `glossary` inputs with the same +media types and required/optional status used by `dnd/npcs`. Reuse the shared +assets through the existing Scriptorium module/shared filesystem pattern; do +not copy their text into this package. + +Do **not** render `common-dnd-extraction-evidence.md`. Its citation instructions +require the model to emit ranges, which conflicts with the application-owned +whole-chunk evidence contract. Put the scene-kind vocabulary, residual +mixed-scene precedence, title constraints, summary constraints, and the rule +that the response describes exactly the supplied accepted chunk in the +package-owned task/instruction assets. Do not ask for IDs, ranges, source IDs, +participants, confidence, or additional fields. + +### Merge, normalization, and conflicts + +Register the existing append-order merger specialized for +`SceneDescriptionList`. Its append function must preserve nil-versus-present +slice semantics and return independently owned values, matching the other D&D +specializations. + +The normalizer is deterministic and has no reference slots. It must: + +1. reject a nil source, malformed current-source reference, blank ID, invalid + kind, or blank title/summary; +2. trim only surrounding whitespace from title and summary; +3. sort by the source document's start-unit position, then scene ID as the + deterministic tie-breaker; +4. remove only records identical in all five durable fields; +5. reject any repeated scene ID whose remaining fields are not identical; and +6. reject records with the same exact source range and different + `kind`, `title`, or `summary`. + +Two records with different IDs but the same range and identical model-owned +content are not exact duplicates and are not a conflict under this contract; +retain both in deterministic ID order. Do not merge adjacent ranges, reconcile +prose, change kinds heuristically, infer missing scenes, or use chunk +annotations. + +### Validation ownership and default chains + +Add these deterministic validators: + +| Key | Owner | +| --- | --- | +| `extract/dnd/scene-descriptions/shape` | Required list and record fields, exact scene-kind vocabulary, trimmed non-empty strings, and exactly one scene during extraction. | +| `extract/dnd/scene-descriptions/source_refs` | Current-source validity at all stages; at extraction, requires a current chunk and exact equality of both scene ID and source range to that chunk. | +| `extract/dnd/scene-descriptions/source_relatedness` | Advisory lexical grounding for title and summary against the scene's current transcript range only. | +| `normalize/dnd/scene-descriptions/invariants` | Normalized order, exact-duplicate removal, unique/conflict rules, and normalized strings. | + +The shape validator requires a present, non-empty `scenes` list at all stages +and exactly one element when `req.Stage == string(pipeline.StageExtract)`. +Stage-independent record validation rejects blank IDs, unsupported kinds, +blank or untrimmed title/summary, and structurally empty source references. + +The source-reference validator first defers malformed shape to the shape +validator. At extraction it rejects a nil `req.Chunk`, an ID unequal to +`req.Chunk.ID`, or a `source_ref` unequal to `req.Chunk.Ref`; containment is not +sufficient. At normalization, where no current chunk is available, it validates +the reference against `req.Source` but does not attempt to reconstruct chunk +identity. + +The source-relatedness validator is warning-only and runs only after shape and +source-reference validity. For each scene, tokenize the cited current-source +text, title, and summary with `shared.NormalizedTokens`. Ignore tokens shorter +than three Unicode code points and these case-normalized function words: + +`a`, `an`, `and`, `are`, `as`, `at`, `be`, `but`, `by`, `for`, `from`, `had`, +`has`, `have`, `he`, `her`, `him`, `his`, `in`, `into`, `is`, `it`, `its`, +`of`, `on`, `or`, `she`, `that`, `the`, `their`, `them`, `they`, `this`, `to`, +`was`, `were`, `with`. + +Emit one warning scoped to `scenes[i].title` if no remaining title token occurs +in the cited text, and independently one warning scoped to +`scenes[i].summary` if no remaining summary token occurs there. If a field has +no remaining significant token, emit its warning. Never use campaign +references for this check. Bound and safely quote diagnostics through the +existing D&D diagnostic helpers; do not include transcript or reference +content in messages. + +Register these exact default chains: + +```text +extract dnd/scene-descriptions: + generic/valid_json + extract/dnd/scene-descriptions/shape + extract/dnd/scene-descriptions/source_refs + generic/valid_json_schema + extract/dnd/scene-descriptions/source_relatedness + +normalize dnd/scene-descriptions: + generic/valid_json + extract/dnd/scene-descriptions/shape + normalize/dnd/scene-descriptions/invariants + extract/dnd/scene-descriptions/source_refs + generic/valid_json_schema + extract/dnd/scene-descriptions/source_relatedness +``` + +Do not add merge-stage validators, LLM-backed validators, options, generated +references, stage dependencies, chunk-map dependencies, or framework changes. + +## Stage 1: Add The Durable Typed Contract And Codec + +### Work + +1. Extend `internal/modules/dnd/types.go` with the fixed artifact kind, scene + kind, constants, list, and record types. +2. Add `internal/modules/dnd/codec/scenedescriptions/` following the strict + candidate/approved encode/decode pattern in the existing D&D codecs. +3. Embed + `assets/schemas/dnd_scene_descriptions.v1.json` with the exact durable shape + and identities above. +4. Return `scene_count` from codec metadata. +5. Add one representative valid JSON fixture under `testdata/`. + +### Tests + +Add focused codec tests covering schema identity and validity, valid +round-trip, strict unknown/trailing JSON rejection, nil `scenes`, invalid +kinds, blank required strings, malformed source fields, candidate decoding of +semantically invalid but structurally decodable values, metadata, and +independent ownership of encoded/schema bytes where the existing codec +contract requires it. + +Do not duplicate shape-validator and durable JSON Schema coverage exhaustively +in the codec suite. + +### Completion gate + +Run: + +```sh +gofmt -w internal/modules/dnd/types.go internal/modules/dnd/codec/scenedescriptions/*.go +go test ./internal/modules/dnd/codec/scenedescriptions +``` + +The repository must compile with no production registration added yet. + +## Stage 2: Add The Private Model Contract, Prompt, And Extractor + +### Work + +1. Add `internal/modules/dnd/extract/scenedescriptions/` using the established + D&D extractor package layout: assets, model, schema loader, prompt + registration/fingerprints, constructor/options handling, extractor, and + deterministic mapper. +2. Define a private Go response with exactly `kind`, `title`, and `summary`. +3. Embed + `assets/schemas/dnd_scene_descriptions_llm.v1.json` with the fixed strict + private contract. +4. Add `assets/prompts/dnd.scene_descriptions.yaml`, `task.md`, and + `instructions.md` with the fixed message order and policy above. +5. Use `shared.ReferenceSlots` with package-owned descriptions for optional + `players`, `party`, and `glossary` slots. The extractor and its module spec + expose no other slots. +6. Map the private response to exactly one durable record using the current + chunk ID and exact `Chunk.Ref`. + +### Tests + +At the package owners, cover: + +- private schema identity, strict shape, closed enum, and rejection of + application-owned or extra fields; +- prompt and response-schema asset registration and fingerprints; +- prepared message roles/order, compatible shared-asset use, explicit empty + campaign placeholders, one transcript rendering, and transcript-last + placement; +- module spec, capabilities, optional reference slots, option rejection, and + registration; +- request validation and provider failure; +- exact application-owned ID/range mapping; +- title/summary trimming and non-repair of kind; and +- prompt inputs using only current chunk material plus optional references. + +Do not assert prompt byte length, shared-prefix length, exact generated prose, +or provider output quality. + +### Completion gate + +Run: + +```sh +gofmt -w internal/modules/dnd/extract/scenedescriptions/*.go +go test ./internal/modules/dnd/extract/scenedescriptions +``` + +Do not register the extractor in the production D&D family in this stage. + +## Stage 3: Add Deterministic Normalization And Validation + +### Work + +1. Add `internal/modules/dnd/normalize/scenedescriptions/` with the fixed + normalization and conflict behavior. Expose no options or reference slots. +2. Add the four validator packages under + `internal/modules/dnd/validate/scenedescriptions/` using the exact keys and + responsibilities above: + `shape`, `source_refs`, `source_relatedness`, and `invariants`. +3. Give each normalizer/validator a checkpoint policy fingerprint that changes + when its owned deterministic behavior changes. +4. Use shared source and diagnostic helpers where their semantics match. Keep + the scene-specific lexical stopword policy local to the relatedness + validator. + +### Tests + +Cover normalization of whitespace and source ordering; ID tie-breaking; exact +deduplication; both conflict classes; preservation of same-range, +same-content, different-ID records; invalid source/shape/kind handling; input +ownership; cancellation/nil requests; and module registration metadata. + +Cover validators at their stable owners, including: + +- extraction requires exactly one scene; +- extraction ID and range must exactly equal the current chunk; +- normalization validates current-source membership without a chunk; +- shape and source-reference deferral between validators; +- supported and unsupported kinds; +- invariant order, trimming, duplicate, and conflict checks; +- separate bounded title and summary relatedness warnings; +- zero-significant-token warnings; +- transcript-only relatedness even when campaign references contain matching + text; and +- safe bounded diagnostics. + +Use table-driven cases where several inputs exercise one behavior. Do not +replicate complete codec or JSON Schema test matrices in every validator. + +### Completion gate + +Run: + +```sh +gofmt -w internal/modules/dnd/normalize/scenedescriptions/*.go internal/modules/dnd/validate/scenedescriptions/*/*.go +go test ./internal/modules/dnd/normalize/scenedescriptions ./internal/modules/dnd/validate/scenedescriptions/... +``` + +Production registration remains deferred until all leaf packages pass. + +## Stage 4: Compose The Production Lane + +### Work + +1. Update `internal/modules/dnd/register/modules.go` to register the codec, + extractor, append-order specialization, normalizer, typed no-op normalizer, + and prompt/schema assets. +2. Add the list append/clone helper in + `internal/modules/dnd/register/merge.go`. +3. Update `internal/modules/dnd/register/validators.go` to register all four + validators plus typed always-accept and always-reject specializations. +4. Update `internal/modules/dnd/register/chains.go` with the exact extract and + normalize chains above. +5. Extend `internal/modules/dnd/register/register_test.go` to assert the new + keys, artifact-kind specializations, assets, module specs, and exact + production chains. +6. Add one offline assembled production workflow test under `internal/cli/` + using the real family registration/configuration/runner/output path and a + fake structured LLM. It must exercise at least two accepted scene chunks and + prove: + - one model description is mapped to each chunk's ID and exact range; + - append merge and normalization produce source order; + - model-owned fields survive with only specified trimming; + - durable schema identity and output payload are correct; and + - no NPC or generated-artifact reference is required. + +Keep this as one representative assembled test. Do not add equivalent +end-to-end cases in multiple packages. + +### Completion gate + +Run: + +```sh +gofmt -w internal/modules/dnd/register/*.go internal/cli/dnd_scene_descriptions_contract_test.go +go test ./internal/modules/dnd/register ./internal/cli +``` + +At this gate the lane must be selectable through production registration, with +no direct production import outside the D&D family registrar and CLI +composition root. + +## Stage 5: Publish Current Documentation And Maintained Example + +### Work + +Only after Stage 4 is green: + +1. Add `docs/integrations/dnd-scene-description-artifacts.md` as the canonical + durable contract. Document identities, exact JSON shape, scene kinds, + application-owned ID/range mapping, merge/normalization rules, conflict + behavior, advisory relatedness warnings, optional campaign references, and + a copyable lane example. +2. Update `docs/config.md` with the maintained example link; extractor, + normalizer, and validator keys; optional `players`, `party`, and `glossary` + slots; and the exact default chains. +3. Update `docs/internal/modules.md` and `docs/internal/overview.md` for the new + codec/extractor/normalizer/validators and family registration. +4. Update `docs/internal/llm.md` with the prompt manifest and the deliberate + omission of the citation-oriented shared extraction-evidence message. Make + clear that compatible shared messages remain canonical shared assets and the + transcript remains last. +5. Update `docs/internal/pipeline.md` only where its current artifact-lane or + source-attachment inventory requires the new lane; do not describe a new + stage or dependency. +6. Update `docs/integrations/json-output.md` to link the new durable artifact + contract alongside the other typed D&D schemas. +7. Add `examples/dnd-scene-descriptions.config.yml`. It should use + `chunk: dnd/scenes`, one scene-description artifact lane, and + `normalize: dnd/scene-descriptions`. Keep it minimal and do not present + chunk annotations as the durable description artifact. +8. Mark `docs/roadmap/dnd-scene-descriptions.md` as `Status: Implemented`, + add a link to the canonical integration document near the top, and remove + the implemented “Extract D&D Scene Descriptions” entry from + `docs/roadmap/future.md`. Leave chunker minimization and combat gating in + future work. + +### Documentation checks + +Verify every new selectable key, schema identity, enum, validator order, and +example against production code. Check all relative Markdown links and run the +maintained example configuration through config loading or the closest +existing example/config test. Do not document private package mechanics in the +integration contract. + +## Stage 6: Final Quality Gate + +1. Review the diff for accidental changes to `dnd/scenes`, combat gating, + framework stages, generated-reference behavior, or unrelated D&D contracts. +2. Confirm all new JSON objects reject unknown fields and the model is never + asked to return application-owned identity/evidence. +3. Confirm diagnostics contain no transcript or campaign-reference content. +4. Confirm the prompt uses shared assets by reference and that the transcript + is the final message; do not introduce a prompt-length or exact-prefix + detector. +5. Run: + +```sh +gofmt -l . +go test ./... +go vet ./... +git diff --check +``` + +`gofmt -l .` must produce no output. Do not mark the feature roadmap +implemented until all commands pass and the current documentation describes +the shipped behavior. + +## Open Questions + +None. The accepted feature roadmap and the fixed decisions in this plan are +sufficient to implement all stages without further product or architecture +choices.