Plan D&D scene description extraction

docs/roadmap/dnd-scene-descriptions.md

@@ -0,0 +1,259 @@
+# D&D Scene Descriptions
+
+Status: Accepted
+
+## Purpose
+
+Add a focused D&D artifact that describes each accepted scene with a bounded
+classification, a concise title, and a concise summary. This is the explicit
+home for the small amount of source-bounded synthesis useful to downstream
+applications; scene chunking should not remain the long-term owner of those
+descriptive fields.
+
+The feature follows
+[ADR-0009](../adr/0009-minimal-evidence-grounded-extraction-artifacts.md).
+It is intentionally richer than a fact-only occurrence extractor, but every
+model-produced field has a specific scene-description purpose and is grounded
+by the exact accepted chunk range.
+
+## Desired End State
+
+A D&D pipeline that uses accepted scene chunks can add a
+`dnd/scene-descriptions` artifact lane. The extractor runs once per accepted
+chunk and returns exactly one description for that chunk. The merged,
+normalized artifact is an ordered list with one independently identified and
+evidenced scene record for every successful chunk extraction.
+
+Use these production identities:
+
+| Concern | Identity |
+| --- | --- |
+| Extractor key | `dnd/scene-descriptions` |
+| Normalizer key | `dnd/scene-descriptions` |
+| Artifact kind | `dnd/scene-description-list` |
+| Durable schema ID | `notarius.dnd.scene_descriptions` |
+| Durable schema name | `notarius_dnd_scene_descriptions_v1` |
+| Durable schema version | `v1` |
+| Durable media type | `application/json` |
+| Private prompt ID | `dnd.scene_descriptions` |
+| Private response schema ID | `notarius.dnd.scene_descriptions.llm` |
+| Private response schema version | `v1` |
+
+The lane uses the existing typed artifact, append-order merge, codec,
+registration, validator-chain, prompt-asset, retry, and output conventions. It
+does not introduce a new pipeline stage or a dependency on the durable
+chunk-map export.
+
+## Durable Artifact Contract
+
+The durable payload is a strict object containing a `scenes` array. The array
+is ordered by accepted source position. Each scene contains exactly:
+
+- `id`: the accepted chunk ID, copied deterministically;
+- `source_ref`: the accepted chunk's exact inclusive current-source range;
+- `kind`: one value from the closed scene-kind vocabulary;
+- `title`: a concise source-grounded label for the scene; and
+- `summary`: a concise source-grounded account of the scene's main activity.
+
+The accepted chunk ID is the scene ID. Do not add a second `chunk_id` field or
+generate an unrelated scene identifier: this artifact has exactly one record
+per accepted scene chunk, so the chunk identity is already the stable identity
+needed for joining with chunk maps and other range-grounded artifacts.
+
+Every fixed object rejects unknown fields. `id` and `source_ref` are
+application-owned evidence and identity fields. The model must not emit or
+reproduce chunk IDs, source IDs, unit IDs, or ranges.
+
+The extractor maps one successful private response into a single-element
+durable `scenes` array before extraction validation. It attaches the current
+chunk ID and range at that boundary rather than relying on normalization to
+manufacture required provenance.
+
+## Model-Owned Fields
+
+The private model response contains exactly:
+
+- `kind`;
+- `title`; and
+- `summary`.
+
+It represents one scene object, not a list. An accepted chunk is already the
+scene boundary, so the extractor must neither split the chunk nor return
+multiple descriptions. It must not return an empty result; inability to
+classify or describe the chunk is an extraction failure subject to the
+configured retry and rejection policy.
+
+`title` should be a short, distinguishing phrase rather than a sentence,
+chapter number, or generic label such as “Scene.” It may use names and places
+established by the transcript or optional campaign references, but must not
+invent a proper noun to make the title more colorful.
+
+`summary` should briefly state the main activity and material transition or
+outcome established within the chunk. It must not add analysis, inferred
+motives, hidden state, future consequences, relationship claims, or facts from
+outside the chunk. Do not require a structured participant list, encounter
+outcome, location, mood, or confidence field.
+
+## Scene-Kind Vocabulary
+
+Use this closed vocabulary:
+
+| Kind | Meaning |
+| --- | --- |
+| `combat` | Active combat materially organizes the scene, including initiative-like exchanges or sustained hostile action. Planning a fight or discussing a completed fight is not combat by itself. |
+| `narrative` | Current-session in-world play that is not principally active combat, a prior-session recap, or sustained out-of-character session discussion. This includes exploration, travel, dialogue, investigation, planning in character, and aftermath. |
+| `recap` | The scene's organizing purpose is to recount events from a previous session for the table. An in-world character recounting history during current play remains narrative. |
+| `meta` | The scene's organizing purpose is sustained out-of-character discussion about the game or session rather than advancing current in-world play. |
+
+`narrative` is the default category for actual current-session gameplay that
+does not meet another definition. The model must choose exactly one kind and
+must not emit an uncertainty or mixed category.
+
+## Residual Mixed-Scene Policy
+
+Scene chunking should normally separate sustained transitions between kinds.
+When an accepted chunk nevertheless contains more than one kind, classify the
+activity that best describes the chunk's organizing purpose and main
+current-session progression:
+
+1. use `combat` when active combat is a substantive central activity, even if
+   the chunk includes brief setup, rules clarification, or immediate aftermath;
+2. use `recap` when recounting a previous session is the chunk's primary table
+   purpose, not merely a brief reminder before current play;
+3. use `meta` when sustained out-of-character session discussion is primary and
+   the chunk contains no more than incidental in-world progression; and
+4. use `narrative` for all remaining current-session in-world play.
+
+Brief table talk, dice resolution, rules clarification, jokes, or administrative
+comments are incidental to the enclosing gameplay scene and do not make it
+`meta`. A short recollection used to orient current action does not make a scene
+`recap`.
+
+This precedence resolves residual ambiguity for description. It does not alter
+the accepted chunk boundary or authorize the extractor to conceal a
+demonstrably poor scene plan.
+
+## Evidence, References, And Prompt Boundary
+
+The exact accepted chunk range is the evidence for all three model-owned
+fields. Do not ask the model for finer-grained source ranges, and do not use
+campaign references as evidence.
+
+The lane requires no generated artifact reference. Existing optional D&D
+campaign references may be exposed through the established shared slots when
+they help disambiguate names or setting terms in titles and summaries. They
+must not introduce events, participants, or claims absent from the current
+chunk. Do not require an NPC registry: structured participants remain owned by
+NPC and NPC-interaction artifacts.
+
+Follow the established D&D prompt ordering and prompt-cache boundary:
+
+1. shared D&D system and identity policy;
+2. stable campaign references, including explicit empty placeholders;
+3. lane task and classification instructions; and
+4. the variable current-chunk transcript last.
+
+Factor wording genuinely shared with other D&D modules into existing shared
+prompt assets. Keep scene-kind definitions and mixed-scene policy in
+scene-description-owned assets rather than broadening unrelated extractors.
+Do not render the existing shared extraction-evidence asset in this lane: that
+asset instructs the model to emit source ranges, while scene identity and the
+entire accepted chunk range are attached deterministically by the application.
+This is a deliberate exception to the shared extraction prompt sequence, not
+permission to duplicate compatible shared wording in lane-owned assets.
+
+## Merge And Normalization
+
+Use the existing append-order merge behavior. Normalization remains
+deterministic and must:
+
+- validate every record against the current source document;
+- require an exact closed-vocabulary kind without case-folding or repairing an
+  invalid model value;
+- trim surrounding title and summary whitespace without otherwise rewriting
+  their prose;
+- order scenes by source-document position, with accepted chunk ID as the
+  deterministic tie-breaker;
+- remove only completely identical duplicate records; and
+- reject conflicting records that reuse one scene ID or exact source range with
+  different model-owned content.
+
+Normalization must not combine adjacent scenes, regenerate titles or summaries,
+change classifications through heuristics, or infer missing records. Exact
+deduplication is defensive cleanup, not semantic reconciliation.
+
+## Validation And Quality
+
+Provide the same layered production validation used by other typed D&D
+artifacts:
+
+- valid JSON and strict private/durable shape;
+- exactly one private description and one mapped durable scene per chunk;
+- closed kind vocabulary;
+- non-empty, trimmed title and summary;
+- non-empty accepted chunk ID;
+- a current-source range exactly equal to the extraction chunk range;
+- source-relatedness checks for title and summary; and
+- strict durable JSON Schema validation after normalization.
+
+Shape, identity, enum, and source-range violations are deterministic
+rejections. Source-relatedness should initially remain warning-only unless
+evaluation establishes a reliable rejection rule. Diagnostics must be bounded
+and must not copy transcript or campaign-reference content.
+
+Evaluate the lane on a small human-reviewed transcript set covering all four
+kinds, incidental table talk, brief and sustained recap, combat setup and
+aftermath, mixed residual chunks, sparse scenes, proper-noun ambiguity, and
+unsupported synthesis. Review classification agreement, title usefulness,
+summary fidelity, evidence attachment, and performance on the cost-effective
+models Notarius is intended to support. Human evaluation is a development aid,
+not an exact-text CI oracle.
+
+Tests should protect structural contracts, deterministic mapping,
+normalization, validation, prompt assembly, registration, and one assembled
+production workflow. Do not assert exact generated prose or use brittle prompt
+length and shared-prefix change detectors.
+
+## Relationship To Scene Chunking
+
+This artifact becomes the intended owner of scene kind, title, and summary.
+The existing `dnd/scenes` chunker remains unchanged in this scope. Downstream
+consumers should prefer the normalized scene-description artifact over chunk
+annotations for kind, title, and summary.
+
+Removing descriptive fields from the chunker is a separate follow-on scope. It
+requires its own review of chunker prompt quality, warning/validator consumers,
+cache behavior, and the compatibility implications of changing accepted plan
+annotations. Do not couple that migration to the initial artifact lane.
+
+Likewise, do not make combat extraction conditional on scene kind in this
+scope. A later ordered-pipeline feature may consume the accepted normalized
+scene-description artifact and no-op combat extraction for non-combat scenes
+without making generic chunk materialization depend on D&D classification.
+
+## Documentation Ownership
+
+Once implemented, the durable contract belongs in `docs/integrations/`;
+selectable keys, optional campaign-reference bindings, and default validator
+chains belong in configuration documentation; and prompt, module, validator,
+and pipeline mechanics belong in current internal documentation. A maintained
+D&D example should demonstrate scene chunking and scene-description extraction
+without presenting chunk annotations as the durable description contract.
+
+## Non-Goals
+
+This scope does not:
+
+- change scene boundaries or run a second chunker;
+- ask the model for IDs, source ranges, segment numbers, or evidence selection;
+- add structured participants, locations, encounters, outcomes, moods,
+  confidence, or relationship fields;
+- duplicate NPC or NPC-interaction extraction;
+- require an NPC registry or another generated artifact;
+- enrich descriptions from campaign references when the chunk lacks support;
+- semantically merge or reconcile adjacent scene descriptions;
+- minimize the existing scene chunker in the same change;
+- gate combat extraction by scene kind;
+- make scene descriptions generated references for later steps;
+- change chunk-map export, chunk caching, or plan validation; or
+- introduce a DAG, a new pipeline stage, or a new reference mechanism.