Add the D&D harmonization roadmap and implementation plan

docs/roadmap/dnd.md

@@ -0,0 +1,186 @@
+# D&D Module Harmonization And Prompt Reuse
+
+This roadmap records proposed improvements to the spell, NPC, and combat-turn
+pipeline lanes. Current implemented behavior remains documented in the
+[module internals](../internal/modules.md) and
+[LLM runtime internals](../internal/llm.md). The work below is not yet an
+implemented contract.
+
+The three lanes already share the same overall decomposition: a typed
+extractor, durable codec, merger, normalizer, shape and provenance validators,
+domain-specific validation, embedded prompt assets, and central production
+registration. Spell catalog resolution, NPC identity and registry support, and
+combat normalization invariants are intentional domain differences. The goal
+is to remove mechanical drift without hiding those differences behind a broad
+generic abstraction.
+
+## Target State
+
+### Prompt Reuse Is A Rendered-Request Invariant
+
+Provider prompt caching depends on the request prefix being byte-for-byte
+identical. Similar prose, duplicated files, or equivalent structured values are
+not sufficient. Treat message role, content bytes, ordering, cache-control
+metadata, input rendering, and any provider-visible separators as part of the
+cache identity.
+
+The target prompt layout has the longest valid common prefix before any
+module-specific message:
+
+- Put shared static system, extraction-evidence, and in-world-identity policy
+  before dynamic transcript and campaign-reference messages so reuse does not
+  depend on identical transcript content.
+- Keep the shared transcript and campaign-reference messages in the same roles
+  and order in every D&D extractor.
+- Keep cache-control declarations identical on corresponding messages.
+- Define one canonical ordering and serialization for shared named inputs.
+  Continue resolving the deprecated `roster` input into the canonical `party`
+  input before prompt rendering.
+- Keep genuinely universal extraction rules in shared message assets. These
+  rules include using only transcript units as evidence, treating references as
+  disambiguation, returning schema-conforming JSON only, using integer
+  `start_unit_id` and `end_unit_id` values, omitting `source_id` for the mapper
+  to assign, citing all factual claims, and preferring narrow ranges over broad
+  ranges that bridge unrelated conversation.
+- Use one shared in-world identity message for lanes whose artifacts name
+  actors or participants. It contains only rules that spells, NPCs, and combat
+  turns can follow verbatim; artifact-specific identity, alias, target, and
+  relationship rules remain local.
+- Keep rules shared by only a subset in subset-specific shared assets. The
+  immediate-declaration-and-resolution boundary and NPC-registry grounding are
+  shared by spells and combat. The NPC extractor does not consume a prior NPC
+  registry.
+- Place subset-specific and module-specific messages only after the longest
+  useful all-lane prefix. Place schema-specific task instructions last.
+- Avoid inserting a nominally shared message when its rendered input or
+  wording differs by lane. Factor the common bytes into one message and leave
+  the differences in later messages instead.
+
+Tests inspect the fully rendered Scriptorium request boundary. Source-level
+message identity alone is not treated as proof of cache identity.
+
+### Exact Prompt And Input Identity Is Protected
+
+Focused tests at the narrowest stable boundary expose the fully prepared
+provider-neutral request.
+
+- Render each extractor prompt with the same transcript and campaign
+  references and assert that the intended common message prefix has identical
+  roles, content bytes, ordering, and cache-control metadata.
+- Assert that spells and combat turns render identical NPC-registry messages
+  for the same bound or unbound registry.
+- Verify common input material identity, including name, media type, content,
+  digest, origin URI, size, empty-value representation, reference ordering,
+  and `roster` fallback behavior.
+- Test both identical and intentionally different chunks and reference sets so
+  the test proves the cache boundary rather than merely snapshotting one
+  request.
+- Add an explicit assertion for the length of the common prefix. A new
+  module-specific message inserted inside that prefix should require deliberate
+  review.
+- Test that every prompt fingerprint includes exactly the assets actually
+  rendered by that prompt. The current grouped reference hash helper should be
+  replaced or refined so a module does not fingerprint an unused shared asset,
+  while no used asset is omitted.
+- Prefer semantic assertions over complete prompt snapshots, except for the
+  common rendered prefix whose exact bytes are the behavior under protection.
+
+### Stable Extraction Preparation Is Centralized
+
+Move the three identical chunk-input preparation implementations into a D&D
+shared helper. The helper should clone the supplied material, fall back to the
+chunk content, verify byte equality, and fill the canonical name, media type,
+and size without retaining mutable request data.
+
+Nil context, cancellation, source, chunk, and empty-unit checks remain local so
+typed result handling and module error context stay explicit. Do not wrap the
+complete LLM call, response DTO mapping, catalog preparation, registry
+preparation, or manifest metadata in a generic extractor framework.
+
+Use the shared helper as the single source of the `transcript` prompt material
+so identical input requests cannot drift between lanes.
+
+### Validator Checkpoint And Composition Policy Is Aligned
+
+- Add explicit versioned policy checkpoint fingerprints to the spell shape,
+  source-reference, and source-relatedness validators. Bump a policy version
+  whenever acceptance, rejection, warning, or diagnostic-selection behavior
+  changes.
+- Shape validators own malformed artifact shape; later validators defer when
+  shape is invalid.
+- Make all source-reference validators collect bounded diagnostics through the
+  shared D&D diagnostics package rather than mixing first-error and aggregate
+  behavior.
+- Use artifact-qualified reason codes consistently unless a reason code is
+  intentionally a stable cross-artifact contract.
+- Add a compact cross-lane validator contract test covering deterministic
+  execution class, strict empty options, registration, policy fingerprinting,
+  prerequisite behavior, and diagnostic bounds.
+
+### Source-Evidence Traversal Is Shared
+
+Extract a D&D helper that validates source ranges and returns cited units or
+text once, in source-document order, without repeating units covered by
+overlapping ranges. Use it from spell, NPC, and combat relatedness validators.
+
+Keep matching policy artifact-specific:
+
+- spell matching may use canonical catalog names and aliases;
+- NPC matching may use NPC comparison keys and aliases; and
+- combat matching may apply separate actor and declaration heuristics.
+
+Review Unicode normalization, apostrophe handling, word boundaries, short-name
+false positives, multiword identities, and overlapping ranges with shared
+table-driven fixtures. Relatedness remains a warning heuristic and should not
+be presented as proof that every semantic claim is supported.
+
+## Structural Cleanup
+
+### Lane Registration
+
+Reduce repetition in the central D&D registrar with focused registration
+helpers grouped by modules, validators, prompt assets, and default chains.
+Retain artifact-specific append and deep-clone behavior in the registrar,
+central ownership of validator ordering, and explicit typed registration.
+Avoid reflection and heterogeneous erased lane descriptors.
+
+### Naming And Package Conventions
+
+Adopt consistent conventions for package aliases, module keys, artifact kinds,
+prompt IDs, schema IDs, reason codes, policy IDs, metadata fields, and
+checkpoint fingerprint names. Compatibility-sensitive identifiers should
+change only through an explicit migration; internal aliases can be harmonized
+independently.
+
+### Schemas And Assets
+
+- Keep private LLM response schemas distinct from durable artifact codec
+  schemas. They represent different trust and compatibility boundaries.
+- Keep source-reference schema definitions package-owned in this work; do not
+  add schema composition or generation machinery solely to deduplicate them.
+- Prefer one embedded shared asset over synchronized copies whenever content
+  must be identical for prompt caching.
+- Make prompt fingerprints derive from an explicit prompt asset manifest, or
+  from the prepared prompt definition, so message composition and provenance
+  cannot drift independently.
+
+Prompt factoring must not be accepted solely because cache reuse improves.
+Retain or restore module-specific wording when evaluation shows a meaningful
+quality regression. Record cache observations using non-secret request and
+usage metadata rather than prompt or transcript payloads.
+
+## Completion Criteria
+
+This roadmap is complete when:
+
+- the three lanes expose the same stable structural conventions while keeping
+  documented domain differences local;
+- shared provider-visible messages and inputs are produced from one source and
+  verified byte-for-byte at the rendered-request boundary;
+- the common prompt prefix is deliberate, tested, and as long as extraction
+  quality permits;
+- every validator policy that affects reusable results participates in
+  checkpoint identity;
+- source-reference traversal and diagnostics no longer drift between lanes;
+- registration remains explicit and type-safe; and
+- focused D&D tests plus repository-wide tests, vetting, and the CLI build pass.