Add feature roadmap and implementation plan for D&D spell extraction and validation upgrades

docs/roadmap/future.md

@@ -2,49 +2,138 @@
 
 Current Notarius behavior is documented in the canonical README, CLI,
 configuration, operations, internal, and integration docs. This roadmap records
-future work only.
+future work only. Items are ordered roughly by current value and specificity,
+not as committed release dates.
 
-## Candidate Product Work
+## Near-Term D&D Pipeline
+
+### Solidify Spell Extraction
+
+- Implement the immediate overlay-capable catalog-grounding and
+  deterministic-validation baseline in
+  [D&D Spell Extraction Quality](dnd-spell-extraction.md), using ordinary
+  `retries: 2` behavior as the comparison point.
+- After evaluating that baseline, reconsider the roadmap's deferred structured
+  diagnostics, retryability classification, repair-capable extractor contract,
+  hybrid repair/fresh retry policy, and narrowly scoped semantic LLM
+  validation.
+- Replace the no-op spell normalizer with a D&D-specific implementation that
+  canonicalizes recognized spell names, performs only high-confidence repairs,
+  retains uncertain values for review, and emits scoped warnings.
+- Conservatively collapse duplicate spell casts when canonical spell, caster,
+  and source evidence establish that they represent the same event. Merge and
+  canonicalize their source references rather than relying on spell name alone.
+- Maintain a small set of human-reviewed transcripts and outputs for prompt,
+  validator, and normalizer development. Treat model-quality review as an
+  iterative human evaluation aid, not a deterministic correctness gate.
+
+### Add Sequential D&D Artifacts
+
+- Add NPC extraction, including identity, aliases, descriptions, relationships,
+  and source evidence suitable for use as later reference material.
+- Add combat-turn extraction with explicit event and source-reference
+  semantics. Use earlier NPC output as a reference to improve participant
+  identity and consistency.
+- Add narrative extraction for scene summaries, party actions, and NPCs
+  encountered when that output proves useful beyond the dedicated NPC artifact.
+- Define the preferred operational sequence for independent pipelines on the
+  same transcript. The initial direction is NPCs first, followed by spells and
+  combat turns as appropriate, with earlier JSON artifacts supplied to later
+  runs as references.
+- Keep this sequencing operator- or script-driven initially. Do not require a
+  general DAG or concurrent cross-lane reconciliation model.
+
+### Improve D&D Scene Classification
+
+- Extend scene annotations with classifications that downstream extractors can
+  use, including reliable combat and narrative indicators.
+- Strengthen the scene prompt so every scene containing combat turns is marked
+  as combat, and add validation capable of detecting missing or inconsistent
+  combat classifications.
+- Allow the combat extractor to no-op for chunks that are not classified as
+  combat, avoiding unnecessary model calls where practical.
+- Allow a narrative extractor to select the corresponding scene classification
+  rather than processing every chunk indiscriminately.
+- Reassess whether one shared scene plan provides enough context for NPC,
+  spell, combat, and narrative pipelines after these extractors have real-world
+  usage. Add more complex chunking only in response to demonstrated failures.
+
+## Shared Normalization And Quality Work
+
+### Generic LLM-Assisted Deduplication
+
+- Add a reusable normalizer that asks an LLM to identify duplicate sets in a
+  list and propose one replacement element for each set.
+- Define the minimum domain-neutral input contract, initially an ordered list
+  whose elements have stable unique IDs. Artifact-kind registrations or
+  adapters may expose that structure without moving domain rules into the
+  generic package.
+- Keep mutation deterministic: parse and validate the model's duplicate groups,
+  require every referenced ID to exist, reject overlapping or malformed groups,
+  prevent unrelated insertion or deletion, and apply only approved replacement
+  operations in code.
+- Preserve provenance needed for audit and downstream validation, and emit
+  warnings describing every collapsed group.
+- Evaluate batching and context-window limits before applying the normalizer to
+  large artifact collections.
+
+The model may use its own domain knowledge to judge semantic duplication; the
+generic implementation is responsible only for the common proposal contract,
+safety checks, and deterministic application of accepted changes.
+
+### Validation And Review
+
+- Add domain validators and production default chains alongside each new D&D
+  artifact.
+- Add production LLM-backed validators only when a concrete review policy
+  benefits from model judgment and deterministic checks are insufficient.
+- Add validator diagnostics and timing summaries if operators need more detail
+  than the current [durable output bundle](../integrations/json-output.md)
+  provides.
+- Add validator compatibility metadata if deployments need config-time proof
+  that a validator is suitable for a particular stage, module, or artifact
+  kind.
+- Add media-type validators when non-JSON artifact representations are
+  introduced.
+
+## Reference And Sequential-Pipeline Evolution
+
+- Make prior-run artifacts easier to bind as references without changing the
+  existing module-facing reference-item contract.
+- Add structured or parsed references, such as typed NPC registries, rosters,
+  or spell catalogs, when opaque UTF-8 prompt material is no longer sufficient.
+- Add per-slot or per-chunk inclusion policies so large references are not
+  repeated in every prompt unnecessarily.
+- Add token budgeting and model context-window management for reference
+  content.
+- Add reference caching, preprocessing, summarization, embedding, or retrieval
+  only when reference size and observed model behavior justify them.
+- Consider non-file reference producers for prior-run artifacts, derived
+  summaries, or entity registries after manual sequential composition becomes
+  burdensome.
+
+## Blue-Sky Platform And Operations
+
+These ideas are intentionally less specified. Promote one into an earlier
+section only after a concrete workflow, contract, and priority emerge.
+
+### Platform Extensions
 
 - Additional input adapters, such as Markdown or note-export formats.
-- Additional D&D extractors beyond spell casts.
-- Add non-file reference producers, such as prior-run artifacts, derived
-  summaries, or entity registries, without changing module-facing reference
-  item contracts.
-- Add token budgeting and model context-window management for reference content.
-- Add per-slot or per-chunk inclusion policies so modules can avoid repeating
-  large reference content in every prompt when that becomes important.
-- Add structured or parsed references, such as typed roster schemas, when a
-  module has a clear need for more than opaque UTF-8 text.
-- Add reference caching, preprocessing, summarization, embedding, or retrieval
-  if references become large enough to require preprocessing.
-- Cross-lane entity normalization.
-- Cross-chunk semantic deduplication.
-- Additional validator packages and production default chains for future
-  modules.
-- Production LLM-backed validators when there is a concrete review policy that
-  benefits from model judgment.
-- Validator diagnostics and timing summaries if operators need more detail than
-  the current [durable output bundle](../integrations/json-output.md) provides.
-- Media-type validators for non-JSON module outputs when such modules are
-  introduced.
-- Validator compatibility metadata if real deployments need config-time
-  enforcement that a validator is suitable for a specific stage or module.
-- Batching or context-window controls for LLM-backed validators if validator
-  inputs become large enough to require them.
 - Additional output encoders.
+- Concurrent cross-lane entity normalization or broader workflow composition.
+- Batching or specialized context-window controls for LLM-backed validators.
 
-## Candidate Operational Work
+### Distribution And Operations
 
 - Packaged release artifacts for alpha distribution.
 - A documented versioning and release process.
-- Optional generated example output fixtures with a regeneration procedure.
-- Additional diagnostics or reporting views if operator workflows need them.
+- Optional generated example-output fixtures with a regeneration procedure.
+- Additional diagnostics or reporting views.
 
-## Candidate Workspace Work
+### Workspace And Storage
 
 - Default-idempotent run behavior with an explicit force override.
 - Remote workspace storage.
-- Workspace garbage collection.
-- Workspace archival policy.
+- Workspace garbage collection and archival policies.
 - Cross-machine checkpoint reuse.