Remove completed roadmap documentation and add future work to the long-term roadmap

docs/internal/llm.md

@@ -102,19 +102,52 @@ The D&D scene chunker and spell, NPC, and combat-turn extractors each declare an
 ordered prompt asset manifest. The manifest lists the package-owned YAML and
 Markdown files, then the exact shared fragments rendered by that prompt; the
 same ordered list drives both filesystem mounting and the prompt fingerprint.
-The three extraction prompts share system, extraction evidence, in-world
+Unused shared assets are neither mounted nor fingerprinted. Universal
-identity, transcript, and campaign-reference messages. Spell and combat prompts
+extraction-evidence and output policy lives only in the shared extraction
-then render immediate resolution and the ephemeral NPC registry before their
+assets; package-owned prompt files retain artifact-specific rules. The scene
-lane-specific material; the NPC prompt renders its task and instructions
+prompt keeps its separate output rule because it does not render the
-instead. The scene prompt uses system, transcript, and campaign-reference
+extraction-evidence asset.
-messages before its task and instructions. Identity, transcript, and
+
-campaign-reference messages are ephemeral in the extraction prompts; the scene
+### D&D Extraction Prompt Ordering And Cache Boundaries
-prompt marks transcript and campaign references ephemeral, and the NPC registry
+
-is ephemeral where spell and combat prompts use it. Unused shared assets are
+D&D extraction prompts order messages from the most reusable content to the
-neither mounted nor fingerprinted. Universal extraction-evidence and output
+most variable content. New extraction lanes use these tiers in order:
-policy lives only in the shared extraction assets; package-owned prompt files
+
-retain artifact-specific rules. The scene prompt keeps its separate output rule
+1. universal shared content, including the system, extraction-evidence, and
-because it does not render the extraction-evidence asset.
+   in-world identity messages;
+2. stable campaign or run context shared across lanes, including campaign
+   references;
+3. stable subset- and lane-specific context and instructions, including an NPC
+   registry, catalog, task, or extraction instructions when applicable;
+4. the chunk transcript as the final user message.
+
+This ordering lets requests reuse the longest identical prefix before the
+per-chunk transcript changes. Cache reuse requires the preceding message
+sequence and content to be exactly identical; semantic similarity is not
+sufficient. Cache boundaries belong at the ends of reusable stable tiers,
+subject to the provider's cache-boundary limit. The shared identity and
+campaign-reference messages form the first two extraction boundaries. Spell
+and combat prompts add a boundary at the shared NPC registry. Each extraction
+prompt places its final boundary on its lane-specific instructions, immediately
+before the transcript. The transcript does not carry cache control because no
+reusable content follows it.
+
+Accordingly, the common prefix of all three extraction prompts is system,
+extraction evidence, identity, and campaign references. The NPC prompt then
+renders task, instructions, and transcript. Spell renders immediate resolution,
+NPC registry, catalog, task, instructions, and transcript. Combat renders
+immediate resolution, NPC registry, task, instructions, and transcript. The
+scene chunker is not an extraction lane: it retains its separate system,
+transcript, campaign-reference, task, and instruction order and marks its
+transcript and campaign-reference messages ephemeral.
+
+Shared wording belongs in the canonical assets under
+`internal/modules/dnd/shared`; extraction packages reference those assets in
+their manifests instead of copying similar text into package-local files.
+Package-local assets contain only lane-specific content. An extraction lane may
+depart from the tier order only when prompt-quality evidence or a provider
+constraint makes the exception necessary; document the exception and rationale
+here when it becomes implemented behavior.
 
 Schema helpers load embedded JSON Schema with identity and digest metadata,
 return defensive copies, and expose a diagnostics map that omits schema bytes.

docs/internal/modules.md

@@ -51,15 +51,12 @@ declarations, prompt-input assembly, and source-unit/citation helpers belong in
 D&D scene chunker and spell, NPC, and combat-turn extractors use ordered
 package-local prompt manifests for both rendering and prompt fingerprinting, so
 only the shared fragments each prompt actually renders participate in either
-operation. The three extraction prompts share the order system, extraction
+operation. Extraction prompts place stable shared and lane-specific context
-evidence, in-world identity, transcript, and campaign references, with
+before the variable transcript and use shared assets for wording common across
-ephemeral cache control on identity, transcript, and references. Spell and
+lanes. The canonical ordering and cache-boundary policy is documented in
-combat prompts append immediate resolution and an ephemeral NPC registry before
+[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). Stage
-lane-specific material; the NPC prompt appends its task and instructions. The
+contracts expose only Notarius structured-completion types, not Scriptorium
-scene prompt uses system, transcript, and campaign references before its
+public types.
-lane-specific task and instructions, with ephemeral cache control on transcript
-and references. Stage contracts expose only Notarius structured-completion
-types, not Scriptorium public types.
 
 The shared `ChunkPromptMaterial` helper owns common transcript material
 preparation for the spell, NPC, and combat-turn extractors. It clones supplied
@@ -240,15 +237,14 @@ assigns source identity and deterministic NPC IDs, and preserves source
 references for deterministic validation. It uses the shared campaign
 references only for disambiguation and does not consume the optional NPC
 registry slot. Its prompt and private response schema are package-owned. The
-prompt uses the common evidence, identity, transcript, and campaign-reference
+prompt follows the shared D&D extraction ordering and cache policy documented
-messages, then the NPC-specific task and instructions; only the identity,
+in [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries).
-transcript, and campaign-reference messages carry ephemeral cache control.
 
 ### `internal/modules/dnd/extract/combatturns`
 
 The combat extractor prepares one structured request per supplied chunk using
-the shared extraction-evidence, identity, transcript, campaign-reference,
+the shared extraction-evidence, identity, campaign-reference,
-immediate-resolution, and NPC-grounding prompt inputs. It
+immediate-resolution, NPC-grounding, and transcript prompt inputs. It
 maps the private response to `dnd.CombatTurnList`, assigns the current source
 identity, removes exact duplicate source ranges, and orders turns by valid
 source-document position while preserving malformed candidate fields for
@@ -256,11 +252,11 @@ deterministic validators. Its package-owned private response schema enforces
 only the structural JSON envelope; semantic artifact constraints remain with
 the validator chain. Its prepared metadata and checkpoint fingerprints contain
 only prompt/schema/mapping identities plus an optional NPC registry digest.
-The prompt renders immediate resolution and the NPC registry before the
+The prompt follows the shared D&D extraction ordering and cache policy
-combat-specific task and instructions; identity, transcript, campaign
+documented in
-references, and the NPC registry use ephemeral cache control. The package
+[LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries). The
-exposes typed registration and is included in the production D&D registrar with
+package exposes typed registration and is included in the production D&D
-the default combat extraction chain.
+registrar with the default combat extraction chain.
 
 The combat normalizer accepts only the optional structured NPC registry.
 Campaign references remain extractor-only LLM context and are not materialized
@@ -448,6 +444,10 @@ When adding a production module or validator:
 2. expose and test its spec, constructor, and registration function;
 3. keep format or domain parsing inside the concrete package;
 4. add package-owned prompt/schema assets when the extension is LLM-backed;
+   new LLM-backed D&D extraction modules must follow the stable-to-variable
+   prompt ordering, shared-asset ownership, and cache-boundary policy in
+   [LLM Runtime](llm.md#dd-extraction-prompt-ordering-and-cache-boundaries), or
+   document the implemented exception and its evidence there;
 5. register it through its package-family registrar and add a default chain
    there only when production policy requires one;
 6. add resolution and composition coverage for capabilities, options,

docs/roadmap/future.md

@@ -5,6 +5,122 @@ configuration, operations, internal, and integration docs. This roadmap records
 future work only. Items are ordered roughly by current value and specificity,
 not as committed release dates.
 
+## Near-Term: Ordered Pipeline Steps
+
+Allow one configured pipeline to contain multiple ordered execution steps so
+accepted artifacts from an earlier step can become generated references for
+later steps in the same run. This is a bounded extension of the fixed pipeline
+model, not an arbitrary DAG or general workflow language.
+
+Input parsing and chunk planning remain pipeline-wide. Each step selects one or
+more artifact lanes; every selected lane completes extraction, validation,
+merge, normalization, and validation before dependent later steps begin. Lanes
+within the same step remain independent and may execute concurrently. The
+runner exposes only accepted normalized artifacts across a step boundary; raw
+extracts, rejected outputs, and intermediate merge results cannot become
+downstream references.
+
+Generated-reference bindings must be explicit in resolved configuration. A
+binding identifies an earlier producing lane and one declared reference slot on
+a later consuming lane. Resolution must reject missing producers, references to
+the same or a later step, incompatible artifact kinds or media types, undeclared
+consumer slots, cycles, and ambiguous bindings. A configured external reference
+and a generated reference cannot bind the same effective target slot; reject
+that pipeline or runtime override instead of applying a precedence rule.
+Each effective target slot accepts at most one producer. One generated artifact
+may fan out to multiple compatible target slots in a later step; aggregation
+from multiple producers into one slot is deferred until a concrete use case
+defines deterministic semantics.
+
+Step-scoped reference defaults mirror existing pipeline-level reference
+defaults. A step binds an earlier artifact once, and the binding automatically
+applies to every target in that step that declares the named slot. Target-local
+bindings remain available when only one module should consume the artifact; do
+not combine a target-local and step-scoped binding for the same effective slot.
+A generated source uses a structured, unambiguous form rather than encoding a
+producer into a path-like string. The target configuration shape is:
+
+```yaml
+steps:
+  - id: identify-npcs
+    artifacts:
+      npcs:
+        extract: dnd/npcs
+        normalize: dnd/npcs
+  - id: grounded-events
+    references:
+      npcs:
+        artifact:
+          step: identify-npcs
+          lane: npcs
+    artifacts:
+      spells:
+        extract: dnd/spells
+        normalize: dnd/spells
+      combat:
+        extract: dnd/combat-turns
+        normalize: dnd/combat-turns
+```
+
+This snippet shows the step and reference portion of a pipeline; pipeline-wide
+input, chunk, output, and other references are omitted. Existing scalar
+reference values continue to mean external file paths; the structured
+`artifact` form means an accepted normalized artifact from the named earlier
+step and lane. Do not infer generated bindings from module keys, matching lane
+names, or D&D-specific knowledge in the framework.
+
+The handoff should use the producer artifact's canonical codec representation
+and retain its artifact kind, schema identity, media type, content digest, and
+producer provenance. Generated references provide context or disambiguation,
+not source evidence. They use the existing module-facing reference contract
+where possible; typed or domain-specific adapters may validate and prepare a
+reference without moving domain concepts into the pipeline framework.
+
+Pipeline identity, manifests, checkpoint dependencies, debug records, and
+errors must include step identity and generated-reference provenance. A
+downstream checkpoint is reusable only when the upstream artifact identity and
+content digest match. Resume should reconstruct an accepted upstream artifact
+through its registered codec instead of requiring the producing lane to run
+again when its checkpoint is reusable.
+
+### Dependency-aware resume and recomputation
+
+Treat generated-reference bindings as checkpoint dependencies. Reusing an
+earlier step is safe only when its existing checkpoint and codec identity are
+valid. Reusing a dependent step additionally requires an exact match for every
+upstream artifact kind, schema identity, media type, and canonical content
+digest it consumed. A changed, missing, rejected, corrupt, or incompatible
+producer artifact invalidates all transitive dependent checkpoints; the runner
+must never combine a newly produced upstream artifact with stale downstream
+output.
+
+Support selectively recomputing one configured step and all of its transitive
+dependents while retaining reusable independent and predecessor work. The
+operator-facing selection mechanism should identify a stable configured step,
+not individual internal stage operations. Resolution must reject a selection
+that would omit a required predecessor without a reusable accepted artifact.
+
+Manifests, checkpoint events, and diagnostics should distinguish work that was
+executed, reused, or invalidated and record a bounded non-secret reason for
+dependency-driven invalidation. Completion order must not affect invalidation,
+public artifact ordering, or the set of dependent steps selected for rerun.
+
+If a required producer finishes without an accepted normalized artifact, fail
+the entire run with a deterministic dependency error. Do not start any
+dependent step. Preserve the upstream rejection or empty-result outcome and
+step provenance in the failed run manifest so the cause remains auditable.
+
+The first production workflow is D&D NPC grounding:
+
+1. the first step runs the NPC lane through accepted normalized output; and
+2. the second step runs spell and combat-turn lanes, binding that NPC artifact
+   to the spell extractor and to the combat extractor and normalizer through
+   their existing `npcs` reference slots.
+
+Spell and combat-turn extraction may run concurrently after the NPC handoff is
+available. The NPC artifact may disambiguate participant identity, but it does
+not prove that a spell cast or combat turn occurred.
+
 ## Near-Term D&D Pipeline
 
 ### Evaluate Spell Extraction And Normalization
@@ -20,10 +136,9 @@ not as committed release dates.
 
 - Add narrative extraction for scene summaries, party actions, and NPCs
   encountered when that output proves useful beyond the dedicated NPC artifact.
-- Continue refining the preferred operational sequence for independent
+- Use ordered pipeline steps when a later artifact needs an accepted earlier
-  pipelines on the same transcript as additional artifacts are introduced.
+  artifact as context. Keep independent lanes in the same step and do not
-- Keep sequencing operator- or script-driven initially. Do not require a
+  introduce a general DAG or concurrent cross-lane reconciliation model.
-  general DAG or concurrent cross-lane reconciliation model.
 
 ### Improve D&D Scene Classification
 
@@ -78,7 +193,7 @@ safety checks, and deterministic application of accepted changes.
 - Add media-type validators when non-JSON artifact representations are
   introduced.
 
-## Reference And Sequential-Pipeline Evolution
+## Further Reference Evolution
 
 - Make prior-run artifacts easier to bind as references without changing the
   existing module-facing reference-item contract.
@@ -90,9 +205,33 @@ safety checks, and deterministic application of accepted changes.
   content.
 - Add reference caching, preprocessing, summarization, embedding, or retrieval
   only when reference size and observed model behavior justify them.
-- Consider non-file reference producers for prior-run artifacts, derived
+- Extend generated references to prior-run artifacts or derived summaries only
-  summaries, or entity registries after manual sequential composition becomes
+  after same-run ordered handoffs establish the required provenance and
-  burdensome.
+  lifecycle semantics.
+
+## Design Considerations To Revisit
+
+These concerns are relevant to ordered artifact dependencies but are not
+committed near-term features.
+
+### Cross-artifact identity links
+
+Evaluate whether downstream D&D artifacts should retain canonical NPC IDs from
+the generated NPC reference in addition to normalized display names. Any such
+contract must define player-character, unknown-actor, missing-NPC, and
+superseded-identity behavior before implementation. Deterministic validation
+may confirm that a linked ID exists in the consumed NPC artifact, but the link
+must never substitute for transcript evidence that the downstream event
+occurred.
+
+### Artifact contract evolution
+
+Define compatibility and migration policy before generated-reference chains
+must span multiple schema versions or long-lived historical artifacts. The
+policy should address stable identifier semantics, which schema changes permit
+checkpoint reuse, when an older artifact may be decoded or adapted, and when a
+producer or all dependents must be recomputed. Do not add a general migration
+framework until an actual contract change requires one.
 
 ## Blue-Sky Platform And Operations
 

internal/modules/dnd/extract/combatturns/assets/prompts/dnd.combat_turns.yaml

@@ -26,10 +26,6 @@ messages:
     content_file: ./sharedassets/common-dnd-identity.md
     cache_control:
       type: ephemeral
-  - role: user
-    content_file: ./sharedassets/common-dnd-transcript.md
-    cache_control:
-      type: ephemeral
   - role: user
     content_file: ./sharedassets/common-dnd-references.md
     cache_control:
@@ -44,6 +40,10 @@ messages:
     content_file: ./task.md
   - role: user
     content_file: ./instructions.md
+    cache_control:
+      type: ephemeral
+  - role: user
+    content_file: ./sharedassets/common-dnd-transcript.md
 output:
   format: json
   validation_mode: json_schema

internal/modules/dnd/extract/npcs/assets/prompts/dnd.npcs.yaml

@@ -23,10 +23,6 @@ messages:
     content_file: ./sharedassets/common-dnd-identity.md
     cache_control:
       type: ephemeral
-  - role: user
-    content_file: ./sharedassets/common-dnd-transcript.md
-    cache_control:
-      type: ephemeral
   - role: user
     content_file: ./sharedassets/common-dnd-references.md
     cache_control:
@@ -35,6 +31,10 @@ messages:
     content_file: ./task.md
   - role: user
     content_file: ./instructions.md
+    cache_control:
+      type: ephemeral
+  - role: user
+    content_file: ./sharedassets/common-dnd-transcript.md
 output:
   format: json
   validation_mode: json_schema

internal/modules/dnd/extract/spells/assets/prompts/dnd.spells.yaml

@@ -29,10 +29,6 @@ messages:
     content_file: ./sharedassets/common-dnd-identity.md
     cache_control:
       type: ephemeral
-  - role: user
-    content_file: ./sharedassets/common-dnd-transcript.md
-    cache_control:
-      type: ephemeral
   - role: user
     content_file: ./sharedassets/common-dnd-references.md
     cache_control:
@@ -49,6 +45,10 @@ messages:
     content_file: ./task.md
   - role: user
     content_file: ./instructions.md
+    cache_control:
+      type: ephemeral
+  - role: user
+    content_file: ./sharedassets/common-dnd-transcript.md
 output:
   format: json
   validation_mode: json_schema