Add a staged implementation plan to address gaps from the initial implementation of the scene chunking module

docs/roadmap/implementation.md

@@ -1,8 +1,7 @@
-# Chunk Module Implementation Plan
+# Chunk Follow-Up Implementation Plan
 
-This plan implements the accepted target state in
+This plan addresses review findings from the first `dnd/scenes` implementation.
-[Chunk Module Roadmap](chunk.md). It is written for an LLM coding agent that
+It is written for an LLM coding agent that will implement each stage in order.
-will implement each stage in order.
 
 Before beginning any stage, review:
 
@@ -14,278 +13,213 @@ Before beginning any stage, review:
 Do not move planned behavior into non-roadmap docs until the corresponding code
 is implemented. Do not revert unrelated user changes.
 
-## Stage 1: Framework Chunk Contract
+## Goals
 
-Goal: make LLM-backed chunking first-class and enforce generic chunk result
+- Record chunker prompt and response-schema provenance in run manifests.
-invariants without adding any D&D-specific framework behavior.
+- Ensure downstream extractors receive canonical source units from the source
+  document, not chunker-mutated unit payloads.
+- Prevent empty or whitespace-only scene caveats from becoming warnings.
+
+## Stage 1: Run-Manifest Module Metadata
+
+Goal: make non-lane module provenance auditable without adding
+chunker-specific fields or D&D-specific framework behavior.
+
+Design decision:
+
+- Add a generic top-level run manifest metadata map for singleton pipeline
+  modules:
+
+```go
+ModuleMetadata map[string]map[string]any `json:"module_metadata,omitempty"`
+```
+
+- Use stable stage keys:
+  - `input`
+  - `chunker`
+  - `output`
+- Keep existing artifact lane metadata under `ArtifactLaneManifest.Metadata`.
+  Do not move extractor, merger, normalizer, or validator metadata into the
+  top-level map in this stage.
+- Record metadata only when a built module implements
+  `contracts.ManifestMetadataProvider` and returns non-empty metadata.
+- Continue to reject raw prompts, raw response schemas, source text, provider
+  payloads, and secrets from manifest metadata by convention and tests.
 
 Code changes:
 
-- Add `LLMClient contracts.StructuredLLMClient` to `contracts.ChunkRequest` in
+- Add `ModuleMetadata` to `internal/core/artifacts.RunManifest`.
-  `internal/framework/contracts/contracts.go`.
+- Add a small helper in `internal/framework/pipeline/runner.go` to attach
-- Update `internal/framework/pipeline/runner.go` so the runner passes
+  top-level module metadata by stage key.
-  `input.LLMClient` to the chunker in `contracts.ChunkRequest`.
+- After building the input adapter, chunker, and output encoder, call that
-- Add framework-level chunk result validation after `chunker.Chunk` returns and
+  helper with keys `input`, `chunker`, and `output` respectively.
-  before lanes execute.
+- Keep `setLaneManifestMetadata` for lane-owned modules. If practical, share
-- Keep validation source-generic. The validator should reject:
+  metadata cloning logic with the new helper.
-  - empty chunk ID;
+- Ensure failed runs that already have a manifest also retain any metadata
-  - duplicate chunk ID;
+  collected before the failure.
-  - chunk `SourceID` that does not match the source document ID;
-  - chunk `Index` that does not match returned order;
-  - empty chunk units;
-  - repeated source unit inside one chunk;
-  - source unit not found in the source document;
-  - chunk units that do not appear in source-document order.
-- The validator must not require complete coverage and must not reject overlap
-  between different chunks.
-- Preserve existing warning behavior: append chunker warnings before returning
-  chunk errors, as the runner does today.
-
-Documentation changes:
-
-- Update implemented internal docs under `docs/internal/` to define the chunk
-  module API and validation invariants once the code exists.
-- Keep examples and user docs unchanged in this stage unless an existing doc
-  becomes inaccurate.
 
 Tests:
 
-- Update contract tests for the new `ChunkRequest.LLMClient` field where useful.
+- Add pipeline runner tests proving top-level metadata is recorded for a fake
-- Add focused pipeline runner tests for each invalid chunk result case listed
+  chunker that implements `ManifestMetadataProvider`.
-  above.
+- Add a test proving the existing lane metadata behavior remains unchanged.
-- Add runner tests proving partial coverage and overlapping chunks remain
+- Add a CLI or output integration test proving a `dnd/scenes` run manifest
-  accepted.
+  contains `module_metadata.chunker` with prompt/schema provenance.
-- Run:
+- Add a negative assertion that raw prompt text, raw schema JSON, source text,
+  provider payloads, and API key-like fields are not present in the scene
+  chunker metadata.
+
+Documentation:
+
+- Update `docs/internal/pipeline.md` to describe top-level metadata for input,
+  chunker, and output modules, and lane metadata for lane modules.
+- Update `docs/internal/modules.md` to say that `dnd/scenes` prompt/schema
+  provenance appears under `module_metadata.chunker`.
+- Update `docs/integrations/json-output.md` and `docs/operations.md` if their
+  manifest descriptions need to mention `module_metadata`.
+
+Validation:
 
 ```sh
-go test ./internal/framework/contracts ./internal/framework/pipeline
+go test ./internal/core/artifacts
-```
-
-Stage completion criteria:
-
-- Existing generic chunking still works.
-- A fake chunker can receive the structured LLM client through `ChunkRequest`.
-- Framework tests prove the accepted generic chunk invariants.
-
-## Stage 2: D&D Scene Assets And Module Skeleton
-
-Goal: revise the draft D&D scene prompt and schema into module-owned assets and
-add load/render plumbing without registering production behavior.
-
-Asset decisions:
-
-- Rename `internal/modules/chunk/dnd/scenes/assets/schemas/scene_map.schema.json`
-  to `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`.
-- Use these schema constants unless a code-local naming conflict requires a
-  mechanical adjustment:
-  - prompt ID: `dnd.scenes`
-  - response schema key: `dnd_scenes`
-  - response schema ID: `notarius.dnd.scenes`
-  - response schema version: `v1`
-  - response schema name: `notarius_dnd_scenes_v1`
-- Use this structured response shape:
-
-```json
-{
-  "scenes": [
-    {
-      "start_unit_id": "seg-001",
-      "end_unit_id": "seg-010",
-      "short_title": "Ambush at the gate",
-      "primary_mode": "Combat",
-      "main_participants": ["Aria", "Bandit mage"],
-      "summary": "The party fights the bandit mage at the gate.",
-      "boundary_note": "The scene begins when combat starts and ends when the immediate threat is resolved.",
-      "boundary_confidence": "High"
-    }
-  ],
-  "boundary_caveats": []
-}
-```
-
-- Required top-level fields: `scenes`, `boundary_caveats`.
-- Required scene fields: `start_unit_id`, `end_unit_id`, `short_title`,
-  `primary_mode`, `main_participants`, `summary`, `boundary_note`,
-  `boundary_confidence`.
-- Boundary fields are source-unit ID strings, not integers.
-- `primary_mode` enum: `Recap`, `Discussion`, `Combat`, `Narrative`.
-- `boundary_confidence` enum: `High`, `Medium`, `Low`.
-- Keep `additionalProperties: false` throughout the schema.
-- Do not include model-authored final chunk IDs or chunk indexes in the schema.
-  The Go module assigns deterministic chunk IDs and indexes.
-
-Prompt decisions:
-
-- Keep D&D-specific scene guidance in the D&D scene module.
-- Make the user prompt a Go template similar to the spell extractor prompt.
-- Include source document ID and ordered source units.
-- Include selected source-unit metadata when present: `speaker`, `start`, and
-  `end`.
-- Align prompt terms exactly with schema field names and enum values.
-- Keep `dnd/scenes` module policy explicit in the prompt: full coverage,
-  sequential scenes, no gaps, no overlap, exact source-unit IDs.
-
-Code changes:
-
-- Add `assets.go` with an `embed.FS` for prompts and schemas.
-- Add `schema.go` with the constants and a `loadResponseSchema` function using
-  `llm.LoadResponseSchema`, following the pattern in
-  `internal/modules/extract/dnd/spells/schema.go`.
-- Add prompt rendering code using `framework/prompt.Bundle`, following the
-  pattern in `internal/modules/extract/dnd/spells/prompt.go`.
-- Add internal response structs for the schema shape.
-- Do not register the module in `internal/cli/catalog.go` in this stage.
-
-Tests:
-
-- Add tests that the schema loads, is valid JSON, has the expected metadata, and
-  rejects the old integer-boundary assumption through Go-side type expectations.
-- Add prompt rendering tests that source unit IDs and selected metadata appear
-  in the rendered user prompt.
-- Run:
-
-```sh
-go test ./internal/modules/chunk/dnd/scenes
-```
-
-Stage completion criteria:
-
-- The scene schema and prompts are loadable embedded assets.
-- The prompt/schema terminology is internally consistent.
-- No production catalog behavior changes yet.
-
-## Stage 3: D&D Scene Chunker Implementation
-
-Goal: implement `dnd/scenes` as a contract-compliant chunk module with strict
-module-owned validation.
-
-Module decisions:
-
-- Package path: `internal/modules/chunk/dnd/scenes`.
-- Package name: `scenes`.
-- Module key: `dnd/scenes`.
-- `ModuleSpec`:
-  - `Stage`: `pipeline.StageChunk`
-  - `Requires`: `source.transcript`
-  - `Provides`: `chunks`, `chunks.scenes`
-- Constructor: `New() *Chunker`.
-- Registration function: `Register(registry *pipeline.ChunkerRegistry) error`.
-- No module options initially. Reject non-empty options with an actionable
-  module-prefixed error unless a clear option is implemented in the same stage.
-
-Chunking behavior:
-
-- Validate `context.Context`, source document, non-empty source units, and
-  non-nil `LLMClient`.
-- Render the scene prompt over the full source document.
-- Call `LLMClient.CompleteStructured` with:
-  - `StageName`: `dnd/scenes`
-  - response schema name and schema JSON from the module schema loader.
-- Validate the decoded response before producing chunks:
-  - `scenes` must be present and non-empty;
-  - every boundary ID must exist in the source document;
-  - each scene start must be at or before its end;
-  - the first scene starts at the first source unit;
-  - the final scene ends at the final source unit;
-  - scenes are contiguous in source order;
-  - scenes do not overlap;
-  - required metadata fields are non-empty after trimming;
-  - `main_participants` entries are trimmed and empty entries rejected.
-- Assign deterministic chunk fields:
-  - `ID`: `scene-000001`, `scene-000002`, and so on;
-  - `SourceID`: source document ID;
-  - `Index`: zero-based returned order;
-  - `Units`: defensive copies of the source units in the scene range.
-- Store per-scene metadata on each chunk:
-  - `scene_title`
-  - `primary_mode`
-  - `main_participants`
-  - `summary`
-  - `boundary_note`
-  - `boundary_confidence`
-  - `start_unit_id`
-  - `end_unit_id`
-  - `unit_count`
-- Convert each `boundary_caveats` entry into a `contracts.Warning` with:
-  - `Scope`: `dnd/scenes`
-  - `ReasonCode`: `scene_boundary_caveat`
-  - `Message`: the caveat text.
-- Fail explicitly for malformed model output. Do not fall back to `generic`.
-- Implement `contracts.ManifestMetadataProvider` and include prompt and
-  response-schema provenance without raw prompts, raw schemas, source text, or
-  secrets.
-
-Tests:
-
-- Registration and `ModuleSpec`.
-- Successful chunking from a fake LLM response.
-- Prompt request uses the expected schema name and schema JSON.
-- Caveats become warnings.
-- Defensive copy behavior for source units and metadata.
-- Errors for nil context, nil source, invalid source, nil LLM client, empty
-  model scenes, unknown boundary ID, out-of-order boundaries, gaps, overlap,
-  incomplete coverage, empty metadata fields, and non-empty unsupported options.
-- Manifest metadata contains prompt/schema provenance.
-- Run:
-
-```sh
-go test ./internal/modules/chunk/dnd/scenes
 go test ./internal/framework/pipeline
-```
-
-Stage completion criteria:
-
-- `dnd/scenes` works in focused tests with fake LLM clients.
-- It is still not production-registered unless Stage 4 is completed.
-
-## Stage 4: Production Registration And Implemented Docs
-
-Goal: make `dnd/scenes` available in production configuration and document only
-the behavior that now exists.
-
-Code changes:
-
-- Register `dnd/scenes` in `internal/cli/catalog.go`.
-- Add or update catalog/default module tests so the production catalog exposes
-  the new chunk module.
-- Add CLI/config validation tests proving a pipeline can select
-  `chunk: dnd/scenes`.
-- Do not change the existing maintained example config unless the related CLI
-  fixture tests are updated to keep it loadable and useful.
-
-Documentation changes:
-
-- Update `docs/config.md` implemented production module tables and chunk module
-  notes.
-- Update `docs/cli.md` implemented production module list.
-- Update `docs/internal/modules.md` with `dnd/scenes` behavior, capabilities,
-  metadata, and failure policy.
-- Update or add internal chunk-module documentation if Stage 1 did not already
-  create a clear API reference.
-- Update `docs/troubleshooting.md` for common scene chunker failures:
-  malformed model output, invalid boundaries, incomplete coverage, and provider
-  failures during chunking.
-- Keep roadmap docs for any deferred options or future prompt tuning.
-
-Tests:
-
-```sh
 go test ./internal/cli
-go test ./internal/core/config
-go test ./internal/framework/pipeline
 go test ./internal/modules/chunk/dnd/scenes
 ```
 
 Stage completion criteria:
 
-- Config resolution can bind `dnd/scenes`.
+- `dnd/scenes` prompt/schema provenance is visible in durable
-- User and internal docs describe the implemented module accurately.
+  `manifest.json` and diagnostics `run-manifest.json`.
-- Existing examples and CLI docs remain truthful.
+- Existing artifact lane metadata remains in the same JSON location as before.
 
-## Stage 5: Full Verification
+## Stage 2: Canonical Source Units In Chunk Results
 
-Goal: verify the complete feature across contracts, production wiring, docs, and
+Goal: preserve the chunker boundary contract while ensuring extractors always
-the command entry point.
+consume source document units, not rewritten units supplied by a chunker.
+
+Design decision:
+
+- Keep the chunker contract expressed in terms of `contracts.SourceChunk`.
+- Continue validating chunk IDs, source IDs, indexes, unit membership, unit
+  uniqueness within each chunk, and source-ordering.
+- After validation, canonicalize chunk units by replacing each returned
+  `SourceUnit` with a defensive copy of the matching source document unit.
+- Preserve `SourceChunk.Metadata` as module-owned chunk metadata.
+- Do not require full source coverage and do not reject overlap between chunks.
+- Do not preserve chunker-mutated per-unit text, kind, or metadata. A chunker
+  that wants to add scene-level information must use `SourceChunk.Metadata`.
+
+Code changes:
+
+- Replace or extend `validateChunkResult` in
+  `internal/framework/pipeline/chunk_validation.go` so it returns canonical
+  chunks, for example:
+
+```go
+func validateAndCanonicalizeChunkResult(doc *source.SourceDocument, chunks []contracts.SourceChunk) ([]contracts.SourceChunk, error)
+```
+
+- Build a source-unit lookup from the validated source document.
+- For each chunk:
+  - validate the existing generic invariants;
+  - copy chunk ID, source ID, index, and chunk metadata;
+  - replace the unit slice with cloned source units from the source document in
+    the returned boundary/order.
+- Update the runner to use canonical chunks for all downstream extraction and
+  merge behavior.
+- Ensure chunk metadata is cloned so later module or caller mutation cannot
+  affect runner state.
+- Keep the implementation source-agnostic. Do not inspect transcript-specific
+  metadata keys.
+
+Tests:
+
+- Add a runner test where a fake chunker returns a valid unit ID with mutated
+  text, kind, and unit metadata. Assert the extractor receives the original
+  source document unit values.
+- Add a runner test proving chunk metadata survives canonicalization and is not
+  aliased to the chunker-returned map.
+- Keep existing tests for invalid chunk IDs, duplicate IDs, wrong source ID,
+  wrong index, empty units, repeated unit IDs, unknown unit IDs, out-of-order
+  units, partial coverage, and overlap.
+- Add or update tests so generic chunking still behaves unchanged.
+
+Documentation:
+
+- Update internal chunk contract docs to state that source units in chunks are
+  canonicalized from the source document by ID before extractors run.
+- Document that chunk metadata is the supported mechanism for passing
+  chunker-owned context to extractors.
+
+Validation:
+
+```sh
+go test ./internal/framework/pipeline
+go test ./internal/modules/chunk/generic
+go test ./internal/modules/chunk/dnd/scenes
+```
+
+Stage completion criteria:
+
+- Extractors cannot observe chunker-rewritten source-unit text, kind, or unit
+  metadata.
+- Chunker-owned scene metadata still reaches extractors through
+  `SourceChunk.Metadata`.
+
+## Stage 3: Scene Caveat Hygiene
+
+Goal: ensure model caveats become useful warnings and never produce blank
+warnings that can confuse operators or affect diagnostics retention.
+
+Design decision:
+
+- Require caveat strings to be non-empty after trimming.
+- Treat whitespace-only caveats as malformed structured output rather than
+  silently dropping them. This is consistent with the `dnd/scenes` policy of
+  failing explicitly for malformed model output.
+- Store warning messages as trimmed caveat text.
+
+Code changes:
+
+- Update `internal/modules/chunk/dnd/scenes/assets/schemas/dnd_scenes.v1.json`
+  so `boundary_caveats.items` has `minLength: 1`.
+- Update `warningsFromCaveats` or response validation in
+  `internal/modules/chunk/dnd/scenes/chunker.go` to trim caveats and reject
+  empty results with a module-prefixed malformed-output error.
+- Prefer validating caveats before constructing chunks so all malformed response
+  checks happen together.
+
+Tests:
+
+- Add schema tests proving `boundary_caveats` items require non-empty strings.
+- Add chunker tests proving:
+  - caveat warning messages are trimmed;
+  - whitespace-only caveats fail explicitly;
+  - valid caveats still produce `scene_boundary_caveat` warnings.
+- Keep existing warning tests passing.
+
+Documentation:
+
+- Update `docs/internal/modules.md` and `docs/troubleshooting.md` if needed to
+  mention that malformed caveats are treated as malformed structured output.
+
+Validation:
+
+```sh
+go test ./internal/modules/chunk/dnd/scenes
+go test ./internal/cli
+```
+
+Stage completion criteria:
+
+- No blank warnings can be emitted from `dnd/scenes` boundary caveats.
+- Valid caveats remain visible as warnings.
+
+## Stage 4: Full Verification
+
+Goal: verify the follow-up work across contracts, production wiring,
+documentation, and the command entry point.
 
 Run:
 
@@ -295,18 +229,17 @@ go vet ./...
 go build ./cmd/notarius
 ```
 
-Inspect diagnostics-sensitive output manually in tests or fixtures where
+Inspect or test representative output manifests:
-relevant:
 
-- no raw prompts, source text, provider payloads, API keys, or secrets in
+- `manifest.json` includes `module_metadata.chunker` for a `dnd/scenes` run;
-  manifest metadata;
+- `module_metadata.chunker` contains prompt and response-schema provenance;
-- errors name the module and operation;
+- no raw prompt, raw schema, source text, provider payload, or secret appears in
-- warnings are preserved in `RunOutput.Warnings`;
+  module metadata;
-- run manifests record the `dnd/scenes` chunker when selected.
+- artifact lane metadata remains under `artifact_lanes[].metadata`;
+- `warnings.json` contains trimmed scene caveats and no blank caveat warnings.
 
 Stage completion criteria:
 
 - Full validation commands pass.
-- The feature is documented as implemented only where code supports it.
+- Current-behavior docs match implemented behavior.
-- `docs/roadmap/chunk.md` retains target-state context and does not duplicate
+- Any remaining planned or deferred behavior stays under `docs/roadmap/`.
-  current-behavior reference material.

docs/roadmap/references.md

@@ -0,0 +1,340 @@
+# Feature Roadmap Proposal: Extraction Reference
+
+## Status
+
+This document captures proposed design and implementation sequencing for the
+extraction-reference feature in Notarius. It describes planned work, not
+implemented behavior. Go snippets are conceptual sketches; the implementing
+agent should adapt names and shapes to the existing contracts, package
+boundaries, and conventions in this repository.
+
+## Goal
+
+Extraction quality improves significantly when the LLM receives reference
+material alongside the source input. For the initial D&D spell extractor,
+useful reference material includes a party roster (mapping players to player
+characters), a player list, and a campaign glossary.
+
+Notarius should support passing this material to extractors as **named reference
+items** without introducing any domain-specific concepts into core or framework
+packages. The framework should know only that:
+
+- extractors declare named reference slots they accept;
+- pipeline config and CLI flags bind content (initially files) to those slots;
+- bound content is rendered into module-owned prompt templates;
+- bound content is digested and recorded as run provenance.
+
+Only extract modules should know what a "roster" or "glossary" means. All
+domain semantics live in module-owned slot declarations and prompt templates.
+
+## Definitions
+
+- **Reference slot**: a named, typed-by-convention input declared by an
+  extractor, with a human-readable description and a required/optional flag.
+  Example: extractor `dnd/spells` declares an optional slot named `roster`.
+- **Reference item**: resolved content bound to a slot for a given run: name,
+  content bytes, media type, content digest, and origin (initially a file
+  path).
+- **Reference binding**: the association of a slot name to a content source,
+  defined in pipeline config and overridable per run via CLI.
+
+## Architectural Principles
+
+- Reference is opaque to the framework. Core and framework packages must not
+  interpret reference content or recognize domain slot names.
+- Reference is an input. Anything that changes extraction output must be
+  digested into the run manifest and participate in any cache key.
+- A Reference is not evidence. `SourceRef` values must only ever reference source
+  units. Reference items must not receive unit IDs and must not be addressable
+  by source references.
+- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting
+  a required slot, should fail at config-load time, before any LLM call.
+- Optional slots degrade gracefully. Prompt templates should render cleanly
+  whether or not an optional slot is bound.
+- Determinism. Identical input, config, prompts, and reference bytes should
+  produce byte-identical rendered prompts. Reference slots should render in a
+  stable, documented order (declaration order).
+
+## Proposed Contracts
+
+### Slot declaration (extractor contract extension)
+
+Extractors should declare the reference slots they accept:
+
+```go
+type ReferenceSlot struct {
+    Name        string
+    Description string
+    Required    bool
+
+    // MVP can leave these empty/defaulted, but having the fields now
+    // makes validation and future docs easier.
+    AcceptedMediaTypes []string
+    Multiple           bool
+    MaxBytes           int64
+}
+```
+
+The extractor interface should gain a method such as:
+
+```go
+ReferenceSlots() []ReferenceSlot
+```
+
+Extractors with no reference needs return an empty slice. Existing extractors
+should require no other changes.
+
+### Resolved reference item
+
+```go
+type ReferenceItem struct {
+    SlotName  string
+    MediaType string
+    Content   []byte
+    Digest    string
+    Origin    ReferenceOrigin
+
+    SizeBytes     int64
+    TokenEstimate int
+}
+
+type ReferenceOrigin struct {
+    Type string // "file" for MVP
+    URI  string // path or future artifact URI
+}
+
+type ReferenceSet struct {
+    // Stable declaration order, then stable binding order within a slot.
+    Slots []ResolvedReferenceSlot
+}
+
+type ResolvedReferenceSlot struct {
+    Name  string
+    Items []ReferenceItem
+}
+```
+
+`ReferenceItem` is a resolved-content type, not a file path. The only MVP
+producer is "read this file," but the shape should permit future producers
+(prior-run artifacts, derived summaries, entity registries) without contract
+changes.
+
+### Binding resolution
+
+A resolver should, at config-load time:
+
+1. Collect declared slots from every extractor selected by the active
+   pipeline (respecting lane selection, e.g. `--only`).
+2. Collect bindings from pipeline config (pipeline-level and lane-level) and
+   CLI overrides, applying the standard layering: config file, then CLI.
+3. Fail with a clear error if a required slot is unbound, or if a binding
+   references a slot no extractor within the selected pipeline declares. 
+   Errors should name the pipeline, lane, slot, and the slot description.
+4. Read, digest, and materialize each bound source into a `ReferenceItem`.
+5. Enforce size guardrails (see Validation and Guardrails).
+
+## Configuration and CLI
+
+### Pipeline config
+
+Reference bindings should live in pipeline config, because the initial use cases
+(roster, glossary) are campaign-invariant rather than run-variant. Bindings
+should be supported at two levels:
+
+- pipeline level: shared by all artifact lanes;
+- lane level: additions or overrides for a single lane.
+
+Illustrative shape (adapt to the existing config format):
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      roster: ./campaign/party_roster.md
+      glossary: ./campaign/glossary.md
+    artifacts:
+      spells:
+        extract: dnd/spells
+      npcs:
+        extract: dnd/npcs
+        reference:
+          npc_registry: ./campaign/npcs.md
+```
+
+### CLI
+
+Per-run override flag, repeatable:
+
+```text
+notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md
+```
+
+CLI bindings override config bindings for the same slot name. The existing
+pipeline-describe/config-validate commands (or their nearest equivalents)
+should surface declared slots, descriptions, required flags, and current
+bindings so users can discover what a pipeline accepts.
+
+## Prompt Template Integration
+
+Prompt templates are module-owned. Template rendering should expose:
+
+- `{{ reference "roster" }}`: renders the content of the bound item;
+- `{{ hasreference "glossary" }}`: predicate for conditional sections, so
+  optional slots can be included only when bound.
+
+Rules:
+
+- Referencing an **undeclared** slot from a template is a module bug and
+  should fail at prompt registration/build time (or earliest feasible point),
+  not silently at render time.
+- Referencing a declared but unbound **optional** slot should render as
+  empty; templates should use `hasreference` to avoid dangling section headers.
+- Rendering must be deterministic and independent of map iteration order.
+- Prompt identity (registry hash) should be computed over the **template**,
+  not the rendered prompt. Reference digests are recorded separately in the
+  manifest, so a reference edit is visible as a reference change, not a prompt
+  change.
+
+Note: reference content is repeated in every per-chunk prompt. Diagnostics
+should record per-slot token or byte counts so reference cost is observable.
+Per-slot inclusion policies (e.g., roster in every chunk, glossary on demand)
+are explicitly out of scope until cost data justifies them.
+
+## Provenance
+
+The run manifest must record, for every bound slot:
+
+- slot name;
+- origin (path);
+- content digest;
+- media type;
+- whether the binding came from config or CLI override.
+
+Reference digests must participate in any idempotency/cache key alongside source
+digests, prompt hashes, schema versions, model, and parameters. Two runs that
+differ only in reference content must be distinguishable from the manifest
+alone.
+
+Diagnostics for a run should include the resolved binding set (with digests,
+not necessarily full content) in the run directory, consistent with the
+existing redacted-effective-config pattern.
+
+## Path Resolution
+
+- Config-relative paths resolve relative to the pipeline config file.
+- CLI-relative paths resolve relative to the current working directory.
+- Manifest records the normalized absolute path or a redacted/display path 
+according to existing diagnostics policy.
+
+## Validation and Guardrails
+
+### References are not evidence
+
+The primary new failure mode: the model extracts facts from references rather
+than from the source input. Example: the roster lists a PC's known spells, and
+the model emits a `SpellCast` for a spell that was never cast in the session,
+with a fabricated or misattributed source reference.
+
+Defenses, in priority order:
+
+1. **Structural.** `SourceRef` remains the only grounding mechanism and can
+   only reference source units. No contract change should make references
+   addressable as evidence.
+2. **Prompt discipline.** Module templates should frame references explicitly as
+   reference material, e.g. "use the roster to resolve speakers to
+   characters; extract only events that occur in the transcript." This
+   guidance belongs in the module prompt guidelines, not framework code.
+3. **Validator support.** The source-reference validator (or a sibling
+   deterministic validator) should support checking that referenced source
+   text plausibly relates to the extracted fact (e.g., spell name or a close
+   variant appears in or near the referenced range). Severity should be
+   `warn`, not `fail`, given paraphrase and nickname casting.
+4. **Regression fixtures.** Golden-file tests must include a fixture in which
+   the bound roster mentions a spell that is never cast in the transcript,
+   asserting no artifact record is produced for it. This regression is likely
+   to be reintroduced by future prompt edits; the fixture is the guard.
+
+### Size and sanity guardrails
+
+- Fail fast, before any LLM call, if bound references plus template plus largest
+  chunk exceeds the configured model context budget, with an error that names
+  the offending slot(s) and sizes.
+- Empty bound files should produce a warning (probable user error).
+- MVP accepts text content only (`utf-8`); other media
+  types should be rejected with a clear error.
+
+## Out of Scope (MVP)
+
+- Non-file reference producers (prior-run artifacts, derived summaries, entity
+  registries). The `ReferenceItem` shape should permit them later.
+- Per-chunk or per-slot inclusion policies and context budgeting beyond the
+  fail-fast guardrail.
+- Structured/parsed references (e.g., typed roster schemas). References are opaque
+  text handed to prompts.
+- Reference caching or preprocessing (summarization, embedding, retrieval).
+- Making reference addressable as evidence, in any form.
+
+## Checkpoint Sequencing
+
+Each checkpoint should leave the repository compiling, with targeted tests
+covering newly introduced contracts or behavior.
+
+1. **Contracts and resolution.** Add `ReferenceSlot`, `ReferenceItem`, and the
+   extractor `ReferenceSlots()` method (empty default for existing extractors).
+   Implement config parsing for pipeline- and lane-level bindings, CLI
+   override flag, layering, and load-time validation (unknown slot, missing
+   required slot, unreadable file, empty file warning). Unit tests for
+   resolution and error cases.
+2. **Prompt rendering.** Add `reference`/`hasreference` template functions,
+   declaration-order rendering, undeclared-slot failure at registration, and
+   deterministic-render tests (byte-identical output across runs).
+3. **Provenance.** Record bindings (name, origin, digest, media type,
+   binding source) in the run manifest and diagnostics; include reference
+   digests in the cache/idempotency key if one exists. Tests: manifest
+   round-trip; two runs differing only in reference content produce differing
+   manifests.
+4. **Guardrails and validation.** Context-window fail-fast check;
+   relatedness `warn` validator (or extension of the source-reference
+   validator); media-type rejection.
+5. **First consumer.** Declare `roster` (optional) and `glossary` (optional)
+   slots on the D&D spells extractor; update its prompt template with
+   conditional reference sections and reference-material framing; add golden
+   fixtures with and without references bound, including the
+   roster-mentions-uncast-spell fixture. This checkpoint is the acceptance
+   test for the feature: spell extraction quality with a roster bound should
+   visibly improve speaker-to-character attribution in fixtures.
+
+## Open Design Questions
+
+The implementing agent should resolve these against existing code and record
+decisions in the implementation plan:
+
+- Should slot names be namespaced per lane in config and CLI (e.g.,
+  `spells.roster=...`) or flat with lane-level config as the only
+  disambiguator? (Recommended default: flat names; lane-level config for
+  overrides; revisit if two extractors in one pipeline want the same slot
+  name with different content.)
+- Where does binding resolution live relative to the existing config and
+  pipeline packages? It must run at load time, alongside existing pipeline
+  validation.
+- Does the existing prompt registry hash templates or rendered prompts? If
+  rendered, this feature requires moving to template hashing as described in
+  Provenance.
+- Should CLI overrides be permitted to bind slots that config leaves unbound
+  (yes, presumably), and to *unbind* a config-bound optional slot (e.g.,
+  `--reference roster=` to clear)? Decide and test both directions.
+
+## Documentation Tasks
+
+Once implemented, move contracts out of this roadmap into canonical docs:
+
+- `docs/cli.md`: `--reference` flag syntax, layering, and examples;
+- `docs/config.md`: pipeline- and lane-level `references` blocks;
+- `docs/internal/`: slot/item contracts, resolution flow, evidence
+  exclusion rule, and template function reference for module authors;
+- module-author guidance: how to declare slots, write conditional reference
+  sections, and frame reference material in prompts;
+- `examples/`: a maintained example pipeline with a roster and glossary
+  bound, plus matching fixture files.
+```