Document extraction reference support

docs/cli.md

@@ -63,6 +63,42 @@ declares that slot. Bound reference files are read before extraction, validated
 as UTF-8 text, and passed only to the lane extractor that declares the slot.
 Reference content is not written to diagnostics, logs, errors, or manifests.
 
+Reference binding precedence is:
+
+1. pipeline-level config `references`;
+2. lane-level config `references`;
+3. `--reference` run flags;
+4. `--without-reference` run flags.
+
+`--reference` binds or replaces one slot for one selected lane. Use
+`slot=path` when the selected lanes declare the slot unambiguously:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --reference roster=./campaign-roster.txt
+```
+
+Use `lane.slot=path` when multiple selected lanes declare the same slot or when
+you want to target a specific lane:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --reference spells.glossary=./campaign-glossary.txt
+```
+
+Use `--without-reference` to remove a configured optional binding for a run:
+
+```sh
+go run ./cmd/notarius run dnd-session \
+  --config examples/dnd-spells.config.yml \
+  --input examples/seriatim-minimal-transcript.json \
+  --without-reference glossary
+```
+
 For durable output, diagnostics, retention, and failure inspection, see
 [Operations](operations.md).
 

docs/config.md

@@ -27,6 +27,9 @@ llm_profiles:
 pipelines:
   dnd-session:
     input: seriatim
+    references:
+      roster: ./dnd-spells-roster.txt
+      glossary: ./dnd-spells-glossary.txt
     chunk:
       module: generic
       options:
@@ -148,6 +151,36 @@ directory. Bound files must be UTF-8 text and are passed only to lane
 extractors that declare the slot. Reference content is not written to
 diagnostics, logs, errors, or manifests.
 
+Pipeline-level `references` are defaults. They apply only to selected lanes
+whose extractor declares the slot:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      roster: ./campaign/party-roster.txt
+      glossary: ./campaign/glossary.txt
+    artifacts:
+      spells:
+        extract: dnd/spells
+```
+
+Lane-level `references` override or add bindings for one lane:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    references:
+      glossary: ./campaign/glossary.txt
+    artifacts:
+      spells:
+        extract: dnd/spells
+        references:
+          roster: ./campaign/session-roster.txt
+```
+
 ## Module Bindings
 
 Every module binding may use shorthand:
@@ -196,6 +229,14 @@ The `generic` chunker accepts:
 The `dnd/scenes` chunker requires transcript source capabilities, calls the
 configured structured LLM provider, and does not accept module options.
 
+The `dnd/spells` extractor declares optional text reference slots:
+
+- `roster`
+- `glossary`
+
+The extractor uses these references only as supporting disambiguation material;
+spell casts still must be present in the source transcript.
+
 ## Diagnostics
 
 `diagnostics` fields:

docs/integrations/dnd-spell-artifacts.md

@@ -17,6 +17,14 @@ The extractor requires source chunks and transcript source capability. It
 returns generic artifact candidates that are serialized by the JSON output
 module.
 
+The extractor accepts optional UTF-8 text references:
+
+- `roster`: campaign roster or player-character notes.
+- `glossary`: campaign glossary or spell/name notes.
+
+References are supporting disambiguation material only. They are not source
+evidence and are not addressable through `source_refs`.
+
 ## Artifact Envelope
 
 Approved artifacts use the generic artifact envelope documented in
@@ -120,6 +128,11 @@ Rejection reason codes:
 - `invalid_source_ref`: at least one source reference fails generic source
   reference validation.
 
+Warning reason codes:
+
+- `spell_not_near_source`: the extracted spell name was not found in the cited
+  source text.
+
 Rejected candidates are written to `rejected.json` by the JSON output module.
 
 ## Manifest Metadata

docs/integrations/json-output.md

@@ -109,7 +109,7 @@ When references are bound, the manifest section has this shape:
 {
   "references": [
     {
-      "lane_id": "events",
+      "lane_id": "spells",
       "slot_name": "roster",
       "origin_type": "file",
       "origin_uri": "file:///absolute/path/roster.txt",

docs/internal/modules.md

@@ -20,6 +20,22 @@ A production module package should provide:
 Module specs should describe capabilities accurately. Resolution uses specs to
 reject incompatible pipelines before execution.
 
+Extractor modules that accept auxiliary reference material must declare slots
+through both `ReferenceSlots()` and `ModuleSpec().ReferenceSlots`. The runtime
+slot list and registry metadata should match so config validation can inspect
+slots without constructing extractor instances. A slot declaration names the
+slot, whether it is required, accepted media types, whether multiple items are
+allowed, and any byte limit.
+
+Reference content is delivered only to the lane extractor through
+`contracts.ExtractionRequest.References`. It is not source evidence and must not
+be converted into `SourceRef` values. If a module prompt uses references, load
+the prompt bundle with the same declared slots and render with
+`RenderUserSystemWithReferences`. Prompt templates may use the `reference`
+function for content and the `hasreference` function for conditional sections.
+Prompt metadata hashes remain based on template source, not rendered reference
+bytes.
+
 Chunk modules receive the structured LLM client through `contracts.ChunkRequest`
 when they need model-backed chunking. The pipeline runner validates generic
 chunk result invariants before extraction; module-owned policies may be stricter
@@ -124,12 +140,18 @@ metadata under `artifact_lanes[].metadata.extractor`. Durable artifact payload
 details belong in the
 [D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
 
+The extractor declares optional `roster` and `glossary` reference slots accepting
+UTF-8 text. Its prompt frames references as supporting disambiguation material
+only; spell-cast artifacts must still be grounded in the source transcript.
+
 ## D&D Spell Validators
 
 The spell extractor returns two built-in validators:
 
 - `dnd/spells/shape`: rejects malformed payloads and missing required fields.
 - `dnd/spells/source_refs`: rejects candidates without valid source references.
+  It also emits a warning when the extracted spell name is not found in the
+  cited source text.
 
 Reason codes include:
 
@@ -137,6 +159,7 @@ Reason codes include:
 - `missing_required_field`
 - `missing_source_ref`
 - `invalid_source_ref`
+- `spell_not_near_source`
 
 These validators are supplied by the extractor when no validators are configured
 for the lane.

docs/internal/pipeline.md

@@ -33,18 +33,23 @@ The CLI writes the resolved pipeline and digest to diagnostics.
 Pipeline profiles and artifact lanes may include reference binding maps keyed by
 extractor reference slot name. During resolution, pipeline-level bindings act as
 defaults for selected lanes whose extractor declares the slot, lane-level
-bindings override or add lane bindings, and runtime unbinds can remove bindings.
+bindings override or add lane bindings, runtime `--reference` requests override
-Resolution validates bindings against extractor specs and records lane-scoped
+config bindings, and runtime unbinds remove optional bindings. Flat runtime slot
-binding metadata. It does not read reference files or include reference bytes in
+names are resolved only when exactly one selected lane declares the slot;
-source digests.
+otherwise the CLI requires `lane.slot`. Resolution validates bindings against
+extractor specs and records lane-scoped binding metadata. It does not read
+reference files or include reference bytes in source digests.
 
 During run preparation, resolved file references are materialized before any
 LLM-backed pipeline work. Config bindings resolve relative to the config file,
 CLI bindings resolve relative to the current working directory, and materialized
-reference content is passed to extractors through `ExtractionRequest`.
+reference content is passed only to the matching lane extractor through
-Reference content is omitted from diagnostics and manifests. The CLI writes
+`ExtractionRequest`. Materialization accepts UTF-8 text files, computes
-provenance-only resolved reference diagnostics, and the run manifest records
+`sha256:` content digests, records file origins, enforces declared byte limits,
-lane-scoped reference provenance separately from source digests.
+and warns for empty bound files. Reference content is omitted from diagnostics
+and manifests. The CLI writes provenance-only resolved reference diagnostics,
+and the run manifest records lane-scoped reference provenance separately from
+source digests.
 
 Prompt bundles can declare reference slots and use `reference` and
 `hasreference` template functions. Bundle loading validates string-literal slot

docs/operations.md

@@ -35,7 +35,8 @@ The `json` output module writes these files:
 - `index.json`: file index with paths to the manifest, artifact files,
   rejected artifacts, and warnings.
 - `manifest.json`: run manifest with resolved pipeline provenance, top-level
-  module metadata, module keys, validation status, and timing.
+  module metadata, module keys, reference provenance, validation status, and
+  timing.
 - `artifacts/<artifact-type>.json`: approved artifacts grouped by artifact
   type. For the current D&D spell extractor, this includes
   `artifacts/dnd.spell_cast.json` when spell-cast artifacts are approved.
@@ -110,6 +111,10 @@ stderr, and writes warnings to durable output and diagnostics when retained.
 The run manifest `validation_status` indicates whether final artifacts were
 approved or rejected after validation.
 
+Reference-related warnings include empty bound reference files and D&D spell
+relatedness warnings such as `spell_not_near_source`. Empty references are still
+passed to extractors so optional slots can be intentionally blank.
+
 ## Cleanup
 
 It is safe to remove specific old run directories after their output and

docs/troubleshooting.md

@@ -102,6 +102,34 @@ go run ./cmd/notarius run dnd-session \
 - For `config validate`, include `--pipeline` when using `--only`.
 - Confirm the lane ID exists under `pipelines.<id>.artifacts`.
 
+## Reference Binding Failure
+
+Symptoms include:
+
+- `reference slot "..." is not declared`
+- `reference slot "..." is declared by multiple selected lanes`
+- `required reference slot "..." is not bound`
+- `--reference must use slot=path or lane.slot=path`
+- `--without-reference must use slot or lane.slot`
+- `read "...": no such file`
+- `must be UTF-8 text`
+- `is ... bytes, limit ...`
+
+Fix:
+
+- Confirm the selected extractor declares the slot. The implemented
+  `dnd/spells` extractor declares optional `roster` and `glossary` slots.
+- Use `lane.slot=path` when more than one selected lane declares the same slot.
+- Use `--without-reference slot` to remove optional config bindings; do not pass
+  an empty `--reference slot=`.
+- Check whether a path came from config or CLI. Config paths are relative to
+  the config file. CLI reference paths are relative to the current working
+  directory.
+- Ensure the file is readable UTF-8 text and within any byte limit declared by
+  the extractor.
+- If diagnostics are retained, inspect `resolved-pipeline.json`,
+  `resolved-references.json`, and `error.log`.
+
 ## Seriatim Input Validation Failure
 
 Symptoms include `seriatim input`, `parse JSON`, `segments must not be empty`,

examples/dnd-spells-glossary.txt

@@ -0,0 +1,2 @@
+Cure Wounds: healing spell cast by touch.
+Shield: defensive reaction spell.

examples/dnd-spells-roster.txt

@@ -0,0 +1,3 @@
+Aria: party cleric and recurring healer.
+Borin: fighter ally.
+Bandit mage: hostile spellcaster.

examples/dnd-spells.config.yml

@@ -7,6 +7,9 @@ llm_profiles:
 pipelines:
   dnd-session:
     input: seriatim
+    references:
+      roster: ./dnd-spells-roster.txt
+      glossary: ./dnd-spells-glossary.txt
     chunk:
       module: generic
       options: