Finish the references implementation for the extraction module and update roadmap documentation

docs/cli.md

@@ -61,7 +61,9 @@ Reference flags are resolved against selected artifact lanes before the run
 starts. Flat slot names are accepted only when exactly one selected lane
 declares that slot. Bound reference files are read before extraction, validated
 as UTF-8 text, and passed only to the lane extractor that declares the slot.
-Reference content is not written to diagnostics, logs, errors, or manifests.
+Notarius infers reference media types from file extensions for provenance and
+for optional slot checks. Reference content is not written to diagnostics, logs,
+errors, or manifests.
 
 Reference binding precedence is:
 

docs/config.md

@@ -148,11 +148,15 @@ lane-level bindings, and run-time `--reference` or `--without-reference`
 overrides are applied. Config-relative paths are resolved relative to the
 config file; CLI reference paths are resolved relative to the current working
 directory. Bound files must be UTF-8 text and are passed only to lane
-extractors that declare the slot. Reference content is not written to
+extractors that declare the slot. Reference media types are inferred from file
-diagnostics, logs, errors, or manifests.
+extensions, recorded as canonical base media types, and checked only when a
+module declares `AcceptedMediaTypes`; unknown extensions are recorded as
+`application/octet-stream`. Reference content is not written to diagnostics,
+logs, errors, or manifests.
 
-Pipeline-level `references` are defaults. They apply only to selected lanes
+Pipeline-level `references` are defaults. They are valid when at least one
-whose extractor declares the slot:
+declared lane in the pipeline has an extractor that declares the slot. During a
+run, they apply only to selected lanes whose extractor declares the slot:
 
 ```yaml
 pipelines:

docs/integrations/json-output.md

@@ -114,7 +114,7 @@ When references are bound, the manifest section has this shape:
       "origin_type": "file",
       "origin_uri": "file:///absolute/path/roster.txt",
       "digest": "sha256:...",
-      "media_type": "text/plain; charset=utf-8",
+      "media_type": "text/plain",
       "size_bytes": 123,
       "binding_source": "config"
     }
@@ -122,6 +122,10 @@ When references are bound, the manifest section has this shape:
 }
 ```
 
+Reference media types are inferred from file extensions and recorded as
+canonical base media types. Unknown extensions are recorded as
+`application/octet-stream`.
+
 `module_metadata` is omitted when no singleton module provides metadata.
 
 `validation_status` is `approved` when no candidates were rejected and

docs/internal/modules.md

@@ -25,7 +25,10 @@ through both `ReferenceSlots()` and `ModuleSpec().ReferenceSlots`. The runtime
 slot list and registry metadata should match so config validation can inspect
 slots without constructing extractor instances. A slot declaration names the
 slot, whether it is required, accepted media types, whether multiple items are
-allowed, and any byte limit.
+allowed, and any byte limit. Empty `AcceptedMediaTypes` means any inferred
+media type is accepted, though the file must still be UTF-8 text. When a slot
+declares accepted media types, Notarius compares the canonical base media type
+inferred from the file extension, case-insensitively and without parameters.
 
 Reference content is delivered only to the lane extractor through
 `contracts.ExtractionRequest.References`. It is not source evidence and must not
@@ -140,9 +143,10 @@ metadata under `artifact_lanes[].metadata.extractor`. Durable artifact payload
 details belong in the
 [D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
 
-The extractor declares optional `roster` and `glossary` reference slots accepting
+The extractor declares optional `roster` and `glossary` reference slots
-UTF-8 text. Its prompt frames references as supporting disambiguation material
+accepting UTF-8 text without narrowing accepted media types. Its prompt frames
-only; spell-cast artifacts must still be grounded in the source transcript.
+references as supporting disambiguation material only; spell-cast artifacts must
+still be grounded in the source transcript.
 
 ## D&D Spell Validators
 

docs/internal/pipeline.md

@@ -45,11 +45,14 @@ LLM-backed pipeline work. Config bindings resolve relative to the config file,
 CLI bindings resolve relative to the current working directory, and materialized
 reference content is passed only to the matching lane extractor through
 `ExtractionRequest`. Materialization accepts UTF-8 text files, computes
-`sha256:` content digests, records file origins, enforces declared byte limits,
+`sha256:` content digests, records file origins, infers canonical base media
-and warns for empty bound files. Reference content is omitted from diagnostics
+types from file extensions, enforces declared byte limits, and warns for empty
-and manifests. The CLI writes provenance-only resolved reference diagnostics,
+bound files. Media-type acceptance is checked only when a slot declares
-and the run manifest records lane-scoped reference provenance separately from
+`AcceptedMediaTypes`; unknown extensions are recorded as
-source digests.
+`application/octet-stream`. Reference content is omitted from diagnostics and
+manifests. The CLI writes provenance-only resolved reference diagnostics, and
+the run manifest records lane-scoped reference provenance separately from source
+digests.
 
 Prompt bundles can declare reference slots and use `reference` and
 `hasreference` template functions. Bundle loading validates string-literal slot

docs/roadmap/future.md

@@ -8,6 +8,21 @@ future work only.
 
 - Additional input adapters, such as Markdown or note-export formats.
 - Additional D&D extractors beyond spell casts.
+- First-class LLM access for all LLM-eligible pipeline stages: `chunk`,
+  `extract`, and `normalize`.
+- Expand reference support beyond extractors to `chunk` and `normalize`
+  modules, using the same declared-slot model and preserving stage-specific
+  ownership boundaries.
+- Add non-file reference producers, such as prior-run artifacts, derived
+  summaries, or entity registries, without changing module-facing reference
+  item contracts.
+- Add token budgeting and model context-window management for reference content.
+- Add per-slot or per-chunk inclusion policies so modules can avoid repeating
+  large reference content in every prompt when that becomes important.
+- Add structured or parsed references, such as typed roster schemas, when a
+  module has a clear need for more than opaque UTF-8 text.
+- Add reference caching, preprocessing, summarization, embedding, or retrieval
+  if references become large enough to require preprocessing.
 - Cross-lane entity normalization.
 - Cross-chunk semantic deduplication.
 - Configurable validator chains with production validator modules.
@@ -15,6 +30,13 @@ future work only.
 - Parallel execution where it preserves deterministic manifests and diagnostics.
 - Additional output encoders.
 
+## Candidate Architecture Work
+
+- Evaluate replacing the local LLM adapter with an import from
+  `gitea.maximumdirect.net/eric/scriptorium`, provided it preserves Notarius
+  boundaries around provider plumbing, prompt ownership, diagnostics, and secret
+  handling.
+
 ## Candidate Operational Work
 
 - Packaged release artifacts for alpha distribution.

docs/roadmap/implementation.md

@@ -1,305 +0,0 @@
-# Extraction References Implementation Plan
-
-## Purpose
-
-Implement the extraction-reference feature described in
-[references.md](references.md). This plan is decision-complete for an LLM coding
-agent: implement each stage in order, keep the repository compiling after each
-stage, and do not move planned behavior into non-roadmap docs until the relevant
-behavior exists.
-
-Core decisions to preserve:
-
-- references are opaque framework inputs and domain semantics stay in extract
-  modules;
-- references are not evidence and must not be addressable through `SourceRef`;
-- reference binding is lane-scoped;
-- extractors expose `ReferenceSlots()` directly on the first-class extractor
-  contract;
-- token budgeting is deferred; enforce only UTF-8 text handling, empty-file
-  warnings, and declared `MaxBytes`;
-- run manifests record references in a dedicated section, separate from
-  `source_digests`;
-- CLI unbinding uses `--without-reference`.
-
-## Stage 1: Contracts and Mechanical Adoption
-
-Add the framework contracts needed to describe references without changing
-runtime behavior. Slot declarations must be available without constructing
-extractor modules, because pipeline/config validation should use registry
-metadata rather than runtime module instances.
-
-Implementation steps:
-
-- In `internal/framework/contracts`, add reference model types:
-  `ReferenceSlot`, `ReferenceOrigin`, `ReferenceItem`,
-  `ResolvedReferenceSlot`, `ReferenceSet`, and a binding-source enum or string
-  constants for `config` and `cli`.
-- Include `Name`, `Description`, `Required`, `AcceptedMediaTypes`, `Multiple`,
-  and `MaxBytes` on `ReferenceSlot`.
-- Include slot name, media type, content bytes, digest, origin, size bytes, and
-  binding source on `ReferenceItem`.
-- Add `References ReferenceSet` to `contracts.ExtractionRequest`.
-- Add `ReferenceSlots() []ReferenceSlot` to `contracts.Extractor`.
-- Extend extractor registration metadata so reference slots are also declared
-  through the extractor's registry spec. Prefer the smallest idiomatic change to
-  the existing registry model, such as adding `ReferenceSlots` to `ModuleSpec`
-  with validation that non-extractor modules leave it empty, unless the codebase
-  shape clearly supports a narrower extractor-specific spec.
-- Update every concrete extractor and all extractor fakes/test doubles to
-  implement `ReferenceSlots()`. Existing extractors without references should
-  return `nil`.
-- Add tests that compare a production extractor's runtime `ReferenceSlots()`
-  with its registered spec slots so the two declarations cannot drift.
-- Add contract tests for empty reference sets, slot copying expectations if
-  helpers are introduced, and compile-time coverage through existing fakes.
-
-Verification:
-
-- `go test ./internal/framework/contracts`
-- `go test ./internal/framework/pipeline`
-- `go test ./...`
-
-## Stage 2: Config Shape and Pipeline-Level Resolution
-
-Add unresolved reference bindings to config and resolved lane bindings to the
-pipeline model. Do not read reference files in this stage.
-
-Implementation steps:
-
-- Add `references` maps to file config parsing at both pipeline and artifact
-  lane level.
-- Add corresponding fields to `pipeline.PipelineProfile` and
-  `pipeline.ArtifactLaneProfile`.
-- Preserve deterministic map handling and duplicate-after-trim validation.
-- Extend config cloning, effective config, redaction, validation, and tests for
-  the new fields.
-- Add resolved reference binding structures to `internal/framework/pipeline`.
-  They should represent lane ID, slot name, source URI/path, and binding source,
-  but not file bytes.
-- During `pipeline.ResolvePipeline`, collect selected lanes, read each lane
-  extractor's declared slots from registry metadata, and validate without
-  building extractor instances:
-  - every bound slot is declared by the lane extractor;
-  - required slots are bound after applying pipeline-level and lane-level config;
-  - required slots remain bound after any CLI unbinds supplied to resolution;
-  - selected lanes under `--only` are the only lanes considered.
-- Apply pipeline-level bindings as defaults only to lanes whose extractor
-  declares the matching slot.
-- Apply lane-level bindings as overrides or additions for that lane.
-- Keep reference bindings out of source digests and artifact source references.
-- Add tests proving reference-slot validation works through registry specs even
-  when extractor constructors would fail if called.
-
-Verification:
-
-- `go test ./internal/core/config`
-- `go test ./internal/framework/pipeline`
-- `go test ./...`
-
-## Stage 3: CLI Reference Overrides and Unbinds
-
-Add run-time CLI syntax for reference binding overrides and optional unbinding.
-
-Implementation steps:
-
-- Add repeatable `--reference` flags to `notarius run`.
-  Accepted forms:
-  - `slot=path` for unambiguous slot names across selected lanes;
-  - `lane.slot=path` for explicit lane-scoped binding.
-- Add repeatable `--without-reference` flags to `notarius run`.
-  Accepted forms:
-  - `slot`;
-  - `lane.slot`.
-- Reject empty paths for `--reference`; use `--without-reference` for unbinding.
-- Reject malformed values with concise CLI errors before expensive work.
-- Pass parsed override/unbind requests into config/pipeline resolution.
-- Resolve flat CLI names only when exactly one selected lane declares the slot.
-  If multiple selected lanes declare the same slot, fail and instruct the user
-  to use `lane.slot`.
-- Let CLI bindings override config bindings for the same lane and slot.
-- Let CLI unbinds remove config-bound optional slots for the same lane and slot.
-- Fail if unbinding leaves a required slot unbound.
-- Add CLI tests for flat binding, lane-qualified binding, ambiguous flat
-  binding, malformed syntax, optional unbind, and required-slot unbind failure.
-
-Verification:
-
-- `go test ./internal/cli`
-- `go test ./internal/core/config`
-- `go test ./internal/framework/pipeline`
-- `go test ./...`
-
-## Stage 4: Run Preparation and Reference Materialization
-
-Read, validate, digest, and materialize resolved file references before any LLM
-call.
-
-Implementation steps:
-
-- Add a reference resolver/materializer near pipeline run preparation. Keep file
-  I/O out of pure config parsing.
-- Ensure run preparation receives the loaded config path or config directory so
-  config-relative reference paths can be resolved after pure config parsing.
-- Resolve config-relative paths relative to the config file path and
-  CLI-relative paths relative to the current working directory.
-- For MVP, accept only UTF-8 text files. Reject non-UTF-8 content with an error
-  naming pipeline, lane, slot, and path.
-- Compute `sha256:` content digests over the raw reference bytes.
-- Populate `ReferenceItem` values with content bytes, media type, digest,
-  origin type `file`, normalized origin URI/path, size bytes, and binding source.
-- Enforce declared `MaxBytes` when greater than zero. The error should name the
-  pipeline, lane, slot, actual size, limit, and path.
-- Emit a warning for empty bound files, but do not fail.
-- Add `ReferenceSet` values to the runner input or resolved pipeline path in a
-  way that keeps lane-scoped references available when calling each extractor.
-- Pass the correct lane-specific `ReferenceSet` into
-  `contracts.ExtractionRequest`.
-- Ensure no reference content is written to ordinary diagnostics, logs, errors,
-  or manifests.
-
-Verification:
-
-- Focused resolver/materializer tests for path resolution, digest stability,
-  UTF-8 rejection, empty-file warning, `MaxBytes`, and binding source.
-- `go test ./internal/cli`
-- `go test ./internal/framework/pipeline`
-- `go test ./...`
-
-## Stage 5: Prompt Template Reference Functions
-
-Make references available to module-owned prompt templates.
-
-Implementation steps:
-
-- Extend `internal/framework/prompt` so prompt bundles can be compiled with
-  declared reference slots.
-- Add `reference` and `hasreference` template functions.
-- Validate at bundle build time, or the earliest feasible equivalent, that
-  templates reference only declared slots.
-- Render a declared but unbound optional slot as an empty string.
-- Ensure `hasreference` returns true only when the slot has at least one bound
-  item with content.
-- Render multiple items deterministically if future `Multiple` support is
-  enabled; for MVP, reject multiple bindings unless the slot declares
-  `Multiple`.
-- Keep prompt metadata hashes based on template source. Do not include rendered
-  reference content in prompt identity.
-- Add deterministic rendering tests proving byte-identical output across runs
-  with the same reference bytes and config.
-
-Verification:
-
-- `go test ./internal/framework/prompt`
-- `go test ./internal/modules/extract/dnd/spells`
-- `go test ./...`
-
-## Stage 6: Manifest and Diagnostics Provenance
-
-Record reference provenance separately from source provenance.
-
-Implementation steps:
-
-- Add a dedicated references section to `artifacts.RunManifest`.
-  The shape should be lane-scoped and include lane ID, slot name, origin type,
-  origin URI/path, digest, media type, size bytes, and binding source.
-- Do not add reference digests to `source_digests`.
-- Include reference digests in any cache/idempotency key if such a key exists.
-  If no cache/idempotency key exists, add a test or comment documenting that no
-  additional key needs updating yet.
-- Write a diagnostics artifact for resolved references that contains provenance
-  only, not full content, consistent with redacted effective config behavior.
-- Ensure durable JSON output manifests include the new manifest section.
-- Add manifest round-trip tests and a CLI/run test where two runs that differ
-  only in reference bytes produce distinguishable manifests.
-
-Verification:
-
-- `go test ./internal/core/artifacts`
-- `go test ./internal/core/diagnostics`
-- `go test ./internal/modules/output/json`
-- `go test ./internal/cli`
-- `go test ./...`
-
-## Stage 7: D&D Spells Consumer
-
-Use the new reference feature in the first production extractor.
-
-Implementation steps:
-
-- Declare optional `roster` and `glossary` slots on `dnd/spells`.
-- Set accepted media type to text/UTF-8. Add conservative `MaxBytes` limits only
-  if a clear module-owned limit is chosen; otherwise leave `MaxBytes` unset.
-- Update the D&D spells prompt bundle to include conditional reference sections
-  using `hasreference` and `reference`.
-- Frame references as supporting material only. The prompt must instruct the
-  model to extract only spell-cast events present in the source transcript and
-  use references only for disambiguation.
-- Update prompt metadata tests as needed while preserving template-hash
-  semantics.
-- Add fixture coverage with no references, with roster/glossary references, and
-  with a roster that mentions a spell never cast in the transcript. The last
-  case must assert no spell-cast artifact is produced for the uncast spell.
-- If existing deterministic source-reference validation can be extended
-  cleanly, add warning-level relatedness checks for spell names or close
-  variants near cited source text. If this becomes large, defer that validator
-  enhancement to a separate roadmap item and keep the prompt/regression fixture
-  guard in this stage.
-
-Verification:
-
-- `go test ./internal/modules/extract/dnd/spells`
-- `go test ./internal/framework/pipeline`
-- `go test ./internal/cli`
-- `go test ./...`
-
-## Stage 8: Canonical Documentation and Examples
-
-Move implemented behavior out of roadmap-only status once code exists.
-
-Implementation steps:
-
-- Update `docs/cli.md` with `--reference` and `--without-reference` syntax,
-  precedence, ambiguity behavior, and examples.
-- Update `docs/config.md` with pipeline-level and lane-level `references`
-  blocks.
-- Update `docs/internal/modules.md` or the most appropriate internal docs with
-  module-author guidance for `ReferenceSlots()`, reference request delivery,
-  prompt functions, evidence exclusion, and provenance.
-- Update `docs/internal/pipeline.md` with reference resolution lifecycle and
-  lane-scoped delivery.
-- Update `docs/integrations/json-output.md` with the manifest reference
-  provenance shape.
-- Update `docs/operations.md` or `docs/troubleshooting.md` for common reference
-  errors such as unknown slot, ambiguous flat override, missing required slot,
-  unreadable file, non-UTF-8 content, and `MaxBytes` failures.
-- Add maintained example reference files and update `examples/dnd-spells.config.yml`
-  only after the CLI/config behavior is implemented and covered by tests.
-- Keep future-only material in `docs/roadmap/references.md`; do not duplicate
-  canonical current behavior there after implementation.
-
-Verification:
-
-- `rg -n "references:|--reference|--without-reference|ReferenceSlots|reference \"|hasreference" docs examples`
-- `go test ./...`
-- `go vet ./...`
-- `go build ./cmd/notarius`
-
-## Final Acceptance Criteria
-
-The feature is complete when:
-
-- extractor modules can declare reference slots through the first-class
-  extractor contract;
-- config and CLI can bind and unbind lane-scoped file references;
-- selected-pipeline validation catches unknown, ambiguous, or missing required
-  references before any LLM call;
-- run preparation materializes UTF-8 text references with digests, size checks,
-  and empty-file warnings;
-- extractors receive lane-scoped resolved references;
-- prompt templates can render `reference` and `hasreference` deterministically;
-- D&D spell extraction uses optional roster and glossary references;
-- run manifests and diagnostics record reference provenance without recording
-  full content or treating references as source evidence;
-- canonical docs and maintained examples describe only implemented behavior;
-- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.

docs/roadmap/references.md

@@ -1,238 +0,0 @@
-# Feature Roadmap: Extraction References
-
-## Status
-
-This document defines the target state and policy choices for the planned
-extraction-reference feature in Notarius. It describes planned behavior, not
-implemented behavior. The staged implementation plan lives in
-[implementation.md](implementation.md).
-
-## Goal
-
-Extraction quality improves when an LLM-backed extractor receives stable
-reference material alongside the source input. For the initial D&D spell
-extractor, useful reference material includes a party roster, a player list, and
-a campaign glossary.
-
-Notarius should support passing this material to extractors as **named reference
-items** without introducing domain-specific concepts into core or framework
-packages. The framework should know only that:
-
-- extractors declare named reference slots they accept;
-- pipeline config and CLI flags bind content, initially files, to those slots;
-- bound content is rendered into module-owned prompt templates;
-- bound content is digested and recorded as run provenance.
-
-Only extract modules should know what a "roster" or "glossary" means. Domain
-semantics live in module-owned slot declarations and prompt templates.
-
-## Definitions
-
-- **Reference slot**: a named, typed-by-convention input declared by an
-  extractor, with a human-readable description, required/optional status, and
-  optional guardrails such as accepted media types and maximum bytes.
-- **Reference item**: resolved content bound to a slot for a given run: slot
-  name, content bytes, media type, content digest, size, origin, and binding
-  source.
-- **Reference binding**: the association of a slot name to a content source,
-  defined in pipeline config and overridable per run via CLI.
-- **Reference set**: the lane-scoped collection of resolved reference items
-  delivered to an extractor.
-
-## Architectural Principles
-
-- References are opaque to the framework. Core and framework packages must not
-  interpret reference content or recognize domain slot names.
-- References are inputs. Anything that can change extraction output must be
-  digested into the run manifest and participate in any cache or idempotency key.
-- References are not evidence. `SourceRef` values must only ever reference
-  source units. Reference items must not receive unit IDs and must not be
-  addressable by source references.
-- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting a
-  required slot, should fail before any LLM call.
-- Reference delivery is lane-scoped. Pipeline-level bindings may apply to
-  multiple lanes, but each lane receives only the references declared by its
-  extractor after pipeline, lane, CLI override, and CLI unbind rules are
-  resolved.
-- Optional slots degrade gracefully. Prompt templates should render cleanly
-  whether or not an optional slot is bound.
-- Rendering must be deterministic. Identical source input, config, prompts, and
-  reference bytes should produce byte-identical rendered prompts. Reference
-  slots should render in declaration order, with stable binding order within a
-  slot.
-
-## Target Contracts
-
-Extractors should declare accepted reference slots directly on the extractor
-contract. This is a first-class feature, so mechanical updates to existing
-extractors and test fakes are acceptable.
-
-The target slot declaration includes:
-
-- `Name`;
-- `Description`;
-- `Required`;
-- `AcceptedMediaTypes`;
-- `Multiple`;
-- `MaxBytes`.
-
-Extractors with no reference needs return an empty slot list.
-
-Resolved reference items should be content-bearing values, not unresolved file
-paths. The MVP producer is "read this file," but the item shape should permit
-future producers such as prior-run artifacts, derived summaries, or entity
-registries without changing extractor-facing contracts.
-
-The extraction request should carry the lane-scoped resolved reference set.
-Framework and core code should treat the set as opaque bytes plus metadata.
-
-## Binding Lifecycle
-
-Reference handling should be split across existing lifecycle boundaries:
-
-1. Config parsing records pipeline-level and lane-level reference bindings
-   without reading files.
-2. Pipeline resolution validates selected lanes, declared extractor slots,
-   missing required slots, unknown bindings, and ambiguous flat CLI bindings.
-3. Run preparation resolves paths, reads files, validates media type and size,
-   computes digests, and materializes reference items.
-4. Extraction receives the lane-specific resolved reference set.
-
-Config-relative paths resolve relative to the config file. CLI-relative paths
-resolve relative to the current working directory.
-
-## Configuration and CLI
-
-Reference bindings should live in pipeline config because the initial use cases
-are campaign-invariant more often than run-variant. Bindings should be supported
-at two levels:
-
-- pipeline level: defaults shared by artifact lanes whose extractors declare
-  matching slots;
-- lane level: additions or overrides for a single artifact lane.
-
-Illustrative config shape:
-
-```yaml
-pipelines:
-  dnd-session:
-    input: seriatim
-    references:
-      roster: ./campaign/party_roster.md
-      glossary: ./campaign/glossary.md
-    artifacts:
-      spells:
-        extract: dnd/spells
-      npcs:
-        extract: dnd/npcs
-        references:
-          npc_registry: ./campaign/npcs.md
-```
-
-Per-run CLI binding overrides should be repeatable:
-
-```text
-notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md
-```
-
-Flat CLI slot names are allowed when unambiguous across selected lanes. Lane
-qualified names, such as `spells.roster=./alt_roster.md`, disambiguate or target
-a specific lane. CLI bindings override config bindings for the same lane and
-slot.
-
-Users should also be able to unbind a config-bound optional slot for a run with
-an explicit repeatable flag:
-
-```text
-notarius run dnd-session --input session-014.json --without-reference roster
-```
-
-Unbinding a required slot should fail during pipeline/reference resolution.
-
-## Prompt Template Integration
-
-Prompt templates are module-owned. Template rendering should expose:
-
-- `{{ reference "roster" }}`: renders the content of the bound item;
-- `{{ hasreference "glossary" }}`: predicate for conditional sections, so
-  optional slots can be included only when bound.
-
-Rules:
-
-- Referencing an undeclared slot from a template is a module bug and should fail
-  at prompt registration/build time or the earliest feasible equivalent.
-- Referencing a declared but unbound optional slot should render as empty;
-  templates should use `hasreference` to avoid dangling section headers.
-- Rendering must be deterministic and independent of map iteration order.
-- Prompt identity should be computed over the template, not the rendered prompt.
-  Reference digests are recorded separately in the manifest so a reference edit
-  is visible as a reference change, not a prompt change.
-
-Reference content is repeated in every per-chunk prompt in the MVP. Per-slot or
-per-chunk inclusion policies are deferred until cost data justifies them.
-
-## Provenance
-
-The run manifest must record resolved references separately from source
-digests. For every bound lane and slot, it should record:
-
-- lane ID;
-- slot name;
-- origin type and URI;
-- content digest;
-- media type;
-- size in bytes;
-- whether the binding came from config or CLI override.
-
-Reference digests must participate in any idempotency/cache key alongside source
-digests, prompt hashes, schema versions, model, and parameters. Two runs that
-differ only in reference content must be distinguishable from the manifest
-alone.
-
-Diagnostics for a run should include the resolved binding set with digests, not
-full reference content, consistent with the existing redacted-effective-config
-pattern.
-
-## Validation and Guardrails
-
-### References Are Not Evidence
-
-The primary new failure mode is the model extracting facts from references
-rather than from the source input. For example, a roster may list a player
-character's known spells, and the model might emit a spell-cast artifact for a
-spell that was never cast in the session.
-
-Defenses, in priority order:
-
-1. **Structural.** `SourceRef` remains the only grounding mechanism and can only
-   reference source units. No contract change should make references
-   addressable as evidence.
-2. **Prompt discipline.** Module templates should frame references explicitly as
-   reference material, such as "use the roster to resolve speakers to
-   characters; extract only events that occur in the transcript."
-3. **Validator support.** The source-reference validator, or a sibling
-   deterministic validator, should warn when referenced source text does not
-   plausibly relate to the extracted fact. Severity should be `warn`, not
-   `fail`, because transcripts can use paraphrase, nicknames, and abbreviations.
-4. **Regression fixtures.** Tests should include a fixture in which a bound
-   roster mentions a spell that is never cast in the transcript, asserting no
-   artifact record is produced for it.
-
-### Size and Sanity Guardrails
-
-- MVP accepts UTF-8 text content only. Other media types should be rejected with
-  a clear error.
-- A slot-level `MaxBytes` value should be enforced when declared.
-- Empty bound files should produce a warning because they are likely user error.
-
-## Out of Scope
-
-- Token budgeting and model context-window management for references.
-- Non-file reference producers, including prior-run artifacts, derived
-  summaries, and entity registries.
-- Per-chunk or per-slot inclusion policies.
-- Structured or parsed references such as typed roster schemas. References are
-  opaque text handed to prompts.
-- Reference caching, preprocessing, summarization, embedding, or retrieval.
-- Making references addressable as evidence in any form.
-

docs/troubleshooting.md

@@ -113,6 +113,7 @@ Symptoms include:
 - `--without-reference must use slot or lane.slot`
 - `read "...": no such file`
 - `must be UTF-8 text`
+- `media type "..." is not accepted`
 - `is ... bytes, limit ...`
 
 Fix:
@@ -127,6 +128,9 @@ Fix:
   directory.
 - Ensure the file is readable UTF-8 text and within any byte limit declared by
   the extractor.
+- If the extractor narrows accepted media types, use a file extension that
+  infers an accepted type such as `text/markdown` or `application/json`.
+  Unknown extensions infer `application/octet-stream`.
 - If diagnostics are retained, inspect `resolved-pipeline.json`,
   `resolved-references.json`, and `error.log`.
 

internal/framework/pipeline/profile.go

@@ -135,6 +135,9 @@ func ResolvePipeline(profile PipelineProfile, options ResolveOptions, catalog Mo
 	if len(selectedLaneIDs) == 0 {
 		return ResolvedPipeline{}, fmt.Errorf("pipeline %q must select at least one artifact lane", pipelineID)
 	}
+	if err := validatePipelineReferenceDefaults(pipelineID, profile.References, lanesByID, catalog); err != nil {
+		return ResolvedPipeline{}, err
+	}
 
 	resolved := ResolvedPipeline{
 		ID:     pipelineID,
@@ -238,6 +241,44 @@ func resolveArtifactLane(
 	return lane, capabilities, nil
 }
 
+func validatePipelineReferenceDefaults(
+	pipelineID string,
+	pipelineReferences map[string]string,
+	lanesByID map[string]ArtifactLaneProfile,
+	catalog ModuleCatalog,
+) error {
+	normalizedPipelineReferences, err := normalizedReferenceMap(pipelineReferences, fmt.Sprintf("pipeline %q reference slot", pipelineID))
+	if err != nil {
+		return err
+	}
+	if len(normalizedPipelineReferences) == 0 {
+		return nil
+	}
+
+	declaredByAnyLane := make(map[string]struct{}, len(normalizedPipelineReferences))
+	for _, laneID := range sortedArtifactLaneProfileKeys(lanesByID) {
+		laneProfile := lanesByID[laneID]
+		extract := resolveBinding(laneProfile.Extract, "")
+		if extract.Module == "" {
+			return fmt.Errorf("pipeline %q lane %q extract module must not be empty", pipelineID, laneID)
+		}
+		extractSpec, err := extractorSpec(catalog, extract.Module)
+		if err != nil {
+			return moduleLookupError(pipelineID, laneID, StageExtract, extract.Module, err)
+		}
+		for _, slot := range extractSpec.ReferenceSlots {
+			declaredByAnyLane[slot.Name] = struct{}{}
+		}
+	}
+
+	for _, slotName := range sortedStringMapKeys(normalizedPipelineReferences) {
+		if _, ok := declaredByAnyLane[slotName]; !ok {
+			return fmt.Errorf("pipeline %q reference slot %q is not declared by any artifact lane", pipelineID, slotName)
+		}
+	}
+	return nil
+}
+
 func resolveReferenceBindings(
 	pipelineID string,
 	laneID string,
@@ -370,6 +411,18 @@ func normalizedReferenceMap(values map[string]string, keyName string) (map[strin
 	return out, nil
 }
 
+func sortedArtifactLaneProfileKeys(values map[string]ArtifactLaneProfile) []string {
+	if len(values) == 0 {
+		return nil
+	}
+	keys := make([]string, 0, len(values))
+	for key := range values {
+		keys = append(keys, key)
+	}
+	sort.Strings(keys)
+	return keys
+}
+
 func sortedStringMapKeys(values map[string]string) []string {
 	if len(values) == 0 {
 		return nil

internal/framework/pipeline/profile_test.go

@@ -130,7 +130,6 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
 	profile := multiLaneProfile()
 	profile.References = map[string]string{
 		" roster ": " ./shared-roster.yml ",
-		"unclaimed": "./ignored.yml",
 	}
 	lane := profile.Artifacts["events"]
 	lane.References = map[string]string{
@@ -168,6 +167,39 @@ func TestResolvePipelineAppliesReferenceBindings(t *testing.T) {
 	}
 }
 
+func TestResolvePipelineAllowsPipelineReferenceDeclaredOnlyByUnselectedLane(t *testing.T) {
+	profile := multiLaneProfile()
+	profile.References = map[string]string{"notes_context": "./notes.md"}
+	catalog := newProfileCatalogWithOverride(t, ModuleSpec{
+		Key:      "note-extractor",
+		Stage:    StageExtract,
+		Requires: []string{"chunk"},
+		Provides: []string{"candidate"},
+		ReferenceSlots: []contracts.ReferenceSlot{
+			{Name: "notes_context"},
+		},
+	})
+
+	resolved, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, catalog)
+	if err != nil {
+		t.Fatalf("ResolvePipeline() error = %v, want nil", err)
+	}
+	if refs := resolved.ArtifactLanes[0].References; len(refs) != 0 {
+		t.Fatalf("selected lane references = %#v, want none", refs)
+	}
+}
+
+func TestResolvePipelineRejectsPipelineReferenceNotDeclaredByAnyLane(t *testing.T) {
+	profile := multiLaneProfile()
+	profile.References = map[string]string{"missing": "./missing.md"}
+
+	_, err := ResolvePipeline(profile, ResolveOptions{Only: []string{"events"}}, newProfileCatalog(t))
+	if err == nil {
+		t.Fatal("ResolvePipeline() error = nil, want error")
+	}
+	assertErrorContains(t, err, "multi", "reference slot", "missing", "not declared")
+}
+
 func TestResolvePipelineRejectsUndeclaredReferenceSlot(t *testing.T) {
 	profile := baselineProfile()
 	lane := profile.Artifacts["events"]

internal/framework/pipeline/references.go

@@ -4,6 +4,7 @@ import (
 	"crypto/sha256"
 	"encoding/hex"
 	"fmt"
+	"mime"
 	"net/url"
 	"os"
 	"path/filepath"
@@ -17,7 +18,8 @@ import (
 
 const (
 	referenceOriginFile = "file"
-	referenceMediaType  = "text/plain; charset=utf-8"
+	referenceMediaType  = "text/plain"
+	unknownMediaType    = "application/octet-stream"
 )
 
 type ReferenceMaterializationOptions struct {
@@ -88,6 +90,10 @@ func materializeLaneReferences(
 		if !utf8.Valid(content) {
 			return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q path %q must be UTF-8 text", pipelineID, lane.ID, slotName, path)
 		}
+		mediaType := referenceMediaTypeForPath(path)
+		if !referenceMediaTypeAccepted(mediaType, slot.AcceptedMediaTypes) {
+			return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q path %q media type %q is not accepted", pipelineID, lane.ID, slotName, path, mediaType)
+		}
 		if slot.MaxBytes > 0 && int64(len(content)) > slot.MaxBytes {
 			return contracts.ReferenceSet{}, nil, fmt.Errorf("pipeline %q lane %q reference slot %q path %q is %d bytes, limit %d", pipelineID, lane.ID, slotName, path, len(content), slot.MaxBytes)
 		}
@@ -101,7 +107,7 @@ func materializeLaneReferences(
 
 		item := contracts.ReferenceItem{
 			SlotName:      slotName,
-			MediaType:     referenceMediaType,
+			MediaType:     mediaType,
 			Content:       append([]byte(nil), content...),
 			Digest:        referenceDigest(content),
 			Origin:        contracts.ReferenceOrigin{Type: referenceOriginFile, URI: fileURI(path)},
@@ -116,6 +122,43 @@ func materializeLaneReferences(
 	return set, warnings, nil
 }
 
+func referenceMediaTypeForPath(path string) string {
+	extension := strings.ToLower(filepath.Ext(path))
+	mediaType := mime.TypeByExtension(extension)
+	if strings.TrimSpace(mediaType) == "" {
+		if extension == ".md" || extension == ".markdown" {
+			return "text/markdown"
+		}
+		return unknownMediaType
+	}
+	return canonicalMediaType(mediaType)
+}
+
+func referenceMediaTypeAccepted(mediaType string, accepted []string) bool {
+	if len(accepted) == 0 {
+		return true
+	}
+	mediaType = canonicalMediaType(mediaType)
+	for _, value := range accepted {
+		if strings.EqualFold(mediaType, canonicalMediaType(value)) {
+			return true
+		}
+	}
+	return false
+}
+
+func canonicalMediaType(mediaType string) string {
+	trimmed := strings.TrimSpace(mediaType)
+	if trimmed == "" {
+		return ""
+	}
+	parsed, _, err := mime.ParseMediaType(trimmed)
+	if err != nil {
+		return strings.ToLower(trimmed)
+	}
+	return strings.ToLower(parsed)
+}
+
 func referencePath(binding ReferenceBinding, options ReferenceMaterializationOptions) (string, error) {
 	source := strings.TrimSpace(binding.Source)
 	if source == "" {

internal/framework/pipeline/references_test.go

@@ -117,6 +117,96 @@ func TestMaterializeReferencesRejectsNonUTF8Content(t *testing.T) {
 	}
 }
 
+func TestMaterializeReferencesAllowsAnyMediaTypeWhenSlotDoesNotRestrictIt(t *testing.T) {
+	configDir := t.TempDir()
+	path := filepath.Join(configDir, "roster.reference")
+	writeReferenceFile(t, path, []byte("plain text"))
+
+	resolved := resolvedPipelineWithReference(t, "roster", "roster.reference", contracts.ReferenceBindingSourceConfig, contracts.ReferenceSlot{Name: "roster"})
+	materialized, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{{Name: "roster"}}), ReferenceMaterializationOptions{
+		ConfigPath: filepath.Join(configDir, "config.yml"),
+	})
+	if err != nil {
+		t.Fatalf("MaterializeReferences() error = %v, want nil", err)
+	}
+	item := materialized.ArtifactLanes[0].ReferenceSet.Slots["roster"].Items[0]
+	if item.MediaType != unknownMediaType {
+		t.Fatalf("MediaType = %q, want %q", item.MediaType, unknownMediaType)
+	}
+}
+
+func TestMaterializeReferencesAcceptsDeclaredMarkdownMediaType(t *testing.T) {
+	configDir := t.TempDir()
+	path := filepath.Join(configDir, "glossary.md")
+	writeReferenceFile(t, path, []byte("# Terms\n"))
+
+	slot := contracts.ReferenceSlot{Name: "glossary", AcceptedMediaTypes: []string{"text/markdown"}}
+	resolved := resolvedPipelineWithReference(t, "glossary", "glossary.md", contracts.ReferenceBindingSourceConfig, slot)
+	materialized, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
+		ConfigPath: filepath.Join(configDir, "config.yml"),
+	})
+	if err != nil {
+		t.Fatalf("MaterializeReferences() error = %v, want nil", err)
+	}
+	item := materialized.ArtifactLanes[0].ReferenceSet.Slots["glossary"].Items[0]
+	if item.MediaType != "text/markdown" {
+		t.Fatalf("MediaType = %q, want text/markdown", item.MediaType)
+	}
+}
+
+func TestMaterializeReferencesAcceptsDeclaredJSONMediaType(t *testing.T) {
+	configDir := t.TempDir()
+	path := filepath.Join(configDir, "roster.json")
+	writeReferenceFile(t, path, []byte(`{"aria":"cleric"}`))
+
+	slot := contracts.ReferenceSlot{Name: "roster", AcceptedMediaTypes: []string{"application/json"}}
+	resolved := resolvedPipelineWithReference(t, "roster", "roster.json", contracts.ReferenceBindingSourceConfig, slot)
+	materialized, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
+		ConfigPath: filepath.Join(configDir, "config.yml"),
+	})
+	if err != nil {
+		t.Fatalf("MaterializeReferences() error = %v, want nil", err)
+	}
+	item := materialized.ArtifactLanes[0].ReferenceSet.Slots["roster"].Items[0]
+	if item.MediaType != "application/json" {
+		t.Fatalf("MediaType = %q, want application/json", item.MediaType)
+	}
+}
+
+func TestMaterializeReferencesRejectsUnacceptedMediaType(t *testing.T) {
+	configDir := t.TempDir()
+	path := filepath.Join(configDir, "roster.json")
+	writeReferenceFile(t, path, []byte(`{"aria":"cleric"}`))
+
+	slot := contracts.ReferenceSlot{Name: "roster", AcceptedMediaTypes: []string{"text/markdown"}}
+	resolved := resolvedPipelineWithReference(t, "roster", "roster.json", contracts.ReferenceBindingSourceConfig, slot)
+	_, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
+		ConfigPath: filepath.Join(configDir, "config.yml"),
+	})
+	if err == nil || !strings.Contains(err.Error(), "media type") || !strings.Contains(err.Error(), "application/json") || !strings.Contains(err.Error(), "roster") {
+		t.Fatalf("error = %v, want media type rejection", err)
+	}
+}
+
+func TestMaterializeReferencesMatchesAcceptedMediaTypesIgnoringParameters(t *testing.T) {
+	configDir := t.TempDir()
+	path := filepath.Join(configDir, "roster.txt")
+	writeReferenceFile(t, path, []byte("Aria\n"))
+
+	slot := contracts.ReferenceSlot{Name: "roster", AcceptedMediaTypes: []string{"text/plain; charset=utf-8"}}
+	resolved := resolvedPipelineWithReference(t, "roster", "roster.txt", contracts.ReferenceBindingSourceConfig, slot)
+	materialized, _, err := MaterializeReferences(resolved, referenceCatalog(t, []contracts.ReferenceSlot{slot}), ReferenceMaterializationOptions{
+		ConfigPath: filepath.Join(configDir, "config.yml"),
+	})
+	if err != nil {
+		t.Fatalf("MaterializeReferences() error = %v, want nil", err)
+	}
+	item := materialized.ArtifactLanes[0].ReferenceSet.Slots["roster"].Items[0]
+	if item.MediaType != referenceMediaType {
+		t.Fatalf("MediaType = %q, want %q", item.MediaType, referenceMediaType)
+	}
+}
+
 func TestMaterializeReferencesWarnsForEmptyFiles(t *testing.T) {
 	configDir := t.TempDir()
 	path := filepath.Join(configDir, "empty.txt")