notarius/docs/roadmap/references.md

# Feature Roadmap: Extraction References

## Status

This document defines the target state and policy choices for the planned
extraction-reference feature in Notarius. It describes planned behavior, not
implemented behavior. The staged implementation plan lives in
[implementation.md](implementation.md).

## Goal

Extraction quality improves when an LLM-backed extractor receives stable
reference material alongside the source input. For the initial D&D spell
extractor, useful reference material includes a party roster, a player list, and
a campaign glossary.

Notarius should support passing this material to extractors as **named reference
items** without introducing domain-specific concepts into core or framework
packages. The framework should know only that:

- extractors declare named reference slots they accept;
- pipeline config and CLI flags bind content, initially files, to those slots;
- bound content is rendered into module-owned prompt templates;
- bound content is digested and recorded as run provenance.

Only extract modules should know what a "roster" or "glossary" means. Domain
semantics live in module-owned slot declarations and prompt templates.

## Definitions

- **Reference slot**: a named, typed-by-convention input declared by an
  extractor, with a human-readable description, required/optional status, and
  optional guardrails such as accepted media types and maximum bytes.
- **Reference item**: resolved content bound to a slot for a given run: slot
  name, content bytes, media type, content digest, size, origin, and binding
  source.
- **Reference binding**: the association of a slot name to a content source,
  defined in pipeline config and overridable per run via CLI.
- **Reference set**: the lane-scoped collection of resolved reference items
  delivered to an extractor.

## Architectural Principles

- References are opaque to the framework. Core and framework packages must not
  interpret reference content or recognize domain slot names.
- References are inputs. Anything that can change extraction output must be
  digested into the run manifest and participate in any cache or idempotency key.
- References are not evidence. `SourceRef` values must only ever reference
  source units. Reference items must not receive unit IDs and must not be
  addressable by source references.
- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting a
  required slot, should fail before any LLM call.
- Reference delivery is lane-scoped. Pipeline-level bindings may apply to
  multiple lanes, but each lane receives only the references declared by its
  extractor after pipeline, lane, CLI override, and CLI unbind rules are
  resolved.
- Optional slots degrade gracefully. Prompt templates should render cleanly
  whether or not an optional slot is bound.
- Rendering must be deterministic. Identical source input, config, prompts, and
  reference bytes should produce byte-identical rendered prompts. Reference
  slots should render in declaration order, with stable binding order within a
  slot.

## Target Contracts

Extractors should declare accepted reference slots directly on the extractor
contract. This is a first-class feature, so mechanical updates to existing
extractors and test fakes are acceptable.

The target slot declaration includes:

- `Name`;
- `Description`;
- `Required`;
- `AcceptedMediaTypes`;
- `Multiple`;
- `MaxBytes`.

Extractors with no reference needs return an empty slot list.

Resolved reference items should be content-bearing values, not unresolved file
paths. The MVP producer is "read this file," but the item shape should permit
future producers such as prior-run artifacts, derived summaries, or entity
registries without changing extractor-facing contracts.

The extraction request should carry the lane-scoped resolved reference set.
Framework and core code should treat the set as opaque bytes plus metadata.

## Binding Lifecycle

Reference handling should be split across existing lifecycle boundaries:

1. Config parsing records pipeline-level and lane-level reference bindings
   without reading files.
2. Pipeline resolution validates selected lanes, declared extractor slots,
   missing required slots, unknown bindings, and ambiguous flat CLI bindings.
3. Run preparation resolves paths, reads files, validates media type and size,
   computes digests, and materializes reference items.
4. Extraction receives the lane-specific resolved reference set.

Config-relative paths resolve relative to the config file. CLI-relative paths
resolve relative to the current working directory.

## Configuration and CLI

Reference bindings should live in pipeline config because the initial use cases
are campaign-invariant more often than run-variant. Bindings should be supported
at two levels:

- pipeline level: defaults shared by artifact lanes whose extractors declare
  matching slots;
- lane level: additions or overrides for a single artifact lane.

Illustrative config shape:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    references:
      roster: ./campaign/party_roster.md
      glossary: ./campaign/glossary.md
    artifacts:
      spells:
        extract: dnd/spells
      npcs:
        extract: dnd/npcs
        references:
          npc_registry: ./campaign/npcs.md
```

Per-run CLI binding overrides should be repeatable:

```text
notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md
```

Flat CLI slot names are allowed when unambiguous across selected lanes. Lane
qualified names, such as `spells.roster=./alt_roster.md`, disambiguate or target
a specific lane. CLI bindings override config bindings for the same lane and
slot.

Users should also be able to unbind a config-bound optional slot for a run with
an explicit repeatable flag:

```text
notarius run dnd-session --input session-014.json --without-reference roster
```

Unbinding a required slot should fail during pipeline/reference resolution.

## Prompt Template Integration

Prompt templates are module-owned. Template rendering should expose:

- `{{ reference "roster" }}`: renders the content of the bound item;
- `{{ hasreference "glossary" }}`: predicate for conditional sections, so
  optional slots can be included only when bound.

Rules:

- Referencing an undeclared slot from a template is a module bug and should fail
  at prompt registration/build time or the earliest feasible equivalent.
- Referencing a declared but unbound optional slot should render as empty;
  templates should use `hasreference` to avoid dangling section headers.
- Rendering must be deterministic and independent of map iteration order.
- Prompt identity should be computed over the template, not the rendered prompt.
  Reference digests are recorded separately in the manifest so a reference edit
  is visible as a reference change, not a prompt change.

Reference content is repeated in every per-chunk prompt in the MVP. Per-slot or
per-chunk inclusion policies are deferred until cost data justifies them.

## Provenance

The run manifest must record resolved references separately from source
digests. For every bound lane and slot, it should record:

- lane ID;
- slot name;
- origin type and URI;
- content digest;
- media type;
- size in bytes;
- whether the binding came from config or CLI override.

Reference digests must participate in any idempotency/cache key alongside source
digests, prompt hashes, schema versions, model, and parameters. Two runs that
differ only in reference content must be distinguishable from the manifest
alone.

Diagnostics for a run should include the resolved binding set with digests, not
full reference content, consistent with the existing redacted-effective-config
pattern.

## Validation and Guardrails

### References Are Not Evidence

The primary new failure mode is the model extracting facts from references
rather than from the source input. For example, a roster may list a player
character's known spells, and the model might emit a spell-cast artifact for a
spell that was never cast in the session.

Defenses, in priority order:

1. **Structural.** `SourceRef` remains the only grounding mechanism and can only
   reference source units. No contract change should make references
   addressable as evidence.
2. **Prompt discipline.** Module templates should frame references explicitly as
   reference material, such as "use the roster to resolve speakers to
   characters; extract only events that occur in the transcript."
3. **Validator support.** The source-reference validator, or a sibling
   deterministic validator, should warn when referenced source text does not
   plausibly relate to the extracted fact. Severity should be `warn`, not
   `fail`, because transcripts can use paraphrase, nicknames, and abbreviations.
4. **Regression fixtures.** Tests should include a fixture in which a bound
   roster mentions a spell that is never cast in the transcript, asserting no
   artifact record is produced for it.

### Size and Sanity Guardrails

- MVP accepts UTF-8 text content only. Other media types should be rejected with
  a clear error.
- A slot-level `MaxBytes` value should be enforced when declared.
- Empty bound files should produce a warning because they are likely user error.

## Out of Scope

- Token budgeting and model context-window management for references.
- Non-file reference producers, including prior-run artifacts, derived
  summaries, and entity registries.
- Per-chunk or per-slot inclusion policies.
- Structured or parsed references such as typed roster schemas. References are
  opaque text handed to prompts.
- Reference caching, preprocessing, summarization, embedding, or retrieval.
- Making references addressable as evidence in any form.