eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add a staged implementation plan for background context references

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-05 09:08:31 -05:00
parent 11d8187052
commit f9999a73df
2 changed files with 439 additions and 236 deletions
Download Patch File Download Diff File Expand all files Collapse all files

370
docs/roadmap/references.md
View File

@@ -1,150 +1,117 @@
# Feature Roadmap Proposal: Extraction Reference
# Feature Roadmap: Extraction References
## Status
This document captures proposed design and implementation sequencing for the
extraction-reference feature in Notarius. It describes planned work, not
implemented behavior. Go snippets are conceptual sketches; the implementing
agent should adapt names and shapes to the existing contracts, package
boundaries, and conventions in this repository.
This document defines the target state and policy choices for the planned
extraction-reference feature in Notarius. It describes planned behavior, not
implemented behavior. The staged implementation plan lives in
[implementation.md](implementation.md).
## Goal
Extraction quality improves significantly when the LLM receives reference
material alongside the source input. For the initial D&D spell extractor,
useful reference material includes a party roster (mapping players to player
characters), a player list, and a campaign glossary.
Extraction quality improves when an LLM-backed extractor receives stable
reference material alongside the source input. For the initial D&D spell
extractor, useful reference material includes a party roster, a player list, and
a campaign glossary.
Notarius should support passing this material to extractors as **named reference
items** without introducing any domain-specific concepts into core or framework
items** without introducing domain-specific concepts into core or framework
packages. The framework should know only that:
- extractors declare named reference slots they accept;
- pipeline config and CLI flags bind content (initially files) to those slots;
- pipeline config and CLI flags bind content, initially files, to those slots;
- bound content is rendered into module-owned prompt templates;
- bound content is digested and recorded as run provenance.
Only extract modules should know what a "roster" or "glossary" means. All
domain semantics live in module-owned slot declarations and prompt templates.
Only extract modules should know what a "roster" or "glossary" means. Domain
semantics live in module-owned slot declarations and prompt templates.
## Definitions
- **Reference slot**: a named, typed-by-convention input declared by an
extractor, with a human-readable description and a required/optional flag.
Example: extractor `dnd/spells` declares an optional slot named `roster`.
- **Reference item**: resolved content bound to a slot for a given run: name,
content bytes, media type, content digest, and origin (initially a file
path).
extractor, with a human-readable description, required/optional status, and
optional guardrails such as accepted media types and maximum bytes.
- **Reference item**: resolved content bound to a slot for a given run: slot
name, content bytes, media type, content digest, size, origin, and binding
source.
- **Reference binding**: the association of a slot name to a content source,
defined in pipeline config and overridable per run via CLI.
- **Reference set**: the lane-scoped collection of resolved reference items
delivered to an extractor.
## Architectural Principles
- Reference is opaque to the framework. Core and framework packages must not
- References are opaque to the framework. Core and framework packages must not
interpret reference content or recognize domain slot names.
- Reference is an input. Anything that changes extraction output must be
digested into the run manifest and participate in any cache key.
- A Reference is not evidence. `SourceRef` values must only ever reference source
units. Reference items must not receive unit IDs and must not be addressable
by source references.
- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting
a required slot, should fail at config-load time, before any LLM call.
- References are inputs. Anything that can change extraction output must be
digested into the run manifest and participate in any cache or idempotency key.
- References are not evidence. `SourceRef` values must only ever reference
source units. Reference items must not receive unit IDs and must not be
addressable by source references.
- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting a
required slot, should fail before any LLM call.
- Reference delivery is lane-scoped. Pipeline-level bindings may apply to
multiple lanes, but each lane receives only the references declared by its
extractor after pipeline, lane, CLI override, and CLI unbind rules are
resolved.
- Optional slots degrade gracefully. Prompt templates should render cleanly
whether or not an optional slot is bound.
- Determinism. Identical input, config, prompts, and reference bytes should
produce byte-identical rendered prompts. Reference slots should render in a
stable, documented order (declaration order).
- Rendering must be deterministic. Identical source input, config, prompts, and
reference bytes should produce byte-identical rendered prompts. Reference
slots should render in declaration order, with stable binding order within a
slot.
## Proposed Contracts
## Target Contracts
### Slot declaration (extractor contract extension)
Extractors should declare accepted reference slots directly on the extractor
contract. This is a first-class feature, so mechanical updates to existing
extractors and test fakes are acceptable.
Extractors should declare the reference slots they accept:
The target slot declaration includes:
```go
type ReferenceSlot struct {
Name string
Description string
Required bool
- `Name`;
- `Description`;
- `Required`;
- `AcceptedMediaTypes`;
- `Multiple`;
- `MaxBytes`.
// MVP can leave these empty/defaulted, but having the fields now
// makes validation and future docs easier.
AcceptedMediaTypes []string
Multiple bool
MaxBytes int64
}
```
Extractors with no reference needs return an empty slot list.
The extractor interface should gain a method such as:
Resolved reference items should be content-bearing values, not unresolved file
paths. The MVP producer is "read this file," but the item shape should permit
future producers such as prior-run artifacts, derived summaries, or entity
registries without changing extractor-facing contracts.
```go
ReferenceSlots() []ReferenceSlot
```
The extraction request should carry the lane-scoped resolved reference set.
Framework and core code should treat the set as opaque bytes plus metadata.
Extractors with no reference needs return an empty slice. Existing extractors
should require no other changes.
## Binding Lifecycle
### Resolved reference item
Reference handling should be split across existing lifecycle boundaries:
```go
type ReferenceItem struct {
SlotName string
MediaType string
Content []byte
Digest string
Origin ReferenceOrigin
1. Config parsing records pipeline-level and lane-level reference bindings
without reading files.
2. Pipeline resolution validates selected lanes, declared extractor slots,
missing required slots, unknown bindings, and ambiguous flat CLI bindings.
3. Run preparation resolves paths, reads files, validates media type and size,
computes digests, and materializes reference items.
4. Extraction receives the lane-specific resolved reference set.
SizeBytes int64
TokenEstimate int
}
type ReferenceOrigin struct {
Type string // "file" for MVP
URI string // path or future artifact URI
}
type ReferenceSet struct {
// Stable declaration order, then stable binding order within a slot.
Slots []ResolvedReferenceSlot
}
type ResolvedReferenceSlot struct {
Name string
Items []ReferenceItem
}
```
`ReferenceItem` is a resolved-content type, not a file path. The only MVP
producer is "read this file," but the shape should permit future producers
(prior-run artifacts, derived summaries, entity registries) without contract
changes.
### Binding resolution
A resolver should, at config-load time:
1. Collect declared slots from every extractor selected by the active
pipeline (respecting lane selection, e.g. `--only`).
2. Collect bindings from pipeline config (pipeline-level and lane-level) and
CLI overrides, applying the standard layering: config file, then CLI.
3. Fail with a clear error if a required slot is unbound, or if a binding
references a slot no extractor within the selected pipeline declares.
Errors should name the pipeline, lane, slot, and the slot description.
4. Read, digest, and materialize each bound source into a `ReferenceItem`.
5. Enforce size guardrails (see Validation and Guardrails).
Config-relative paths resolve relative to the config file. CLI-relative paths
resolve relative to the current working directory.
## Configuration and CLI
### Pipeline config
Reference bindings should live in pipeline config because the initial use cases
are campaign-invariant more often than run-variant. Bindings should be supported
at two levels:
Reference bindings should live in pipeline config, because the initial use cases
(roster, glossary) are campaign-invariant rather than run-variant. Bindings
should be supported at two levels:
- pipeline level: defaults shared by artifact lanes whose extractors declare
matching slots;
- lane level: additions or overrides for a single artifact lane.
- pipeline level: shared by all artifact lanes;
- lane level: additions or overrides for a single lane.
Illustrative shape (adapt to the existing config format):
Illustrative config shape:
```yaml
pipelines:
@@ -158,22 +125,29 @@ pipelines:
extract: dnd/spells
npcs:
extract: dnd/npcs
reference:
references:
npc_registry: ./campaign/npcs.md
```
### CLI
Per-run override flag, repeatable:
Per-run CLI binding overrides should be repeatable:
```text
notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md
```
CLI bindings override config bindings for the same slot name. The existing
pipeline-describe/config-validate commands (or their nearest equivalents)
should surface declared slots, descriptions, required flags, and current
bindings so users can discover what a pipeline accepts.
Flat CLI slot names are allowed when unambiguous across selected lanes. Lane
qualified names, such as `spells.roster=./alt_roster.md`, disambiguate or target
a specific lane. CLI bindings override config bindings for the same lane and
slot.
Users should also be able to unbind a config-bound optional slot for a run with
an explicit repeatable flag:
```text
notarius run dnd-session --input session-014.json --without-reference roster
```
Unbinding a required slot should fail during pipeline/reference resolution.
## Prompt Template Integration
@@ -185,30 +159,29 @@ Prompt templates are module-owned. Template rendering should expose:
Rules:
- Referencing an **undeclared** slot from a template is a module bug and
should fail at prompt registration/build time (or earliest feasible point),
not silently at render time.
- Referencing a declared but unbound **optional** slot should render as
empty; templates should use `hasreference` to avoid dangling section headers.
- Referencing an undeclared slot from a template is a module bug and should fail
at prompt registration/build time or the earliest feasible equivalent.
- Referencing a declared but unbound optional slot should render as empty;
templates should use `hasreference` to avoid dangling section headers.
- Rendering must be deterministic and independent of map iteration order.
- Prompt identity (registry hash) should be computed over the **template**,
not the rendered prompt. Reference digests are recorded separately in the
manifest, so a reference edit is visible as a reference change, not a prompt
change.
- Prompt identity should be computed over the template, not the rendered prompt.
Reference digests are recorded separately in the manifest so a reference edit
is visible as a reference change, not a prompt change.
Note: reference content is repeated in every per-chunk prompt. Diagnostics
should record per-slot token or byte counts so reference cost is observable.
Per-slot inclusion policies (e.g., roster in every chunk, glossary on demand)
are explicitly out of scope until cost data justifies them.
Reference content is repeated in every per-chunk prompt in the MVP. Per-slot or
per-chunk inclusion policies are deferred until cost data justifies them.
## Provenance
The run manifest must record, for every bound slot:
The run manifest must record resolved references separately from source
digests. For every bound lane and slot, it should record:
- lane ID;
- slot name;
- origin (path);
- origin type and URI;
- content digest;
- media type;
- size in bytes;
- whether the binding came from config or CLI override.
Reference digests must participate in any idempotency/cache key alongside source
@@ -216,125 +189,50 @@ digests, prompt hashes, schema versions, model, and parameters. Two runs that
differ only in reference content must be distinguishable from the manifest
alone.
Diagnostics for a run should include the resolved binding set (with digests,
not necessarily full content) in the run directory, consistent with the
existing redacted-effective-config pattern.
## Path Resolution
- Config-relative paths resolve relative to the pipeline config file.
- CLI-relative paths resolve relative to the current working directory.
- Manifest records the normalized absolute path or a redacted/display path
according to existing diagnostics policy.
Diagnostics for a run should include the resolved binding set with digests, not
full reference content, consistent with the existing redacted-effective-config
pattern.
## Validation and Guardrails
### References are not evidence
### References Are Not Evidence
The primary new failure mode: the model extracts facts from references rather
than from the source input. Example: the roster lists a PC's known spells, and
the model emits a `SpellCast` for a spell that was never cast in the session,
with a fabricated or misattributed source reference.
The primary new failure mode is the model extracting facts from references
rather than from the source input. For example, a roster may list a player
character's known spells, and the model might emit a spell-cast artifact for a
spell that was never cast in the session.
Defenses, in priority order:
1. **Structural.** `SourceRef` remains the only grounding mechanism and can
only reference source units. No contract change should make references
1. **Structural.** `SourceRef` remains the only grounding mechanism and can only
reference source units. No contract change should make references
addressable as evidence.
2. **Prompt discipline.** Module templates should frame references explicitly as
reference material, e.g. "use the roster to resolve speakers to
characters; extract only events that occur in the transcript." This
guidance belongs in the module prompt guidelines, not framework code.
3. **Validator support.** The source-reference validator (or a sibling
deterministic validator) should support checking that referenced source
text plausibly relates to the extracted fact (e.g., spell name or a close
variant appears in or near the referenced range). Severity should be
`warn`, not `fail`, given paraphrase and nickname casting.
4. **Regression fixtures.** Golden-file tests must include a fixture in which
the bound roster mentions a spell that is never cast in the transcript,
asserting no artifact record is produced for it. This regression is likely
to be reintroduced by future prompt edits; the fixture is the guard.
reference material, such as "use the roster to resolve speakers to
characters; extract only events that occur in the transcript."
3. **Validator support.** The source-reference validator, or a sibling
deterministic validator, should warn when referenced source text does not
plausibly relate to the extracted fact. Severity should be `warn`, not
`fail`, because transcripts can use paraphrase, nicknames, and abbreviations.
4. **Regression fixtures.** Tests should include a fixture in which a bound
roster mentions a spell that is never cast in the transcript, asserting no
artifact record is produced for it.
### Size and sanity guardrails
### Size and Sanity Guardrails
- Fail fast, before any LLM call, if bound references plus template plus largest
chunk exceeds the configured model context budget, with an error that names
the offending slot(s) and sizes.
- Empty bound files should produce a warning (probable user error).
- MVP accepts text content only (`utf-8`); other media
types should be rejected with a clear error.
- MVP accepts UTF-8 text content only. Other media types should be rejected with
a clear error.
- A slot-level `MaxBytes` value should be enforced when declared.
- Empty bound files should produce a warning because they are likely user error.
## Out of Scope (MVP)
## Out of Scope
- Non-file reference producers (prior-run artifacts, derived summaries, entity
registries). The `ReferenceItem` shape should permit them later.
- Per-chunk or per-slot inclusion policies and context budgeting beyond the
fail-fast guardrail.
- Structured/parsed references (e.g., typed roster schemas). References are opaque
text handed to prompts.
- Reference caching or preprocessing (summarization, embedding, retrieval).
- Making reference addressable as evidence, in any form.
- Token budgeting and model context-window management for references.
- Non-file reference producers, including prior-run artifacts, derived
summaries, and entity registries.
- Per-chunk or per-slot inclusion policies.
- Structured or parsed references such as typed roster schemas. References are
opaque text handed to prompts.
- Reference caching, preprocessing, summarization, embedding, or retrieval.
- Making references addressable as evidence in any form.
## Checkpoint Sequencing
Each checkpoint should leave the repository compiling, with targeted tests
covering newly introduced contracts or behavior.
1. **Contracts and resolution.** Add `ReferenceSlot`, `ReferenceItem`, and the
extractor `ReferenceSlots()` method (empty default for existing extractors).
Implement config parsing for pipeline- and lane-level bindings, CLI
override flag, layering, and load-time validation (unknown slot, missing
required slot, unreadable file, empty file warning). Unit tests for
resolution and error cases.
2. **Prompt rendering.** Add `reference`/`hasreference` template functions,
declaration-order rendering, undeclared-slot failure at registration, and
deterministic-render tests (byte-identical output across runs).
3. **Provenance.** Record bindings (name, origin, digest, media type,
binding source) in the run manifest and diagnostics; include reference
digests in the cache/idempotency key if one exists. Tests: manifest
round-trip; two runs differing only in reference content produce differing
manifests.
4. **Guardrails and validation.** Context-window fail-fast check;
relatedness `warn` validator (or extension of the source-reference
validator); media-type rejection.
5. **First consumer.** Declare `roster` (optional) and `glossary` (optional)
slots on the D&D spells extractor; update its prompt template with
conditional reference sections and reference-material framing; add golden
fixtures with and without references bound, including the
roster-mentions-uncast-spell fixture. This checkpoint is the acceptance
test for the feature: spell extraction quality with a roster bound should
visibly improve speaker-to-character attribution in fixtures.
## Open Design Questions
The implementing agent should resolve these against existing code and record
decisions in the implementation plan:
- Should slot names be namespaced per lane in config and CLI (e.g.,
`spells.roster=...`) or flat with lane-level config as the only
disambiguator? (Recommended default: flat names; lane-level config for
overrides; revisit if two extractors in one pipeline want the same slot
name with different content.)
- Where does binding resolution live relative to the existing config and
pipeline packages? It must run at load time, alongside existing pipeline
validation.
- Does the existing prompt registry hash templates or rendered prompts? If
rendered, this feature requires moving to template hashing as described in
Provenance.
- Should CLI overrides be permitted to bind slots that config leaves unbound
(yes, presumably), and to *unbind* a config-bound optional slot (e.g.,
`--reference roster=` to clear)? Decide and test both directions.
## Documentation Tasks
Once implemented, move contracts out of this roadmap into canonical docs:
- `docs/cli.md`: `--reference` flag syntax, layering, and examples;
- `docs/config.md`: pipeline- and lane-level `references` blocks;
- `docs/internal/`: slot/item contracts, resolution flow, evidence
exclusion rule, and template function reference for module authors;
- module-author guidance: how to declare slots, write conditional reference
sections, and frame reference material in prompts;
- `examples/`: a maintained example pipeline with a roster and glossary
bound, plus matching fixture files.
```