From 8a5419448f0549a44eb16646e4c26c5577357568 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 5 Jul 2026 08:11:54 -0500 Subject: [PATCH] Update feature roadmaps to reflect future work --- docs/roadmap/{mvp.md => future.md} | 0 docs/roadmap/references.md | 340 +++++++++++++++++++++++++++++ 2 files changed, 340 insertions(+) rename docs/roadmap/{mvp.md => future.md} (100%) create mode 100644 docs/roadmap/references.md diff --git a/docs/roadmap/mvp.md b/docs/roadmap/future.md similarity index 100% rename from docs/roadmap/mvp.md rename to docs/roadmap/future.md diff --git a/docs/roadmap/references.md b/docs/roadmap/references.md new file mode 100644 index 0000000..1376d49 --- /dev/null +++ b/docs/roadmap/references.md @@ -0,0 +1,340 @@ +# Feature Roadmap Proposal: Extraction Reference + +## Status + +This document captures proposed design and implementation sequencing for the +extraction-reference feature in Notarius. It describes planned work, not +implemented behavior. Go snippets are conceptual sketches; the implementing +agent should adapt names and shapes to the existing contracts, package +boundaries, and conventions in this repository. + +## Goal + +Extraction quality improves significantly when the LLM receives reference +material alongside the source input. For the initial D&D spell extractor, +useful reference material includes a party roster (mapping players to player +characters), a player list, and a campaign glossary. + +Notarius should support passing this material to extractors as **named reference +items** without introducing any domain-specific concepts into core or framework +packages. The framework should know only that: + +- extractors declare named reference slots they accept; +- pipeline config and CLI flags bind content (initially files) to those slots; +- bound content is rendered into module-owned prompt templates; +- bound content is digested and recorded as run provenance. + +Only extract modules should know what a "roster" or "glossary" means. All +domain semantics live in module-owned slot declarations and prompt templates. + +## Definitions + +- **Reference slot**: a named, typed-by-convention input declared by an + extractor, with a human-readable description and a required/optional flag. + Example: extractor `dnd/spells` declares an optional slot named `roster`. +- **Reference item**: resolved content bound to a slot for a given run: name, + content bytes, media type, content digest, and origin (initially a file + path). +- **Reference binding**: the association of a slot name to a content source, + defined in pipeline config and overridable per run via CLI. + +## Architectural Principles + +- Reference is opaque to the framework. Core and framework packages must not + interpret reference content or recognize domain slot names. +- Reference is an input. Anything that changes extraction output must be + digested into the run manifest and participate in any cache key. +- A Reference is not evidence. `SourceRef` values must only ever reference source + units. Reference items must not receive unit IDs and must not be addressable + by source references. +- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting + a required slot, should fail at config-load time, before any LLM call. +- Optional slots degrade gracefully. Prompt templates should render cleanly + whether or not an optional slot is bound. +- Determinism. Identical input, config, prompts, and reference bytes should + produce byte-identical rendered prompts. Reference slots should render in a + stable, documented order (declaration order). + +## Proposed Contracts + +### Slot declaration (extractor contract extension) + +Extractors should declare the reference slots they accept: + +```go +type ReferenceSlot struct { + Name string + Description string + Required bool + + // MVP can leave these empty/defaulted, but having the fields now + // makes validation and future docs easier. + AcceptedMediaTypes []string + Multiple bool + MaxBytes int64 +} +``` + +The extractor interface should gain a method such as: + +```go +ReferenceSlots() []ReferenceSlot +``` + +Extractors with no reference needs return an empty slice. Existing extractors +should require no other changes. + +### Resolved reference item + +```go +type ReferenceItem struct { + SlotName string + MediaType string + Content []byte + Digest string + Origin ReferenceOrigin + + SizeBytes int64 + TokenEstimate int +} + +type ReferenceOrigin struct { + Type string // "file" for MVP + URI string // path or future artifact URI +} + +type ReferenceSet struct { + // Stable declaration order, then stable binding order within a slot. + Slots []ResolvedReferenceSlot +} + +type ResolvedReferenceSlot struct { + Name string + Items []ReferenceItem +} +``` + +`ReferenceItem` is a resolved-content type, not a file path. The only MVP +producer is "read this file," but the shape should permit future producers +(prior-run artifacts, derived summaries, entity registries) without contract +changes. + +### Binding resolution + +A resolver should, at config-load time: + +1. Collect declared slots from every extractor selected by the active + pipeline (respecting lane selection, e.g. `--only`). +2. Collect bindings from pipeline config (pipeline-level and lane-level) and + CLI overrides, applying the standard layering: config file, then CLI. +3. Fail with a clear error if a required slot is unbound, or if a binding + references a slot no extractor within the selected pipeline declares. + Errors should name the pipeline, lane, slot, and the slot description. +4. Read, digest, and materialize each bound source into a `ReferenceItem`. +5. Enforce size guardrails (see Validation and Guardrails). + +## Configuration and CLI + +### Pipeline config + +Reference bindings should live in pipeline config, because the initial use cases +(roster, glossary) are campaign-invariant rather than run-variant. Bindings +should be supported at two levels: + +- pipeline level: shared by all artifact lanes; +- lane level: additions or overrides for a single lane. + +Illustrative shape (adapt to the existing config format): + +```yaml +pipelines: + dnd-session: + input: seriatim + references: + roster: ./campaign/party_roster.md + glossary: ./campaign/glossary.md + artifacts: + spells: + extract: dnd/spells + npcs: + extract: dnd/npcs + reference: + npc_registry: ./campaign/npcs.md +``` + +### CLI + +Per-run override flag, repeatable: + +```text +notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md +``` + +CLI bindings override config bindings for the same slot name. The existing +pipeline-describe/config-validate commands (or their nearest equivalents) +should surface declared slots, descriptions, required flags, and current +bindings so users can discover what a pipeline accepts. + +## Prompt Template Integration + +Prompt templates are module-owned. Template rendering should expose: + +- `{{ reference "roster" }}`: renders the content of the bound item; +- `{{ hasreference "glossary" }}`: predicate for conditional sections, so + optional slots can be included only when bound. + +Rules: + +- Referencing an **undeclared** slot from a template is a module bug and + should fail at prompt registration/build time (or earliest feasible point), + not silently at render time. +- Referencing a declared but unbound **optional** slot should render as + empty; templates should use `hasreference` to avoid dangling section headers. +- Rendering must be deterministic and independent of map iteration order. +- Prompt identity (registry hash) should be computed over the **template**, + not the rendered prompt. Reference digests are recorded separately in the + manifest, so a reference edit is visible as a reference change, not a prompt + change. + +Note: reference content is repeated in every per-chunk prompt. Diagnostics +should record per-slot token or byte counts so reference cost is observable. +Per-slot inclusion policies (e.g., roster in every chunk, glossary on demand) +are explicitly out of scope until cost data justifies them. + +## Provenance + +The run manifest must record, for every bound slot: + +- slot name; +- origin (path); +- content digest; +- media type; +- whether the binding came from config or CLI override. + +Reference digests must participate in any idempotency/cache key alongside source +digests, prompt hashes, schema versions, model, and parameters. Two runs that +differ only in reference content must be distinguishable from the manifest +alone. + +Diagnostics for a run should include the resolved binding set (with digests, +not necessarily full content) in the run directory, consistent with the +existing redacted-effective-config pattern. + +## Path Resolution + +- Config-relative paths resolve relative to the pipeline config file. +- CLI-relative paths resolve relative to the current working directory. +- Manifest records the normalized absolute path or a redacted/display path +according to existing diagnostics policy. + +## Validation and Guardrails + +### References are not evidence + +The primary new failure mode: the model extracts facts from references rather +than from the source input. Example: the roster lists a PC's known spells, and +the model emits a `SpellCast` for a spell that was never cast in the session, +with a fabricated or misattributed source reference. + +Defenses, in priority order: + +1. **Structural.** `SourceRef` remains the only grounding mechanism and can + only reference source units. No contract change should make references + addressable as evidence. +2. **Prompt discipline.** Module templates should frame references explicitly as + reference material, e.g. "use the roster to resolve speakers to + characters; extract only events that occur in the transcript." This + guidance belongs in the module prompt guidelines, not framework code. +3. **Validator support.** The source-reference validator (or a sibling + deterministic validator) should support checking that referenced source + text plausibly relates to the extracted fact (e.g., spell name or a close + variant appears in or near the referenced range). Severity should be + `warn`, not `fail`, given paraphrase and nickname casting. +4. **Regression fixtures.** Golden-file tests must include a fixture in which + the bound roster mentions a spell that is never cast in the transcript, + asserting no artifact record is produced for it. This regression is likely + to be reintroduced by future prompt edits; the fixture is the guard. + +### Size and sanity guardrails + +- Fail fast, before any LLM call, if bound references plus template plus largest + chunk exceeds the configured model context budget, with an error that names + the offending slot(s) and sizes. +- Empty bound files should produce a warning (probable user error). +- MVP accepts text content only (`utf-8`); other media + types should be rejected with a clear error. + +## Out of Scope (MVP) + +- Non-file reference producers (prior-run artifacts, derived summaries, entity + registries). The `ReferenceItem` shape should permit them later. +- Per-chunk or per-slot inclusion policies and context budgeting beyond the + fail-fast guardrail. +- Structured/parsed references (e.g., typed roster schemas). References are opaque + text handed to prompts. +- Reference caching or preprocessing (summarization, embedding, retrieval). +- Making reference addressable as evidence, in any form. + +## Checkpoint Sequencing + +Each checkpoint should leave the repository compiling, with targeted tests +covering newly introduced contracts or behavior. + +1. **Contracts and resolution.** Add `ReferenceSlot`, `ReferenceItem`, and the + extractor `ReferenceSlots()` method (empty default for existing extractors). + Implement config parsing for pipeline- and lane-level bindings, CLI + override flag, layering, and load-time validation (unknown slot, missing + required slot, unreadable file, empty file warning). Unit tests for + resolution and error cases. +2. **Prompt rendering.** Add `reference`/`hasreference` template functions, + declaration-order rendering, undeclared-slot failure at registration, and + deterministic-render tests (byte-identical output across runs). +3. **Provenance.** Record bindings (name, origin, digest, media type, + binding source) in the run manifest and diagnostics; include reference + digests in the cache/idempotency key if one exists. Tests: manifest + round-trip; two runs differing only in reference content produce differing + manifests. +4. **Guardrails and validation.** Context-window fail-fast check; + relatedness `warn` validator (or extension of the source-reference + validator); media-type rejection. +5. **First consumer.** Declare `roster` (optional) and `glossary` (optional) + slots on the D&D spells extractor; update its prompt template with + conditional reference sections and reference-material framing; add golden + fixtures with and without references bound, including the + roster-mentions-uncast-spell fixture. This checkpoint is the acceptance + test for the feature: spell extraction quality with a roster bound should + visibly improve speaker-to-character attribution in fixtures. + +## Open Design Questions + +The implementing agent should resolve these against existing code and record +decisions in the implementation plan: + +- Should slot names be namespaced per lane in config and CLI (e.g., + `spells.roster=...`) or flat with lane-level config as the only + disambiguator? (Recommended default: flat names; lane-level config for + overrides; revisit if two extractors in one pipeline want the same slot + name with different content.) +- Where does binding resolution live relative to the existing config and + pipeline packages? It must run at load time, alongside existing pipeline + validation. +- Does the existing prompt registry hash templates or rendered prompts? If + rendered, this feature requires moving to template hashing as described in + Provenance. +- Should CLI overrides be permitted to bind slots that config leaves unbound + (yes, presumably), and to *unbind* a config-bound optional slot (e.g., + `--reference roster=` to clear)? Decide and test both directions. + +## Documentation Tasks + +Once implemented, move contracts out of this roadmap into canonical docs: + +- `docs/cli.md`: `--reference` flag syntax, layering, and examples; +- `docs/config.md`: pipeline- and lane-level `references` blocks; +- `docs/internal/`: slot/item contracts, resolution flow, evidence + exclusion rule, and template function reference for module authors; +- module-author guidance: how to declare slots, write conditional reference + sections, and frame reference material in prompts; +- `examples/`: a maintained example pipeline with a roster and glossary + bound, plus matching fixture files. +``` \ No newline at end of file