Finish the references implementation for the extraction module and update roadmap documentation
This commit is contained in:
@@ -8,6 +8,21 @@ future work only.
|
||||
|
||||
- Additional input adapters, such as Markdown or note-export formats.
|
||||
- Additional D&D extractors beyond spell casts.
|
||||
- First-class LLM access for all LLM-eligible pipeline stages: `chunk`,
|
||||
`extract`, and `normalize`.
|
||||
- Expand reference support beyond extractors to `chunk` and `normalize`
|
||||
modules, using the same declared-slot model and preserving stage-specific
|
||||
ownership boundaries.
|
||||
- Add non-file reference producers, such as prior-run artifacts, derived
|
||||
summaries, or entity registries, without changing module-facing reference
|
||||
item contracts.
|
||||
- Add token budgeting and model context-window management for reference content.
|
||||
- Add per-slot or per-chunk inclusion policies so modules can avoid repeating
|
||||
large reference content in every prompt when that becomes important.
|
||||
- Add structured or parsed references, such as typed roster schemas, when a
|
||||
module has a clear need for more than opaque UTF-8 text.
|
||||
- Add reference caching, preprocessing, summarization, embedding, or retrieval
|
||||
if references become large enough to require preprocessing.
|
||||
- Cross-lane entity normalization.
|
||||
- Cross-chunk semantic deduplication.
|
||||
- Configurable validator chains with production validator modules.
|
||||
@@ -15,6 +30,13 @@ future work only.
|
||||
- Parallel execution where it preserves deterministic manifests and diagnostics.
|
||||
- Additional output encoders.
|
||||
|
||||
## Candidate Architecture Work
|
||||
|
||||
- Evaluate replacing the local LLM adapter with an import from
|
||||
`gitea.maximumdirect.net/eric/scriptorium`, provided it preserves Notarius
|
||||
boundaries around provider plumbing, prompt ownership, diagnostics, and secret
|
||||
handling.
|
||||
|
||||
## Candidate Operational Work
|
||||
|
||||
- Packaged release artifacts for alpha distribution.
|
||||
|
||||
@@ -1,305 +0,0 @@
|
||||
# Extraction References Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement the extraction-reference feature described in
|
||||
[references.md](references.md). This plan is decision-complete for an LLM coding
|
||||
agent: implement each stage in order, keep the repository compiling after each
|
||||
stage, and do not move planned behavior into non-roadmap docs until the relevant
|
||||
behavior exists.
|
||||
|
||||
Core decisions to preserve:
|
||||
|
||||
- references are opaque framework inputs and domain semantics stay in extract
|
||||
modules;
|
||||
- references are not evidence and must not be addressable through `SourceRef`;
|
||||
- reference binding is lane-scoped;
|
||||
- extractors expose `ReferenceSlots()` directly on the first-class extractor
|
||||
contract;
|
||||
- token budgeting is deferred; enforce only UTF-8 text handling, empty-file
|
||||
warnings, and declared `MaxBytes`;
|
||||
- run manifests record references in a dedicated section, separate from
|
||||
`source_digests`;
|
||||
- CLI unbinding uses `--without-reference`.
|
||||
|
||||
## Stage 1: Contracts and Mechanical Adoption
|
||||
|
||||
Add the framework contracts needed to describe references without changing
|
||||
runtime behavior. Slot declarations must be available without constructing
|
||||
extractor modules, because pipeline/config validation should use registry
|
||||
metadata rather than runtime module instances.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- In `internal/framework/contracts`, add reference model types:
|
||||
`ReferenceSlot`, `ReferenceOrigin`, `ReferenceItem`,
|
||||
`ResolvedReferenceSlot`, `ReferenceSet`, and a binding-source enum or string
|
||||
constants for `config` and `cli`.
|
||||
- Include `Name`, `Description`, `Required`, `AcceptedMediaTypes`, `Multiple`,
|
||||
and `MaxBytes` on `ReferenceSlot`.
|
||||
- Include slot name, media type, content bytes, digest, origin, size bytes, and
|
||||
binding source on `ReferenceItem`.
|
||||
- Add `References ReferenceSet` to `contracts.ExtractionRequest`.
|
||||
- Add `ReferenceSlots() []ReferenceSlot` to `contracts.Extractor`.
|
||||
- Extend extractor registration metadata so reference slots are also declared
|
||||
through the extractor's registry spec. Prefer the smallest idiomatic change to
|
||||
the existing registry model, such as adding `ReferenceSlots` to `ModuleSpec`
|
||||
with validation that non-extractor modules leave it empty, unless the codebase
|
||||
shape clearly supports a narrower extractor-specific spec.
|
||||
- Update every concrete extractor and all extractor fakes/test doubles to
|
||||
implement `ReferenceSlots()`. Existing extractors without references should
|
||||
return `nil`.
|
||||
- Add tests that compare a production extractor's runtime `ReferenceSlots()`
|
||||
with its registered spec slots so the two declarations cannot drift.
|
||||
- Add contract tests for empty reference sets, slot copying expectations if
|
||||
helpers are introduced, and compile-time coverage through existing fakes.
|
||||
|
||||
Verification:
|
||||
|
||||
- `go test ./internal/framework/contracts`
|
||||
- `go test ./internal/framework/pipeline`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 2: Config Shape and Pipeline-Level Resolution
|
||||
|
||||
Add unresolved reference bindings to config and resolved lane bindings to the
|
||||
pipeline model. Do not read reference files in this stage.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Add `references` maps to file config parsing at both pipeline and artifact
|
||||
lane level.
|
||||
- Add corresponding fields to `pipeline.PipelineProfile` and
|
||||
`pipeline.ArtifactLaneProfile`.
|
||||
- Preserve deterministic map handling and duplicate-after-trim validation.
|
||||
- Extend config cloning, effective config, redaction, validation, and tests for
|
||||
the new fields.
|
||||
- Add resolved reference binding structures to `internal/framework/pipeline`.
|
||||
They should represent lane ID, slot name, source URI/path, and binding source,
|
||||
but not file bytes.
|
||||
- During `pipeline.ResolvePipeline`, collect selected lanes, read each lane
|
||||
extractor's declared slots from registry metadata, and validate without
|
||||
building extractor instances:
|
||||
- every bound slot is declared by the lane extractor;
|
||||
- required slots are bound after applying pipeline-level and lane-level config;
|
||||
- required slots remain bound after any CLI unbinds supplied to resolution;
|
||||
- selected lanes under `--only` are the only lanes considered.
|
||||
- Apply pipeline-level bindings as defaults only to lanes whose extractor
|
||||
declares the matching slot.
|
||||
- Apply lane-level bindings as overrides or additions for that lane.
|
||||
- Keep reference bindings out of source digests and artifact source references.
|
||||
- Add tests proving reference-slot validation works through registry specs even
|
||||
when extractor constructors would fail if called.
|
||||
|
||||
Verification:
|
||||
|
||||
- `go test ./internal/core/config`
|
||||
- `go test ./internal/framework/pipeline`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 3: CLI Reference Overrides and Unbinds
|
||||
|
||||
Add run-time CLI syntax for reference binding overrides and optional unbinding.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Add repeatable `--reference` flags to `notarius run`.
|
||||
Accepted forms:
|
||||
- `slot=path` for unambiguous slot names across selected lanes;
|
||||
- `lane.slot=path` for explicit lane-scoped binding.
|
||||
- Add repeatable `--without-reference` flags to `notarius run`.
|
||||
Accepted forms:
|
||||
- `slot`;
|
||||
- `lane.slot`.
|
||||
- Reject empty paths for `--reference`; use `--without-reference` for unbinding.
|
||||
- Reject malformed values with concise CLI errors before expensive work.
|
||||
- Pass parsed override/unbind requests into config/pipeline resolution.
|
||||
- Resolve flat CLI names only when exactly one selected lane declares the slot.
|
||||
If multiple selected lanes declare the same slot, fail and instruct the user
|
||||
to use `lane.slot`.
|
||||
- Let CLI bindings override config bindings for the same lane and slot.
|
||||
- Let CLI unbinds remove config-bound optional slots for the same lane and slot.
|
||||
- Fail if unbinding leaves a required slot unbound.
|
||||
- Add CLI tests for flat binding, lane-qualified binding, ambiguous flat
|
||||
binding, malformed syntax, optional unbind, and required-slot unbind failure.
|
||||
|
||||
Verification:
|
||||
|
||||
- `go test ./internal/cli`
|
||||
- `go test ./internal/core/config`
|
||||
- `go test ./internal/framework/pipeline`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 4: Run Preparation and Reference Materialization
|
||||
|
||||
Read, validate, digest, and materialize resolved file references before any LLM
|
||||
call.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Add a reference resolver/materializer near pipeline run preparation. Keep file
|
||||
I/O out of pure config parsing.
|
||||
- Ensure run preparation receives the loaded config path or config directory so
|
||||
config-relative reference paths can be resolved after pure config parsing.
|
||||
- Resolve config-relative paths relative to the config file path and
|
||||
CLI-relative paths relative to the current working directory.
|
||||
- For MVP, accept only UTF-8 text files. Reject non-UTF-8 content with an error
|
||||
naming pipeline, lane, slot, and path.
|
||||
- Compute `sha256:` content digests over the raw reference bytes.
|
||||
- Populate `ReferenceItem` values with content bytes, media type, digest,
|
||||
origin type `file`, normalized origin URI/path, size bytes, and binding source.
|
||||
- Enforce declared `MaxBytes` when greater than zero. The error should name the
|
||||
pipeline, lane, slot, actual size, limit, and path.
|
||||
- Emit a warning for empty bound files, but do not fail.
|
||||
- Add `ReferenceSet` values to the runner input or resolved pipeline path in a
|
||||
way that keeps lane-scoped references available when calling each extractor.
|
||||
- Pass the correct lane-specific `ReferenceSet` into
|
||||
`contracts.ExtractionRequest`.
|
||||
- Ensure no reference content is written to ordinary diagnostics, logs, errors,
|
||||
or manifests.
|
||||
|
||||
Verification:
|
||||
|
||||
- Focused resolver/materializer tests for path resolution, digest stability,
|
||||
UTF-8 rejection, empty-file warning, `MaxBytes`, and binding source.
|
||||
- `go test ./internal/cli`
|
||||
- `go test ./internal/framework/pipeline`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 5: Prompt Template Reference Functions
|
||||
|
||||
Make references available to module-owned prompt templates.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Extend `internal/framework/prompt` so prompt bundles can be compiled with
|
||||
declared reference slots.
|
||||
- Add `reference` and `hasreference` template functions.
|
||||
- Validate at bundle build time, or the earliest feasible equivalent, that
|
||||
templates reference only declared slots.
|
||||
- Render a declared but unbound optional slot as an empty string.
|
||||
- Ensure `hasreference` returns true only when the slot has at least one bound
|
||||
item with content.
|
||||
- Render multiple items deterministically if future `Multiple` support is
|
||||
enabled; for MVP, reject multiple bindings unless the slot declares
|
||||
`Multiple`.
|
||||
- Keep prompt metadata hashes based on template source. Do not include rendered
|
||||
reference content in prompt identity.
|
||||
- Add deterministic rendering tests proving byte-identical output across runs
|
||||
with the same reference bytes and config.
|
||||
|
||||
Verification:
|
||||
|
||||
- `go test ./internal/framework/prompt`
|
||||
- `go test ./internal/modules/extract/dnd/spells`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 6: Manifest and Diagnostics Provenance
|
||||
|
||||
Record reference provenance separately from source provenance.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Add a dedicated references section to `artifacts.RunManifest`.
|
||||
The shape should be lane-scoped and include lane ID, slot name, origin type,
|
||||
origin URI/path, digest, media type, size bytes, and binding source.
|
||||
- Do not add reference digests to `source_digests`.
|
||||
- Include reference digests in any cache/idempotency key if such a key exists.
|
||||
If no cache/idempotency key exists, add a test or comment documenting that no
|
||||
additional key needs updating yet.
|
||||
- Write a diagnostics artifact for resolved references that contains provenance
|
||||
only, not full content, consistent with redacted effective config behavior.
|
||||
- Ensure durable JSON output manifests include the new manifest section.
|
||||
- Add manifest round-trip tests and a CLI/run test where two runs that differ
|
||||
only in reference bytes produce distinguishable manifests.
|
||||
|
||||
Verification:
|
||||
|
||||
- `go test ./internal/core/artifacts`
|
||||
- `go test ./internal/core/diagnostics`
|
||||
- `go test ./internal/modules/output/json`
|
||||
- `go test ./internal/cli`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 7: D&D Spells Consumer
|
||||
|
||||
Use the new reference feature in the first production extractor.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Declare optional `roster` and `glossary` slots on `dnd/spells`.
|
||||
- Set accepted media type to text/UTF-8. Add conservative `MaxBytes` limits only
|
||||
if a clear module-owned limit is chosen; otherwise leave `MaxBytes` unset.
|
||||
- Update the D&D spells prompt bundle to include conditional reference sections
|
||||
using `hasreference` and `reference`.
|
||||
- Frame references as supporting material only. The prompt must instruct the
|
||||
model to extract only spell-cast events present in the source transcript and
|
||||
use references only for disambiguation.
|
||||
- Update prompt metadata tests as needed while preserving template-hash
|
||||
semantics.
|
||||
- Add fixture coverage with no references, with roster/glossary references, and
|
||||
with a roster that mentions a spell never cast in the transcript. The last
|
||||
case must assert no spell-cast artifact is produced for the uncast spell.
|
||||
- If existing deterministic source-reference validation can be extended
|
||||
cleanly, add warning-level relatedness checks for spell names or close
|
||||
variants near cited source text. If this becomes large, defer that validator
|
||||
enhancement to a separate roadmap item and keep the prompt/regression fixture
|
||||
guard in this stage.
|
||||
|
||||
Verification:
|
||||
|
||||
- `go test ./internal/modules/extract/dnd/spells`
|
||||
- `go test ./internal/framework/pipeline`
|
||||
- `go test ./internal/cli`
|
||||
- `go test ./...`
|
||||
|
||||
## Stage 8: Canonical Documentation and Examples
|
||||
|
||||
Move implemented behavior out of roadmap-only status once code exists.
|
||||
|
||||
Implementation steps:
|
||||
|
||||
- Update `docs/cli.md` with `--reference` and `--without-reference` syntax,
|
||||
precedence, ambiguity behavior, and examples.
|
||||
- Update `docs/config.md` with pipeline-level and lane-level `references`
|
||||
blocks.
|
||||
- Update `docs/internal/modules.md` or the most appropriate internal docs with
|
||||
module-author guidance for `ReferenceSlots()`, reference request delivery,
|
||||
prompt functions, evidence exclusion, and provenance.
|
||||
- Update `docs/internal/pipeline.md` with reference resolution lifecycle and
|
||||
lane-scoped delivery.
|
||||
- Update `docs/integrations/json-output.md` with the manifest reference
|
||||
provenance shape.
|
||||
- Update `docs/operations.md` or `docs/troubleshooting.md` for common reference
|
||||
errors such as unknown slot, ambiguous flat override, missing required slot,
|
||||
unreadable file, non-UTF-8 content, and `MaxBytes` failures.
|
||||
- Add maintained example reference files and update `examples/dnd-spells.config.yml`
|
||||
only after the CLI/config behavior is implemented and covered by tests.
|
||||
- Keep future-only material in `docs/roadmap/references.md`; do not duplicate
|
||||
canonical current behavior there after implementation.
|
||||
|
||||
Verification:
|
||||
|
||||
- `rg -n "references:|--reference|--without-reference|ReferenceSlots|reference \"|hasreference" docs examples`
|
||||
- `go test ./...`
|
||||
- `go vet ./...`
|
||||
- `go build ./cmd/notarius`
|
||||
|
||||
## Final Acceptance Criteria
|
||||
|
||||
The feature is complete when:
|
||||
|
||||
- extractor modules can declare reference slots through the first-class
|
||||
extractor contract;
|
||||
- config and CLI can bind and unbind lane-scoped file references;
|
||||
- selected-pipeline validation catches unknown, ambiguous, or missing required
|
||||
references before any LLM call;
|
||||
- run preparation materializes UTF-8 text references with digests, size checks,
|
||||
and empty-file warnings;
|
||||
- extractors receive lane-scoped resolved references;
|
||||
- prompt templates can render `reference` and `hasreference` deterministically;
|
||||
- D&D spell extraction uses optional roster and glossary references;
|
||||
- run manifests and diagnostics record reference provenance without recording
|
||||
full content or treating references as source evidence;
|
||||
- canonical docs and maintained examples describe only implemented behavior;
|
||||
- `go test ./...`, `go vet ./...`, and `go build ./cmd/notarius` pass.
|
||||
@@ -1,238 +0,0 @@
|
||||
# Feature Roadmap: Extraction References
|
||||
|
||||
## Status
|
||||
|
||||
This document defines the target state and policy choices for the planned
|
||||
extraction-reference feature in Notarius. It describes planned behavior, not
|
||||
implemented behavior. The staged implementation plan lives in
|
||||
[implementation.md](implementation.md).
|
||||
|
||||
## Goal
|
||||
|
||||
Extraction quality improves when an LLM-backed extractor receives stable
|
||||
reference material alongside the source input. For the initial D&D spell
|
||||
extractor, useful reference material includes a party roster, a player list, and
|
||||
a campaign glossary.
|
||||
|
||||
Notarius should support passing this material to extractors as **named reference
|
||||
items** without introducing domain-specific concepts into core or framework
|
||||
packages. The framework should know only that:
|
||||
|
||||
- extractors declare named reference slots they accept;
|
||||
- pipeline config and CLI flags bind content, initially files, to those slots;
|
||||
- bound content is rendered into module-owned prompt templates;
|
||||
- bound content is digested and recorded as run provenance.
|
||||
|
||||
Only extract modules should know what a "roster" or "glossary" means. Domain
|
||||
semantics live in module-owned slot declarations and prompt templates.
|
||||
|
||||
## Definitions
|
||||
|
||||
- **Reference slot**: a named, typed-by-convention input declared by an
|
||||
extractor, with a human-readable description, required/optional status, and
|
||||
optional guardrails such as accepted media types and maximum bytes.
|
||||
- **Reference item**: resolved content bound to a slot for a given run: slot
|
||||
name, content bytes, media type, content digest, size, origin, and binding
|
||||
source.
|
||||
- **Reference binding**: the association of a slot name to a content source,
|
||||
defined in pipeline config and overridable per run via CLI.
|
||||
- **Reference set**: the lane-scoped collection of resolved reference items
|
||||
delivered to an extractor.
|
||||
|
||||
## Architectural Principles
|
||||
|
||||
- References are opaque to the framework. Core and framework packages must not
|
||||
interpret reference content or recognize domain slot names.
|
||||
- References are inputs. Anything that can change extraction output must be
|
||||
digested into the run manifest and participate in any cache or idempotency key.
|
||||
- References are not evidence. `SourceRef` values must only ever reference
|
||||
source units. Reference items must not receive unit IDs and must not be
|
||||
addressable by source references.
|
||||
- Slots are declared, not ad hoc. Binding an undeclared slot name, or omitting a
|
||||
required slot, should fail before any LLM call.
|
||||
- Reference delivery is lane-scoped. Pipeline-level bindings may apply to
|
||||
multiple lanes, but each lane receives only the references declared by its
|
||||
extractor after pipeline, lane, CLI override, and CLI unbind rules are
|
||||
resolved.
|
||||
- Optional slots degrade gracefully. Prompt templates should render cleanly
|
||||
whether or not an optional slot is bound.
|
||||
- Rendering must be deterministic. Identical source input, config, prompts, and
|
||||
reference bytes should produce byte-identical rendered prompts. Reference
|
||||
slots should render in declaration order, with stable binding order within a
|
||||
slot.
|
||||
|
||||
## Target Contracts
|
||||
|
||||
Extractors should declare accepted reference slots directly on the extractor
|
||||
contract. This is a first-class feature, so mechanical updates to existing
|
||||
extractors and test fakes are acceptable.
|
||||
|
||||
The target slot declaration includes:
|
||||
|
||||
- `Name`;
|
||||
- `Description`;
|
||||
- `Required`;
|
||||
- `AcceptedMediaTypes`;
|
||||
- `Multiple`;
|
||||
- `MaxBytes`.
|
||||
|
||||
Extractors with no reference needs return an empty slot list.
|
||||
|
||||
Resolved reference items should be content-bearing values, not unresolved file
|
||||
paths. The MVP producer is "read this file," but the item shape should permit
|
||||
future producers such as prior-run artifacts, derived summaries, or entity
|
||||
registries without changing extractor-facing contracts.
|
||||
|
||||
The extraction request should carry the lane-scoped resolved reference set.
|
||||
Framework and core code should treat the set as opaque bytes plus metadata.
|
||||
|
||||
## Binding Lifecycle
|
||||
|
||||
Reference handling should be split across existing lifecycle boundaries:
|
||||
|
||||
1. Config parsing records pipeline-level and lane-level reference bindings
|
||||
without reading files.
|
||||
2. Pipeline resolution validates selected lanes, declared extractor slots,
|
||||
missing required slots, unknown bindings, and ambiguous flat CLI bindings.
|
||||
3. Run preparation resolves paths, reads files, validates media type and size,
|
||||
computes digests, and materializes reference items.
|
||||
4. Extraction receives the lane-specific resolved reference set.
|
||||
|
||||
Config-relative paths resolve relative to the config file. CLI-relative paths
|
||||
resolve relative to the current working directory.
|
||||
|
||||
## Configuration and CLI
|
||||
|
||||
Reference bindings should live in pipeline config because the initial use cases
|
||||
are campaign-invariant more often than run-variant. Bindings should be supported
|
||||
at two levels:
|
||||
|
||||
- pipeline level: defaults shared by artifact lanes whose extractors declare
|
||||
matching slots;
|
||||
- lane level: additions or overrides for a single artifact lane.
|
||||
|
||||
Illustrative config shape:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
dnd-session:
|
||||
input: seriatim
|
||||
references:
|
||||
roster: ./campaign/party_roster.md
|
||||
glossary: ./campaign/glossary.md
|
||||
artifacts:
|
||||
spells:
|
||||
extract: dnd/spells
|
||||
npcs:
|
||||
extract: dnd/npcs
|
||||
references:
|
||||
npc_registry: ./campaign/npcs.md
|
||||
```
|
||||
|
||||
Per-run CLI binding overrides should be repeatable:
|
||||
|
||||
```text
|
||||
notarius run dnd-session --input session-014.json --reference roster=./alt_roster.md
|
||||
```
|
||||
|
||||
Flat CLI slot names are allowed when unambiguous across selected lanes. Lane
|
||||
qualified names, such as `spells.roster=./alt_roster.md`, disambiguate or target
|
||||
a specific lane. CLI bindings override config bindings for the same lane and
|
||||
slot.
|
||||
|
||||
Users should also be able to unbind a config-bound optional slot for a run with
|
||||
an explicit repeatable flag:
|
||||
|
||||
```text
|
||||
notarius run dnd-session --input session-014.json --without-reference roster
|
||||
```
|
||||
|
||||
Unbinding a required slot should fail during pipeline/reference resolution.
|
||||
|
||||
## Prompt Template Integration
|
||||
|
||||
Prompt templates are module-owned. Template rendering should expose:
|
||||
|
||||
- `{{ reference "roster" }}`: renders the content of the bound item;
|
||||
- `{{ hasreference "glossary" }}`: predicate for conditional sections, so
|
||||
optional slots can be included only when bound.
|
||||
|
||||
Rules:
|
||||
|
||||
- Referencing an undeclared slot from a template is a module bug and should fail
|
||||
at prompt registration/build time or the earliest feasible equivalent.
|
||||
- Referencing a declared but unbound optional slot should render as empty;
|
||||
templates should use `hasreference` to avoid dangling section headers.
|
||||
- Rendering must be deterministic and independent of map iteration order.
|
||||
- Prompt identity should be computed over the template, not the rendered prompt.
|
||||
Reference digests are recorded separately in the manifest so a reference edit
|
||||
is visible as a reference change, not a prompt change.
|
||||
|
||||
Reference content is repeated in every per-chunk prompt in the MVP. Per-slot or
|
||||
per-chunk inclusion policies are deferred until cost data justifies them.
|
||||
|
||||
## Provenance
|
||||
|
||||
The run manifest must record resolved references separately from source
|
||||
digests. For every bound lane and slot, it should record:
|
||||
|
||||
- lane ID;
|
||||
- slot name;
|
||||
- origin type and URI;
|
||||
- content digest;
|
||||
- media type;
|
||||
- size in bytes;
|
||||
- whether the binding came from config or CLI override.
|
||||
|
||||
Reference digests must participate in any idempotency/cache key alongside source
|
||||
digests, prompt hashes, schema versions, model, and parameters. Two runs that
|
||||
differ only in reference content must be distinguishable from the manifest
|
||||
alone.
|
||||
|
||||
Diagnostics for a run should include the resolved binding set with digests, not
|
||||
full reference content, consistent with the existing redacted-effective-config
|
||||
pattern.
|
||||
|
||||
## Validation and Guardrails
|
||||
|
||||
### References Are Not Evidence
|
||||
|
||||
The primary new failure mode is the model extracting facts from references
|
||||
rather than from the source input. For example, a roster may list a player
|
||||
character's known spells, and the model might emit a spell-cast artifact for a
|
||||
spell that was never cast in the session.
|
||||
|
||||
Defenses, in priority order:
|
||||
|
||||
1. **Structural.** `SourceRef` remains the only grounding mechanism and can only
|
||||
reference source units. No contract change should make references
|
||||
addressable as evidence.
|
||||
2. **Prompt discipline.** Module templates should frame references explicitly as
|
||||
reference material, such as "use the roster to resolve speakers to
|
||||
characters; extract only events that occur in the transcript."
|
||||
3. **Validator support.** The source-reference validator, or a sibling
|
||||
deterministic validator, should warn when referenced source text does not
|
||||
plausibly relate to the extracted fact. Severity should be `warn`, not
|
||||
`fail`, because transcripts can use paraphrase, nicknames, and abbreviations.
|
||||
4. **Regression fixtures.** Tests should include a fixture in which a bound
|
||||
roster mentions a spell that is never cast in the transcript, asserting no
|
||||
artifact record is produced for it.
|
||||
|
||||
### Size and Sanity Guardrails
|
||||
|
||||
- MVP accepts UTF-8 text content only. Other media types should be rejected with
|
||||
a clear error.
|
||||
- A slot-level `MaxBytes` value should be enforced when declared.
|
||||
- Empty bound files should produce a warning because they are likely user error.
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Token budgeting and model context-window management for references.
|
||||
- Non-file reference producers, including prior-run artifacts, derived
|
||||
summaries, and entity registries.
|
||||
- Per-chunk or per-slot inclusion policies.
|
||||
- Structured or parsed references such as typed roster schemas. References are
|
||||
opaque text handed to prompts.
|
||||
- Reference caching, preprocessing, summarization, embedding, or retrieval.
|
||||
- Making references addressable as evidence in any form.
|
||||
|
||||
Reference in New Issue
Block a user