Compare commits

..

4 Commits

18 changed files with 290 additions and 718 deletions

View File

@@ -319,7 +319,8 @@ Unknown outer or nested option fields are rejected, as are incompatible YAML
types. The allowlist remains valid when a run uses lane filtering: a configured
lane that is not active for that invocation simply contributes no evidence.
Evidence publication is opt-in because it can persist source text and metadata.
Its payload contract is [Published Evidence Context](integrations/evidence-context.md).
When enabled, it publishes the selected source-unit excerpt defined by the
[Published Evidence Context contract](integrations/evidence-context.md).
## References And Ordered Handoffs

View File

@@ -55,10 +55,10 @@ contract. The JSON bundle contract links to the available lane contracts.
If `index.json` has an `evidence_context` descriptor, treat it as a
pipeline-wide artifact rather than a lane entry. Verify its six descriptor
fields before decoding the linked file according to the [Published Evidence
Context contract](../integrations/evidence-context.md). Use each
`evidence_refs` entry as the citation to source material. Its surrounding
context range and included units explain the citation, but do not widen or
replace the cited source reference.
Context contract](../integrations/evidence-context.md). Decode its top-level
source-unit array as a reading excerpt. Obtain authoritative citations and lane
provenance from the normalized lane artifacts; the excerpt has neither and its
nearby units do not widen a lane artifact's cited source reference.
A zero exit status may still report rejected outputs, warnings, or absent
lanes. The caller decides which lane IDs are required for its own work and
@@ -73,5 +73,5 @@ them. Treat the input, output bundle, cache, debug bundle, and captured process
logs as potentially sensitive data. Apply the caller's access controls and
retention policy, and avoid copying secrets into arguments, logs, or
provenance records. An evidence-context artifact contains source-unit text and
metadata, and selected lanes can cover most of an input; preserve and share it
only when that source content is authorized for the recipient.
metadata and can cover most of an input; preserve and share it only when that
source content is authorized for the recipient.

View File

@@ -1,9 +1,11 @@
# Published Evidence Context
This contract defines the optional `source/evidence-context` artifact emitted
by the production JSON output. Its configuration is owned by
[Configuration](../config.md#module-bindings-and-validators); its logical-file
discovery is owned by [Published JSON Output](json-output.md).
by the production JSON output. It is a selected source-unit excerpt for
convenient reading alongside normalized lane artifacts; it is not a second
citation or provenance model. Its configuration is owned by
[Configuration](../config.md#module-bindings-and-validators), and its
logical-file discovery is owned by [Published JSON Output](json-output.md).
## Identity And Discovery
@@ -26,91 +28,80 @@ its absence means evidence publication was not enabled for that bundle.
## Payload
The v1 payload is a JSON object with required `source_id`, `source_digest`,
`window_units`, `selected_lanes`, and `contexts` fields. `selected_lanes` and
`contexts` are always arrays; an enabled configuration with no accepted direct
evidence publishes `contexts: []`.
The v1 payload is a top-level JSON array of generic source units. There is no
wrapper, source-level metadata, context grouping, lane identifier, or evidence
reference in the payload. An enabled configuration with no contributing
accepted evidence publishes `[]`.
```json
{
"source_id": "session-alpha",
"source_digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"window_units": 1,
"selected_lanes": ["npc_registry", "spells"],
"contexts": [
{
"context_ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 20
},
"evidence_refs": [
{
"lane_id": "spells",
"source_ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 10
}
}
],
"units": [
{
"id": 10,
"kind": "transcript_segment",
"text": "Aria casts Cure Wounds.",
"ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 10
}
},
{
"id": 20,
"kind": "transcript_segment",
"text": "The party regroups.",
"ref": {
"source_id": "session-alpha",
"start_unit_id": 20,
"end_unit_id": 20
}
}
]
[
{
"id": 10,
"kind": "transcript_segment",
"text": "Aria casts Cure Wounds.",
"ref": {
"source_id": "session-alpha",
"start_unit_id": 10,
"end_unit_id": 10
}
]
}
},
{
"id": 20,
"kind": "transcript_segment",
"text": "The party regroups.",
"ref": {
"source_id": "session-alpha",
"start_unit_id": 20,
"end_unit_id": 20
}
}
]
```
Each context requires a `context_ref` object and `evidence_refs` and `units`
arrays. `context_ref` identifies the first and last included unit. Each
evidence entry contains a selected `lane_id` and an original `source_ref`. A
unit uses the existing source-unit shape: required `id`, `kind`, `text`, and
self `ref`, plus optional JSON-object `metadata`. Fixed payload objects reject
unknown fields; unit metadata may contain application-defined JSON values.
Each source unit has required `id`, `kind`, `text`, and self `ref` fields.
`ref` contains `source_id`, `start_unit_id`, and `end_unit_id`, and both unit
endpoints identify that unit's `id`. A unit may also contain source-owned
`metadata`, an open-ended JSON object. Fixed unit and reference fields are
strict: consumers must reject unknown fixed fields, malformed units, invalid
self-references, units whose `source_id` differs from other units in the same
excerpt, and a payload that is not the array described here.
## Citations And Context
The excerpt preserves each selected unit exactly as represented by the
validated generic source document. It does not add evidence-context-specific
annotations or reshape source-owned metadata.
`evidence_refs` are the authoritative citations. They identify the direct
references emitted by accepted normalized artifacts. `context_ref` and the
units collection include those cited units plus nearby source units selected by
the configured window. They are explanatory context, not widened citations.
## Selection And Citations
Only accepted outputs from the configured lane allowlist contribute. Rejected,
failed, absent, and lane-filtered outputs do not contribute. The artifact never
contains raw input bytes, prompts, model responses, auxiliary reference
content, credentials, or filesystem paths.
The framework obtains direct source references only through typed evidence
projections of accepted normalized artifacts in the configured lane allowlist.
It validates each reference against the current source document, expands its
range by `window_units` source-unit positions on each side, clamps at document
boundaries, and takes the union of all expanded ranges. The output contains
each selected source unit once in source-document position order, regardless
of numeric unit IDs. Repeated references, overlapping windows, and citations
from multiple lanes do not duplicate a unit. Rejected, failed, absent,
inactive, and unselected lanes contribute nothing.
## Ordering And Compatibility
Normalized lane artifacts remain authoritative for citations and for which lane
cited a range. The excerpt has no lane attribution and must not be used to
reconstruct it. Its included nearby units provide reading context only; they
do not widen any citation in a lane artifact.
The selected lane allowlist is lexical. Contexts and units are in source
document position order, not numeric unit-ID order. Direct evidence entries
are deterministically ordered by lane and source reference. Overlapping or
contiguous windows merge, and each source unit appears at most once in the
resulting contexts.
The excerpt contains at most every generic source unit once. It can therefore
equal the complete generic source document when coverage is broad or the
window is large. No byte-, token-, or compression-size guarantee is made, and
the framework does not truncate the excerpt to meet an arbitrary size limit.
## Consumer Responsibilities And Data Handling
The artifact is additive to the JSON bundle and is not a lane payload,
normalized-output count, checkpoint, or generated reference. Consumers that
do not need it must tolerate the absent optional descriptor. Consumers that do
use it should preserve the artifact and its schema identity with the run
provenance, and should treat its source text and metadata as sensitive durable
content.
do not need it must tolerate an absent descriptor. Consumers that do use it
should validate the descriptor and payload before use, retain the artifact with
its schema identity when needed for a run record, and read citations from the
corresponding normalized lane artifacts.
The excerpt contains source-unit text and source-owned metadata and is durable
output. Treat it as sensitive source content, apply appropriate access controls
and retention, and do not assume its selected form is materially smaller or
less sensitive than the original input.

View File

@@ -24,7 +24,7 @@ root for the logical discovery described here.
| `warnings.json` | Accepted-output and run warnings. |
| `lanes/<safe-lane-id>.json` | One normalized artifact payload for each lane. |
| `chunk-map.json` | Optional accepted chunk map, when its export is enabled and available. |
| `evidence-context.json` | Optional source-context artifact, when evidence publication is enabled. |
| `evidence-context.json` | Optional selected source-unit excerpt, when evidence publication is enabled. |
JSON files are pretty-printed with a trailing newline. Lane payloads are
accepted only when their media type is `application/json`.

View File

@@ -42,7 +42,8 @@ generic source references and must use the codec's exact Go type. It does not
interpret surrounding context or publish files; the pipeline validates the
capability during preparation and the output boundary owns publication. See
the [Published Evidence Context contract](../integrations/evidence-context.md)
for the durable result.
for the durable source-unit excerpt. Lane artifacts retain citation and lane
provenance; the framework does not add either to that published excerpt.
An artifact family is broader than a module: it owns the cohesive domain
feature across its artifact type, codec, stage modules, validators, prompt

View File

@@ -110,9 +110,9 @@ are defined in [Accepted Chunk Map](integrations/chunk-map.md). An optional
[evidence context](integrations/evidence-context.md) contains source-unit text
and metadata. It is not a cache or debug artifact: retain it with the output
bundle only for as long as consumers need it, and apply source-content access
controls to the entire bundle. Selected lanes may collectively cite most of a
transcript, so a broad allowlist can make the evidence artifact nearly as
sensitive and large as the source itself.
controls to the entire bundle. Its selected source-unit excerpt may include
every source unit once when coverage is broad or its configured window is
large, so do not assume a byte or token reduction or reduced sensitivity.
## Chunk-Plan Cache

View File

@@ -189,10 +189,18 @@ func TestMaintainedCompleteExamplePublishesRegistryBackedEntityOccurrences(t *te
}
evidence := readProductionJSON[evidencecontext.Document](t, filepath.Join(runRoot, "evidence-context.json"))
for _, laneID := range []string{"enemy-events", "npc-registry", "npc-occurrences", "item-registry", "item-occurrences", "location-registry", "location-occurrences"} {
if !containsString(evidence.SelectedLanes, laneID) || !evidenceHasLane(evidence, laneID) {
t.Fatalf("evidence context = %#v, want direct %s evidence", evidence, laneID)
if len(evidence) == 0 {
t.Fatalf("evidence context = %#v, want selected source-unit evidence", evidence)
}
seenEvidenceUnits := make(map[int]struct{}, len(evidence))
for _, unit := range evidence {
if unit.Ref.SourceID != "session-ravenfall" || unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
t.Fatalf("evidence unit = %#v, want unchanged source-unit self-reference", unit)
}
if _, exists := seenEvidenceUnits[unit.ID]; exists {
t.Fatalf("evidence context = %#v, want each source unit once", evidence)
}
seenEvidenceUnits[unit.ID] = struct{}{}
}
requests := client.requestsFor(enemyevents.PromptID)
@@ -415,17 +423,6 @@ func containsString(values []string, want string) bool {
return false
}
func evidenceHasLane(value evidencecontext.Document, laneID string) bool {
for _, context := range value.Contexts {
for _, reference := range context.EvidenceRefs {
if reference.LaneID == laneID {
return true
}
}
}
return false
}
func generatedReferenceBinding(bindings []pipeline.ReferenceBinding, slotName string) (pipeline.ReferenceBinding, bool) {
for _, binding := range bindings {
if binding.SlotName == slotName && binding.Artifact != nil {

View File

@@ -2,24 +2,8 @@
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "notarius.source.evidence_context",
"title": "notarius_source_evidence_context_v1",
"type": "object",
"additionalProperties": false,
"required": ["source_id", "source_digest", "window_units", "selected_lanes", "contexts"],
"properties": {
"source_id": {"type": "string", "minLength": 1},
"source_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
"window_units": {"type": "integer", "minimum": 0},
"selected_lanes": {
"type": "array",
"minItems": 1,
"uniqueItems": true,
"items": {"type": "string", "minLength": 1}
},
"contexts": {
"type": "array",
"items": {"$ref": "#/$defs/context"}
}
},
"type": "array",
"items": {"$ref": "#/$defs/unit"},
"$defs": {
"source_ref": {
"type": "object",
@@ -42,25 +26,6 @@
"ref": {"$ref": "#/$defs/source_ref"},
"metadata": {"type": "object", "additionalProperties": true}
}
},
"evidence_ref": {
"type": "object",
"additionalProperties": false,
"required": ["lane_id", "source_ref"],
"properties": {
"lane_id": {"type": "string", "minLength": 1},
"source_ref": {"$ref": "#/$defs/source_ref"}
}
},
"context": {
"type": "object",
"additionalProperties": false,
"required": ["context_ref", "evidence_refs", "units"],
"properties": {
"context_ref": {"$ref": "#/$defs/source_ref"},
"evidence_refs": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/evidence_ref"}},
"units": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/unit"}}
}
}
}
}

View File

@@ -3,118 +3,62 @@ package evidencecontext
import (
"fmt"
"sort"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
type contribution struct {
laneID string
ref source.SourceRef
type expandedRange struct {
startPos int
endPos int
}
type expandedRange struct {
startPos int
endPos int
contributions []contribution
}
// Build validates accepted direct references, expands them by source-document
// position, and returns their deterministic context union.
// Build validates projected source references, expands them by source-document
// position, and returns their ordered union as an owned source-unit excerpt.
func Build(request BuildRequest) (Document, error) {
if request.WindowUnits < 0 {
return Document{}, fmt.Errorf("window_units must not be negative")
}
lanes, err := normalizeSelectedLanes(request.SelectedLanes)
if err != nil {
return Document{}, err
return nil, fmt.Errorf("window_units must not be negative")
}
if err := source.ValidateDocument(request.Source); err != nil {
return Document{}, fmt.Errorf("validate source document: %w", err)
return nil, fmt.Errorf("validate source document: %w", err)
}
digest, err := source.DigestDocument(request.Source)
if err != nil {
return Document{}, fmt.Errorf("digest source document: %w", err)
return nil, fmt.Errorf("digest source document: %w", err)
}
if digest != request.Source.Digest {
return Document{}, fmt.Errorf("source digest does not match source document digest")
return nil, fmt.Errorf("source digest does not match source document digest")
}
selected := make(map[string]struct{}, len(lanes))
for _, laneID := range lanes {
selected[laneID] = struct{}{}
}
index := source.NewDocumentIndex(request.Source)
seen := make(map[evidenceKey]struct{})
contributions := make([]contribution, 0)
for laneIndex, laneEvidence := range request.LaneEvidence {
laneID := strings.TrimSpace(laneEvidence.LaneID)
if _, ok := selected[laneID]; !ok {
return Document{}, fmt.Errorf("lane evidence[%d] lane %q is not selected", laneIndex, laneID)
ranges := make([]expandedRange, 0, len(request.SourceRefs))
for refIndex, ref := range request.SourceRefs {
if err := index.ValidateRef(ref); err != nil {
return nil, fmt.Errorf("source reference[%d]: %w", refIndex, err)
}
for refIndex, ref := range laneEvidence.SourceRefs {
if err := index.ValidateRef(ref); err != nil {
return Document{}, fmt.Errorf("lane %q source reference[%d]: %w", laneID, refIndex, err)
startPos, _ := index.Position(ref.StartUnitID)
endPos, _ := index.Position(ref.EndUnitID)
ranges = append(ranges, expandedRange{
startPos: expandStart(startPos, request.WindowUnits),
endPos: expandEnd(endPos, len(request.Source.Units), request.WindowUnits),
})
}
merged := mergeRanges(ranges)
unitCount := 0
for _, value := range merged {
unitCount += value.endPos - value.startPos + 1
}
document := make(Document, 0, unitCount)
for _, value := range merged {
for position := value.startPos; position <= value.endPos; position++ {
unit, err := cloneSourceUnit(request.Source.Units[position])
if err != nil {
return nil, fmt.Errorf("clone source unit at position %d: %w", position, err)
}
key := evidenceKey{laneID: laneID, ref: ref}
if _, exists := seen[key]; exists {
continue
}
seen[key] = struct{}{}
startPos, _ := index.Position(ref.StartUnitID)
endPos, _ := index.Position(ref.EndUnitID)
contributions = append(contributions, contribution{laneID: laneID, ref: ref, startPos: expandStart(startPos, request.WindowUnits), endPos: expandEnd(endPos, len(request.Source.Units), request.WindowUnits)})
document = append(document, unit)
}
}
sort.Slice(contributions, func(i, j int) bool { return lessContribution(contributions[i], contributions[j]) })
document := Document{
SourceID: request.Source.ID,
SourceDigest: digest,
WindowUnits: request.WindowUnits,
SelectedLanes: lanes,
Contexts: make([]Context, 0),
}
for _, rangeValue := range mergeRanges(contributions) {
context, err := buildContext(request.Source, rangeValue)
if err != nil {
return Document{}, err
}
document.Contexts = append(document.Contexts, context)
}
canonical, err := canonicalizeOwned(document)
if err != nil {
return Document{}, fmt.Errorf("validate evidence context: %w", err)
}
return canonical, nil
}
type evidenceKey struct {
laneID string
ref source.SourceRef
}
func normalizeSelectedLanes(values []string) ([]string, error) {
if len(values) == 0 {
return nil, fmt.Errorf("selected_lanes must not be empty")
}
seen := make(map[string]struct{}, len(values))
lanes := make([]string, 0, len(values))
for index, raw := range values {
laneID := strings.TrimSpace(raw)
if laneID == "" {
return nil, fmt.Errorf("selected_lanes[%d] must not be empty", index)
}
if _, exists := seen[laneID]; exists {
return nil, fmt.Errorf("selected_lanes lane %q is duplicated", laneID)
}
seen[laneID] = struct{}{}
lanes = append(lanes, laneID)
}
sort.Strings(lanes)
return lanes, nil
return document, nil
}
func expandStart(position, window int) int {
@@ -132,65 +76,25 @@ func expandEnd(position, length, window int) int {
return position + window
}
func lessContribution(left, right contribution) bool {
if left.startPos != right.startPos {
return left.startPos < right.startPos
}
if left.endPos != right.endPos {
return left.endPos < right.endPos
}
return lessEvidenceRef(EvidenceRef{LaneID: left.laneID, SourceRef: left.ref}, EvidenceRef{LaneID: right.laneID, SourceRef: right.ref})
}
func mergeRanges(values []contribution) []expandedRange {
func mergeRanges(values []expandedRange) []expandedRange {
if len(values) == 0 {
return nil
}
ranges := make([]expandedRange, 0, len(values))
sort.Slice(values, func(i, j int) bool {
if values[i].startPos != values[j].startPos {
return values[i].startPos < values[j].startPos
}
return values[i].endPos < values[j].endPos
})
merged := make([]expandedRange, 0, len(values))
for _, value := range values {
if len(ranges) == 0 || value.startPos > ranges[len(ranges)-1].endPos+1 {
ranges = append(ranges, expandedRange{startPos: value.startPos, endPos: value.endPos, contributions: []contribution{value}})
if len(merged) == 0 || value.startPos > merged[len(merged)-1].endPos+1 {
merged = append(merged, value)
continue
}
current := &ranges[len(ranges)-1]
if value.endPos > current.endPos {
current.endPos = value.endPos
if value.endPos > merged[len(merged)-1].endPos {
merged[len(merged)-1].endPos = value.endPos
}
current.contributions = append(current.contributions, value)
}
return ranges
}
func buildContext(document *source.SourceDocument, value expandedRange) (Context, error) {
evidenceRefs := make([]EvidenceRef, 0, len(value.contributions))
for _, contribution := range value.contributions {
evidenceRefs = append(evidenceRefs, EvidenceRef{LaneID: contribution.laneID, SourceRef: contribution.ref})
}
sort.Slice(evidenceRefs, func(i, j int) bool { return lessEvidenceRef(evidenceRefs[i], evidenceRefs[j]) })
units := make([]source.SourceUnit, 0, value.endPos-value.startPos+1)
for position := value.startPos; position <= value.endPos; position++ {
unit, err := cloneSourceUnit(document.Units[position])
if err != nil {
return Context{}, fmt.Errorf("clone source unit at position %d: %w", position, err)
}
units = append(units, unit)
}
return Context{
ContextRef: source.SourceRef{SourceID: document.ID, StartUnitID: units[0].ID, EndUnitID: units[len(units)-1].ID},
EvidenceRefs: evidenceRefs,
Units: units,
}, nil
}
func lessEvidenceRef(left, right EvidenceRef) bool {
if left.LaneID != right.LaneID {
return left.LaneID < right.LaneID
}
if left.SourceRef.SourceID != right.SourceRef.SourceID {
return left.SourceRef.SourceID < right.SourceRef.SourceID
}
if left.SourceRef.StartUnitID != right.SourceRef.StartUnitID {
return left.SourceRef.StartUnitID < right.SourceRef.StartUnitID
}
return left.SourceRef.EndUnitID < right.SourceRef.EndUnitID
return merged
}

View File

@@ -6,7 +6,6 @@ import (
"encoding/json"
"fmt"
"io"
"regexp"
"strings"
"sync"
@@ -18,8 +17,6 @@ import (
//go:embed assets/schemas/source_evidence_context.v1.json
var schemaAssets embed.FS
var digestPattern = regexp.MustCompile(`^sha256:[0-9a-f]{64}$`)
var (
loadSchemaOnce sync.Once
loadedSchema []byte
@@ -78,24 +75,24 @@ func (c *Codec) Encode(value Document) ([]byte, error) {
func (c *Codec) Decode(content []byte) (Document, error) {
if _, err := c.schemaBytes(); err != nil {
return Document{}, err
return nil, err
}
if err := validateSchemaInstance(content); err != nil {
return Document{}, fmt.Errorf("decode evidence context: %w", err)
return nil, fmt.Errorf("decode evidence context: %w", err)
}
decoder := json.NewDecoder(bytes.NewReader(content))
decoder.DisallowUnknownFields()
var value Document
if err := decoder.Decode(&value); err != nil {
return Document{}, fmt.Errorf("decode evidence context: %w", err)
return nil, fmt.Errorf("decode evidence context: %w", err)
}
var trailing any
if err := decoder.Decode(&trailing); err != io.EOF {
return Document{}, fmt.Errorf("decode evidence context: multiple JSON values")
return nil, fmt.Errorf("decode evidence context: multiple JSON values")
}
canonical, err := canonicalizeOwned(value)
if err != nil {
return Document{}, fmt.Errorf("decode evidence context: %w", err)
return nil, fmt.Errorf("decode evidence context: %w", err)
}
return canonical, nil
}
@@ -115,17 +112,16 @@ func loadAndCompileSchema() {
return
}
var identity struct {
ID string `json:"$id"`
Title string `json:"title"`
Type string `json:"type"`
Required []string `json:"required"`
ID string `json:"$id"`
Title string `json:"title"`
Type string `json:"type"`
}
if err := json.Unmarshal(raw, &identity); err != nil {
loadSchemaErr = fmt.Errorf("decode source evidence context schema: %w", err)
return
}
if identity.ID != SchemaID || identity.Title != SchemaName || identity.Type != "object" || !hasRequiredFields(identity.Required) {
loadSchemaErr = fmt.Errorf("source evidence context schema identity or required fields are invalid")
if identity.ID != SchemaID || identity.Title != SchemaName || identity.Type != "array" {
loadSchemaErr = fmt.Errorf("source evidence context schema identity is invalid")
return
}
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(raw))
@@ -158,164 +154,65 @@ func validateSchemaInstance(content []byte) error {
return nil
}
func hasRequiredFields(required []string) bool {
want := map[string]bool{"source_id": true, "source_digest": true, "window_units": true, "selected_lanes": true, "contexts": true}
for _, field := range required {
delete(want, field)
}
return len(want) == 0
}
func canonicalize(value Document) (Document, error) {
owned, err := clone(value)
if err != nil {
return Document{}, err
return nil, err
}
return canonicalizeOwned(owned)
}
func canonicalizeOwned(value Document) (Document, error) {
if err := requireIdentity("source_id", value.SourceID); err != nil {
return Document{}, err
if value == nil {
return nil, fmt.Errorf("document must be a JSON array")
}
if !digestPattern.MatchString(value.SourceDigest) {
return Document{}, fmt.Errorf("source_digest must be a sha256 digest")
}
if value.WindowUnits < 0 {
return Document{}, fmt.Errorf("window_units must not be negative")
}
if err := validateSelectedLanes(value.SelectedLanes); err != nil {
return Document{}, err
}
if value.Contexts == nil {
value.Contexts = make([]Context, 0)
}
selected := make(map[string]struct{}, len(value.SelectedLanes))
for _, laneID := range value.SelectedLanes {
selected[laneID] = struct{}{}
}
seenUnits := make(map[int]struct{})
for contextIndex := range value.Contexts {
context, err := canonicalizeContext(value.SourceID, selected, seenUnits, value.Contexts[contextIndex], contextIndex)
if err != nil {
return Document{}, err
}
value.Contexts[contextIndex] = context
}
return value, nil
}
func validateSelectedLanes(lanes []string) error {
if len(lanes) == 0 {
return fmt.Errorf("selected_lanes must not be empty")
}
for index, laneID := range lanes {
if err := requireIdentity(fmt.Sprintf("selected_lanes[%d]", index), laneID); err != nil {
return err
}
if index > 0 && lanes[index-1] >= laneID {
return fmt.Errorf("selected_lanes must be unique and in lexical order")
}
}
return nil
}
func canonicalizeContext(sourceID string, selected map[string]struct{}, seenUnits map[int]struct{}, value Context, contextIndex int) (Context, error) {
prefix := fmt.Sprintf("contexts[%d]", contextIndex)
if len(value.EvidenceRefs) == 0 {
return Context{}, fmt.Errorf("%s.evidence_refs must not be empty", prefix)
}
if len(value.Units) == 0 {
return Context{}, fmt.Errorf("%s.units must not be empty", prefix)
}
if err := validateRefIdentity(sourceID, value.ContextRef, prefix+".context_ref"); err != nil {
return Context{}, err
}
positions := make(map[int]int, len(value.Units))
for unitIndex := range value.Units {
unit := value.Units[unitIndex]
seenUnitIDs := make(map[int]struct{}, len(value))
sourceID := ""
for unitIndex := range value {
unit := value[unitIndex]
if unit.ID <= 0 || strings.TrimSpace(unit.Kind) == "" || strings.TrimSpace(unit.Text) == "" {
return Context{}, fmt.Errorf("%s.units[%d] has invalid required fields", prefix, unitIndex)
return nil, fmt.Errorf("units[%d] has invalid required fields", unitIndex)
}
if err := validateRefIdentity(sourceID, unit.Ref, fmt.Sprintf("%s.units[%d].ref", prefix, unitIndex)); err != nil {
return Context{}, err
if err := validateUnitRef(unit, unitIndex); err != nil {
return nil, err
}
if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
return Context{}, fmt.Errorf("%s.units[%d].ref must identify unit id %d", prefix, unitIndex, unit.ID)
if sourceID == "" {
sourceID = unit.Ref.SourceID
} else if unit.Ref.SourceID != sourceID {
return nil, fmt.Errorf("units[%d].ref.source_id must match units[0].ref.source_id", unitIndex)
}
if _, exists := positions[unit.ID]; exists {
return Context{}, fmt.Errorf("%s.units contains duplicate unit id %d", prefix, unit.ID)
}
if _, exists := seenUnits[unit.ID]; exists {
return Context{}, fmt.Errorf("contexts contain duplicate unit id %d", unit.ID)
}
positions[unit.ID] = unitIndex
seenUnits[unit.ID] = struct{}{}
}
if value.ContextRef.StartUnitID != value.Units[0].ID || value.ContextRef.EndUnitID != value.Units[len(value.Units)-1].ID {
return Context{}, fmt.Errorf("%s.context_ref must identify the first and last units", prefix)
}
for evidenceIndex := range value.EvidenceRefs {
evidence := value.EvidenceRefs[evidenceIndex]
if _, ok := selected[evidence.LaneID]; !ok {
return Context{}, fmt.Errorf("%s.evidence_refs[%d].lane_id is not selected", prefix, evidenceIndex)
}
if err := requireIdentity(fmt.Sprintf("%s.evidence_refs[%d].lane_id", prefix, evidenceIndex), evidence.LaneID); err != nil {
return Context{}, err
}
if err := validateRefIdentity(sourceID, evidence.SourceRef, fmt.Sprintf("%s.evidence_refs[%d].source_ref", prefix, evidenceIndex)); err != nil {
return Context{}, err
}
start, startOK := positions[evidence.SourceRef.StartUnitID]
end, endOK := positions[evidence.SourceRef.EndUnitID]
if !startOK || !endOK || start > end {
return Context{}, fmt.Errorf("%s.evidence_refs[%d].source_ref is outside context units", prefix, evidenceIndex)
}
if evidenceIndex > 0 && !lessEvidenceRef(value.EvidenceRefs[evidenceIndex-1], evidence) {
return Context{}, fmt.Errorf("%s.evidence_refs must be unique and in deterministic order", prefix)
if _, exists := seenUnitIDs[unit.ID]; exists {
return nil, fmt.Errorf("units contains duplicate unit id %d", unit.ID)
}
seenUnitIDs[unit.ID] = struct{}{}
}
return value, nil
}
func validateRefIdentity(sourceID string, ref source.SourceRef, field string) error {
if ref.SourceID != sourceID {
return fmt.Errorf("%s.source_id does not match source_id", field)
func validateUnitRef(unit source.SourceUnit, unitIndex int) error {
prefix := fmt.Sprintf("units[%d].ref", unitIndex)
if strings.TrimSpace(unit.Ref.SourceID) == "" || strings.TrimSpace(unit.Ref.SourceID) != unit.Ref.SourceID {
return fmt.Errorf("%s.source_id must be a non-empty trimmed string", prefix)
}
if ref.StartUnitID <= 0 || ref.EndUnitID <= 0 {
return fmt.Errorf("%s endpoints must be positive", field)
}
return nil
}
func requireIdentity(field, value string) error {
if strings.TrimSpace(value) == "" || strings.TrimSpace(value) != value {
return fmt.Errorf("%s must be a non-empty trimmed string", field)
if unit.Ref.StartUnitID != unit.ID || unit.Ref.EndUnitID != unit.ID {
return fmt.Errorf("%s must identify unit id %d", prefix, unit.ID)
}
return nil
}
func clone(value Document) (Document, error) {
value.SelectedLanes = append([]string(nil), value.SelectedLanes...)
if value.Contexts == nil {
value.Contexts = make([]Context, 0)
} else {
contexts := make([]Context, len(value.Contexts))
for contextIndex, context := range value.Contexts {
contexts[contextIndex].ContextRef = context.ContextRef
contexts[contextIndex].EvidenceRefs = append([]EvidenceRef(nil), context.EvidenceRefs...)
contexts[contextIndex].Units = make([]source.SourceUnit, len(context.Units))
for unitIndex, unit := range context.Units {
cloned, err := cloneSourceUnit(unit)
if err != nil {
return Document{}, fmt.Errorf("clone contexts[%d].units[%d]: %w", contextIndex, unitIndex, err)
}
contexts[contextIndex].Units[unitIndex] = cloned
}
}
value.Contexts = contexts
if value == nil {
return nil, nil
}
return value, nil
cloned := make(Document, len(value))
for unitIndex, unit := range value {
owned, err := cloneSourceUnit(unit)
if err != nil {
return nil, fmt.Errorf("clone units[%d]: %w", unitIndex, err)
}
cloned[unitIndex] = owned
}
return cloned, nil
}
func cloneSourceUnit(unit source.SourceUnit) (source.SourceUnit, error) {

View File

@@ -2,7 +2,6 @@ package evidencecontext
import (
"bytes"
"encoding/json"
"math"
"os"
"reflect"
@@ -12,126 +11,57 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
)
func TestBuildExpandsAndMergesEvidenceByDocumentPosition(t *testing.T) {
func TestBuildSelectsExpandedSourceUnitUnion(t *testing.T) {
for _, test := range []struct {
name string
window int
evidence []LaneEvidence
wantUnits [][]int
wantRefs [][]EvidenceRef
name string
window int
refs []source.SourceRef
wantIDs []int
}{
{
name: "zero window",
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}},
wantUnits: [][]int{{3}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(3, 3)}}},
},
{
name: "non monotonic ids use positions and clip boundaries",
window: 1,
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}},
wantUnits: [][]int{{10, 3, 30}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(3, 3)}}},
},
{
name: "separate gaps stay separate",
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(10, 10), ref(50, 50)}}},
wantUnits: [][]int{{10}, {50}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(10, 10)}}, {{LaneID: "npcs", SourceRef: ref(50, 50)}}},
},
{
name: "overlapping windows merge",
window: 1,
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3), ref(30, 30)}}},
wantUnits: [][]int{{10, 3, 30, 7}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(3, 3)}, {LaneID: "npcs", SourceRef: ref(30, 30)}}},
},
{
name: "contiguous windows merge",
window: 1,
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(10, 10), ref(7, 7)}}},
wantUnits: [][]int{{10, 3, 30, 7, 50}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(7, 7)}, {LaneID: "npcs", SourceRef: ref(10, 10)}}},
},
{
name: "duplicate contributions retain unique lane attribution",
evidence: []LaneEvidence{
{LaneID: "spells", SourceRefs: []source.SourceRef{ref(30, 30), ref(30, 30)}},
{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(30, 30)}},
},
wantUnits: [][]int{{30}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(30, 30)}, {LaneID: "spells", SourceRef: ref(30, 30)}}},
},
{
name: "empty contributions retain explicit empty contexts",
evidence: []LaneEvidence{{LaneID: "npcs"}},
wantUnits: [][]int{},
wantRefs: [][]EvidenceRef{},
},
{
name: "largest window clips without overflow",
window: math.MaxInt,
evidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(30, 30)}}},
wantUnits: [][]int{{10, 3, 30, 7, 50}},
wantRefs: [][]EvidenceRef{{{LaneID: "npcs", SourceRef: ref(30, 30)}}},
},
{name: "zero window", refs: []source.SourceRef{ref(3, 3)}, wantIDs: []int{3}},
{name: "multi unit citation includes complete range", refs: []source.SourceRef{ref(3, 7)}, wantIDs: []int{3, 30, 7}},
{name: "non monotonic IDs use document positions", window: 1, refs: []source.SourceRef{ref(3, 3)}, wantIDs: []int{10, 3, 30}},
{name: "boundary clamping", window: 1, refs: []source.SourceRef{ref(10, 10), ref(50, 50)}, wantIDs: []int{10, 3, 7, 50}},
{name: "overlapping and adjacent windows merge", window: 1, refs: []source.SourceRef{ref(3, 3), ref(30, 30), ref(30, 30)}, wantIDs: []int{10, 3, 30, 7}},
{name: "adjacent expanded ranges merge", window: 1, refs: []source.SourceRef{ref(10, 10), ref(7, 7)}, wantIDs: []int{10, 3, 30, 7, 50}},
{name: "largest window clips without overflow", window: math.MaxInt, refs: []source.SourceRef{ref(30, 30)}, wantIDs: []int{10, 3, 30, 7, 50}},
{name: "no references returns an initialized empty document", wantIDs: []int{}},
} {
t.Run(test.name, func(t *testing.T) {
document := testDocument(t)
got, err := Build(BuildRequest{Source: document, WindowUnits: test.window, SelectedLanes: []string{"spells", "npcs"}, LaneEvidence: test.evidence})
got, err := Build(BuildRequest{Source: testDocument(t), WindowUnits: test.window, SourceRefs: test.refs})
if err != nil {
t.Fatalf("Build() error = %v", err)
}
if want := []string{"npcs", "spells"}; !reflect.DeepEqual(got.SelectedLanes, want) {
t.Fatalf("SelectedLanes = %#v, want %#v", got.SelectedLanes, want)
if got == nil {
t.Fatal("Build() returned a nil document")
}
if got.WindowUnits != test.window || got.SourceID != document.ID || got.SourceDigest != document.Digest {
t.Fatalf("Build() identity = %#v, want source and window identity", got)
}
if actual := contextUnitIDs(got.Contexts); !reflect.DeepEqual(actual, test.wantUnits) {
t.Fatalf("context unit ids = %#v, want %#v", actual, test.wantUnits)
}
if actual := contextEvidenceRefs(got.Contexts); !reflect.DeepEqual(actual, test.wantRefs) {
t.Fatalf("context evidence refs = %#v, want %#v", actual, test.wantRefs)
if actual := unitIDs(got); !reflect.DeepEqual(actual, test.wantIDs) {
t.Fatalf("unit IDs = %#v, want %#v", actual, test.wantIDs)
}
})
}
}
func TestBuildIsStableAndOwnsSourceAndInputs(t *testing.T) {
func TestBuildCopiesSelectedUnitsAndMetadata(t *testing.T) {
document := testDocument(t)
refs := []source.SourceRef{ref(30, 30), ref(3, 3)}
request := BuildRequest{
Source: document,
WindowUnits: 1,
SelectedLanes: []string{"spells", "npcs"},
LaneEvidence: []LaneEvidence{{LaneID: "spells", SourceRefs: refs}, {LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}},
}
first, err := Build(request)
first, err := Build(BuildRequest{Source: document, WindowUnits: 1, SourceRefs: []source.SourceRef{ref(3, 3)}})
if err != nil {
t.Fatal(err)
}
secondRequest := request
secondRequest.LaneEvidence = []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}, {LaneID: "spells", SourceRefs: []source.SourceRef{ref(3, 3), ref(30, 30)}}}
second, err := Build(secondRequest)
second, err := Build(BuildRequest{Source: document, WindowUnits: 1, SourceRefs: []source.SourceRef{ref(3, 3)}})
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(first, second) {
t.Fatalf("Build() order differs:\nfirst: %#v\nsecond: %#v", first, second)
if !reflect.DeepEqual(first[0], document.Units[0]) {
t.Fatalf("first unit = %#v, want unchanged source unit %#v", first[0], document.Units[0])
}
first.SelectedLanes[0] = "changed"
first.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] = "changed"
first[0].Metadata["nested"].(map[string]any)["value"] = "changed"
if document.Units[0].Metadata["nested"].(map[string]any)["value"] != "original" {
t.Fatal("Build() returned metadata aliases to source document")
t.Fatal("Build() returned metadata aliases to the source document")
}
document.Units[0].Metadata["nested"].(map[string]any)["value"] = "later"
if second.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] != "original" {
t.Fatal("Build() retained metadata aliases to source document")
}
refs[0].StartUnitID = 999
if !containsEvidenceRef(second.Contexts[0].EvidenceRefs, ref(30, 30)) {
t.Fatal("Build() retained source-reference input aliases")
if second[0].Metadata["nested"].(map[string]any)["value"] != "original" {
t.Fatal("Build() retained metadata aliases to the source document")
}
}
@@ -142,18 +72,11 @@ func TestBuildRejectsInvalidInputs(t *testing.T) {
want string
}{
{name: "negative window", mutate: func(request *BuildRequest) { request.WindowUnits = -1 }, want: "window_units"},
{name: "blank selected lane", mutate: func(request *BuildRequest) { request.SelectedLanes = []string{" "} }, want: "selected_lanes"},
{name: "duplicate selected lane", mutate: func(request *BuildRequest) { request.SelectedLanes = []string{"npcs", " npcs "} }, want: "duplicated"},
{name: "unselected contribution", mutate: func(request *BuildRequest) {
request.LaneEvidence = []LaneEvidence{{LaneID: "other", SourceRefs: []source.SourceRef{ref(3, 3)}}}
}, want: "not selected"},
{name: "source digest mismatch", mutate: func(request *BuildRequest) { request.Source.Digest = "sha256:" + strings.Repeat("0", 64) }, want: "does not match"},
{name: "invalid reference", mutate: func(request *BuildRequest) {
request.LaneEvidence = []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(99, 99)}}}
}, want: "source reference[0]"},
{name: "invalid reference", mutate: func(request *BuildRequest) { request.SourceRefs = []source.SourceRef{ref(99, 99)} }, want: "source reference[0]"},
} {
t.Run(test.name, func(t *testing.T) {
request := BuildRequest{Source: testDocument(t), SelectedLanes: []string{"npcs"}, LaneEvidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}}}
request := BuildRequest{Source: testDocument(t), SourceRefs: []source.SourceRef{ref(3, 3)}}
test.mutate(&request)
if _, err := Build(request); err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Build() error = %v, want %q", err, test.want)
@@ -162,7 +85,7 @@ func TestBuildRejectsInvalidInputs(t *testing.T) {
}
}
func TestCodecRoundTripsCompactFixtureAndOwnsDecodedValues(t *testing.T) {
func TestCodecRoundTripsFixtureAndOwnsValues(t *testing.T) {
fixture, err := os.ReadFile("testdata/source_evidence_context.v1.json")
if err != nil {
t.Fatal(err)
@@ -179,15 +102,16 @@ func TestCodecRoundTripsCompactFixtureAndOwnsDecodedValues(t *testing.T) {
if !bytes.Equal(encoded, bytes.TrimSpace(fixture)) {
t.Fatalf("fixture does not use canonical encoding\nwant: %s\n got: %s", fixture, encoded)
}
value.Contexts[0].Units[0].Text = "changed"
value[0].Text = "changed"
decoded, err := codec.Decode(encoded)
if err != nil {
t.Fatal(err)
}
if decoded.Contexts[0].Units[0].Text != "The party meets Rowan." {
if decoded[0].Text != "The party meets Rowan." {
t.Fatal("Encode() retained mutable document storage")
}
built, err := Build(BuildRequest{Source: testDocument(t), WindowUnits: 1, SelectedLanes: []string{"npcs"}, LaneEvidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}}})
built, err := Build(BuildRequest{Source: testDocument(t), WindowUnits: 1, SourceRefs: []source.SourceRef{ref(3, 3)}})
if err != nil {
t.Fatal(err)
}
@@ -203,82 +127,52 @@ func TestCodecRoundTripsCompactFixtureAndOwnsDecodedValues(t *testing.T) {
if err != nil {
t.Fatal(err)
}
first.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] = "changed"
if second.Contexts[0].Units[0].Metadata["nested"].(map[string]any)["value"] != "original" {
first[0].Metadata["nested"].(map[string]any)["value"] = "changed"
if second[0].Metadata["nested"].(map[string]any)["value"] != "original" {
t.Fatal("Decode() returned metadata aliases")
}
}
func TestCodecRejectsInvalidDurableBoundaries(t *testing.T) {
value, err := Build(BuildRequest{Source: testDocument(t), SelectedLanes: []string{"npcs"}, LaneEvidence: []LaneEvidence{{LaneID: "npcs", SourceRefs: []source.SourceRef{ref(3, 3)}}}})
if err != nil {
t.Fatal(err)
}
func TestCodecRejectsInvalidDurablePayloads(t *testing.T) {
for _, test := range []struct {
name string
mutate func(*Document)
name string
content string
}{
{name: "unsorted lanes", mutate: func(value *Document) { value.SelectedLanes = []string{"z", "a"} }},
{name: "context range mismatch", mutate: func(value *Document) { value.Contexts[0].ContextRef.EndUnitID = 999 }},
{name: "mismatched evidence source", mutate: func(value *Document) { value.Contexts[0].EvidenceRefs[0].SourceRef.SourceID = "other" }},
{name: "invalid evidence range", mutate: func(value *Document) {
value.Contexts[0].EvidenceRefs[0].SourceRef.StartUnitID = 10
}},
{name: "duplicate context unit", mutate: func(value *Document) { value.Contexts = append(value.Contexts, value.Contexts[0]) }},
{name: "null", content: "null"},
{name: "wrapper object", content: `{"units":[]}`},
{name: "missing required unit field", content: `[{"id":1,"kind":"segment","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1}}]`},
{name: "unknown unit field", content: `[{"id":1,"kind":"segment","text":"text","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1},"unknown":true}]`},
{name: "unknown reference field", content: `[{"id":1,"kind":"segment","text":"text","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1,"unknown":true}}]`},
{name: "invalid self reference", content: `[{"id":1,"kind":"segment","text":"text","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":2}}]`},
{name: "mixed source documents", content: `[{"id":1,"kind":"segment","text":"one","ref":{"source_id":"session-one","start_unit_id":1,"end_unit_id":1}},{"id":2,"kind":"segment","text":"two","ref":{"source_id":"session-two","start_unit_id":2,"end_unit_id":2}}]`},
{name: "duplicate units", content: `[{"id":1,"kind":"segment","text":"one","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1}},{"id":1,"kind":"segment","text":"two","ref":{"source_id":"session","start_unit_id":1,"end_unit_id":1}}]`},
{name: "multiple JSON values", content: `[] []`},
} {
t.Run(test.name, func(t *testing.T) {
candidate, err := clone(value)
if err != nil {
t.Fatal(err)
}
test.mutate(&candidate)
if _, err := New().Encode(candidate); err == nil {
t.Fatal("Encode() error = nil, want durable model rejection")
}
})
}
content, err := New().Encode(value)
if err != nil {
t.Fatal(err)
}
for _, test := range []struct {
name string
mutate func(map[string]any)
}{
{name: "missing contexts", mutate: func(value map[string]any) { delete(value, "contexts") }},
{name: "null contexts", mutate: func(value map[string]any) { value["contexts"] = nil }},
{name: "unknown fixed field", mutate: func(value map[string]any) { value["unknown"] = true }},
{name: "missing units", mutate: func(value map[string]any) { delete(contextObject(value, 0), "units") }},
{name: "null evidence refs", mutate: func(value map[string]any) { contextObject(value, 0)["evidence_refs"] = nil }},
} {
t.Run(test.name, func(t *testing.T) {
raw := decodeJSON(t, content)
test.mutate(raw)
mutated, err := json.Marshal(raw)
if err != nil {
t.Fatal(err)
}
if _, err := New().Decode(mutated); err == nil {
if _, err := New().Decode([]byte(test.content)); err == nil {
t.Fatal("Decode() error = nil, want strict payload rejection")
}
})
}
if _, err := New().Decode(append(content, []byte(" {}")...)); err == nil {
t.Fatal("Decode() error = nil, want trailing JSON rejection")
if _, err := New().Encode(nil); err == nil {
t.Fatal("Encode(nil) error = nil, want array rejection")
}
}
func TestSerializeUsesFixedArtifactIdentity(t *testing.T) {
artifact, err := Serialize(BuildRequest{Source: testDocument(t), SelectedLanes: []string{"npcs"}})
func TestSerializeUsesFixedArtifactIdentityAndEmptyArray(t *testing.T) {
artifact, err := Serialize(BuildRequest{Source: testDocument(t)})
if err != nil {
t.Fatal(err)
}
if artifact.Kind != ArtifactKind || artifact.MediaType != MediaType || artifact.Schema.ID != SchemaID || artifact.Schema.Name != SchemaName || artifact.Schema.Version != SchemaVersion {
t.Fatalf("Serialize() = %#v, want fixed artifact identity", artifact)
}
if string(artifact.Content) != "[]" {
t.Fatalf("Serialize() content = %s, want []", artifact.Content)
}
decoded, err := New().Decode(artifact.Content)
if err != nil || len(decoded.Contexts) != 0 || decoded.Contexts == nil {
t.Fatalf("Decode(Serialize()) = %#v, %v; want explicit empty contexts", decoded, err)
if err != nil || decoded == nil || len(decoded) != 0 {
t.Fatalf("Decode(Serialize()) = %#v, %v; want explicit empty array", decoded, err)
}
}
@@ -306,45 +200,10 @@ func ref(start, end int) source.SourceRef {
return source.SourceRef{SourceID: "session", StartUnitID: start, EndUnitID: end}
}
func contextUnitIDs(contexts []Context) [][]int {
values := make([][]int, len(contexts))
for index, context := range contexts {
values[index] = make([]int, len(context.Units))
for unitIndex, unit := range context.Units {
values[index][unitIndex] = unit.ID
}
func unitIDs(units Document) []int {
values := make([]int, len(units))
for index, unit := range units {
values[index] = unit.ID
}
return values
}
func contextEvidenceRefs(contexts []Context) [][]EvidenceRef {
values := make([][]EvidenceRef, len(contexts))
for index, context := range contexts {
values[index] = append([]EvidenceRef(nil), context.EvidenceRefs...)
}
return values
}
func containsEvidenceRef(values []EvidenceRef, want source.SourceRef) bool {
for _, value := range values {
if value.SourceRef == want {
return true
}
}
return false
}
func decodeJSON(t *testing.T, content []byte) map[string]any {
t.Helper()
decoder := json.NewDecoder(bytes.NewReader(content))
decoder.UseNumber()
var value map[string]any
if err := decoder.Decode(&value); err != nil {
t.Fatal(err)
}
return value
}
func contextObject(value map[string]any, index int) map[string]any {
return value["contexts"].([]any)[index].(map[string]any)
}

View File

@@ -14,37 +14,12 @@ const (
MediaType = "application/json"
)
// Document is the durable union of direct evidence and surrounding source
// context selected for one accepted source document.
type Document struct {
SourceID string `json:"source_id"`
SourceDigest string `json:"source_digest"`
WindowUnits int `json:"window_units"`
SelectedLanes []string `json:"selected_lanes"`
Contexts []Context `json:"contexts"`
}
// Document is the durable selected source-unit excerpt.
type Document []source.SourceUnit
type Context struct {
ContextRef source.SourceRef `json:"context_ref"`
EvidenceRefs []EvidenceRef `json:"evidence_refs"`
Units []source.SourceUnit `json:"units"`
}
type EvidenceRef struct {
LaneID string `json:"lane_id"`
SourceRef source.SourceRef `json:"source_ref"`
}
// LaneEvidence attributes direct source references to one selected lane.
type LaneEvidence struct {
LaneID string `json:"lane_id"`
SourceRefs []source.SourceRef `json:"source_refs"`
}
// BuildRequest supplies accepted source material and direct lane evidence.
// BuildRequest supplies accepted source material and projected source references.
type BuildRequest struct {
Source *source.SourceDocument
WindowUnits int
SelectedLanes []string
LaneEvidence []LaneEvidence
Source *source.SourceDocument
WindowUnits int
SourceRefs []source.SourceRef
}

View File

@@ -1 +1 @@
{"source_id":"session-alpha","source_digest":"sha256:0000000000000000000000000000000000000000000000000000000000000000","window_units":0,"selected_lanes":["npcs"],"contexts":[{"context_ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13},"evidence_refs":[{"lane_id":"npcs","source_ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13}}],"units":[{"id":13,"kind":"transcript_segment","text":"The party meets Rowan.","ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13}}]}]}
[{"id":13,"kind":"transcript_segment","text":"The party meets Rowan.","ref":{"source_id":"session-alpha","start_unit_id":13,"end_unit_id":13}}]

View File

@@ -20,7 +20,6 @@ type debugEvidenceContextSummary struct {
SchemaVersion string `json:"schema_version"`
SelectedLanes []string `json:"selected_lanes"`
WindowUnits int `json:"window_units"`
ContextCount int `json:"context_count"`
UnitCount int `json:"unit_count"`
SourceDigest string `json:"source_digest"`
}
@@ -49,10 +48,9 @@ func buildOutputEvidenceContext(prepared *PreparedPipeline, doc *source.SourceDo
}
request := evidencecontext.BuildRequest{
Source: doc,
WindowUnits: prepared.evidencePlan.policy.WindowUnits,
SelectedLanes: append([]string(nil), prepared.evidencePlan.policy.LaneIDs...),
LaneEvidence: make([]evidencecontext.LaneEvidence, 0, len(prepared.evidencePlan.lanes)),
Source: doc,
WindowUnits: prepared.evidencePlan.policy.WindowUnits,
SourceRefs: make([]source.SourceRef, 0),
}
for _, lane := range prepared.evidencePlan.lanes {
output, ok := byLane[lane.laneID]
@@ -73,10 +71,7 @@ func buildOutputEvidenceContext(prepared *PreparedPipeline, doc *source.SourceDo
if err != nil {
return nil, nil, fmt.Errorf("evidence context output lane %q: accepted normalized artifact cannot be projected", lane.laneID)
}
request.LaneEvidence = append(request.LaneEvidence, evidencecontext.LaneEvidence{
LaneID: lane.laneID,
SourceRefs: append([]source.SourceRef(nil), references...),
})
request.SourceRefs = append(request.SourceRefs, references...)
}
document, err := evidencecontext.Build(request)
@@ -99,13 +94,10 @@ func buildOutputEvidenceContext(prepared *PreparedPipeline, doc *source.SourceDo
SchemaID: artifact.Schema.ID,
SchemaName: artifact.Schema.Name,
SchemaVersion: artifact.Schema.Version,
SelectedLanes: append([]string(nil), document.SelectedLanes...),
WindowUnits: document.WindowUnits,
ContextCount: len(document.Contexts),
SourceDigest: document.SourceDigest,
}
for _, context := range document.Contexts {
summary.UnitCount += len(context.Units)
SelectedLanes: append([]string(nil), prepared.evidencePlan.policy.LaneIDs...),
WindowUnits: prepared.evidencePlan.policy.WindowUnits,
SourceDigest: doc.Digest,
UnitCount: len(document),
}
return contracts.CloneSerializedArtifactPointer(artifact), &summary, nil
}

View File

@@ -97,14 +97,11 @@ func TestRunnerBuildsEvidenceContextFromSelectedNormalizedOutputs(t *testing.T)
}
value := decodeCapturedEvidence(t, encoder)
if !reflect.DeepEqual(value.SelectedLanes, []string{"alpha", "beta", "inactive"}) || len(value.Contexts) != 1 || len(value.Contexts[0].Units) != 3 {
if actual := []int{value[0].ID, value[1].ID, value[2].ID}; !reflect.DeepEqual(actual, []int{1, 2, 3}) {
t.Fatalf("evidence context = %#v, want selected union", value)
}
if got := value.Contexts[0].EvidenceRefs; len(got) != 2 || got[0].LaneID != "alpha" || got[1].LaneID != "beta" {
t.Fatalf("evidence refs = %#v, want both selected lanes", got)
}
debugJSON := string(debug.json["output/evidence-context.json"])
if strings.Contains(debugJSON, "text-1") || strings.Contains(debugJSON, "metadata") || !strings.Contains(debugJSON, `"artifact_kind":"source/evidence-context"`) || !strings.Contains(debugJSON, `"schema_id":"notarius.source.evidence_context"`) || !strings.Contains(debugJSON, `"context_count":1`) || !strings.Contains(debugJSON, `"unit_count":3`) {
if strings.Contains(debugJSON, "text-1") || strings.Contains(debugJSON, "metadata") || !strings.Contains(debugJSON, `"artifact_kind":"source/evidence-context"`) || !strings.Contains(debugJSON, `"schema_id":"notarius.source.evidence_context"`) || strings.Contains(debugJSON, "context_count") || !strings.Contains(debugJSON, `"unit_count":3`) {
t.Fatalf("evidence debug envelope = %s, want only allowlisted summary", debugJSON)
}
}
@@ -134,7 +131,7 @@ func TestRunnerEvidenceContextOmitsAbsentAndRejectedLanes(t *testing.T) {
t.Fatalf("rejections = %#v, want rejected lane unchanged", result.Rejected)
}
value := decodeCapturedEvidence(t, encoder)
if len(value.Contexts) != 1 || len(value.Contexts[0].EvidenceRefs) != 1 || value.Contexts[0].EvidenceRefs[0].LaneID != "present" {
if len(value) != 1 || value[0].ID != 1 {
t.Fatalf("evidence context = %#v, want present lane only", value)
}
}
@@ -232,7 +229,7 @@ func TestRunnerEvidenceContextRebuildsFromAcceptedCheckpoint(t *testing.T) {
t.Fatalf("Run() error = %v", err)
}
value := decodeCapturedEvidence(t, encoder)
if len(value.Contexts) != 1 || value.Contexts[0].EvidenceRefs[0].LaneID != "notes" {
if len(value) != 1 || value[0].ID != 1 {
t.Fatalf("evidence context = %#v, want checkpointed normalized output", value)
}
}

View File

@@ -353,8 +353,8 @@ func TestEncodeIncludesValidatedEvidenceContext(t *testing.T) {
if err != nil {
t.Fatalf("Decode(evidence context file) error = %v", err)
}
if len(value.Contexts) != 0 {
t.Fatalf("evidence context = %#v, want explicit empty contexts", value)
if len(value) != 1 || value[0].ID != 7 || value[0].Text != "Source content retained only in the evidence artifact." {
t.Fatalf("evidence context = %#v, want the published source-unit array", value)
}
index := decodeObject(t, fileBytes(t, result.Files, "index.json"))
if got, want := index["evidence_context"], map[string]any{
@@ -786,7 +786,8 @@ func acceptedEvidenceContextArtifact(t *testing.T) contracts.SerializedArtifact
}
document.Digest = digest
artifact, err := evidencecontext.Serialize(evidencecontext.BuildRequest{
Source: document, WindowUnits: 3, SelectedLanes: []string{"spells"},
Source: document, WindowUnits: 3,
SourceRefs: []source.SourceRef{{SourceID: "source-1", StartUnitID: 7, EndUnitID: 7}},
})
if err != nil {
t.Fatal(err)

View File

@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"reflect"
"strings"
"sync"
"testing"
@@ -71,20 +72,17 @@ func TestLocationRegistryHandoffProducesOccurrencesAndEvidence(t *testing.T) {
if err != nil {
t.Fatalf("Decode(evidence context) error = %v", err)
}
if !locationEvidenceHasLane(evidence, "locations") || !locationEvidenceHasLane(evidence, "occurrences") {
t.Fatalf("evidence context = %#v, want registry and occurrence evidence from their own artifacts", evidence)
if actual := locationEvidenceUnitIDs(evidence); !reflect.DeepEqual(actual, []int{1, 2, 3, 4, 5}) {
t.Fatalf("evidence context = %#v, want deduplicated registry and occurrence source-unit evidence", evidence)
}
}
func locationEvidenceHasLane(document evidencecontext.Document, laneID string) bool {
for _, context := range document.Contexts {
for _, reference := range context.EvidenceRefs {
if reference.LaneID == laneID {
return true
}
}
func locationEvidenceUnitIDs(document evidencecontext.Document) []int {
ids := make([]int, len(document))
for index, unit := range document {
ids[index] = unit.ID
}
return false
return ids
}
func TestLocationOccurrenceConsumerDoesNotRunAfterRejectedRegistry(t *testing.T) {

View File

@@ -279,22 +279,8 @@ func TestProductionDNDOutputPublishesSelectedEvidenceContext(t *testing.T) {
if err != nil {
t.Fatalf("Decode(evidence context) error = %v", err)
}
if !reflect.DeepEqual(value.SelectedLanes, []string{"combat", "npc_registry", "spells"}) {
t.Fatalf("selected lanes = %#v, want configured production lanes without scene descriptions", value.SelectedLanes)
}
if len(value.Contexts) != 2 || len(value.Contexts[0].Units) != 1 || len(value.Contexts[1].Units) != 1 || value.Contexts[0].Units[0].ID != 10 || value.Contexts[1].Units[0].ID != 20 {
t.Fatalf("evidence contexts = %#v, want source-position union with non-monotonic unit IDs", value.Contexts)
}
firstRefs := value.Contexts[0].EvidenceRefs
if len(firstRefs) != 3 || firstRefs[0].LaneID != "combat" || firstRefs[1].LaneID != "npc_registry" || firstRefs[2].LaneID != "spells" {
t.Fatalf("first context evidence = %#v, want overlapping selected lane references", firstRefs)
}
for _, context := range value.Contexts {
for _, reference := range context.EvidenceRefs {
if reference.LaneID == "scene-descriptions" {
t.Fatalf("evidence refs = %#v, want scene descriptions excluded by allowlist", value.Contexts)
}
}
if actual := evidenceUnitIDs(value); !reflect.DeepEqual(actual, []int{10, 20}) {
t.Fatalf("evidence units = %#v, want deduplicated source-position union without scene descriptions", actual)
}
}
@@ -312,11 +298,19 @@ func TestProductionDNDOutputCanExplicitlySelectSceneDescriptionEvidence(t *testi
if err != nil {
t.Fatalf("Decode(evidence context) error = %v", err)
}
if !reflect.DeepEqual(value.SelectedLanes, []string{"scene-descriptions"}) || len(value.Contexts) == 0 || len(value.Contexts[0].EvidenceRefs) == 0 || value.Contexts[0].EvidenceRefs[0].LaneID != "scene-descriptions" {
t.Fatalf("evidence context = %#v, want explicitly selected scene-description evidence", value)
if len(value) == 0 {
t.Fatalf("evidence context = %#v, want explicitly selected scene-description source units", value)
}
}
func evidenceUnitIDs(value evidencecontext.Document) []int {
ids := make([]int, len(value))
for index, unit := range value {
ids[index] = unit.ID
}
return ids
}
func TestGroundedPipelineSkipsCombatForExactNarrativeScene(t *testing.T) {
registries := productionNPCRegistries(t)
configValue := loadGroundedPipelineConfig(t)