Compare commits

..

5 Commits

66 changed files with 5051 additions and 57 deletions

View File

@@ -94,6 +94,16 @@ go run ./cmd/notarius run dnd-session \
--reference spells.extract.glossary=./campaign-glossary.txt
```
For the operator-driven NPC-to-spell workflow, bind the normalized NPC lane
payload from the completed NPC run to the spell extractor:
```sh
go run ./cmd/notarius run dnd-spells \
--config examples/dnd-npc-spell-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference spells.extract.npcs=./npc-run/lanes/npcs.json
```
The same grammar can target chunk, merge, and normalize slots when the configured
modules declare them:

View File

@@ -21,8 +21,10 @@ The explicit-path option is defined in the [CLI reference](cli.md).
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
- [D&D NPC configuration](../examples/dnd-npcs.config.yml)
- [Sequential D&D NPC and spell configuration](../examples/dnd-npc-spell-sequential.config.yml)
Both are complete version 3 files. The fragments below illustrate individual
All are complete version 3 files. The fragments below illustrate individual
fields and are not alternate complete configurations.
## Top-Level Fields
@@ -283,9 +285,11 @@ production validators do not call the LLM and must not set `llm_profile`.
| chunk | `generic` | Splits source units into ordered chunks. |
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
| extract | `dnd/npcs` | Extracts typed D&D NPC-list artifacts. |
| merge | `appendorder` | Combines typed artifacts in chunk order. |
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical identity and aliases. |
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
## Implemented Production Validators
@@ -300,6 +304,10 @@ production validators do not call the LLM and must not set `llm_profile`.
| `extract/dnd/spells/catalog` | deterministic | Rejects spell-list artifacts containing names outside the effective SRD and overlay catalog. |
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
| `extract/dnd/npcs/shape` | deterministic | Rejects malformed D&D NPC-list artifacts. |
| `extract/dnd/npcs/source_refs` | deterministic | Rejects missing or invalid D&D NPC source references. |
| `extract/dnd/npcs/source_relatedness` | deterministic | Emits warnings when an NPC name or alias is not found near its cited source text. |
| `normalize/dnd/npcs/identity` | deterministic | Rejects invalid canonical IDs, aliases, and cross-record identity collisions. |
The production default chain for `dnd/spells` is used for both its extract and
normalize stages:
@@ -314,8 +322,27 @@ validators:
- extract/dnd/spells/source_relatedness
```
No other production module currently has a default validator chain. Empty
chains approve output by default.
The production default chain for `dnd/npcs` uses the extraction chain for the
extract stage and the identity chain for normalize-stage output:
```yaml
extract:
validators:
- generic/valid_json
- generic/valid_json_schema
- extract/dnd/npcs/shape
- extract/dnd/npcs/source_refs
- extract/dnd/npcs/source_relatedness
normalize:
validators:
- generic/valid_json
- generic/valid_json_schema
- normalize/dnd/npcs/identity
- extract/dnd/npcs/source_refs
- extract/dnd/npcs/source_relatedness
```
Empty chains approve output by default.
The `generic` chunker accepts:
@@ -344,6 +371,17 @@ not allow multiple files. Its format is defined in the
The extractor uses campaign references only as supporting disambiguation
material; spell casts still must be present in the source transcript.
It also declares an optional `npcs` slot for a normalized NPC artifact. The
slot accepts exactly one `application/json` file no larger than 1 MiB. During
extractor preparation Notarius strictly decodes and identity-validates the
artifact, then gives the model canonical JSON for caster-name grounding.
Registry source references may belong to the NPC-producing session and are
provenance only; they are not spell evidence. The bound registry contributes a
semantic digest and NPC count to extractor metadata and checkpoint identity,
while its names, aliases, content, and path do not appear there. When absent,
the prompt receives the exact empty value `{"npcs":[]}` and no registry
provenance or fingerprint is recorded.
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
When an overlay is used, bind it independently under
`artifacts.<lane>.normalize.references.spell_catalog`; normalize-stage
@@ -351,6 +389,12 @@ references are local to that stage and are not inherited from extraction. The
normalizer uses the embedded SRD catalog when no normalize-stage overlay is
bound.
The `dnd/npcs` extractor declares the same optional campaign slots as the spell
extractor, but it does not declare the `npcs` registry slot. Its normalizer
accepts no references. To pass an NPC result to a later spell run, bind the
normalized payload explicitly at runtime; the maintained sequential example
documents that operator workflow.
## State Surfaces
The `output`, `cache`, and `debug` top-level fields select independent physical

View File

@@ -0,0 +1,106 @@
# D&D NPC Artifact
This document defines the durable D&D NPC-list artifact, its JSON codec, and
the selectable production NPC pipeline. The normalized JSON payload can be
passed explicitly to the spell extractor as an optional caster-name registry;
it remains a reference, not spell evidence.
## Identity
- Artifact kind: `dnd/npc-list`
- Durable schema ID: `notarius.dnd.npcs`
- Durable schema name: `notarius_dnd_npcs_v1`
- Durable schema version: `v1`
- Media type: `application/json`
- Identity policy: `dnd.npcs.identity.v1`
The durable JSON Schema is owned by the D&D NPC codec. NPC IDs are derived from
the Unicode-normalized, case-folded canonical name using the identity policy.
The durable codec enforces the artifact shape and ID syntax; registry identity
validation remains a separate deterministic concern.
## Output Shape
The payload is one object with a required top-level `npcs` array:
```json
{"npcs": []}
```
The array may be empty. Every object and nested object rejects unknown fields.
## NPC Fields
Each NPC contains exactly these required fields:
- `id`: `npc:sha256:` followed by 64 lowercase hexadecimal characters;
- `name`: the canonical display name;
- `aliases`: an array of alternate display names, which may be empty;
- `description`: a concise description;
- `relationships`: an array of target/relationship objects, which may be empty;
- `source_refs`: at least one source reference supporting the NPC record.
Each relationship contains required `target` and `relationship` strings. Each
source reference contains required `source_id`, `start_unit_id`, and
`end_unit_id`; unit IDs are positive integers. Source document identity, unit
existence, and range ordering are validated by the source-reference validator
when the artifact is used by a pipeline.
## Codec Boundary
`EncodeCandidate` and `DecodeCandidate` provide strict single-value JSON
serialization while preserving typed values that still need semantic
validation. `Encode` and `Decode` are the approved-artifact boundary and
require all durable structural fields, non-empty required strings, valid source
reference shapes, and the NPC ID pattern.
Codec metadata contains only `npc_count`. Schema bytes and returned metadata
are independent values so callers cannot mutate codec-owned state.
## Production Pipeline
The production identities are:
- extractor: `dnd/npcs`;
- artifact kind: `dnd/npc-list`;
- normalizer: `dnd/npcs`; and
- durable schema: `notarius.dnd.npcs`, version `v1`, media type
`application/json`.
The extractor maps private model records to the current source identity and
assigns deterministic IDs. Extraction validation checks shape, source
references, and source relatedness. The normalizer then consolidates records
by canonical identity or canonical-name/alias matches, preserves the first
record's display and output position, unions relationships and exact evidence,
rewrites unambiguous relationship targets to canonical names, and validates
the retained registry's identity. No LLM is used for consolidation.
The default extraction chain is `generic/valid_json`,
`generic/valid_json_schema`, `extract/dnd/npcs/shape`,
`extract/dnd/npcs/source_refs`, and
`extract/dnd/npcs/source_relatedness`. The normalize chain adds
`normalize/dnd/npcs/identity` before the source-reference and relatedness
checks. Relatedness emits bounded warnings when an NPC canonical name or
alias is not present near its cited transcript text; opaque campaign
references may explain such a warning but do not become evidence.
## Manifest And Sequential Consumption
The NPC extractor records prompt and response-schema identities. The durable
codec records only `npc_count`; raw names, aliases, descriptions, source
references, and payload bytes stay in the lane file rather than manifest
metadata. The normalized lane is independently reusable as a file reference:
```sh
go run ./cmd/notarius run dnd-spells \
--config examples/dnd-npc-spell-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
```
The spell extractor strictly decodes and identity-validates this file, accepts
source references belonging to another session as registry provenance, and
uses only canonical names and aliases for caster grounding. Those NPC source
references are never accepted as spell evidence. The spell run's manifest
keeps raw file provenance under `references` and records only the prepared
registry's semantic digest and count in extractor metadata.

View File

@@ -94,6 +94,22 @@ Reference slot keys and accepted file types are defined in
supporting disambiguation material, not source evidence, and are not
addressable through `source_refs`.
## Optional NPC Grounding
The `dnd/spells` extractor accepts an optional `npcs` reference containing one
normalized NPC artifact as `application/json`, up to 1 MiB. Preparation uses
the approved NPC codec and identity policy to validate the file, re-encodes
canonical durable JSON, and supplies that JSON as a spell-owned prompt input.
It helps the model prefer canonical caster names and recognize aliases; it
does not establish that a spell was cast.
NPC source references may identify the run that produced the registry or any
other session. They remain registry provenance and are never copied into a
spell cast's `source_refs`; every spell evidence range must still identify the
current transcript. When the slot is absent, the prompt receives exactly
`{"npcs":[]}` and the run has no NPC reference provenance or NPC checkpoint
fingerprint.
## Normalization Behavior
When the `dnd/spells` normalizer is selected, each recognized spell name is
@@ -153,7 +169,9 @@ manifest metadata:
"response_schema_sha256": "sha256:...",
"catalog_base_id": "dnd-5e-2014-srd-spells",
"catalog_digest": "sha256:...",
"catalog_overlay_ids": ["campaign.example"]
"catalog_overlay_ids": ["campaign.example"],
"npc_registry_digest": "sha256:...",
"npc_count": 3
},
"normalizer": {
"catalog_base_id": "dnd-5e-2014-srd-spells",
@@ -172,8 +190,15 @@ identity fields when that module is selected. Overlay origin, media type, byte
size, and raw digest are recorded separately in the manifest's reference
provenance; see the [JSON output contract](json-output.md#manifestjson).
The `npc_registry_digest` and `npc_count` fields in the example are present only
when the optional NPC registry is bound. They contain no NPC names, aliases,
source references, paths, or raw bytes.
The extractor's prompt hash, private response-schema hash, and effective catalog
digest also contribute independently scoped semantic checkpoint fingerprints.
Changing any of those prepared contracts intentionally produces a cold
checkpoint miss. Fingerprints contain only digests, never prompt, schema,
catalog, or reference content.
catalog, or reference content. When an NPC registry is bound, its semantic
digest contributes an additional local `npc_registry` fingerprint; the
manifest metadata contains only that digest and `npc_count`. Raw NPC file
provenance remains independently recorded in the manifest's `references` list.

View File

@@ -151,7 +151,8 @@ accepts only artifacts whose codec media type is `application/json`. The file
contains the codec-owned JSON bytes pretty-printed.
The schema of each lane payload is owned by that artifact contract. For the
current D&D spell lane, see [D&D Spell Artifact](dnd-spell-artifacts.md).
current D&D lanes, see [D&D Spell Artifact](dnd-spell-artifacts.md) and
[D&D NPC Artifact](dnd-npc-artifacts.md).
## `rejected.json`

View File

@@ -39,9 +39,9 @@ without exposing Scriptorium types through stage contracts.
7. injecting that one shared client into complete pipeline preparation before
the source file is read or the runner is invoked.
The D&D scene chunker and spell extractor retain this injected client and use
it for every structured completion. Operation requests do not carry an LLM
client.
The D&D scene chunker and spell and NPC extractors retain this injected client
and use it for every structured completion. Operation requests do not carry an
LLM client.
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
and validator bindings. It prepares a small internal check prompt for each ID so
@@ -104,11 +104,15 @@ The small framework registry contains only generic test schemas; production
schemas remain package-owned.
The spell extractor's package-owned prompt declares a required
`application/json` `spell_catalog` input. The extractor generates that input
from its prepared effective catalog as `{"spell_names":[...]}` using sorted
canonical names only. Its input digest covers those generated bytes; manifests
record catalog identity and digest rather than names, aliases, overlay bytes,
or source metadata.
`application/json` `spell_catalog` input and an optional `application/json`
`npcs` input. The extractor generates the catalog input from its prepared
effective catalog as `{"spell_names":[...]}` using sorted canonical names only.
When an NPC registry is bound, it strictly decodes and identity-validates one
durable artifact, re-encodes canonical JSON, and generates a semantic digest
over those bytes. The unbound input is exactly `{"npcs":[]}`. Input digests
cover the generated bytes; manifests record catalog identity and optional NPC
registry digest/count rather than names, aliases, overlay bytes, registry
paths, or source metadata.
## Debug And Redaction Boundaries

View File

@@ -25,8 +25,9 @@ of that set; input and output builders receive no references. The spell
extractor is typed over the canonical D&D model. D&D validators, merge, and
normalize use typed variants; JSON representation validators use serialized
requests; and unconditional validators expose separate chunk and typed
variants. The D&D production registrar registers only the canonical typed spell
implementations.
variants. The D&D production registrar registers the canonical typed spell and
NPC implementations, including their kind-specific merge and normalize
behavior.
Prepared extractors, extract validators, and codecs may be reused concurrently
by the run-wide extract pool. Production implementations are immutable after
@@ -81,6 +82,15 @@ semantic digest; overlay content remains contextual reference material rather
than source evidence. Its external JSON contract is defined in the
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).
### `internal/modules/dnd/npcs/identity` and `internal/modules/dnd/codec/npcs`
The NPC identity package owns Unicode comparison keys, deterministic
`npc:sha256:` IDs, display normalization, and whole-registry collision issues.
The NPC codec owns the strict durable `dnd/npc-list` JSON boundary and exposes
candidate versus approved encode/decode operations. NPC source references are
durable provenance and may later be consumed by another pipeline as registry
context without being treated as evidence for that pipeline.
## Input Adapter
### `internal/modules/seriatim/input/transcript`
@@ -186,9 +196,34 @@ stages, using the codec only for checkpoint, debug, and output boundaries.
Shared D&D helpers keep prompt input
names and source-unit reference conversion consistent with the scene chunker.
The extractor also declares the optional `npcs` registry slot. Preparation
requires one approved `application/json` item no larger than 1 MiB, validates
identity without relating registry source references to the current transcript,
and supplies canonical JSON to a spell-owned prompt message. A bound registry
adds only `npc_registry_digest` and `npc_count` to manifest metadata and an
`npc_registry` checkpoint fingerprint. The unbound prompt input is exactly
`{"npcs":[]}` and has no registry provenance or fingerprint.
The durable payload and manifest metadata shapes are defined in the
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
### `internal/modules/dnd/extract/npcs`
The NPC extractor maps private model output to the canonical `dnd.NPCList`,
assigns source identity and deterministic NPC IDs, and preserves source
references for deterministic validation. It uses the shared campaign
references only for disambiguation and does not consume the spell-owned NPC
registry slot. Its prompt and private response schema are package-owned.
### `internal/modules/dnd/normalize/npcs`
The NPC normalizer performs deterministic identity-aware consolidation in
merged input order. It unions only canonical identity or canonical/alias
matches, retains the first display record, unions exact relationships and
source references, rewrites unambiguous relationship targets, and leaves
ambiguous collisions for identity validation. It exposes the identity policy
as its local checkpoint fingerprint and emits bounded normalization warnings.
## Merger And Normalizer
### `internal/modules/generic/merge/appendorder`
@@ -272,6 +307,17 @@ are defined in
payload rules are defined in the
[artifact contract](../integrations/dnd-spell-artifacts.md).
## D&D NPC Validators
NPC shape validation checks required strings, arrays, and source-reference
shape. The source-reference validator checks current-document identity, unit
existence, and range ordering; source relatedness emits at most one bounded
warning per record when neither the canonical name nor an alias occurs near
its cited text. Normalize identity validation checks deterministic IDs,
canonical names, aliases, and cross-record ownership or canonical collisions.
All are deterministic and expose the policy fingerprints used by the
production chains.
## Production Registration
Production composition occurs through family registrars. The CLI allocates one
@@ -280,8 +326,8 @@ complete framework registry set and one LLM asset registry. It invokes
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
that order, then exposes the matching catalog for resolution. The generic and
Seriatim registrars own their production leaf registrations. The D&D registrar
owns D&D leaf registrations, the typed spell default-validator chains, and D&D
prompt/schema asset collection.
owns D&D leaf registrations, typed spell and NPC default-validator chains,
typed append-order specializations, and D&D prompt/schema asset collection.
Concrete implementation packages do not import generic implementation
packages directly. A concrete family's `register` package is its composition

View File

@@ -65,9 +65,9 @@ run-local construction closures. Preparation injects shared dependencies and
constructs input, chunk, validators, ordered lanes, and output before source
parsing. Production modules use strict construction-time option decoding, and
LLM-backed modules retain the injected shared client. The D&D family registers
the canonical `dnd/spell-list` codec, typed spell extractor, normalizer, and
validators, plus kind-specific generic merge strategies; generic JSON validators
use the serialized-validation contract. The runner executes lanes through
the canonical `dnd/spell-list` and `dnd/npc-list` codecs, typed spell and NPC
extractors and normalizers, validators, plus kind-specific generic merge
strategies; generic JSON validators use the serialized-validation contract. The runner executes lanes through
private exact-type-checked closures, coordinates extract results independently
of completion timing, and serializes artifacts only through their codec at
checkpoint, debug, and output boundaries.
@@ -84,13 +84,17 @@ Configuration. The implemented module packages are:
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, and relationship artifact types. |
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. |
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. |
| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. |
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by identity and aliases, rewrites unambiguous relationship targets, and reports bounded warnings. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
@@ -98,6 +102,14 @@ reference declarations, prompt input assembly, and source-unit reference
helpers. Domain-neutral prompt filesystem composition lives in
`internal/framework/promptfs`.
The spell extractor owns its optional `npcs` registry boundary. Preparation
strictly decodes and identity-validates one normalized JSON artifact, emits
canonical registry JSON to the spell prompt, and records only its semantic
digest and count in prepared metadata. The raw reference remains independently
tracked by pipeline provenance. An absent registry is represented only by the
empty prompt value `{"npcs":[]}`; the shared D&D reference fragment is not
changed.
Generic validators under `internal/modules/generic/validate` provide
unconditional test decisions, JSON syntax validation, and JSON Schema
validation. D&D spell validators under `internal/modules/dnd/validate/spells`

View File

@@ -40,6 +40,36 @@ names, schemas, and media types inside a run directory.
Remove an output run directory only after its consumer data is no longer
needed. This is data deletion, not cache cleanup.
## Sequential NPC And Spell Runs
The maintained [sequential configuration](../examples/dnd-npc-spell-sequential.config.yml)
contains two independent pipelines over the same Seriatim input shape. Run the
NPC pipeline first and retain its normalized payload:
```sh
go run ./cmd/notarius run dnd-npcs \
--config examples/dnd-npc-spell-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--output-dir ./npc-output
```
Then bind that completed run's `lanes/npcs.json` file to the spell extractor:
```sh
go run ./cmd/notarius run dnd-spells \
--config examples/dnd-npc-spell-sequential.config.yml \
--input examples/seriatim-minimal-transcript.json \
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
```
The NPC file is a reference for canonical caster names and aliases, not spell
evidence. The spell manifest records the bound file's raw reference provenance
and the prepared registry's count and semantic digest separately. The NPC
payload, names, aliases, source references, and file bytes can be sensitive
campaign data; protect both output roots and any checkpoint or debug roots that
retain derived application data. A registry from another session is allowed,
but its source references are never copied into spell output evidence.
## Chunk-Plan Cache
Chunk plans are stored at:

View File

@@ -1,6 +1,6 @@
# D&D NPC Extraction And Registry
Status: Accepted.
Status: Complete.
## Purpose

View File

@@ -18,10 +18,6 @@ not as committed release dates.
### Add Sequential D&D Artifacts
- Add the proposed [D&D NPC extraction and registry](dnd-npc-extraction.md) as
the next sequential artifact. It defines canonical identity, aliases,
descriptions, relationships, evidence, deterministic consolidation, and
direct reuse as spell-pipeline reference material.
- Add combat-turn extraction with explicit event and source-reference
semantics. Use earlier NPC output as a reference to improve participant
identity and consistency.

View File

@@ -1,6 +1,6 @@
# D&D NPC Extraction And Registry Implementation Plan
Status: Ready for implementation.
Status: Complete.
Implement this plan in order. The feature policy and target state are defined
in [D&D NPC Extraction And Registry](dnd-npc-extraction.md); this document owns

View File

@@ -0,0 +1,28 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints:
enabled: false
directory: ""
debug:
directory: ./notarius-debug
pipelines:
dnd-npcs:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
retries: 2
normalize: dnd/npcs
dnd-spells:
input: seriatim
chunk: generic
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells

View File

@@ -0,0 +1,21 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints:
enabled: false
directory: ""
debug:
directory: ./notarius-debug
pipelines:
dnd-session:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
retries: 2
normalize: dnd/npcs

View File

@@ -0,0 +1,132 @@
package cli
import (
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
)
func TestProductionNPCConfigurationResolvesTypedLane(t *testing.T) {
components := productionTestComponents(t)
catalog := catalogFromRegistries(components.registries)
configPath := repositoryPath("examples", "dnd-npcs.config.yml")
cfg := loadMaintainedExample(t, configPath)
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalog})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
}
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 {
t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.ArtifactLanes)
}
lane := effective.ResolvedPipeline.ArtifactLanes[0]
if lane.ID != "npcs" || lane.ArtifactKind != dnd.NPCListKind || lane.Extract.Module != npcextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != npcnormalize.Key {
t.Fatalf("resolved NPC lane = %#v, want typed production composition", lane)
}
if len(lane.ExtractReferences.Bindings) != 0 || len(lane.NormalizeReferences.Bindings) != 0 {
t.Fatalf("unbound NPC references = %#v / %#v, want none", lane.ExtractReferences, lane.NormalizeReferences)
}
extractSpec, ok := catalog.Extractors.Spec(npcextract.Key)
if !ok || !reflect.DeepEqual(extractSpec.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(extractSpec.Provides, []string{"dnd.npcs"}) {
t.Fatalf("NPC extractor spec = %#v, want source and artifact capabilities", extractSpec)
}
mergeSpec, ok := catalog.Mergers.SpecForArtifact(pipeline.DefaultMergeModule, dnd.NPCListKind)
if !ok || !reflect.DeepEqual(mergeSpec.Provides, []string{"merged"}) {
t.Fatalf("NPC merger spec = %#v, want merged capability", mergeSpec)
}
normalizeSpec, ok := catalog.Normalizers.SpecForArtifact(npcnormalize.Key, dnd.NPCListKind)
if !ok || !reflect.DeepEqual(normalizeSpec.Requires, []string{"merged"}) || !reflect.DeepEqual(normalizeSpec.Provides, []string{"normalized"}) {
t.Fatalf("NPC normalizer spec = %#v, want merged/normalized capabilities", normalizeSpec)
}
wantExtractChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/npcs/shape"),
pipeline.Binding("extract/dnd/npcs/source_refs"),
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
}
wantNormalizeChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/npcs/shape"),
pipeline.Binding("normalize/dnd/npcs/identity"),
pipeline.Binding("extract/dnd/npcs/source_refs"),
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageExtract, npcextract.Key); !reflect.DeepEqual(got, wantExtractChain) {
t.Fatalf("NPC extract chain = %#v, want %#v", got, wantExtractChain)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageNormalize, npcnormalize.Key); !reflect.DeepEqual(got, wantNormalizeChain) {
t.Fatalf("NPC normalize chain = %#v, want %#v", got, wantNormalizeChain)
}
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageMerge, pipeline.DefaultMergeModule); len(got) != 0 {
t.Fatalf("NPC merge chain = %#v, want empty", got)
}
}
func TestProductionNPCConfigurationValidatesOptionsReferencesAndPlacement(t *testing.T) {
components := productionTestComponents(t)
configPath := repositoryPath("examples", "dnd-npcs.config.yml")
resolve := func(mutate func(*pipeline.PipelineProfile)) error {
cfg := loadMaintainedExample(t, configPath)
profile := cfg.Pipelines["dnd-session"]
mutate(&profile)
cfg.Pipelines["dnd-session"] = profile
_, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(components.registries)})
return err
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["npcs"]
lane.Extract.Options = map[string]any{"unexpected": true}
profile.Artifacts["npcs"] = lane
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("unknown extractor option error = %v, want strict option rejection", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["npcs"]
lane.Normalize.Options = map[string]any{"unexpected": true}
profile.Artifacts["npcs"] = lane
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
profile.References = map[string]string{
"players": "players.txt",
"party": "party.txt",
"glossary": "glossary.txt",
}
}); err != nil {
t.Fatalf("optional NPC references error = %v, want resolution success", err)
}
if err := resolve(func(profile *pipeline.PipelineProfile) {
lane := profile.Artifacts["npcs"]
lane.Validators = []pipeline.ModuleBinding{pipeline.Binding("normalize/dnd/npcs/identity")}
profile.Artifacts["npcs"] = lane
}); err == nil || !strings.Contains(err.Error(), "artifact lane level") {
t.Fatalf("lane-level validator error = %v, want invalid placement rejection", err)
}
}
func validatorChain(resolved pipeline.ResolvedPipeline, stage pipeline.ModuleStage, module string) []pipeline.ModuleBinding {
for _, chain := range resolved.ValidatorChains {
if chain.Stage == stage && chain.ModuleKey == module {
bindings := make([]pipeline.ModuleBinding, len(chain.Validators))
for index, validator := range chain.Validators {
bindings[index] = validator.Binding
}
return bindings
}
}
return nil
}

View File

@@ -20,16 +20,17 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
for _, example := range maintainedExampleFiles(t) {
t.Run(example.name, func(t *testing.T) {
cfg := loadMaintainedExample(t, example.path)
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
for _, pipelineID := range example.pipelineIDs {
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID))
if err != nil {
t.Fatalf("resolve maintained example: %v", err)
t.Fatalf("resolve maintained example %q: %v", pipelineID, err)
}
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{
ConfigPath: example.path,
WorkingDir: filepath.Dir(example.path),
})
if err != nil {
t.Fatalf("materialize maintained example references: %v", err)
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
}
if example.name == "production" {
if len(materialized.ArtifactLanes) != 1 ||
@@ -38,9 +39,10 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.ArtifactLanes)
}
}
}
var stdout, stderr strings.Builder
code := RunWithOptions([]string{"pipelines", "list", "--config", example.path}, &stdout, &stderr, productionOptionsFromComponents(components))
if code != 0 || stdout.String() != "dnd-session\n" || stderr.Len() != 0 {
if code != 0 || stdout.String() != strings.Join(example.pipelineIDs, "\n")+"\n" || stderr.Len() != 0 {
t.Fatalf("pipelines list: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
}
})

View File

@@ -0,0 +1,64 @@
package cli
import (
"context"
"errors"
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
)
func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *testing.T) {
components := productionTestComponents(t)
npcPath := filepath.Join(t.TempDir(), "npcs.json")
if err := os.WriteFile(npcPath, []byte(strings.Repeat("x", 1048577)), 0o600); err != nil {
t.Fatal(err)
}
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
content := string(readRepositoryFile(t, "examples", "dnd-npc-spell-sequential.config.yml"))
content = replaceRequiredOnce(t, content, " extract: dnd/spells", " extract:\n module: dnd/spells\n references:\n npcs: "+npcPath)
content = replaceRequiredOnce(t, content, " enabled: false\n directory: \"\"", " enabled: true\n directory: "+checkpointRoot)
configPath := filepath.Join(t.TempDir(), "config.yml")
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
llmConstructed := false
chunkStoreConstructed := false
options := Options{
Catalog: catalogFromRegistries(components.registries),
Registries: components.registries,
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
llmConstructed = true
return nil, nil, errors.New("LLM client must not be constructed")
},
ChunkPlanStoreFactory: func(string) (pipeline.ChunkPlanStore, error) {
chunkStoreConstructed = true
return nil, errors.New("chunk-plan store must not be constructed")
},
}
var stdout, stderr strings.Builder
code := RunWithOptions([]string{
"run", "dnd-spells", "--config", configPath,
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
"--chunk_cache", "bypass", "--output-dir", t.TempDir(),
}, &stdout, &stderr, options)
for _, fragment := range []string{`pipeline "dnd-spells"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} {
if code == 0 || !strings.Contains(stderr.String(), fragment) {
t.Fatalf("RunWithOptions() code = %d stderr = %q, want context fragment %q", code, stderr.String(), fragment)
}
}
if llmConstructed || chunkStoreConstructed {
t.Fatalf("runtime construction = LLM %t, chunk store %t; want materialization failure first", llmConstructed, chunkStoreConstructed)
}
if _, err := os.Stat(checkpointRoot); !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("checkpoint root stat error = %v, want no checkpoint allocation", err)
}
}

View File

@@ -456,13 +456,16 @@ func TestProductionSceneRunRecordsChunkerWarningsAndProvenance(t *testing.T) {
type maintainedExample struct {
name string
path string
pipelineIDs []string
}
func maintainedExampleFiles(t *testing.T) []maintainedExample {
t.Helper()
return []maintainedExample{
{name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml")},
{name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml")},
{name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml"), pipelineIDs: []string{"dnd-session"}},
{name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml"), pipelineIDs: []string{"dnd-session"}},
{name: "npcs", path: repositoryPath("examples", "dnd-npcs.config.yml"), pipelineIDs: []string{"dnd-session"}},
{name: "sequential", path: repositoryPath("examples", "dnd-npc-spell-sequential.config.yml"), pipelineIDs: []string{"dnd-npcs", "dnd-spells"}},
}
}

View File

@@ -0,0 +1,86 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "notarius.dnd.npcs",
"type": "object",
"additionalProperties": false,
"required": ["npcs"],
"properties": {
"npcs": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"id",
"name",
"aliases",
"description",
"relationships",
"source_refs"
],
"properties": {
"id": {
"type": "string",
"pattern": "^npc:sha256:[0-9a-f]{64}$"
},
"name": {
"type": "string",
"minLength": 1
},
"aliases": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"description": {
"type": "string",
"minLength": 1
},
"relationships": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["target", "relationship"],
"properties": {
"target": {
"type": "string",
"minLength": 1
},
"relationship": {
"type": "string",
"minLength": 1
}
}
}
},
"source_refs": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["source_id", "start_unit_id", "end_unit_id"],
"properties": {
"source_id": {
"type": "string",
"minLength": 1
},
"start_unit_id": {
"type": "integer",
"minimum": 1
},
"end_unit_id": {
"type": "integer",
"minimum": 1
}
}
}
}
}
}
}
}
}

View File

@@ -0,0 +1,149 @@
package npcs
import (
"bytes"
"embed"
"encoding/json"
"fmt"
"io"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
const (
SchemaID = "notarius.dnd.npcs"
SchemaName = "notarius_dnd_npcs_v1"
SchemaVersion = "v1"
MediaType = "application/json"
)
//go:embed assets/schemas/dnd_npcs.v1.json
var schemaAssets embed.FS
var _ contracts.ArtifactCodec[dnd.NPCList] = (*Codec)(nil)
type Codec struct{}
func New() *Codec { return &Codec{} }
func (c *Codec) Kind() contracts.ArtifactKind { return dnd.NPCListKind }
func (c *Codec) Schema() contracts.ArtifactSchema {
raw, err := schemaAssets.ReadFile("assets/schemas/dnd_npcs.v1.json")
if err != nil {
return contracts.ArtifactSchema{}
}
return contracts.ArtifactSchema{
ID: SchemaID,
Name: SchemaName,
Version: SchemaVersion,
JSONSchema: append([]byte(nil), raw...),
}
}
func (c *Codec) MediaType() string { return MediaType }
func (c *Codec) Metadata(value dnd.NPCList) map[string]any {
return map[string]any{"npc_count": len(value.NPCs)}
}
func (c *Codec) Encode(value dnd.NPCList) ([]byte, error) {
if err := validate(value); err != nil {
return nil, fmt.Errorf("encode dnd npc list: %w", err)
}
return c.EncodeCandidate(value)
}
// EncodeCandidate provides the durable representation before semantic
// validators have approved a value.
func (c *Codec) EncodeCandidate(value dnd.NPCList) ([]byte, error) {
content, err := json.Marshal(value)
if err != nil {
return nil, fmt.Errorf("encode dnd npc list: %w", err)
}
return content, nil
}
func (c *Codec) Decode(content []byte) (dnd.NPCList, error) {
value, err := c.DecodeCandidate(content)
if err != nil {
return dnd.NPCList{}, err
}
if err := validate(value); err != nil {
return dnd.NPCList{}, fmt.Errorf("decode dnd npc list: %w", err)
}
return value, nil
}
// DecodeCandidate reads one strict durable JSON value before semantic
// validators have approved it.
func (c *Codec) DecodeCandidate(content []byte) (dnd.NPCList, error) {
decoder := json.NewDecoder(bytes.NewReader(content))
decoder.DisallowUnknownFields()
var value dnd.NPCList
if err := decoder.Decode(&value); err != nil {
return dnd.NPCList{}, fmt.Errorf("decode dnd npc list: %w", err)
}
var trailing any
if err := decoder.Decode(&trailing); err != io.EOF {
return dnd.NPCList{}, fmt.Errorf("decode dnd npc list: multiple JSON values")
}
return value, nil
}
func validate(value dnd.NPCList) error {
if value.NPCs == nil {
return fmt.Errorf("npcs must be present")
}
for index, npc := range value.NPCs {
prefix := fmt.Sprintf("npcs[%d]", index)
if !identity.IsValidID(npc.ID) {
return fmt.Errorf("%s.id must match npc ID pattern", prefix)
}
if strings.TrimSpace(npc.Name) == "" {
return fmt.Errorf("%s.name must not be empty", prefix)
}
if npc.Aliases == nil {
return fmt.Errorf("%s.aliases must be present", prefix)
}
for aliasIndex, alias := range npc.Aliases {
if strings.TrimSpace(alias) == "" {
return fmt.Errorf("%s.aliases[%d] must not be empty", prefix, aliasIndex)
}
}
if strings.TrimSpace(npc.Description) == "" {
return fmt.Errorf("%s.description must not be empty", prefix)
}
if npc.Relationships == nil {
return fmt.Errorf("%s.relationships must be present", prefix)
}
for relationshipIndex, relationship := range npc.Relationships {
relationshipPrefix := fmt.Sprintf("%s.relationships[%d]", prefix, relationshipIndex)
if strings.TrimSpace(relationship.Target) == "" {
return fmt.Errorf("%s.target must not be empty", relationshipPrefix)
}
if strings.TrimSpace(relationship.Relationship) == "" {
return fmt.Errorf("%s.relationship must not be empty", relationshipPrefix)
}
}
if len(npc.SourceRefs) == 0 {
return fmt.Errorf("%s.source_refs must not be empty", prefix)
}
for refIndex, ref := range npc.SourceRefs {
refPrefix := fmt.Sprintf("%s.source_refs[%d]", prefix, refIndex)
if strings.TrimSpace(ref.SourceID) == "" {
return fmt.Errorf("%s.source_id must not be empty", refPrefix)
}
if ref.StartUnitID <= 0 {
return fmt.Errorf("%s.start_unit_id must be positive", refPrefix)
}
if ref.EndUnitID <= 0 {
return fmt.Errorf("%s.end_unit_id must be positive", refPrefix)
}
}
}
return nil
}

View File

@@ -0,0 +1,155 @@
package npcs
import (
"bytes"
"encoding/json"
"os"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
func validList() dnd.NPCList {
return dnd.NPCList{NPCs: []dnd.NPC{{
ID: identity.DeriveID("Mira Thorn"),
Name: "Mira Thorn",
Aliases: []string{"The Greencloak"},
Description: "A guarded ranger who watches the northern road.",
Relationships: []dnd.NPCRelationship{{
Target: "Captain Vale", Relationship: "reports to",
}},
SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}},
}}}
}
func TestCodecMatchesMaintainedDurableFixture(t *testing.T) {
raw, err := os.ReadFile("testdata/dnd_npcs.v1.json")
if err != nil {
t.Fatalf("read durable fixture: %v", err)
}
codec := New()
value, err := codec.Decode(raw)
if err != nil {
t.Fatalf("Decode() error = %v, want nil", err)
}
want := validList()
if !reflect.DeepEqual(value, want) {
t.Fatalf("Decode() = %#v, want %#v", value, want)
}
encoded, err := codec.Encode(value)
if err != nil {
t.Fatalf("Encode() error = %v, want nil", err)
}
var compact bytes.Buffer
if err := json.Compact(&compact, raw); err != nil {
t.Fatalf("compact durable fixture: %v", err)
}
if !bytes.Equal(encoded, compact.Bytes()) {
t.Fatalf("Encode() = %s, want stable durable JSON %s", encoded, compact.Bytes())
}
}
func TestCodecOwnsDurableSchemaAndRegistersExactType(t *testing.T) {
codec := New()
schema := codec.Schema()
if codec.Kind() != dnd.NPCListKind || codec.MediaType() != MediaType {
t.Fatalf("codec identity = %q/%q", codec.Kind(), codec.MediaType())
}
if schema.ID != SchemaID || schema.Name != SchemaName || schema.Version != SchemaVersion || !json.Valid(schema.JSONSchema) {
t.Fatalf("schema = %#v, want durable NPC schema", schema)
}
var document map[string]any
if err := json.Unmarshal(schema.JSONSchema, &document); err != nil || document["$id"] != SchemaID {
t.Fatalf("durable schema document = %#v, %v", document, err)
}
registry := pipeline.NewArtifactCodecRegistry()
if err := pipeline.RegisterArtifactCodec(registry, codec); err != nil {
t.Fatalf("RegisterArtifactCodec() error = %v", err)
}
spec, ok := registry.Spec(dnd.NPCListKind)
if !ok || spec.SchemaDigest != contracts.DigestArtifactSchema(schema) {
t.Fatalf("registered spec = %#v, %t", spec, ok)
}
}
func TestCodecStrictlyRejectsMalformedOrUnknownJSON(t *testing.T) {
codec := New()
tests := []struct {
name string
raw string
want string
}{
{name: "unknown top-level", raw: `{"npcs":[],"unexpected":true}`, want: "unknown field"},
{name: "unknown nested", raw: `{"npcs":[{"id":"x","name":"Mira","aliases":[],"description":"desc","relationships":[],"source_refs":[{"source_id":"s","start_unit_id":1,"end_unit_id":1}],"unexpected":true}]}`, want: "unknown field"},
{name: "trailing", raw: `{"npcs":[]} {}`, want: "multiple JSON values"},
{name: "missing array", raw: `{}`, want: "npcs must be present"},
{name: "invalid reference", raw: `{"npcs":[{"id":"npc:sha256:0000000000000000000000000000000000000000000000000000000000000000","name":"Mira","aliases":[],"description":"desc","relationships":[],"source_refs":[{"source_id":"s","start_unit_id":0,"end_unit_id":1}]}]}`, want: "start_unit_id"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := codec.Decode([]byte(test.raw))
if err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Decode() error = %v, want %q", err, test.want)
}
})
}
}
func TestCodecCandidatePreservesInvalidTypedValues(t *testing.T) {
codec := New()
candidate := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn", Aliases: []string{}, Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{}}}}
content, err := codec.EncodeCandidate(candidate)
if err != nil || !json.Valid(content) {
t.Fatalf("EncodeCandidate() = %s, %v; want JSON", content, err)
}
decoded, err := codec.DecodeCandidate(content)
if err != nil || !reflect.DeepEqual(decoded, candidate) {
t.Fatalf("DecodeCandidate() = %#v, %v; want %#v", decoded, err, candidate)
}
if _, err := codec.Encode(candidate); err == nil || !strings.Contains(err.Error(), ".id must match npc ID pattern") {
t.Fatalf("Encode() error = %v, want strict validation error", err)
}
}
func TestCodecRejectsEveryRequiredShapeBoundary(t *testing.T) {
base := validList().NPCs[0]
tests := []struct {
name string
value dnd.NPCList
want string
}{
{name: "nil aliases", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Description: base.Description, Relationships: []dnd.NPCRelationship{}, SourceRefs: base.SourceRefs}}}, want: "aliases must be present"},
{name: "empty alias", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{" "}, Description: base.Description, Relationships: []dnd.NPCRelationship{}, SourceRefs: base.SourceRefs}}}, want: "aliases[0] must not be empty"},
{name: "nil relationships", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{}, Description: base.Description, SourceRefs: base.SourceRefs}}}, want: "relationships must be present"},
{name: "empty relationship target", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{}, Description: base.Description, Relationships: []dnd.NPCRelationship{{Relationship: "knows"}}, SourceRefs: base.SourceRefs}}}, want: "target must not be empty"},
{name: "empty source refs", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{}, Description: base.Description, Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{}}}}, want: "source_refs must not be empty"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if _, err := New().Encode(test.value); err == nil || !strings.Contains(err.Error(), test.want) {
t.Fatalf("Encode() error = %v, want %q", err, test.want)
}
})
}
}
func TestCodecSchemaAndMetadataAreDefensive(t *testing.T) {
codec := New()
first := codec.Schema()
first.JSONSchema[0] = '['
second := codec.Schema()
if !json.Valid(second.JSONSchema) || second.JSONSchema[0] == '[' {
t.Fatal("Schema() returned shared bytes")
}
metadata := codec.Metadata(validList())
metadata["other"] = true
if next := codec.Metadata(validList()); len(next) != 1 || next["npc_count"] != 1 {
t.Fatalf("Metadata() = %#v, want only npc_count", next)
}
}

View File

@@ -0,0 +1,12 @@
{
"npcs": [
{
"id": "npc:sha256:99a16589618a04f535a7d21fdcc71a0b1c05d22f752cd492065b1086d97bc3d7",
"name": "Mira Thorn",
"aliases": ["The Greencloak"],
"description": "A guarded ranger who watches the northern road.",
"relationships": [{"target": "Captain Vale", "relationship": "reports to"}],
"source_refs": [{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 2}]
}
]
}

View File

@@ -0,0 +1,6 @@
package npcs
import "embed"
//go:embed assets/schemas/dnd_npcs_llm.v1.json assets/prompts/*.yaml assets/prompts/*.md
var embeddedAssets embed.FS

View File

@@ -0,0 +1,36 @@
id: dnd.npcs
version: "v1"
default_profile: gemini-2-flash
inputs:
- name: transcript
required: true
content_type: application/json
- name: players
required: false
content_type: text/plain
- name: party
required: false
content_type: text/plain
- name: glossary
required: false
content_type: text/plain
messages:
- role: system
content_file: ./sharedassets/common-dnd-system.md
- role: user
content_file: ./sharedassets/common-dnd-transcript.md
cache_control:
type: ephemeral
- role: user
content_file: ./sharedassets/common-dnd-references.md
cache_control:
type: ephemeral
- role: user
content_file: ./task.md
- role: user
content_file: ./instructions.md
output:
format: json
validation_mode: json_schema
schema_path: dnd_npcs_llm.v1.json
repair_attempts: 0

View File

@@ -0,0 +1,25 @@
Return exactly one JSON object and no explanatory text.
For every NPC record, cite one or more transcript source-unit ranges that
collectively support the canonical name, every alias, the description, and
every relationship. Use integer start_unit_id and end_unit_id values from the
transcript. Do not provide source_id; the extractor assigns it automatically.
Use narrow ranges when evidence is not contiguous and do not bridge unrelated
conversation with a broad range.
A canonical name must be the most specific in-world identity supported by the
transcript. Never use a human player, transcript speaker, or GM name when an
associated in-world character or creature is identified. Player, party, and
glossary references may disambiguate identities, but they are not transcript
evidence and cannot establish that an NPC appeared, acted, or was discussed.
Do not return a person or entity mentioned only in those references.
Descriptions must be short session records, not biographies, statistics,
alignment, motivations, or lore inferred from general D&D knowledge. Do not
summarize every action or follow a participant through unrelated scenes.
Relationships must be stated or directly demonstrated by cited transcript
units, not inferred from game lore.
Return aliases and relationships as arrays, including empty arrays when there
are none. Return only NPC records supported by the transcript and preserve
observed display spelling.

View File

@@ -0,0 +1,16 @@
Extract a concise Dungeons & Dragons non-player-character registry from the
provided transcript.
Include an in-world non-PC participant when the transcript establishes that it
appears, acts, speaks, or is materially discussed and gives it a proper name,
a stable alias or title, or an individually useful distinguishing description.
Exclude human players, transcript speakers, the GM as an out-of-world person,
player characters identified by the player or party references, incidental or
hypothetical name drops, corrected transcription mistakes, characters mentioned
only by reference material, indistinguishable crowds or groups, and temporary
summoned creatures or spell effects without a persistent individual identity.
Return canonical in-world names rather than player names or transcript speaker
names. Keep each description concise and limited to facts established by the
transcript. Include only explicitly supported aliases and relationships.

View File

@@ -0,0 +1,77 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "notarius.dnd.npcs.llm",
"type": "object",
"additionalProperties": false,
"required": ["npcs"],
"properties": {
"npcs": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": [
"name",
"aliases",
"description",
"relationships",
"source_refs"
],
"properties": {
"name": {
"type": "string",
"minLength": 1
},
"aliases": {
"type": "array",
"items": {
"type": "string",
"minLength": 1
}
},
"description": {
"type": "string",
"minLength": 1
},
"relationships": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["target", "relationship"],
"properties": {
"target": {
"type": "string",
"minLength": 1
},
"relationship": {
"type": "string",
"minLength": 1
}
}
}
},
"source_refs": {
"type": "array",
"minItems": 1,
"items": {
"type": "object",
"additionalProperties": false,
"required": ["start_unit_id", "end_unit_id"],
"properties": {
"start_unit_id": {
"type": "integer",
"minimum": 1
},
"end_unit_id": {
"type": "integer",
"minimum": 1
}
}
}
}
}
}
}
}
}

View File

@@ -0,0 +1,153 @@
package npcs
import (
"sort"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
func canonicalizeResponse(response *extractionResponse, doc *source.SourceDocument) {
if response == nil {
return
}
for index := range response.NPCs {
canonicalizeNPC(&response.NPCs[index])
}
sort.SliceStable(response.NPCs, func(i, j int) bool {
left, leftOK := earliestSourceUnit(doc, response.NPCs[i])
right, rightOK := earliestSourceUnit(doc, response.NPCs[j])
if leftOK != rightOK {
return leftOK
}
if !leftOK {
return false
}
return left < right
})
}
func canonicalizeNPC(npc *npcResponse) {
if npc == nil {
return
}
for index := range npc.SourceRefs {
npc.SourceRefs[index].StartUnitID = canonicalUnitRef(npc.SourceRefs[index].StartUnitID)
npc.SourceRefs[index].EndUnitID = canonicalUnitRef(npc.SourceRefs[index].EndUnitID)
}
sort.SliceStable(npc.SourceRefs, func(i, j int) bool {
left := npc.SourceRefs[i]
right := npc.SourceRefs[j]
if unitSortValue(left.StartUnitID) != unitSortValue(right.StartUnitID) {
return unitSortValue(left.StartUnitID) < unitSortValue(right.StartUnitID)
}
return unitSortValue(left.EndUnitID) < unitSortValue(right.EndUnitID)
})
npc.SourceRefs = dedupeSourceRefs(npc.SourceRefs)
}
func canonicalUnitRef(ref shared.UnitRef) shared.UnitRef {
value := ref.Int()
if value <= 0 {
return ref
}
return shared.UnitRefFromInt(value)
}
func dedupeSourceRefs(refs []npcSourceRefResponse) []npcSourceRefResponse {
if len(refs) < 2 {
return refs
}
out := refs[:0]
var previous npcSourceRefResponse
for index, ref := range refs {
if index > 0 && sameSourceRef(previous, ref) {
continue
}
out = append(out, ref)
previous = ref
}
return out
}
func sameSourceRef(left npcSourceRefResponse, right npcSourceRefResponse) bool {
return left.StartUnitID.Int() == right.StartUnitID.Int() &&
left.EndUnitID.Int() == right.EndUnitID.Int()
}
func earliestSourceUnit(doc *source.SourceDocument, npc npcResponse) (int, bool) {
for _, ref := range npc.SourceRefs {
start := ref.StartUnitID.Int()
end := ref.EndUnitID.Int()
if start > 0 && end > 0 {
startIndex, startOK := source.UnitIndex(doc, start)
endIndex, endOK := source.UnitIndex(doc, end)
if !startOK || !endOK || startIndex > endIndex {
continue
}
return start, true
}
}
return 0, false
}
func unitSortValue(ref shared.UnitRef) int {
value := ref.Int()
if value <= 0 {
return int(^uint(0) >> 1)
}
return value
}
func canonicalNPCList(response extractionResponse, sourceID string) dnd.NPCList {
if response.NPCs == nil {
return dnd.NPCList{NPCs: nil}
}
npcs := make([]dnd.NPC, len(response.NPCs))
for index, npc := range response.NPCs {
npcs[index] = dnd.NPC{
ID: identity.DeriveID(npc.Name),
Name: npc.Name,
Aliases: cloneStrings(npc.Aliases),
Description: npc.Description,
Relationships: cloneRelationships(npc.Relationships),
SourceRefs: canonicalSourceRefs(npc.SourceRefs, sourceID),
}
}
return dnd.NPCList{NPCs: npcs}
}
func cloneStrings(values []string) []string {
if values == nil {
return nil
}
return append([]string{}, values...)
}
func cloneRelationships(values []npcRelationshipResponse) []dnd.NPCRelationship {
if values == nil {
return nil
}
out := make([]dnd.NPCRelationship, len(values))
for index, value := range values {
out[index] = dnd.NPCRelationship{Target: value.Target, Relationship: value.Relationship}
}
return out
}
func canonicalSourceRefs(values []npcSourceRefResponse, sourceID string) []source.SourceRef {
if values == nil {
return nil
}
out := make([]source.SourceRef, len(values))
for index, value := range values {
out[index] = source.SourceRef{
SourceID: sourceID,
StartUnitID: value.StartUnitID.Int(),
EndUnitID: value.EndUnitID.Int(),
}
}
return out
}

View File

@@ -0,0 +1,200 @@
package npcs
import (
"bytes"
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
const Key = "dnd/npcs"
var requiredCapabilities = []string{
"chunks",
"source.transcript",
}
var providedCapabilities = []string{
"dnd.npcs",
}
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
Glossary: "Optional campaign glossary reference material used only for NPC disambiguation.",
Party: "Optional party roster reference material used only for NPC disambiguation.",
Players: "Optional player list reference material used only for NPC disambiguation.",
Roster: "Deprecated alias for party roster reference material used only for NPC disambiguation.",
}
func referenceSlots() []contracts.ReferenceSlot {
return shared.ReferenceSlots(referenceSlotDescriptions)
}
var _ contracts.Extractor[dnd.NPCList] = (*Extractor)(nil)
var _ contracts.ManifestMetadataProvider = (*Extractor)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Extractor)(nil)
type Options struct{}
type Extractor struct {
llm contracts.StructuredLLMClient
promptSHA string
responseSchemaSHA string
}
func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contracts.ReferenceSet) (*Extractor, error) {
if llmClient == nil {
return nil, extractorErrorf("LLM client must not be nil")
}
if len(references) > 1 {
return nil, extractorErrorf("at most one reference set may be supplied")
}
promptSHA, err := scriptoriumPromptMetadata()
if err != nil {
return nil, extractorErrorf("load prompt metadata: %w", err)
}
responseSchema, err := loadResponseSchema()
if err != nil {
return nil, extractorErrorf("load response schema: %w", err)
}
return &Extractor{
llm: llmClient,
promptSHA: promptSHA,
responseSchemaSHA: responseSchema.SHA256,
}, nil
}
func (e *Extractor) Key() string { return Key }
func (e *Extractor) ReferenceSlots() []contracts.ReferenceSlot { return referenceSlots() }
func (e *Extractor) ManifestMetadata() map[string]any {
if e == nil {
return nil
}
return map[string]any{
"prompt_id": PromptID,
"prompt_version": SchemaVersion,
"prompt_sha256": e.promptSHA,
"response_schema_key": string(ResponseSchemaKey),
"response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName,
"response_schema_version": SchemaVersion,
"response_schema_sha256": e.responseSchemaSHA,
"identity_policy": identity.Policy,
}
}
func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
if e == nil {
return nil
}
return []pipeline.CheckpointFingerprint{
{Name: "prompt", Value: e.promptSHA},
{Name: "response_schema", Value: e.responseSchemaSHA},
{Name: "identity_policy", Value: identity.Policy},
}
}
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.NPCList], error) {
if e == nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("extractor must not be nil")
}
if e.llm == nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("LLM client must not be nil")
}
if ctx == nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("context error before extraction: %w", err)
}
if req.Source == nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("source must not be nil")
}
if req.Chunk == nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("chunk must not be nil")
}
if len(req.Chunk.Units) == 0 {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("chunk %q units must not be empty", req.Chunk.ID)
}
sourceInput, err := chunkSourceInput(req)
if err != nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, err
}
var response extractionResponse
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,
PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile,
SessionID: req.SessionID,
Inputs: shared.PromptInputs(sourceInput, req.References),
}, &response); err != nil {
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("complete structured output: %w", err)
}
canonicalizeResponse(&response, req.Source)
return contracts.TypedExtractionResult[dnd.NPCList]{Value: canonicalNPCList(response, req.Source.ID)}, nil
}
func chunkSourceInput(req contracts.TypedExtractionRequest) (contracts.LLMInputMaterial, error) {
material := req.SourceInput.Clone()
if len(material.Content) == 0 {
material = contracts.NewLLMInputMaterial("source", req.Chunk.MediaType, req.Chunk.Content, "", "")
}
if !bytes.Equal(material.Content, req.Chunk.Content) {
return contracts.LLMInputMaterial{}, extractorErrorf("source input must match chunk %q content", req.Chunk.ID)
}
if material.Name == "" {
material.Name = "source"
}
if material.MediaType == "" {
material.MediaType = req.Chunk.MediaType
}
if material.SizeBytes == 0 {
material.SizeBytes = int64(len(material.Content))
}
return material, nil
}
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCListKind,
ReferenceSlots: referenceSlots(),
}
}
func Register(registry *pipeline.ExtractorRegistry) error {
return pipeline.RegisterExtractorBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Extractor[dnd.NPCList], error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(request.Dependencies.LLM, options, request.References)
})
}
func validateOptions(options map[string]any) error {
_, err := DecodeOptions(options)
return err
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, extractorErrorf("%w", err)
}
return Options{}, nil
}
func extractorErrorf(format string, args ...any) error {
return fmt.Errorf("dnd npcs extractor: "+format, args...)
}

View File

@@ -0,0 +1,110 @@
package npcs
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
func TestExtractReturnsCanonicalNPCListFromPrivateResponse(t *testing.T) {
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{
{
Name: "Captain Vale", Aliases: []string{"The Captain"}, Description: "A road captain.",
Relationships: []npcRelationshipResponse{{Target: "Mira Thorn", Relationship: "reports to"}},
SourceRefs: responseSourceRefs(3, 3),
},
{
Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A guarded ranger.",
Relationships: []npcRelationshipResponse{{Target: "Captain Vale", Relationship: "commands"}},
SourceRefs: []npcSourceRefResponse{
{StartUnitID: sharedUnitRef(2), EndUnitID: sharedUnitRef(2)},
{StartUnitID: sharedUnitRef(1), EndUnitID: sharedUnitRef(2)},
{StartUnitID: sharedUnitRef(1), EndUnitID: sharedUnitRef(2)},
},
},
}}}
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
want := dnd.NPCList{NPCs: []dnd.NPC{
{ID: identity.DeriveID("Mira Thorn"), Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A guarded ranger.", Relationships: []dnd.NPCRelationship{{Target: "Captain Vale", Relationship: "commands"}}, SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}, {SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}},
{ID: identity.DeriveID("Captain Vale"), Name: "Captain Vale", Aliases: []string{"The Captain"}, Description: "A road captain.", Relationships: []dnd.NPCRelationship{{Target: "Mira Thorn", Relationship: "reports to"}}, SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 3}}},
}}
if !reflect.DeepEqual(result.Value, want) {
t.Fatalf("Value = %#v, want %#v", result.Value, want)
}
if len(client.requests) != 1 {
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
}
request := client.requests[0]
if request.StageName != Key || request.PromptID != PromptID || request.PromptVersion != SchemaVersion || request.ProfileID != "profile-npcs" || request.SessionID != "session-123" {
t.Fatalf("LLM request identity = %#v", request)
}
transcript := request.Inputs["transcript"]
if transcript.Name != "transcript" || transcript.MediaType != "application/json" || transcript.Digest != "sha256:chunk" || transcript.OriginURI != "file:///session-alpha.json" || string(transcript.Content) != string(extractionRequest().Chunk.Content) {
t.Fatalf("transcript input = %#v, want chunk-scoped material", transcript)
}
}
func TestExtractPassesCampaignReferencesAsPromptInputs(t *testing.T) {
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{}}}
req := extractionRequest()
req.References = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
"players": {Slot: contracts.ReferenceSlot{Name: "players"}, Items: []contracts.ReferenceItem{{SlotName: "players", Content: []byte("Dana: Mira")}}},
"party": {Slot: contracts.ReferenceSlot{Name: "party"}, Items: []contracts.ReferenceItem{{SlotName: "party", Content: []byte("Mira: ranger")}}},
"glossary": {Slot: contracts.ReferenceSlot{Name: "glossary"}, Items: []contracts.ReferenceItem{{SlotName: "glossary", Content: []byte("Greencloak: local title")}}},
}}
if _, err := newExtractor(t, client).Extract(context.Background(), req); err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
inputs := client.requests[0].Inputs
if string(inputs["players"].Content) != "Dana: Mira" || string(inputs["party"].Content) != "Mira: ranger" || string(inputs["glossary"].Content) != "Greencloak: local title" {
t.Fatalf("reference inputs = %#v", inputs)
}
if strings.Contains(string(inputs["transcript"].Content), "local title") {
t.Fatal("transcript input contains reference content")
}
}
func TestExtractPreservesMalformedCandidatesForValidators(t *testing.T) {
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{{
Name: "", Aliases: nil, Description: "", Relationships: nil,
SourceRefs: []npcSourceRefResponse{{StartUnitID: sharedUnitRef(99), EndUnitID: shared.UnitRefFromString("missing")}, {StartUnitID: sharedUnitRef(99), EndUnitID: shared.UnitRefFromInt(0)}},
}}}}
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
if err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
if len(result.Value.NPCs) != 1 || result.Value.NPCs[0].ID != "" || result.Value.NPCs[0].Name != "" || result.Value.NPCs[0].Aliases != nil || result.Value.NPCs[0].Relationships != nil {
t.Fatalf("malformed candidate = %#v, want invalid values preserved", result.Value)
}
if refs := result.Value.NPCs[0].SourceRefs; len(refs) != 1 || refs[0].SourceID != "session-alpha" || refs[0].StartUnitID != 99 || refs[0].EndUnitID != 0 {
t.Fatalf("malformed source refs = %#v, want invalid range preserved after exact deduplication", refs)
}
}
func TestExtractHandlesCancellationAndProviderErrors(t *testing.T) {
request := extractionRequest()
extractor := newExtractor(t, &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{}}})
canceled, cancel := context.WithCancel(context.Background())
cancel()
if _, err := extractor.Extract(canceled, request); err == nil || !strings.Contains(err.Error(), "context") {
t.Fatalf("canceled Extract() error = %v, want context error", err)
}
_, err := newExtractor(t, &fakeNPCsLLMClient{err: errors.New("provider unavailable")}).Extract(context.Background(), request)
if err == nil || !strings.Contains(err.Error(), "dnd npcs") || !strings.Contains(err.Error(), "provider unavailable") {
t.Fatalf("provider Extract() error = %v, want contextual provider error", err)
}
}
func sharedUnitRef(value int) shared.UnitRef { return shared.UnitRefFromInt(value) }

View File

@@ -0,0 +1,25 @@
package npcs
import "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
type extractionResponse struct {
NPCs []npcResponse `json:"npcs"`
}
type npcResponse struct {
Name string `json:"name"`
Aliases []string `json:"aliases"`
Description string `json:"description"`
Relationships []npcRelationshipResponse `json:"relationships"`
SourceRefs []npcSourceRefResponse `json:"source_refs"`
}
type npcRelationshipResponse struct {
Target string `json:"target"`
Relationship string `json:"relationship"`
}
type npcSourceRefResponse struct {
StartUnitID shared.UnitRef `json:"start_unit_id"`
EndUnitID shared.UnitRef `json:"end_unit_id"`
}

View File

@@ -0,0 +1,100 @@
package npcs
import (
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
t.Fatalf("New(nil) error = %v, want dependency error", err)
}
if _, err := New(&fakeNPCsLLMClient{}, Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one reference set") {
t.Fatalf("New() error = %v, want reference-set error", err)
}
if got := newExtractor(t, &fakeNPCsLLMClient{}).Key(); got != Key {
t.Fatalf("Key() = %q, want %q", got, Key)
}
}
func TestModuleSpecAndReferenceSlots(t *testing.T) {
got := ModuleSpec()
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageExtract,
Requires: []string{"chunks", "source.transcript"},
Provides: []string{"dnd.npcs"},
ArtifactKind: dnd.NPCListKind,
ReferenceSlots: []contracts.ReferenceSlot{
{Name: "glossary", Description: "Optional campaign glossary reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "party", Description: "Optional party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "players", Description: "Optional player list reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
{Name: "roster", Description: "Deprecated alias for party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
got.Requires[0] = "changed"
got.Provides[0] = "changed"
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
if !reflect.DeepEqual(ModuleSpec(), want) {
t.Fatal("ModuleSpec() returned mutable shared slices")
}
extractor := newExtractor(t, &fakeNPCsLLMClient{})
if !reflect.DeepEqual(extractor.ReferenceSlots(), want.ReferenceSlots) {
t.Fatalf("ReferenceSlots() = %#v, want %#v", extractor.ReferenceSlots(), want.ReferenceSlots)
}
}
func TestRegisterStoresTypedModuleSpec(t *testing.T) {
registry := pipeline.NewExtractorRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
got, ok := registry.Spec(Key)
if !ok || !reflect.DeepEqual(got, ModuleSpec()) {
t.Fatalf("registry.Spec(%q) = %#v, present = %t", Key, got, ok)
}
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "extractor registry") {
t.Fatalf("Register(nil) error = %v, want registry error", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("DecodeOptions() error = %v, want strict options error", err)
}
}
func TestExtractorMetadataAndCheckpointIdentity(t *testing.T) {
extractor := newExtractor(t, &fakeNPCsLLMClient{})
metadata := extractor.ManifestMetadata()
for key, want := range map[string]string{
"prompt_id": PromptID, "prompt_version": SchemaVersion,
"response_schema_key": string(ResponseSchemaKey), "response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName, "response_schema_version": SchemaVersion,
"identity_policy": "dnd.npcs.identity.v1",
} {
if metadata[key] != want {
t.Fatalf("metadata[%q] = %#v, want %q", key, metadata[key], want)
}
}
for _, key := range []string{"prompt_sha256", "response_schema_sha256"} {
if value, ok := metadata[key].(string); !ok || !strings.HasPrefix(value, "sha256:") {
t.Fatalf("metadata[%q] = %#v, want hash", key, metadata[key])
}
}
fingerprints := extractor.CheckpointFingerprints()
want := map[string]string{"prompt": metadata["prompt_sha256"].(string), "response_schema": metadata["response_schema_sha256"].(string), "identity_policy": "dnd.npcs.identity.v1"}
if len(fingerprints) != len(want) {
t.Fatalf("CheckpointFingerprints() = %#v, want %d entries", fingerprints, len(want))
}
for _, fingerprint := range fingerprints {
if fingerprint.Value != want[fingerprint.Name] {
t.Fatalf("fingerprint %q = %q, want %#v", fingerprint.Name, fingerprint.Value, want)
}
}
}

View File

@@ -0,0 +1,21 @@
package npcs
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
const (
PromptID = "dnd.npcs"
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_npcs_llm")
ResponseSchemaID = "notarius.dnd.npcs.llm"
ResponseSchemaName = "notarius_dnd_npcs_llm_v1"
SchemaVersion = "v1"
)
func loadResponseSchema() (llm.ResponseSchema, error) {
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
Key: ResponseSchemaKey,
ID: ResponseSchemaID,
Version: SchemaVersion,
Name: ResponseSchemaName,
AssetPath: "assets/schemas/dnd_npcs_llm.v1.json",
})
}

View File

@@ -0,0 +1,78 @@
package npcs
import (
"bytes"
"encoding/json"
"strings"
"testing"
"github.com/santhosh-tekuri/jsonschema/v6"
)
func TestLoadResponseSchemaUsesPrivateNPCSchema(t *testing.T) {
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
}
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Version != SchemaVersion || schema.Name != ResponseSchemaName || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
t.Fatalf("schema = %#v, want private NPC schema identity", schema)
}
valid := map[string]any{"npcs": []any{map[string]any{
"name": "Mira Thorn", "aliases": []any{}, "description": "A ranger.", "relationships": []any{},
"source_refs": []any{map[string]any{"start_unit_id": 1, "end_unit_id": 2}},
}}}
validJSON, err := json.Marshal(valid)
if err != nil {
t.Fatal(err)
}
if err := validateJSONSchema(validJSON, schema.JSONSchema); err != nil {
t.Fatalf("valid private NPC response rejected: %v", err)
}
withID := map[string]any{"npcs": []any{map[string]any{
"name": "Mira Thorn", "id": "assigned-later", "aliases": []any{}, "description": "A ranger.", "relationships": []any{},
"source_refs": []any{map[string]any{"start_unit_id": 1, "end_unit_id": 2}},
}}}
withIDJSON, err := json.Marshal(withID)
if err != nil {
t.Fatal(err)
}
if err := validateJSONSchema(withIDJSON, schema.JSONSchema); err == nil {
t.Fatal("private schema accepted framework-assigned id")
}
}
func TestResponseSchemaIsMutationSafeAndDiagnosticsRedactContent(t *testing.T) {
first, err := loadResponseSchema()
if err != nil {
t.Fatal(err)
}
first.JSONSchema[0] = '['
second, err := loadResponseSchema()
if err != nil || !json.Valid(second.JSONSchema) || bytes.Equal(first.JSONSchema, second.JSONSchema) {
t.Fatalf("second schema = %s, %v; want defensive copy", second.JSONSchema, err)
}
diagnostics := second.DiagnosticsMap()
if _, ok := diagnostics["json_schema"]; ok {
t.Fatalf("schema diagnostics included raw content: %#v", diagnostics)
}
}
func validateJSONSchema(instanceContent, schemaContent []byte) error {
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(instanceContent))
if err != nil {
return err
}
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
if err != nil {
return err
}
compiler := jsonschema.NewCompiler()
if err := compiler.AddResource("schema.json", schemaDocument); err != nil {
return err
}
compiled, err := compiler.Compile("schema.json")
if err != nil {
return err
}
return compiled.Validate(instance)
}

View File

@@ -0,0 +1,45 @@
package npcs
import (
"fmt"
"sync"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
const scriptoriumPromptRoot = "assets/prompts"
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
promptFS, err := shared.ModulePromptFS("dnd.npcs", embeddedAssets, []promptfs.ModulePromptFile{
{Name: "dnd.npcs.yaml", Path: "assets/prompts/dnd.npcs.yaml"},
{Name: "task.md", Path: "assets/prompts/task.md"},
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
})
if err != nil {
return fmt.Errorf("prepare NPC prompt assets: %w", err)
}
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
return err
}
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
}
func scriptoriumPromptMetadata() (string, error) {
scriptoriumPromptHashOnce.Do(func() {
parts := append([]llm.AssetHashPart{
{FS: embeddedAssets, Path: "assets/prompts/dnd.npcs.yaml"},
{FS: embeddedAssets, Path: "assets/prompts/task.md"},
{FS: embeddedAssets, Path: "assets/prompts/instructions.md"},
}, append(shared.CommonHashParts(), shared.ReferenceHashParts()...)...)
scriptoriumPromptHash, scriptoriumPromptHashErr = llm.HashAssets(parts)
})
return scriptoriumPromptHash, scriptoriumPromptHashErr
}
var (
scriptoriumPromptHashOnce sync.Once
scriptoriumPromptHash string
scriptoriumPromptHashErr error
)

View File

@@ -0,0 +1,68 @@
package npcs
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/scriptorium"
)
func TestRegisterPromptAssetsAndPrepareNPCPrompt(t *testing.T) {
registry := llm.NewAssetRegistry()
if err := RegisterPromptAssets(registry); err != nil {
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
}
options, err := registry.ScriptoriumOptions()
if err != nil {
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
}
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
ID: "npc-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "npc-test-model",
})))
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
if err != nil {
t.Fatalf("NewEngine() error = %v, want nil", err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "npc-test-profile",
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"units":[1]}`),
"players": scriptorium.Inline("Dana: Mira"),
"party": scriptorium.Inline("Mira: ranger"),
"glossary": scriptorium.Inline("Greencloak: title"),
},
})
if err != nil {
t.Fatalf("Prepare() error = %v, want nil", err)
}
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_npcs_llm.v1.json" || len(prepared.Messages) != 5 {
t.Fatalf("prepared prompt = %#v, want NPC prompt identity and wiring", prepared)
}
if !strings.Contains(prepared.Messages[1].Content, `{"units":[1]}`) || !strings.Contains(prepared.Messages[2].Content, "Dana: Mira") || !strings.Contains(prepared.Messages[2].Content, "Mira: ranger") {
t.Fatalf("prepared prompt inputs do not include transcript/references")
}
if strings.Contains(prepared.Messages[3].Content, `{"units":[1]}`) || strings.Contains(prepared.Messages[4].Content, `{"units":[1]}`) {
t.Fatal("task or instruction prompt leaked raw transcript")
}
}
func TestPromptMetadataAndDiagnosticsDoNotContainRawAssets(t *testing.T) {
hash, err := scriptoriumPromptMetadata()
if err != nil || !strings.HasPrefix(hash, "sha256:") {
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want hash", hash, err)
}
metadata := newExtractor(t, &fakeNPCsLLMClient{}).ManifestMetadata()
payload, err := json.Marshal(metadata)
if err != nil {
t.Fatal(err)
}
for _, forbidden := range []string{"Include an in-world", "common-dnd-system", "source-unit", "dnd_npcs_llm.v1.json"} {
if strings.Contains(string(payload), forbidden) {
t.Fatalf("metadata leaked raw prompt/schema content %q: %s", forbidden, payload)
}
}
}

View File

@@ -0,0 +1,112 @@
package npcs
import (
"context"
"encoding/json"
"errors"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
func extractionRequest() contracts.TypedExtractionRequest {
doc := sourceDocument()
chunk := &source.Chunk{
ID: "session-alpha:chunk:0",
SourceID: doc.ID,
Index: 0,
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 3},
Content: []byte(`{"units":[1,2,3]}`),
MediaType: "application/json",
Units: append([]source.SourceUnit(nil), doc.Units...),
Metadata: map[string]any{"ignored": "chunk metadata"},
}
return contracts.TypedExtractionRequest{
Source: doc,
Chunk: chunk,
SourceInput: contracts.NewLLMInputMaterial("source", chunk.MediaType, chunk.Content, "sha256:chunk", "file:///session-alpha.json"),
SessionID: "session-123",
LLMProfile: "profile-npcs",
}
}
func sourceDocument() *source.SourceDocument {
return &source.SourceDocument{
ID: "session-alpha",
Kind: "transcript",
Format: "application/vnd.seriatim.minimal+json",
Digest: "sha256:test",
Units: []source.SourceUnit{
{ID: 1, Kind: "transcript_segment", Text: "Mira Thorn greets the party."},
{ID: 2, Kind: "transcript_segment", Text: "The Greencloak watches the northern road."},
{ID: 3, Kind: "transcript_segment", Text: "Captain Vale reports to Mira Thorn."},
},
}
}
func responseSourceRefs(startUnitID, endUnitID int) []npcSourceRefResponse {
return []npcSourceRefResponse{{StartUnitID: shared.UnitRefFromInt(startUnitID), EndUnitID: shared.UnitRefFromInt(endUnitID)}}
}
func newExtractor(t *testing.T, client contracts.StructuredLLMClient, references ...contracts.ReferenceSet) *Extractor {
t.Helper()
extractor, err := New(client, Options{}, references...)
if err != nil {
t.Fatalf("New() error = %v, want nil", err)
}
return extractor
}
func emptyChunkRequest(req contracts.TypedExtractionRequest) contracts.TypedExtractionRequest {
req.Chunk = &source.Chunk{ID: req.Chunk.ID, SourceID: req.Chunk.SourceID, Index: req.Chunk.Index}
return req
}
func mismatchedSourceInputRequest(req contracts.TypedExtractionRequest) contracts.TypedExtractionRequest {
req.SourceInput = contracts.NewLLMInputMaterial("source", "application/json", []byte(`{"different":true}`), "sha256:other", "file:///other.json")
return req
}
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
req.Inputs = req.Inputs.Clone()
if len(req.Vars) == 0 {
req.Vars = nil
return req
}
vars := make(map[string]any, len(req.Vars))
for key, value := range req.Vars {
vars[key] = value
}
req.Vars = vars
return req
}
type fakeNPCsLLMClient struct {
response extractionResponse
content []byte
err error
requests []contracts.StructuredCompletionRequest
}
func (client *fakeNPCsLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
client.requests = append(client.requests, cloneStructuredCompletionRequest(req))
if client.err != nil {
return contracts.StructuredCompletionResponse{}, client.err
}
target, ok := out.(*extractionResponse)
if !ok {
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
}
*target = client.response
content := append([]byte(nil), client.content...)
if len(content) == 0 {
var err error
content, err = json.Marshal(client.response)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
}
return contracts.StructuredCompletionResponse{Content: content}, nil
}

View File

@@ -17,6 +17,9 @@ inputs:
- name: glossary
required: false
content_type: text/plain
- name: npcs
required: false
content_type: application/json
messages:
- role: system
content_file: ./sharedassets/common-dnd-system.md
@@ -30,6 +33,8 @@ messages:
type: ephemeral
- role: user
content_file: ./catalog.md
- role: user
content_file: ./npc_registry.md
- role: user
content_file: ./task.md
- role: user

View File

@@ -0,0 +1,11 @@
The optional canonical NPC registry for this extraction is provided below as
durable JSON. Use it only to prefer exact canonical NPC names and recognize
their aliases when the transcript identifies a caster.
Registry entries are grounding material, not evidence that a spell was cast.
Do not extract a spell, caster, effect, or source reference from the registry.
NPC source references describe registry provenance and may belong to another
session; they are never spell evidence. Preserve the existing player and party
policy for identifying PCs from the transcript.
{{ input "npcs" }}

View File

@@ -4,6 +4,7 @@ import (
"bytes"
"context"
"fmt"
"sort"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
@@ -34,12 +35,19 @@ var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
func referenceSlots() []contracts.ReferenceSlot {
slots := shared.ReferenceSlots(referenceSlotDescriptions)
return append(slots, contracts.ReferenceSlot{
slots = append(slots, contracts.ReferenceSlot{
Name: spellcatalog.SpellCatalogReferenceSlot,
Description: "Optional canonical spell-name catalog used for extraction grounding.",
AcceptedMediaTypes: []string{"application/json"},
MaxBytes: 1048576,
}, contracts.ReferenceSlot{
Name: NPCRegistryReferenceSlot,
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
AcceptedMediaTypes: []string{"application/json"},
MaxBytes: NPCRegistryMaxBytes,
})
sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name })
return slots
}
var _ contracts.Extractor[dnd.SpellList] = (*Extractor)(nil)
@@ -51,6 +59,7 @@ type Extractor struct {
llm contracts.StructuredLLMClient
effectiveCatalog spellcatalog.EffectiveCatalog
catalogPromptInput contracts.LLMInputMaterial
npcRegistry npcRegistryPromptInput
promptSHA string
responseSchemaSHA string
}
@@ -74,6 +83,10 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
if err != nil {
return nil, extractorErrorf("prepare spell catalog prompt input: %w", err)
}
npcRegistry, err := resolveNPCRegistry(referenceSet)
if err != nil {
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
}
promptSHA, err := scriptoriumPromptMetadata()
if err != nil {
return nil, extractorErrorf("load prompt metadata: %w", err)
@@ -86,6 +99,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
llm: llmClient,
effectiveCatalog: effectiveCatalog,
catalogPromptInput: catalogPromptInput,
npcRegistry: npcRegistry,
promptSHA: promptSHA,
responseSchemaSHA: responseSchema.SHA256,
}, nil
@@ -113,6 +127,10 @@ func (e *Extractor) ManifestMetadata() map[string]any {
"response_schema_version": SchemaVersion,
"response_schema_sha256": e.responseSchemaSHA,
}
if e.npcRegistry.bound {
metadata["npc_registry_digest"] = e.npcRegistry.digest
metadata["npc_count"] = e.npcRegistry.count
}
return metadata
}
@@ -120,11 +138,15 @@ func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
if e == nil {
return nil
}
return []pipeline.CheckpointFingerprint{
fingerprints := []pipeline.CheckpointFingerprint{
{Name: "effective_catalog", Value: e.effectiveCatalog.Digest()},
{Name: "prompt", Value: e.promptSHA},
{Name: "response_schema", Value: e.responseSchemaSHA},
}
if e.npcRegistry.bound {
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: e.npcRegistry.digest})
}
return fingerprints
}
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.SpellList], error) {
@@ -157,6 +179,7 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
var response extractionResponse
inputs := shared.PromptInputs(sourceInput, req.References)
inputs[spellcatalog.SpellCatalogReferenceSlot] = e.catalogPromptInput.Clone()
inputs[NPCRegistryReferenceSlot] = e.npcRegistry.input.Clone()
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key,
PromptID: PromptID,

View File

@@ -0,0 +1,89 @@
package spells
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"mime"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
const (
NPCRegistryReferenceSlot = "npcs"
NPCRegistryMaxBytes = 1048576
)
type npcRegistryPromptInput struct {
input contracts.LLMInputMaterial
digest string
count int
bound bool
}
func resolveNPCRegistry(references contracts.ReferenceSet) (npcRegistryPromptInput, error) {
slot, ok := references.Slots[NPCRegistryReferenceSlot]
if !ok {
return npcRegistryPromptInput{
input: contracts.NewLLMInputMaterial(
NPCRegistryReferenceSlot,
"application/json",
[]byte(`{"npcs":[]}`),
"",
"",
),
}, nil
}
if len(slot.Items) != 1 {
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q must contain exactly one item", NPCRegistryReferenceSlot)
}
item := slot.Items[0]
mediaType, _, err := mime.ParseMediaType(item.MediaType)
if err != nil {
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q item media type %q is invalid: %w", NPCRegistryReferenceSlot, item.MediaType, err)
}
if !strings.EqualFold(mediaType, "application/json") {
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q item media type %q must be application/json", NPCRegistryReferenceSlot, item.MediaType)
}
if len(item.Content) > NPCRegistryMaxBytes {
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q item is %d bytes, limit %d", NPCRegistryReferenceSlot, len(item.Content), NPCRegistryMaxBytes)
}
codec := npccodec.New()
value, err := codec.Decode(item.Content)
if err != nil {
return npcRegistryPromptInput{}, fmt.Errorf("decode NPC registry: %w", err)
}
if issues := identity.ValidateList(value); len(issues) > 0 {
return npcRegistryPromptInput{}, fmt.Errorf("validate NPC registry identity: %s", formatNPCIdentityIssues(issues))
}
content, err := codec.Encode(value)
if err != nil {
return npcRegistryPromptInput{}, fmt.Errorf("encode canonical NPC registry: %w", err)
}
digest := semanticNPCRegistryDigest(content)
return npcRegistryPromptInput{
input: contracts.NewLLMInputMaterial(NPCRegistryReferenceSlot, "application/json", content, digest, ""),
digest: digest,
count: len(value.NPCs),
bound: true,
}, nil
}
func semanticNPCRegistryDigest(content []byte) string {
sum := sha256.Sum256(content)
return "sha256:" + hex.EncodeToString(sum[:])
}
func formatNPCIdentityIssues(issues []identity.Issue) string {
parts := make([]string, len(issues))
for index, issue := range issues {
parts[index] = fmt.Sprintf("%s at record %d", issue.Code, issue.RecordIndex)
}
return strings.Join(parts, ", ")
}

View File

@@ -0,0 +1,236 @@
package spells
import (
"bytes"
"context"
"encoding/json"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
func TestResolveNPCRegistryUsesExactEmptyPromptWhenUnbound(t *testing.T) {
resolved, err := resolveNPCRegistry(contracts.ReferenceSet{})
if err != nil {
t.Fatalf("resolveNPCRegistry() error = %v, want nil", err)
}
if resolved.bound || resolved.digest != "" || resolved.count != 0 {
t.Fatalf("resolved unbound registry = %#v, want no semantic metadata", resolved)
}
if resolved.input.Name != NPCRegistryReferenceSlot || resolved.input.MediaType != "application/json" || resolved.input.Digest != "" || resolved.input.OriginURI != "" {
t.Fatalf("unbound prompt input metadata = %#v, want name/media type only", resolved.input)
}
if got := string(resolved.input.Content); got != `{"npcs":[]}` {
t.Fatalf("unbound prompt input = %q, want exact empty registry", got)
}
if resolved.input.SizeBytes != int64(len(`{"npcs":[]}`)) {
t.Fatalf("unbound prompt input size = %d, want %d", resolved.input.SizeBytes, len(`{"npcs":[]}`))
}
if metadata := newExtractor(t, &fakeSpellsLLMClient{}).ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
t.Fatalf("unbound extractor metadata = %#v, want no NPC registry fields", metadata)
}
fingerprints := newExtractor(t, &fakeSpellsLLMClient{}).CheckpointFingerprints()
for _, fingerprint := range fingerprints {
if fingerprint.Name == "npc_registry" {
t.Fatalf("unbound checkpoint fingerprints = %#v, want no NPC registry fingerprint", fingerprints)
}
}
}
func TestResolveNPCRegistryCanonicalizesContentAndUsesSemanticDigest(t *testing.T) {
value := validNPCRegistryList()
canonical := encodeNPCRegistry(t, value)
raw := append([]byte(" \n"), canonical...)
raw = append(raw, []byte("\n ")...)
resolved, err := resolveNPCRegistry(npcRegistryReference(raw, "file:///another-session/npcs.json"))
if err != nil {
t.Fatalf("resolveNPCRegistry() error = %v, want nil", err)
}
if !resolved.bound || resolved.count != len(value.NPCs) {
t.Fatalf("resolved registry = %#v, want bound registry with %d NPC", resolved, len(value.NPCs))
}
if !bytes.Equal(resolved.input.Content, canonical) {
t.Fatalf("canonical prompt input = %s, want %s", resolved.input.Content, canonical)
}
if resolved.input.Digest != semanticNPCRegistryDigest(canonical) || resolved.digest != resolved.input.Digest {
t.Fatalf("semantic digest = %q/%q, want %q", resolved.input.Digest, resolved.digest, semanticNPCRegistryDigest(canonical))
}
if resolved.input.OriginURI != "" {
t.Fatalf("prompt input origin = %q, want no provenance path", resolved.input.OriginURI)
}
resolved.input.Content[0] = 'X'
again, err := resolveNPCRegistry(npcRegistryReference(raw, "file:///another-session/npcs.json"))
if err != nil {
t.Fatalf("second resolveNPCRegistry() error = %v, want nil", err)
}
if !bytes.Equal(again.input.Content, canonical) {
t.Fatalf("canonical content changed after caller mutation = %s, want %s", again.input.Content, canonical)
}
}
func TestResolveNPCRegistryRejectsInvalidBoundaryValues(t *testing.T) {
valid := validNPCRegistryList()
second := validNPCRegistryList().NPCs[0]
second.ID = identity.DeriveID("Captain Vale")
second.Name = "Captain Vale"
second.Aliases = []string{"The Greencloak"}
valueWithAliasCollision := dnd.NPCList{NPCs: []dnd.NPC{valid.NPCs[0], second}}
invalidID := valid
invalidID.NPCs[0].ID = "not-an-npc-id"
tests := []struct {
name string
reference contracts.ReferenceSet
wantError string
}{
{name: "zero items", reference: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{NPCRegistryReferenceSlot: {Items: []contracts.ReferenceItem{}}}}, wantError: "exactly one"},
{name: "multiple", reference: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{NPCRegistryReferenceSlot: {Items: []contracts.ReferenceItem{{Content: []byte(`{"npcs":[]}`)}, {Content: []byte(`{"npcs":[]}`)}}}}}, wantError: "exactly one"},
{name: "wrong media type", reference: npcRegistryReferenceWithMedia([]byte(`{"npcs":[]}`), "text/plain"), wantError: "must be application/json"},
{name: "malformed JSON", reference: npcRegistryReference([]byte(`{"npcs":[`), "file:///private.json"), wantError: "decode NPC registry"},
{name: "unknown field", reference: npcRegistryReference([]byte(`{"npcs":[],"unexpected":true}`), "file:///private.json"), wantError: "unknown field"},
{name: "invalid ID", reference: npcRegistryReference(marshalNPCRegistry(t, invalidID), "file:///private.json"), wantError: "decode NPC registry"},
{name: "alias collision", reference: npcRegistryReference(encodeNPCRegistry(t, valueWithAliasCollision), "file:///private.json"), wantError: string(identity.IssueAliasOwnershipCollision)},
{name: "byte limit", reference: npcRegistryReference(bytes.Repeat([]byte("x"), NPCRegistryMaxBytes+1), "file:///private.json"), wantError: "limit"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
_, err := resolveNPCRegistry(test.reference)
if err == nil || !strings.Contains(err.Error(), test.wantError) {
t.Fatalf("resolveNPCRegistry() error = %v, want %q", err, test.wantError)
}
if strings.Contains(err.Error(), "Mira Thorn") || strings.Contains(err.Error(), "The Greencloak") || strings.Contains(err.Error(), "private.json") {
t.Fatalf("error leaked registry content or provenance: %v", err)
}
})
}
}
func TestNPCRegistryFingerprintIsSemanticAndDefensive(t *testing.T) {
value := validNPCRegistryList()
canonical := encodeNPCRegistry(t, value)
pretty, err := json.MarshalIndent(value, "", " ")
if err != nil {
t.Fatalf("MarshalIndent() error = %v", err)
}
first := newExtractor(t, &fakeSpellsLLMClient{}, npcRegistryReference(canonical, "file:///one.json"))
second := newExtractor(t, &fakeSpellsLLMClient{}, npcRegistryReference(pretty, "file:///two.json"))
firstFingerprints := checkpointFingerprintMap(first.CheckpointFingerprints())
secondFingerprints := checkpointFingerprintMap(second.CheckpointFingerprints())
if firstFingerprints["npc_registry"] == "" || firstFingerprints["npc_registry"] != secondFingerprints["npc_registry"] {
t.Fatalf("semantic NPC fingerprints = %#v and %#v, want same npc_registry value", firstFingerprints, secondFingerprints)
}
returned := first.CheckpointFingerprints()
returned[0].Name = "caller-mutated"
if first.CheckpointFingerprints()[0].Name == "caller-mutated" {
t.Fatal("CheckpointFingerprints() returned caller-mutable slice state")
}
changed := validNPCRegistryList()
changed.NPCs[0].Description = "A different description."
changedFingerprint := checkpointFingerprintMap(newExtractor(t, &fakeSpellsLLMClient{}, npcRegistryReference(encodeNPCRegistry(t, changed), "file:///three.json")).CheckpointFingerprints())
if changedFingerprint["npc_registry"] == firstFingerprints["npc_registry"] {
t.Fatalf("semantic NPC fingerprint did not change: %#v", changedFingerprint)
}
metadata := first.ManifestMetadata()
encoded, err := json.Marshal(map[string]any{"metadata": metadata, "fingerprints": firstFingerprints})
if err != nil {
t.Fatalf("marshal metadata: %v", err)
}
for _, forbidden := range []string{"Mira Thorn", "The Greencloak", "another-session", "one.json"} {
if strings.Contains(string(encoded), forbidden) {
t.Fatalf("metadata or fingerprints leaked %q: %s", forbidden, encoded)
}
}
if metadata["npc_registry_digest"] != firstFingerprints["npc_registry"] || metadata["npc_count"] != 1 {
t.Fatalf("NPC registry metadata = %#v, want digest and count only", metadata)
}
}
func TestExtractPassesCanonicalNPCRegistryToLLMWithoutProvenance(t *testing.T) {
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
canonical := encodeNPCRegistry(t, validNPCRegistryList())
extractor := newExtractor(t, client, npcRegistryReference(append([]byte("\n"), canonical...), "file:///npc-session.json"))
if _, err := extractor.Extract(context.Background(), extractionRequest()); err != nil {
t.Fatalf("Extract() error = %v, want nil", err)
}
input := client.requests[0].Inputs[NPCRegistryReferenceSlot]
if input.Name != NPCRegistryReferenceSlot || input.MediaType != "application/json" || input.Digest != semanticNPCRegistryDigest(canonical) || input.OriginURI != "" {
t.Fatalf("NPC prompt input metadata = %#v, want semantic metadata without provenance", input)
}
if !bytes.Equal(input.Content, canonical) {
t.Fatalf("NPC prompt input = %s, want canonical JSON %s", input.Content, canonical)
}
}
func validNPCRegistryList() dnd.NPCList {
return dnd.NPCList{NPCs: []dnd.NPC{{
ID: identity.DeriveID("Mira Thorn"),
Name: "Mira Thorn",
Aliases: []string{"The Greencloak"},
Description: "A guarded ranger who watches the northern road.",
Relationships: []dnd.NPCRelationship{{
Target: "Captain Vale", Relationship: "reports to",
}},
SourceRefs: []source.SourceRef{{SourceID: "npc-session", StartUnitID: 41, EndUnitID: 43}},
}}}
}
func encodeNPCRegistry(t *testing.T, value dnd.NPCList) []byte {
t.Helper()
content, err := npccodec.New().Encode(value)
if err != nil {
t.Fatalf("encode NPC registry: %v", err)
}
return content
}
func marshalNPCRegistry(t *testing.T, value dnd.NPCList) []byte {
t.Helper()
content, err := json.Marshal(value)
if err != nil {
t.Fatalf("marshal NPC registry: %v", err)
}
return content
}
func npcRegistryReference(content []byte, origin string) contracts.ReferenceSet {
references := npcRegistryReferenceWithMedia(content, "application/json; charset=utf-8")
item := references.Slots[NPCRegistryReferenceSlot].Items[0]
item.Origin.URI = origin
slot := references.Slots[NPCRegistryReferenceSlot]
slot.Items[0] = item
references.Slots[NPCRegistryReferenceSlot] = slot
return references
}
func npcRegistryReferenceWithMedia(content []byte, mediaType string) contracts.ReferenceSet {
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
NPCRegistryReferenceSlot: {
Slot: contracts.ReferenceSlot{Name: NPCRegistryReferenceSlot, AcceptedMediaTypes: []string{"application/json"}, MaxBytes: NPCRegistryMaxBytes},
Items: []contracts.ReferenceItem{{
SlotName: NPCRegistryReferenceSlot,
MediaType: mediaType,
Content: append([]byte(nil), content...),
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///npc-registry.json"},
}},
},
}}
}
func checkpointFingerprintMap(values []pipeline.CheckpointFingerprint) map[string]string {
result := make(map[string]string, len(values))
for _, value := range values {
result[value.Name] = value.Value
}
return result
}

View File

@@ -39,6 +39,12 @@ func TestModuleSpec(t *testing.T) {
Description: "Optional campaign glossary reference material used only for disambiguation.",
AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"},
},
{
Name: NPCRegistryReferenceSlot,
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
AcceptedMediaTypes: []string{"application/json"},
MaxBytes: NPCRegistryMaxBytes,
},
{
Name: "party",
Description: "Optional party roster reference material used only for disambiguation.",

View File

@@ -15,6 +15,7 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
promptFS, err := shared.ModulePromptFS("dnd.spells", embeddedAssets, []promptfs.ModulePromptFile{
{Name: "dnd.spells.yaml", Path: "assets/prompts/dnd.spells.yaml"},
{Name: "catalog.md", Path: "assets/prompts/catalog.md"},
{Name: "npc_registry.md", Path: "assets/prompts/npc_registry.md"},
{Name: "task.md", Path: "assets/prompts/task.md"},
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
})
@@ -32,6 +33,7 @@ func scriptoriumPromptMetadata() (string, error) {
parts := append([]llm.AssetHashPart{
{FS: embeddedAssets, Path: "assets/prompts/dnd.spells.yaml"},
{FS: embeddedAssets, Path: "assets/prompts/catalog.md"},
{FS: embeddedAssets, Path: "assets/prompts/npc_registry.md"},
{FS: embeddedAssets, Path: "assets/prompts/task.md"},
{FS: embeddedAssets, Path: "assets/prompts/instructions.md"},
}, append(shared.CommonHashParts(), shared.ReferenceHashParts()...)...)

View File

@@ -21,8 +21,8 @@ func TestScriptoriumPromptPreparesTranscriptReferencesAndTaskMessages(t *testing
if prepared.OutputContract.SchemaPath != "dnd_spells_llm.v1.json" {
t.Fatalf("schema path = %q, want LLM-only schema", prepared.OutputContract.SchemaPath)
}
if got := len(prepared.Messages); got != 6 {
t.Fatalf("message count = %d, want 6", got)
if got := len(prepared.Messages); got != 7 {
t.Fatalf("message count = %d, want 7", got)
}
if !strings.Contains(prepared.Messages[1].Content, string(transcript)) {
t.Fatalf("transcript message did not include source input")
@@ -42,7 +42,10 @@ func TestScriptoriumPromptPreparesTranscriptReferencesAndTaskMessages(t *testing
if !strings.Contains(prepared.Messages[3].Content, `{"spell_names":["Cure Wounds"]}`) {
t.Fatalf("catalog message missing canonical spell-name input: %s", prepared.Messages[3].Content)
}
if strings.Contains(prepared.Messages[4].Content, string(transcript)) {
if !strings.Contains(prepared.Messages[4].Content, `{"npcs":[]}`) {
t.Fatalf("NPC registry message missing empty registry input: %s", prepared.Messages[4].Content)
}
if strings.Contains(prepared.Messages[5].Content, string(transcript)) {
t.Fatalf("task message leaked transcript bytes")
}
}
@@ -129,6 +132,7 @@ func prepareSpellsPrompt(t *testing.T, transcript []byte, players string, party
Inputs: map[string]scriptorium.ArtifactRef{
"transcript": scriptorium.InlineWithURI("file:///session.json", string(transcript)),
"spell_catalog": scriptorium.Inline(`{"spell_names":["Cure Wounds"]}`),
"npcs": scriptorium.Inline(`{"npcs":[]}`),
"players": scriptorium.Inline(players),
"party": scriptorium.Inline(party),
"glossary": scriptorium.Inline(glossary),

View File

@@ -0,0 +1,547 @@
// Package npcs normalizes merged D&D non-player character records.
package npcs
import (
"context"
"fmt"
"reflect"
"sort"
"strconv"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
)
const (
Key = "dnd/npcs"
NormalizationPolicy = "dnd.npcs.normalize.v1"
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
ReasonCodeNPCIDRecomputed = "npc_id_recomputed"
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
ReasonCodeDuplicateNPCCollapsed = "duplicate_npc_collapsed"
ReasonCodeRelationshipTargetCanonicalized = "relationship_target_canonicalized"
)
var requiredCapabilities = []string{"merged"}
var providedCapabilities = []string{"normalized"}
var _ contracts.Normalizer[dnd.NPCList] = (*Normalizer)(nil)
var _ contracts.ManifestMetadataProvider = (*Normalizer)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
type Options struct{}
type Normalizer struct{}
func New(Options) *Normalizer { return &Normalizer{} }
func (n *Normalizer) Key() string { return Key }
func (n *Normalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (n *Normalizer) ManifestMetadata() map[string]any {
if n == nil {
return nil
}
return map[string]any{
"identity_policy": identity.Policy,
"normalization_policy": NormalizationPolicy,
}
}
func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
if n == nil {
return nil
}
return []pipeline.CheckpointFingerprint{
{Name: "identity_policy", Value: identity.Policy},
{Name: "normalization_policy", Value: NormalizationPolicy},
}
}
func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[dnd.NPCList]) (contracts.TypedNormalizeResult[dnd.NPCList], error) {
if n == nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("normalizer must not be nil")
}
if ctx == nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context error before normalize: %w", err)
}
value, warnings := normalizeList(req.MergeOutput.Value)
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: value, Warnings: warnings}, nil
}
type normalizedRecord struct {
npc dnd.NPC
fieldsChanged bool
referencesChanged bool
}
func normalizeList(input dnd.NPCList) (dnd.NPCList, []contracts.Warning) {
if input.NPCs == nil {
return dnd.NPCList{}, nil
}
records := make([]normalizedRecord, len(input.NPCs))
warnings := make([]contracts.Warning, 0)
for index, inputNPC := range input.NPCs {
npc, fieldsChanged, referencesChanged := normalizeRecord(inputNPC)
records[index] = normalizedRecord{
npc: npc,
fieldsChanged: fieldsChanged,
referencesChanged: referencesChanged,
}
if fieldsChanged {
warnings = append(warnings, contracts.Warning{
Scope: npcScope(index),
ReasonCode: ReasonCodeNPCFieldsNormalized,
Message: fmt.Sprintf("input index %d: NPC fields normalized for %s",
index, diagnostics.Quote(inputNPC.Name)),
})
}
if referencesChanged {
warnings = append(warnings, contracts.Warning{
Scope: npcScope(index),
ReasonCode: ReasonCodeSourceReferencesNormalized,
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
index, len(inputNPC.SourceRefs), len(npc.SourceRefs)),
})
}
if inputNPC.ID != npc.ID {
warnings = append(warnings, contracts.Warning{
Scope: npcScope(index),
ReasonCode: ReasonCodeNPCIDRecomputed,
Message: fmt.Sprintf("input index %d: NPC ID recomputed from %s",
index, diagnostics.Quote(npc.Name)),
})
}
}
components := identityComponents(records)
output := dnd.NPCList{NPCs: make([]dnd.NPC, 0, len(components))}
retainedIndexes := make([]int, 0, len(components))
for _, members := range components {
consolidated, sourceChanged := consolidate(records, members)
retainedIndex := members[0]
output.NPCs = append(output.NPCs, consolidated)
retainedIndexes = append(retainedIndexes, retainedIndex)
if sourceChanged {
warnings = append(warnings, contracts.Warning{
Scope: npcScope(retainedIndex),
ReasonCode: ReasonCodeSourceReferencesNormalized,
Message: fmt.Sprintf("input index %d: source references normalized during identity consolidation (final count %d)",
retainedIndex, len(consolidated.SourceRefs)),
})
}
if len(members) > 1 {
warnings = append(warnings, duplicateWarning(retainedIndex, members[1:]))
}
}
warnings = append(warnings, canonicalizeRelationshipTargets(&output, retainedIndexes)...)
return output, warnings
}
func normalizeRecord(input dnd.NPC) (dnd.NPC, bool, bool) {
output := cloneNPC(input)
output.Name = identity.NormalizeDisplay(input.Name)
output.Aliases = normalizeAliases(input.Aliases, output.Name)
output.Description = strings.TrimSpace(input.Description)
output.Relationships = normalizeRelationships(input.Relationships)
output.SourceRefs, _, _ = canonicalizeSourceRefs(input.SourceRefs)
output.ID = identity.DeriveID(output.Name)
fieldsChanged := input.Name != output.Name ||
!reflect.DeepEqual(input.Aliases, output.Aliases) ||
input.Description != output.Description ||
!reflect.DeepEqual(input.Relationships, output.Relationships)
referencesChanged := !reflect.DeepEqual(input.SourceRefs, output.SourceRefs)
return output, fieldsChanged, referencesChanged
}
func cloneNPC(input dnd.NPC) dnd.NPC {
output := input
if input.Aliases != nil {
output.Aliases = make([]string, len(input.Aliases))
copy(output.Aliases, input.Aliases)
}
if input.Relationships != nil {
output.Relationships = make([]dnd.NPCRelationship, len(input.Relationships))
copy(output.Relationships, input.Relationships)
}
if input.SourceRefs != nil {
output.SourceRefs = make([]source.SourceRef, len(input.SourceRefs))
copy(output.SourceRefs, input.SourceRefs)
}
return output
}
func normalizeAliases(input []string, canonicalName string) []string {
if input == nil {
return nil
}
canonicalKey := identity.ComparisonKey(canonicalName)
output := make([]string, 0, len(input))
seen := make(map[string]struct{}, len(input))
for _, alias := range input {
normalized := identity.NormalizeDisplay(alias)
key := identity.ComparisonKey(normalized)
if canonicalKey != "" && key == canonicalKey {
continue
}
if _, exists := seen[key]; exists {
continue
}
seen[key] = struct{}{}
output = append(output, normalized)
}
return output
}
func normalizeRelationships(input []dnd.NPCRelationship) []dnd.NPCRelationship {
if input == nil {
return nil
}
output := make([]dnd.NPCRelationship, 0, len(input))
seen := make(map[relationshipIdentity]struct{}, len(input))
for _, relationship := range input {
normalized := dnd.NPCRelationship{
Target: identity.NormalizeDisplay(relationship.Target),
Relationship: strings.TrimSpace(relationship.Relationship),
}
key := relationshipKey(normalized)
if _, exists := seen[key]; exists {
continue
}
seen[key] = struct{}{}
output = append(output, normalized)
}
return output
}
type relationshipIdentity struct {
target string
relationship string
}
func relationshipKey(relationship dnd.NPCRelationship) relationshipIdentity {
return relationshipIdentity{
target: identity.ComparisonKey(relationship.Target),
relationship: identity.ComparisonKey(relationship.Relationship),
}
}
func canonicalizeSourceRefs(input []source.SourceRef) ([]source.SourceRef, bool, int) {
if input == nil {
return nil, false, 0
}
canonical := make([]source.SourceRef, len(input))
copy(canonical, input)
sort.SliceStable(canonical, func(left, right int) bool {
return sourceRefLess(canonical[left], canonical[right])
})
orderChanged := false
for index := range input {
if input[index] != canonical[index] {
orderChanged = true
break
}
}
unique := make([]source.SourceRef, 0, len(canonical))
for _, ref := range canonical {
if len(unique) == 0 || unique[len(unique)-1] != ref {
unique = append(unique, ref)
}
}
return unique, orderChanged, len(input) - len(unique)
}
func sourceRefLess(left, right source.SourceRef) bool {
if left.SourceID != right.SourceID {
return left.SourceID < right.SourceID
}
if left.StartUnitID != right.StartUnitID {
return left.StartUnitID < right.StartUnitID
}
return left.EndUnitID < right.EndUnitID
}
func identityComponents(records []normalizedRecord) [][]int {
parent := make([]int, len(records))
for index := range parent {
parent[index] = index
}
for left := 0; left < len(records); left++ {
for right := left + 1; right < len(records); right++ {
if recordsCanMerge(records[left].npc, records[right].npc) {
union(parent, left, right)
}
}
}
byRoot := make(map[int][]int, len(records))
for index := range records {
root := find(parent, index)
byRoot[root] = append(byRoot[root], index)
}
roots := make([]int, 0, len(byRoot))
for root := range byRoot {
roots = append(roots, root)
}
sort.Slice(roots, func(left, right int) bool {
return byRoot[roots[left]][0] < byRoot[roots[right]][0]
})
components := make([][]int, 0, len(roots))
for _, root := range roots {
components = append(components, byRoot[root])
}
return components
}
func recordsCanMerge(left, right dnd.NPC) bool {
leftCanonical := identity.ComparisonKey(left.Name)
rightCanonical := identity.ComparisonKey(right.Name)
return leftCanonical == rightCanonical ||
containsAliasKey(right.Aliases, leftCanonical) ||
containsAliasKey(left.Aliases, rightCanonical)
}
func containsAliasKey(aliases []string, wanted string) bool {
for _, alias := range aliases {
if identity.ComparisonKey(alias) == wanted {
return true
}
}
return false
}
func find(parent []int, index int) int {
for parent[index] != index {
parent[index] = parent[parent[index]]
index = parent[index]
}
return index
}
func union(parent []int, left, right int) {
leftRoot := find(parent, left)
rightRoot := find(parent, right)
if leftRoot == rightRoot {
return
}
if leftRoot < rightRoot {
parent[rightRoot] = leftRoot
} else {
parent[leftRoot] = rightRoot
}
}
func consolidate(records []normalizedRecord, members []int) (dnd.NPC, bool) {
output := cloneNPC(records[members[0]].npc)
originalRefs := cloneSourceRefs(output.SourceRefs)
canonicalKey := identity.ComparisonKey(output.Name)
for _, member := range members[1:] {
candidate := records[member].npc
appendAlias(&output.Aliases, candidate.Name, canonicalKey)
for _, alias := range candidate.Aliases {
appendAlias(&output.Aliases, alias, canonicalKey)
}
for _, relationship := range candidate.Relationships {
appendRelationship(&output.Relationships, relationship)
}
output.SourceRefs = append(output.SourceRefs, candidate.SourceRefs...)
}
output.SourceRefs, _, _ = canonicalizeSourceRefs(output.SourceRefs)
output.ID = identity.DeriveID(output.Name)
return output, !reflect.DeepEqual(originalRefs, output.SourceRefs)
}
func cloneSourceRefs(input []source.SourceRef) []source.SourceRef {
if input == nil {
return nil
}
output := make([]source.SourceRef, len(input))
copy(output, input)
return output
}
func appendAlias(aliases *[]string, value, canonicalKey string) {
key := identity.ComparisonKey(value)
if canonicalKey != "" && key == canonicalKey {
return
}
for _, existing := range *aliases {
if identity.ComparisonKey(existing) == key {
return
}
}
*aliases = append(*aliases, value)
}
func appendRelationship(relationships *[]dnd.NPCRelationship, relationship dnd.NPCRelationship) {
key := relationshipKey(relationship)
for _, existing := range *relationships {
if relationshipKey(existing) == key {
return
}
}
*relationships = append(*relationships, relationship)
}
func canonicalizeRelationshipTargets(list *dnd.NPCList, retainedIndexes []int) []contracts.Warning {
if list == nil || len(list.NPCs) == 0 {
return nil
}
owners := make(map[string][]int)
for index, npc := range list.NPCs {
if key := identity.ComparisonKey(npc.Name); key != "" {
owners[key] = append(owners[key], index)
}
for _, alias := range npc.Aliases {
if key := identity.ComparisonKey(alias); key != "" {
owners[key] = appendUniqueIndex(owners[key], index)
}
}
}
warnings := make([]contracts.Warning, 0)
for outputIndex := range list.NPCs {
npc := &list.NPCs[outputIndex]
originalRelationshipCount := len(npc.Relationships)
for relationshipIndex := range npc.Relationships {
relationship := &npc.Relationships[relationshipIndex]
key := identity.ComparisonKey(relationship.Target)
if key == "" || len(owners[key]) != 1 {
continue
}
target := list.NPCs[owners[key][0]].Name
if relationship.Target == target {
continue
}
oldTarget := relationship.Target
relationship.Target = target
warnings = append(warnings, contracts.Warning{
Scope: npcScope(retainedIndexes[outputIndex]),
ReasonCode: ReasonCodeRelationshipTargetCanonicalized,
Message: fmt.Sprintf("input index %d: relationship target canonicalized from %s to %s",
retainedIndexes[outputIndex], diagnostics.Quote(oldTarget), diagnostics.Quote(target)),
})
}
npc.Relationships = deduplicateRelationships(npc.Relationships)
if len(npc.Relationships) != originalRelationshipCount {
warnings = append(warnings, contracts.Warning{
Scope: npcScope(retainedIndexes[outputIndex]),
ReasonCode: ReasonCodeNPCFieldsNormalized,
Message: fmt.Sprintf("input index %d: duplicate relationships removed after target canonicalization",
retainedIndexes[outputIndex]),
})
}
}
return warnings
}
func deduplicateRelationships(input []dnd.NPCRelationship) []dnd.NPCRelationship {
if input == nil {
return nil
}
output := make([]dnd.NPCRelationship, 0, len(input))
seen := make(map[relationshipIdentity]struct{}, len(input))
for _, relationship := range input {
key := relationshipKey(relationship)
if _, exists := seen[key]; exists {
continue
}
seen[key] = struct{}{}
output = append(output, relationship)
}
return output
}
func appendUniqueIndex(values []int, wanted int) []int {
for _, value := range values {
if value == wanted {
return values
}
}
return append(values, wanted)
}
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
const maxDisplayedIndices = 20
displayed := removed
if len(displayed) > maxDisplayedIndices {
displayed = displayed[:maxDisplayedIndices]
}
indices := make([]string, len(displayed))
for index, removedIndex := range displayed {
indices[index] = strconv.Itoa(removedIndex)
}
message := fmt.Sprintf("retained input index %d; removed input indices [%s]", retainedIndex, strings.Join(indices, ", "))
if omitted := len(removed) - len(displayed); omitted > 0 {
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
}
return contracts.Warning{
Scope: npcScope(retainedIndex),
ReasonCode: ReasonCodeDuplicateNPCCollapsed,
Message: message,
}
}
func npcScope(index int) string { return fmt.Sprintf("npcs[%d]", index) }
func ModuleSpec() pipeline.ModuleSpec {
return pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: append([]string(nil), requiredCapabilities...),
Provides: append([]string(nil), providedCapabilities...),
ArtifactKind: dnd.NPCListKind,
}
}
func Register(registry *pipeline.NormalizerRegistry) error {
return pipeline.RegisterNormalizerBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Normalizer[dnd.NPCList], error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(options), nil
})
}
func validateOptions(options map[string]any) error {
_, err := DecodeOptions(options)
return err
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, normalizerErrorf("%w", err)
}
return Options{}, nil
}
func normalizerErrorf(format string, args ...any) error {
return fmt.Errorf("dnd npcs normalizer: "+format, args...)
}

View File

@@ -0,0 +1,292 @@
package npcs
import (
"context"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
domainidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
func TestModuleContractAndIdentity(t *testing.T) {
if _, err := DecodeOptions(nil); err != nil {
t.Fatalf("DecodeOptions(nil) error = %v, want nil", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
}
want := pipeline.ModuleSpec{
Key: Key,
Stage: pipeline.StageNormalize,
Requires: []string{"merged"},
Provides: []string{"normalized"},
ArtifactKind: dnd.NPCListKind,
}
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
}
if slots := New(Options{}).ReferenceSlots(); slots != nil {
t.Fatalf("ReferenceSlots() = %#v, want nil", slots)
}
registry := pipeline.NewNormalizerRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
if registered, ok := registry.SpecForArtifact(Key, dnd.NPCListKind); !ok || !reflect.DeepEqual(registered, want) {
t.Fatalf("registered spec = %#v, ok = %t, want %#v", registered, ok, want)
}
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "normalizer registry") {
t.Fatalf("Register(nil) error = %v, want registry error", err)
}
normalizer := New(Options{})
metadata := normalizer.ManifestMetadata()
if metadata["identity_policy"] != domainidentity.Policy || metadata["normalization_policy"] != NormalizationPolicy {
t.Fatalf("metadata = %#v, want identity and normalization policies", metadata)
}
fingerprints := normalizer.CheckpointFingerprints()
wantFingerprints := []pipeline.CheckpointFingerprint{
{Name: "identity_policy", Value: domainidentity.Policy},
{Name: "normalization_policy", Value: NormalizationPolicy},
}
if !reflect.DeepEqual(fingerprints, wantFingerprints) {
t.Fatalf("fingerprints = %#v, want %#v", fingerprints, wantFingerprints)
}
fingerprints[0].Value = "changed"
if got := normalizer.CheckpointFingerprints(); !reflect.DeepEqual(got, wantFingerprints) {
t.Fatalf("fingerprints were not defensive: %#v", got)
}
}
func TestNormalizePerRecordFieldsAndEvidence(t *testing.T) {
input := dnd.NPCList{NPCs: []dnd.NPC{{
ID: "wrong",
Name: " Lady\tAsh ",
Aliases: []string{" Ash ", " L.A. ", "l.a.", " Lady Ash "},
Description: " first description\n",
Relationships: []dnd.NPCRelationship{
{Target: " Lord\nOak ", Relationship: " friend "},
{Target: "lord oak", Relationship: "friend"},
},
SourceRefs: []source.SourceRef{
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
{SourceID: "a", StartUnitID: 4, EndUnitID: 4},
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
},
}}}
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
got := result.Value.NPCs[0]
if got.Name != "Lady Ash" || got.Description != "first description" {
t.Fatalf("normalized fields = %#v, want normalized display and description", got)
}
if !reflect.DeepEqual(got.Aliases, []string{"Ash", "L.A."}) {
t.Fatalf("aliases = %#v, want canonical alias removal and deduplication", got.Aliases)
}
wantRelationships := []dnd.NPCRelationship{{Target: "Lord Oak", Relationship: "friend"}}
if !reflect.DeepEqual(got.Relationships, wantRelationships) {
t.Fatalf("relationships = %#v, want %#v", got.Relationships, wantRelationships)
}
wantRefs := []source.SourceRef{
{SourceID: "a", StartUnitID: 4, EndUnitID: 4},
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
}
if !reflect.DeepEqual(got.SourceRefs, wantRefs) {
t.Fatalf("source refs = %#v, want %#v", got.SourceRefs, wantRefs)
}
if got.ID != domainidentity.DeriveID("Lady Ash") {
t.Fatalf("ID = %q, want derived ID", got.ID)
}
if !hasWarning(result.Warnings, ReasonCodeNPCFieldsNormalized, "npcs[0]") ||
!hasWarning(result.Warnings, ReasonCodeSourceReferencesNormalized, "npcs[0]") ||
!hasWarning(result.Warnings, ReasonCodeNPCIDRecomputed, "npcs[0]") {
t.Fatalf("warnings = %#v, want field, source, and ID warnings", result.Warnings)
}
}
func TestNormalizeConsolidatesIdentityComponentsInStableOrder(t *testing.T) {
input := dnd.NPCList{NPCs: []dnd.NPC{
{
Name: " Captain Vale ",
Description: "first description",
Aliases: []string{"Vale"},
Relationships: []dnd.NPCRelationship{{Target: "Archivist", Relationship: "knows"}},
SourceRefs: []source.SourceRef{{SourceID: "a", StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "Captain Vale",
Description: "second description",
Aliases: []string{"CV"},
SourceRefs: []source.SourceRef{{SourceID: "b", StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "The Sage",
Description: "sage description",
Aliases: []string{"Archivist"},
SourceRefs: []source.SourceRef{{SourceID: "c", StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "Archivist",
Description: "later sage description",
Aliases: []string{"Chronicler"},
SourceRefs: []source.SourceRef{{SourceID: "d", StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "North",
Description: "north description",
SourceRefs: []source.SourceRef{{SourceID: "e", StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "North Old",
Description: "old north description",
Aliases: []string{"North"},
SourceRefs: []source.SourceRef{{SourceID: "f", StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "North Renamed",
Description: "renamed north description",
Aliases: []string{"North Old"},
SourceRefs: []source.SourceRef{{SourceID: "g", StartUnitID: 1, EndUnitID: 1}},
},
{Name: "Red", Description: "red", Aliases: []string{"Shared"}, SourceRefs: []source.SourceRef{{SourceID: "h", StartUnitID: 1, EndUnitID: 1}}},
{Name: "Blue", Description: "blue", Aliases: []string{"Shared"}, SourceRefs: []source.SourceRef{{SourceID: "i", StartUnitID: 1, EndUnitID: 1}}},
}}
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if got := len(result.Value.NPCs); got != 5 {
t.Fatalf("normalized NPC count = %d, want five components", got)
}
first := result.Value.NPCs[0]
if first.Name != "Captain Vale" || first.Description != "first description" || !reflect.DeepEqual(first.Aliases, []string{"Vale", "CV"}) {
t.Fatalf("first component = %#v, want first description and ordered aliases", first)
}
if !reflect.DeepEqual(first.SourceRefs, []source.SourceRef{
{SourceID: "a", StartUnitID: 1, EndUnitID: 1},
{SourceID: "b", StartUnitID: 1, EndUnitID: 1},
}) {
t.Fatalf("first provenance = %#v, want unioned refs", first.SourceRefs)
}
second := result.Value.NPCs[1]
if second.Name != "The Sage" || !reflect.DeepEqual(second.Aliases, []string{"Archivist", "Chronicler"}) || second.Description != "sage description" {
t.Fatalf("canonical-to-alias component = %#v, want consolidated sage", second)
}
if first.Relationships[0].Target != "The Sage" {
t.Fatalf("relationship target = %q, want The Sage", first.Relationships[0].Target)
}
if !hasWarning(result.Warnings, ReasonCodeRelationshipTargetCanonicalized, "npcs[0]") ||
!hasWarning(result.Warnings, ReasonCodeDuplicateNPCCollapsed, "npcs[0]") ||
!hasWarning(result.Warnings, ReasonCodeDuplicateNPCCollapsed, "npcs[2]") {
t.Fatalf("warnings = %#v, want collapse and target warnings", result.Warnings)
}
if got := result.Value.NPCs[2].Aliases; !reflect.DeepEqual(got, []string{"North Old", "North Renamed"}) {
t.Fatalf("transitive aliases = %#v, want ordered canonical members", got)
}
if result.Value.NPCs[3].Name != "Red" || result.Value.NPCs[4].Name != "Blue" {
t.Fatalf("shared-alias ordering = %#v, want Red then Blue", result.Value.NPCs[3:])
}
}
func TestNormalizePreservesInvalidEvidenceAndDoesNotAliasInput(t *testing.T) {
invalid := dnd.NPCList{NPCs: []dnd.NPC{{
Name: " ",
Aliases: []string{""},
Description: " ",
Relationships: []dnd.NPCRelationship{{Target: " ", Relationship: " "}},
SourceRefs: []source.SourceRef{{SourceID: "source", StartUnitID: 9, EndUnitID: 9}},
}}}
original := cloneNPCListForTest(invalid)
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(invalid))
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if !reflect.DeepEqual(invalid, original) {
t.Fatalf("normalizer mutated input: got %#v, want %#v", invalid, original)
}
if result.Value.NPCs[0].Name != "" || result.Value.NPCs[0].Aliases[0] != "" || result.Value.NPCs[0].Relationships[0].Target != "" {
t.Fatalf("invalid evidence was unexpectedly removed: %#v", result.Value.NPCs[0])
}
result.Value.NPCs[0].Aliases[0] = "changed"
result.Value.NPCs[0].Relationships[0].Target = "changed"
result.Value.NPCs[0].SourceRefs[0].SourceID = "changed"
if invalid.NPCs[0].Aliases[0] != "" || invalid.NPCs[0].Relationships[0].Target != " " || invalid.NPCs[0].SourceRefs[0].SourceID != "source" {
t.Fatalf("output aliases input storage: input = %#v", invalid)
}
}
func TestNormalizeHandlesNilAndCanceledCalls(t *testing.T) {
normalizer := New(Options{})
result, err := normalizer.Normalize(context.Background(), normalizeRequest(dnd.NPCList{NPCs: nil}))
if err != nil || result.Value.NPCs != nil {
t.Fatalf("nil list result = %#v, error = %v, want nil NPC slice", result.Value, err)
}
canceled, cancel := context.WithCancel(context.Background())
cancel()
if _, err := normalizer.Normalize(canceled, normalizeRequest(dnd.NPCList{})); err == nil || !strings.Contains(err.Error(), "context error") {
t.Fatalf("canceled Normalize() error = %v, want context error", err)
}
if _, err := normalizer.Normalize(nil, normalizeRequest(dnd.NPCList{})); err == nil || !strings.Contains(err.Error(), "context must not be nil") {
t.Fatalf("nil context Normalize() error = %v, want context error", err)
}
var nilNormalizer *Normalizer
if _, err := nilNormalizer.Normalize(context.Background(), normalizeRequest(dnd.NPCList{})); err == nil {
t.Fatal("nil normalizer Normalize() error = nil, want error")
}
}
func TestNormalizePreservesPresentEmptyAliases(t *testing.T) {
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(dnd.NPCList{NPCs: []dnd.NPC{{
Name: "Hooded Guard",
Aliases: []string{},
Description: "A distinguishable sentry.",
SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}},
}}}))
if err != nil {
t.Fatalf("Normalize() error = %v, want nil", err)
}
if result.Value.NPCs[0].Aliases == nil || len(result.Value.NPCs[0].Aliases) != 0 {
t.Fatalf("aliases = %#v, want present empty array", result.Value.NPCs[0].Aliases)
}
}
func normalizeRequest(value dnd.NPCList) contracts.TypedNormalizeRequest[dnd.NPCList] {
return contracts.TypedNormalizeRequest[dnd.NPCList]{
MergeOutput: contracts.MergeArtifact[dnd.NPCList]{Value: value},
}
}
func cloneNPCListForTest(input dnd.NPCList) dnd.NPCList {
output := dnd.NPCList{}
if input.NPCs != nil {
output.NPCs = make([]dnd.NPC, len(input.NPCs))
for index, npc := range input.NPCs {
output.NPCs[index] = cloneNPC(npc)
}
}
return output
}
func hasWarning(warnings []contracts.Warning, reason, scope string) bool {
for _, warning := range warnings {
if warning.ReasonCode == reason && warning.Scope == scope {
return true
}
}
return false
}

View File

@@ -0,0 +1,230 @@
// Package identity owns the stable identity policy for D&D non-player
// characters.
package identity
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"strings"
"golang.org/x/text/cases"
"golang.org/x/text/unicode/norm"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
const (
// Policy identifies the complete identity comparison and ID derivation
// policy. A future semantic change must use a new value.
Policy = "dnd.npcs.identity.v1"
// IdentityPolicy is an explicit alias for callers recording policy
// fingerprints.
IdentityPolicy = Policy
)
const idPrefix = "npc:sha256:"
// IssueCode identifies one deterministic registry identity problem.
type IssueCode string
const (
IssueEmptyCanonicalName IssueCode = "empty_canonical_name"
IssueEmptyAlias IssueCode = "empty_alias"
IssueInvalidID IssueCode = "invalid_id"
IssueIDMismatch IssueCode = "id_mismatch"
IssueDuplicateCanonical IssueCode = "duplicate_canonical_identity"
IssueDuplicateID IssueCode = "duplicate_id"
IssueDuplicateAlias IssueCode = "duplicate_alias"
IssueOwnCanonicalAlias IssueCode = "alias_matches_canonical_name"
IssueAliasCanonicalCollision IssueCode = "alias_canonical_collision"
IssueAliasOwnershipCollision IssueCode = "alias_owned_by_multiple_records"
)
// Issue is an inspectable identity validation problem. AliasIndex is -1 when
// the issue applies to an NPC as a whole rather than a particular alias.
type Issue struct {
Code IssueCode
RecordIndex int
AliasIndex int
Value string
}
// NormalizeDisplay trims and collapses Unicode whitespace while retaining all
// other observed spelling and punctuation.
func NormalizeDisplay(value string) string {
return strings.Join(strings.Fields(value), " ")
}
// ComparisonKey returns the stable key used for NPC identity comparisons.
func ComparisonKey(value string) string {
value = norm.NFKC.String(value)
value = strings.Map(func(r rune) rune {
switch r {
case '\u2018', '\u2019', '\u02bc':
return '\''
default:
return r
}
}, value)
value = strings.Join(strings.Fields(value), " ")
return cases.Fold().String(value)
}
// DeriveID returns the deterministic ID for a canonical NPC name. Empty
// identity keys intentionally produce an empty ID so shape validation can
// report the missing identity instead of manufacturing one.
func DeriveID(name string) string {
key := ComparisonKey(name)
if key == "" {
return ""
}
digest := sha256.Sum256([]byte(key))
return idPrefix + hex.EncodeToString(digest[:])
}
// IDFor is a concise alias for DeriveID for callers that work with IDs as
// values rather than derivation operations.
func IDFor(name string) string { return DeriveID(name) }
// IsValidID reports whether value has the exact durable NPC ID syntax.
func IsValidID(value string) bool {
if len(value) != len(idPrefix)+sha256.Size*2 || !strings.HasPrefix(value, idPrefix) {
return false
}
for _, r := range value[len(idPrefix):] {
if !(r >= '0' && r <= '9') && !(r >= 'a' && r <= 'f') {
return false
}
}
return true
}
// ValidID is an alias for IsValidID.
func ValidID(value string) bool { return IsValidID(value) }
// ValidateRegistry checks all identity invariants without changing the input.
// It accepts the NPC slice used by typed pipeline artifacts. Use ValidateList
// when the enclosing NPCList is more convenient at the call site.
func ValidateRegistry(npcs []dnd.NPC) []Issue {
type record struct {
canonical string
aliases []string
}
records := make([]record, len(npcs))
issues := make([]Issue, 0)
canonicalOwners := make(map[string][]int)
idOwners := make(map[string][]int)
aliasOwners := make(map[string][]int)
for recordIndex, npc := range npcs {
canonical := ComparisonKey(npc.Name)
records[recordIndex].canonical = canonical
if canonical == "" {
issues = append(issues, Issue{Code: IssueEmptyCanonicalName, RecordIndex: recordIndex, AliasIndex: -1, Value: npc.Name})
} else {
canonicalOwners[canonical] = append(canonicalOwners[canonical], recordIndex)
}
if !IsValidID(npc.ID) {
issues = append(issues, Issue{Code: IssueInvalidID, RecordIndex: recordIndex, AliasIndex: -1, Value: npc.ID})
} else if expected := DeriveID(npc.Name); npc.ID != expected {
issues = append(issues, Issue{Code: IssueIDMismatch, RecordIndex: recordIndex, AliasIndex: -1, Value: npc.ID})
}
if npc.ID != "" {
idOwners[npc.ID] = append(idOwners[npc.ID], recordIndex)
}
seenAliases := make(map[string]int, len(npc.Aliases))
for aliasIndex, alias := range npc.Aliases {
key := ComparisonKey(alias)
records[recordIndex].aliases = append(records[recordIndex].aliases, key)
if key == "" {
issues = append(issues, Issue{Code: IssueEmptyAlias, RecordIndex: recordIndex, AliasIndex: aliasIndex, Value: alias})
continue
}
if _, ok := seenAliases[key]; ok {
issues = append(issues, Issue{Code: IssueDuplicateAlias, RecordIndex: recordIndex, AliasIndex: aliasIndex, Value: alias})
} else {
seenAliases[key] = aliasIndex
}
if key == canonical {
issues = append(issues, Issue{Code: IssueOwnCanonicalAlias, RecordIndex: recordIndex, AliasIndex: aliasIndex, Value: alias})
}
aliasOwners[key] = append(aliasOwners[key], recordIndex)
}
}
for recordIndex, record := range records {
if record.canonical != "" && len(canonicalOwners[record.canonical]) > 1 && canonicalOwners[record.canonical][0] != recordIndex {
issues = append(issues, Issue{Code: IssueDuplicateCanonical, RecordIndex: recordIndex, AliasIndex: -1, Value: npcs[recordIndex].Name})
}
if id := npcs[recordIndex].ID; id != "" && len(idOwners[id]) > 1 && idOwners[id][0] != recordIndex {
issues = append(issues, Issue{Code: IssueDuplicateID, RecordIndex: recordIndex, AliasIndex: -1, Value: id})
}
}
seenAliasKeys := make(map[string]struct{})
for _, record := range records {
for _, alias := range record.aliases {
if alias == "" {
continue
}
if _, alreadyProcessed := seenAliasKeys[alias]; alreadyProcessed {
continue
}
seenAliasKeys[alias] = struct{}{}
owners := uniqueIndexes(aliasOwners[alias])
if len(owners) > 1 {
for _, recordIndex := range owners {
issues = append(issues, Issue{Code: IssueAliasOwnershipCollision, RecordIndex: recordIndex, AliasIndex: aliasIndexFor(records[recordIndex].aliases, alias), Value: alias})
}
}
for _, recordIndex := range owners {
for _, canonicalOwner := range canonicalOwners[alias] {
if canonicalOwner != recordIndex {
issues = append(issues, Issue{Code: IssueAliasCanonicalCollision, RecordIndex: recordIndex, AliasIndex: aliasIndexFor(records[recordIndex].aliases, alias), Value: alias})
break
}
}
}
}
}
return issues
}
// ValidateList validates the identity members of list.
func ValidateList(list dnd.NPCList) []Issue { return ValidateRegistry(list.NPCs) }
// Validate is a convenience alias for ValidateList.
func Validate(list dnd.NPCList) []Issue { return ValidateList(list) }
func uniqueIndexes(values []int) []int {
seen := make(map[int]struct{}, len(values))
unique := make([]int, 0, len(values))
for _, value := range values {
if _, ok := seen[value]; ok {
continue
}
seen[value] = struct{}{}
unique = append(unique, value)
}
return unique
}
func aliasIndexFor(aliases []string, key string) int {
for index, alias := range aliases {
if alias == key {
return index
}
}
return -1
}
// Error makes an issue useful in simple callers while preserving its
// structured fields for aggregate diagnostics.
func (i Issue) Error() string {
return fmt.Sprintf("%s at record %d", i.Code, i.RecordIndex)
}

View File

@@ -0,0 +1,114 @@
package identity
import (
"strings"
"sync"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestComparisonKeyNormalizesSupportedEquivalences(t *testing.T) {
tests := []struct {
name string
left string
right string
}{
{name: "case", left: "Captain Vale", right: "cAPtAiN vALE"},
{name: "compatibility", left: "", right: "Ally"},
{name: "whitespace", left: " Mira\u2003Thorn ", right: "Mira Thorn"},
{name: "apostrophe", left: "ORin", right: "o'Rin"},
{name: "modifier apostrophe", left: "OʼRin", right: "o'Rin"},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
if ComparisonKey(test.left) != ComparisonKey(test.right) {
t.Fatalf("ComparisonKey(%q) = %q, ComparisonKey(%q) = %q", test.left, ComparisonKey(test.left), test.right, ComparisonKey(test.right))
}
})
}
if ComparisonKey("Mira Thorn") == ComparisonKey("Mira Thorne") {
t.Fatal("different names received the same comparison key")
}
}
func TestNormalizeDisplayOnlyChangesWhitespace(t *testing.T) {
if got := NormalizeDisplay(" ORin\u2003Thorn "); got != "ORin Thorn" {
t.Fatalf("NormalizeDisplay() = %q", got)
}
}
func TestDeriveIDAndIDSyntax(t *testing.T) {
got := DeriveID(" Mira\u2003Thorn ")
if len(got) != len("npc:sha256:")+64 || !strings.HasPrefix(got, "npc:sha256:") || !IsValidID(got) {
t.Fatalf("DeriveID() = %q, want exact NPC ID syntax", got)
}
if got != DeriveID("Mira Thorn") || got != IDFor("Mira Thorn") {
t.Fatal("DeriveID() is not deterministic across equivalent names")
}
if DeriveID(" \u2003 ") != "" {
t.Fatal("empty identity produced an ID")
}
for _, invalid := range []string{"", "npc:sha256:", "npc:sha256:ABC", "npc:sha256:" + strings.Repeat("0", 63), "npc:sha256:" + strings.Repeat("0", 65)} {
if IsValidID(invalid) {
t.Fatalf("IsValidID(%q) = true, want false", invalid)
}
}
}
func TestIdentityFunctionsAreSafeForConcurrentUse(t *testing.T) {
const workers = 32
var group sync.WaitGroup
for i := 0; i < workers; i++ {
group.Add(1)
go func() {
defer group.Done()
for j := 0; j < 100; j++ {
if !IsValidID(DeriveID("Mira Thorn")) {
t.Errorf("derived ID failed syntax check")
return
}
}
}()
}
group.Wait()
}
func TestValidateRegistryReportsIdentityCollisionCategories(t *testing.T) {
validID := DeriveID("Mira Thorn")
npcs := []dnd.NPC{
{ID: validID, Name: "Mira Thorn", Aliases: []string{"The Greencloak", "the greencloak", "Mira Thorn"}},
{ID: validID, Name: "Mira Thorn", Aliases: []string{"The Greencloak"}},
{ID: DeriveID("Captain Vale"), Name: "Captain Vale", Aliases: []string{"Mira Thorn"}},
}
issues := ValidateRegistry(npcs)
want := map[IssueCode]bool{
IssueDuplicateAlias: false,
IssueOwnCanonicalAlias: false,
IssueDuplicateCanonical: false,
IssueDuplicateID: false,
IssueAliasOwnershipCollision: false,
IssueAliasCanonicalCollision: false,
}
for _, issue := range issues {
if _, ok := want[issue.Code]; ok {
want[issue.Code] = true
}
}
for code, found := range want {
if !found {
t.Errorf("ValidateRegistry() did not report %s", code)
}
}
}
func TestValidateRegistryReportsIDProblems(t *testing.T) {
issues := ValidateRegistry([]dnd.NPC{{ID: "bad", Name: "Mira Thorn"}, {ID: DeriveID("Mira Thorn"), Name: "Other Name"}})
seen := map[IssueCode]bool{}
for _, issue := range issues {
seen[issue.Code] = true
}
if !seen[IssueInvalidID] || !seen[IssueIDMismatch] {
t.Fatalf("ValidateRegistry() issues = %#v, want invalid and mismatched ID issues", issues)
}
}

View File

@@ -8,9 +8,16 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
spellcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/spells"
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
npcidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/identity"
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
npcsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/source_refs"
npcrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/source_relatedness"
spellcatalog "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/catalog"
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
@@ -34,25 +41,43 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
register func() error
}{
{name: "spells codec", register: func() error { return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, codec) }},
{name: "npcs codec", register: func() error { return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, npccodec.New()) }},
{name: "scenes chunker", register: func() error { return scenes.Register(registries.Chunkers) }},
{name: "spells extractor", register: func() error { return spells.Register(registries.Extractors) }},
{name: "npcs extractor", register: func() error { return npcextract.Register(registries.Extractors) }},
{name: "spell-list appendorder merger", register: func() error {
return appendorder.RegisterTyped(registries.Mergers, dnd.SpellListKind, appendSpellLists)
}},
{name: "npc-list appendorder merger", register: func() error {
return appendorder.RegisterTyped(registries.Mergers, dnd.NPCListKind, appendNPCLists)
}},
{name: "spells normalizer", register: func() error { return spellnormalize.Register(registries.Normalizers) }},
{name: "npcs normalizer", register: func() error { return npcnormalize.Register(registries.Normalizers) }},
{name: "spell-list noop normalizer", register: func() error { return noop.RegisterTyped[dnd.SpellList](registries.Normalizers, dnd.SpellListKind) }},
{name: "npc-list noop normalizer", register: func() error { return noop.RegisterTyped[dnd.NPCList](registries.Normalizers, dnd.NPCListKind) }},
{name: "spell shape validator", register: func() error { return spellshape.Register(registries.Validators) }},
{name: "spell catalog validator", register: func() error { return spellcatalog.Register(registries.Validators) }},
{name: "spell source references validator", register: func() error { return spellsourcerefs.Register(registries.Validators) }},
{name: "spell source relatedness validator", register: func() error { return spellrelatedness.Register(registries.Validators) }},
{name: "npc shape validator", register: func() error { return npcshape.Register(registries.Validators) }},
{name: "npc identity validator", register: func() error { return npcidentity.Register(registries.Validators) }},
{name: "npc source references validator", register: func() error { return npcsourcerefs.Register(registries.Validators) }},
{name: "npc source relatedness validator", register: func() error { return npcrelatedness.Register(registries.Validators) }},
{name: "spell-list always accept validator", register: func() error {
return alwaysaccept.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
}},
{name: "spell-list always reject validator", register: func() error {
return alwaysreject.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
}},
{name: "npc-list always accept validator", register: func() error {
return alwaysaccept.RegisterTyped[dnd.NPCList](registries.Validators, dnd.NPCListKind)
}},
{name: "npc-list always reject validator", register: func() error {
return alwaysreject.RegisterTyped[dnd.NPCList](registries.Validators, dnd.NPCListKind)
}},
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
{name: "spells prompt assets", register: func() error { return spells.RegisterPromptAssets(assets) }},
{name: "npcs prompt assets", register: func() error { return npcextract.RegisterPromptAssets(assets) }},
}
for _, registration := range registrations {
if err := registration.register(); err != nil {
@@ -87,6 +112,33 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
}); err != nil {
return fmt.Errorf("register dnd spells normalize validator chain: %w", err)
}
if err := registries.ValidatorChains.Register(pipeline.ValidatorChainMapping{
Stage: pipeline.StageExtract,
Module: npcextract.Key,
Validators: []pipeline.ModuleBinding{
pipeline.Binding(validjson.Key),
pipeline.Binding(validjsonschema.Key),
pipeline.Binding(npcshape.Key),
pipeline.Binding(npcsourcerefs.Key),
pipeline.Binding(npcrelatedness.Key),
},
}); err != nil {
return fmt.Errorf("register dnd npcs validator chain: %w", err)
}
if err := registries.ValidatorChains.Register(pipeline.ValidatorChainMapping{
Stage: pipeline.StageNormalize,
Module: npcnormalize.Key,
Validators: []pipeline.ModuleBinding{
pipeline.Binding(validjson.Key),
pipeline.Binding(validjsonschema.Key),
pipeline.Binding(npcshape.Key),
pipeline.Binding(npcidentity.Key),
pipeline.Binding(npcsourcerefs.Key),
pipeline.Binding(npcrelatedness.Key),
},
}); err != nil {
return fmt.Errorf("register dnd npcs normalize validator chain: %w", err)
}
return nil
}
@@ -102,6 +154,25 @@ func appendSpellLists(values []dnd.SpellList) (dnd.SpellList, error) {
return combined, nil
}
func appendNPCLists(values []dnd.NPCList) (dnd.NPCList, error) {
count := 0
present := false
for _, value := range values {
if value.NPCs != nil {
present = true
}
count += len(value.NPCs)
}
if !present {
return dnd.NPCList{}, nil
}
combined := dnd.NPCList{NPCs: make([]dnd.NPC, 0, count)}
for _, value := range values {
combined.NPCs = append(combined.NPCs, value.NPCs...)
}
return combined, nil
}
func validateRegistries(registries pipeline.Registries, assets *llm.AssetRegistry) error {
switch {
case registries.Chunkers == nil:

View File

@@ -11,7 +11,9 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
)
@@ -22,10 +24,17 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
t.Fatalf("Register() error = %v, want nil", err)
}
assertContainsKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells"})
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, pipeline.DefaultNormalizeModule})
assertContainsArtifactKinds(t, registries.ArtifactCodecs.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind})
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells", npcextract.Key})
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, npcnormalize.Key, pipeline.DefaultNormalizeModule})
assertContainsArtifactKinds(t, registries.ArtifactCodecs.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind})
assertContainsArtifactKinds(t, registries.Mergers.RegisteredArtifactKinds(pipeline.DefaultMergeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind})
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(pipeline.DefaultNormalizeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind})
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(npcnormalize.Key), []contracts.ArtifactKind{dnd.NPCListKind})
assertContainsKeys(t, "validators", registries.Validators.RegisteredKeys(), []string{
"extract/dnd/npcs/shape",
"extract/dnd/npcs/source_refs",
"extract/dnd/npcs/source_relatedness",
"normalize/dnd/npcs/identity",
"extract/dnd/spells/catalog",
"extract/dnd/spells/shape",
"extract/dnd/spells/source_refs",
@@ -47,6 +56,30 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
if got := registries.ValidatorChains.Validators(pipeline.StageNormalize, spellnormalize.Key); !reflect.DeepEqual(got, wantChain) {
t.Fatalf("spell normalize validator chain = %#v, want %#v", got, wantChain)
}
npcExtractChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/npcs/shape"),
pipeline.Binding("extract/dnd/npcs/source_refs"),
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
}
if got := registries.ValidatorChains.Validators(pipeline.StageExtract, npcextract.Key); !reflect.DeepEqual(got, npcExtractChain) {
t.Fatalf("NPC extract validator chain = %#v, want %#v", got, npcExtractChain)
}
npcNormalizeChain := []pipeline.ModuleBinding{
pipeline.Binding("generic/valid_json"),
pipeline.Binding("generic/valid_json_schema"),
pipeline.Binding("extract/dnd/npcs/shape"),
pipeline.Binding("normalize/dnd/npcs/identity"),
pipeline.Binding("extract/dnd/npcs/source_refs"),
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
}
if got := registries.ValidatorChains.Validators(pipeline.StageNormalize, npcnormalize.Key); !reflect.DeepEqual(got, npcNormalizeChain) {
t.Fatalf("NPC normalize validator chain = %#v, want %#v", got, npcNormalizeChain)
}
if got := registries.ValidatorChains.Validators(pipeline.StageMerge, npcextract.Key); got != nil {
t.Fatalf("NPC merge validator chain = %#v, want absent", got)
}
assertAssetNamesContain(t, assets.PromptFS, []string{
"dnd.scenes/dnd.scenes.yaml",
"dnd.scenes/instructions.md",
@@ -60,10 +93,17 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
"dnd.spells/sharedassets/common-dnd-system.md",
"dnd.spells/sharedassets/common-dnd-transcript.md",
"dnd.spells/task.md",
"dnd.npcs/dnd.npcs.yaml",
"dnd.npcs/instructions.md",
"dnd.npcs/sharedassets/common-dnd-references.md",
"dnd.npcs/sharedassets/common-dnd-system.md",
"dnd.npcs/sharedassets/common-dnd-transcript.md",
"dnd.npcs/task.md",
})
assertAssetNamesContain(t, assets.SchemaFS, []string{
"dnd_scenes.v1.json",
"dnd_spells_llm.v1.json",
"dnd_npcs_llm.v1.json",
})
if spec, ok := registries.Chunkers.Spec("dnd/scenes"); !ok || spec.Key != "dnd/scenes" {
t.Fatalf("scene chunker spec = %#v, present = %t; want family-owned spec", spec, ok)
@@ -74,6 +114,33 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
if spec, ok := registries.Normalizers.Spec(spellnormalize.Key); !ok || spec.ArtifactKind != dnd.SpellListKind || spec.Stage != pipeline.StageNormalize {
t.Fatalf("spell normalizer spec = %#v, present = %t; want dnd spell-list artifact", spec, ok)
}
if spec, ok := registries.Extractors.Spec(npcextract.Key); !ok || spec.ArtifactKind != dnd.NPCListKind {
t.Fatalf("NPC extractor spec = %#v, present = %t; want dnd NPC-list artifact", spec, ok)
}
if spec, ok := registries.Normalizers.Spec(npcnormalize.Key); !ok || spec.ArtifactKind != dnd.NPCListKind || spec.Stage != pipeline.StageNormalize {
t.Fatalf("NPC normalizer spec = %#v, present = %t; want dnd NPC-list artifact", spec, ok)
}
}
func TestAppendNPCListsPreservesOrderAndArrayPresence(t *testing.T) {
tests := []struct {
name string
in []dnd.NPCList
want dnd.NPCList
}{
{name: "no values", in: nil, want: dnd.NPCList{}},
{name: "nil values", in: []dnd.NPCList{{}, {}}, want: dnd.NPCList{}},
{name: "present empty", in: []dnd.NPCList{{NPCs: []dnd.NPC{}}}, want: dnd.NPCList{NPCs: []dnd.NPC{}}},
{name: "ordered values", in: []dnd.NPCList{{NPCs: []dnd.NPC{{Name: "first"}}}, {NPCs: []dnd.NPC{{Name: "second"}}}}, want: dnd.NPCList{NPCs: []dnd.NPC{{Name: "first"}, {Name: "second"}}}},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := appendNPCLists(tt.in)
if err != nil || !reflect.DeepEqual(got, tt.want) {
t.Fatalf("appendNPCLists() = %#v, error = %v, want %#v", got, err, tt.want)
}
})
}
}
func TestRegisterRejectsMissingDNDDependenciesBeforeMutation(t *testing.T) {

View File

@@ -8,6 +8,8 @@ import (
const SpellListKind contracts.ArtifactKind = "dnd/spell-list"
const NPCListKind contracts.ArtifactKind = "dnd/npc-list"
type SpellList struct {
SpellCasts []SpellCast `json:"spell_casts"`
}
@@ -19,3 +21,21 @@ type SpellCast struct {
NarrativeDescription string `json:"narrative_description"`
SourceRefs []source.SourceRef `json:"source_refs"`
}
type NPCList struct {
NPCs []NPC `json:"npcs"`
}
type NPC struct {
ID string `json:"id"`
Name string `json:"name"`
Aliases []string `json:"aliases"`
Description string `json:"description"`
Relationships []NPCRelationship `json:"relationships"`
SourceRefs []source.SourceRef `json:"source_refs"`
}
type NPCRelationship struct {
Target string `json:"target"`
Relationship string `json:"relationship"`
}

View File

@@ -0,0 +1,53 @@
// Package diagnostics provides bounded, safe text for deterministic NPC
// validator decisions and warnings.
package diagnostics
import (
"fmt"
"strconv"
"strings"
)
const (
MaxIssues = 20
MaxDisplayedRunes = 128
MaxMessageBytes = 4096
)
func Truncate(value string) string {
runes := []rune(value)
if len(runes) <= MaxDisplayedRunes {
return value
}
return string(runes[:MaxDisplayedRunes-1]) + "…"
}
func Quote(value string) string { return strconv.Quote(Truncate(value)) }
func Aggregate(prefix string, issues []string) string {
displayed := make([]string, 0, min(len(issues), MaxIssues))
for len(displayed) < len(issues) && len(displayed) < MaxIssues {
issue := Truncate(issues[len(displayed)])
candidate := aggregateMessage(prefix, append(displayed, issue), len(issues)-len(displayed)-1)
if len([]byte(candidate)) > MaxMessageBytes {
break
}
displayed = append(displayed, issue)
}
return aggregateMessage(prefix, displayed, len(issues)-len(displayed))
}
func aggregateMessage(prefix string, issues []string, omitted int) string {
message := prefix + ": " + strings.Join(issues, ", ")
if omitted > 0 {
message += fmt.Sprintf("; %d additional issue(s) omitted", omitted)
}
return message
}
func min(left, right int) int {
if left < right {
return left
}
return right
}

View File

@@ -0,0 +1,84 @@
// Package identity validates the stable identity invariants of D&D NPC lists.
package identity
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
domainidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
)
const (
Key = "normalize/dnd/npcs/identity"
ReasonCode = "invalid_npc_identity"
policy = domainidentity.Policy
)
type Options struct{}
type Validator struct{}
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
func New(Options) *Validator { return &Validator{} }
func (v *Validator) Name() string { return Key }
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
}
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
if err := npcshape.Validate(req.Value); err != nil {
return contracts.ValidationResult{Approved: true}, nil
}
identityIssues := domainidentity.ValidateRegistry(req.Value.NPCs)
if len(identityIssues) == 0 {
return contracts.ValidationResult{Approved: true}, nil
}
issues := make([]string, len(identityIssues))
for index, issue := range identityIssues {
location := fmt.Sprintf("npcs[%d]", issue.RecordIndex)
if issue.AliasIndex >= 0 {
location += fmt.Sprintf(".aliases[%d]", issue.AliasIndex)
}
issues[index] = fmt.Sprintf("%s %s: %s", location, issue.Code, diagnostics.Quote(issue.Value))
}
return contracts.ValidationResult{
Approved: false,
ReasonCode: ReasonCode,
Message: diagnostics.Aggregate("invalid NPC identity", issues),
}, nil
}
func Spec() pipeline.ValidatorSpec {
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
}
func Register(registry *pipeline.ValidatorRegistry) error {
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(options), nil
})
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, err
}
return Options{}, nil
}
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }

View File

@@ -0,0 +1,107 @@
package identity
import (
"context"
"fmt"
"reflect"
"strings"
"testing"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestValidatorContractAndRegistration(t *testing.T) {
if _, err := DecodeOptions(nil); err != nil {
t.Fatalf("DecodeOptions(nil) error = %v, want nil", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
}
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != policy {
t.Fatalf("fingerprints = %#v, want identity policy", got)
}
want := pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
if got := Spec(); !reflect.DeepEqual(got, want) {
t.Fatalf("Spec() = %#v, want %#v", got, want)
}
registry := pipeline.NewValidatorRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
if registered, ok := registry.Spec(Key); !ok || !reflect.DeepEqual(registered, want) {
t.Fatalf("registered spec = %#v, ok = %t, want %#v", registered, ok, want)
}
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "validator registry") {
t.Fatalf("Register(nil) error = %v, want registry error", err)
}
}
func TestValidatorDefersShapeAndRejectsIdentityIssues(t *testing.T) {
validator := New(Options{})
shapeInvalid := dnd.NPCList{NPCs: []dnd.NPC{{Name: "missing description"}}}
result, err := validator.Validate(context.Background(), validationRequest(shapeInvalid))
if err != nil || !result.Approved {
t.Fatalf("shape-invalid result = %#v, error = %v, want deferred approval", result, err)
}
value := validNPCList(2)
value.NPCs[0].ID = "not-an-id"
value.NPCs[1].Aliases = []string{"Shared Alias"}
value.NPCs[0].Aliases = []string{"Shared Alias"}
result, err = validator.Validate(context.Background(), validationRequest(value))
if err != nil || result.Approved || result.ReasonCode != ReasonCode {
t.Fatalf("identity result = %#v, error = %v, want rejection", result, err)
}
for _, want := range []string{"invalid_id", "alias_owned_by_multiple_records", "npcs[0]", "npcs[1]"} {
if !strings.Contains(result.Message, want) {
t.Fatalf("identity message %q missing %q", result.Message, want)
}
}
}
func TestValidatorBoundsUnicodeDiagnostics(t *testing.T) {
value := validNPCList(30)
for index := range value.NPCs {
value.NPCs[index].ID = fmt.Sprintf("bad-%d", index)
value.NPCs[index].Name = strings.Repeat("火", 140) + fmt.Sprintf("-%d", index)
}
result, err := New(Options{}).Validate(context.Background(), validationRequest(value))
if err != nil || result.Approved {
t.Fatalf("bounded result = %#v, error = %v, want rejection", result, err)
}
if !utf8.ValidString(result.Message) || len([]rune(result.Message)) == 0 || len([]byte(result.Message)) > 4096 {
t.Fatalf("bounded message has invalid size/encoding: bytes=%d message=%q", len([]byte(result.Message)), result.Message)
}
if !strings.Contains(result.Message, "additional issue(s) omitted") || strings.Contains(result.Message, strings.Repeat("火", 140)) {
t.Fatalf("bounded message = %q, want omission and truncation", result.Message)
}
}
func validationRequest(value dnd.NPCList) contracts.TypedValidationRequest[dnd.NPCList] {
return contracts.TypedValidationRequest[dnd.NPCList]{Value: value}
}
func validNPCList(count int) dnd.NPCList {
value := dnd.NPCList{NPCs: make([]dnd.NPC, count)}
for index := range value.NPCs {
name := fmt.Sprintf("NPC %d", index)
value.NPCs[index] = dnd.NPC{
ID: "npc:sha256:0000000000000000000000000000000000000000000000000000000000000000",
Name: name,
Aliases: []string{},
Description: "description",
Relationships: []dnd.NPCRelationship{},
SourceRefs: []source.SourceRef{sourceRefForTest()},
}
}
return value
}
func sourceRefForTest() source.SourceRef {
return source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}
}

View File

@@ -0,0 +1,121 @@
package shape
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
)
const (
Key = "extract/dnd/npcs/shape"
ReasonCode = "invalid_npc_shape"
policy = "dnd.npcs.validator.shape.v1"
)
type Options struct{}
type Validator struct{}
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
func New(Options) *Validator { return &Validator{} }
func (v *Validator) Name() string { return Key }
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
}
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
issues := issuesFor(req.Value)
if len(issues) > 0 {
return rejection(diagnostics.Aggregate("invalid NPC shape", issues)), nil
}
return contracts.ValidationResult{Approved: true}, nil
}
func Validate(value dnd.NPCList) error {
issues := issuesFor(value)
if len(issues) == 0 {
return nil
}
return fmt.Errorf("%s", diagnostics.Aggregate("invalid NPC shape", issues))
}
func issuesFor(value dnd.NPCList) []string {
issues := make([]string, 0)
if value.NPCs == nil {
return []string{"npcs must be present"}
}
for index, npc := range value.NPCs {
prefix := fmt.Sprintf("npcs[%d]", index)
if strings.TrimSpace(npc.ID) == "" {
issues = append(issues, prefix+".id must not be empty")
}
if strings.TrimSpace(npc.Name) == "" {
issues = append(issues, prefix+".name must not be empty")
}
if npc.Aliases == nil {
issues = append(issues, prefix+".aliases must be present")
} else {
for aliasIndex, alias := range npc.Aliases {
if strings.TrimSpace(alias) == "" {
issues = append(issues, fmt.Sprintf("%s.aliases[%d] must not be empty: %s", prefix, aliasIndex, diagnostics.Quote(alias)))
}
}
}
if strings.TrimSpace(npc.Description) == "" {
issues = append(issues, prefix+".description must not be empty")
}
if npc.Relationships == nil {
issues = append(issues, prefix+".relationships must be present")
} else {
for relationshipIndex, relationship := range npc.Relationships {
relationshipPrefix := fmt.Sprintf("%s.relationships[%d]", prefix, relationshipIndex)
if strings.TrimSpace(relationship.Target) == "" {
issues = append(issues, relationshipPrefix+".target must not be empty: "+diagnostics.Quote(relationship.Target))
}
if strings.TrimSpace(relationship.Relationship) == "" {
issues = append(issues, relationshipPrefix+".relationship must not be empty: "+diagnostics.Quote(relationship.Relationship))
}
}
}
if len(npc.SourceRefs) == 0 {
issues = append(issues, prefix+".source_refs must not be empty")
}
}
return issues
}
func Spec() pipeline.ValidatorSpec {
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
}
func Register(registry *pipeline.ValidatorRegistry) error {
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(options), nil
})
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, err
}
return Options{}, nil
}
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
func rejection(message string) contracts.ValidationResult {
return contracts.ValidationResult{Approved: false, ReasonCode: ReasonCode, Message: message}
}

View File

@@ -0,0 +1,92 @@
package shape
import (
"context"
"strings"
"testing"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestValidatorApprovesWellFormedNPCPayload(t *testing.T) {
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validNPCList()))
if err != nil || !result.Approved {
t.Fatalf("Validate() = %#v, %v; want approval", result, err)
}
}
func TestValidatorRejectsRequiredShapeValues(t *testing.T) {
value := validNPCList()
value.NPCs[0].Aliases = nil
result, err := New(Options{}).Validate(context.Background(), requestWithValue(value))
if err != nil || result.Approved || result.ReasonCode != ReasonCode || !strings.Contains(result.Message, "aliases must be present") {
t.Fatalf("Validate() = %#v, %v; want bounded shape rejection", result, err)
}
missing := dnd.NPCList{}
result, err = New(Options{}).Validate(context.Background(), requestWithValue(missing))
if err != nil || result.Approved || result.ReasonCode != ReasonCode {
t.Fatalf("missing Validate() = %#v, %v; want shape rejection", result, err)
}
}
func TestValidatorBoundsDiagnosticsAndQuotesUnicode(t *testing.T) {
value := dnd.NPCList{NPCs: make([]dnd.NPC, 24)}
long := strings.Repeat("火", 220) + "\n\t"
for index := range value.NPCs {
value.NPCs[index] = dnd.NPC{ID: "candidate", Name: long, Aliases: []string{"\n\t"}, Description: "", Relationships: []dnd.NPCRelationship{{Target: " ", Relationship: " "}}, SourceRefs: []source.SourceRef{}}
}
result, err := New(Options{}).Validate(context.Background(), requestWithValue(value))
if err != nil || result.Approved || len([]byte(result.Message)) > diagnosticsMaxMessageBytes || !utf8.ValidString(result.Message) {
t.Fatalf("Validate() = %#v, %v; want bounded valid UTF-8 rejection", result, err)
}
if strings.Count(result.Message, "npcs[") > diagnosticsMaxIssues || !strings.Contains(result.Message, "additional issue(s) omitted") || !strings.Contains(result.Message, `\n\t`) {
t.Fatalf("message = %q, want bounded quoted diagnostics", result.Message)
}
}
func TestValidatorSpecCheckpointAndRegistration(t *testing.T) {
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != "dnd.npcs.validator.shape.v1" {
t.Fatalf("CheckpointFingerprints() = %#v, want local policy", got)
}
if spec := Spec(); spec.Key != Key || spec.ExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("Spec() = %#v, want deterministic shape validator", spec)
}
registry := pipeline.NewValidatorRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
t.Fatal("DecodeOptions() accepted unknown option")
}
}
func TestValidatorDoesNotMutateValue(t *testing.T) {
value := validNPCList()
before := value
_, err := New(Options{}).Validate(context.Background(), requestWithValue(value))
if err != nil || value.NPCs[0].Aliases[0] != before.NPCs[0].Aliases[0] {
t.Fatalf("Validate() mutated value: %#v", value)
}
}
func requestWithValue(value dnd.NPCList) contracts.TypedValidationRequest[dnd.NPCList] {
return contracts.TypedValidationRequest[dnd.NPCList]{Value: value}
}
func validNPCList() dnd.NPCList {
return dnd.NPCList{NPCs: []dnd.NPC{{
ID: "candidate", Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A guarded ranger.",
Relationships: []dnd.NPCRelationship{{Target: "Captain Vale", Relationship: "reports to"}},
SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}},
}}}
}
const (
diagnosticsMaxIssues = 20
diagnosticsMaxMessageBytes = 4096
)

View File

@@ -0,0 +1,79 @@
package sourcerefs
import (
"context"
"fmt"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
)
const (
Key = "extract/dnd/npcs/source_refs"
ReasonCode = "invalid_npc_source_refs"
policy = "dnd.npcs.validator.source_refs.v1"
)
type Options struct{}
type Validator struct{}
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
func New(Options) *Validator { return &Validator{} }
func (v *Validator) Name() string { return Key }
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
}
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
if err := npcshape.Validate(req.Value); err != nil {
return contracts.ValidationResult{Approved: true}, nil
}
issues := make([]string, 0)
for npcIndex, npc := range req.Value.NPCs {
for refIndex, ref := range npc.SourceRefs {
if err := source.ValidateRef(req.Source, ref); err != nil {
issues = append(issues, fmt.Sprintf("npcs[%d].source_refs[%d]: %s", npcIndex, refIndex, diagnostics.Truncate(err.Error())))
}
}
}
if len(issues) == 0 {
return contracts.ValidationResult{Approved: true}, nil
}
return rejection(diagnostics.Aggregate("invalid NPC source references", issues)), nil
}
func Spec() pipeline.ValidatorSpec {
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
}
func Register(registry *pipeline.ValidatorRegistry) error {
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(options), nil
})
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, err
}
return Options{}, nil
}
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
func rejection(message string) contracts.ValidationResult {
return contracts.ValidationResult{Approved: false, ReasonCode: ReasonCode, Message: message}
}

View File

@@ -0,0 +1,88 @@
package sourcerefs
import (
"context"
"fmt"
"strings"
"testing"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestValidatorApprovesValidSourceReferences(t *testing.T) {
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validDocument(), validNPCList()))
if err != nil || !result.Approved || len(result.Warnings) != 0 {
t.Fatalf("Validate() = %#v, %v; want approval", result, err)
}
}
func TestValidatorRejectsInvalidSourceReferences(t *testing.T) {
value := validNPCList()
value.NPCs[0].SourceRefs = []source.SourceRef{
{SourceID: "foreign", StartUnitID: 1, EndUnitID: 1},
{SourceID: "session", StartUnitID: 2, EndUnitID: 1},
{SourceID: "session", StartUnitID: 99, EndUnitID: 99},
}
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validDocument(), value))
if err != nil || result.Approved || result.ReasonCode != ReasonCode || !strings.Contains(result.Message, "npcs[0].source_refs[0]") {
t.Fatalf("Validate() = %#v, %v; want source-reference rejection", result, err)
}
}
func TestValidatorDefersMalformedShape(t *testing.T) {
value := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn"}}}
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validDocument(), value))
if err != nil || !result.Approved || result.ReasonCode != "" || result.Message != "" {
t.Fatalf("Validate() = %#v, %v; want shape deferral", result, err)
}
}
func TestValidatorBoundsDiagnosticsAndHandlesMissingDocument(t *testing.T) {
value := validNPCList()
value.NPCs[0].SourceRefs = make([]source.SourceRef, 24)
for index := range value.NPCs[0].SourceRefs {
value.NPCs[0].SourceRefs[index] = source.SourceRef{SourceID: strings.Repeat("火", 220) + "\n\t", StartUnitID: index + 1, EndUnitID: index + 1}
}
result, err := New(Options{}).Validate(context.Background(), requestWithValue(nil, value))
if err != nil || result.Approved || len([]byte(result.Message)) > 4096 || !utf8.ValidString(result.Message) {
t.Fatalf("Validate() = %#v, %v; want bounded missing-document rejection", result, err)
}
if !strings.Contains(result.Message, "additional issue(s) omitted") || !strings.Contains(result.Message, fmt.Sprintf("npcs[0].source_refs[%d]", 19)) {
t.Fatalf("message = %q, want bounded aggregate diagnostics", result.Message)
}
}
func TestValidatorSpecCheckpointAndRegistration(t *testing.T) {
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != "dnd.npcs.validator.source_refs.v1" {
t.Fatalf("CheckpointFingerprints() = %#v, want local policy", got)
}
if Spec().ExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("Spec() = %#v, want deterministic validator", Spec())
}
registry := pipeline.NewValidatorRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
t.Fatal("DecodeOptions() accepted unknown option")
}
}
func requestWithValue(doc *source.SourceDocument, value dnd.NPCList) contracts.TypedValidationRequest[dnd.NPCList] {
return contracts.TypedValidationRequest[dnd.NPCList]{Source: doc, Value: value}
}
func validDocument() *source.SourceDocument {
return &source.SourceDocument{ID: "session", Kind: "transcript", Format: "application/json", Digest: "sha256:session", Units: []source.SourceUnit{
{ID: 1, Kind: "message", Text: "Mira Thorn enters."},
{ID: 2, Kind: "message", Text: "The ranger watches."},
}}
}
func validNPCList() dnd.NPCList {
return dnd.NPCList{NPCs: []dnd.NPC{{ID: "candidate", Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A ranger.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 2}}}}}
}

View File

@@ -0,0 +1,114 @@
package sourcerelatedness
import (
"context"
"fmt"
"strings"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
)
const (
Key = "extract/dnd/npcs/source_relatedness"
WarningReasonCode = "npc_not_near_source"
policy = "dnd.npcs.validator.source_relatedness.v1"
)
type Options struct{}
type Validator struct{}
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
func New(Options) *Validator { return &Validator{} }
func (v *Validator) Name() string { return Key }
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
return contracts.ExecutionClassDeterministic
}
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
}
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
if err := npcshape.Validate(req.Value); err != nil {
return contracts.ValidationResult{Approved: true}, nil
}
var warnings []contracts.Warning
for npcIndex, npc := range req.Value.NPCs {
if npcAppearsInCitedText(req.Source, npc) {
continue
}
warnings = append(warnings, contracts.Warning{
Scope: fmt.Sprintf("npcs[%d]", npcIndex),
ReasonCode: WarningReasonCode,
Message: fmt.Sprintf("NPC %s was not found in cited source text", diagnostics.Quote(npc.Name)),
})
}
return contracts.ValidationResult{Approved: true, Warnings: warnings}, nil
}
func npcAppearsInCitedText(doc *source.SourceDocument, npc dnd.NPC) bool {
cited := citedTextKey(doc, npc.SourceRefs)
if cited == "" {
return false
}
if strings.Contains(cited, identity.ComparisonKey(npc.Name)) {
return true
}
for _, alias := range npc.Aliases {
if strings.Contains(cited, identity.ComparisonKey(alias)) {
return true
}
}
return false
}
func citedTextKey(doc *source.SourceDocument, refs []source.SourceRef) string {
if doc == nil {
return ""
}
var builder strings.Builder
for _, ref := range refs {
if err := source.ValidateRef(doc, ref); err != nil {
continue
}
start, _ := source.UnitIndex(doc, ref.StartUnitID)
end, _ := source.UnitIndex(doc, ref.EndUnitID)
for index := start; index <= end; index++ {
if builder.Len() > 0 {
builder.WriteByte(' ')
}
builder.WriteString(doc.Units[index].Text)
}
}
return identity.ComparisonKey(builder.String())
}
func Spec() pipeline.ValidatorSpec {
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
}
func Register(registry *pipeline.ValidatorRegistry) error {
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
options, err := DecodeOptions(request.Options)
if err != nil {
return nil, err
}
return New(options), nil
})
}
func DecodeOptions(options map[string]any) (Options, error) {
if err := pipeline.RejectUnknownOptions(options); err != nil {
return Options{}, err
}
return Options{}, nil
}
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }

View File

@@ -0,0 +1,81 @@
package sourcerelatedness
import (
"context"
"strings"
"testing"
"unicode/utf8"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestValidatorMatchesCanonicalNamesAndAliasesWithUnicodeVariants(t *testing.T) {
value := dnd.NPCList{NPCs: []dnd.NPC{
{ID: "one", Name: "O'Rin Thorn", Aliases: []string{}, Description: "A ranger.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
{ID: "two", Name: "Missing Name", Aliases: []string{"The Greencloak"}, Description: "A guard.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}},
}}
doc := &source.SourceDocument{ID: "session", Kind: "transcript", Format: "application/json", Digest: "sha256:session", Units: []source.SourceUnit{
{ID: 1, Kind: "message", Text: " orin\u2003thorn appears."},
{ID: 2, Kind: "message", Text: "The greencloak watches."},
}}
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: doc, Value: value})
if err != nil || !result.Approved || len(result.Warnings) != 0 {
t.Fatalf("Validate() = %#v, %v; want alias/canonical relatedness approval", result, err)
}
}
func TestValidatorWarnsAtMostOncePerNPCForUnrelatedCitations(t *testing.T) {
value := dnd.NPCList{NPCs: []dnd.NPC{
{ID: "one", Name: "Missing\nName", Aliases: []string{"Also Missing"}, Description: "A guard.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
}}
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), Value: value})
if err != nil || !result.Approved || len(result.Warnings) != 1 {
t.Fatalf("Validate() = %#v, %v; want one warning", result, err)
}
warning := result.Warnings[0]
if warning.Scope != "npcs[0]" || warning.ReasonCode != WarningReasonCode || !strings.Contains(warning.Message, `Missing\nName`) || strings.Contains(warning.Message, "Missing\nName") || !utf8.ValidString(warning.Message) {
t.Fatalf("warning = %#v, want safely quoted bounded warning", warning)
}
}
func TestValidatorDefersMalformedShapeAndInvalidRangesDoNotPanic(t *testing.T) {
invalidShape := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn"}}}
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), Value: invalidShape})
if err != nil || !result.Approved || len(result.Warnings) != 0 {
t.Fatalf("shape deferral = %#v, %v; want approval without warning", result, err)
}
invalidRange := dnd.NPCList{NPCs: []dnd.NPC{{ID: "one", Name: "Mira Thorn", Aliases: []string{}, Description: "A ranger.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 99, EndUnitID: 99}}}}}
result, err = New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), Value: invalidRange})
if err != nil || !result.Approved || len(result.Warnings) != 1 || result.Warnings[0].ReasonCode != WarningReasonCode {
t.Fatalf("invalid-range relatedness = %#v, %v; want one warning", result, err)
}
}
func TestValidatorUsesOnlyTranscriptEvidenceAndRegistersPolicy(t *testing.T) {
value := dnd.NPCList{NPCs: []dnd.NPC{{ID: "one", Name: "Opaque NPC", Aliases: []string{}, Description: "A guard.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}}}}
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{"glossary": {Items: []contracts.ReferenceItem{{Content: []byte("Opaque NPC")}}}}}
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), References: references, Value: value})
if err != nil || len(result.Warnings) != 1 {
t.Fatalf("reference-only relatedness = %#v, %v; want warning", result, err)
}
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != "dnd.npcs.validator.source_relatedness.v1" {
t.Fatalf("CheckpointFingerprints() = %#v, want local policy", got)
}
if Spec().ExecutionClass != contracts.ExecutionClassDeterministic {
t.Fatalf("Spec() = %#v, want deterministic validator", Spec())
}
registry := pipeline.NewValidatorRegistry()
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v", err)
}
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
t.Fatal("DecodeOptions() accepted unknown option")
}
}
func relatednessDocument() *source.SourceDocument {
return &source.SourceDocument{ID: "session", Kind: "transcript", Format: "application/json", Digest: "sha256:session", Units: []source.SourceUnit{{ID: 1, Kind: "message", Text: "The party waits."}}}
}

View File

@@ -0,0 +1,130 @@
package integration_test
import (
"os"
"path/filepath"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
)
func TestSequentialNPCOutputCanGroundIndependentSpellRun(t *testing.T) {
registries := productionNPCRegistries(t)
catalog := moduleCatalog(registries)
configValue := loadSequentialPipelineConfig(t)
npcEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs", Catalog: catalog})
if err != nil {
t.Fatalf("resolve NPC pipeline: %v", err)
}
npcClient := &fakeNPCProductionLLMClient{response: npcProductionResponse{NPCs: []npcProductionRecord{{
Name: "Mira Thorn",
Aliases: []string{"The Greencloak"},
Description: "A guarded ranger who watches the northern road.",
Relationships: []npcProductionRelationship{{
Target: "Captain Vale", Relationship: "reports to",
}},
SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 2}},
}}}}
npcOutput, err := runPreparedPipeline(t, registries, npcEffective.ResolvedPipeline, npcClient, pipeline.RunInput{RawInput: readNPCFixture(t)})
if err != nil {
t.Fatalf("run NPC pipeline: %v", err)
}
if len(npcOutput.NormalizeOutputs) != 1 || npcOutput.NormalizeOutputs[0].LaneID != "npcs" {
t.Fatalf("NPC normalized outputs = %#v, want one npcs lane", npcOutput.NormalizeOutputs)
}
npcPayload := npcOutput.NormalizeOutputs[0].Artifact.Content
if _, err := npccodec.New().Decode(npcPayload); err != nil {
t.Fatalf("decode normalized NPC payload: %v", err)
}
npcRunDir := t.TempDir()
npcPath := filepath.Join(npcRunDir, "lanes", "npcs.json")
if err := os.MkdirAll(filepath.Dir(npcPath), 0o700); err != nil {
t.Fatalf("create NPC output directory: %v", err)
}
if err := os.WriteFile(npcPath, npcPayload, 0o600); err != nil {
t.Fatalf("write NPC output payload: %v", err)
}
spellEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-spells", Catalog: catalog})
if err != nil {
t.Fatalf("resolve spell pipeline: %v", err)
}
spellEffective.ResolvedPipeline.ArtifactLanes[0].ExtractReferences.Bindings = []pipeline.ReferenceBinding{{
Stage: pipeline.StageExtract,
LaneID: "spells",
SlotName: spells.NPCRegistryReferenceSlot,
Source: npcPath,
BindingSource: contracts.ReferenceBindingSourceCLI,
}}
materialized, warnings, err := pipeline.MaterializeReferences(spellEffective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{WorkingDir: npcRunDir})
if err != nil {
t.Fatalf("materialize NPC registry reference: %v", err)
}
if len(warnings) != 0 {
t.Fatalf("reference materialization warnings = %#v, want none", warnings)
}
spellClient := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{{
Caster: "Mira Thorn",
Spell: "Cure Wounds",
Effect: "Restores the injured ally.",
NarrativeDescription: "Mira Thorn restores the ally after the fight.",
SourceRefs: responseSourceRefs("spell-session", 1, 1),
}}}}
spellOutput, err := runPreparedPipeline(t, registries, materialized, spellClient, pipeline.RunInput{RawInput: readDNDSpellsFixture(t)})
if err != nil {
t.Fatalf("run spell pipeline: %v", err)
}
if len(spellOutput.NormalizeOutputs) != 1 || spellOutput.NormalizeOutputs[0].LaneID != "spells" {
t.Fatalf("spell normalized outputs = %#v, want one spells lane", spellOutput.NormalizeOutputs)
}
spellValue := decodeRunnerSpellResponse(t, spellOutput.NormalizeOutputs[0].Artifact.Content)
if len(spellValue.SpellCasts) != 1 || spellValue.SpellCasts[0].Caster != "Mira Thorn" {
t.Fatalf("spell output = %#v, want one registry-grounded caster", spellValue)
}
if len(spellValue.SpellCasts[0].SourceRefs) != 1 || spellValue.SpellCasts[0].SourceRefs[0].SourceID != "spell-session" {
t.Fatalf("spell source refs = %#v, want current spell session only", spellValue.SpellCasts[0].SourceRefs)
}
if len(spellClient.requests) != 1 {
t.Fatalf("spell LLM requests = %d, want one", len(spellClient.requests))
}
registryInput := spellClient.requests[0].Inputs[spells.NPCRegistryReferenceSlot]
if string(registryInput.Content) != string(npcPayload) || registryInput.MediaType != npccodec.MediaType || registryInput.OriginURI != "" {
t.Fatalf("spell NPC prompt input = %#v, want canonical payload without origin", registryInput)
}
if len(spellOutput.Manifest.References) != 1 {
t.Fatalf("spell manifest references = %#v, want one NPC provenance entry", spellOutput.Manifest.References)
}
provenance := spellOutput.Manifest.References[0]
if provenance.Stage != "extract" || provenance.LaneID != "spells" || provenance.SlotName != spells.NPCRegistryReferenceSlot || provenance.BindingSource != contracts.ReferenceBindingSourceCLI || !strings.Contains(provenance.OriginURI, "npcs.json") {
t.Fatalf("spell NPC provenance = %#v, want extract CLI reference provenance", provenance)
}
metadata, ok := spellOutput.Manifest.ArtifactLanes[0].Metadata["extractor"].(map[string]any)
if !ok || metadata["npc_count"] != 1 || metadata["npc_registry_digest"] != registryInput.Digest {
t.Fatalf("spell extractor metadata = %#v, want NPC count and semantic digest", spellOutput.Manifest.ArtifactLanes[0].Metadata)
}
}
func loadSequentialPipelineConfig(t *testing.T) config.Config {
t.Helper()
data, err := os.ReadFile("testdata/dnd_npc_spell_sequential_pipeline.yml")
if err != nil {
t.Fatalf("read sequential pipeline config: %v", err)
}
fileConfig, err := config.ParseFileConfigYAML(data)
if err != nil {
t.Fatalf("parse sequential pipeline config: %v", err)
}
configValue := config.Default()
if err := configValue.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("apply sequential pipeline config: %v", err)
}
return configValue
}

View File

@@ -0,0 +1,242 @@
package integration_test
import (
"context"
"encoding/json"
"fmt"
"os"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
)
func TestRunnerProcessesSeriatimInputWithProductionDNDNPCPipeline(t *testing.T) {
raw := readNPCFixture(t)
doc, err := transcript.New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
if err != nil {
t.Fatalf("Parse() error = %v, want nil", err)
}
registries := productionNPCRegistries(t)
configValue := loadNPCPipelineConfig(t)
effective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs-fixture", Catalog: moduleCatalog(registries)})
if err != nil {
t.Fatalf("Resolve() error = %v, want nil", err)
}
client := &fakeNPCProductionLLMClient{response: npcProductionResponse{
NPCs: []npcProductionRecord{
{
Name: "Mira Thorn",
Aliases: []string{"The Greencloak"},
Description: "The first named NPC encountered.",
Relationships: []npcProductionRelationship{
{Target: "Hooded Guard", Relationship: "works with"},
},
SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 1}},
},
{
Name: "The Greencloak",
Aliases: []string{"Mira"},
Description: "A later description that must not replace the first.",
Relationships: []npcProductionRelationship{
{Target: "Hooded Guard", Relationship: "trusts"},
},
SourceRefs: []npcProductionSourceRef{{StartUnitID: 2, EndUnitID: 2}},
},
{
Name: "Hooded Guard",
Aliases: []string{},
Description: "An unnamed but distinguishable sentry.",
Relationships: []npcProductionRelationship{
{Target: "The Greencloak", Relationship: "reports to"},
},
SourceRefs: []npcProductionSourceRef{{StartUnitID: 3, EndUnitID: 3}},
},
}},
}
output, err := runPreparedPipeline(t, registries, effective.ResolvedPipeline, client, pipeline.RunInput{RawInput: raw})
if err != nil {
t.Fatalf("Run() error = %v, want nil", err)
}
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("normalize outputs = %d, want one NPC output; rejected=%#v", len(output.NormalizeOutputs), output.Rejected)
}
serialized := output.NormalizeOutputs[0]
if serialized.LaneID != "npcs" || serialized.NormalizerKey != npcs.Key || serialized.Artifact.Schema.ID != npccodec.SchemaID || serialized.Artifact.Schema.Version != npccodec.SchemaVersion {
t.Fatalf("serialized output = %#v, want durable NPC lane schema", serialized)
}
value, err := npccodec.New().Decode(serialized.Artifact.Content)
if err != nil {
t.Fatalf("Decode(output) error = %v, want durable NPC payload", err)
}
if len(value.NPCs) != 2 {
t.Fatalf("NPC output = %#v, want repeated name consolidated and group/PC omitted", value.NPCs)
}
first, second := value.NPCs[0], value.NPCs[1]
if first.Name != "Mira Thorn" || first.Description != "The first named NPC encountered." || !reflect.DeepEqual(first.Aliases, []string{"The Greencloak", "Mira"}) {
t.Fatalf("first NPC = %#v, want consolidated Mira identity", first)
}
if first.ID != identity.DeriveID(first.Name) || second.Name != "Hooded Guard" || second.ID != identity.DeriveID(second.Name) {
t.Fatalf("NPC IDs = %q/%q, want derived IDs", first.ID, second.ID)
}
if first.Relationships[0].Target != "Hooded Guard" || second.Relationships[0].Target != "Mira Thorn" {
t.Fatalf("relationship targets = %q/%q, want canonical target rewrite", first.Relationships[0].Target, second.Relationships[0].Target)
}
for _, npc := range value.NPCs {
for _, ref := range npc.SourceRefs {
if ref.SourceID != doc.ID {
t.Fatalf("NPC source ref = %#v, want source document %q", ref, doc.ID)
}
}
}
if !hasNPCWarning(output.Warnings, "duplicate_npc_collapsed") || !hasNPCWarning(output.Warnings, "relationship_target_canonicalized") {
t.Fatalf("warnings = %#v, want consolidation and target warnings", output.Warnings)
}
if output.Manifest.ValidationStatus != "approved" || len(output.Manifest.ArtifactLanes) != 1 {
t.Fatalf("manifest = %#v, want approved NPC lane", output.Manifest)
}
lane := output.Manifest.ArtifactLanes[0]
if lane.ID != "npcs" || lane.Extractor != npcs.Key || lane.Merger != pipeline.DefaultMergeModule || lane.Normalizer != npcs.Key {
t.Fatalf("manifest lane = %#v, want NPC production composition", lane)
}
normalizerMetadata, ok := lane.Metadata["normalizer"].(map[string]any)
if !ok || normalizerMetadata["identity_policy"] != identity.Policy || normalizerMetadata["normalization_policy"] != "dnd.npcs.normalize.v1" {
t.Fatalf("normalizer metadata = %#v, want identity and normalization policies", lane.Metadata)
}
var npcOutputFile *contracts.OutputFile
for index := range output.OutputFiles {
if output.OutputFiles[index].Name == "lanes/npcs.json" {
npcOutputFile = &output.OutputFiles[index]
break
}
}
if npcOutputFile == nil || npcOutputFile.ContentType != npccodec.MediaType {
t.Fatalf("output files = %#v, want JSON NPC lane file", output.OutputFiles)
}
if len(client.requests) != 1 || client.requests[0].PromptID != npcs.PromptID {
t.Fatalf("LLM requests = %#v, want one NPC prompt request", client.requests)
}
}
type npcProductionResponse struct {
NPCs []npcProductionRecord `json:"npcs"`
}
type npcProductionRecord struct {
Name string `json:"name"`
Aliases []string `json:"aliases"`
Description string `json:"description"`
Relationships []npcProductionRelationship `json:"relationships"`
SourceRefs []npcProductionSourceRef `json:"source_refs"`
}
type npcProductionRelationship struct {
Target string `json:"target"`
Relationship string `json:"relationship"`
}
type npcProductionSourceRef struct {
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
}
type fakeNPCProductionLLMClient struct {
response npcProductionResponse
requests []contracts.StructuredCompletionRequest
}
func (client *fakeNPCProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
client.requests = append(client.requests, req)
content, err := json.Marshal(client.response)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
if err := json.Unmarshal(content, out); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate NPC structured target: %w", err)
}
return contracts.StructuredCompletionResponse{Content: content}, nil
}
func productionNPCRegistries(t *testing.T) pipeline.Registries {
t.Helper()
registries := pipeline.Registries{
Inputs: pipeline.NewInputAdapterRegistry(),
Chunkers: pipeline.NewChunkerRegistry(),
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
Extractors: pipeline.NewExtractorRegistry(),
Mergers: pipeline.NewMergerRegistry(),
Normalizers: pipeline.NewNormalizerRegistry(),
Validators: pipeline.NewValidatorRegistry(),
ValidatorChains: pipeline.NewValidatorChainRegistry(),
Outputs: pipeline.NewOutputEncoderRegistry(),
}
assets := llm.NewAssetRegistry()
for _, registration := range []struct {
name string
fn func(pipeline.Registries, *llm.AssetRegistry) error
}{
{name: "generic", fn: genericregister.Register},
{name: "seriatim", fn: seriatimregister.Register},
{name: "dnd", fn: dndregister.Register},
} {
if err := registration.fn(registries, assets); err != nil {
t.Fatalf("register %s modules: %v", registration.name, err)
}
}
return registries
}
func moduleCatalog(registries pipeline.Registries) pipeline.ModuleCatalog {
return pipeline.ModuleCatalog{
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs,
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
}
}
func loadNPCPipelineConfig(t *testing.T) config.Config {
t.Helper()
data, err := os.ReadFile("testdata/dnd_npcs_pipeline.yml")
if err != nil {
t.Fatalf("ReadFile(dnd_npcs_pipeline.yml) error = %v", err)
}
fileConfig, err := config.ParseFileConfigYAML(data)
if err != nil {
t.Fatalf("ParseFileConfigYAML() error = %v", err)
}
cfg := config.Default()
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
t.Fatalf("ApplyFileConfig() error = %v", err)
}
return cfg
}
func readNPCFixture(t *testing.T) []byte {
t.Helper()
raw, err := os.ReadFile("testdata/seriatim_npc_session.json")
if err != nil {
t.Fatalf("ReadFile(seriatim_npc_session.json) error = %v", err)
}
return raw
}
func hasNPCWarning(warnings []contracts.Warning, reason string) bool {
for _, warning := range warnings {
if warning.ReasonCode == reason && strings.HasPrefix(warning.Scope, "npcs[") {
return true
}
}
return false
}

View File

@@ -0,0 +1,25 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints: {}
debug:
directory: ./notarius-debug
pipelines:
dnd-npcs:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
normalize: dnd/npcs
dnd-spells:
input: seriatim
chunk: generic
artifacts:
spells:
extract: dnd/spells
normalize: dnd/spells

View File

@@ -0,0 +1,20 @@
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
mode: bypass
checkpoints: {}
debug:
directory: ./notarius-debug
pipelines:
dnd-npcs-fixture:
input: seriatim
chunk: generic
artifacts:
npcs:
extract:
module: dnd/npcs
retries: 2
normalize: dnd/npcs
output: json

View File

@@ -0,0 +1,43 @@
{
"metadata": {
"id": "npc-session",
"title": "Synthetic D&D NPC session"
},
"segments": [
{
"id": 1,
"start": 0,
"end": 4,
"speaker": "Aria",
"text": "Aria watches Mira Thorn, the Greencloak, enter the ruined hall."
},
{
"id": 2,
"start": 4,
"end": 8,
"speaker": "DM",
"text": "Mira Thorn asks the party to follow the old road."
},
{
"id": 3,
"start": 8,
"end": 12,
"speaker": "DM",
"text": "A hooded guard opens the side gate and waits in silence."
},
{
"id": 4,
"start": 12,
"end": 16,
"speaker": "DM",
"text": "Three identical guards surround the interchangeable group."
},
{
"id": 5,
"start": 16,
"end": 20,
"speaker": "Aria",
"text": "Aria keeps watch while the named NPCs leave the hall."
}
]
}