Compare commits
5 Commits
e2ab01f9d2
...
20cfbfd311
| Author | SHA1 | Date | |
|---|---|---|---|
| 20cfbfd311 | |||
| fb043325e1 | |||
| 06c0259788 | |||
| 3d5fd9dc05 | |||
| 3ba2c62cc1 |
10
docs/cli.md
10
docs/cli.md
@@ -94,6 +94,16 @@ go run ./cmd/notarius run dnd-session \
|
|||||||
--reference spells.extract.glossary=./campaign-glossary.txt
|
--reference spells.extract.glossary=./campaign-glossary.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For the operator-driven NPC-to-spell workflow, bind the normalized NPC lane
|
||||||
|
payload from the completed NPC run to the spell extractor:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/notarius run dnd-spells \
|
||||||
|
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||||
|
--input examples/seriatim-minimal-transcript.json \
|
||||||
|
--reference spells.extract.npcs=./npc-run/lanes/npcs.json
|
||||||
|
```
|
||||||
|
|
||||||
The same grammar can target chunk, merge, and normalize slots when the configured
|
The same grammar can target chunk, merge, and normalize slots when the configured
|
||||||
modules declare them:
|
modules declare them:
|
||||||
|
|
||||||
|
|||||||
@@ -21,8 +21,10 @@ The explicit-path option is defined in the [CLI reference](cli.md).
|
|||||||
|
|
||||||
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
|
- [Minimal D&D spell configuration](../examples/dnd-spells.config.yml)
|
||||||
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
|
- [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml)
|
||||||
|
- [D&D NPC configuration](../examples/dnd-npcs.config.yml)
|
||||||
|
- [Sequential D&D NPC and spell configuration](../examples/dnd-npc-spell-sequential.config.yml)
|
||||||
|
|
||||||
Both are complete version 3 files. The fragments below illustrate individual
|
All are complete version 3 files. The fragments below illustrate individual
|
||||||
fields and are not alternate complete configurations.
|
fields and are not alternate complete configurations.
|
||||||
|
|
||||||
## Top-Level Fields
|
## Top-Level Fields
|
||||||
@@ -283,9 +285,11 @@ production validators do not call the LLM and must not set `llm_profile`.
|
|||||||
| chunk | `generic` | Splits source units into ordered chunks. |
|
| chunk | `generic` | Splits source units into ordered chunks. |
|
||||||
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
| chunk | `dnd/scenes` | Uses an LLM to split transcript source units into D&D scenes. |
|
||||||
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
|
| extract | `dnd/spells` | Extracts typed D&D spell-list artifacts. |
|
||||||
|
| extract | `dnd/npcs` | Extracts typed D&D NPC-list artifacts. |
|
||||||
| merge | `appendorder` | Combines typed artifacts in chunk order. |
|
| merge | `appendorder` | Combines typed artifacts in chunk order. |
|
||||||
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
|
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
|
||||||
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
|
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
|
||||||
|
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical identity and aliases. |
|
||||||
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
| output | `json` | Produces JSON output files for normalized `application/json` lanes. |
|
||||||
|
|
||||||
## Implemented Production Validators
|
## Implemented Production Validators
|
||||||
@@ -300,6 +304,10 @@ production validators do not call the LLM and must not set `llm_profile`.
|
|||||||
| `extract/dnd/spells/catalog` | deterministic | Rejects spell-list artifacts containing names outside the effective SRD and overlay catalog. |
|
| `extract/dnd/spells/catalog` | deterministic | Rejects spell-list artifacts containing names outside the effective SRD and overlay catalog. |
|
||||||
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
| `extract/dnd/spells/source_refs` | deterministic | Rejects missing or invalid D&D spell source references. |
|
||||||
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
| `extract/dnd/spells/source_relatedness` | deterministic | Emits warnings when a spell name is not found near its cited source text. |
|
||||||
|
| `extract/dnd/npcs/shape` | deterministic | Rejects malformed D&D NPC-list artifacts. |
|
||||||
|
| `extract/dnd/npcs/source_refs` | deterministic | Rejects missing or invalid D&D NPC source references. |
|
||||||
|
| `extract/dnd/npcs/source_relatedness` | deterministic | Emits warnings when an NPC name or alias is not found near its cited source text. |
|
||||||
|
| `normalize/dnd/npcs/identity` | deterministic | Rejects invalid canonical IDs, aliases, and cross-record identity collisions. |
|
||||||
|
|
||||||
The production default chain for `dnd/spells` is used for both its extract and
|
The production default chain for `dnd/spells` is used for both its extract and
|
||||||
normalize stages:
|
normalize stages:
|
||||||
@@ -314,8 +322,27 @@ validators:
|
|||||||
- extract/dnd/spells/source_relatedness
|
- extract/dnd/spells/source_relatedness
|
||||||
```
|
```
|
||||||
|
|
||||||
No other production module currently has a default validator chain. Empty
|
The production default chain for `dnd/npcs` uses the extraction chain for the
|
||||||
chains approve output by default.
|
extract stage and the identity chain for normalize-stage output:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
extract:
|
||||||
|
validators:
|
||||||
|
- generic/valid_json
|
||||||
|
- generic/valid_json_schema
|
||||||
|
- extract/dnd/npcs/shape
|
||||||
|
- extract/dnd/npcs/source_refs
|
||||||
|
- extract/dnd/npcs/source_relatedness
|
||||||
|
normalize:
|
||||||
|
validators:
|
||||||
|
- generic/valid_json
|
||||||
|
- generic/valid_json_schema
|
||||||
|
- normalize/dnd/npcs/identity
|
||||||
|
- extract/dnd/npcs/source_refs
|
||||||
|
- extract/dnd/npcs/source_relatedness
|
||||||
|
```
|
||||||
|
|
||||||
|
Empty chains approve output by default.
|
||||||
|
|
||||||
The `generic` chunker accepts:
|
The `generic` chunker accepts:
|
||||||
|
|
||||||
@@ -344,6 +371,17 @@ not allow multiple files. Its format is defined in the
|
|||||||
The extractor uses campaign references only as supporting disambiguation
|
The extractor uses campaign references only as supporting disambiguation
|
||||||
material; spell casts still must be present in the source transcript.
|
material; spell casts still must be present in the source transcript.
|
||||||
|
|
||||||
|
It also declares an optional `npcs` slot for a normalized NPC artifact. The
|
||||||
|
slot accepts exactly one `application/json` file no larger than 1 MiB. During
|
||||||
|
extractor preparation Notarius strictly decodes and identity-validates the
|
||||||
|
artifact, then gives the model canonical JSON for caster-name grounding.
|
||||||
|
Registry source references may belong to the NPC-producing session and are
|
||||||
|
provenance only; they are not spell evidence. The bound registry contributes a
|
||||||
|
semantic digest and NPC count to extractor metadata and checkpoint identity,
|
||||||
|
while its names, aliases, content, and path do not appear there. When absent,
|
||||||
|
the prompt receives the exact empty value `{"npcs":[]}` and no registry
|
||||||
|
provenance or fingerprint is recorded.
|
||||||
|
|
||||||
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
|
The `dnd/spells` normalizer declares the same optional `spell_catalog` slot.
|
||||||
When an overlay is used, bind it independently under
|
When an overlay is used, bind it independently under
|
||||||
`artifacts.<lane>.normalize.references.spell_catalog`; normalize-stage
|
`artifacts.<lane>.normalize.references.spell_catalog`; normalize-stage
|
||||||
@@ -351,6 +389,12 @@ references are local to that stage and are not inherited from extraction. The
|
|||||||
normalizer uses the embedded SRD catalog when no normalize-stage overlay is
|
normalizer uses the embedded SRD catalog when no normalize-stage overlay is
|
||||||
bound.
|
bound.
|
||||||
|
|
||||||
|
The `dnd/npcs` extractor declares the same optional campaign slots as the spell
|
||||||
|
extractor, but it does not declare the `npcs` registry slot. Its normalizer
|
||||||
|
accepts no references. To pass an NPC result to a later spell run, bind the
|
||||||
|
normalized payload explicitly at runtime; the maintained sequential example
|
||||||
|
documents that operator workflow.
|
||||||
|
|
||||||
## State Surfaces
|
## State Surfaces
|
||||||
|
|
||||||
The `output`, `cache`, and `debug` top-level fields select independent physical
|
The `output`, `cache`, and `debug` top-level fields select independent physical
|
||||||
|
|||||||
106
docs/integrations/dnd-npc-artifacts.md
Normal file
106
docs/integrations/dnd-npc-artifacts.md
Normal file
@@ -0,0 +1,106 @@
|
|||||||
|
# D&D NPC Artifact
|
||||||
|
|
||||||
|
This document defines the durable D&D NPC-list artifact, its JSON codec, and
|
||||||
|
the selectable production NPC pipeline. The normalized JSON payload can be
|
||||||
|
passed explicitly to the spell extractor as an optional caster-name registry;
|
||||||
|
it remains a reference, not spell evidence.
|
||||||
|
|
||||||
|
## Identity
|
||||||
|
|
||||||
|
- Artifact kind: `dnd/npc-list`
|
||||||
|
- Durable schema ID: `notarius.dnd.npcs`
|
||||||
|
- Durable schema name: `notarius_dnd_npcs_v1`
|
||||||
|
- Durable schema version: `v1`
|
||||||
|
- Media type: `application/json`
|
||||||
|
- Identity policy: `dnd.npcs.identity.v1`
|
||||||
|
|
||||||
|
The durable JSON Schema is owned by the D&D NPC codec. NPC IDs are derived from
|
||||||
|
the Unicode-normalized, case-folded canonical name using the identity policy.
|
||||||
|
The durable codec enforces the artifact shape and ID syntax; registry identity
|
||||||
|
validation remains a separate deterministic concern.
|
||||||
|
|
||||||
|
## Output Shape
|
||||||
|
|
||||||
|
The payload is one object with a required top-level `npcs` array:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"npcs": []}
|
||||||
|
```
|
||||||
|
|
||||||
|
The array may be empty. Every object and nested object rejects unknown fields.
|
||||||
|
|
||||||
|
## NPC Fields
|
||||||
|
|
||||||
|
Each NPC contains exactly these required fields:
|
||||||
|
|
||||||
|
- `id`: `npc:sha256:` followed by 64 lowercase hexadecimal characters;
|
||||||
|
- `name`: the canonical display name;
|
||||||
|
- `aliases`: an array of alternate display names, which may be empty;
|
||||||
|
- `description`: a concise description;
|
||||||
|
- `relationships`: an array of target/relationship objects, which may be empty;
|
||||||
|
- `source_refs`: at least one source reference supporting the NPC record.
|
||||||
|
|
||||||
|
Each relationship contains required `target` and `relationship` strings. Each
|
||||||
|
source reference contains required `source_id`, `start_unit_id`, and
|
||||||
|
`end_unit_id`; unit IDs are positive integers. Source document identity, unit
|
||||||
|
existence, and range ordering are validated by the source-reference validator
|
||||||
|
when the artifact is used by a pipeline.
|
||||||
|
|
||||||
|
## Codec Boundary
|
||||||
|
|
||||||
|
`EncodeCandidate` and `DecodeCandidate` provide strict single-value JSON
|
||||||
|
serialization while preserving typed values that still need semantic
|
||||||
|
validation. `Encode` and `Decode` are the approved-artifact boundary and
|
||||||
|
require all durable structural fields, non-empty required strings, valid source
|
||||||
|
reference shapes, and the NPC ID pattern.
|
||||||
|
|
||||||
|
Codec metadata contains only `npc_count`. Schema bytes and returned metadata
|
||||||
|
are independent values so callers cannot mutate codec-owned state.
|
||||||
|
|
||||||
|
## Production Pipeline
|
||||||
|
|
||||||
|
The production identities are:
|
||||||
|
|
||||||
|
- extractor: `dnd/npcs`;
|
||||||
|
- artifact kind: `dnd/npc-list`;
|
||||||
|
- normalizer: `dnd/npcs`; and
|
||||||
|
- durable schema: `notarius.dnd.npcs`, version `v1`, media type
|
||||||
|
`application/json`.
|
||||||
|
|
||||||
|
The extractor maps private model records to the current source identity and
|
||||||
|
assigns deterministic IDs. Extraction validation checks shape, source
|
||||||
|
references, and source relatedness. The normalizer then consolidates records
|
||||||
|
by canonical identity or canonical-name/alias matches, preserves the first
|
||||||
|
record's display and output position, unions relationships and exact evidence,
|
||||||
|
rewrites unambiguous relationship targets to canonical names, and validates
|
||||||
|
the retained registry's identity. No LLM is used for consolidation.
|
||||||
|
|
||||||
|
The default extraction chain is `generic/valid_json`,
|
||||||
|
`generic/valid_json_schema`, `extract/dnd/npcs/shape`,
|
||||||
|
`extract/dnd/npcs/source_refs`, and
|
||||||
|
`extract/dnd/npcs/source_relatedness`. The normalize chain adds
|
||||||
|
`normalize/dnd/npcs/identity` before the source-reference and relatedness
|
||||||
|
checks. Relatedness emits bounded warnings when an NPC canonical name or
|
||||||
|
alias is not present near its cited transcript text; opaque campaign
|
||||||
|
references may explain such a warning but do not become evidence.
|
||||||
|
|
||||||
|
## Manifest And Sequential Consumption
|
||||||
|
|
||||||
|
The NPC extractor records prompt and response-schema identities. The durable
|
||||||
|
codec records only `npc_count`; raw names, aliases, descriptions, source
|
||||||
|
references, and payload bytes stay in the lane file rather than manifest
|
||||||
|
metadata. The normalized lane is independently reusable as a file reference:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/notarius run dnd-spells \
|
||||||
|
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||||
|
--input examples/seriatim-minimal-transcript.json \
|
||||||
|
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The spell extractor strictly decodes and identity-validates this file, accepts
|
||||||
|
source references belonging to another session as registry provenance, and
|
||||||
|
uses only canonical names and aliases for caster grounding. Those NPC source
|
||||||
|
references are never accepted as spell evidence. The spell run's manifest
|
||||||
|
keeps raw file provenance under `references` and records only the prepared
|
||||||
|
registry's semantic digest and count in extractor metadata.
|
||||||
@@ -94,6 +94,22 @@ Reference slot keys and accepted file types are defined in
|
|||||||
supporting disambiguation material, not source evidence, and are not
|
supporting disambiguation material, not source evidence, and are not
|
||||||
addressable through `source_refs`.
|
addressable through `source_refs`.
|
||||||
|
|
||||||
|
## Optional NPC Grounding
|
||||||
|
|
||||||
|
The `dnd/spells` extractor accepts an optional `npcs` reference containing one
|
||||||
|
normalized NPC artifact as `application/json`, up to 1 MiB. Preparation uses
|
||||||
|
the approved NPC codec and identity policy to validate the file, re-encodes
|
||||||
|
canonical durable JSON, and supplies that JSON as a spell-owned prompt input.
|
||||||
|
It helps the model prefer canonical caster names and recognize aliases; it
|
||||||
|
does not establish that a spell was cast.
|
||||||
|
|
||||||
|
NPC source references may identify the run that produced the registry or any
|
||||||
|
other session. They remain registry provenance and are never copied into a
|
||||||
|
spell cast's `source_refs`; every spell evidence range must still identify the
|
||||||
|
current transcript. When the slot is absent, the prompt receives exactly
|
||||||
|
`{"npcs":[]}` and the run has no NPC reference provenance or NPC checkpoint
|
||||||
|
fingerprint.
|
||||||
|
|
||||||
## Normalization Behavior
|
## Normalization Behavior
|
||||||
|
|
||||||
When the `dnd/spells` normalizer is selected, each recognized spell name is
|
When the `dnd/spells` normalizer is selected, each recognized spell name is
|
||||||
@@ -153,7 +169,9 @@ manifest metadata:
|
|||||||
"response_schema_sha256": "sha256:...",
|
"response_schema_sha256": "sha256:...",
|
||||||
"catalog_base_id": "dnd-5e-2014-srd-spells",
|
"catalog_base_id": "dnd-5e-2014-srd-spells",
|
||||||
"catalog_digest": "sha256:...",
|
"catalog_digest": "sha256:...",
|
||||||
"catalog_overlay_ids": ["campaign.example"]
|
"catalog_overlay_ids": ["campaign.example"],
|
||||||
|
"npc_registry_digest": "sha256:...",
|
||||||
|
"npc_count": 3
|
||||||
},
|
},
|
||||||
"normalizer": {
|
"normalizer": {
|
||||||
"catalog_base_id": "dnd-5e-2014-srd-spells",
|
"catalog_base_id": "dnd-5e-2014-srd-spells",
|
||||||
@@ -172,8 +190,15 @@ identity fields when that module is selected. Overlay origin, media type, byte
|
|||||||
size, and raw digest are recorded separately in the manifest's reference
|
size, and raw digest are recorded separately in the manifest's reference
|
||||||
provenance; see the [JSON output contract](json-output.md#manifestjson).
|
provenance; see the [JSON output contract](json-output.md#manifestjson).
|
||||||
|
|
||||||
|
The `npc_registry_digest` and `npc_count` fields in the example are present only
|
||||||
|
when the optional NPC registry is bound. They contain no NPC names, aliases,
|
||||||
|
source references, paths, or raw bytes.
|
||||||
|
|
||||||
The extractor's prompt hash, private response-schema hash, and effective catalog
|
The extractor's prompt hash, private response-schema hash, and effective catalog
|
||||||
digest also contribute independently scoped semantic checkpoint fingerprints.
|
digest also contribute independently scoped semantic checkpoint fingerprints.
|
||||||
Changing any of those prepared contracts intentionally produces a cold
|
Changing any of those prepared contracts intentionally produces a cold
|
||||||
checkpoint miss. Fingerprints contain only digests, never prompt, schema,
|
checkpoint miss. Fingerprints contain only digests, never prompt, schema,
|
||||||
catalog, or reference content.
|
catalog, or reference content. When an NPC registry is bound, its semantic
|
||||||
|
digest contributes an additional local `npc_registry` fingerprint; the
|
||||||
|
manifest metadata contains only that digest and `npc_count`. Raw NPC file
|
||||||
|
provenance remains independently recorded in the manifest's `references` list.
|
||||||
|
|||||||
@@ -151,7 +151,8 @@ accepts only artifacts whose codec media type is `application/json`. The file
|
|||||||
contains the codec-owned JSON bytes pretty-printed.
|
contains the codec-owned JSON bytes pretty-printed.
|
||||||
|
|
||||||
The schema of each lane payload is owned by that artifact contract. For the
|
The schema of each lane payload is owned by that artifact contract. For the
|
||||||
current D&D spell lane, see [D&D Spell Artifact](dnd-spell-artifacts.md).
|
current D&D lanes, see [D&D Spell Artifact](dnd-spell-artifacts.md) and
|
||||||
|
[D&D NPC Artifact](dnd-npc-artifacts.md).
|
||||||
|
|
||||||
## `rejected.json`
|
## `rejected.json`
|
||||||
|
|
||||||
|
|||||||
@@ -39,9 +39,9 @@ without exposing Scriptorium types through stage contracts.
|
|||||||
7. injecting that one shared client into complete pipeline preparation before
|
7. injecting that one shared client into complete pipeline preparation before
|
||||||
the source file is read or the runner is invoked.
|
the source file is read or the runner is invoked.
|
||||||
|
|
||||||
The D&D scene chunker and spell extractor retain this injected client and use
|
The D&D scene chunker and spell and NPC extractors retain this injected client
|
||||||
it for every structured completion. Operation requests do not carry an LLM
|
and use it for every structured completion. Operation requests do not carry an
|
||||||
client.
|
LLM client.
|
||||||
|
|
||||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
||||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
and validator bindings. It prepares a small internal check prompt for each ID so
|
||||||
@@ -104,11 +104,15 @@ The small framework registry contains only generic test schemas; production
|
|||||||
schemas remain package-owned.
|
schemas remain package-owned.
|
||||||
|
|
||||||
The spell extractor's package-owned prompt declares a required
|
The spell extractor's package-owned prompt declares a required
|
||||||
`application/json` `spell_catalog` input. The extractor generates that input
|
`application/json` `spell_catalog` input and an optional `application/json`
|
||||||
from its prepared effective catalog as `{"spell_names":[...]}` using sorted
|
`npcs` input. The extractor generates the catalog input from its prepared
|
||||||
canonical names only. Its input digest covers those generated bytes; manifests
|
effective catalog as `{"spell_names":[...]}` using sorted canonical names only.
|
||||||
record catalog identity and digest rather than names, aliases, overlay bytes,
|
When an NPC registry is bound, it strictly decodes and identity-validates one
|
||||||
or source metadata.
|
durable artifact, re-encodes canonical JSON, and generates a semantic digest
|
||||||
|
over those bytes. The unbound input is exactly `{"npcs":[]}`. Input digests
|
||||||
|
cover the generated bytes; manifests record catalog identity and optional NPC
|
||||||
|
registry digest/count rather than names, aliases, overlay bytes, registry
|
||||||
|
paths, or source metadata.
|
||||||
|
|
||||||
## Debug And Redaction Boundaries
|
## Debug And Redaction Boundaries
|
||||||
|
|
||||||
|
|||||||
@@ -25,8 +25,9 @@ of that set; input and output builders receive no references. The spell
|
|||||||
extractor is typed over the canonical D&D model. D&D validators, merge, and
|
extractor is typed over the canonical D&D model. D&D validators, merge, and
|
||||||
normalize use typed variants; JSON representation validators use serialized
|
normalize use typed variants; JSON representation validators use serialized
|
||||||
requests; and unconditional validators expose separate chunk and typed
|
requests; and unconditional validators expose separate chunk and typed
|
||||||
variants. The D&D production registrar registers only the canonical typed spell
|
variants. The D&D production registrar registers the canonical typed spell and
|
||||||
implementations.
|
NPC implementations, including their kind-specific merge and normalize
|
||||||
|
behavior.
|
||||||
|
|
||||||
Prepared extractors, extract validators, and codecs may be reused concurrently
|
Prepared extractors, extract validators, and codecs may be reused concurrently
|
||||||
by the run-wide extract pool. Production implementations are immutable after
|
by the run-wide extract pool. Production implementations are immutable after
|
||||||
@@ -81,6 +82,15 @@ semantic digest; overlay content remains contextual reference material rather
|
|||||||
than source evidence. Its external JSON contract is defined in the
|
than source evidence. Its external JSON contract is defined in the
|
||||||
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).
|
[spell-catalog overlay contract](../integrations/dnd-spell-catalog-overlays.md).
|
||||||
|
|
||||||
|
### `internal/modules/dnd/npcs/identity` and `internal/modules/dnd/codec/npcs`
|
||||||
|
|
||||||
|
The NPC identity package owns Unicode comparison keys, deterministic
|
||||||
|
`npc:sha256:` IDs, display normalization, and whole-registry collision issues.
|
||||||
|
The NPC codec owns the strict durable `dnd/npc-list` JSON boundary and exposes
|
||||||
|
candidate versus approved encode/decode operations. NPC source references are
|
||||||
|
durable provenance and may later be consumed by another pipeline as registry
|
||||||
|
context without being treated as evidence for that pipeline.
|
||||||
|
|
||||||
## Input Adapter
|
## Input Adapter
|
||||||
|
|
||||||
### `internal/modules/seriatim/input/transcript`
|
### `internal/modules/seriatim/input/transcript`
|
||||||
@@ -186,9 +196,34 @@ stages, using the codec only for checkpoint, debug, and output boundaries.
|
|||||||
Shared D&D helpers keep prompt input
|
Shared D&D helpers keep prompt input
|
||||||
names and source-unit reference conversion consistent with the scene chunker.
|
names and source-unit reference conversion consistent with the scene chunker.
|
||||||
|
|
||||||
|
The extractor also declares the optional `npcs` registry slot. Preparation
|
||||||
|
requires one approved `application/json` item no larger than 1 MiB, validates
|
||||||
|
identity without relating registry source references to the current transcript,
|
||||||
|
and supplies canonical JSON to a spell-owned prompt message. A bound registry
|
||||||
|
adds only `npc_registry_digest` and `npc_count` to manifest metadata and an
|
||||||
|
`npc_registry` checkpoint fingerprint. The unbound prompt input is exactly
|
||||||
|
`{"npcs":[]}` and has no registry provenance or fingerprint.
|
||||||
|
|
||||||
The durable payload and manifest metadata shapes are defined in the
|
The durable payload and manifest metadata shapes are defined in the
|
||||||
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
[D&D spell artifact contract](../integrations/dnd-spell-artifacts.md).
|
||||||
|
|
||||||
|
### `internal/modules/dnd/extract/npcs`
|
||||||
|
|
||||||
|
The NPC extractor maps private model output to the canonical `dnd.NPCList`,
|
||||||
|
assigns source identity and deterministic NPC IDs, and preserves source
|
||||||
|
references for deterministic validation. It uses the shared campaign
|
||||||
|
references only for disambiguation and does not consume the spell-owned NPC
|
||||||
|
registry slot. Its prompt and private response schema are package-owned.
|
||||||
|
|
||||||
|
### `internal/modules/dnd/normalize/npcs`
|
||||||
|
|
||||||
|
The NPC normalizer performs deterministic identity-aware consolidation in
|
||||||
|
merged input order. It unions only canonical identity or canonical/alias
|
||||||
|
matches, retains the first display record, unions exact relationships and
|
||||||
|
source references, rewrites unambiguous relationship targets, and leaves
|
||||||
|
ambiguous collisions for identity validation. It exposes the identity policy
|
||||||
|
as its local checkpoint fingerprint and emits bounded normalization warnings.
|
||||||
|
|
||||||
## Merger And Normalizer
|
## Merger And Normalizer
|
||||||
|
|
||||||
### `internal/modules/generic/merge/appendorder`
|
### `internal/modules/generic/merge/appendorder`
|
||||||
@@ -272,6 +307,17 @@ are defined in
|
|||||||
payload rules are defined in the
|
payload rules are defined in the
|
||||||
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
[artifact contract](../integrations/dnd-spell-artifacts.md).
|
||||||
|
|
||||||
|
## D&D NPC Validators
|
||||||
|
|
||||||
|
NPC shape validation checks required strings, arrays, and source-reference
|
||||||
|
shape. The source-reference validator checks current-document identity, unit
|
||||||
|
existence, and range ordering; source relatedness emits at most one bounded
|
||||||
|
warning per record when neither the canonical name nor an alias occurs near
|
||||||
|
its cited text. Normalize identity validation checks deterministic IDs,
|
||||||
|
canonical names, aliases, and cross-record ownership or canonical collisions.
|
||||||
|
All are deterministic and expose the policy fingerprints used by the
|
||||||
|
production chains.
|
||||||
|
|
||||||
## Production Registration
|
## Production Registration
|
||||||
|
|
||||||
Production composition occurs through family registrars. The CLI allocates one
|
Production composition occurs through family registrars. The CLI allocates one
|
||||||
@@ -280,8 +326,8 @@ complete framework registry set and one LLM asset registry. It invokes
|
|||||||
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
`internal/modules/seriatim/register`, and `internal/modules/dnd/register` in
|
||||||
that order, then exposes the matching catalog for resolution. The generic and
|
that order, then exposes the matching catalog for resolution. The generic and
|
||||||
Seriatim registrars own their production leaf registrations. The D&D registrar
|
Seriatim registrars own their production leaf registrations. The D&D registrar
|
||||||
owns D&D leaf registrations, the typed spell default-validator chains, and D&D
|
owns D&D leaf registrations, typed spell and NPC default-validator chains,
|
||||||
prompt/schema asset collection.
|
typed append-order specializations, and D&D prompt/schema asset collection.
|
||||||
|
|
||||||
Concrete implementation packages do not import generic implementation
|
Concrete implementation packages do not import generic implementation
|
||||||
packages directly. A concrete family's `register` package is its composition
|
packages directly. A concrete family's `register` package is its composition
|
||||||
|
|||||||
@@ -65,9 +65,9 @@ run-local construction closures. Preparation injects shared dependencies and
|
|||||||
constructs input, chunk, validators, ordered lanes, and output before source
|
constructs input, chunk, validators, ordered lanes, and output before source
|
||||||
parsing. Production modules use strict construction-time option decoding, and
|
parsing. Production modules use strict construction-time option decoding, and
|
||||||
LLM-backed modules retain the injected shared client. The D&D family registers
|
LLM-backed modules retain the injected shared client. The D&D family registers
|
||||||
the canonical `dnd/spell-list` codec, typed spell extractor, normalizer, and
|
the canonical `dnd/spell-list` and `dnd/npc-list` codecs, typed spell and NPC
|
||||||
validators, plus kind-specific generic merge strategies; generic JSON validators
|
extractors and normalizers, validators, plus kind-specific generic merge
|
||||||
use the serialized-validation contract. The runner executes lanes through
|
strategies; generic JSON validators use the serialized-validation contract. The runner executes lanes through
|
||||||
private exact-type-checked closures, coordinates extract results independently
|
private exact-type-checked closures, coordinates extract results independently
|
||||||
of completion timing, and serializes artifacts only through their codec at
|
of completion timing, and serializes artifacts only through their codec at
|
||||||
checkpoint, debug, and output boundaries.
|
checkpoint, debug, and output boundaries.
|
||||||
@@ -84,13 +84,17 @@ Configuration. The implemented module packages are:
|
|||||||
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
|
| `internal/modules/seriatim/input/transcript` | Parses the supported Seriatim transcript format into the generic source model. |
|
||||||
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
|
| `internal/modules/generic/chunk/units` | Splits ordered source units by unit count and overlap. |
|
||||||
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
| `internal/modules/dnd/chunk/scenes` | Produces contiguous D&D scene chunks from structured model output. |
|
||||||
| `internal/modules/dnd` | Owns the canonical D&D spell-list and spell-cast artifact types. |
|
| `internal/modules/dnd` | Owns the canonical D&D spell-list, spell-cast, NPC-list, NPC, and relationship artifact types. |
|
||||||
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
|
| `internal/modules/dnd/codec/spells` | Strictly decodes and stably encodes the durable D&D spell-list representation. |
|
||||||
|
| `internal/modules/dnd/codec/npcs` | Strictly decodes and stably encodes the durable D&D NPC-list representation. |
|
||||||
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
|
| `internal/modules/dnd/extract/spells` | Maps private structured model output to canonical source-grounded D&D spell lists. |
|
||||||
|
| `internal/modules/dnd/extract/npcs` | Maps private structured model output to canonical source-grounded D&D NPC lists. |
|
||||||
|
| `internal/modules/dnd/npcs/identity` | Owns Unicode-aware NPC identity, ID derivation, and registry collision validation. |
|
||||||
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
|
| `internal/modules/dnd/spells/catalog` | Embeds and validates the versioned D&D 5e 2014 SRD catalog, composes optional overlays, and provides immutable effective lookup. |
|
||||||
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
||||||
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
||||||
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
|
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
|
||||||
|
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by identity and aliases, rewrites unambiguous relationship targets, and reports bounded warnings. |
|
||||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, and rejections as logical JSON files. |
|
||||||
|
|
||||||
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
||||||
@@ -98,6 +102,14 @@ reference declarations, prompt input assembly, and source-unit reference
|
|||||||
helpers. Domain-neutral prompt filesystem composition lives in
|
helpers. Domain-neutral prompt filesystem composition lives in
|
||||||
`internal/framework/promptfs`.
|
`internal/framework/promptfs`.
|
||||||
|
|
||||||
|
The spell extractor owns its optional `npcs` registry boundary. Preparation
|
||||||
|
strictly decodes and identity-validates one normalized JSON artifact, emits
|
||||||
|
canonical registry JSON to the spell prompt, and records only its semantic
|
||||||
|
digest and count in prepared metadata. The raw reference remains independently
|
||||||
|
tracked by pipeline provenance. An absent registry is represented only by the
|
||||||
|
empty prompt value `{"npcs":[]}`; the shared D&D reference fragment is not
|
||||||
|
changed.
|
||||||
|
|
||||||
Generic validators under `internal/modules/generic/validate` provide
|
Generic validators under `internal/modules/generic/validate` provide
|
||||||
unconditional test decisions, JSON syntax validation, and JSON Schema
|
unconditional test decisions, JSON syntax validation, and JSON Schema
|
||||||
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
|
validation. D&D spell validators under `internal/modules/dnd/validate/spells`
|
||||||
|
|||||||
@@ -40,6 +40,36 @@ names, schemas, and media types inside a run directory.
|
|||||||
Remove an output run directory only after its consumer data is no longer
|
Remove an output run directory only after its consumer data is no longer
|
||||||
needed. This is data deletion, not cache cleanup.
|
needed. This is data deletion, not cache cleanup.
|
||||||
|
|
||||||
|
## Sequential NPC And Spell Runs
|
||||||
|
|
||||||
|
The maintained [sequential configuration](../examples/dnd-npc-spell-sequential.config.yml)
|
||||||
|
contains two independent pipelines over the same Seriatim input shape. Run the
|
||||||
|
NPC pipeline first and retain its normalized payload:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/notarius run dnd-npcs \
|
||||||
|
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||||
|
--input examples/seriatim-minimal-transcript.json \
|
||||||
|
--output-dir ./npc-output
|
||||||
|
```
|
||||||
|
|
||||||
|
Then bind that completed run's `lanes/npcs.json` file to the spell extractor:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go run ./cmd/notarius run dnd-spells \
|
||||||
|
--config examples/dnd-npc-spell-sequential.config.yml \
|
||||||
|
--input examples/seriatim-minimal-transcript.json \
|
||||||
|
--reference spells.extract.npcs=./npc-output/<run-id>/lanes/npcs.json
|
||||||
|
```
|
||||||
|
|
||||||
|
The NPC file is a reference for canonical caster names and aliases, not spell
|
||||||
|
evidence. The spell manifest records the bound file's raw reference provenance
|
||||||
|
and the prepared registry's count and semantic digest separately. The NPC
|
||||||
|
payload, names, aliases, source references, and file bytes can be sensitive
|
||||||
|
campaign data; protect both output roots and any checkpoint or debug roots that
|
||||||
|
retain derived application data. A registry from another session is allowed,
|
||||||
|
but its source references are never copied into spell output evidence.
|
||||||
|
|
||||||
## Chunk-Plan Cache
|
## Chunk-Plan Cache
|
||||||
|
|
||||||
Chunk plans are stored at:
|
Chunk plans are stored at:
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# D&D NPC Extraction And Registry
|
# D&D NPC Extraction And Registry
|
||||||
|
|
||||||
Status: Accepted.
|
Status: Complete.
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
|
|||||||
@@ -18,10 +18,6 @@ not as committed release dates.
|
|||||||
|
|
||||||
### Add Sequential D&D Artifacts
|
### Add Sequential D&D Artifacts
|
||||||
|
|
||||||
- Add the proposed [D&D NPC extraction and registry](dnd-npc-extraction.md) as
|
|
||||||
the next sequential artifact. It defines canonical identity, aliases,
|
|
||||||
descriptions, relationships, evidence, deterministic consolidation, and
|
|
||||||
direct reuse as spell-pipeline reference material.
|
|
||||||
- Add combat-turn extraction with explicit event and source-reference
|
- Add combat-turn extraction with explicit event and source-reference
|
||||||
semantics. Use earlier NPC output as a reference to improve participant
|
semantics. Use earlier NPC output as a reference to improve participant
|
||||||
identity and consistency.
|
identity and consistency.
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# D&D NPC Extraction And Registry Implementation Plan
|
# D&D NPC Extraction And Registry Implementation Plan
|
||||||
|
|
||||||
Status: Ready for implementation.
|
Status: Complete.
|
||||||
|
|
||||||
Implement this plan in order. The feature policy and target state are defined
|
Implement this plan in order. The feature policy and target state are defined
|
||||||
in [D&D NPC Extraction And Registry](dnd-npc-extraction.md); this document owns
|
in [D&D NPC Extraction And Registry](dnd-npc-extraction.md); this document owns
|
||||||
|
|||||||
28
examples/dnd-npc-spell-sequential.config.yml
Normal file
28
examples/dnd-npc-spell-sequential.config.yml
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
version: 3
|
||||||
|
output:
|
||||||
|
directory: ./notarius-output
|
||||||
|
cache:
|
||||||
|
chunk_plans:
|
||||||
|
mode: bypass
|
||||||
|
checkpoints:
|
||||||
|
enabled: false
|
||||||
|
directory: ""
|
||||||
|
debug:
|
||||||
|
directory: ./notarius-debug
|
||||||
|
pipelines:
|
||||||
|
dnd-npcs:
|
||||||
|
input: seriatim
|
||||||
|
chunk: generic
|
||||||
|
artifacts:
|
||||||
|
npcs:
|
||||||
|
extract:
|
||||||
|
module: dnd/npcs
|
||||||
|
retries: 2
|
||||||
|
normalize: dnd/npcs
|
||||||
|
dnd-spells:
|
||||||
|
input: seriatim
|
||||||
|
chunk: generic
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
normalize: dnd/spells
|
||||||
21
examples/dnd-npcs.config.yml
Normal file
21
examples/dnd-npcs.config.yml
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
version: 3
|
||||||
|
output:
|
||||||
|
directory: ./notarius-output
|
||||||
|
cache:
|
||||||
|
chunk_plans:
|
||||||
|
mode: bypass
|
||||||
|
checkpoints:
|
||||||
|
enabled: false
|
||||||
|
directory: ""
|
||||||
|
debug:
|
||||||
|
directory: ./notarius-debug
|
||||||
|
pipelines:
|
||||||
|
dnd-session:
|
||||||
|
input: seriatim
|
||||||
|
chunk: generic
|
||||||
|
artifacts:
|
||||||
|
npcs:
|
||||||
|
extract:
|
||||||
|
module: dnd/npcs
|
||||||
|
retries: 2
|
||||||
|
normalize: dnd/npcs
|
||||||
132
internal/cli/dnd_npc_contract_test.go
Normal file
132
internal/cli/dnd_npc_contract_test.go
Normal file
@@ -0,0 +1,132 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||||
|
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestProductionNPCConfigurationResolvesTypedLane(t *testing.T) {
|
||||||
|
components := productionTestComponents(t)
|
||||||
|
catalog := catalogFromRegistries(components.registries)
|
||||||
|
configPath := repositoryPath("examples", "dnd-npcs.config.yml")
|
||||||
|
cfg := loadMaintainedExample(t, configPath)
|
||||||
|
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalog})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if effective.ResolvedPipeline.Chunk.Module != pipeline.DefaultChunkModule {
|
||||||
|
t.Fatalf("chunk module = %q, want %q", effective.ResolvedPipeline.Chunk.Module, pipeline.DefaultChunkModule)
|
||||||
|
}
|
||||||
|
if len(effective.ResolvedPipeline.ArtifactLanes) != 1 {
|
||||||
|
t.Fatalf("artifact lanes = %#v, want one NPC lane", effective.ResolvedPipeline.ArtifactLanes)
|
||||||
|
}
|
||||||
|
lane := effective.ResolvedPipeline.ArtifactLanes[0]
|
||||||
|
if lane.ID != "npcs" || lane.ArtifactKind != dnd.NPCListKind || lane.Extract.Module != npcextract.Key || lane.Extract.Retries != 2 || lane.Merge.Module != pipeline.DefaultMergeModule || lane.Normalize.Module != npcnormalize.Key {
|
||||||
|
t.Fatalf("resolved NPC lane = %#v, want typed production composition", lane)
|
||||||
|
}
|
||||||
|
if len(lane.ExtractReferences.Bindings) != 0 || len(lane.NormalizeReferences.Bindings) != 0 {
|
||||||
|
t.Fatalf("unbound NPC references = %#v / %#v, want none", lane.ExtractReferences, lane.NormalizeReferences)
|
||||||
|
}
|
||||||
|
|
||||||
|
extractSpec, ok := catalog.Extractors.Spec(npcextract.Key)
|
||||||
|
if !ok || !reflect.DeepEqual(extractSpec.Requires, []string{"chunks", "source.transcript"}) || !reflect.DeepEqual(extractSpec.Provides, []string{"dnd.npcs"}) {
|
||||||
|
t.Fatalf("NPC extractor spec = %#v, want source and artifact capabilities", extractSpec)
|
||||||
|
}
|
||||||
|
mergeSpec, ok := catalog.Mergers.SpecForArtifact(pipeline.DefaultMergeModule, dnd.NPCListKind)
|
||||||
|
if !ok || !reflect.DeepEqual(mergeSpec.Provides, []string{"merged"}) {
|
||||||
|
t.Fatalf("NPC merger spec = %#v, want merged capability", mergeSpec)
|
||||||
|
}
|
||||||
|
normalizeSpec, ok := catalog.Normalizers.SpecForArtifact(npcnormalize.Key, dnd.NPCListKind)
|
||||||
|
if !ok || !reflect.DeepEqual(normalizeSpec.Requires, []string{"merged"}) || !reflect.DeepEqual(normalizeSpec.Provides, []string{"normalized"}) {
|
||||||
|
t.Fatalf("NPC normalizer spec = %#v, want merged/normalized capabilities", normalizeSpec)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantExtractChain := []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding("generic/valid_json"),
|
||||||
|
pipeline.Binding("generic/valid_json_schema"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/shape"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_refs"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
|
||||||
|
}
|
||||||
|
wantNormalizeChain := []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding("generic/valid_json"),
|
||||||
|
pipeline.Binding("generic/valid_json_schema"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/shape"),
|
||||||
|
pipeline.Binding("normalize/dnd/npcs/identity"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_refs"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
|
||||||
|
}
|
||||||
|
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageExtract, npcextract.Key); !reflect.DeepEqual(got, wantExtractChain) {
|
||||||
|
t.Fatalf("NPC extract chain = %#v, want %#v", got, wantExtractChain)
|
||||||
|
}
|
||||||
|
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageNormalize, npcnormalize.Key); !reflect.DeepEqual(got, wantNormalizeChain) {
|
||||||
|
t.Fatalf("NPC normalize chain = %#v, want %#v", got, wantNormalizeChain)
|
||||||
|
}
|
||||||
|
if got := validatorChain(effective.ResolvedPipeline, pipeline.StageMerge, pipeline.DefaultMergeModule); len(got) != 0 {
|
||||||
|
t.Fatalf("NPC merge chain = %#v, want empty", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProductionNPCConfigurationValidatesOptionsReferencesAndPlacement(t *testing.T) {
|
||||||
|
components := productionTestComponents(t)
|
||||||
|
configPath := repositoryPath("examples", "dnd-npcs.config.yml")
|
||||||
|
resolve := func(mutate func(*pipeline.PipelineProfile)) error {
|
||||||
|
cfg := loadMaintainedExample(t, configPath)
|
||||||
|
profile := cfg.Pipelines["dnd-session"]
|
||||||
|
mutate(&profile)
|
||||||
|
cfg.Pipelines["dnd-session"] = profile
|
||||||
|
_, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-session", Catalog: catalogFromRegistries(components.registries)})
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := resolve(func(profile *pipeline.PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["npcs"]
|
||||||
|
lane.Extract.Options = map[string]any{"unexpected": true}
|
||||||
|
profile.Artifacts["npcs"] = lane
|
||||||
|
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("unknown extractor option error = %v, want strict option rejection", err)
|
||||||
|
}
|
||||||
|
if err := resolve(func(profile *pipeline.PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["npcs"]
|
||||||
|
lane.Normalize.Options = map[string]any{"unexpected": true}
|
||||||
|
profile.Artifacts["npcs"] = lane
|
||||||
|
}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("unknown normalizer option error = %v, want strict option rejection", err)
|
||||||
|
}
|
||||||
|
if err := resolve(func(profile *pipeline.PipelineProfile) {
|
||||||
|
profile.References = map[string]string{
|
||||||
|
"players": "players.txt",
|
||||||
|
"party": "party.txt",
|
||||||
|
"glossary": "glossary.txt",
|
||||||
|
}
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("optional NPC references error = %v, want resolution success", err)
|
||||||
|
}
|
||||||
|
if err := resolve(func(profile *pipeline.PipelineProfile) {
|
||||||
|
lane := profile.Artifacts["npcs"]
|
||||||
|
lane.Validators = []pipeline.ModuleBinding{pipeline.Binding("normalize/dnd/npcs/identity")}
|
||||||
|
profile.Artifacts["npcs"] = lane
|
||||||
|
}); err == nil || !strings.Contains(err.Error(), "artifact lane level") {
|
||||||
|
t.Fatalf("lane-level validator error = %v, want invalid placement rejection", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validatorChain(resolved pipeline.ResolvedPipeline, stage pipeline.ModuleStage, module string) []pipeline.ModuleBinding {
|
||||||
|
for _, chain := range resolved.ValidatorChains {
|
||||||
|
if chain.Stage == stage && chain.ModuleKey == module {
|
||||||
|
bindings := make([]pipeline.ModuleBinding, len(chain.Validators))
|
||||||
|
for index, validator := range chain.Validators {
|
||||||
|
bindings[index] = validator.Binding
|
||||||
|
}
|
||||||
|
return bindings
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -20,16 +20,17 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
|||||||
for _, example := range maintainedExampleFiles(t) {
|
for _, example := range maintainedExampleFiles(t) {
|
||||||
t.Run(example.name, func(t *testing.T) {
|
t.Run(example.name, func(t *testing.T) {
|
||||||
cfg := loadMaintainedExample(t, example.path)
|
cfg := loadMaintainedExample(t, example.path)
|
||||||
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, "dnd-session"))
|
for _, pipelineID := range example.pipelineIDs {
|
||||||
|
effective, err := cfg.Resolve(resolveInputForMaintainedExample(components, pipelineID))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("resolve maintained example: %v", err)
|
t.Fatalf("resolve maintained example %q: %v", pipelineID, err)
|
||||||
}
|
}
|
||||||
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{
|
materialized, _, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, catalogFromRegistries(components.registries), pipeline.ReferenceMaterializationOptions{
|
||||||
ConfigPath: example.path,
|
ConfigPath: example.path,
|
||||||
WorkingDir: filepath.Dir(example.path),
|
WorkingDir: filepath.Dir(example.path),
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("materialize maintained example references: %v", err)
|
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
|
||||||
}
|
}
|
||||||
if example.name == "production" {
|
if example.name == "production" {
|
||||||
if len(materialized.ArtifactLanes) != 1 ||
|
if len(materialized.ArtifactLanes) != 1 ||
|
||||||
@@ -38,9 +39,10 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
|||||||
t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.ArtifactLanes)
|
t.Fatalf("production spell catalog reference was not materialized: %#v", materialized.ArtifactLanes)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
var stdout, stderr strings.Builder
|
var stdout, stderr strings.Builder
|
||||||
code := RunWithOptions([]string{"pipelines", "list", "--config", example.path}, &stdout, &stderr, productionOptionsFromComponents(components))
|
code := RunWithOptions([]string{"pipelines", "list", "--config", example.path}, &stdout, &stderr, productionOptionsFromComponents(components))
|
||||||
if code != 0 || stdout.String() != "dnd-session\n" || stderr.Len() != 0 {
|
if code != 0 || stdout.String() != strings.Join(example.pipelineIDs, "\n")+"\n" || stderr.Len() != 0 {
|
||||||
t.Fatalf("pipelines list: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
t.Fatalf("pipelines list: code=%d stdout=%q stderr=%q", code, stdout.String(), stderr.String())
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
64
internal/cli/npc_registry_contract_test.go
Normal file
64
internal/cli/npc_registry_contract_test.go
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
package cli
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestOversizedNPCRegistryFailsBeforeRuntimeAndCheckpointConstruction(t *testing.T) {
|
||||||
|
components := productionTestComponents(t)
|
||||||
|
npcPath := filepath.Join(t.TempDir(), "npcs.json")
|
||||||
|
if err := os.WriteFile(npcPath, []byte(strings.Repeat("x", 1048577)), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
checkpointRoot := filepath.Join(t.TempDir(), "checkpoints")
|
||||||
|
content := string(readRepositoryFile(t, "examples", "dnd-npc-spell-sequential.config.yml"))
|
||||||
|
content = replaceRequiredOnce(t, content, " extract: dnd/spells", " extract:\n module: dnd/spells\n references:\n npcs: "+npcPath)
|
||||||
|
content = replaceRequiredOnce(t, content, " enabled: false\n directory: \"\"", " enabled: true\n directory: "+checkpointRoot)
|
||||||
|
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||||
|
if err := os.WriteFile(configPath, []byte(content), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
llmConstructed := false
|
||||||
|
chunkStoreConstructed := false
|
||||||
|
options := Options{
|
||||||
|
Catalog: catalogFromRegistries(components.registries),
|
||||||
|
Registries: components.registries,
|
||||||
|
LLMClientFactory: func(context.Context, config.Config, string) (contracts.StructuredLLMClient, []artifacts.LLMProfileManifest, error) {
|
||||||
|
llmConstructed = true
|
||||||
|
return nil, nil, errors.New("LLM client must not be constructed")
|
||||||
|
},
|
||||||
|
ChunkPlanStoreFactory: func(string) (pipeline.ChunkPlanStore, error) {
|
||||||
|
chunkStoreConstructed = true
|
||||||
|
return nil, errors.New("chunk-plan store must not be constructed")
|
||||||
|
},
|
||||||
|
}
|
||||||
|
var stdout, stderr strings.Builder
|
||||||
|
code := RunWithOptions([]string{
|
||||||
|
"run", "dnd-spells", "--config", configPath,
|
||||||
|
"--input", repositoryPath("examples", "seriatim-minimal-transcript.json"),
|
||||||
|
"--chunk_cache", "bypass", "--output-dir", t.TempDir(),
|
||||||
|
}, &stdout, &stderr, options)
|
||||||
|
for _, fragment := range []string{`pipeline "dnd-spells"`, `reference slot "npcs"`, "1048577 bytes", "limit 1048576"} {
|
||||||
|
if code == 0 || !strings.Contains(stderr.String(), fragment) {
|
||||||
|
t.Fatalf("RunWithOptions() code = %d stderr = %q, want context fragment %q", code, stderr.String(), fragment)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if llmConstructed || chunkStoreConstructed {
|
||||||
|
t.Fatalf("runtime construction = LLM %t, chunk store %t; want materialization failure first", llmConstructed, chunkStoreConstructed)
|
||||||
|
}
|
||||||
|
if _, err := os.Stat(checkpointRoot); !errors.Is(err, fs.ErrNotExist) {
|
||||||
|
t.Fatalf("checkpoint root stat error = %v, want no checkpoint allocation", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -456,13 +456,16 @@ func TestProductionSceneRunRecordsChunkerWarningsAndProvenance(t *testing.T) {
|
|||||||
type maintainedExample struct {
|
type maintainedExample struct {
|
||||||
name string
|
name string
|
||||||
path string
|
path string
|
||||||
|
pipelineIDs []string
|
||||||
}
|
}
|
||||||
|
|
||||||
func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
func maintainedExampleFiles(t *testing.T) []maintainedExample {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
return []maintainedExample{
|
return []maintainedExample{
|
||||||
{name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml")},
|
{name: "minimal", path: repositoryPath("examples", "dnd-spells.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
||||||
{name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml")},
|
{name: "production", path: repositoryPath("examples", "dnd-spells-production.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
||||||
|
{name: "npcs", path: repositoryPath("examples", "dnd-npcs.config.yml"), pipelineIDs: []string{"dnd-session"}},
|
||||||
|
{name: "sequential", path: repositoryPath("examples", "dnd-npc-spell-sequential.config.yml"), pipelineIDs: []string{"dnd-npcs", "dnd-spells"}},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,86 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "notarius.dnd.npcs",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["npcs"],
|
||||||
|
"properties": {
|
||||||
|
"npcs": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"id",
|
||||||
|
"name",
|
||||||
|
"aliases",
|
||||||
|
"description",
|
||||||
|
"relationships",
|
||||||
|
"source_refs"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"id": {
|
||||||
|
"type": "string",
|
||||||
|
"pattern": "^npc:sha256:[0-9a-f]{64}$"
|
||||||
|
},
|
||||||
|
"name": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"aliases": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"relationships": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["target", "relationship"],
|
||||||
|
"properties": {
|
||||||
|
"target": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"relationship": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"source_refs": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["source_id", "start_unit_id", "end_unit_id"],
|
||||||
|
"properties": {
|
||||||
|
"source_id": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"start_unit_id": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 1
|
||||||
|
},
|
||||||
|
"end_unit_id": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
149
internal/modules/dnd/codec/npcs/codec.go
Normal file
149
internal/modules/dnd/codec/npcs/codec.go
Normal file
@@ -0,0 +1,149 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"embed"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
SchemaID = "notarius.dnd.npcs"
|
||||||
|
SchemaName = "notarius_dnd_npcs_v1"
|
||||||
|
SchemaVersion = "v1"
|
||||||
|
MediaType = "application/json"
|
||||||
|
)
|
||||||
|
|
||||||
|
//go:embed assets/schemas/dnd_npcs.v1.json
|
||||||
|
var schemaAssets embed.FS
|
||||||
|
|
||||||
|
var _ contracts.ArtifactCodec[dnd.NPCList] = (*Codec)(nil)
|
||||||
|
|
||||||
|
type Codec struct{}
|
||||||
|
|
||||||
|
func New() *Codec { return &Codec{} }
|
||||||
|
|
||||||
|
func (c *Codec) Kind() contracts.ArtifactKind { return dnd.NPCListKind }
|
||||||
|
|
||||||
|
func (c *Codec) Schema() contracts.ArtifactSchema {
|
||||||
|
raw, err := schemaAssets.ReadFile("assets/schemas/dnd_npcs.v1.json")
|
||||||
|
if err != nil {
|
||||||
|
return contracts.ArtifactSchema{}
|
||||||
|
}
|
||||||
|
return contracts.ArtifactSchema{
|
||||||
|
ID: SchemaID,
|
||||||
|
Name: SchemaName,
|
||||||
|
Version: SchemaVersion,
|
||||||
|
JSONSchema: append([]byte(nil), raw...),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) MediaType() string { return MediaType }
|
||||||
|
|
||||||
|
func (c *Codec) Metadata(value dnd.NPCList) map[string]any {
|
||||||
|
return map[string]any{"npc_count": len(value.NPCs)}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) Encode(value dnd.NPCList) ([]byte, error) {
|
||||||
|
if err := validate(value); err != nil {
|
||||||
|
return nil, fmt.Errorf("encode dnd npc list: %w", err)
|
||||||
|
}
|
||||||
|
return c.EncodeCandidate(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
// EncodeCandidate provides the durable representation before semantic
|
||||||
|
// validators have approved a value.
|
||||||
|
func (c *Codec) EncodeCandidate(value dnd.NPCList) ([]byte, error) {
|
||||||
|
content, err := json.Marshal(value)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("encode dnd npc list: %w", err)
|
||||||
|
}
|
||||||
|
return content, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *Codec) Decode(content []byte) (dnd.NPCList, error) {
|
||||||
|
value, err := c.DecodeCandidate(content)
|
||||||
|
if err != nil {
|
||||||
|
return dnd.NPCList{}, err
|
||||||
|
}
|
||||||
|
if err := validate(value); err != nil {
|
||||||
|
return dnd.NPCList{}, fmt.Errorf("decode dnd npc list: %w", err)
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DecodeCandidate reads one strict durable JSON value before semantic
|
||||||
|
// validators have approved it.
|
||||||
|
func (c *Codec) DecodeCandidate(content []byte) (dnd.NPCList, error) {
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(content))
|
||||||
|
decoder.DisallowUnknownFields()
|
||||||
|
var value dnd.NPCList
|
||||||
|
if err := decoder.Decode(&value); err != nil {
|
||||||
|
return dnd.NPCList{}, fmt.Errorf("decode dnd npc list: %w", err)
|
||||||
|
}
|
||||||
|
var trailing any
|
||||||
|
if err := decoder.Decode(&trailing); err != io.EOF {
|
||||||
|
return dnd.NPCList{}, fmt.Errorf("decode dnd npc list: multiple JSON values")
|
||||||
|
}
|
||||||
|
return value, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validate(value dnd.NPCList) error {
|
||||||
|
if value.NPCs == nil {
|
||||||
|
return fmt.Errorf("npcs must be present")
|
||||||
|
}
|
||||||
|
for index, npc := range value.NPCs {
|
||||||
|
prefix := fmt.Sprintf("npcs[%d]", index)
|
||||||
|
if !identity.IsValidID(npc.ID) {
|
||||||
|
return fmt.Errorf("%s.id must match npc ID pattern", prefix)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(npc.Name) == "" {
|
||||||
|
return fmt.Errorf("%s.name must not be empty", prefix)
|
||||||
|
}
|
||||||
|
if npc.Aliases == nil {
|
||||||
|
return fmt.Errorf("%s.aliases must be present", prefix)
|
||||||
|
}
|
||||||
|
for aliasIndex, alias := range npc.Aliases {
|
||||||
|
if strings.TrimSpace(alias) == "" {
|
||||||
|
return fmt.Errorf("%s.aliases[%d] must not be empty", prefix, aliasIndex)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(npc.Description) == "" {
|
||||||
|
return fmt.Errorf("%s.description must not be empty", prefix)
|
||||||
|
}
|
||||||
|
if npc.Relationships == nil {
|
||||||
|
return fmt.Errorf("%s.relationships must be present", prefix)
|
||||||
|
}
|
||||||
|
for relationshipIndex, relationship := range npc.Relationships {
|
||||||
|
relationshipPrefix := fmt.Sprintf("%s.relationships[%d]", prefix, relationshipIndex)
|
||||||
|
if strings.TrimSpace(relationship.Target) == "" {
|
||||||
|
return fmt.Errorf("%s.target must not be empty", relationshipPrefix)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(relationship.Relationship) == "" {
|
||||||
|
return fmt.Errorf("%s.relationship must not be empty", relationshipPrefix)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(npc.SourceRefs) == 0 {
|
||||||
|
return fmt.Errorf("%s.source_refs must not be empty", prefix)
|
||||||
|
}
|
||||||
|
for refIndex, ref := range npc.SourceRefs {
|
||||||
|
refPrefix := fmt.Sprintf("%s.source_refs[%d]", prefix, refIndex)
|
||||||
|
if strings.TrimSpace(ref.SourceID) == "" {
|
||||||
|
return fmt.Errorf("%s.source_id must not be empty", refPrefix)
|
||||||
|
}
|
||||||
|
if ref.StartUnitID <= 0 {
|
||||||
|
return fmt.Errorf("%s.start_unit_id must be positive", refPrefix)
|
||||||
|
}
|
||||||
|
if ref.EndUnitID <= 0 {
|
||||||
|
return fmt.Errorf("%s.end_unit_id must be positive", refPrefix)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
155
internal/modules/dnd/codec/npcs/codec_test.go
Normal file
155
internal/modules/dnd/codec/npcs/codec_test.go
Normal file
@@ -0,0 +1,155 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validList() dnd.NPCList {
|
||||||
|
return dnd.NPCList{NPCs: []dnd.NPC{{
|
||||||
|
ID: identity.DeriveID("Mira Thorn"),
|
||||||
|
Name: "Mira Thorn",
|
||||||
|
Aliases: []string{"The Greencloak"},
|
||||||
|
Description: "A guarded ranger who watches the northern road.",
|
||||||
|
Relationships: []dnd.NPCRelationship{{
|
||||||
|
Target: "Captain Vale", Relationship: "reports to",
|
||||||
|
}},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}},
|
||||||
|
}}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecMatchesMaintainedDurableFixture(t *testing.T) {
|
||||||
|
raw, err := os.ReadFile("testdata/dnd_npcs.v1.json")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read durable fixture: %v", err)
|
||||||
|
}
|
||||||
|
codec := New()
|
||||||
|
value, err := codec.Decode(raw)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Decode() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
want := validList()
|
||||||
|
if !reflect.DeepEqual(value, want) {
|
||||||
|
t.Fatalf("Decode() = %#v, want %#v", value, want)
|
||||||
|
}
|
||||||
|
encoded, err := codec.Encode(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Encode() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
var compact bytes.Buffer
|
||||||
|
if err := json.Compact(&compact, raw); err != nil {
|
||||||
|
t.Fatalf("compact durable fixture: %v", err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(encoded, compact.Bytes()) {
|
||||||
|
t.Fatalf("Encode() = %s, want stable durable JSON %s", encoded, compact.Bytes())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecOwnsDurableSchemaAndRegistersExactType(t *testing.T) {
|
||||||
|
codec := New()
|
||||||
|
schema := codec.Schema()
|
||||||
|
if codec.Kind() != dnd.NPCListKind || codec.MediaType() != MediaType {
|
||||||
|
t.Fatalf("codec identity = %q/%q", codec.Kind(), codec.MediaType())
|
||||||
|
}
|
||||||
|
if schema.ID != SchemaID || schema.Name != SchemaName || schema.Version != SchemaVersion || !json.Valid(schema.JSONSchema) {
|
||||||
|
t.Fatalf("schema = %#v, want durable NPC schema", schema)
|
||||||
|
}
|
||||||
|
var document map[string]any
|
||||||
|
if err := json.Unmarshal(schema.JSONSchema, &document); err != nil || document["$id"] != SchemaID {
|
||||||
|
t.Fatalf("durable schema document = %#v, %v", document, err)
|
||||||
|
}
|
||||||
|
registry := pipeline.NewArtifactCodecRegistry()
|
||||||
|
if err := pipeline.RegisterArtifactCodec(registry, codec); err != nil {
|
||||||
|
t.Fatalf("RegisterArtifactCodec() error = %v", err)
|
||||||
|
}
|
||||||
|
spec, ok := registry.Spec(dnd.NPCListKind)
|
||||||
|
if !ok || spec.SchemaDigest != contracts.DigestArtifactSchema(schema) {
|
||||||
|
t.Fatalf("registered spec = %#v, %t", spec, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecStrictlyRejectsMalformedOrUnknownJSON(t *testing.T) {
|
||||||
|
codec := New()
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
raw string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "unknown top-level", raw: `{"npcs":[],"unexpected":true}`, want: "unknown field"},
|
||||||
|
{name: "unknown nested", raw: `{"npcs":[{"id":"x","name":"Mira","aliases":[],"description":"desc","relationships":[],"source_refs":[{"source_id":"s","start_unit_id":1,"end_unit_id":1}],"unexpected":true}]}`, want: "unknown field"},
|
||||||
|
{name: "trailing", raw: `{"npcs":[]} {}`, want: "multiple JSON values"},
|
||||||
|
{name: "missing array", raw: `{}`, want: "npcs must be present"},
|
||||||
|
{name: "invalid reference", raw: `{"npcs":[{"id":"npc:sha256:0000000000000000000000000000000000000000000000000000000000000000","name":"Mira","aliases":[],"description":"desc","relationships":[],"source_refs":[{"source_id":"s","start_unit_id":0,"end_unit_id":1}]}]}`, want: "start_unit_id"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
_, err := codec.Decode([]byte(test.raw))
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Decode() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecCandidatePreservesInvalidTypedValues(t *testing.T) {
|
||||||
|
codec := New()
|
||||||
|
candidate := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn", Aliases: []string{}, Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{}}}}
|
||||||
|
content, err := codec.EncodeCandidate(candidate)
|
||||||
|
if err != nil || !json.Valid(content) {
|
||||||
|
t.Fatalf("EncodeCandidate() = %s, %v; want JSON", content, err)
|
||||||
|
}
|
||||||
|
decoded, err := codec.DecodeCandidate(content)
|
||||||
|
if err != nil || !reflect.DeepEqual(decoded, candidate) {
|
||||||
|
t.Fatalf("DecodeCandidate() = %#v, %v; want %#v", decoded, err, candidate)
|
||||||
|
}
|
||||||
|
if _, err := codec.Encode(candidate); err == nil || !strings.Contains(err.Error(), ".id must match npc ID pattern") {
|
||||||
|
t.Fatalf("Encode() error = %v, want strict validation error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecRejectsEveryRequiredShapeBoundary(t *testing.T) {
|
||||||
|
base := validList().NPCs[0]
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
value dnd.NPCList
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "nil aliases", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Description: base.Description, Relationships: []dnd.NPCRelationship{}, SourceRefs: base.SourceRefs}}}, want: "aliases must be present"},
|
||||||
|
{name: "empty alias", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{" "}, Description: base.Description, Relationships: []dnd.NPCRelationship{}, SourceRefs: base.SourceRefs}}}, want: "aliases[0] must not be empty"},
|
||||||
|
{name: "nil relationships", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{}, Description: base.Description, SourceRefs: base.SourceRefs}}}, want: "relationships must be present"},
|
||||||
|
{name: "empty relationship target", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{}, Description: base.Description, Relationships: []dnd.NPCRelationship{{Relationship: "knows"}}, SourceRefs: base.SourceRefs}}}, want: "target must not be empty"},
|
||||||
|
{name: "empty source refs", value: dnd.NPCList{NPCs: []dnd.NPC{{ID: base.ID, Name: base.Name, Aliases: []string{}, Description: base.Description, Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{}}}}, want: "source_refs must not be empty"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
if _, err := New().Encode(test.value); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||||
|
t.Fatalf("Encode() error = %v, want %q", err, test.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestCodecSchemaAndMetadataAreDefensive(t *testing.T) {
|
||||||
|
codec := New()
|
||||||
|
first := codec.Schema()
|
||||||
|
first.JSONSchema[0] = '['
|
||||||
|
second := codec.Schema()
|
||||||
|
if !json.Valid(second.JSONSchema) || second.JSONSchema[0] == '[' {
|
||||||
|
t.Fatal("Schema() returned shared bytes")
|
||||||
|
}
|
||||||
|
metadata := codec.Metadata(validList())
|
||||||
|
metadata["other"] = true
|
||||||
|
if next := codec.Metadata(validList()); len(next) != 1 || next["npc_count"] != 1 {
|
||||||
|
t.Fatalf("Metadata() = %#v, want only npc_count", next)
|
||||||
|
}
|
||||||
|
}
|
||||||
12
internal/modules/dnd/codec/npcs/testdata/dnd_npcs.v1.json
vendored
Normal file
12
internal/modules/dnd/codec/npcs/testdata/dnd_npcs.v1.json
vendored
Normal file
@@ -0,0 +1,12 @@
|
|||||||
|
{
|
||||||
|
"npcs": [
|
||||||
|
{
|
||||||
|
"id": "npc:sha256:99a16589618a04f535a7d21fdcc71a0b1c05d22f752cd492065b1086d97bc3d7",
|
||||||
|
"name": "Mira Thorn",
|
||||||
|
"aliases": ["The Greencloak"],
|
||||||
|
"description": "A guarded ranger who watches the northern road.",
|
||||||
|
"relationships": [{"target": "Captain Vale", "relationship": "reports to"}],
|
||||||
|
"source_refs": [{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 2}]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
6
internal/modules/dnd/extract/npcs/assets.go
Normal file
6
internal/modules/dnd/extract/npcs/assets.go
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import "embed"
|
||||||
|
|
||||||
|
//go:embed assets/schemas/dnd_npcs_llm.v1.json assets/prompts/*.yaml assets/prompts/*.md
|
||||||
|
var embeddedAssets embed.FS
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
id: dnd.npcs
|
||||||
|
version: "v1"
|
||||||
|
default_profile: gemini-2-flash
|
||||||
|
inputs:
|
||||||
|
- name: transcript
|
||||||
|
required: true
|
||||||
|
content_type: application/json
|
||||||
|
- name: players
|
||||||
|
required: false
|
||||||
|
content_type: text/plain
|
||||||
|
- name: party
|
||||||
|
required: false
|
||||||
|
content_type: text/plain
|
||||||
|
- name: glossary
|
||||||
|
required: false
|
||||||
|
content_type: text/plain
|
||||||
|
messages:
|
||||||
|
- role: system
|
||||||
|
content_file: ./sharedassets/common-dnd-system.md
|
||||||
|
- role: user
|
||||||
|
content_file: ./sharedassets/common-dnd-transcript.md
|
||||||
|
cache_control:
|
||||||
|
type: ephemeral
|
||||||
|
- role: user
|
||||||
|
content_file: ./sharedassets/common-dnd-references.md
|
||||||
|
cache_control:
|
||||||
|
type: ephemeral
|
||||||
|
- role: user
|
||||||
|
content_file: ./task.md
|
||||||
|
- role: user
|
||||||
|
content_file: ./instructions.md
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: dnd_npcs_llm.v1.json
|
||||||
|
repair_attempts: 0
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
Return exactly one JSON object and no explanatory text.
|
||||||
|
|
||||||
|
For every NPC record, cite one or more transcript source-unit ranges that
|
||||||
|
collectively support the canonical name, every alias, the description, and
|
||||||
|
every relationship. Use integer start_unit_id and end_unit_id values from the
|
||||||
|
transcript. Do not provide source_id; the extractor assigns it automatically.
|
||||||
|
Use narrow ranges when evidence is not contiguous and do not bridge unrelated
|
||||||
|
conversation with a broad range.
|
||||||
|
|
||||||
|
A canonical name must be the most specific in-world identity supported by the
|
||||||
|
transcript. Never use a human player, transcript speaker, or GM name when an
|
||||||
|
associated in-world character or creature is identified. Player, party, and
|
||||||
|
glossary references may disambiguate identities, but they are not transcript
|
||||||
|
evidence and cannot establish that an NPC appeared, acted, or was discussed.
|
||||||
|
Do not return a person or entity mentioned only in those references.
|
||||||
|
|
||||||
|
Descriptions must be short session records, not biographies, statistics,
|
||||||
|
alignment, motivations, or lore inferred from general D&D knowledge. Do not
|
||||||
|
summarize every action or follow a participant through unrelated scenes.
|
||||||
|
Relationships must be stated or directly demonstrated by cited transcript
|
||||||
|
units, not inferred from game lore.
|
||||||
|
|
||||||
|
Return aliases and relationships as arrays, including empty arrays when there
|
||||||
|
are none. Return only NPC records supported by the transcript and preserve
|
||||||
|
observed display spelling.
|
||||||
16
internal/modules/dnd/extract/npcs/assets/prompts/task.md
Normal file
16
internal/modules/dnd/extract/npcs/assets/prompts/task.md
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
Extract a concise Dungeons & Dragons non-player-character registry from the
|
||||||
|
provided transcript.
|
||||||
|
|
||||||
|
Include an in-world non-PC participant when the transcript establishes that it
|
||||||
|
appears, acts, speaks, or is materially discussed and gives it a proper name,
|
||||||
|
a stable alias or title, or an individually useful distinguishing description.
|
||||||
|
|
||||||
|
Exclude human players, transcript speakers, the GM as an out-of-world person,
|
||||||
|
player characters identified by the player or party references, incidental or
|
||||||
|
hypothetical name drops, corrected transcription mistakes, characters mentioned
|
||||||
|
only by reference material, indistinguishable crowds or groups, and temporary
|
||||||
|
summoned creatures or spell effects without a persistent individual identity.
|
||||||
|
|
||||||
|
Return canonical in-world names rather than player names or transcript speaker
|
||||||
|
names. Keep each description concise and limited to facts established by the
|
||||||
|
transcript. Include only explicitly supported aliases and relationships.
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "notarius.dnd.npcs.llm",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["npcs"],
|
||||||
|
"properties": {
|
||||||
|
"npcs": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"name",
|
||||||
|
"aliases",
|
||||||
|
"description",
|
||||||
|
"relationships",
|
||||||
|
"source_refs"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"name": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"aliases": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"description": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"relationships": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["target", "relationship"],
|
||||||
|
"properties": {
|
||||||
|
"target": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"relationship": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"source_refs": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["start_unit_id", "end_unit_id"],
|
||||||
|
"properties": {
|
||||||
|
"start_unit_id": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 1
|
||||||
|
},
|
||||||
|
"end_unit_id": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
153
internal/modules/dnd/extract/npcs/canonicalize.go
Normal file
153
internal/modules/dnd/extract/npcs/canonicalize.go
Normal file
@@ -0,0 +1,153 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"sort"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
)
|
||||||
|
|
||||||
|
func canonicalizeResponse(response *extractionResponse, doc *source.SourceDocument) {
|
||||||
|
if response == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for index := range response.NPCs {
|
||||||
|
canonicalizeNPC(&response.NPCs[index])
|
||||||
|
}
|
||||||
|
sort.SliceStable(response.NPCs, func(i, j int) bool {
|
||||||
|
left, leftOK := earliestSourceUnit(doc, response.NPCs[i])
|
||||||
|
right, rightOK := earliestSourceUnit(doc, response.NPCs[j])
|
||||||
|
if leftOK != rightOK {
|
||||||
|
return leftOK
|
||||||
|
}
|
||||||
|
if !leftOK {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return left < right
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalizeNPC(npc *npcResponse) {
|
||||||
|
if npc == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for index := range npc.SourceRefs {
|
||||||
|
npc.SourceRefs[index].StartUnitID = canonicalUnitRef(npc.SourceRefs[index].StartUnitID)
|
||||||
|
npc.SourceRefs[index].EndUnitID = canonicalUnitRef(npc.SourceRefs[index].EndUnitID)
|
||||||
|
}
|
||||||
|
sort.SliceStable(npc.SourceRefs, func(i, j int) bool {
|
||||||
|
left := npc.SourceRefs[i]
|
||||||
|
right := npc.SourceRefs[j]
|
||||||
|
if unitSortValue(left.StartUnitID) != unitSortValue(right.StartUnitID) {
|
||||||
|
return unitSortValue(left.StartUnitID) < unitSortValue(right.StartUnitID)
|
||||||
|
}
|
||||||
|
return unitSortValue(left.EndUnitID) < unitSortValue(right.EndUnitID)
|
||||||
|
})
|
||||||
|
npc.SourceRefs = dedupeSourceRefs(npc.SourceRefs)
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalUnitRef(ref shared.UnitRef) shared.UnitRef {
|
||||||
|
value := ref.Int()
|
||||||
|
if value <= 0 {
|
||||||
|
return ref
|
||||||
|
}
|
||||||
|
return shared.UnitRefFromInt(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func dedupeSourceRefs(refs []npcSourceRefResponse) []npcSourceRefResponse {
|
||||||
|
if len(refs) < 2 {
|
||||||
|
return refs
|
||||||
|
}
|
||||||
|
out := refs[:0]
|
||||||
|
var previous npcSourceRefResponse
|
||||||
|
for index, ref := range refs {
|
||||||
|
if index > 0 && sameSourceRef(previous, ref) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out = append(out, ref)
|
||||||
|
previous = ref
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func sameSourceRef(left npcSourceRefResponse, right npcSourceRefResponse) bool {
|
||||||
|
return left.StartUnitID.Int() == right.StartUnitID.Int() &&
|
||||||
|
left.EndUnitID.Int() == right.EndUnitID.Int()
|
||||||
|
}
|
||||||
|
|
||||||
|
func earliestSourceUnit(doc *source.SourceDocument, npc npcResponse) (int, bool) {
|
||||||
|
for _, ref := range npc.SourceRefs {
|
||||||
|
start := ref.StartUnitID.Int()
|
||||||
|
end := ref.EndUnitID.Int()
|
||||||
|
if start > 0 && end > 0 {
|
||||||
|
startIndex, startOK := source.UnitIndex(doc, start)
|
||||||
|
endIndex, endOK := source.UnitIndex(doc, end)
|
||||||
|
if !startOK || !endOK || startIndex > endIndex {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return start, true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return 0, false
|
||||||
|
}
|
||||||
|
|
||||||
|
func unitSortValue(ref shared.UnitRef) int {
|
||||||
|
value := ref.Int()
|
||||||
|
if value <= 0 {
|
||||||
|
return int(^uint(0) >> 1)
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalNPCList(response extractionResponse, sourceID string) dnd.NPCList {
|
||||||
|
if response.NPCs == nil {
|
||||||
|
return dnd.NPCList{NPCs: nil}
|
||||||
|
}
|
||||||
|
npcs := make([]dnd.NPC, len(response.NPCs))
|
||||||
|
for index, npc := range response.NPCs {
|
||||||
|
npcs[index] = dnd.NPC{
|
||||||
|
ID: identity.DeriveID(npc.Name),
|
||||||
|
Name: npc.Name,
|
||||||
|
Aliases: cloneStrings(npc.Aliases),
|
||||||
|
Description: npc.Description,
|
||||||
|
Relationships: cloneRelationships(npc.Relationships),
|
||||||
|
SourceRefs: canonicalSourceRefs(npc.SourceRefs, sourceID),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return dnd.NPCList{NPCs: npcs}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneStrings(values []string) []string {
|
||||||
|
if values == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return append([]string{}, values...)
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneRelationships(values []npcRelationshipResponse) []dnd.NPCRelationship {
|
||||||
|
if values == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]dnd.NPCRelationship, len(values))
|
||||||
|
for index, value := range values {
|
||||||
|
out[index] = dnd.NPCRelationship{Target: value.Target, Relationship: value.Relationship}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalSourceRefs(values []npcSourceRefResponse, sourceID string) []source.SourceRef {
|
||||||
|
if values == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := make([]source.SourceRef, len(values))
|
||||||
|
for index, value := range values {
|
||||||
|
out[index] = source.SourceRef{
|
||||||
|
SourceID: sourceID,
|
||||||
|
StartUnitID: value.StartUnitID.Int(),
|
||||||
|
EndUnitID: value.EndUnitID.Int(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
200
internal/modules/dnd/extract/npcs/extractor.go
Normal file
200
internal/modules/dnd/extract/npcs/extractor.go
Normal file
@@ -0,0 +1,200 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "dnd/npcs"
|
||||||
|
|
||||||
|
var requiredCapabilities = []string{
|
||||||
|
"chunks",
|
||||||
|
"source.transcript",
|
||||||
|
}
|
||||||
|
|
||||||
|
var providedCapabilities = []string{
|
||||||
|
"dnd.npcs",
|
||||||
|
}
|
||||||
|
|
||||||
|
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
||||||
|
Glossary: "Optional campaign glossary reference material used only for NPC disambiguation.",
|
||||||
|
Party: "Optional party roster reference material used only for NPC disambiguation.",
|
||||||
|
Players: "Optional player list reference material used only for NPC disambiguation.",
|
||||||
|
Roster: "Deprecated alias for party roster reference material used only for NPC disambiguation.",
|
||||||
|
}
|
||||||
|
|
||||||
|
func referenceSlots() []contracts.ReferenceSlot {
|
||||||
|
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ contracts.Extractor[dnd.NPCList] = (*Extractor)(nil)
|
||||||
|
var _ contracts.ManifestMetadataProvider = (*Extractor)(nil)
|
||||||
|
var _ pipeline.CheckpointFingerprintProvider = (*Extractor)(nil)
|
||||||
|
|
||||||
|
type Options struct{}
|
||||||
|
|
||||||
|
type Extractor struct {
|
||||||
|
llm contracts.StructuredLLMClient
|
||||||
|
promptSHA string
|
||||||
|
responseSchemaSHA string
|
||||||
|
}
|
||||||
|
|
||||||
|
func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contracts.ReferenceSet) (*Extractor, error) {
|
||||||
|
if llmClient == nil {
|
||||||
|
return nil, extractorErrorf("LLM client must not be nil")
|
||||||
|
}
|
||||||
|
if len(references) > 1 {
|
||||||
|
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||||
|
}
|
||||||
|
promptSHA, err := scriptoriumPromptMetadata()
|
||||||
|
if err != nil {
|
||||||
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
|
}
|
||||||
|
responseSchema, err := loadResponseSchema()
|
||||||
|
if err != nil {
|
||||||
|
return nil, extractorErrorf("load response schema: %w", err)
|
||||||
|
}
|
||||||
|
return &Extractor{
|
||||||
|
llm: llmClient,
|
||||||
|
promptSHA: promptSHA,
|
||||||
|
responseSchemaSHA: responseSchema.SHA256,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) Key() string { return Key }
|
||||||
|
|
||||||
|
func (e *Extractor) ReferenceSlots() []contracts.ReferenceSlot { return referenceSlots() }
|
||||||
|
|
||||||
|
func (e *Extractor) ManifestMetadata() map[string]any {
|
||||||
|
if e == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return map[string]any{
|
||||||
|
"prompt_id": PromptID,
|
||||||
|
"prompt_version": SchemaVersion,
|
||||||
|
"prompt_sha256": e.promptSHA,
|
||||||
|
"response_schema_key": string(ResponseSchemaKey),
|
||||||
|
"response_schema_id": ResponseSchemaID,
|
||||||
|
"response_schema_name": ResponseSchemaName,
|
||||||
|
"response_schema_version": SchemaVersion,
|
||||||
|
"response_schema_sha256": e.responseSchemaSHA,
|
||||||
|
"identity_policy": identity.Policy,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||||
|
if e == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []pipeline.CheckpointFingerprint{
|
||||||
|
{Name: "prompt", Value: e.promptSHA},
|
||||||
|
{Name: "response_schema", Value: e.responseSchemaSHA},
|
||||||
|
{Name: "identity_policy", Value: identity.Policy},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.NPCList], error) {
|
||||||
|
if e == nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("extractor must not be nil")
|
||||||
|
}
|
||||||
|
if e.llm == nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("LLM client must not be nil")
|
||||||
|
}
|
||||||
|
if ctx == nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("context must not be nil")
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("context error before extraction: %w", err)
|
||||||
|
}
|
||||||
|
if req.Source == nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("source must not be nil")
|
||||||
|
}
|
||||||
|
if req.Chunk == nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("chunk must not be nil")
|
||||||
|
}
|
||||||
|
if len(req.Chunk.Units) == 0 {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("chunk %q units must not be empty", req.Chunk.ID)
|
||||||
|
}
|
||||||
|
sourceInput, err := chunkSourceInput(req)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, err
|
||||||
|
}
|
||||||
|
|
||||||
|
var response extractionResponse
|
||||||
|
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||||
|
StageName: Key,
|
||||||
|
PromptID: PromptID,
|
||||||
|
PromptVersion: SchemaVersion,
|
||||||
|
ProfileID: req.LLMProfile,
|
||||||
|
SessionID: req.SessionID,
|
||||||
|
Inputs: shared.PromptInputs(sourceInput, req.References),
|
||||||
|
}, &response); err != nil {
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{}, extractorErrorf("complete structured output: %w", err)
|
||||||
|
}
|
||||||
|
canonicalizeResponse(&response, req.Source)
|
||||||
|
return contracts.TypedExtractionResult[dnd.NPCList]{Value: canonicalNPCList(response, req.Source.ID)}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func chunkSourceInput(req contracts.TypedExtractionRequest) (contracts.LLMInputMaterial, error) {
|
||||||
|
material := req.SourceInput.Clone()
|
||||||
|
if len(material.Content) == 0 {
|
||||||
|
material = contracts.NewLLMInputMaterial("source", req.Chunk.MediaType, req.Chunk.Content, "", "")
|
||||||
|
}
|
||||||
|
if !bytes.Equal(material.Content, req.Chunk.Content) {
|
||||||
|
return contracts.LLMInputMaterial{}, extractorErrorf("source input must match chunk %q content", req.Chunk.ID)
|
||||||
|
}
|
||||||
|
if material.Name == "" {
|
||||||
|
material.Name = "source"
|
||||||
|
}
|
||||||
|
if material.MediaType == "" {
|
||||||
|
material.MediaType = req.Chunk.MediaType
|
||||||
|
}
|
||||||
|
if material.SizeBytes == 0 {
|
||||||
|
material.SizeBytes = int64(len(material.Content))
|
||||||
|
}
|
||||||
|
return material, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
|
return pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
|
ArtifactKind: dnd.NPCListKind,
|
||||||
|
ReferenceSlots: referenceSlots(),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.ExtractorRegistry) error {
|
||||||
|
return pipeline.RegisterExtractorBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Extractor[dnd.NPCList], error) {
|
||||||
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(request.Dependencies.LLM, options, request.References)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOptions(options map[string]any) error {
|
||||||
|
_, err := DecodeOptions(options)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, extractorErrorf("%w", err)
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractorErrorf(format string, args ...any) error {
|
||||||
|
return fmt.Errorf("dnd npcs extractor: "+format, args...)
|
||||||
|
}
|
||||||
110
internal/modules/dnd/extract/npcs/extractor_test.go
Normal file
110
internal/modules/dnd/extract/npcs/extractor_test.go
Normal file
@@ -0,0 +1,110 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExtractReturnsCanonicalNPCListFromPrivateResponse(t *testing.T) {
|
||||||
|
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{
|
||||||
|
{
|
||||||
|
Name: "Captain Vale", Aliases: []string{"The Captain"}, Description: "A road captain.",
|
||||||
|
Relationships: []npcRelationshipResponse{{Target: "Mira Thorn", Relationship: "reports to"}},
|
||||||
|
SourceRefs: responseSourceRefs(3, 3),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A guarded ranger.",
|
||||||
|
Relationships: []npcRelationshipResponse{{Target: "Captain Vale", Relationship: "commands"}},
|
||||||
|
SourceRefs: []npcSourceRefResponse{
|
||||||
|
{StartUnitID: sharedUnitRef(2), EndUnitID: sharedUnitRef(2)},
|
||||||
|
{StartUnitID: sharedUnitRef(1), EndUnitID: sharedUnitRef(2)},
|
||||||
|
{StartUnitID: sharedUnitRef(1), EndUnitID: sharedUnitRef(2)},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}}}
|
||||||
|
|
||||||
|
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
want := dnd.NPCList{NPCs: []dnd.NPC{
|
||||||
|
{ID: identity.DeriveID("Mira Thorn"), Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A guarded ranger.", Relationships: []dnd.NPCRelationship{{Target: "Captain Vale", Relationship: "commands"}}, SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}, {SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}},
|
||||||
|
{ID: identity.DeriveID("Captain Vale"), Name: "Captain Vale", Aliases: []string{"The Captain"}, Description: "A road captain.", Relationships: []dnd.NPCRelationship{{Target: "Mira Thorn", Relationship: "reports to"}}, SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 3}}},
|
||||||
|
}}
|
||||||
|
if !reflect.DeepEqual(result.Value, want) {
|
||||||
|
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 1 {
|
||||||
|
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
||||||
|
}
|
||||||
|
request := client.requests[0]
|
||||||
|
if request.StageName != Key || request.PromptID != PromptID || request.PromptVersion != SchemaVersion || request.ProfileID != "profile-npcs" || request.SessionID != "session-123" {
|
||||||
|
t.Fatalf("LLM request identity = %#v", request)
|
||||||
|
}
|
||||||
|
transcript := request.Inputs["transcript"]
|
||||||
|
if transcript.Name != "transcript" || transcript.MediaType != "application/json" || transcript.Digest != "sha256:chunk" || transcript.OriginURI != "file:///session-alpha.json" || string(transcript.Content) != string(extractionRequest().Chunk.Content) {
|
||||||
|
t.Fatalf("transcript input = %#v, want chunk-scoped material", transcript)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractPassesCampaignReferencesAsPromptInputs(t *testing.T) {
|
||||||
|
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{}}}
|
||||||
|
req := extractionRequest()
|
||||||
|
req.References = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
|
"players": {Slot: contracts.ReferenceSlot{Name: "players"}, Items: []contracts.ReferenceItem{{SlotName: "players", Content: []byte("Dana: Mira")}}},
|
||||||
|
"party": {Slot: contracts.ReferenceSlot{Name: "party"}, Items: []contracts.ReferenceItem{{SlotName: "party", Content: []byte("Mira: ranger")}}},
|
||||||
|
"glossary": {Slot: contracts.ReferenceSlot{Name: "glossary"}, Items: []contracts.ReferenceItem{{SlotName: "glossary", Content: []byte("Greencloak: local title")}}},
|
||||||
|
}}
|
||||||
|
if _, err := newExtractor(t, client).Extract(context.Background(), req); err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
inputs := client.requests[0].Inputs
|
||||||
|
if string(inputs["players"].Content) != "Dana: Mira" || string(inputs["party"].Content) != "Mira: ranger" || string(inputs["glossary"].Content) != "Greencloak: local title" {
|
||||||
|
t.Fatalf("reference inputs = %#v", inputs)
|
||||||
|
}
|
||||||
|
if strings.Contains(string(inputs["transcript"].Content), "local title") {
|
||||||
|
t.Fatal("transcript input contains reference content")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractPreservesMalformedCandidatesForValidators(t *testing.T) {
|
||||||
|
client := &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{{
|
||||||
|
Name: "", Aliases: nil, Description: "", Relationships: nil,
|
||||||
|
SourceRefs: []npcSourceRefResponse{{StartUnitID: sharedUnitRef(99), EndUnitID: shared.UnitRefFromString("missing")}, {StartUnitID: sharedUnitRef(99), EndUnitID: shared.UnitRefFromInt(0)}},
|
||||||
|
}}}}
|
||||||
|
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(result.Value.NPCs) != 1 || result.Value.NPCs[0].ID != "" || result.Value.NPCs[0].Name != "" || result.Value.NPCs[0].Aliases != nil || result.Value.NPCs[0].Relationships != nil {
|
||||||
|
t.Fatalf("malformed candidate = %#v, want invalid values preserved", result.Value)
|
||||||
|
}
|
||||||
|
if refs := result.Value.NPCs[0].SourceRefs; len(refs) != 1 || refs[0].SourceID != "session-alpha" || refs[0].StartUnitID != 99 || refs[0].EndUnitID != 0 {
|
||||||
|
t.Fatalf("malformed source refs = %#v, want invalid range preserved after exact deduplication", refs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractHandlesCancellationAndProviderErrors(t *testing.T) {
|
||||||
|
request := extractionRequest()
|
||||||
|
extractor := newExtractor(t, &fakeNPCsLLMClient{response: extractionResponse{NPCs: []npcResponse{}}})
|
||||||
|
canceled, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
if _, err := extractor.Extract(canceled, request); err == nil || !strings.Contains(err.Error(), "context") {
|
||||||
|
t.Fatalf("canceled Extract() error = %v, want context error", err)
|
||||||
|
}
|
||||||
|
_, err := newExtractor(t, &fakeNPCsLLMClient{err: errors.New("provider unavailable")}).Extract(context.Background(), request)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "dnd npcs") || !strings.Contains(err.Error(), "provider unavailable") {
|
||||||
|
t.Fatalf("provider Extract() error = %v, want contextual provider error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func sharedUnitRef(value int) shared.UnitRef { return shared.UnitRefFromInt(value) }
|
||||||
25
internal/modules/dnd/extract/npcs/model.go
Normal file
25
internal/modules/dnd/extract/npcs/model.go
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
|
||||||
|
type extractionResponse struct {
|
||||||
|
NPCs []npcResponse `json:"npcs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcResponse struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Aliases []string `json:"aliases"`
|
||||||
|
Description string `json:"description"`
|
||||||
|
Relationships []npcRelationshipResponse `json:"relationships"`
|
||||||
|
SourceRefs []npcSourceRefResponse `json:"source_refs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcRelationshipResponse struct {
|
||||||
|
Target string `json:"target"`
|
||||||
|
Relationship string `json:"relationship"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcSourceRefResponse struct {
|
||||||
|
StartUnitID shared.UnitRef `json:"start_unit_id"`
|
||||||
|
EndUnitID shared.UnitRef `json:"end_unit_id"`
|
||||||
|
}
|
||||||
100
internal/modules/dnd/extract/npcs/registry_test.go
Normal file
100
internal/modules/dnd/extract/npcs/registry_test.go
Normal file
@@ -0,0 +1,100 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNewRequiresLLMClientAndRejectsAmbiguousReferences(t *testing.T) {
|
||||||
|
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
|
||||||
|
t.Fatalf("New(nil) error = %v, want dependency error", err)
|
||||||
|
}
|
||||||
|
if _, err := New(&fakeNPCsLLMClient{}, Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one reference set") {
|
||||||
|
t.Fatalf("New() error = %v, want reference-set error", err)
|
||||||
|
}
|
||||||
|
if got := newExtractor(t, &fakeNPCsLLMClient{}).Key(); got != Key {
|
||||||
|
t.Fatalf("Key() = %q, want %q", got, Key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestModuleSpecAndReferenceSlots(t *testing.T) {
|
||||||
|
got := ModuleSpec()
|
||||||
|
want := pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
Requires: []string{"chunks", "source.transcript"},
|
||||||
|
Provides: []string{"dnd.npcs"},
|
||||||
|
ArtifactKind: dnd.NPCListKind,
|
||||||
|
ReferenceSlots: []contracts.ReferenceSlot{
|
||||||
|
{Name: "glossary", Description: "Optional campaign glossary reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
{Name: "party", Description: "Optional party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
{Name: "players", Description: "Optional player list reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
{Name: "roster", Description: "Deprecated alias for party roster reference material used only for NPC disambiguation.", AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"}},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
got.Requires[0] = "changed"
|
||||||
|
got.Provides[0] = "changed"
|
||||||
|
got.ReferenceSlots[0].AcceptedMediaTypes[0] = "changed"
|
||||||
|
if !reflect.DeepEqual(ModuleSpec(), want) {
|
||||||
|
t.Fatal("ModuleSpec() returned mutable shared slices")
|
||||||
|
}
|
||||||
|
extractor := newExtractor(t, &fakeNPCsLLMClient{})
|
||||||
|
if !reflect.DeepEqual(extractor.ReferenceSlots(), want.ReferenceSlots) {
|
||||||
|
t.Fatalf("ReferenceSlots() = %#v, want %#v", extractor.ReferenceSlots(), want.ReferenceSlots)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterStoresTypedModuleSpec(t *testing.T) {
|
||||||
|
registry := pipeline.NewExtractorRegistry()
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
got, ok := registry.Spec(Key)
|
||||||
|
if !ok || !reflect.DeepEqual(got, ModuleSpec()) {
|
||||||
|
t.Fatalf("registry.Spec(%q) = %#v, present = %t", Key, got, ok)
|
||||||
|
}
|
||||||
|
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "extractor registry") {
|
||||||
|
t.Fatalf("Register(nil) error = %v, want registry error", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("DecodeOptions() error = %v, want strict options error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractorMetadataAndCheckpointIdentity(t *testing.T) {
|
||||||
|
extractor := newExtractor(t, &fakeNPCsLLMClient{})
|
||||||
|
metadata := extractor.ManifestMetadata()
|
||||||
|
for key, want := range map[string]string{
|
||||||
|
"prompt_id": PromptID, "prompt_version": SchemaVersion,
|
||||||
|
"response_schema_key": string(ResponseSchemaKey), "response_schema_id": ResponseSchemaID,
|
||||||
|
"response_schema_name": ResponseSchemaName, "response_schema_version": SchemaVersion,
|
||||||
|
"identity_policy": "dnd.npcs.identity.v1",
|
||||||
|
} {
|
||||||
|
if metadata[key] != want {
|
||||||
|
t.Fatalf("metadata[%q] = %#v, want %q", key, metadata[key], want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, key := range []string{"prompt_sha256", "response_schema_sha256"} {
|
||||||
|
if value, ok := metadata[key].(string); !ok || !strings.HasPrefix(value, "sha256:") {
|
||||||
|
t.Fatalf("metadata[%q] = %#v, want hash", key, metadata[key])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
fingerprints := extractor.CheckpointFingerprints()
|
||||||
|
want := map[string]string{"prompt": metadata["prompt_sha256"].(string), "response_schema": metadata["response_schema_sha256"].(string), "identity_policy": "dnd.npcs.identity.v1"}
|
||||||
|
if len(fingerprints) != len(want) {
|
||||||
|
t.Fatalf("CheckpointFingerprints() = %#v, want %d entries", fingerprints, len(want))
|
||||||
|
}
|
||||||
|
for _, fingerprint := range fingerprints {
|
||||||
|
if fingerprint.Value != want[fingerprint.Name] {
|
||||||
|
t.Fatalf("fingerprint %q = %q, want %#v", fingerprint.Name, fingerprint.Value, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
21
internal/modules/dnd/extract/npcs/schema.go
Normal file
21
internal/modules/dnd/extract/npcs/schema.go
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
|
||||||
|
const (
|
||||||
|
PromptID = "dnd.npcs"
|
||||||
|
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_npcs_llm")
|
||||||
|
ResponseSchemaID = "notarius.dnd.npcs.llm"
|
||||||
|
ResponseSchemaName = "notarius_dnd_npcs_llm_v1"
|
||||||
|
SchemaVersion = "v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
func loadResponseSchema() (llm.ResponseSchema, error) {
|
||||||
|
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
|
||||||
|
Key: ResponseSchemaKey,
|
||||||
|
ID: ResponseSchemaID,
|
||||||
|
Version: SchemaVersion,
|
||||||
|
Name: ResponseSchemaName,
|
||||||
|
AssetPath: "assets/schemas/dnd_npcs_llm.v1.json",
|
||||||
|
})
|
||||||
|
}
|
||||||
78
internal/modules/dnd/extract/npcs/schema_test.go
Normal file
78
internal/modules/dnd/extract/npcs/schema_test.go
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLoadResponseSchemaUsesPrivateNPCSchema(t *testing.T) {
|
||||||
|
schema, err := loadResponseSchema()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("loadResponseSchema() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Version != SchemaVersion || schema.Name != ResponseSchemaName || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
|
||||||
|
t.Fatalf("schema = %#v, want private NPC schema identity", schema)
|
||||||
|
}
|
||||||
|
valid := map[string]any{"npcs": []any{map[string]any{
|
||||||
|
"name": "Mira Thorn", "aliases": []any{}, "description": "A ranger.", "relationships": []any{},
|
||||||
|
"source_refs": []any{map[string]any{"start_unit_id": 1, "end_unit_id": 2}},
|
||||||
|
}}}
|
||||||
|
validJSON, err := json.Marshal(valid)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := validateJSONSchema(validJSON, schema.JSONSchema); err != nil {
|
||||||
|
t.Fatalf("valid private NPC response rejected: %v", err)
|
||||||
|
}
|
||||||
|
withID := map[string]any{"npcs": []any{map[string]any{
|
||||||
|
"name": "Mira Thorn", "id": "assigned-later", "aliases": []any{}, "description": "A ranger.", "relationships": []any{},
|
||||||
|
"source_refs": []any{map[string]any{"start_unit_id": 1, "end_unit_id": 2}},
|
||||||
|
}}}
|
||||||
|
withIDJSON, err := json.Marshal(withID)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := validateJSONSchema(withIDJSON, schema.JSONSchema); err == nil {
|
||||||
|
t.Fatal("private schema accepted framework-assigned id")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResponseSchemaIsMutationSafeAndDiagnosticsRedactContent(t *testing.T) {
|
||||||
|
first, err := loadResponseSchema()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
first.JSONSchema[0] = '['
|
||||||
|
second, err := loadResponseSchema()
|
||||||
|
if err != nil || !json.Valid(second.JSONSchema) || bytes.Equal(first.JSONSchema, second.JSONSchema) {
|
||||||
|
t.Fatalf("second schema = %s, %v; want defensive copy", second.JSONSchema, err)
|
||||||
|
}
|
||||||
|
diagnostics := second.DiagnosticsMap()
|
||||||
|
if _, ok := diagnostics["json_schema"]; ok {
|
||||||
|
t.Fatalf("schema diagnostics included raw content: %#v", diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateJSONSchema(instanceContent, schemaContent []byte) error {
|
||||||
|
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(instanceContent))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
schemaDocument, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
compiler := jsonschema.NewCompiler()
|
||||||
|
if err := compiler.AddResource("schema.json", schemaDocument); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
compiled, err := compiler.Compile("schema.json")
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return compiled.Validate(instance)
|
||||||
|
}
|
||||||
45
internal/modules/dnd/extract/npcs/scriptorium_assets.go
Normal file
45
internal/modules/dnd/extract/npcs/scriptorium_assets.go
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
)
|
||||||
|
|
||||||
|
const scriptoriumPromptRoot = "assets/prompts"
|
||||||
|
|
||||||
|
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
||||||
|
promptFS, err := shared.ModulePromptFS("dnd.npcs", embeddedAssets, []promptfs.ModulePromptFile{
|
||||||
|
{Name: "dnd.npcs.yaml", Path: "assets/prompts/dnd.npcs.yaml"},
|
||||||
|
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||||
|
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("prepare NPC prompt assets: %w", err)
|
||||||
|
}
|
||||||
|
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||||
|
}
|
||||||
|
|
||||||
|
func scriptoriumPromptMetadata() (string, error) {
|
||||||
|
scriptoriumPromptHashOnce.Do(func() {
|
||||||
|
parts := append([]llm.AssetHashPart{
|
||||||
|
{FS: embeddedAssets, Path: "assets/prompts/dnd.npcs.yaml"},
|
||||||
|
{FS: embeddedAssets, Path: "assets/prompts/task.md"},
|
||||||
|
{FS: embeddedAssets, Path: "assets/prompts/instructions.md"},
|
||||||
|
}, append(shared.CommonHashParts(), shared.ReferenceHashParts()...)...)
|
||||||
|
scriptoriumPromptHash, scriptoriumPromptHashErr = llm.HashAssets(parts)
|
||||||
|
})
|
||||||
|
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
scriptoriumPromptHashOnce sync.Once
|
||||||
|
scriptoriumPromptHash string
|
||||||
|
scriptoriumPromptHashErr error
|
||||||
|
)
|
||||||
68
internal/modules/dnd/extract/npcs/scriptorium_assets_test.go
Normal file
68
internal/modules/dnd/extract/npcs/scriptorium_assets_test.go
Normal file
@@ -0,0 +1,68 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
"gitea.maximumdirect.net/eric/scriptorium"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRegisterPromptAssetsAndPrepareNPCPrompt(t *testing.T) {
|
||||||
|
registry := llm.NewAssetRegistry()
|
||||||
|
if err := RegisterPromptAssets(registry); err != nil {
|
||||||
|
t.Fatalf("RegisterPromptAssets() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
options, err := registry.ScriptoriumOptions()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ScriptoriumOptions() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
||||||
|
ID: "npc-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "npc-test-model",
|
||||||
|
})))
|
||||||
|
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewEngine() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
||||||
|
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "npc-test-profile",
|
||||||
|
Inputs: map[string]scriptorium.ArtifactRef{
|
||||||
|
"transcript": scriptorium.InlineWithURI("file:///session.json", `{"units":[1]}`),
|
||||||
|
"players": scriptorium.Inline("Dana: Mira"),
|
||||||
|
"party": scriptorium.Inline("Mira: ranger"),
|
||||||
|
"glossary": scriptorium.Inline("Greencloak: title"),
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Prepare() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_npcs_llm.v1.json" || len(prepared.Messages) != 5 {
|
||||||
|
t.Fatalf("prepared prompt = %#v, want NPC prompt identity and wiring", prepared)
|
||||||
|
}
|
||||||
|
if !strings.Contains(prepared.Messages[1].Content, `{"units":[1]}`) || !strings.Contains(prepared.Messages[2].Content, "Dana: Mira") || !strings.Contains(prepared.Messages[2].Content, "Mira: ranger") {
|
||||||
|
t.Fatalf("prepared prompt inputs do not include transcript/references")
|
||||||
|
}
|
||||||
|
if strings.Contains(prepared.Messages[3].Content, `{"units":[1]}`) || strings.Contains(prepared.Messages[4].Content, `{"units":[1]}`) {
|
||||||
|
t.Fatal("task or instruction prompt leaked raw transcript")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptMetadataAndDiagnosticsDoNotContainRawAssets(t *testing.T) {
|
||||||
|
hash, err := scriptoriumPromptMetadata()
|
||||||
|
if err != nil || !strings.HasPrefix(hash, "sha256:") {
|
||||||
|
t.Fatalf("scriptoriumPromptMetadata() = %q, %v; want hash", hash, err)
|
||||||
|
}
|
||||||
|
metadata := newExtractor(t, &fakeNPCsLLMClient{}).ManifestMetadata()
|
||||||
|
payload, err := json.Marshal(metadata)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
for _, forbidden := range []string{"Include an in-world", "common-dnd-system", "source-unit", "dnd_npcs_llm.v1.json"} {
|
||||||
|
if strings.Contains(string(payload), forbidden) {
|
||||||
|
t.Fatalf("metadata leaked raw prompt/schema content %q: %s", forbidden, payload)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
112
internal/modules/dnd/extract/npcs/test_helpers_test.go
Normal file
112
internal/modules/dnd/extract/npcs/test_helpers_test.go
Normal file
@@ -0,0 +1,112 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||||
|
)
|
||||||
|
|
||||||
|
func extractionRequest() contracts.TypedExtractionRequest {
|
||||||
|
doc := sourceDocument()
|
||||||
|
chunk := &source.Chunk{
|
||||||
|
ID: "session-alpha:chunk:0",
|
||||||
|
SourceID: doc.ID,
|
||||||
|
Index: 0,
|
||||||
|
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 3},
|
||||||
|
Content: []byte(`{"units":[1,2,3]}`),
|
||||||
|
MediaType: "application/json",
|
||||||
|
Units: append([]source.SourceUnit(nil), doc.Units...),
|
||||||
|
Metadata: map[string]any{"ignored": "chunk metadata"},
|
||||||
|
}
|
||||||
|
return contracts.TypedExtractionRequest{
|
||||||
|
Source: doc,
|
||||||
|
Chunk: chunk,
|
||||||
|
SourceInput: contracts.NewLLMInputMaterial("source", chunk.MediaType, chunk.Content, "sha256:chunk", "file:///session-alpha.json"),
|
||||||
|
SessionID: "session-123",
|
||||||
|
LLMProfile: "profile-npcs",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func sourceDocument() *source.SourceDocument {
|
||||||
|
return &source.SourceDocument{
|
||||||
|
ID: "session-alpha",
|
||||||
|
Kind: "transcript",
|
||||||
|
Format: "application/vnd.seriatim.minimal+json",
|
||||||
|
Digest: "sha256:test",
|
||||||
|
Units: []source.SourceUnit{
|
||||||
|
{ID: 1, Kind: "transcript_segment", Text: "Mira Thorn greets the party."},
|
||||||
|
{ID: 2, Kind: "transcript_segment", Text: "The Greencloak watches the northern road."},
|
||||||
|
{ID: 3, Kind: "transcript_segment", Text: "Captain Vale reports to Mira Thorn."},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func responseSourceRefs(startUnitID, endUnitID int) []npcSourceRefResponse {
|
||||||
|
return []npcSourceRefResponse{{StartUnitID: shared.UnitRefFromInt(startUnitID), EndUnitID: shared.UnitRefFromInt(endUnitID)}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newExtractor(t *testing.T, client contracts.StructuredLLMClient, references ...contracts.ReferenceSet) *Extractor {
|
||||||
|
t.Helper()
|
||||||
|
extractor, err := New(client, Options{}, references...)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return extractor
|
||||||
|
}
|
||||||
|
|
||||||
|
func emptyChunkRequest(req contracts.TypedExtractionRequest) contracts.TypedExtractionRequest {
|
||||||
|
req.Chunk = &source.Chunk{ID: req.Chunk.ID, SourceID: req.Chunk.SourceID, Index: req.Chunk.Index}
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
func mismatchedSourceInputRequest(req contracts.TypedExtractionRequest) contracts.TypedExtractionRequest {
|
||||||
|
req.SourceInput = contracts.NewLLMInputMaterial("source", "application/json", []byte(`{"different":true}`), "sha256:other", "file:///other.json")
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||||
|
req.Inputs = req.Inputs.Clone()
|
||||||
|
if len(req.Vars) == 0 {
|
||||||
|
req.Vars = nil
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
vars := make(map[string]any, len(req.Vars))
|
||||||
|
for key, value := range req.Vars {
|
||||||
|
vars[key] = value
|
||||||
|
}
|
||||||
|
req.Vars = vars
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeNPCsLLMClient struct {
|
||||||
|
response extractionResponse
|
||||||
|
content []byte
|
||||||
|
err error
|
||||||
|
requests []contracts.StructuredCompletionRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *fakeNPCsLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
client.requests = append(client.requests, cloneStructuredCompletionRequest(req))
|
||||||
|
if client.err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, client.err
|
||||||
|
}
|
||||||
|
target, ok := out.(*extractionResponse)
|
||||||
|
if !ok {
|
||||||
|
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
|
||||||
|
}
|
||||||
|
*target = client.response
|
||||||
|
content := append([]byte(nil), client.content...)
|
||||||
|
if len(content) == 0 {
|
||||||
|
var err error
|
||||||
|
content, err = json.Marshal(client.response)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||||
|
}
|
||||||
@@ -17,6 +17,9 @@ inputs:
|
|||||||
- name: glossary
|
- name: glossary
|
||||||
required: false
|
required: false
|
||||||
content_type: text/plain
|
content_type: text/plain
|
||||||
|
- name: npcs
|
||||||
|
required: false
|
||||||
|
content_type: application/json
|
||||||
messages:
|
messages:
|
||||||
- role: system
|
- role: system
|
||||||
content_file: ./sharedassets/common-dnd-system.md
|
content_file: ./sharedassets/common-dnd-system.md
|
||||||
@@ -30,6 +33,8 @@ messages:
|
|||||||
type: ephemeral
|
type: ephemeral
|
||||||
- role: user
|
- role: user
|
||||||
content_file: ./catalog.md
|
content_file: ./catalog.md
|
||||||
|
- role: user
|
||||||
|
content_file: ./npc_registry.md
|
||||||
- role: user
|
- role: user
|
||||||
content_file: ./task.md
|
content_file: ./task.md
|
||||||
- role: user
|
- role: user
|
||||||
|
|||||||
@@ -0,0 +1,11 @@
|
|||||||
|
The optional canonical NPC registry for this extraction is provided below as
|
||||||
|
durable JSON. Use it only to prefer exact canonical NPC names and recognize
|
||||||
|
their aliases when the transcript identifies a caster.
|
||||||
|
|
||||||
|
Registry entries are grounding material, not evidence that a spell was cast.
|
||||||
|
Do not extract a spell, caster, effect, or source reference from the registry.
|
||||||
|
NPC source references describe registry provenance and may belong to another
|
||||||
|
session; they are never spell evidence. Preserve the existing player and party
|
||||||
|
policy for identifying PCs from the transcript.
|
||||||
|
|
||||||
|
{{ input "npcs" }}
|
||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"sort"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
@@ -34,12 +35,19 @@ var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
|||||||
|
|
||||||
func referenceSlots() []contracts.ReferenceSlot {
|
func referenceSlots() []contracts.ReferenceSlot {
|
||||||
slots := shared.ReferenceSlots(referenceSlotDescriptions)
|
slots := shared.ReferenceSlots(referenceSlotDescriptions)
|
||||||
return append(slots, contracts.ReferenceSlot{
|
slots = append(slots, contracts.ReferenceSlot{
|
||||||
Name: spellcatalog.SpellCatalogReferenceSlot,
|
Name: spellcatalog.SpellCatalogReferenceSlot,
|
||||||
Description: "Optional canonical spell-name catalog used for extraction grounding.",
|
Description: "Optional canonical spell-name catalog used for extraction grounding.",
|
||||||
AcceptedMediaTypes: []string{"application/json"},
|
AcceptedMediaTypes: []string{"application/json"},
|
||||||
MaxBytes: 1048576,
|
MaxBytes: 1048576,
|
||||||
|
}, contracts.ReferenceSlot{
|
||||||
|
Name: NPCRegistryReferenceSlot,
|
||||||
|
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
|
||||||
|
AcceptedMediaTypes: []string{"application/json"},
|
||||||
|
MaxBytes: NPCRegistryMaxBytes,
|
||||||
})
|
})
|
||||||
|
sort.Slice(slots, func(i, j int) bool { return slots[i].Name < slots[j].Name })
|
||||||
|
return slots
|
||||||
}
|
}
|
||||||
|
|
||||||
var _ contracts.Extractor[dnd.SpellList] = (*Extractor)(nil)
|
var _ contracts.Extractor[dnd.SpellList] = (*Extractor)(nil)
|
||||||
@@ -51,6 +59,7 @@ type Extractor struct {
|
|||||||
llm contracts.StructuredLLMClient
|
llm contracts.StructuredLLMClient
|
||||||
effectiveCatalog spellcatalog.EffectiveCatalog
|
effectiveCatalog spellcatalog.EffectiveCatalog
|
||||||
catalogPromptInput contracts.LLMInputMaterial
|
catalogPromptInput contracts.LLMInputMaterial
|
||||||
|
npcRegistry npcRegistryPromptInput
|
||||||
promptSHA string
|
promptSHA string
|
||||||
responseSchemaSHA string
|
responseSchemaSHA string
|
||||||
}
|
}
|
||||||
@@ -74,6 +83,10 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("prepare spell catalog prompt input: %w", err)
|
return nil, extractorErrorf("prepare spell catalog prompt input: %w", err)
|
||||||
}
|
}
|
||||||
|
npcRegistry, err := resolveNPCRegistry(referenceSet)
|
||||||
|
if err != nil {
|
||||||
|
return nil, extractorErrorf("prepare NPC registry prompt input: %w", err)
|
||||||
|
}
|
||||||
promptSHA, err := scriptoriumPromptMetadata()
|
promptSHA, err := scriptoriumPromptMetadata()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||||
@@ -86,6 +99,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contr
|
|||||||
llm: llmClient,
|
llm: llmClient,
|
||||||
effectiveCatalog: effectiveCatalog,
|
effectiveCatalog: effectiveCatalog,
|
||||||
catalogPromptInput: catalogPromptInput,
|
catalogPromptInput: catalogPromptInput,
|
||||||
|
npcRegistry: npcRegistry,
|
||||||
promptSHA: promptSHA,
|
promptSHA: promptSHA,
|
||||||
responseSchemaSHA: responseSchema.SHA256,
|
responseSchemaSHA: responseSchema.SHA256,
|
||||||
}, nil
|
}, nil
|
||||||
@@ -113,6 +127,10 @@ func (e *Extractor) ManifestMetadata() map[string]any {
|
|||||||
"response_schema_version": SchemaVersion,
|
"response_schema_version": SchemaVersion,
|
||||||
"response_schema_sha256": e.responseSchemaSHA,
|
"response_schema_sha256": e.responseSchemaSHA,
|
||||||
}
|
}
|
||||||
|
if e.npcRegistry.bound {
|
||||||
|
metadata["npc_registry_digest"] = e.npcRegistry.digest
|
||||||
|
metadata["npc_count"] = e.npcRegistry.count
|
||||||
|
}
|
||||||
return metadata
|
return metadata
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -120,11 +138,15 @@ func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
|||||||
if e == nil {
|
if e == nil {
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
return []pipeline.CheckpointFingerprint{
|
fingerprints := []pipeline.CheckpointFingerprint{
|
||||||
{Name: "effective_catalog", Value: e.effectiveCatalog.Digest()},
|
{Name: "effective_catalog", Value: e.effectiveCatalog.Digest()},
|
||||||
{Name: "prompt", Value: e.promptSHA},
|
{Name: "prompt", Value: e.promptSHA},
|
||||||
{Name: "response_schema", Value: e.responseSchemaSHA},
|
{Name: "response_schema", Value: e.responseSchemaSHA},
|
||||||
}
|
}
|
||||||
|
if e.npcRegistry.bound {
|
||||||
|
fingerprints = append(fingerprints, pipeline.CheckpointFingerprint{Name: "npc_registry", Value: e.npcRegistry.digest})
|
||||||
|
}
|
||||||
|
return fingerprints
|
||||||
}
|
}
|
||||||
|
|
||||||
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.SpellList], error) {
|
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.SpellList], error) {
|
||||||
@@ -157,6 +179,7 @@ func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRe
|
|||||||
var response extractionResponse
|
var response extractionResponse
|
||||||
inputs := shared.PromptInputs(sourceInput, req.References)
|
inputs := shared.PromptInputs(sourceInput, req.References)
|
||||||
inputs[spellcatalog.SpellCatalogReferenceSlot] = e.catalogPromptInput.Clone()
|
inputs[spellcatalog.SpellCatalogReferenceSlot] = e.catalogPromptInput.Clone()
|
||||||
|
inputs[NPCRegistryReferenceSlot] = e.npcRegistry.input.Clone()
|
||||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||||
StageName: Key,
|
StageName: Key,
|
||||||
PromptID: PromptID,
|
PromptID: PromptID,
|
||||||
|
|||||||
89
internal/modules/dnd/extract/spells/npc_registry.go
Normal file
89
internal/modules/dnd/extract/spells/npc_registry.go
Normal file
@@ -0,0 +1,89 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"mime"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
NPCRegistryReferenceSlot = "npcs"
|
||||||
|
NPCRegistryMaxBytes = 1048576
|
||||||
|
)
|
||||||
|
|
||||||
|
type npcRegistryPromptInput struct {
|
||||||
|
input contracts.LLMInputMaterial
|
||||||
|
digest string
|
||||||
|
count int
|
||||||
|
bound bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveNPCRegistry(references contracts.ReferenceSet) (npcRegistryPromptInput, error) {
|
||||||
|
slot, ok := references.Slots[NPCRegistryReferenceSlot]
|
||||||
|
if !ok {
|
||||||
|
return npcRegistryPromptInput{
|
||||||
|
input: contracts.NewLLMInputMaterial(
|
||||||
|
NPCRegistryReferenceSlot,
|
||||||
|
"application/json",
|
||||||
|
[]byte(`{"npcs":[]}`),
|
||||||
|
"",
|
||||||
|
"",
|
||||||
|
),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
if len(slot.Items) != 1 {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q must contain exactly one item", NPCRegistryReferenceSlot)
|
||||||
|
}
|
||||||
|
|
||||||
|
item := slot.Items[0]
|
||||||
|
mediaType, _, err := mime.ParseMediaType(item.MediaType)
|
||||||
|
if err != nil {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q item media type %q is invalid: %w", NPCRegistryReferenceSlot, item.MediaType, err)
|
||||||
|
}
|
||||||
|
if !strings.EqualFold(mediaType, "application/json") {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q item media type %q must be application/json", NPCRegistryReferenceSlot, item.MediaType)
|
||||||
|
}
|
||||||
|
if len(item.Content) > NPCRegistryMaxBytes {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("reference slot %q item is %d bytes, limit %d", NPCRegistryReferenceSlot, len(item.Content), NPCRegistryMaxBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
codec := npccodec.New()
|
||||||
|
value, err := codec.Decode(item.Content)
|
||||||
|
if err != nil {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("decode NPC registry: %w", err)
|
||||||
|
}
|
||||||
|
if issues := identity.ValidateList(value); len(issues) > 0 {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("validate NPC registry identity: %s", formatNPCIdentityIssues(issues))
|
||||||
|
}
|
||||||
|
content, err := codec.Encode(value)
|
||||||
|
if err != nil {
|
||||||
|
return npcRegistryPromptInput{}, fmt.Errorf("encode canonical NPC registry: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
digest := semanticNPCRegistryDigest(content)
|
||||||
|
return npcRegistryPromptInput{
|
||||||
|
input: contracts.NewLLMInputMaterial(NPCRegistryReferenceSlot, "application/json", content, digest, ""),
|
||||||
|
digest: digest,
|
||||||
|
count: len(value.NPCs),
|
||||||
|
bound: true,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func semanticNPCRegistryDigest(content []byte) string {
|
||||||
|
sum := sha256.Sum256(content)
|
||||||
|
return "sha256:" + hex.EncodeToString(sum[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatNPCIdentityIssues(issues []identity.Issue) string {
|
||||||
|
parts := make([]string, len(issues))
|
||||||
|
for index, issue := range issues {
|
||||||
|
parts[index] = fmt.Sprintf("%s at record %d", issue.Code, issue.RecordIndex)
|
||||||
|
}
|
||||||
|
return strings.Join(parts, ", ")
|
||||||
|
}
|
||||||
236
internal/modules/dnd/extract/spells/npc_registry_test.go
Normal file
236
internal/modules/dnd/extract/spells/npc_registry_test.go
Normal file
@@ -0,0 +1,236 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestResolveNPCRegistryUsesExactEmptyPromptWhenUnbound(t *testing.T) {
|
||||||
|
resolved, err := resolveNPCRegistry(contracts.ReferenceSet{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resolveNPCRegistry() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if resolved.bound || resolved.digest != "" || resolved.count != 0 {
|
||||||
|
t.Fatalf("resolved unbound registry = %#v, want no semantic metadata", resolved)
|
||||||
|
}
|
||||||
|
if resolved.input.Name != NPCRegistryReferenceSlot || resolved.input.MediaType != "application/json" || resolved.input.Digest != "" || resolved.input.OriginURI != "" {
|
||||||
|
t.Fatalf("unbound prompt input metadata = %#v, want name/media type only", resolved.input)
|
||||||
|
}
|
||||||
|
if got := string(resolved.input.Content); got != `{"npcs":[]}` {
|
||||||
|
t.Fatalf("unbound prompt input = %q, want exact empty registry", got)
|
||||||
|
}
|
||||||
|
if resolved.input.SizeBytes != int64(len(`{"npcs":[]}`)) {
|
||||||
|
t.Fatalf("unbound prompt input size = %d, want %d", resolved.input.SizeBytes, len(`{"npcs":[]}`))
|
||||||
|
}
|
||||||
|
if metadata := newExtractor(t, &fakeSpellsLLMClient{}).ManifestMetadata(); metadata["npc_registry_digest"] != nil || metadata["npc_count"] != nil {
|
||||||
|
t.Fatalf("unbound extractor metadata = %#v, want no NPC registry fields", metadata)
|
||||||
|
}
|
||||||
|
fingerprints := newExtractor(t, &fakeSpellsLLMClient{}).CheckpointFingerprints()
|
||||||
|
for _, fingerprint := range fingerprints {
|
||||||
|
if fingerprint.Name == "npc_registry" {
|
||||||
|
t.Fatalf("unbound checkpoint fingerprints = %#v, want no NPC registry fingerprint", fingerprints)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveNPCRegistryCanonicalizesContentAndUsesSemanticDigest(t *testing.T) {
|
||||||
|
value := validNPCRegistryList()
|
||||||
|
canonical := encodeNPCRegistry(t, value)
|
||||||
|
raw := append([]byte(" \n"), canonical...)
|
||||||
|
raw = append(raw, []byte("\n ")...)
|
||||||
|
|
||||||
|
resolved, err := resolveNPCRegistry(npcRegistryReference(raw, "file:///another-session/npcs.json"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resolveNPCRegistry() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if !resolved.bound || resolved.count != len(value.NPCs) {
|
||||||
|
t.Fatalf("resolved registry = %#v, want bound registry with %d NPC", resolved, len(value.NPCs))
|
||||||
|
}
|
||||||
|
if !bytes.Equal(resolved.input.Content, canonical) {
|
||||||
|
t.Fatalf("canonical prompt input = %s, want %s", resolved.input.Content, canonical)
|
||||||
|
}
|
||||||
|
if resolved.input.Digest != semanticNPCRegistryDigest(canonical) || resolved.digest != resolved.input.Digest {
|
||||||
|
t.Fatalf("semantic digest = %q/%q, want %q", resolved.input.Digest, resolved.digest, semanticNPCRegistryDigest(canonical))
|
||||||
|
}
|
||||||
|
if resolved.input.OriginURI != "" {
|
||||||
|
t.Fatalf("prompt input origin = %q, want no provenance path", resolved.input.OriginURI)
|
||||||
|
}
|
||||||
|
|
||||||
|
resolved.input.Content[0] = 'X'
|
||||||
|
again, err := resolveNPCRegistry(npcRegistryReference(raw, "file:///another-session/npcs.json"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second resolveNPCRegistry() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(again.input.Content, canonical) {
|
||||||
|
t.Fatalf("canonical content changed after caller mutation = %s, want %s", again.input.Content, canonical)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResolveNPCRegistryRejectsInvalidBoundaryValues(t *testing.T) {
|
||||||
|
valid := validNPCRegistryList()
|
||||||
|
second := validNPCRegistryList().NPCs[0]
|
||||||
|
second.ID = identity.DeriveID("Captain Vale")
|
||||||
|
second.Name = "Captain Vale"
|
||||||
|
second.Aliases = []string{"The Greencloak"}
|
||||||
|
valueWithAliasCollision := dnd.NPCList{NPCs: []dnd.NPC{valid.NPCs[0], second}}
|
||||||
|
invalidID := valid
|
||||||
|
invalidID.NPCs[0].ID = "not-an-npc-id"
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
reference contracts.ReferenceSet
|
||||||
|
wantError string
|
||||||
|
}{
|
||||||
|
{name: "zero items", reference: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{NPCRegistryReferenceSlot: {Items: []contracts.ReferenceItem{}}}}, wantError: "exactly one"},
|
||||||
|
{name: "multiple", reference: contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{NPCRegistryReferenceSlot: {Items: []contracts.ReferenceItem{{Content: []byte(`{"npcs":[]}`)}, {Content: []byte(`{"npcs":[]}`)}}}}}, wantError: "exactly one"},
|
||||||
|
{name: "wrong media type", reference: npcRegistryReferenceWithMedia([]byte(`{"npcs":[]}`), "text/plain"), wantError: "must be application/json"},
|
||||||
|
{name: "malformed JSON", reference: npcRegistryReference([]byte(`{"npcs":[`), "file:///private.json"), wantError: "decode NPC registry"},
|
||||||
|
{name: "unknown field", reference: npcRegistryReference([]byte(`{"npcs":[],"unexpected":true}`), "file:///private.json"), wantError: "unknown field"},
|
||||||
|
{name: "invalid ID", reference: npcRegistryReference(marshalNPCRegistry(t, invalidID), "file:///private.json"), wantError: "decode NPC registry"},
|
||||||
|
{name: "alias collision", reference: npcRegistryReference(encodeNPCRegistry(t, valueWithAliasCollision), "file:///private.json"), wantError: string(identity.IssueAliasOwnershipCollision)},
|
||||||
|
{name: "byte limit", reference: npcRegistryReference(bytes.Repeat([]byte("x"), NPCRegistryMaxBytes+1), "file:///private.json"), wantError: "limit"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
_, err := resolveNPCRegistry(test.reference)
|
||||||
|
if err == nil || !strings.Contains(err.Error(), test.wantError) {
|
||||||
|
t.Fatalf("resolveNPCRegistry() error = %v, want %q", err, test.wantError)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "Mira Thorn") || strings.Contains(err.Error(), "The Greencloak") || strings.Contains(err.Error(), "private.json") {
|
||||||
|
t.Fatalf("error leaked registry content or provenance: %v", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNPCRegistryFingerprintIsSemanticAndDefensive(t *testing.T) {
|
||||||
|
value := validNPCRegistryList()
|
||||||
|
canonical := encodeNPCRegistry(t, value)
|
||||||
|
pretty, err := json.MarshalIndent(value, "", " ")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("MarshalIndent() error = %v", err)
|
||||||
|
}
|
||||||
|
first := newExtractor(t, &fakeSpellsLLMClient{}, npcRegistryReference(canonical, "file:///one.json"))
|
||||||
|
second := newExtractor(t, &fakeSpellsLLMClient{}, npcRegistryReference(pretty, "file:///two.json"))
|
||||||
|
firstFingerprints := checkpointFingerprintMap(first.CheckpointFingerprints())
|
||||||
|
secondFingerprints := checkpointFingerprintMap(second.CheckpointFingerprints())
|
||||||
|
if firstFingerprints["npc_registry"] == "" || firstFingerprints["npc_registry"] != secondFingerprints["npc_registry"] {
|
||||||
|
t.Fatalf("semantic NPC fingerprints = %#v and %#v, want same npc_registry value", firstFingerprints, secondFingerprints)
|
||||||
|
}
|
||||||
|
returned := first.CheckpointFingerprints()
|
||||||
|
returned[0].Name = "caller-mutated"
|
||||||
|
if first.CheckpointFingerprints()[0].Name == "caller-mutated" {
|
||||||
|
t.Fatal("CheckpointFingerprints() returned caller-mutable slice state")
|
||||||
|
}
|
||||||
|
|
||||||
|
changed := validNPCRegistryList()
|
||||||
|
changed.NPCs[0].Description = "A different description."
|
||||||
|
changedFingerprint := checkpointFingerprintMap(newExtractor(t, &fakeSpellsLLMClient{}, npcRegistryReference(encodeNPCRegistry(t, changed), "file:///three.json")).CheckpointFingerprints())
|
||||||
|
if changedFingerprint["npc_registry"] == firstFingerprints["npc_registry"] {
|
||||||
|
t.Fatalf("semantic NPC fingerprint did not change: %#v", changedFingerprint)
|
||||||
|
}
|
||||||
|
|
||||||
|
metadata := first.ManifestMetadata()
|
||||||
|
encoded, err := json.Marshal(map[string]any{"metadata": metadata, "fingerprints": firstFingerprints})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal metadata: %v", err)
|
||||||
|
}
|
||||||
|
for _, forbidden := range []string{"Mira Thorn", "The Greencloak", "another-session", "one.json"} {
|
||||||
|
if strings.Contains(string(encoded), forbidden) {
|
||||||
|
t.Fatalf("metadata or fingerprints leaked %q: %s", forbidden, encoded)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if metadata["npc_registry_digest"] != firstFingerprints["npc_registry"] || metadata["npc_count"] != 1 {
|
||||||
|
t.Fatalf("NPC registry metadata = %#v, want digest and count only", metadata)
|
||||||
|
}
|
||||||
|
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractPassesCanonicalNPCRegistryToLLMWithoutProvenance(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
|
||||||
|
canonical := encodeNPCRegistry(t, validNPCRegistryList())
|
||||||
|
extractor := newExtractor(t, client, npcRegistryReference(append([]byte("\n"), canonical...), "file:///npc-session.json"))
|
||||||
|
if _, err := extractor.Extract(context.Background(), extractionRequest()); err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
input := client.requests[0].Inputs[NPCRegistryReferenceSlot]
|
||||||
|
if input.Name != NPCRegistryReferenceSlot || input.MediaType != "application/json" || input.Digest != semanticNPCRegistryDigest(canonical) || input.OriginURI != "" {
|
||||||
|
t.Fatalf("NPC prompt input metadata = %#v, want semantic metadata without provenance", input)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(input.Content, canonical) {
|
||||||
|
t.Fatalf("NPC prompt input = %s, want canonical JSON %s", input.Content, canonical)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validNPCRegistryList() dnd.NPCList {
|
||||||
|
return dnd.NPCList{NPCs: []dnd.NPC{{
|
||||||
|
ID: identity.DeriveID("Mira Thorn"),
|
||||||
|
Name: "Mira Thorn",
|
||||||
|
Aliases: []string{"The Greencloak"},
|
||||||
|
Description: "A guarded ranger who watches the northern road.",
|
||||||
|
Relationships: []dnd.NPCRelationship{{
|
||||||
|
Target: "Captain Vale", Relationship: "reports to",
|
||||||
|
}},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "npc-session", StartUnitID: 41, EndUnitID: 43}},
|
||||||
|
}}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func encodeNPCRegistry(t *testing.T, value dnd.NPCList) []byte {
|
||||||
|
t.Helper()
|
||||||
|
content, err := npccodec.New().Encode(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("encode NPC registry: %v", err)
|
||||||
|
}
|
||||||
|
return content
|
||||||
|
}
|
||||||
|
|
||||||
|
func marshalNPCRegistry(t *testing.T, value dnd.NPCList) []byte {
|
||||||
|
t.Helper()
|
||||||
|
content, err := json.Marshal(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("marshal NPC registry: %v", err)
|
||||||
|
}
|
||||||
|
return content
|
||||||
|
}
|
||||||
|
|
||||||
|
func npcRegistryReference(content []byte, origin string) contracts.ReferenceSet {
|
||||||
|
references := npcRegistryReferenceWithMedia(content, "application/json; charset=utf-8")
|
||||||
|
item := references.Slots[NPCRegistryReferenceSlot].Items[0]
|
||||||
|
item.Origin.URI = origin
|
||||||
|
slot := references.Slots[NPCRegistryReferenceSlot]
|
||||||
|
slot.Items[0] = item
|
||||||
|
references.Slots[NPCRegistryReferenceSlot] = slot
|
||||||
|
return references
|
||||||
|
}
|
||||||
|
|
||||||
|
func npcRegistryReferenceWithMedia(content []byte, mediaType string) contracts.ReferenceSet {
|
||||||
|
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||||
|
NPCRegistryReferenceSlot: {
|
||||||
|
Slot: contracts.ReferenceSlot{Name: NPCRegistryReferenceSlot, AcceptedMediaTypes: []string{"application/json"}, MaxBytes: NPCRegistryMaxBytes},
|
||||||
|
Items: []contracts.ReferenceItem{{
|
||||||
|
SlotName: NPCRegistryReferenceSlot,
|
||||||
|
MediaType: mediaType,
|
||||||
|
Content: append([]byte(nil), content...),
|
||||||
|
Origin: contracts.ReferenceOrigin{Type: "file", URI: "file:///npc-registry.json"},
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func checkpointFingerprintMap(values []pipeline.CheckpointFingerprint) map[string]string {
|
||||||
|
result := make(map[string]string, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
result[value.Name] = value.Value
|
||||||
|
}
|
||||||
|
return result
|
||||||
|
}
|
||||||
@@ -39,6 +39,12 @@ func TestModuleSpec(t *testing.T) {
|
|||||||
Description: "Optional campaign glossary reference material used only for disambiguation.",
|
Description: "Optional campaign glossary reference material used only for disambiguation.",
|
||||||
AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"},
|
AcceptedMediaTypes: []string{"application/json", "application/x-yaml", "application/yaml", "text/markdown", "text/plain"},
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
Name: NPCRegistryReferenceSlot,
|
||||||
|
Description: "Optional normalized NPC registry used for canonical caster-name grounding.",
|
||||||
|
AcceptedMediaTypes: []string{"application/json"},
|
||||||
|
MaxBytes: NPCRegistryMaxBytes,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
Name: "party",
|
Name: "party",
|
||||||
Description: "Optional party roster reference material used only for disambiguation.",
|
Description: "Optional party roster reference material used only for disambiguation.",
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
|||||||
promptFS, err := shared.ModulePromptFS("dnd.spells", embeddedAssets, []promptfs.ModulePromptFile{
|
promptFS, err := shared.ModulePromptFS("dnd.spells", embeddedAssets, []promptfs.ModulePromptFile{
|
||||||
{Name: "dnd.spells.yaml", Path: "assets/prompts/dnd.spells.yaml"},
|
{Name: "dnd.spells.yaml", Path: "assets/prompts/dnd.spells.yaml"},
|
||||||
{Name: "catalog.md", Path: "assets/prompts/catalog.md"},
|
{Name: "catalog.md", Path: "assets/prompts/catalog.md"},
|
||||||
|
{Name: "npc_registry.md", Path: "assets/prompts/npc_registry.md"},
|
||||||
{Name: "task.md", Path: "assets/prompts/task.md"},
|
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||||
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
||||||
})
|
})
|
||||||
@@ -32,6 +33,7 @@ func scriptoriumPromptMetadata() (string, error) {
|
|||||||
parts := append([]llm.AssetHashPart{
|
parts := append([]llm.AssetHashPart{
|
||||||
{FS: embeddedAssets, Path: "assets/prompts/dnd.spells.yaml"},
|
{FS: embeddedAssets, Path: "assets/prompts/dnd.spells.yaml"},
|
||||||
{FS: embeddedAssets, Path: "assets/prompts/catalog.md"},
|
{FS: embeddedAssets, Path: "assets/prompts/catalog.md"},
|
||||||
|
{FS: embeddedAssets, Path: "assets/prompts/npc_registry.md"},
|
||||||
{FS: embeddedAssets, Path: "assets/prompts/task.md"},
|
{FS: embeddedAssets, Path: "assets/prompts/task.md"},
|
||||||
{FS: embeddedAssets, Path: "assets/prompts/instructions.md"},
|
{FS: embeddedAssets, Path: "assets/prompts/instructions.md"},
|
||||||
}, append(shared.CommonHashParts(), shared.ReferenceHashParts()...)...)
|
}, append(shared.CommonHashParts(), shared.ReferenceHashParts()...)...)
|
||||||
|
|||||||
@@ -21,8 +21,8 @@ func TestScriptoriumPromptPreparesTranscriptReferencesAndTaskMessages(t *testing
|
|||||||
if prepared.OutputContract.SchemaPath != "dnd_spells_llm.v1.json" {
|
if prepared.OutputContract.SchemaPath != "dnd_spells_llm.v1.json" {
|
||||||
t.Fatalf("schema path = %q, want LLM-only schema", prepared.OutputContract.SchemaPath)
|
t.Fatalf("schema path = %q, want LLM-only schema", prepared.OutputContract.SchemaPath)
|
||||||
}
|
}
|
||||||
if got := len(prepared.Messages); got != 6 {
|
if got := len(prepared.Messages); got != 7 {
|
||||||
t.Fatalf("message count = %d, want 6", got)
|
t.Fatalf("message count = %d, want 7", got)
|
||||||
}
|
}
|
||||||
if !strings.Contains(prepared.Messages[1].Content, string(transcript)) {
|
if !strings.Contains(prepared.Messages[1].Content, string(transcript)) {
|
||||||
t.Fatalf("transcript message did not include source input")
|
t.Fatalf("transcript message did not include source input")
|
||||||
@@ -42,7 +42,10 @@ func TestScriptoriumPromptPreparesTranscriptReferencesAndTaskMessages(t *testing
|
|||||||
if !strings.Contains(prepared.Messages[3].Content, `{"spell_names":["Cure Wounds"]}`) {
|
if !strings.Contains(prepared.Messages[3].Content, `{"spell_names":["Cure Wounds"]}`) {
|
||||||
t.Fatalf("catalog message missing canonical spell-name input: %s", prepared.Messages[3].Content)
|
t.Fatalf("catalog message missing canonical spell-name input: %s", prepared.Messages[3].Content)
|
||||||
}
|
}
|
||||||
if strings.Contains(prepared.Messages[4].Content, string(transcript)) {
|
if !strings.Contains(prepared.Messages[4].Content, `{"npcs":[]}`) {
|
||||||
|
t.Fatalf("NPC registry message missing empty registry input: %s", prepared.Messages[4].Content)
|
||||||
|
}
|
||||||
|
if strings.Contains(prepared.Messages[5].Content, string(transcript)) {
|
||||||
t.Fatalf("task message leaked transcript bytes")
|
t.Fatalf("task message leaked transcript bytes")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -129,6 +132,7 @@ func prepareSpellsPrompt(t *testing.T, transcript []byte, players string, party
|
|||||||
Inputs: map[string]scriptorium.ArtifactRef{
|
Inputs: map[string]scriptorium.ArtifactRef{
|
||||||
"transcript": scriptorium.InlineWithURI("file:///session.json", string(transcript)),
|
"transcript": scriptorium.InlineWithURI("file:///session.json", string(transcript)),
|
||||||
"spell_catalog": scriptorium.Inline(`{"spell_names":["Cure Wounds"]}`),
|
"spell_catalog": scriptorium.Inline(`{"spell_names":["Cure Wounds"]}`),
|
||||||
|
"npcs": scriptorium.Inline(`{"npcs":[]}`),
|
||||||
"players": scriptorium.Inline(players),
|
"players": scriptorium.Inline(players),
|
||||||
"party": scriptorium.Inline(party),
|
"party": scriptorium.Inline(party),
|
||||||
"glossary": scriptorium.Inline(glossary),
|
"glossary": scriptorium.Inline(glossary),
|
||||||
|
|||||||
547
internal/modules/dnd/normalize/npcs/normalizer.go
Normal file
547
internal/modules/dnd/normalize/npcs/normalizer.go
Normal file
@@ -0,0 +1,547 @@
|
|||||||
|
// Package npcs normalizes merged D&D non-player character records.
|
||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"sort"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Key = "dnd/npcs"
|
||||||
|
NormalizationPolicy = "dnd.npcs.normalize.v1"
|
||||||
|
|
||||||
|
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
|
||||||
|
ReasonCodeNPCIDRecomputed = "npc_id_recomputed"
|
||||||
|
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
|
||||||
|
ReasonCodeDuplicateNPCCollapsed = "duplicate_npc_collapsed"
|
||||||
|
ReasonCodeRelationshipTargetCanonicalized = "relationship_target_canonicalized"
|
||||||
|
)
|
||||||
|
|
||||||
|
var requiredCapabilities = []string{"merged"}
|
||||||
|
var providedCapabilities = []string{"normalized"}
|
||||||
|
|
||||||
|
var _ contracts.Normalizer[dnd.NPCList] = (*Normalizer)(nil)
|
||||||
|
var _ contracts.ManifestMetadataProvider = (*Normalizer)(nil)
|
||||||
|
var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
|
||||||
|
|
||||||
|
type Options struct{}
|
||||||
|
|
||||||
|
type Normalizer struct{}
|
||||||
|
|
||||||
|
func New(Options) *Normalizer { return &Normalizer{} }
|
||||||
|
|
||||||
|
func (n *Normalizer) Key() string { return Key }
|
||||||
|
|
||||||
|
func (n *Normalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||||
|
|
||||||
|
func (n *Normalizer) ManifestMetadata() map[string]any {
|
||||||
|
if n == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return map[string]any{
|
||||||
|
"identity_policy": identity.Policy,
|
||||||
|
"normalization_policy": NormalizationPolicy,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||||
|
if n == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return []pipeline.CheckpointFingerprint{
|
||||||
|
{Name: "identity_policy", Value: identity.Policy},
|
||||||
|
{Name: "normalization_policy", Value: NormalizationPolicy},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[dnd.NPCList]) (contracts.TypedNormalizeResult[dnd.NPCList], error) {
|
||||||
|
if n == nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("normalizer must not be nil")
|
||||||
|
}
|
||||||
|
if ctx == nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context must not be nil")
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context error before normalize: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
value, warnings := normalizeList(req.MergeOutput.Value)
|
||||||
|
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: value, Warnings: warnings}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type normalizedRecord struct {
|
||||||
|
npc dnd.NPC
|
||||||
|
fieldsChanged bool
|
||||||
|
referencesChanged bool
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeList(input dnd.NPCList) (dnd.NPCList, []contracts.Warning) {
|
||||||
|
if input.NPCs == nil {
|
||||||
|
return dnd.NPCList{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
records := make([]normalizedRecord, len(input.NPCs))
|
||||||
|
warnings := make([]contracts.Warning, 0)
|
||||||
|
for index, inputNPC := range input.NPCs {
|
||||||
|
npc, fieldsChanged, referencesChanged := normalizeRecord(inputNPC)
|
||||||
|
records[index] = normalizedRecord{
|
||||||
|
npc: npc,
|
||||||
|
fieldsChanged: fieldsChanged,
|
||||||
|
referencesChanged: referencesChanged,
|
||||||
|
}
|
||||||
|
if fieldsChanged {
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: npcScope(index),
|
||||||
|
ReasonCode: ReasonCodeNPCFieldsNormalized,
|
||||||
|
Message: fmt.Sprintf("input index %d: NPC fields normalized for %s",
|
||||||
|
index, diagnostics.Quote(inputNPC.Name)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if referencesChanged {
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: npcScope(index),
|
||||||
|
ReasonCode: ReasonCodeSourceReferencesNormalized,
|
||||||
|
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)",
|
||||||
|
index, len(inputNPC.SourceRefs), len(npc.SourceRefs)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if inputNPC.ID != npc.ID {
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: npcScope(index),
|
||||||
|
ReasonCode: ReasonCodeNPCIDRecomputed,
|
||||||
|
Message: fmt.Sprintf("input index %d: NPC ID recomputed from %s",
|
||||||
|
index, diagnostics.Quote(npc.Name)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
components := identityComponents(records)
|
||||||
|
output := dnd.NPCList{NPCs: make([]dnd.NPC, 0, len(components))}
|
||||||
|
retainedIndexes := make([]int, 0, len(components))
|
||||||
|
for _, members := range components {
|
||||||
|
consolidated, sourceChanged := consolidate(records, members)
|
||||||
|
retainedIndex := members[0]
|
||||||
|
output.NPCs = append(output.NPCs, consolidated)
|
||||||
|
retainedIndexes = append(retainedIndexes, retainedIndex)
|
||||||
|
|
||||||
|
if sourceChanged {
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: npcScope(retainedIndex),
|
||||||
|
ReasonCode: ReasonCodeSourceReferencesNormalized,
|
||||||
|
Message: fmt.Sprintf("input index %d: source references normalized during identity consolidation (final count %d)",
|
||||||
|
retainedIndex, len(consolidated.SourceRefs)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if len(members) > 1 {
|
||||||
|
warnings = append(warnings, duplicateWarning(retainedIndex, members[1:]))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
warnings = append(warnings, canonicalizeRelationshipTargets(&output, retainedIndexes)...)
|
||||||
|
return output, warnings
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeRecord(input dnd.NPC) (dnd.NPC, bool, bool) {
|
||||||
|
output := cloneNPC(input)
|
||||||
|
output.Name = identity.NormalizeDisplay(input.Name)
|
||||||
|
output.Aliases = normalizeAliases(input.Aliases, output.Name)
|
||||||
|
output.Description = strings.TrimSpace(input.Description)
|
||||||
|
output.Relationships = normalizeRelationships(input.Relationships)
|
||||||
|
output.SourceRefs, _, _ = canonicalizeSourceRefs(input.SourceRefs)
|
||||||
|
output.ID = identity.DeriveID(output.Name)
|
||||||
|
|
||||||
|
fieldsChanged := input.Name != output.Name ||
|
||||||
|
!reflect.DeepEqual(input.Aliases, output.Aliases) ||
|
||||||
|
input.Description != output.Description ||
|
||||||
|
!reflect.DeepEqual(input.Relationships, output.Relationships)
|
||||||
|
referencesChanged := !reflect.DeepEqual(input.SourceRefs, output.SourceRefs)
|
||||||
|
return output, fieldsChanged, referencesChanged
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneNPC(input dnd.NPC) dnd.NPC {
|
||||||
|
output := input
|
||||||
|
if input.Aliases != nil {
|
||||||
|
output.Aliases = make([]string, len(input.Aliases))
|
||||||
|
copy(output.Aliases, input.Aliases)
|
||||||
|
}
|
||||||
|
if input.Relationships != nil {
|
||||||
|
output.Relationships = make([]dnd.NPCRelationship, len(input.Relationships))
|
||||||
|
copy(output.Relationships, input.Relationships)
|
||||||
|
}
|
||||||
|
if input.SourceRefs != nil {
|
||||||
|
output.SourceRefs = make([]source.SourceRef, len(input.SourceRefs))
|
||||||
|
copy(output.SourceRefs, input.SourceRefs)
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeAliases(input []string, canonicalName string) []string {
|
||||||
|
if input == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
canonicalKey := identity.ComparisonKey(canonicalName)
|
||||||
|
output := make([]string, 0, len(input))
|
||||||
|
seen := make(map[string]struct{}, len(input))
|
||||||
|
for _, alias := range input {
|
||||||
|
normalized := identity.NormalizeDisplay(alias)
|
||||||
|
key := identity.ComparisonKey(normalized)
|
||||||
|
if canonicalKey != "" && key == canonicalKey {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, exists := seen[key]; exists {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
output = append(output, normalized)
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeRelationships(input []dnd.NPCRelationship) []dnd.NPCRelationship {
|
||||||
|
if input == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
output := make([]dnd.NPCRelationship, 0, len(input))
|
||||||
|
seen := make(map[relationshipIdentity]struct{}, len(input))
|
||||||
|
for _, relationship := range input {
|
||||||
|
normalized := dnd.NPCRelationship{
|
||||||
|
Target: identity.NormalizeDisplay(relationship.Target),
|
||||||
|
Relationship: strings.TrimSpace(relationship.Relationship),
|
||||||
|
}
|
||||||
|
key := relationshipKey(normalized)
|
||||||
|
if _, exists := seen[key]; exists {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
output = append(output, normalized)
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
type relationshipIdentity struct {
|
||||||
|
target string
|
||||||
|
relationship string
|
||||||
|
}
|
||||||
|
|
||||||
|
func relationshipKey(relationship dnd.NPCRelationship) relationshipIdentity {
|
||||||
|
return relationshipIdentity{
|
||||||
|
target: identity.ComparisonKey(relationship.Target),
|
||||||
|
relationship: identity.ComparisonKey(relationship.Relationship),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalizeSourceRefs(input []source.SourceRef) ([]source.SourceRef, bool, int) {
|
||||||
|
if input == nil {
|
||||||
|
return nil, false, 0
|
||||||
|
}
|
||||||
|
|
||||||
|
canonical := make([]source.SourceRef, len(input))
|
||||||
|
copy(canonical, input)
|
||||||
|
sort.SliceStable(canonical, func(left, right int) bool {
|
||||||
|
return sourceRefLess(canonical[left], canonical[right])
|
||||||
|
})
|
||||||
|
|
||||||
|
orderChanged := false
|
||||||
|
for index := range input {
|
||||||
|
if input[index] != canonical[index] {
|
||||||
|
orderChanged = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
unique := make([]source.SourceRef, 0, len(canonical))
|
||||||
|
for _, ref := range canonical {
|
||||||
|
if len(unique) == 0 || unique[len(unique)-1] != ref {
|
||||||
|
unique = append(unique, ref)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return unique, orderChanged, len(input) - len(unique)
|
||||||
|
}
|
||||||
|
|
||||||
|
func sourceRefLess(left, right source.SourceRef) bool {
|
||||||
|
if left.SourceID != right.SourceID {
|
||||||
|
return left.SourceID < right.SourceID
|
||||||
|
}
|
||||||
|
if left.StartUnitID != right.StartUnitID {
|
||||||
|
return left.StartUnitID < right.StartUnitID
|
||||||
|
}
|
||||||
|
return left.EndUnitID < right.EndUnitID
|
||||||
|
}
|
||||||
|
|
||||||
|
func identityComponents(records []normalizedRecord) [][]int {
|
||||||
|
parent := make([]int, len(records))
|
||||||
|
for index := range parent {
|
||||||
|
parent[index] = index
|
||||||
|
}
|
||||||
|
|
||||||
|
for left := 0; left < len(records); left++ {
|
||||||
|
for right := left + 1; right < len(records); right++ {
|
||||||
|
if recordsCanMerge(records[left].npc, records[right].npc) {
|
||||||
|
union(parent, left, right)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
byRoot := make(map[int][]int, len(records))
|
||||||
|
for index := range records {
|
||||||
|
root := find(parent, index)
|
||||||
|
byRoot[root] = append(byRoot[root], index)
|
||||||
|
}
|
||||||
|
roots := make([]int, 0, len(byRoot))
|
||||||
|
for root := range byRoot {
|
||||||
|
roots = append(roots, root)
|
||||||
|
}
|
||||||
|
sort.Slice(roots, func(left, right int) bool {
|
||||||
|
return byRoot[roots[left]][0] < byRoot[roots[right]][0]
|
||||||
|
})
|
||||||
|
|
||||||
|
components := make([][]int, 0, len(roots))
|
||||||
|
for _, root := range roots {
|
||||||
|
components = append(components, byRoot[root])
|
||||||
|
}
|
||||||
|
return components
|
||||||
|
}
|
||||||
|
|
||||||
|
func recordsCanMerge(left, right dnd.NPC) bool {
|
||||||
|
leftCanonical := identity.ComparisonKey(left.Name)
|
||||||
|
rightCanonical := identity.ComparisonKey(right.Name)
|
||||||
|
return leftCanonical == rightCanonical ||
|
||||||
|
containsAliasKey(right.Aliases, leftCanonical) ||
|
||||||
|
containsAliasKey(left.Aliases, rightCanonical)
|
||||||
|
}
|
||||||
|
|
||||||
|
func containsAliasKey(aliases []string, wanted string) bool {
|
||||||
|
for _, alias := range aliases {
|
||||||
|
if identity.ComparisonKey(alias) == wanted {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func find(parent []int, index int) int {
|
||||||
|
for parent[index] != index {
|
||||||
|
parent[index] = parent[parent[index]]
|
||||||
|
index = parent[index]
|
||||||
|
}
|
||||||
|
return index
|
||||||
|
}
|
||||||
|
|
||||||
|
func union(parent []int, left, right int) {
|
||||||
|
leftRoot := find(parent, left)
|
||||||
|
rightRoot := find(parent, right)
|
||||||
|
if leftRoot == rightRoot {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if leftRoot < rightRoot {
|
||||||
|
parent[rightRoot] = leftRoot
|
||||||
|
} else {
|
||||||
|
parent[leftRoot] = rightRoot
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func consolidate(records []normalizedRecord, members []int) (dnd.NPC, bool) {
|
||||||
|
output := cloneNPC(records[members[0]].npc)
|
||||||
|
originalRefs := cloneSourceRefs(output.SourceRefs)
|
||||||
|
canonicalKey := identity.ComparisonKey(output.Name)
|
||||||
|
|
||||||
|
for _, member := range members[1:] {
|
||||||
|
candidate := records[member].npc
|
||||||
|
appendAlias(&output.Aliases, candidate.Name, canonicalKey)
|
||||||
|
for _, alias := range candidate.Aliases {
|
||||||
|
appendAlias(&output.Aliases, alias, canonicalKey)
|
||||||
|
}
|
||||||
|
for _, relationship := range candidate.Relationships {
|
||||||
|
appendRelationship(&output.Relationships, relationship)
|
||||||
|
}
|
||||||
|
output.SourceRefs = append(output.SourceRefs, candidate.SourceRefs...)
|
||||||
|
}
|
||||||
|
|
||||||
|
output.SourceRefs, _, _ = canonicalizeSourceRefs(output.SourceRefs)
|
||||||
|
output.ID = identity.DeriveID(output.Name)
|
||||||
|
return output, !reflect.DeepEqual(originalRefs, output.SourceRefs)
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneSourceRefs(input []source.SourceRef) []source.SourceRef {
|
||||||
|
if input == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
output := make([]source.SourceRef, len(input))
|
||||||
|
copy(output, input)
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func appendAlias(aliases *[]string, value, canonicalKey string) {
|
||||||
|
key := identity.ComparisonKey(value)
|
||||||
|
if canonicalKey != "" && key == canonicalKey {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
for _, existing := range *aliases {
|
||||||
|
if identity.ComparisonKey(existing) == key {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
*aliases = append(*aliases, value)
|
||||||
|
}
|
||||||
|
|
||||||
|
func appendRelationship(relationships *[]dnd.NPCRelationship, relationship dnd.NPCRelationship) {
|
||||||
|
key := relationshipKey(relationship)
|
||||||
|
for _, existing := range *relationships {
|
||||||
|
if relationshipKey(existing) == key {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
*relationships = append(*relationships, relationship)
|
||||||
|
}
|
||||||
|
|
||||||
|
func canonicalizeRelationshipTargets(list *dnd.NPCList, retainedIndexes []int) []contracts.Warning {
|
||||||
|
if list == nil || len(list.NPCs) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
owners := make(map[string][]int)
|
||||||
|
for index, npc := range list.NPCs {
|
||||||
|
if key := identity.ComparisonKey(npc.Name); key != "" {
|
||||||
|
owners[key] = append(owners[key], index)
|
||||||
|
}
|
||||||
|
for _, alias := range npc.Aliases {
|
||||||
|
if key := identity.ComparisonKey(alias); key != "" {
|
||||||
|
owners[key] = appendUniqueIndex(owners[key], index)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
warnings := make([]contracts.Warning, 0)
|
||||||
|
for outputIndex := range list.NPCs {
|
||||||
|
npc := &list.NPCs[outputIndex]
|
||||||
|
originalRelationshipCount := len(npc.Relationships)
|
||||||
|
for relationshipIndex := range npc.Relationships {
|
||||||
|
relationship := &npc.Relationships[relationshipIndex]
|
||||||
|
key := identity.ComparisonKey(relationship.Target)
|
||||||
|
if key == "" || len(owners[key]) != 1 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
target := list.NPCs[owners[key][0]].Name
|
||||||
|
if relationship.Target == target {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
oldTarget := relationship.Target
|
||||||
|
relationship.Target = target
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: npcScope(retainedIndexes[outputIndex]),
|
||||||
|
ReasonCode: ReasonCodeRelationshipTargetCanonicalized,
|
||||||
|
Message: fmt.Sprintf("input index %d: relationship target canonicalized from %s to %s",
|
||||||
|
retainedIndexes[outputIndex], diagnostics.Quote(oldTarget), diagnostics.Quote(target)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
npc.Relationships = deduplicateRelationships(npc.Relationships)
|
||||||
|
if len(npc.Relationships) != originalRelationshipCount {
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: npcScope(retainedIndexes[outputIndex]),
|
||||||
|
ReasonCode: ReasonCodeNPCFieldsNormalized,
|
||||||
|
Message: fmt.Sprintf("input index %d: duplicate relationships removed after target canonicalization",
|
||||||
|
retainedIndexes[outputIndex]),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return warnings
|
||||||
|
}
|
||||||
|
|
||||||
|
func deduplicateRelationships(input []dnd.NPCRelationship) []dnd.NPCRelationship {
|
||||||
|
if input == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
output := make([]dnd.NPCRelationship, 0, len(input))
|
||||||
|
seen := make(map[relationshipIdentity]struct{}, len(input))
|
||||||
|
for _, relationship := range input {
|
||||||
|
key := relationshipKey(relationship)
|
||||||
|
if _, exists := seen[key]; exists {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[key] = struct{}{}
|
||||||
|
output = append(output, relationship)
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func appendUniqueIndex(values []int, wanted int) []int {
|
||||||
|
for _, value := range values {
|
||||||
|
if value == wanted {
|
||||||
|
return values
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return append(values, wanted)
|
||||||
|
}
|
||||||
|
|
||||||
|
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
||||||
|
const maxDisplayedIndices = 20
|
||||||
|
displayed := removed
|
||||||
|
if len(displayed) > maxDisplayedIndices {
|
||||||
|
displayed = displayed[:maxDisplayedIndices]
|
||||||
|
}
|
||||||
|
indices := make([]string, len(displayed))
|
||||||
|
for index, removedIndex := range displayed {
|
||||||
|
indices[index] = strconv.Itoa(removedIndex)
|
||||||
|
}
|
||||||
|
|
||||||
|
message := fmt.Sprintf("retained input index %d; removed input indices [%s]", retainedIndex, strings.Join(indices, ", "))
|
||||||
|
if omitted := len(removed) - len(displayed); omitted > 0 {
|
||||||
|
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
|
||||||
|
}
|
||||||
|
return contracts.Warning{
|
||||||
|
Scope: npcScope(retainedIndex),
|
||||||
|
ReasonCode: ReasonCodeDuplicateNPCCollapsed,
|
||||||
|
Message: message,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func npcScope(index int) string { return fmt.Sprintf("npcs[%d]", index) }
|
||||||
|
|
||||||
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
|
return pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageNormalize,
|
||||||
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
|
ArtifactKind: dnd.NPCListKind,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.NormalizerRegistry) error {
|
||||||
|
return pipeline.RegisterNormalizerBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Normalizer[dnd.NPCList], error) {
|
||||||
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(options), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOptions(options map[string]any) error {
|
||||||
|
_, err := DecodeOptions(options)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, normalizerErrorf("%w", err)
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizerErrorf(format string, args ...any) error {
|
||||||
|
return fmt.Errorf("dnd npcs normalizer: "+format, args...)
|
||||||
|
}
|
||||||
292
internal/modules/dnd/normalize/npcs/normalizer_test.go
Normal file
292
internal/modules/dnd/normalize/npcs/normalizer_test.go
Normal file
@@ -0,0 +1,292 @@
|
|||||||
|
package npcs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
domainidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestModuleContractAndIdentity(t *testing.T) {
|
||||||
|
if _, err := DecodeOptions(nil); err != nil {
|
||||||
|
t.Fatalf("DecodeOptions(nil) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageNormalize,
|
||||||
|
Requires: []string{"merged"},
|
||||||
|
Provides: []string{"normalized"},
|
||||||
|
ArtifactKind: dnd.NPCListKind,
|
||||||
|
}
|
||||||
|
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
if slots := New(Options{}).ReferenceSlots(); slots != nil {
|
||||||
|
t.Fatalf("ReferenceSlots() = %#v, want nil", slots)
|
||||||
|
}
|
||||||
|
|
||||||
|
registry := pipeline.NewNormalizerRegistry()
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if registered, ok := registry.SpecForArtifact(Key, dnd.NPCListKind); !ok || !reflect.DeepEqual(registered, want) {
|
||||||
|
t.Fatalf("registered spec = %#v, ok = %t, want %#v", registered, ok, want)
|
||||||
|
}
|
||||||
|
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "normalizer registry") {
|
||||||
|
t.Fatalf("Register(nil) error = %v, want registry error", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
normalizer := New(Options{})
|
||||||
|
metadata := normalizer.ManifestMetadata()
|
||||||
|
if metadata["identity_policy"] != domainidentity.Policy || metadata["normalization_policy"] != NormalizationPolicy {
|
||||||
|
t.Fatalf("metadata = %#v, want identity and normalization policies", metadata)
|
||||||
|
}
|
||||||
|
fingerprints := normalizer.CheckpointFingerprints()
|
||||||
|
wantFingerprints := []pipeline.CheckpointFingerprint{
|
||||||
|
{Name: "identity_policy", Value: domainidentity.Policy},
|
||||||
|
{Name: "normalization_policy", Value: NormalizationPolicy},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(fingerprints, wantFingerprints) {
|
||||||
|
t.Fatalf("fingerprints = %#v, want %#v", fingerprints, wantFingerprints)
|
||||||
|
}
|
||||||
|
fingerprints[0].Value = "changed"
|
||||||
|
if got := normalizer.CheckpointFingerprints(); !reflect.DeepEqual(got, wantFingerprints) {
|
||||||
|
t.Fatalf("fingerprints were not defensive: %#v", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizePerRecordFieldsAndEvidence(t *testing.T) {
|
||||||
|
input := dnd.NPCList{NPCs: []dnd.NPC{{
|
||||||
|
ID: "wrong",
|
||||||
|
Name: " Lady\tAsh ",
|
||||||
|
Aliases: []string{" Ash ", " L.A. ", "l.a.", " Lady Ash "},
|
||||||
|
Description: " first description\n",
|
||||||
|
Relationships: []dnd.NPCRelationship{
|
||||||
|
{Target: " Lord\nOak ", Relationship: " friend "},
|
||||||
|
{Target: "lord oak", Relationship: "friend"},
|
||||||
|
},
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
|
||||||
|
{SourceID: "a", StartUnitID: 4, EndUnitID: 4},
|
||||||
|
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
|
||||||
|
},
|
||||||
|
}}}
|
||||||
|
|
||||||
|
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
got := result.Value.NPCs[0]
|
||||||
|
if got.Name != "Lady Ash" || got.Description != "first description" {
|
||||||
|
t.Fatalf("normalized fields = %#v, want normalized display and description", got)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got.Aliases, []string{"Ash", "L.A."}) {
|
||||||
|
t.Fatalf("aliases = %#v, want canonical alias removal and deduplication", got.Aliases)
|
||||||
|
}
|
||||||
|
wantRelationships := []dnd.NPCRelationship{{Target: "Lord Oak", Relationship: "friend"}}
|
||||||
|
if !reflect.DeepEqual(got.Relationships, wantRelationships) {
|
||||||
|
t.Fatalf("relationships = %#v, want %#v", got.Relationships, wantRelationships)
|
||||||
|
}
|
||||||
|
wantRefs := []source.SourceRef{
|
||||||
|
{SourceID: "a", StartUnitID: 4, EndUnitID: 4},
|
||||||
|
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got.SourceRefs, wantRefs) {
|
||||||
|
t.Fatalf("source refs = %#v, want %#v", got.SourceRefs, wantRefs)
|
||||||
|
}
|
||||||
|
if got.ID != domainidentity.DeriveID("Lady Ash") {
|
||||||
|
t.Fatalf("ID = %q, want derived ID", got.ID)
|
||||||
|
}
|
||||||
|
if !hasWarning(result.Warnings, ReasonCodeNPCFieldsNormalized, "npcs[0]") ||
|
||||||
|
!hasWarning(result.Warnings, ReasonCodeSourceReferencesNormalized, "npcs[0]") ||
|
||||||
|
!hasWarning(result.Warnings, ReasonCodeNPCIDRecomputed, "npcs[0]") {
|
||||||
|
t.Fatalf("warnings = %#v, want field, source, and ID warnings", result.Warnings)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeConsolidatesIdentityComponentsInStableOrder(t *testing.T) {
|
||||||
|
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||||
|
{
|
||||||
|
Name: " Captain Vale ",
|
||||||
|
Description: "first description",
|
||||||
|
Aliases: []string{"Vale"},
|
||||||
|
Relationships: []dnd.NPCRelationship{{Target: "Archivist", Relationship: "knows"}},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "a", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "Captain Vale",
|
||||||
|
Description: "second description",
|
||||||
|
Aliases: []string{"CV"},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "b", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "The Sage",
|
||||||
|
Description: "sage description",
|
||||||
|
Aliases: []string{"Archivist"},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "c", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "Archivist",
|
||||||
|
Description: "later sage description",
|
||||||
|
Aliases: []string{"Chronicler"},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "d", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "North",
|
||||||
|
Description: "north description",
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "e", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "North Old",
|
||||||
|
Description: "old north description",
|
||||||
|
Aliases: []string{"North"},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "f", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "North Renamed",
|
||||||
|
Description: "renamed north description",
|
||||||
|
Aliases: []string{"North Old"},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "g", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{Name: "Red", Description: "red", Aliases: []string{"Shared"}, SourceRefs: []source.SourceRef{{SourceID: "h", StartUnitID: 1, EndUnitID: 1}}},
|
||||||
|
{Name: "Blue", Description: "blue", Aliases: []string{"Shared"}, SourceRefs: []source.SourceRef{{SourceID: "i", StartUnitID: 1, EndUnitID: 1}}},
|
||||||
|
}}
|
||||||
|
|
||||||
|
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if got := len(result.Value.NPCs); got != 5 {
|
||||||
|
t.Fatalf("normalized NPC count = %d, want five components", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
first := result.Value.NPCs[0]
|
||||||
|
if first.Name != "Captain Vale" || first.Description != "first description" || !reflect.DeepEqual(first.Aliases, []string{"Vale", "CV"}) {
|
||||||
|
t.Fatalf("first component = %#v, want first description and ordered aliases", first)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(first.SourceRefs, []source.SourceRef{
|
||||||
|
{SourceID: "a", StartUnitID: 1, EndUnitID: 1},
|
||||||
|
{SourceID: "b", StartUnitID: 1, EndUnitID: 1},
|
||||||
|
}) {
|
||||||
|
t.Fatalf("first provenance = %#v, want unioned refs", first.SourceRefs)
|
||||||
|
}
|
||||||
|
|
||||||
|
second := result.Value.NPCs[1]
|
||||||
|
if second.Name != "The Sage" || !reflect.DeepEqual(second.Aliases, []string{"Archivist", "Chronicler"}) || second.Description != "sage description" {
|
||||||
|
t.Fatalf("canonical-to-alias component = %#v, want consolidated sage", second)
|
||||||
|
}
|
||||||
|
if first.Relationships[0].Target != "The Sage" {
|
||||||
|
t.Fatalf("relationship target = %q, want The Sage", first.Relationships[0].Target)
|
||||||
|
}
|
||||||
|
if !hasWarning(result.Warnings, ReasonCodeRelationshipTargetCanonicalized, "npcs[0]") ||
|
||||||
|
!hasWarning(result.Warnings, ReasonCodeDuplicateNPCCollapsed, "npcs[0]") ||
|
||||||
|
!hasWarning(result.Warnings, ReasonCodeDuplicateNPCCollapsed, "npcs[2]") {
|
||||||
|
t.Fatalf("warnings = %#v, want collapse and target warnings", result.Warnings)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := result.Value.NPCs[2].Aliases; !reflect.DeepEqual(got, []string{"North Old", "North Renamed"}) {
|
||||||
|
t.Fatalf("transitive aliases = %#v, want ordered canonical members", got)
|
||||||
|
}
|
||||||
|
if result.Value.NPCs[3].Name != "Red" || result.Value.NPCs[4].Name != "Blue" {
|
||||||
|
t.Fatalf("shared-alias ordering = %#v, want Red then Blue", result.Value.NPCs[3:])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizePreservesInvalidEvidenceAndDoesNotAliasInput(t *testing.T) {
|
||||||
|
invalid := dnd.NPCList{NPCs: []dnd.NPC{{
|
||||||
|
Name: " ",
|
||||||
|
Aliases: []string{""},
|
||||||
|
Description: " ",
|
||||||
|
Relationships: []dnd.NPCRelationship{{Target: " ", Relationship: " "}},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "source", StartUnitID: 9, EndUnitID: 9}},
|
||||||
|
}}}
|
||||||
|
original := cloneNPCListForTest(invalid)
|
||||||
|
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(invalid))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(invalid, original) {
|
||||||
|
t.Fatalf("normalizer mutated input: got %#v, want %#v", invalid, original)
|
||||||
|
}
|
||||||
|
if result.Value.NPCs[0].Name != "" || result.Value.NPCs[0].Aliases[0] != "" || result.Value.NPCs[0].Relationships[0].Target != "" {
|
||||||
|
t.Fatalf("invalid evidence was unexpectedly removed: %#v", result.Value.NPCs[0])
|
||||||
|
}
|
||||||
|
|
||||||
|
result.Value.NPCs[0].Aliases[0] = "changed"
|
||||||
|
result.Value.NPCs[0].Relationships[0].Target = "changed"
|
||||||
|
result.Value.NPCs[0].SourceRefs[0].SourceID = "changed"
|
||||||
|
if invalid.NPCs[0].Aliases[0] != "" || invalid.NPCs[0].Relationships[0].Target != " " || invalid.NPCs[0].SourceRefs[0].SourceID != "source" {
|
||||||
|
t.Fatalf("output aliases input storage: input = %#v", invalid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeHandlesNilAndCanceledCalls(t *testing.T) {
|
||||||
|
normalizer := New(Options{})
|
||||||
|
result, err := normalizer.Normalize(context.Background(), normalizeRequest(dnd.NPCList{NPCs: nil}))
|
||||||
|
if err != nil || result.Value.NPCs != nil {
|
||||||
|
t.Fatalf("nil list result = %#v, error = %v, want nil NPC slice", result.Value, err)
|
||||||
|
}
|
||||||
|
canceled, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
if _, err := normalizer.Normalize(canceled, normalizeRequest(dnd.NPCList{})); err == nil || !strings.Contains(err.Error(), "context error") {
|
||||||
|
t.Fatalf("canceled Normalize() error = %v, want context error", err)
|
||||||
|
}
|
||||||
|
if _, err := normalizer.Normalize(nil, normalizeRequest(dnd.NPCList{})); err == nil || !strings.Contains(err.Error(), "context must not be nil") {
|
||||||
|
t.Fatalf("nil context Normalize() error = %v, want context error", err)
|
||||||
|
}
|
||||||
|
var nilNormalizer *Normalizer
|
||||||
|
if _, err := nilNormalizer.Normalize(context.Background(), normalizeRequest(dnd.NPCList{})); err == nil {
|
||||||
|
t.Fatal("nil normalizer Normalize() error = nil, want error")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizePreservesPresentEmptyAliases(t *testing.T) {
|
||||||
|
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(dnd.NPCList{NPCs: []dnd.NPC{{
|
||||||
|
Name: "Hooded Guard",
|
||||||
|
Aliases: []string{},
|
||||||
|
Description: "A distinguishable sentry.",
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
}}}))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Normalize() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if result.Value.NPCs[0].Aliases == nil || len(result.Value.NPCs[0].Aliases) != 0 {
|
||||||
|
t.Fatalf("aliases = %#v, want present empty array", result.Value.NPCs[0].Aliases)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func normalizeRequest(value dnd.NPCList) contracts.TypedNormalizeRequest[dnd.NPCList] {
|
||||||
|
return contracts.TypedNormalizeRequest[dnd.NPCList]{
|
||||||
|
MergeOutput: contracts.MergeArtifact[dnd.NPCList]{Value: value},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func cloneNPCListForTest(input dnd.NPCList) dnd.NPCList {
|
||||||
|
output := dnd.NPCList{}
|
||||||
|
if input.NPCs != nil {
|
||||||
|
output.NPCs = make([]dnd.NPC, len(input.NPCs))
|
||||||
|
for index, npc := range input.NPCs {
|
||||||
|
output.NPCs[index] = cloneNPC(npc)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return output
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasWarning(warnings []contracts.Warning, reason, scope string) bool {
|
||||||
|
for _, warning := range warnings {
|
||||||
|
if warning.ReasonCode == reason && warning.Scope == scope {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
230
internal/modules/dnd/npcs/identity/identity.go
Normal file
230
internal/modules/dnd/npcs/identity/identity.go
Normal file
@@ -0,0 +1,230 @@
|
|||||||
|
// Package identity owns the stable identity policy for D&D non-player
|
||||||
|
// characters.
|
||||||
|
package identity
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"golang.org/x/text/cases"
|
||||||
|
"golang.org/x/text/unicode/norm"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
// Policy identifies the complete identity comparison and ID derivation
|
||||||
|
// policy. A future semantic change must use a new value.
|
||||||
|
Policy = "dnd.npcs.identity.v1"
|
||||||
|
// IdentityPolicy is an explicit alias for callers recording policy
|
||||||
|
// fingerprints.
|
||||||
|
IdentityPolicy = Policy
|
||||||
|
)
|
||||||
|
|
||||||
|
const idPrefix = "npc:sha256:"
|
||||||
|
|
||||||
|
// IssueCode identifies one deterministic registry identity problem.
|
||||||
|
type IssueCode string
|
||||||
|
|
||||||
|
const (
|
||||||
|
IssueEmptyCanonicalName IssueCode = "empty_canonical_name"
|
||||||
|
IssueEmptyAlias IssueCode = "empty_alias"
|
||||||
|
IssueInvalidID IssueCode = "invalid_id"
|
||||||
|
IssueIDMismatch IssueCode = "id_mismatch"
|
||||||
|
IssueDuplicateCanonical IssueCode = "duplicate_canonical_identity"
|
||||||
|
IssueDuplicateID IssueCode = "duplicate_id"
|
||||||
|
IssueDuplicateAlias IssueCode = "duplicate_alias"
|
||||||
|
IssueOwnCanonicalAlias IssueCode = "alias_matches_canonical_name"
|
||||||
|
IssueAliasCanonicalCollision IssueCode = "alias_canonical_collision"
|
||||||
|
IssueAliasOwnershipCollision IssueCode = "alias_owned_by_multiple_records"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Issue is an inspectable identity validation problem. AliasIndex is -1 when
|
||||||
|
// the issue applies to an NPC as a whole rather than a particular alias.
|
||||||
|
type Issue struct {
|
||||||
|
Code IssueCode
|
||||||
|
RecordIndex int
|
||||||
|
AliasIndex int
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
// NormalizeDisplay trims and collapses Unicode whitespace while retaining all
|
||||||
|
// other observed spelling and punctuation.
|
||||||
|
func NormalizeDisplay(value string) string {
|
||||||
|
return strings.Join(strings.Fields(value), " ")
|
||||||
|
}
|
||||||
|
|
||||||
|
// ComparisonKey returns the stable key used for NPC identity comparisons.
|
||||||
|
func ComparisonKey(value string) string {
|
||||||
|
value = norm.NFKC.String(value)
|
||||||
|
value = strings.Map(func(r rune) rune {
|
||||||
|
switch r {
|
||||||
|
case '\u2018', '\u2019', '\u02bc':
|
||||||
|
return '\''
|
||||||
|
default:
|
||||||
|
return r
|
||||||
|
}
|
||||||
|
}, value)
|
||||||
|
value = strings.Join(strings.Fields(value), " ")
|
||||||
|
return cases.Fold().String(value)
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeriveID returns the deterministic ID for a canonical NPC name. Empty
|
||||||
|
// identity keys intentionally produce an empty ID so shape validation can
|
||||||
|
// report the missing identity instead of manufacturing one.
|
||||||
|
func DeriveID(name string) string {
|
||||||
|
key := ComparisonKey(name)
|
||||||
|
if key == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
digest := sha256.Sum256([]byte(key))
|
||||||
|
return idPrefix + hex.EncodeToString(digest[:])
|
||||||
|
}
|
||||||
|
|
||||||
|
// IDFor is a concise alias for DeriveID for callers that work with IDs as
|
||||||
|
// values rather than derivation operations.
|
||||||
|
func IDFor(name string) string { return DeriveID(name) }
|
||||||
|
|
||||||
|
// IsValidID reports whether value has the exact durable NPC ID syntax.
|
||||||
|
func IsValidID(value string) bool {
|
||||||
|
if len(value) != len(idPrefix)+sha256.Size*2 || !strings.HasPrefix(value, idPrefix) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for _, r := range value[len(idPrefix):] {
|
||||||
|
if !(r >= '0' && r <= '9') && !(r >= 'a' && r <= 'f') {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidID is an alias for IsValidID.
|
||||||
|
func ValidID(value string) bool { return IsValidID(value) }
|
||||||
|
|
||||||
|
// ValidateRegistry checks all identity invariants without changing the input.
|
||||||
|
// It accepts the NPC slice used by typed pipeline artifacts. Use ValidateList
|
||||||
|
// when the enclosing NPCList is more convenient at the call site.
|
||||||
|
func ValidateRegistry(npcs []dnd.NPC) []Issue {
|
||||||
|
type record struct {
|
||||||
|
canonical string
|
||||||
|
aliases []string
|
||||||
|
}
|
||||||
|
|
||||||
|
records := make([]record, len(npcs))
|
||||||
|
issues := make([]Issue, 0)
|
||||||
|
canonicalOwners := make(map[string][]int)
|
||||||
|
idOwners := make(map[string][]int)
|
||||||
|
aliasOwners := make(map[string][]int)
|
||||||
|
|
||||||
|
for recordIndex, npc := range npcs {
|
||||||
|
canonical := ComparisonKey(npc.Name)
|
||||||
|
records[recordIndex].canonical = canonical
|
||||||
|
if canonical == "" {
|
||||||
|
issues = append(issues, Issue{Code: IssueEmptyCanonicalName, RecordIndex: recordIndex, AliasIndex: -1, Value: npc.Name})
|
||||||
|
} else {
|
||||||
|
canonicalOwners[canonical] = append(canonicalOwners[canonical], recordIndex)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !IsValidID(npc.ID) {
|
||||||
|
issues = append(issues, Issue{Code: IssueInvalidID, RecordIndex: recordIndex, AliasIndex: -1, Value: npc.ID})
|
||||||
|
} else if expected := DeriveID(npc.Name); npc.ID != expected {
|
||||||
|
issues = append(issues, Issue{Code: IssueIDMismatch, RecordIndex: recordIndex, AliasIndex: -1, Value: npc.ID})
|
||||||
|
}
|
||||||
|
if npc.ID != "" {
|
||||||
|
idOwners[npc.ID] = append(idOwners[npc.ID], recordIndex)
|
||||||
|
}
|
||||||
|
|
||||||
|
seenAliases := make(map[string]int, len(npc.Aliases))
|
||||||
|
for aliasIndex, alias := range npc.Aliases {
|
||||||
|
key := ComparisonKey(alias)
|
||||||
|
records[recordIndex].aliases = append(records[recordIndex].aliases, key)
|
||||||
|
if key == "" {
|
||||||
|
issues = append(issues, Issue{Code: IssueEmptyAlias, RecordIndex: recordIndex, AliasIndex: aliasIndex, Value: alias})
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, ok := seenAliases[key]; ok {
|
||||||
|
issues = append(issues, Issue{Code: IssueDuplicateAlias, RecordIndex: recordIndex, AliasIndex: aliasIndex, Value: alias})
|
||||||
|
} else {
|
||||||
|
seenAliases[key] = aliasIndex
|
||||||
|
}
|
||||||
|
if key == canonical {
|
||||||
|
issues = append(issues, Issue{Code: IssueOwnCanonicalAlias, RecordIndex: recordIndex, AliasIndex: aliasIndex, Value: alias})
|
||||||
|
}
|
||||||
|
aliasOwners[key] = append(aliasOwners[key], recordIndex)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
for recordIndex, record := range records {
|
||||||
|
if record.canonical != "" && len(canonicalOwners[record.canonical]) > 1 && canonicalOwners[record.canonical][0] != recordIndex {
|
||||||
|
issues = append(issues, Issue{Code: IssueDuplicateCanonical, RecordIndex: recordIndex, AliasIndex: -1, Value: npcs[recordIndex].Name})
|
||||||
|
}
|
||||||
|
if id := npcs[recordIndex].ID; id != "" && len(idOwners[id]) > 1 && idOwners[id][0] != recordIndex {
|
||||||
|
issues = append(issues, Issue{Code: IssueDuplicateID, RecordIndex: recordIndex, AliasIndex: -1, Value: id})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
seenAliasKeys := make(map[string]struct{})
|
||||||
|
for _, record := range records {
|
||||||
|
for _, alias := range record.aliases {
|
||||||
|
if alias == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if _, alreadyProcessed := seenAliasKeys[alias]; alreadyProcessed {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seenAliasKeys[alias] = struct{}{}
|
||||||
|
owners := uniqueIndexes(aliasOwners[alias])
|
||||||
|
if len(owners) > 1 {
|
||||||
|
for _, recordIndex := range owners {
|
||||||
|
issues = append(issues, Issue{Code: IssueAliasOwnershipCollision, RecordIndex: recordIndex, AliasIndex: aliasIndexFor(records[recordIndex].aliases, alias), Value: alias})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for _, recordIndex := range owners {
|
||||||
|
for _, canonicalOwner := range canonicalOwners[alias] {
|
||||||
|
if canonicalOwner != recordIndex {
|
||||||
|
issues = append(issues, Issue{Code: IssueAliasCanonicalCollision, RecordIndex: recordIndex, AliasIndex: aliasIndexFor(records[recordIndex].aliases, alias), Value: alias})
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return issues
|
||||||
|
}
|
||||||
|
|
||||||
|
// ValidateList validates the identity members of list.
|
||||||
|
func ValidateList(list dnd.NPCList) []Issue { return ValidateRegistry(list.NPCs) }
|
||||||
|
|
||||||
|
// Validate is a convenience alias for ValidateList.
|
||||||
|
func Validate(list dnd.NPCList) []Issue { return ValidateList(list) }
|
||||||
|
|
||||||
|
func uniqueIndexes(values []int) []int {
|
||||||
|
seen := make(map[int]struct{}, len(values))
|
||||||
|
unique := make([]int, 0, len(values))
|
||||||
|
for _, value := range values {
|
||||||
|
if _, ok := seen[value]; ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
seen[value] = struct{}{}
|
||||||
|
unique = append(unique, value)
|
||||||
|
}
|
||||||
|
return unique
|
||||||
|
}
|
||||||
|
|
||||||
|
func aliasIndexFor(aliases []string, key string) int {
|
||||||
|
for index, alias := range aliases {
|
||||||
|
if alias == key {
|
||||||
|
return index
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return -1
|
||||||
|
}
|
||||||
|
|
||||||
|
// Error makes an issue useful in simple callers while preserving its
|
||||||
|
// structured fields for aggregate diagnostics.
|
||||||
|
func (i Issue) Error() string {
|
||||||
|
return fmt.Sprintf("%s at record %d", i.Code, i.RecordIndex)
|
||||||
|
}
|
||||||
114
internal/modules/dnd/npcs/identity/identity_test.go
Normal file
114
internal/modules/dnd/npcs/identity/identity_test.go
Normal file
@@ -0,0 +1,114 @@
|
|||||||
|
package identity
|
||||||
|
|
||||||
|
import (
|
||||||
|
"strings"
|
||||||
|
"sync"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestComparisonKeyNormalizesSupportedEquivalences(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
left string
|
||||||
|
right string
|
||||||
|
}{
|
||||||
|
{name: "case", left: "Captain Vale", right: "cAPtAiN vALE"},
|
||||||
|
{name: "compatibility", left: "Ally", right: "Ally"},
|
||||||
|
{name: "whitespace", left: " Mira\u2003Thorn ", right: "Mira Thorn"},
|
||||||
|
{name: "apostrophe", left: "O’Rin", right: "o'Rin"},
|
||||||
|
{name: "modifier apostrophe", left: "OʼRin", right: "o'Rin"},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
if ComparisonKey(test.left) != ComparisonKey(test.right) {
|
||||||
|
t.Fatalf("ComparisonKey(%q) = %q, ComparisonKey(%q) = %q", test.left, ComparisonKey(test.left), test.right, ComparisonKey(test.right))
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
if ComparisonKey("Mira Thorn") == ComparisonKey("Mira Thorne") {
|
||||||
|
t.Fatal("different names received the same comparison key")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestNormalizeDisplayOnlyChangesWhitespace(t *testing.T) {
|
||||||
|
if got := NormalizeDisplay(" O’Rin\u2003Thorn "); got != "O’Rin Thorn" {
|
||||||
|
t.Fatalf("NormalizeDisplay() = %q", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDeriveIDAndIDSyntax(t *testing.T) {
|
||||||
|
got := DeriveID(" Mira\u2003Thorn ")
|
||||||
|
if len(got) != len("npc:sha256:")+64 || !strings.HasPrefix(got, "npc:sha256:") || !IsValidID(got) {
|
||||||
|
t.Fatalf("DeriveID() = %q, want exact NPC ID syntax", got)
|
||||||
|
}
|
||||||
|
if got != DeriveID("Mira Thorn") || got != IDFor("Mira Thorn") {
|
||||||
|
t.Fatal("DeriveID() is not deterministic across equivalent names")
|
||||||
|
}
|
||||||
|
if DeriveID(" \u2003 ") != "" {
|
||||||
|
t.Fatal("empty identity produced an ID")
|
||||||
|
}
|
||||||
|
for _, invalid := range []string{"", "npc:sha256:", "npc:sha256:ABC", "npc:sha256:" + strings.Repeat("0", 63), "npc:sha256:" + strings.Repeat("0", 65)} {
|
||||||
|
if IsValidID(invalid) {
|
||||||
|
t.Fatalf("IsValidID(%q) = true, want false", invalid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestIdentityFunctionsAreSafeForConcurrentUse(t *testing.T) {
|
||||||
|
const workers = 32
|
||||||
|
var group sync.WaitGroup
|
||||||
|
for i := 0; i < workers; i++ {
|
||||||
|
group.Add(1)
|
||||||
|
go func() {
|
||||||
|
defer group.Done()
|
||||||
|
for j := 0; j < 100; j++ {
|
||||||
|
if !IsValidID(DeriveID("Mira Thorn")) {
|
||||||
|
t.Errorf("derived ID failed syntax check")
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}()
|
||||||
|
}
|
||||||
|
group.Wait()
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRegistryReportsIdentityCollisionCategories(t *testing.T) {
|
||||||
|
validID := DeriveID("Mira Thorn")
|
||||||
|
npcs := []dnd.NPC{
|
||||||
|
{ID: validID, Name: "Mira Thorn", Aliases: []string{"The Greencloak", "the greencloak", "Mira Thorn"}},
|
||||||
|
{ID: validID, Name: "Mira Thorn", Aliases: []string{"The Greencloak"}},
|
||||||
|
{ID: DeriveID("Captain Vale"), Name: "Captain Vale", Aliases: []string{"Mira Thorn"}},
|
||||||
|
}
|
||||||
|
issues := ValidateRegistry(npcs)
|
||||||
|
want := map[IssueCode]bool{
|
||||||
|
IssueDuplicateAlias: false,
|
||||||
|
IssueOwnCanonicalAlias: false,
|
||||||
|
IssueDuplicateCanonical: false,
|
||||||
|
IssueDuplicateID: false,
|
||||||
|
IssueAliasOwnershipCollision: false,
|
||||||
|
IssueAliasCanonicalCollision: false,
|
||||||
|
}
|
||||||
|
for _, issue := range issues {
|
||||||
|
if _, ok := want[issue.Code]; ok {
|
||||||
|
want[issue.Code] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for code, found := range want {
|
||||||
|
if !found {
|
||||||
|
t.Errorf("ValidateRegistry() did not report %s", code)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRegistryReportsIDProblems(t *testing.T) {
|
||||||
|
issues := ValidateRegistry([]dnd.NPC{{ID: "bad", Name: "Mira Thorn"}, {ID: DeriveID("Mira Thorn"), Name: "Other Name"}})
|
||||||
|
seen := map[IssueCode]bool{}
|
||||||
|
for _, issue := range issues {
|
||||||
|
seen[issue.Code] = true
|
||||||
|
}
|
||||||
|
if !seen[IssueInvalidID] || !seen[IssueIDMismatch] {
|
||||||
|
t.Fatalf("ValidateRegistry() issues = %#v, want invalid and mismatched ID issues", issues)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,9 +8,16 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||||
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||||
spellcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/spells"
|
spellcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/spells"
|
||||||
|
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||||
|
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
|
npcidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/identity"
|
||||||
|
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
|
||||||
|
npcsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/source_refs"
|
||||||
|
npcrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/source_relatedness"
|
||||||
spellcatalog "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/catalog"
|
spellcatalog "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/catalog"
|
||||||
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
|
spellshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/shape"
|
||||||
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
|
spellsourcerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/spells/source_refs"
|
||||||
@@ -34,25 +41,43 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
|||||||
register func() error
|
register func() error
|
||||||
}{
|
}{
|
||||||
{name: "spells codec", register: func() error { return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, codec) }},
|
{name: "spells codec", register: func() error { return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, codec) }},
|
||||||
|
{name: "npcs codec", register: func() error { return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, npccodec.New()) }},
|
||||||
{name: "scenes chunker", register: func() error { return scenes.Register(registries.Chunkers) }},
|
{name: "scenes chunker", register: func() error { return scenes.Register(registries.Chunkers) }},
|
||||||
{name: "spells extractor", register: func() error { return spells.Register(registries.Extractors) }},
|
{name: "spells extractor", register: func() error { return spells.Register(registries.Extractors) }},
|
||||||
|
{name: "npcs extractor", register: func() error { return npcextract.Register(registries.Extractors) }},
|
||||||
{name: "spell-list appendorder merger", register: func() error {
|
{name: "spell-list appendorder merger", register: func() error {
|
||||||
return appendorder.RegisterTyped(registries.Mergers, dnd.SpellListKind, appendSpellLists)
|
return appendorder.RegisterTyped(registries.Mergers, dnd.SpellListKind, appendSpellLists)
|
||||||
}},
|
}},
|
||||||
|
{name: "npc-list appendorder merger", register: func() error {
|
||||||
|
return appendorder.RegisterTyped(registries.Mergers, dnd.NPCListKind, appendNPCLists)
|
||||||
|
}},
|
||||||
{name: "spells normalizer", register: func() error { return spellnormalize.Register(registries.Normalizers) }},
|
{name: "spells normalizer", register: func() error { return spellnormalize.Register(registries.Normalizers) }},
|
||||||
|
{name: "npcs normalizer", register: func() error { return npcnormalize.Register(registries.Normalizers) }},
|
||||||
{name: "spell-list noop normalizer", register: func() error { return noop.RegisterTyped[dnd.SpellList](registries.Normalizers, dnd.SpellListKind) }},
|
{name: "spell-list noop normalizer", register: func() error { return noop.RegisterTyped[dnd.SpellList](registries.Normalizers, dnd.SpellListKind) }},
|
||||||
|
{name: "npc-list noop normalizer", register: func() error { return noop.RegisterTyped[dnd.NPCList](registries.Normalizers, dnd.NPCListKind) }},
|
||||||
{name: "spell shape validator", register: func() error { return spellshape.Register(registries.Validators) }},
|
{name: "spell shape validator", register: func() error { return spellshape.Register(registries.Validators) }},
|
||||||
{name: "spell catalog validator", register: func() error { return spellcatalog.Register(registries.Validators) }},
|
{name: "spell catalog validator", register: func() error { return spellcatalog.Register(registries.Validators) }},
|
||||||
{name: "spell source references validator", register: func() error { return spellsourcerefs.Register(registries.Validators) }},
|
{name: "spell source references validator", register: func() error { return spellsourcerefs.Register(registries.Validators) }},
|
||||||
{name: "spell source relatedness validator", register: func() error { return spellrelatedness.Register(registries.Validators) }},
|
{name: "spell source relatedness validator", register: func() error { return spellrelatedness.Register(registries.Validators) }},
|
||||||
|
{name: "npc shape validator", register: func() error { return npcshape.Register(registries.Validators) }},
|
||||||
|
{name: "npc identity validator", register: func() error { return npcidentity.Register(registries.Validators) }},
|
||||||
|
{name: "npc source references validator", register: func() error { return npcsourcerefs.Register(registries.Validators) }},
|
||||||
|
{name: "npc source relatedness validator", register: func() error { return npcrelatedness.Register(registries.Validators) }},
|
||||||
{name: "spell-list always accept validator", register: func() error {
|
{name: "spell-list always accept validator", register: func() error {
|
||||||
return alwaysaccept.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
|
return alwaysaccept.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
|
||||||
}},
|
}},
|
||||||
{name: "spell-list always reject validator", register: func() error {
|
{name: "spell-list always reject validator", register: func() error {
|
||||||
return alwaysreject.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
|
return alwaysreject.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
|
||||||
}},
|
}},
|
||||||
|
{name: "npc-list always accept validator", register: func() error {
|
||||||
|
return alwaysaccept.RegisterTyped[dnd.NPCList](registries.Validators, dnd.NPCListKind)
|
||||||
|
}},
|
||||||
|
{name: "npc-list always reject validator", register: func() error {
|
||||||
|
return alwaysreject.RegisterTyped[dnd.NPCList](registries.Validators, dnd.NPCListKind)
|
||||||
|
}},
|
||||||
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
|
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
|
||||||
{name: "spells prompt assets", register: func() error { return spells.RegisterPromptAssets(assets) }},
|
{name: "spells prompt assets", register: func() error { return spells.RegisterPromptAssets(assets) }},
|
||||||
|
{name: "npcs prompt assets", register: func() error { return npcextract.RegisterPromptAssets(assets) }},
|
||||||
}
|
}
|
||||||
for _, registration := range registrations {
|
for _, registration := range registrations {
|
||||||
if err := registration.register(); err != nil {
|
if err := registration.register(); err != nil {
|
||||||
@@ -87,6 +112,33 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
|||||||
}); err != nil {
|
}); err != nil {
|
||||||
return fmt.Errorf("register dnd spells normalize validator chain: %w", err)
|
return fmt.Errorf("register dnd spells normalize validator chain: %w", err)
|
||||||
}
|
}
|
||||||
|
if err := registries.ValidatorChains.Register(pipeline.ValidatorChainMapping{
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
Module: npcextract.Key,
|
||||||
|
Validators: []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding(validjson.Key),
|
||||||
|
pipeline.Binding(validjsonschema.Key),
|
||||||
|
pipeline.Binding(npcshape.Key),
|
||||||
|
pipeline.Binding(npcsourcerefs.Key),
|
||||||
|
pipeline.Binding(npcrelatedness.Key),
|
||||||
|
},
|
||||||
|
}); err != nil {
|
||||||
|
return fmt.Errorf("register dnd npcs validator chain: %w", err)
|
||||||
|
}
|
||||||
|
if err := registries.ValidatorChains.Register(pipeline.ValidatorChainMapping{
|
||||||
|
Stage: pipeline.StageNormalize,
|
||||||
|
Module: npcnormalize.Key,
|
||||||
|
Validators: []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding(validjson.Key),
|
||||||
|
pipeline.Binding(validjsonschema.Key),
|
||||||
|
pipeline.Binding(npcshape.Key),
|
||||||
|
pipeline.Binding(npcidentity.Key),
|
||||||
|
pipeline.Binding(npcsourcerefs.Key),
|
||||||
|
pipeline.Binding(npcrelatedness.Key),
|
||||||
|
},
|
||||||
|
}); err != nil {
|
||||||
|
return fmt.Errorf("register dnd npcs normalize validator chain: %w", err)
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -102,6 +154,25 @@ func appendSpellLists(values []dnd.SpellList) (dnd.SpellList, error) {
|
|||||||
return combined, nil
|
return combined, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func appendNPCLists(values []dnd.NPCList) (dnd.NPCList, error) {
|
||||||
|
count := 0
|
||||||
|
present := false
|
||||||
|
for _, value := range values {
|
||||||
|
if value.NPCs != nil {
|
||||||
|
present = true
|
||||||
|
}
|
||||||
|
count += len(value.NPCs)
|
||||||
|
}
|
||||||
|
if !present {
|
||||||
|
return dnd.NPCList{}, nil
|
||||||
|
}
|
||||||
|
combined := dnd.NPCList{NPCs: make([]dnd.NPC, 0, count)}
|
||||||
|
for _, value := range values {
|
||||||
|
combined.NPCs = append(combined.NPCs, value.NPCs...)
|
||||||
|
}
|
||||||
|
return combined, nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateRegistries(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
func validateRegistries(registries pipeline.Registries, assets *llm.AssetRegistry) error {
|
||||||
switch {
|
switch {
|
||||||
case registries.Chunkers == nil:
|
case registries.Chunkers == nil:
|
||||||
|
|||||||
@@ -11,7 +11,9 @@ import (
|
|||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||||
|
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -22,10 +24,17 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
|||||||
t.Fatalf("Register() error = %v, want nil", err)
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
}
|
}
|
||||||
assertContainsKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
|
assertContainsKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
|
||||||
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells"})
|
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells", npcextract.Key})
|
||||||
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, pipeline.DefaultNormalizeModule})
|
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, npcnormalize.Key, pipeline.DefaultNormalizeModule})
|
||||||
assertContainsArtifactKinds(t, registries.ArtifactCodecs.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind})
|
assertContainsArtifactKinds(t, registries.ArtifactCodecs.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind})
|
||||||
|
assertContainsArtifactKinds(t, registries.Mergers.RegisteredArtifactKinds(pipeline.DefaultMergeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind})
|
||||||
|
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(pipeline.DefaultNormalizeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind})
|
||||||
|
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(npcnormalize.Key), []contracts.ArtifactKind{dnd.NPCListKind})
|
||||||
assertContainsKeys(t, "validators", registries.Validators.RegisteredKeys(), []string{
|
assertContainsKeys(t, "validators", registries.Validators.RegisteredKeys(), []string{
|
||||||
|
"extract/dnd/npcs/shape",
|
||||||
|
"extract/dnd/npcs/source_refs",
|
||||||
|
"extract/dnd/npcs/source_relatedness",
|
||||||
|
"normalize/dnd/npcs/identity",
|
||||||
"extract/dnd/spells/catalog",
|
"extract/dnd/spells/catalog",
|
||||||
"extract/dnd/spells/shape",
|
"extract/dnd/spells/shape",
|
||||||
"extract/dnd/spells/source_refs",
|
"extract/dnd/spells/source_refs",
|
||||||
@@ -47,6 +56,30 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
|||||||
if got := registries.ValidatorChains.Validators(pipeline.StageNormalize, spellnormalize.Key); !reflect.DeepEqual(got, wantChain) {
|
if got := registries.ValidatorChains.Validators(pipeline.StageNormalize, spellnormalize.Key); !reflect.DeepEqual(got, wantChain) {
|
||||||
t.Fatalf("spell normalize validator chain = %#v, want %#v", got, wantChain)
|
t.Fatalf("spell normalize validator chain = %#v, want %#v", got, wantChain)
|
||||||
}
|
}
|
||||||
|
npcExtractChain := []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding("generic/valid_json"),
|
||||||
|
pipeline.Binding("generic/valid_json_schema"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/shape"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_refs"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
|
||||||
|
}
|
||||||
|
if got := registries.ValidatorChains.Validators(pipeline.StageExtract, npcextract.Key); !reflect.DeepEqual(got, npcExtractChain) {
|
||||||
|
t.Fatalf("NPC extract validator chain = %#v, want %#v", got, npcExtractChain)
|
||||||
|
}
|
||||||
|
npcNormalizeChain := []pipeline.ModuleBinding{
|
||||||
|
pipeline.Binding("generic/valid_json"),
|
||||||
|
pipeline.Binding("generic/valid_json_schema"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/shape"),
|
||||||
|
pipeline.Binding("normalize/dnd/npcs/identity"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_refs"),
|
||||||
|
pipeline.Binding("extract/dnd/npcs/source_relatedness"),
|
||||||
|
}
|
||||||
|
if got := registries.ValidatorChains.Validators(pipeline.StageNormalize, npcnormalize.Key); !reflect.DeepEqual(got, npcNormalizeChain) {
|
||||||
|
t.Fatalf("NPC normalize validator chain = %#v, want %#v", got, npcNormalizeChain)
|
||||||
|
}
|
||||||
|
if got := registries.ValidatorChains.Validators(pipeline.StageMerge, npcextract.Key); got != nil {
|
||||||
|
t.Fatalf("NPC merge validator chain = %#v, want absent", got)
|
||||||
|
}
|
||||||
assertAssetNamesContain(t, assets.PromptFS, []string{
|
assertAssetNamesContain(t, assets.PromptFS, []string{
|
||||||
"dnd.scenes/dnd.scenes.yaml",
|
"dnd.scenes/dnd.scenes.yaml",
|
||||||
"dnd.scenes/instructions.md",
|
"dnd.scenes/instructions.md",
|
||||||
@@ -60,10 +93,17 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
|||||||
"dnd.spells/sharedassets/common-dnd-system.md",
|
"dnd.spells/sharedassets/common-dnd-system.md",
|
||||||
"dnd.spells/sharedassets/common-dnd-transcript.md",
|
"dnd.spells/sharedassets/common-dnd-transcript.md",
|
||||||
"dnd.spells/task.md",
|
"dnd.spells/task.md",
|
||||||
|
"dnd.npcs/dnd.npcs.yaml",
|
||||||
|
"dnd.npcs/instructions.md",
|
||||||
|
"dnd.npcs/sharedassets/common-dnd-references.md",
|
||||||
|
"dnd.npcs/sharedassets/common-dnd-system.md",
|
||||||
|
"dnd.npcs/sharedassets/common-dnd-transcript.md",
|
||||||
|
"dnd.npcs/task.md",
|
||||||
})
|
})
|
||||||
assertAssetNamesContain(t, assets.SchemaFS, []string{
|
assertAssetNamesContain(t, assets.SchemaFS, []string{
|
||||||
"dnd_scenes.v1.json",
|
"dnd_scenes.v1.json",
|
||||||
"dnd_spells_llm.v1.json",
|
"dnd_spells_llm.v1.json",
|
||||||
|
"dnd_npcs_llm.v1.json",
|
||||||
})
|
})
|
||||||
if spec, ok := registries.Chunkers.Spec("dnd/scenes"); !ok || spec.Key != "dnd/scenes" {
|
if spec, ok := registries.Chunkers.Spec("dnd/scenes"); !ok || spec.Key != "dnd/scenes" {
|
||||||
t.Fatalf("scene chunker spec = %#v, present = %t; want family-owned spec", spec, ok)
|
t.Fatalf("scene chunker spec = %#v, present = %t; want family-owned spec", spec, ok)
|
||||||
@@ -74,6 +114,33 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
|||||||
if spec, ok := registries.Normalizers.Spec(spellnormalize.Key); !ok || spec.ArtifactKind != dnd.SpellListKind || spec.Stage != pipeline.StageNormalize {
|
if spec, ok := registries.Normalizers.Spec(spellnormalize.Key); !ok || spec.ArtifactKind != dnd.SpellListKind || spec.Stage != pipeline.StageNormalize {
|
||||||
t.Fatalf("spell normalizer spec = %#v, present = %t; want dnd spell-list artifact", spec, ok)
|
t.Fatalf("spell normalizer spec = %#v, present = %t; want dnd spell-list artifact", spec, ok)
|
||||||
}
|
}
|
||||||
|
if spec, ok := registries.Extractors.Spec(npcextract.Key); !ok || spec.ArtifactKind != dnd.NPCListKind {
|
||||||
|
t.Fatalf("NPC extractor spec = %#v, present = %t; want dnd NPC-list artifact", spec, ok)
|
||||||
|
}
|
||||||
|
if spec, ok := registries.Normalizers.Spec(npcnormalize.Key); !ok || spec.ArtifactKind != dnd.NPCListKind || spec.Stage != pipeline.StageNormalize {
|
||||||
|
t.Fatalf("NPC normalizer spec = %#v, present = %t; want dnd NPC-list artifact", spec, ok)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAppendNPCListsPreservesOrderAndArrayPresence(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
in []dnd.NPCList
|
||||||
|
want dnd.NPCList
|
||||||
|
}{
|
||||||
|
{name: "no values", in: nil, want: dnd.NPCList{}},
|
||||||
|
{name: "nil values", in: []dnd.NPCList{{}, {}}, want: dnd.NPCList{}},
|
||||||
|
{name: "present empty", in: []dnd.NPCList{{NPCs: []dnd.NPC{}}}, want: dnd.NPCList{NPCs: []dnd.NPC{}}},
|
||||||
|
{name: "ordered values", in: []dnd.NPCList{{NPCs: []dnd.NPC{{Name: "first"}}}, {NPCs: []dnd.NPC{{Name: "second"}}}}, want: dnd.NPCList{NPCs: []dnd.NPC{{Name: "first"}, {Name: "second"}}}},
|
||||||
|
}
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
got, err := appendNPCLists(tt.in)
|
||||||
|
if err != nil || !reflect.DeepEqual(got, tt.want) {
|
||||||
|
t.Fatalf("appendNPCLists() = %#v, error = %v, want %#v", got, err, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRegisterRejectsMissingDNDDependenciesBeforeMutation(t *testing.T) {
|
func TestRegisterRejectsMissingDNDDependenciesBeforeMutation(t *testing.T) {
|
||||||
|
|||||||
@@ -8,6 +8,8 @@ import (
|
|||||||
|
|
||||||
const SpellListKind contracts.ArtifactKind = "dnd/spell-list"
|
const SpellListKind contracts.ArtifactKind = "dnd/spell-list"
|
||||||
|
|
||||||
|
const NPCListKind contracts.ArtifactKind = "dnd/npc-list"
|
||||||
|
|
||||||
type SpellList struct {
|
type SpellList struct {
|
||||||
SpellCasts []SpellCast `json:"spell_casts"`
|
SpellCasts []SpellCast `json:"spell_casts"`
|
||||||
}
|
}
|
||||||
@@ -19,3 +21,21 @@ type SpellCast struct {
|
|||||||
NarrativeDescription string `json:"narrative_description"`
|
NarrativeDescription string `json:"narrative_description"`
|
||||||
SourceRefs []source.SourceRef `json:"source_refs"`
|
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type NPCList struct {
|
||||||
|
NPCs []NPC `json:"npcs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type NPC struct {
|
||||||
|
ID string `json:"id"`
|
||||||
|
Name string `json:"name"`
|
||||||
|
Aliases []string `json:"aliases"`
|
||||||
|
Description string `json:"description"`
|
||||||
|
Relationships []NPCRelationship `json:"relationships"`
|
||||||
|
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type NPCRelationship struct {
|
||||||
|
Target string `json:"target"`
|
||||||
|
Relationship string `json:"relationship"`
|
||||||
|
}
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
// Package diagnostics provides bounded, safe text for deterministic NPC
|
||||||
|
// validator decisions and warnings.
|
||||||
|
package diagnostics
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
MaxIssues = 20
|
||||||
|
MaxDisplayedRunes = 128
|
||||||
|
MaxMessageBytes = 4096
|
||||||
|
)
|
||||||
|
|
||||||
|
func Truncate(value string) string {
|
||||||
|
runes := []rune(value)
|
||||||
|
if len(runes) <= MaxDisplayedRunes {
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
return string(runes[:MaxDisplayedRunes-1]) + "…"
|
||||||
|
}
|
||||||
|
|
||||||
|
func Quote(value string) string { return strconv.Quote(Truncate(value)) }
|
||||||
|
|
||||||
|
func Aggregate(prefix string, issues []string) string {
|
||||||
|
displayed := make([]string, 0, min(len(issues), MaxIssues))
|
||||||
|
for len(displayed) < len(issues) && len(displayed) < MaxIssues {
|
||||||
|
issue := Truncate(issues[len(displayed)])
|
||||||
|
candidate := aggregateMessage(prefix, append(displayed, issue), len(issues)-len(displayed)-1)
|
||||||
|
if len([]byte(candidate)) > MaxMessageBytes {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
displayed = append(displayed, issue)
|
||||||
|
}
|
||||||
|
return aggregateMessage(prefix, displayed, len(issues)-len(displayed))
|
||||||
|
}
|
||||||
|
|
||||||
|
func aggregateMessage(prefix string, issues []string, omitted int) string {
|
||||||
|
message := prefix + ": " + strings.Join(issues, ", ")
|
||||||
|
if omitted > 0 {
|
||||||
|
message += fmt.Sprintf("; %d additional issue(s) omitted", omitted)
|
||||||
|
}
|
||||||
|
return message
|
||||||
|
}
|
||||||
|
|
||||||
|
func min(left, right int) int {
|
||||||
|
if left < right {
|
||||||
|
return left
|
||||||
|
}
|
||||||
|
return right
|
||||||
|
}
|
||||||
84
internal/modules/dnd/validate/npcs/identity/validator.go
Normal file
84
internal/modules/dnd/validate/npcs/identity/validator.go
Normal file
@@ -0,0 +1,84 @@
|
|||||||
|
// Package identity validates the stable identity invariants of D&D NPC lists.
|
||||||
|
package identity
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
domainidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
|
||||||
|
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Key = "normalize/dnd/npcs/identity"
|
||||||
|
ReasonCode = "invalid_npc_identity"
|
||||||
|
policy = domainidentity.Policy
|
||||||
|
)
|
||||||
|
|
||||||
|
type Options struct{}
|
||||||
|
type Validator struct{}
|
||||||
|
|
||||||
|
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
|
||||||
|
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
|
||||||
|
|
||||||
|
func New(Options) *Validator { return &Validator{} }
|
||||||
|
func (v *Validator) Name() string { return Key }
|
||||||
|
|
||||||
|
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||||
|
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
|
||||||
|
if err := npcshape.Validate(req.Value); err != nil {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
identityIssues := domainidentity.ValidateRegistry(req.Value.NPCs)
|
||||||
|
if len(identityIssues) == 0 {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
issues := make([]string, len(identityIssues))
|
||||||
|
for index, issue := range identityIssues {
|
||||||
|
location := fmt.Sprintf("npcs[%d]", issue.RecordIndex)
|
||||||
|
if issue.AliasIndex >= 0 {
|
||||||
|
location += fmt.Sprintf(".aliases[%d]", issue.AliasIndex)
|
||||||
|
}
|
||||||
|
issues[index] = fmt.Sprintf("%s %s: %s", location, issue.Code, diagnostics.Quote(issue.Value))
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{
|
||||||
|
Approved: false,
|
||||||
|
ReasonCode: ReasonCode,
|
||||||
|
Message: diagnostics.Aggregate("invalid NPC identity", issues),
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Spec() pipeline.ValidatorSpec {
|
||||||
|
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.ValidatorRegistry) error {
|
||||||
|
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
|
||||||
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(options), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||||
107
internal/modules/dnd/validate/npcs/identity/validator_test.go
Normal file
107
internal/modules/dnd/validate/npcs/identity/validator_test.go
Normal file
@@ -0,0 +1,107 @@
|
|||||||
|
package identity
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestValidatorContractAndRegistration(t *testing.T) {
|
||||||
|
if _, err := DecodeOptions(nil); err != nil {
|
||||||
|
t.Fatalf("DecodeOptions(nil) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil || !strings.Contains(err.Error(), "unknown option") {
|
||||||
|
t.Fatalf("DecodeOptions() error = %v, want unknown option error", err)
|
||||||
|
}
|
||||||
|
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != policy {
|
||||||
|
t.Fatalf("fingerprints = %#v, want identity policy", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
if got := Spec(); !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
registry := pipeline.NewValidatorRegistry()
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if registered, ok := registry.Spec(Key); !ok || !reflect.DeepEqual(registered, want) {
|
||||||
|
t.Fatalf("registered spec = %#v, ok = %t, want %#v", registered, ok, want)
|
||||||
|
}
|
||||||
|
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "validator registry") {
|
||||||
|
t.Fatalf("Register(nil) error = %v, want registry error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorDefersShapeAndRejectsIdentityIssues(t *testing.T) {
|
||||||
|
validator := New(Options{})
|
||||||
|
shapeInvalid := dnd.NPCList{NPCs: []dnd.NPC{{Name: "missing description"}}}
|
||||||
|
result, err := validator.Validate(context.Background(), validationRequest(shapeInvalid))
|
||||||
|
if err != nil || !result.Approved {
|
||||||
|
t.Fatalf("shape-invalid result = %#v, error = %v, want deferred approval", result, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
value := validNPCList(2)
|
||||||
|
value.NPCs[0].ID = "not-an-id"
|
||||||
|
value.NPCs[1].Aliases = []string{"Shared Alias"}
|
||||||
|
value.NPCs[0].Aliases = []string{"Shared Alias"}
|
||||||
|
result, err = validator.Validate(context.Background(), validationRequest(value))
|
||||||
|
if err != nil || result.Approved || result.ReasonCode != ReasonCode {
|
||||||
|
t.Fatalf("identity result = %#v, error = %v, want rejection", result, err)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"invalid_id", "alias_owned_by_multiple_records", "npcs[0]", "npcs[1]"} {
|
||||||
|
if !strings.Contains(result.Message, want) {
|
||||||
|
t.Fatalf("identity message %q missing %q", result.Message, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorBoundsUnicodeDiagnostics(t *testing.T) {
|
||||||
|
value := validNPCList(30)
|
||||||
|
for index := range value.NPCs {
|
||||||
|
value.NPCs[index].ID = fmt.Sprintf("bad-%d", index)
|
||||||
|
value.NPCs[index].Name = strings.Repeat("火", 140) + fmt.Sprintf("-%d", index)
|
||||||
|
}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), validationRequest(value))
|
||||||
|
if err != nil || result.Approved {
|
||||||
|
t.Fatalf("bounded result = %#v, error = %v, want rejection", result, err)
|
||||||
|
}
|
||||||
|
if !utf8.ValidString(result.Message) || len([]rune(result.Message)) == 0 || len([]byte(result.Message)) > 4096 {
|
||||||
|
t.Fatalf("bounded message has invalid size/encoding: bytes=%d message=%q", len([]byte(result.Message)), result.Message)
|
||||||
|
}
|
||||||
|
if !strings.Contains(result.Message, "additional issue(s) omitted") || strings.Contains(result.Message, strings.Repeat("火", 140)) {
|
||||||
|
t.Fatalf("bounded message = %q, want omission and truncation", result.Message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validationRequest(value dnd.NPCList) contracts.TypedValidationRequest[dnd.NPCList] {
|
||||||
|
return contracts.TypedValidationRequest[dnd.NPCList]{Value: value}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validNPCList(count int) dnd.NPCList {
|
||||||
|
value := dnd.NPCList{NPCs: make([]dnd.NPC, count)}
|
||||||
|
for index := range value.NPCs {
|
||||||
|
name := fmt.Sprintf("NPC %d", index)
|
||||||
|
value.NPCs[index] = dnd.NPC{
|
||||||
|
ID: "npc:sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||||
|
Name: name,
|
||||||
|
Aliases: []string{},
|
||||||
|
Description: "description",
|
||||||
|
Relationships: []dnd.NPCRelationship{},
|
||||||
|
SourceRefs: []source.SourceRef{sourceRefForTest()},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
|
||||||
|
func sourceRefForTest() source.SourceRef {
|
||||||
|
return source.SourceRef{SourceID: "source", StartUnitID: 1, EndUnitID: 1}
|
||||||
|
}
|
||||||
121
internal/modules/dnd/validate/npcs/shape/validator.go
Normal file
121
internal/modules/dnd/validate/npcs/shape/validator.go
Normal file
@@ -0,0 +1,121 @@
|
|||||||
|
package shape
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Key = "extract/dnd/npcs/shape"
|
||||||
|
ReasonCode = "invalid_npc_shape"
|
||||||
|
policy = "dnd.npcs.validator.shape.v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Options struct{}
|
||||||
|
type Validator struct{}
|
||||||
|
|
||||||
|
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
|
||||||
|
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
|
||||||
|
|
||||||
|
func New(Options) *Validator { return &Validator{} }
|
||||||
|
func (v *Validator) Name() string { return Key }
|
||||||
|
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||||
|
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
|
||||||
|
issues := issuesFor(req.Value)
|
||||||
|
if len(issues) > 0 {
|
||||||
|
return rejection(diagnostics.Aggregate("invalid NPC shape", issues)), nil
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Validate(value dnd.NPCList) error {
|
||||||
|
issues := issuesFor(value)
|
||||||
|
if len(issues) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%s", diagnostics.Aggregate("invalid NPC shape", issues))
|
||||||
|
}
|
||||||
|
|
||||||
|
func issuesFor(value dnd.NPCList) []string {
|
||||||
|
issues := make([]string, 0)
|
||||||
|
if value.NPCs == nil {
|
||||||
|
return []string{"npcs must be present"}
|
||||||
|
}
|
||||||
|
for index, npc := range value.NPCs {
|
||||||
|
prefix := fmt.Sprintf("npcs[%d]", index)
|
||||||
|
if strings.TrimSpace(npc.ID) == "" {
|
||||||
|
issues = append(issues, prefix+".id must not be empty")
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(npc.Name) == "" {
|
||||||
|
issues = append(issues, prefix+".name must not be empty")
|
||||||
|
}
|
||||||
|
if npc.Aliases == nil {
|
||||||
|
issues = append(issues, prefix+".aliases must be present")
|
||||||
|
} else {
|
||||||
|
for aliasIndex, alias := range npc.Aliases {
|
||||||
|
if strings.TrimSpace(alias) == "" {
|
||||||
|
issues = append(issues, fmt.Sprintf("%s.aliases[%d] must not be empty: %s", prefix, aliasIndex, diagnostics.Quote(alias)))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(npc.Description) == "" {
|
||||||
|
issues = append(issues, prefix+".description must not be empty")
|
||||||
|
}
|
||||||
|
if npc.Relationships == nil {
|
||||||
|
issues = append(issues, prefix+".relationships must be present")
|
||||||
|
} else {
|
||||||
|
for relationshipIndex, relationship := range npc.Relationships {
|
||||||
|
relationshipPrefix := fmt.Sprintf("%s.relationships[%d]", prefix, relationshipIndex)
|
||||||
|
if strings.TrimSpace(relationship.Target) == "" {
|
||||||
|
issues = append(issues, relationshipPrefix+".target must not be empty: "+diagnostics.Quote(relationship.Target))
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(relationship.Relationship) == "" {
|
||||||
|
issues = append(issues, relationshipPrefix+".relationship must not be empty: "+diagnostics.Quote(relationship.Relationship))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(npc.SourceRefs) == 0 {
|
||||||
|
issues = append(issues, prefix+".source_refs must not be empty")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return issues
|
||||||
|
}
|
||||||
|
|
||||||
|
func Spec() pipeline.ValidatorSpec {
|
||||||
|
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.ValidatorRegistry) error {
|
||||||
|
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
|
||||||
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(options), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||||
|
|
||||||
|
func rejection(message string) contracts.ValidationResult {
|
||||||
|
return contracts.ValidationResult{Approved: false, ReasonCode: ReasonCode, Message: message}
|
||||||
|
}
|
||||||
92
internal/modules/dnd/validate/npcs/shape/validator_test.go
Normal file
92
internal/modules/dnd/validate/npcs/shape/validator_test.go
Normal file
@@ -0,0 +1,92 @@
|
|||||||
|
package shape
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestValidatorApprovesWellFormedNPCPayload(t *testing.T) {
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validNPCList()))
|
||||||
|
if err != nil || !result.Approved {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want approval", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorRejectsRequiredShapeValues(t *testing.T) {
|
||||||
|
value := validNPCList()
|
||||||
|
value.NPCs[0].Aliases = nil
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(value))
|
||||||
|
if err != nil || result.Approved || result.ReasonCode != ReasonCode || !strings.Contains(result.Message, "aliases must be present") {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want bounded shape rejection", result, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
missing := dnd.NPCList{}
|
||||||
|
result, err = New(Options{}).Validate(context.Background(), requestWithValue(missing))
|
||||||
|
if err != nil || result.Approved || result.ReasonCode != ReasonCode {
|
||||||
|
t.Fatalf("missing Validate() = %#v, %v; want shape rejection", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorBoundsDiagnosticsAndQuotesUnicode(t *testing.T) {
|
||||||
|
value := dnd.NPCList{NPCs: make([]dnd.NPC, 24)}
|
||||||
|
long := strings.Repeat("火", 220) + "\n\t"
|
||||||
|
for index := range value.NPCs {
|
||||||
|
value.NPCs[index] = dnd.NPC{ID: "candidate", Name: long, Aliases: []string{"\n\t"}, Description: "", Relationships: []dnd.NPCRelationship{{Target: " ", Relationship: " "}}, SourceRefs: []source.SourceRef{}}
|
||||||
|
}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(value))
|
||||||
|
if err != nil || result.Approved || len([]byte(result.Message)) > diagnosticsMaxMessageBytes || !utf8.ValidString(result.Message) {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want bounded valid UTF-8 rejection", result, err)
|
||||||
|
}
|
||||||
|
if strings.Count(result.Message, "npcs[") > diagnosticsMaxIssues || !strings.Contains(result.Message, "additional issue(s) omitted") || !strings.Contains(result.Message, `\n\t`) {
|
||||||
|
t.Fatalf("message = %q, want bounded quoted diagnostics", result.Message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorSpecCheckpointAndRegistration(t *testing.T) {
|
||||||
|
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != "dnd.npcs.validator.shape.v1" {
|
||||||
|
t.Fatalf("CheckpointFingerprints() = %#v, want local policy", got)
|
||||||
|
}
|
||||||
|
if spec := Spec(); spec.Key != Key || spec.ExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("Spec() = %#v, want deterministic shape validator", spec)
|
||||||
|
}
|
||||||
|
registry := pipeline.NewValidatorRegistry()
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||||
|
t.Fatal("DecodeOptions() accepted unknown option")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorDoesNotMutateValue(t *testing.T) {
|
||||||
|
value := validNPCList()
|
||||||
|
before := value
|
||||||
|
_, err := New(Options{}).Validate(context.Background(), requestWithValue(value))
|
||||||
|
if err != nil || value.NPCs[0].Aliases[0] != before.NPCs[0].Aliases[0] {
|
||||||
|
t.Fatalf("Validate() mutated value: %#v", value)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func requestWithValue(value dnd.NPCList) contracts.TypedValidationRequest[dnd.NPCList] {
|
||||||
|
return contracts.TypedValidationRequest[dnd.NPCList]{Value: value}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validNPCList() dnd.NPCList {
|
||||||
|
return dnd.NPCList{NPCs: []dnd.NPC{{
|
||||||
|
ID: "candidate", Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A guarded ranger.",
|
||||||
|
Relationships: []dnd.NPCRelationship{{Target: "Captain Vale", Relationship: "reports to"}},
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
}}}
|
||||||
|
}
|
||||||
|
|
||||||
|
const (
|
||||||
|
diagnosticsMaxIssues = 20
|
||||||
|
diagnosticsMaxMessageBytes = 4096
|
||||||
|
)
|
||||||
79
internal/modules/dnd/validate/npcs/source_refs/validator.go
Normal file
79
internal/modules/dnd/validate/npcs/source_refs/validator.go
Normal file
@@ -0,0 +1,79 @@
|
|||||||
|
package sourcerefs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
|
||||||
|
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Key = "extract/dnd/npcs/source_refs"
|
||||||
|
ReasonCode = "invalid_npc_source_refs"
|
||||||
|
policy = "dnd.npcs.validator.source_refs.v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Options struct{}
|
||||||
|
type Validator struct{}
|
||||||
|
|
||||||
|
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
|
||||||
|
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
|
||||||
|
|
||||||
|
func New(Options) *Validator { return &Validator{} }
|
||||||
|
func (v *Validator) Name() string { return Key }
|
||||||
|
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||||
|
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
|
||||||
|
if err := npcshape.Validate(req.Value); err != nil {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
issues := make([]string, 0)
|
||||||
|
for npcIndex, npc := range req.Value.NPCs {
|
||||||
|
for refIndex, ref := range npc.SourceRefs {
|
||||||
|
if err := source.ValidateRef(req.Source, ref); err != nil {
|
||||||
|
issues = append(issues, fmt.Sprintf("npcs[%d].source_refs[%d]: %s", npcIndex, refIndex, diagnostics.Truncate(err.Error())))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if len(issues) == 0 {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
return rejection(diagnostics.Aggregate("invalid NPC source references", issues)), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Spec() pipeline.ValidatorSpec {
|
||||||
|
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.ValidatorRegistry) error {
|
||||||
|
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
|
||||||
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(options), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||||
|
|
||||||
|
func rejection(message string) contracts.ValidationResult {
|
||||||
|
return contracts.ValidationResult{Approved: false, ReasonCode: ReasonCode, Message: message}
|
||||||
|
}
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
package sourcerefs
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestValidatorApprovesValidSourceReferences(t *testing.T) {
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validDocument(), validNPCList()))
|
||||||
|
if err != nil || !result.Approved || len(result.Warnings) != 0 {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want approval", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorRejectsInvalidSourceReferences(t *testing.T) {
|
||||||
|
value := validNPCList()
|
||||||
|
value.NPCs[0].SourceRefs = []source.SourceRef{
|
||||||
|
{SourceID: "foreign", StartUnitID: 1, EndUnitID: 1},
|
||||||
|
{SourceID: "session", StartUnitID: 2, EndUnitID: 1},
|
||||||
|
{SourceID: "session", StartUnitID: 99, EndUnitID: 99},
|
||||||
|
}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validDocument(), value))
|
||||||
|
if err != nil || result.Approved || result.ReasonCode != ReasonCode || !strings.Contains(result.Message, "npcs[0].source_refs[0]") {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want source-reference rejection", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorDefersMalformedShape(t *testing.T) {
|
||||||
|
value := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn"}}}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(validDocument(), value))
|
||||||
|
if err != nil || !result.Approved || result.ReasonCode != "" || result.Message != "" {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want shape deferral", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorBoundsDiagnosticsAndHandlesMissingDocument(t *testing.T) {
|
||||||
|
value := validNPCList()
|
||||||
|
value.NPCs[0].SourceRefs = make([]source.SourceRef, 24)
|
||||||
|
for index := range value.NPCs[0].SourceRefs {
|
||||||
|
value.NPCs[0].SourceRefs[index] = source.SourceRef{SourceID: strings.Repeat("火", 220) + "\n\t", StartUnitID: index + 1, EndUnitID: index + 1}
|
||||||
|
}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), requestWithValue(nil, value))
|
||||||
|
if err != nil || result.Approved || len([]byte(result.Message)) > 4096 || !utf8.ValidString(result.Message) {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want bounded missing-document rejection", result, err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(result.Message, "additional issue(s) omitted") || !strings.Contains(result.Message, fmt.Sprintf("npcs[0].source_refs[%d]", 19)) {
|
||||||
|
t.Fatalf("message = %q, want bounded aggregate diagnostics", result.Message)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorSpecCheckpointAndRegistration(t *testing.T) {
|
||||||
|
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != "dnd.npcs.validator.source_refs.v1" {
|
||||||
|
t.Fatalf("CheckpointFingerprints() = %#v, want local policy", got)
|
||||||
|
}
|
||||||
|
if Spec().ExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("Spec() = %#v, want deterministic validator", Spec())
|
||||||
|
}
|
||||||
|
registry := pipeline.NewValidatorRegistry()
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||||
|
t.Fatal("DecodeOptions() accepted unknown option")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func requestWithValue(doc *source.SourceDocument, value dnd.NPCList) contracts.TypedValidationRequest[dnd.NPCList] {
|
||||||
|
return contracts.TypedValidationRequest[dnd.NPCList]{Source: doc, Value: value}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validDocument() *source.SourceDocument {
|
||||||
|
return &source.SourceDocument{ID: "session", Kind: "transcript", Format: "application/json", Digest: "sha256:session", Units: []source.SourceUnit{
|
||||||
|
{ID: 1, Kind: "message", Text: "Mira Thorn enters."},
|
||||||
|
{ID: 2, Kind: "message", Text: "The ranger watches."},
|
||||||
|
}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validNPCList() dnd.NPCList {
|
||||||
|
return dnd.NPCList{NPCs: []dnd.NPC{{ID: "candidate", Name: "Mira Thorn", Aliases: []string{"The Greencloak"}, Description: "A ranger.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 2}}}}}
|
||||||
|
}
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
package sourcerelatedness
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/diagnostics"
|
||||||
|
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Key = "extract/dnd/npcs/source_relatedness"
|
||||||
|
WarningReasonCode = "npc_not_near_source"
|
||||||
|
policy = "dnd.npcs.validator.source_relatedness.v1"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Options struct{}
|
||||||
|
type Validator struct{}
|
||||||
|
|
||||||
|
var _ contracts.TypedValidator[dnd.NPCList] = (*Validator)(nil)
|
||||||
|
var _ pipeline.CheckpointFingerprintProvider = (*Validator)(nil)
|
||||||
|
|
||||||
|
func New(Options) *Validator { return &Validator{} }
|
||||||
|
func (v *Validator) Name() string { return Key }
|
||||||
|
func (v *Validator) ExecutionClass() contracts.ExecutionClass {
|
||||||
|
return contracts.ExecutionClassDeterministic
|
||||||
|
}
|
||||||
|
func (v *Validator) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||||
|
return []pipeline.CheckpointFingerprint{{Name: "policy", Value: policy}}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (v *Validator) Validate(_ context.Context, req contracts.TypedValidationRequest[dnd.NPCList]) (contracts.ValidationResult, error) {
|
||||||
|
if err := npcshape.Validate(req.Value); err != nil {
|
||||||
|
return contracts.ValidationResult{Approved: true}, nil
|
||||||
|
}
|
||||||
|
var warnings []contracts.Warning
|
||||||
|
for npcIndex, npc := range req.Value.NPCs {
|
||||||
|
if npcAppearsInCitedText(req.Source, npc) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
warnings = append(warnings, contracts.Warning{
|
||||||
|
Scope: fmt.Sprintf("npcs[%d]", npcIndex),
|
||||||
|
ReasonCode: WarningReasonCode,
|
||||||
|
Message: fmt.Sprintf("NPC %s was not found in cited source text", diagnostics.Quote(npc.Name)),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{Approved: true, Warnings: warnings}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func npcAppearsInCitedText(doc *source.SourceDocument, npc dnd.NPC) bool {
|
||||||
|
cited := citedTextKey(doc, npc.SourceRefs)
|
||||||
|
if cited == "" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
if strings.Contains(cited, identity.ComparisonKey(npc.Name)) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
for _, alias := range npc.Aliases {
|
||||||
|
if strings.Contains(cited, identity.ComparisonKey(alias)) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func citedTextKey(doc *source.SourceDocument, refs []source.SourceRef) string {
|
||||||
|
if doc == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
var builder strings.Builder
|
||||||
|
for _, ref := range refs {
|
||||||
|
if err := source.ValidateRef(doc, ref); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
start, _ := source.UnitIndex(doc, ref.StartUnitID)
|
||||||
|
end, _ := source.UnitIndex(doc, ref.EndUnitID)
|
||||||
|
for index := start; index <= end; index++ {
|
||||||
|
if builder.Len() > 0 {
|
||||||
|
builder.WriteByte(' ')
|
||||||
|
}
|
||||||
|
builder.WriteString(doc.Units[index].Text)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return identity.ComparisonKey(builder.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
func Spec() pipeline.ValidatorSpec {
|
||||||
|
return pipeline.ValidatorSpec{Key: Key, ExecutionClass: contracts.ExecutionClassDeterministic}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.ValidatorRegistry) error {
|
||||||
|
return pipeline.RegisterTypedValidatorBuilder(registry, dnd.NPCListKind, Spec(), validateOptions, func(request pipeline.BuildRequest) (contracts.TypedValidator[dnd.NPCList], error) {
|
||||||
|
options, err := DecodeOptions(request.Options)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
return New(options), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func DecodeOptions(options map[string]any) (Options, error) {
|
||||||
|
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||||
|
return Options{}, err
|
||||||
|
}
|
||||||
|
return Options{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
package sourcerelatedness
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestValidatorMatchesCanonicalNamesAndAliasesWithUnicodeVariants(t *testing.T) {
|
||||||
|
value := dnd.NPCList{NPCs: []dnd.NPC{
|
||||||
|
{ID: "one", Name: "O'Rin Thorn", Aliases: []string{}, Description: "A ranger.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
|
||||||
|
{ID: "two", Name: "Missing Name", Aliases: []string{"The Greencloak"}, Description: "A guard.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}},
|
||||||
|
}}
|
||||||
|
doc := &source.SourceDocument{ID: "session", Kind: "transcript", Format: "application/json", Digest: "sha256:session", Units: []source.SourceUnit{
|
||||||
|
{ID: 1, Kind: "message", Text: " o’rin\u2003thorn appears."},
|
||||||
|
{ID: 2, Kind: "message", Text: "The greencloak watches."},
|
||||||
|
}}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: doc, Value: value})
|
||||||
|
if err != nil || !result.Approved || len(result.Warnings) != 0 {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want alias/canonical relatedness approval", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorWarnsAtMostOncePerNPCForUnrelatedCitations(t *testing.T) {
|
||||||
|
value := dnd.NPCList{NPCs: []dnd.NPC{
|
||||||
|
{ID: "one", Name: "Missing\nName", Aliases: []string{"Also Missing"}, Description: "A guard.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
|
||||||
|
}}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), Value: value})
|
||||||
|
if err != nil || !result.Approved || len(result.Warnings) != 1 {
|
||||||
|
t.Fatalf("Validate() = %#v, %v; want one warning", result, err)
|
||||||
|
}
|
||||||
|
warning := result.Warnings[0]
|
||||||
|
if warning.Scope != "npcs[0]" || warning.ReasonCode != WarningReasonCode || !strings.Contains(warning.Message, `Missing\nName`) || strings.Contains(warning.Message, "Missing\nName") || !utf8.ValidString(warning.Message) {
|
||||||
|
t.Fatalf("warning = %#v, want safely quoted bounded warning", warning)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorDefersMalformedShapeAndInvalidRangesDoNotPanic(t *testing.T) {
|
||||||
|
invalidShape := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn"}}}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), Value: invalidShape})
|
||||||
|
if err != nil || !result.Approved || len(result.Warnings) != 0 {
|
||||||
|
t.Fatalf("shape deferral = %#v, %v; want approval without warning", result, err)
|
||||||
|
}
|
||||||
|
invalidRange := dnd.NPCList{NPCs: []dnd.NPC{{ID: "one", Name: "Mira Thorn", Aliases: []string{}, Description: "A ranger.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 99, EndUnitID: 99}}}}}
|
||||||
|
result, err = New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), Value: invalidRange})
|
||||||
|
if err != nil || !result.Approved || len(result.Warnings) != 1 || result.Warnings[0].ReasonCode != WarningReasonCode {
|
||||||
|
t.Fatalf("invalid-range relatedness = %#v, %v; want one warning", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorUsesOnlyTranscriptEvidenceAndRegistersPolicy(t *testing.T) {
|
||||||
|
value := dnd.NPCList{NPCs: []dnd.NPC{{ID: "one", Name: "Opaque NPC", Aliases: []string{}, Description: "A guard.", Relationships: []dnd.NPCRelationship{}, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}}}}
|
||||||
|
references := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{"glossary": {Items: []contracts.ReferenceItem{{Content: []byte("Opaque NPC")}}}}}
|
||||||
|
result, err := New(Options{}).Validate(context.Background(), contracts.TypedValidationRequest[dnd.NPCList]{Source: relatednessDocument(), References: references, Value: value})
|
||||||
|
if err != nil || len(result.Warnings) != 1 {
|
||||||
|
t.Fatalf("reference-only relatedness = %#v, %v; want warning", result, err)
|
||||||
|
}
|
||||||
|
if got := New(Options{}).CheckpointFingerprints(); len(got) != 1 || got[0].Name != "policy" || got[0].Value != "dnd.npcs.validator.source_relatedness.v1" {
|
||||||
|
t.Fatalf("CheckpointFingerprints() = %#v, want local policy", got)
|
||||||
|
}
|
||||||
|
if Spec().ExecutionClass != contracts.ExecutionClassDeterministic {
|
||||||
|
t.Fatalf("Spec() = %#v, want deterministic validator", Spec())
|
||||||
|
}
|
||||||
|
registry := pipeline.NewValidatorRegistry()
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||||
|
t.Fatal("DecodeOptions() accepted unknown option")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func relatednessDocument() *source.SourceDocument {
|
||||||
|
return &source.SourceDocument{ID: "session", Kind: "transcript", Format: "application/json", Digest: "sha256:session", Units: []source.SourceUnit{{ID: 1, Kind: "message", Text: "The party waits."}}}
|
||||||
|
}
|
||||||
130
internal/modules/integration/dnd_npc_spell_sequential_test.go
Normal file
130
internal/modules/integration/dnd_npc_spell_sequential_test.go
Normal file
@@ -0,0 +1,130 @@
|
|||||||
|
package integration_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestSequentialNPCOutputCanGroundIndependentSpellRun(t *testing.T) {
|
||||||
|
registries := productionNPCRegistries(t)
|
||||||
|
catalog := moduleCatalog(registries)
|
||||||
|
configValue := loadSequentialPipelineConfig(t)
|
||||||
|
|
||||||
|
npcEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs", Catalog: catalog})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resolve NPC pipeline: %v", err)
|
||||||
|
}
|
||||||
|
npcClient := &fakeNPCProductionLLMClient{response: npcProductionResponse{NPCs: []npcProductionRecord{{
|
||||||
|
Name: "Mira Thorn",
|
||||||
|
Aliases: []string{"The Greencloak"},
|
||||||
|
Description: "A guarded ranger who watches the northern road.",
|
||||||
|
Relationships: []npcProductionRelationship{{
|
||||||
|
Target: "Captain Vale", Relationship: "reports to",
|
||||||
|
}},
|
||||||
|
SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 2}},
|
||||||
|
}}}}
|
||||||
|
npcOutput, err := runPreparedPipeline(t, registries, npcEffective.ResolvedPipeline, npcClient, pipeline.RunInput{RawInput: readNPCFixture(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("run NPC pipeline: %v", err)
|
||||||
|
}
|
||||||
|
if len(npcOutput.NormalizeOutputs) != 1 || npcOutput.NormalizeOutputs[0].LaneID != "npcs" {
|
||||||
|
t.Fatalf("NPC normalized outputs = %#v, want one npcs lane", npcOutput.NormalizeOutputs)
|
||||||
|
}
|
||||||
|
npcPayload := npcOutput.NormalizeOutputs[0].Artifact.Content
|
||||||
|
if _, err := npccodec.New().Decode(npcPayload); err != nil {
|
||||||
|
t.Fatalf("decode normalized NPC payload: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
npcRunDir := t.TempDir()
|
||||||
|
npcPath := filepath.Join(npcRunDir, "lanes", "npcs.json")
|
||||||
|
if err := os.MkdirAll(filepath.Dir(npcPath), 0o700); err != nil {
|
||||||
|
t.Fatalf("create NPC output directory: %v", err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(npcPath, npcPayload, 0o600); err != nil {
|
||||||
|
t.Fatalf("write NPC output payload: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
spellEffective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-spells", Catalog: catalog})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("resolve spell pipeline: %v", err)
|
||||||
|
}
|
||||||
|
spellEffective.ResolvedPipeline.ArtifactLanes[0].ExtractReferences.Bindings = []pipeline.ReferenceBinding{{
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
LaneID: "spells",
|
||||||
|
SlotName: spells.NPCRegistryReferenceSlot,
|
||||||
|
Source: npcPath,
|
||||||
|
BindingSource: contracts.ReferenceBindingSourceCLI,
|
||||||
|
}}
|
||||||
|
materialized, warnings, err := pipeline.MaterializeReferences(spellEffective.ResolvedPipeline, catalog, pipeline.ReferenceMaterializationOptions{WorkingDir: npcRunDir})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("materialize NPC registry reference: %v", err)
|
||||||
|
}
|
||||||
|
if len(warnings) != 0 {
|
||||||
|
t.Fatalf("reference materialization warnings = %#v, want none", warnings)
|
||||||
|
}
|
||||||
|
|
||||||
|
spellClient := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{{
|
||||||
|
Caster: "Mira Thorn",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Restores the injured ally.",
|
||||||
|
NarrativeDescription: "Mira Thorn restores the ally after the fight.",
|
||||||
|
SourceRefs: responseSourceRefs("spell-session", 1, 1),
|
||||||
|
}}}}
|
||||||
|
spellOutput, err := runPreparedPipeline(t, registries, materialized, spellClient, pipeline.RunInput{RawInput: readDNDSpellsFixture(t)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("run spell pipeline: %v", err)
|
||||||
|
}
|
||||||
|
if len(spellOutput.NormalizeOutputs) != 1 || spellOutput.NormalizeOutputs[0].LaneID != "spells" {
|
||||||
|
t.Fatalf("spell normalized outputs = %#v, want one spells lane", spellOutput.NormalizeOutputs)
|
||||||
|
}
|
||||||
|
spellValue := decodeRunnerSpellResponse(t, spellOutput.NormalizeOutputs[0].Artifact.Content)
|
||||||
|
if len(spellValue.SpellCasts) != 1 || spellValue.SpellCasts[0].Caster != "Mira Thorn" {
|
||||||
|
t.Fatalf("spell output = %#v, want one registry-grounded caster", spellValue)
|
||||||
|
}
|
||||||
|
if len(spellValue.SpellCasts[0].SourceRefs) != 1 || spellValue.SpellCasts[0].SourceRefs[0].SourceID != "spell-session" {
|
||||||
|
t.Fatalf("spell source refs = %#v, want current spell session only", spellValue.SpellCasts[0].SourceRefs)
|
||||||
|
}
|
||||||
|
if len(spellClient.requests) != 1 {
|
||||||
|
t.Fatalf("spell LLM requests = %d, want one", len(spellClient.requests))
|
||||||
|
}
|
||||||
|
registryInput := spellClient.requests[0].Inputs[spells.NPCRegistryReferenceSlot]
|
||||||
|
if string(registryInput.Content) != string(npcPayload) || registryInput.MediaType != npccodec.MediaType || registryInput.OriginURI != "" {
|
||||||
|
t.Fatalf("spell NPC prompt input = %#v, want canonical payload without origin", registryInput)
|
||||||
|
}
|
||||||
|
if len(spellOutput.Manifest.References) != 1 {
|
||||||
|
t.Fatalf("spell manifest references = %#v, want one NPC provenance entry", spellOutput.Manifest.References)
|
||||||
|
}
|
||||||
|
provenance := spellOutput.Manifest.References[0]
|
||||||
|
if provenance.Stage != "extract" || provenance.LaneID != "spells" || provenance.SlotName != spells.NPCRegistryReferenceSlot || provenance.BindingSource != contracts.ReferenceBindingSourceCLI || !strings.Contains(provenance.OriginURI, "npcs.json") {
|
||||||
|
t.Fatalf("spell NPC provenance = %#v, want extract CLI reference provenance", provenance)
|
||||||
|
}
|
||||||
|
metadata, ok := spellOutput.Manifest.ArtifactLanes[0].Metadata["extractor"].(map[string]any)
|
||||||
|
if !ok || metadata["npc_count"] != 1 || metadata["npc_registry_digest"] != registryInput.Digest {
|
||||||
|
t.Fatalf("spell extractor metadata = %#v, want NPC count and semantic digest", spellOutput.Manifest.ArtifactLanes[0].Metadata)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadSequentialPipelineConfig(t *testing.T) config.Config {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile("testdata/dnd_npc_spell_sequential_pipeline.yml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read sequential pipeline config: %v", err)
|
||||||
|
}
|
||||||
|
fileConfig, err := config.ParseFileConfigYAML(data)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("parse sequential pipeline config: %v", err)
|
||||||
|
}
|
||||||
|
configValue := config.Default()
|
||||||
|
if err := configValue.ApplyFileConfig(fileConfig); err != nil {
|
||||||
|
t.Fatalf("apply sequential pipeline config: %v", err)
|
||||||
|
}
|
||||||
|
return configValue
|
||||||
|
}
|
||||||
242
internal/modules/integration/dnd_npcs_runner_test.go
Normal file
242
internal/modules/integration/dnd_npcs_runner_test.go
Normal file
@@ -0,0 +1,242 @@
|
|||||||
|
package integration_test
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||||
|
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
|
||||||
|
genericregister "gitea.maximumdirect.net/eric/notarius/internal/modules/generic/register"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
|
||||||
|
seriatimregister "gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/register"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRunnerProcessesSeriatimInputWithProductionDNDNPCPipeline(t *testing.T) {
|
||||||
|
raw := readNPCFixture(t)
|
||||||
|
doc, err := transcript.New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
registries := productionNPCRegistries(t)
|
||||||
|
configValue := loadNPCPipelineConfig(t)
|
||||||
|
effective, err := configValue.Resolve(config.ResolveInput{PipelineID: "dnd-npcs-fixture", Catalog: moduleCatalog(registries)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
client := &fakeNPCProductionLLMClient{response: npcProductionResponse{
|
||||||
|
NPCs: []npcProductionRecord{
|
||||||
|
{
|
||||||
|
Name: "Mira Thorn",
|
||||||
|
Aliases: []string{"The Greencloak"},
|
||||||
|
Description: "The first named NPC encountered.",
|
||||||
|
Relationships: []npcProductionRelationship{
|
||||||
|
{Target: "Hooded Guard", Relationship: "works with"},
|
||||||
|
},
|
||||||
|
SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 1}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "The Greencloak",
|
||||||
|
Aliases: []string{"Mira"},
|
||||||
|
Description: "A later description that must not replace the first.",
|
||||||
|
Relationships: []npcProductionRelationship{
|
||||||
|
{Target: "Hooded Guard", Relationship: "trusts"},
|
||||||
|
},
|
||||||
|
SourceRefs: []npcProductionSourceRef{{StartUnitID: 2, EndUnitID: 2}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Name: "Hooded Guard",
|
||||||
|
Aliases: []string{},
|
||||||
|
Description: "An unnamed but distinguishable sentry.",
|
||||||
|
Relationships: []npcProductionRelationship{
|
||||||
|
{Target: "The Greencloak", Relationship: "reports to"},
|
||||||
|
},
|
||||||
|
SourceRefs: []npcProductionSourceRef{{StartUnitID: 3, EndUnitID: 3}},
|
||||||
|
},
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
output, err := runPreparedPipeline(t, registries, effective.ResolvedPipeline, client, pipeline.RunInput{RawInput: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(output.NormalizeOutputs) != 1 {
|
||||||
|
t.Fatalf("normalize outputs = %d, want one NPC output; rejected=%#v", len(output.NormalizeOutputs), output.Rejected)
|
||||||
|
}
|
||||||
|
serialized := output.NormalizeOutputs[0]
|
||||||
|
if serialized.LaneID != "npcs" || serialized.NormalizerKey != npcs.Key || serialized.Artifact.Schema.ID != npccodec.SchemaID || serialized.Artifact.Schema.Version != npccodec.SchemaVersion {
|
||||||
|
t.Fatalf("serialized output = %#v, want durable NPC lane schema", serialized)
|
||||||
|
}
|
||||||
|
value, err := npccodec.New().Decode(serialized.Artifact.Content)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Decode(output) error = %v, want durable NPC payload", err)
|
||||||
|
}
|
||||||
|
if len(value.NPCs) != 2 {
|
||||||
|
t.Fatalf("NPC output = %#v, want repeated name consolidated and group/PC omitted", value.NPCs)
|
||||||
|
}
|
||||||
|
first, second := value.NPCs[0], value.NPCs[1]
|
||||||
|
if first.Name != "Mira Thorn" || first.Description != "The first named NPC encountered." || !reflect.DeepEqual(first.Aliases, []string{"The Greencloak", "Mira"}) {
|
||||||
|
t.Fatalf("first NPC = %#v, want consolidated Mira identity", first)
|
||||||
|
}
|
||||||
|
if first.ID != identity.DeriveID(first.Name) || second.Name != "Hooded Guard" || second.ID != identity.DeriveID(second.Name) {
|
||||||
|
t.Fatalf("NPC IDs = %q/%q, want derived IDs", first.ID, second.ID)
|
||||||
|
}
|
||||||
|
if first.Relationships[0].Target != "Hooded Guard" || second.Relationships[0].Target != "Mira Thorn" {
|
||||||
|
t.Fatalf("relationship targets = %q/%q, want canonical target rewrite", first.Relationships[0].Target, second.Relationships[0].Target)
|
||||||
|
}
|
||||||
|
for _, npc := range value.NPCs {
|
||||||
|
for _, ref := range npc.SourceRefs {
|
||||||
|
if ref.SourceID != doc.ID {
|
||||||
|
t.Fatalf("NPC source ref = %#v, want source document %q", ref, doc.ID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !hasNPCWarning(output.Warnings, "duplicate_npc_collapsed") || !hasNPCWarning(output.Warnings, "relationship_target_canonicalized") {
|
||||||
|
t.Fatalf("warnings = %#v, want consolidation and target warnings", output.Warnings)
|
||||||
|
}
|
||||||
|
if output.Manifest.ValidationStatus != "approved" || len(output.Manifest.ArtifactLanes) != 1 {
|
||||||
|
t.Fatalf("manifest = %#v, want approved NPC lane", output.Manifest)
|
||||||
|
}
|
||||||
|
lane := output.Manifest.ArtifactLanes[0]
|
||||||
|
if lane.ID != "npcs" || lane.Extractor != npcs.Key || lane.Merger != pipeline.DefaultMergeModule || lane.Normalizer != npcs.Key {
|
||||||
|
t.Fatalf("manifest lane = %#v, want NPC production composition", lane)
|
||||||
|
}
|
||||||
|
normalizerMetadata, ok := lane.Metadata["normalizer"].(map[string]any)
|
||||||
|
if !ok || normalizerMetadata["identity_policy"] != identity.Policy || normalizerMetadata["normalization_policy"] != "dnd.npcs.normalize.v1" {
|
||||||
|
t.Fatalf("normalizer metadata = %#v, want identity and normalization policies", lane.Metadata)
|
||||||
|
}
|
||||||
|
var npcOutputFile *contracts.OutputFile
|
||||||
|
for index := range output.OutputFiles {
|
||||||
|
if output.OutputFiles[index].Name == "lanes/npcs.json" {
|
||||||
|
npcOutputFile = &output.OutputFiles[index]
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if npcOutputFile == nil || npcOutputFile.ContentType != npccodec.MediaType {
|
||||||
|
t.Fatalf("output files = %#v, want JSON NPC lane file", output.OutputFiles)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 1 || client.requests[0].PromptID != npcs.PromptID {
|
||||||
|
t.Fatalf("LLM requests = %#v, want one NPC prompt request", client.requests)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcProductionResponse struct {
|
||||||
|
NPCs []npcProductionRecord `json:"npcs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcProductionRecord struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Aliases []string `json:"aliases"`
|
||||||
|
Description string `json:"description"`
|
||||||
|
Relationships []npcProductionRelationship `json:"relationships"`
|
||||||
|
SourceRefs []npcProductionSourceRef `json:"source_refs"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcProductionRelationship struct {
|
||||||
|
Target string `json:"target"`
|
||||||
|
Relationship string `json:"relationship"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type npcProductionSourceRef struct {
|
||||||
|
StartUnitID int `json:"start_unit_id"`
|
||||||
|
EndUnitID int `json:"end_unit_id"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeNPCProductionLLMClient struct {
|
||||||
|
response npcProductionResponse
|
||||||
|
requests []contracts.StructuredCompletionRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *fakeNPCProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
client.requests = append(client.requests, req)
|
||||||
|
content, err := json.Marshal(client.response)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(content, out); err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate NPC structured target: %w", err)
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func productionNPCRegistries(t *testing.T) pipeline.Registries {
|
||||||
|
t.Helper()
|
||||||
|
registries := pipeline.Registries{
|
||||||
|
Inputs: pipeline.NewInputAdapterRegistry(),
|
||||||
|
Chunkers: pipeline.NewChunkerRegistry(),
|
||||||
|
ArtifactCodecs: pipeline.NewArtifactCodecRegistry(),
|
||||||
|
Extractors: pipeline.NewExtractorRegistry(),
|
||||||
|
Mergers: pipeline.NewMergerRegistry(),
|
||||||
|
Normalizers: pipeline.NewNormalizerRegistry(),
|
||||||
|
Validators: pipeline.NewValidatorRegistry(),
|
||||||
|
ValidatorChains: pipeline.NewValidatorChainRegistry(),
|
||||||
|
Outputs: pipeline.NewOutputEncoderRegistry(),
|
||||||
|
}
|
||||||
|
assets := llm.NewAssetRegistry()
|
||||||
|
for _, registration := range []struct {
|
||||||
|
name string
|
||||||
|
fn func(pipeline.Registries, *llm.AssetRegistry) error
|
||||||
|
}{
|
||||||
|
{name: "generic", fn: genericregister.Register},
|
||||||
|
{name: "seriatim", fn: seriatimregister.Register},
|
||||||
|
{name: "dnd", fn: dndregister.Register},
|
||||||
|
} {
|
||||||
|
if err := registration.fn(registries, assets); err != nil {
|
||||||
|
t.Fatalf("register %s modules: %v", registration.name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return registries
|
||||||
|
}
|
||||||
|
|
||||||
|
func moduleCatalog(registries pipeline.Registries) pipeline.ModuleCatalog {
|
||||||
|
return pipeline.ModuleCatalog{
|
||||||
|
Inputs: registries.Inputs, Chunkers: registries.Chunkers, ArtifactCodecs: registries.ArtifactCodecs,
|
||||||
|
Extractors: registries.Extractors, Mergers: registries.Mergers, Normalizers: registries.Normalizers,
|
||||||
|
Validators: registries.Validators, ValidatorChains: registries.ValidatorChains, Outputs: registries.Outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadNPCPipelineConfig(t *testing.T) config.Config {
|
||||||
|
t.Helper()
|
||||||
|
data, err := os.ReadFile("testdata/dnd_npcs_pipeline.yml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(dnd_npcs_pipeline.yml) error = %v", err)
|
||||||
|
}
|
||||||
|
fileConfig, err := config.ParseFileConfigYAML(data)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v", err)
|
||||||
|
}
|
||||||
|
cfg := config.Default()
|
||||||
|
if err := cfg.ApplyFileConfig(fileConfig); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v", err)
|
||||||
|
}
|
||||||
|
return cfg
|
||||||
|
}
|
||||||
|
|
||||||
|
func readNPCFixture(t *testing.T) []byte {
|
||||||
|
t.Helper()
|
||||||
|
raw, err := os.ReadFile("testdata/seriatim_npc_session.json")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(seriatim_npc_session.json) error = %v", err)
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasNPCWarning(warnings []contracts.Warning, reason string) bool {
|
||||||
|
for _, warning := range warnings {
|
||||||
|
if warning.ReasonCode == reason && strings.HasPrefix(warning.Scope, "npcs[") {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
25
internal/modules/integration/testdata/dnd_npc_spell_sequential_pipeline.yml
vendored
Normal file
25
internal/modules/integration/testdata/dnd_npc_spell_sequential_pipeline.yml
vendored
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
version: 3
|
||||||
|
output:
|
||||||
|
directory: ./notarius-output
|
||||||
|
cache:
|
||||||
|
chunk_plans:
|
||||||
|
mode: bypass
|
||||||
|
checkpoints: {}
|
||||||
|
debug:
|
||||||
|
directory: ./notarius-debug
|
||||||
|
pipelines:
|
||||||
|
dnd-npcs:
|
||||||
|
input: seriatim
|
||||||
|
chunk: generic
|
||||||
|
artifacts:
|
||||||
|
npcs:
|
||||||
|
extract:
|
||||||
|
module: dnd/npcs
|
||||||
|
normalize: dnd/npcs
|
||||||
|
dnd-spells:
|
||||||
|
input: seriatim
|
||||||
|
chunk: generic
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
normalize: dnd/spells
|
||||||
20
internal/modules/integration/testdata/dnd_npcs_pipeline.yml
vendored
Normal file
20
internal/modules/integration/testdata/dnd_npcs_pipeline.yml
vendored
Normal file
@@ -0,0 +1,20 @@
|
|||||||
|
version: 3
|
||||||
|
output:
|
||||||
|
directory: ./notarius-output
|
||||||
|
cache:
|
||||||
|
chunk_plans:
|
||||||
|
mode: bypass
|
||||||
|
checkpoints: {}
|
||||||
|
debug:
|
||||||
|
directory: ./notarius-debug
|
||||||
|
pipelines:
|
||||||
|
dnd-npcs-fixture:
|
||||||
|
input: seriatim
|
||||||
|
chunk: generic
|
||||||
|
artifacts:
|
||||||
|
npcs:
|
||||||
|
extract:
|
||||||
|
module: dnd/npcs
|
||||||
|
retries: 2
|
||||||
|
normalize: dnd/npcs
|
||||||
|
output: json
|
||||||
43
internal/modules/integration/testdata/seriatim_npc_session.json
vendored
Normal file
43
internal/modules/integration/testdata/seriatim_npc_session.json
vendored
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"id": "npc-session",
|
||||||
|
"title": "Synthetic D&D NPC session"
|
||||||
|
},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"start": 0,
|
||||||
|
"end": 4,
|
||||||
|
"speaker": "Aria",
|
||||||
|
"text": "Aria watches Mira Thorn, the Greencloak, enter the ruined hall."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 2,
|
||||||
|
"start": 4,
|
||||||
|
"end": 8,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Mira Thorn asks the party to follow the old road."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 3,
|
||||||
|
"start": 8,
|
||||||
|
"end": 12,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "A hooded guard opens the side gate and waits in silence."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 4,
|
||||||
|
"start": 12,
|
||||||
|
"end": 16,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "Three identical guards surround the interchangeable group."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": 5,
|
||||||
|
"start": 16,
|
||||||
|
"end": 20,
|
||||||
|
"speaker": "Aria",
|
||||||
|
"text": "Aria keeps watch while the named NPCs leave the hall."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user