Compare commits
21 Commits
b6aad4fa98
...
29fcad6e9b
| Author | SHA1 | Date | |
|---|---|---|---|
| 29fcad6e9b | |||
| f5fd115046 | |||
| 7f28899730 | |||
| 84c0758455 | |||
| 5002864e88 | |||
| 55b188fd84 | |||
| d1f43df88e | |||
| d52387c1f7 | |||
| 9c5e3cff14 | |||
| 811d5b8bd9 | |||
| a168c13b85 | |||
| dd61a4efda | |||
| 228cc6ee83 | |||
| 06170e1f65 | |||
| 7715baa1f6 | |||
| 98506db1a9 | |||
| bb4855f0c6 | |||
| c51934d5c6 | |||
| c3513da880 | |||
| c7d853ea52 | |||
| da7fcdaffd |
@@ -2,8 +2,9 @@
|
||||
|
||||
Notarius is a Go CLI for turning source material into structured artifacts with
|
||||
configured extraction pipelines. The implemented D&D workflow reads Seriatim
|
||||
transcript JSON and can produce scene descriptions, item and currency events,
|
||||
NPC identities, combat turns, NPC interactions, enemy events, and spell casts.
|
||||
transcript JSON and can produce location registries and occurrences, scene
|
||||
descriptions, item and currency events, NPC identities, combat turns, NPC
|
||||
interactions, enemy events, and spell casts.
|
||||
|
||||
## Quickstart
|
||||
|
||||
|
||||
@@ -370,14 +370,16 @@ selected target declares them:
|
||||
| **players** | Optional text player context. |
|
||||
| **glossary** | Optional text campaign glossary. |
|
||||
| **spell_catalog** | Optional JSON spell-catalog overlay for spell extraction and normalization. See [spell-catalog overlays](integrations/dnd-spell-catalog-overlays.md). |
|
||||
| **locations** | Required normalized location registry for location-occurrence extraction and normalization. |
|
||||
| **npcs** | Normalized NPC registry. Optional for spells and combat turns; required for NPC interactions and enemy-event extraction and normalization. |
|
||||
| **scene_descriptions** | Required normalized scene-description artifact for combat-turn and enemy-event extraction. |
|
||||
| **combat_turns** | Required normalized combat-turn artifact for enemy-event extraction. |
|
||||
| **npc_interactions** | Required normalized NPC-interaction artifact for enemy-event extraction. |
|
||||
|
||||
Enemy-event artifact slots have the following exact binding contract. Durable
|
||||
event semantics and wire shape remain in the
|
||||
[enemy-event artifact contract](integrations/dnd-enemy-event-artifacts.md).
|
||||
Location-occurrence and enemy-event artifact slots have the following exact
|
||||
binding contracts. Durable semantics and wire shapes remain in their
|
||||
[location-occurrence](integrations/dnd-location-occurrence-artifacts.md) and
|
||||
[enemy-event](integrations/dnd-enemy-event-artifacts.md) contracts.
|
||||
|
||||
| Slot | Accepted artifact kind | Media type | Maximum size | Required stage |
|
||||
| --- | --- | --- | --- | --- |
|
||||
@@ -385,13 +387,16 @@ event semantics and wire shape remain in the
|
||||
| `scene_descriptions` | `dnd/scene-description-list` | `application/json` | 1,048,576 bytes | extract only |
|
||||
| `combat_turns` | `dnd/combat-turn-list` | `application/json` | 1,048,576 bytes | extract only |
|
||||
| `npc_interactions` | `dnd/npc-interaction-list` | `application/json` | 1,048,576 bytes | extract only |
|
||||
| `locations` | `dnd/location-list` | `application/json` | 1,048,576 bytes | location-occurrence extract and normalize |
|
||||
|
||||
Scene descriptions accept **party**, **players**, and **glossary**, but not
|
||||
**roster**. NPC interactions require **npcs** for both extraction and
|
||||
normalization. Combat turns require **scene_descriptions** for extraction; the
|
||||
normalized combat-turn module may use optional **npcs**. Enemy-event extraction
|
||||
requires all four JSON artifact slots; its normalizer requires **npcs**. The
|
||||
complete example shows the ordered generated bindings.
|
||||
normalized combat-turn module may use optional **npcs**. Location occurrences
|
||||
require **locations** for extraction and normalization. Enemy-event extraction
|
||||
requires all four of its JSON artifact slots; its normalizer requires **npcs**.
|
||||
The [complete example](../examples/dnd-complete.config.yml) shows the ordered
|
||||
generated bindings.
|
||||
|
||||
## Production Module Keys
|
||||
|
||||
@@ -399,11 +404,17 @@ complete example shows the ordered generated bindings.
|
||||
| --- | --- |
|
||||
| Input | **seriatim** |
|
||||
| Chunk | **generic**, **dnd/scenes** |
|
||||
| Extract | **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions**, **dnd/enemy-events** |
|
||||
| Extract | **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions**, **dnd/enemy-events**, **dnd/locations**, **dnd/location-occurrences** |
|
||||
| Merge | **appendorder** |
|
||||
| Normalize | **noop**, **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions**, **dnd/enemy-events** |
|
||||
| Normalize | **noop**, **dnd/spells**, **dnd/npcs**, **dnd/combat-turns**, **dnd/item-events**, **dnd/npc-interactions**, **dnd/scene-descriptions**, **dnd/enemy-events**, **dnd/locations**, **dnd/location-occurrences** |
|
||||
| Output | **json** |
|
||||
|
||||
`dnd/locations` extraction and normalization are `llm_backed`; location
|
||||
normalization may use the pipeline's selected LLM profile for bounded duplicate
|
||||
proposals. `dnd/location-occurrences` extraction is `llm_backed`, while its
|
||||
normalizer is `deterministic`. The complete example binds the registry in one
|
||||
step and the occurrence lane in the next.
|
||||
|
||||
The D&D artifact contracts define each emitted schema:
|
||||
[spells](integrations/dnd-spell-artifacts.md),
|
||||
[NPCs](integrations/dnd-npc-artifacts.md),
|
||||
@@ -411,7 +422,9 @@ The D&D artifact contracts define each emitted schema:
|
||||
[combat turns](integrations/dnd-combat-turn-artifacts.md),
|
||||
[item events](integrations/dnd-item-event-artifacts.md),
|
||||
[scene descriptions](integrations/dnd-scene-description-artifacts.md), and
|
||||
[enemy events](integrations/dnd-enemy-event-artifacts.md).
|
||||
[enemy events](integrations/dnd-enemy-event-artifacts.md),
|
||||
[locations](integrations/dnd-location-artifacts.md), and
|
||||
[location occurrences](integrations/dnd-location-occurrence-artifacts.md).
|
||||
|
||||
## Production Validator Keys And Default Chains
|
||||
|
||||
@@ -427,6 +440,8 @@ Available validator keys are:
|
||||
| NPC interactions | **extract/dnd/npc-interactions/shape**, **extract/dnd/npc-interactions/registry**, **extract/dnd/npc-interactions/source_refs**, **extract/dnd/npc-interactions/source_relatedness**, **normalize/dnd/npc-interactions/invariants** |
|
||||
| Scene descriptions | **extract/dnd/scene-descriptions/shape**, **extract/dnd/scene-descriptions/source_refs**, **extract/dnd/scene-descriptions/source_relatedness**, **normalize/dnd/scene-descriptions/invariants** |
|
||||
| Enemy events | **extract/dnd/enemy-events/shape**, **extract/dnd/enemy-events/engagements**, **extract/dnd/enemy-events/source_refs**, **extract/dnd/enemy-events/source_relatedness**, **normalize/dnd/enemy-events/invariants** |
|
||||
| Locations | **extract/dnd/locations/shape**, **extract/dnd/locations/source_refs**, **extract/dnd/locations/source_relatedness**, **normalize/dnd/locations/identity** |
|
||||
| Location occurrences | **extract/dnd/location-occurrences/shape**, **extract/dnd/location-occurrences/registry**, **extract/dnd/location-occurrences/source_refs**, **extract/dnd/location-occurrences/source_relatedness**, **normalize/dnd/location-occurrences/invariants** |
|
||||
|
||||
When no override is configured, production D&D bindings use the following
|
||||
ordered chains. Each row lists extract then normalize; spell chains are the
|
||||
@@ -441,6 +456,8 @@ same at both stages.
|
||||
| NPC interactions | generic/valid_json, extract/dnd/npc-interactions/shape, extract/dnd/npc-interactions/registry, extract/dnd/npc-interactions/source_refs, generic/valid_json_schema, extract/dnd/npc-interactions/source_relatedness | generic/valid_json, extract/dnd/npc-interactions/shape, extract/dnd/npc-interactions/registry, normalize/dnd/npc-interactions/invariants, extract/dnd/npc-interactions/source_refs, generic/valid_json_schema, extract/dnd/npc-interactions/source_relatedness |
|
||||
| Scene descriptions | generic/valid_json, extract/dnd/scene-descriptions/shape, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness | generic/valid_json, extract/dnd/scene-descriptions/shape, normalize/dnd/scene-descriptions/invariants, extract/dnd/scene-descriptions/source_refs, generic/valid_json_schema, extract/dnd/scene-descriptions/source_relatedness |
|
||||
| Enemy events | generic/valid_json, extract/dnd/enemy-events/shape, extract/dnd/enemy-events/engagements, extract/dnd/enemy-events/source_refs, generic/valid_json_schema, extract/dnd/enemy-events/source_relatedness | generic/valid_json, extract/dnd/enemy-events/shape, normalize/dnd/enemy-events/invariants, extract/dnd/enemy-events/source_refs, generic/valid_json_schema, extract/dnd/enemy-events/source_relatedness |
|
||||
| Locations | generic/valid_json, extract/dnd/locations/shape, extract/dnd/locations/source_refs, generic/valid_json_schema, extract/dnd/locations/source_relatedness | generic/valid_json, extract/dnd/locations/shape, normalize/dnd/locations/identity, extract/dnd/locations/source_refs, generic/valid_json_schema, extract/dnd/locations/source_relatedness |
|
||||
| Location occurrences | generic/valid_json, extract/dnd/location-occurrences/shape, extract/dnd/location-occurrences/registry, extract/dnd/location-occurrences/source_refs, generic/valid_json_schema, extract/dnd/location-occurrences/source_relatedness | generic/valid_json, extract/dnd/location-occurrences/shape, extract/dnd/location-occurrences/registry, normalize/dnd/location-occurrences/invariants, extract/dnd/location-occurrences/source_refs, generic/valid_json_schema, extract/dnd/location-occurrences/source_relatedness |
|
||||
|
||||
Chains are only registered for the D&D extract and normalize modules shown
|
||||
above; select an explicit override when a different compatible chain is
|
||||
|
||||
89
docs/integrations/dnd-location-artifacts.md
Normal file
89
docs/integrations/dnd-location-artifacts.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# D&D Location Artifact
|
||||
|
||||
This contract defines the durable, source-grounded location registry produced
|
||||
by `dnd/locations`. It records transcript-established physical places for one
|
||||
source document; it is not a map, location hierarchy, campaign-wide world
|
||||
registry, or location description.
|
||||
|
||||
## Identity and compatibility
|
||||
|
||||
| Property | Value |
|
||||
| --- | --- |
|
||||
| Artifact kind | `dnd/location-list` |
|
||||
| Schema ID | `notarius.dnd.locations` |
|
||||
| Schema name | `notarius_dnd_locations_v1` |
|
||||
| Schema version | `v1` |
|
||||
| Media type | `application/json` |
|
||||
| Identity policy | `dnd.locations.identity.v1` |
|
||||
|
||||
`v1` accepts one strict JSON object with required `locations`; the array may be
|
||||
empty. Location and source-reference objects reject unknown fields. An
|
||||
incompatible artifact shape or identity-policy change uses a new version or
|
||||
policy.
|
||||
|
||||
## Wire shape and identity
|
||||
|
||||
Each location has these required fields:
|
||||
|
||||
| Field | Contract |
|
||||
| --- | --- |
|
||||
| `id` | `location:sha256:` followed by 64 lowercase hexadecimal characters. |
|
||||
| `name` | Non-empty transcript-established display name. |
|
||||
| `source_refs` | One or more transcript evidence ranges that identify the place. |
|
||||
|
||||
A source reference has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
|
||||
The source ID identifies the transcript, unit IDs are positive inclusive unit
|
||||
identifiers, and the start may not follow the end.
|
||||
|
||||
```json
|
||||
{
|
||||
"locations": [
|
||||
{
|
||||
"id": "location:sha256:5c1a91f15729df0b8c257093865fdf2452b43c215375e8cf2341aa9c37bb99aa",
|
||||
"name": "Moon Gate",
|
||||
"source_refs": [
|
||||
{"source_id": "session-7", "start_unit_id": 4, "end_unit_id": 5}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The ID is deterministic and scoped to the source document. Notarius normalizes
|
||||
the display name for comparison with Unicode NFKC, supported apostrophe
|
||||
normalization, collapsed whitespace, and case folding. It hashes compact JSON
|
||||
for this array, using the earliest canonical source reference as the anchor:
|
||||
|
||||
```text
|
||||
["dnd.locations.identity.v1", comparison_name, source_id, start_unit_id, end_unit_id]
|
||||
```
|
||||
|
||||
The canonical ID is the lowercase SHA-256 digest of those bytes with the
|
||||
`location:sha256:` prefix. Equal display names are allowed when their evidence
|
||||
anchors differ, so a generic name does not force distinct places to collapse.
|
||||
|
||||
## Scope, reconciliation, and evidence
|
||||
|
||||
Locations are physical or spatial places established by the transcript, such
|
||||
as planes, regions, settlements, districts, buildings, rooms, landmarks,
|
||||
routes, and geographic features. A generic label is permitted only when it
|
||||
identifies a specific place in the transcript. Notarius does not infer an
|
||||
unstated place or add hierarchy, coordinates, descriptions, participants, or
|
||||
ownership.
|
||||
|
||||
Normalization first applies deterministic display, evidence, and ID rules. It
|
||||
then may use a bounded LLM-assisted proposal to reconcile semantically duplicate
|
||||
records. The proposal is validated and applied conservatively; invalid or
|
||||
unusable proposals retain the deterministic result with retry or fallback
|
||||
diagnostics. The registry's source references establish registry provenance,
|
||||
not evidence for later artifacts.
|
||||
|
||||
## Consumers and publication
|
||||
|
||||
`dnd/location-occurrences` requires one approved location registry through its
|
||||
`locations` reference slot. Its prompt receives an ordered source-free `{id,
|
||||
name}` projection and must not treat registry references as occurrence
|
||||
evidence. See the [location-occurrence artifact](dnd-location-occurrence-artifacts.md)
|
||||
for that contract, [Configuration](../config.md#references-and-ordered-handoffs)
|
||||
for binding rules, and the [JSON output contract](json-output.md) for
|
||||
publication.
|
||||
84
docs/integrations/dnd-location-occurrence-artifacts.md
Normal file
84
docs/integrations/dnd-location-occurrence-artifacts.md
Normal file
@@ -0,0 +1,84 @@
|
||||
# D&D Location-Occurrence Artifact
|
||||
|
||||
This contract defines the durable occurrence list produced by
|
||||
`dnd/location-occurrences`. It records source-grounded ways the party relates
|
||||
to locations in a required normalized location registry; it does not extend
|
||||
that registry or infer a place absent from it.
|
||||
|
||||
## Identity and compatibility
|
||||
|
||||
| Property | Value |
|
||||
| --- | --- |
|
||||
| Artifact kind | `dnd/location-occurrence-list` |
|
||||
| Schema ID | `notarius.dnd.location_occurrences` |
|
||||
| Schema name | `notarius_dnd_location_occurrences_v1` |
|
||||
| Schema version | `v1` |
|
||||
| Media type | `application/json` |
|
||||
|
||||
`v1` accepts one strict JSON object with required `occurrences`; the array may
|
||||
be empty. Occurrence and source-reference objects reject unknown fields. An
|
||||
incompatible shape change requires a new schema version.
|
||||
|
||||
## Wire shape
|
||||
|
||||
Each occurrence has these required fields:
|
||||
|
||||
| Field | Contract |
|
||||
| --- | --- |
|
||||
| `location_id` | Exact ID from the required normalized [location registry](dnd-location-artifacts.md). |
|
||||
| `name` | Exact canonical display name for `location_id` in that registry. |
|
||||
| `kind` | One of `visited`, `planned`, `recalled`, or `mentioned`. |
|
||||
| `source_refs` | One or more current-transcript evidence ranges for this occurrence. |
|
||||
|
||||
A source reference has exactly `source_id`, `start_unit_id`, and `end_unit_id`.
|
||||
It identifies an inclusive range in the current transcript; unit IDs are
|
||||
positive and the start may not follow the end.
|
||||
|
||||
```json
|
||||
{
|
||||
"occurrences": [
|
||||
{
|
||||
"location_id": "location:sha256:5c1a91f15729df0b8c257093865fdf2452b43c215375e8cf2341aa9c37bb99aa",
|
||||
"name": "Moon Gate",
|
||||
"kind": "visited",
|
||||
"source_refs": [
|
||||
{"source_id": "session-7", "start_unit_id": 12, "end_unit_id": 13}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Occurrence categories
|
||||
|
||||
| Kind | Meaning |
|
||||
| --- | --- |
|
||||
| `visited` | The transcript establishes physical party presence, including arrival, continuing presence, or departure. |
|
||||
| `planned` | The party explicitly proposes, intends, or agrees to future travel; speculation alone is not enough. |
|
||||
| `recalled` | The transcript explicitly recounts prior party presence before the current live events. |
|
||||
| `mentioned` | The location is explicit but no stronger category applies, including lore, directions, third-party activity, non-actionable speculation, a mere hypothetical reference, or out-of-character discussion. |
|
||||
|
||||
For overlapping evidence, precedence is `visited`, then `planned`, then
|
||||
`recalled`, then `mentioned`. For example, “What if we went to Moon Gate?” is
|
||||
eligible as `mentioned` when its narrow evidence explicitly references that
|
||||
registry location, but it is not `planned` without an actual proposal,
|
||||
intention, or agreement to travel. Inferred, unstated, uncertain, and
|
||||
unsupported places or occurrences are omitted. Normalization
|
||||
canonicalizes the registry name, orders and deduplicates source references, and
|
||||
orders occurrences by source chronology, location ID, name, kind, and reference
|
||||
sequence. It collapses only exact duplicates with the same ID, kind, and
|
||||
complete canonical evidence sequence.
|
||||
|
||||
## Required grounding and evidence
|
||||
|
||||
Both extraction and normalization require exactly one `locations` reference of
|
||||
kind `dnd/location-list`, media type `application/json`, and at most 1 MiB. The
|
||||
registry provides identity grounding only: unknown IDs and mismatched ID/name
|
||||
pairs are rejected rather than guessed or reassigned. The current transcript is
|
||||
the only evidence source for an occurrence; registry evidence and provenance
|
||||
never become occurrence evidence.
|
||||
|
||||
See [Configuration](../config.md#d-d-reference-slots) for the selectable slot
|
||||
and generated-handoff compatibility, [D&D module internals](../internal/dnd.md)
|
||||
for implementation behavior, and the [JSON output contract](json-output.md)
|
||||
for publication.
|
||||
@@ -55,6 +55,14 @@ with the same canonical identity, retains their earliest position, and merges
|
||||
their canonicalized evidence; it does not add aliases, roles, descriptions, or
|
||||
relationship fields.
|
||||
|
||||
When evidence supports a semantically duplicate group, the canonical display
|
||||
name is one of that group's supplied candidates. A complete, stable proper name
|
||||
is preferred over an abbreviation. An unadorned proper name is preferred over
|
||||
the same name plus a contextual class, role, title, or relationship descriptor
|
||||
unless the transcript establishes that descriptor as part of the person's
|
||||
name. A longer candidate is not preferred solely because it includes such a
|
||||
descriptor.
|
||||
|
||||
## Scope and consumers
|
||||
|
||||
Only individually identifiable NPC names with transcript evidence belong in
|
||||
|
||||
@@ -76,7 +76,9 @@ than infer a lane schema from its name. The current D&D payload contracts are
|
||||
[combat turns](dnd-combat-turn-artifacts.md),
|
||||
[item events](dnd-item-event-artifacts.md),
|
||||
[scene descriptions](dnd-scene-description-artifacts.md), and
|
||||
[enemy events](dnd-enemy-event-artifacts.md).
|
||||
[enemy events](dnd-enemy-event-artifacts.md),
|
||||
[locations](dnd-location-artifacts.md), and
|
||||
[location occurrences](dnd-location-occurrence-artifacts.md).
|
||||
|
||||
## `manifest.json`
|
||||
|
||||
|
||||
@@ -7,7 +7,7 @@ selectable keys, bindings, reference syntax, and default validator chains.
|
||||
|
||||
## Durable Artifact Contracts
|
||||
|
||||
The seven lanes have separate durable wire contracts. This guide deliberately
|
||||
The nine lanes have separate durable wire contracts. This guide deliberately
|
||||
does not repeat their JSON shapes or schemas.
|
||||
|
||||
| Lane | Durable contract |
|
||||
@@ -19,6 +19,8 @@ does not repeat their JSON shapes or schemas.
|
||||
| NPC interactions | [NPC-interaction artifacts](../integrations/dnd-npc-interaction-artifacts.md) |
|
||||
| Scene descriptions | [scene-description artifacts](../integrations/dnd-scene-description-artifacts.md) |
|
||||
| Enemy events | [enemy-event artifacts](../integrations/dnd-enemy-event-artifacts.md) |
|
||||
| Locations | [location artifacts](../integrations/dnd-location-artifacts.md) |
|
||||
| Location occurrences | [location-occurrence artifacts](../integrations/dnd-location-occurrence-artifacts.md) |
|
||||
|
||||
## Family Composition
|
||||
|
||||
@@ -26,9 +28,9 @@ The D&D registrar registers the family’s artifact codecs, extractors, typed
|
||||
append-order mergers, normalizers, validators, prompt assets, fallback LLM
|
||||
profile asset, and default validator chains. Each extractor and normalizer has
|
||||
a stable module spec, explicit execution class, strict option decoding, and a
|
||||
typed builder. Scene chunking, every extractor, and NPC normalization are
|
||||
registered as `llm_backed`; the remaining current D&D mergers and normalizers
|
||||
are `deterministic`. The metadata is available to catalog inspection and
|
||||
typed builder. Scene chunking, every extractor, NPC normalization, and location
|
||||
normalization are registered as `llm_backed`; the remaining current D&D mergers
|
||||
and normalizers are `deterministic`. The metadata is available to catalog inspection and
|
||||
resolved-pipeline debug data and determines which selected bindings inherit the
|
||||
pipeline profile. Configuration remains the canonical owner of the exact keys,
|
||||
profile precedence, and validator order.
|
||||
@@ -42,10 +44,10 @@ the contracts above define durable data.
|
||||
## Prompt Construction
|
||||
|
||||
D&D extractors assemble prompts from an ordered manifest of shared and
|
||||
module-owned assets. Reuse the shared D&D system, evidence, identity,
|
||||
reference, and transcript assets instead of copying their text into individual
|
||||
modules. A manifest’s declared sequence, including cache-control placement, is
|
||||
part of the prompt behavior.
|
||||
module-owned assets. The location extractor and occurrence extractor reuse the
|
||||
shared D&D system, evidence, identity, reference, and transcript assets instead
|
||||
of copying their text into individual modules. A manifest’s declared sequence,
|
||||
including cache-control placement, is part of the prompt behavior.
|
||||
|
||||
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
|
||||
profile. The D&D registrar embeds that fallback profile with the maintained
|
||||
@@ -72,7 +74,9 @@ remains stable.
|
||||
The other D&D LLM prompts intentionally follow different patterns. Scene
|
||||
chunking has no sibling extraction lane with which to share its full transcript,
|
||||
so it renders campaign references before its task and instructions, then places
|
||||
the cacheable full transcript last. NPC normalization keeps its task and
|
||||
the cacheable full transcript last. NPC and location normalization share the
|
||||
entity-reconciliation response schema and safety boundary while retaining their
|
||||
own task and identity rules. NPC normalization keeps its task and
|
||||
cacheable instructions before the candidate collection, followed by the
|
||||
cacheable transcript windows: candidates must be available before their
|
||||
supporting evidence is evaluated, and those windows are not a cross-lane
|
||||
@@ -108,13 +112,14 @@ combine results from distinct scenes, so it intentionally does not apply that
|
||||
rule. Configuration owns the exact validator key and chain position.
|
||||
|
||||
Normalizers are deterministic for spells, combat turns, item events, NPC
|
||||
interactions, scene descriptions, and enemy events. They canonicalize display
|
||||
interactions, scene descriptions, enemy events, and location occurrences. They canonicalize display
|
||||
values and evidence, use source-document order for stable output, and issue
|
||||
bounded warnings for changes or collapsed duplicates. The NPC normalizer is
|
||||
the intentional exception: it first produces a deterministic candidate set,
|
||||
then uses a bounded structured-LLM proposal to reconcile identity groups.
|
||||
Invalid or unusable proposals retain the deterministic result and surface retry
|
||||
or fallback diagnostics; the model does not directly replace durable records.
|
||||
bounded warnings for changes or collapsed duplicates. The NPC and location
|
||||
normalizers are intentional exceptions: each first produces a deterministic
|
||||
candidate set, then may use a bounded structured-LLM proposal to reconcile
|
||||
identity groups. Invalid or unusable proposals retain the deterministic result
|
||||
and surface retry or fallback diagnostics; the model does not directly replace
|
||||
durable records.
|
||||
|
||||
## Generated References And Grounding
|
||||
|
||||
@@ -130,8 +135,11 @@ they do not supply evidence. Scene-description registries are eligibility-only
|
||||
projections: they retain the current chunk’s classification data, not scene
|
||||
prose or evidence, and exist to route combat extraction. Enemy-event extraction
|
||||
also projects combat turns to `actor` and `turn_kind` and filters NPC
|
||||
interactions to `combat_opponent` names and kinds. These compact projections,
|
||||
like NPC grounding, are source-free guidance and never event evidence.
|
||||
interactions to `combat_opponent` names and kinds. Location registries project
|
||||
ordered `{id, name}` pairs to location-occurrence extraction and normalization;
|
||||
exact ID/name matching keeps same-name locations distinguishable. These compact
|
||||
projections, like NPC grounding, are source-free guidance and never event
|
||||
evidence.
|
||||
|
||||
## Lane-Specific Rules
|
||||
|
||||
@@ -147,6 +155,8 @@ shared helper changes.
|
||||
| NPC interactions | Requires the normalized NPC registry at extraction and normalization, using it for canonical actor grounding only. |
|
||||
| Scene descriptions | Produces the classifications consumed by combat routing; it does not consume an NPC registry or provide evidence for combat artifacts. |
|
||||
| Enemy events | Requires NPC, scene-description, combat-turn, and NPC-interaction artifacts. It calls the LLM only for an exact `combat` classification, records ordered observations rather than terminal state, and normalizes recognized names through the NPC registry while preserving grounded collective labels. |
|
||||
| Locations | Produces a source-anchored, session-scoped registry. Its LLM-assisted reconciliation is proposal-only and never collapses same-name places without validated identity and evidence rules. |
|
||||
| Location occurrences | Requires the normalized location registry for both extraction and normalization. Its [durable occurrence categories](../integrations/dnd-location-occurrence-artifacts.md#occurrence-categories) distinguish explicit speculation from unsupported inference; the deterministic normalizer enforces exact registry grounding and never turns registry provenance into occurrence evidence. |
|
||||
|
||||
The combat and scene-description contracts describe their exact handoff and
|
||||
empty-result behavior in more detail:
|
||||
|
||||
@@ -7,19 +7,6 @@ not as committed release dates.
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
### Location Extraction
|
||||
|
||||
- Add a D&D artifact for locations visited by the party or otherwise mentioned
|
||||
in the transcript.
|
||||
- Distinguish observed visits from references, plans, recalled places, and
|
||||
uncertain or inferred locations so a mention alone is not reported as a
|
||||
visit.
|
||||
- Preserve transcript evidence for each visit or mention and reconcile aliases,
|
||||
nested places, and repeated appearances without collapsing distinct
|
||||
locations that share a generic name.
|
||||
- Define how the location artifact should ground later narrative reports and
|
||||
whether future event artifacts should retain canonical location identities.
|
||||
|
||||
### Evaluate Spell Extraction And Normalization
|
||||
|
||||
- Evaluate ordinary extraction retries and the completed normalization path
|
||||
|
||||
@@ -1,649 +0,0 @@
|
||||
# D&D Location Tracking Implementation Plan
|
||||
|
||||
## Objective
|
||||
|
||||
Implement the target state in [D&D Location Tracking](location.md): an
|
||||
evidence-grounded `dnd/locations` registry lane and a dependent
|
||||
`dnd/location-occurrences` lane, including conservative location identity,
|
||||
shared D&D entity-reconciliation infrastructure, production validation,
|
||||
generated-reference wiring, maintained examples, and current documentation.
|
||||
|
||||
This is an ordered implementation plan for a `gpt-5.6-terra` coding agent.
|
||||
Implement one stage per prompt, in order. Finish each stage's tests and leave
|
||||
the repository coherent before proceeding. Do not implement later-stage
|
||||
production registrations early merely to make an incomplete feature selectable.
|
||||
|
||||
All stages must follow:
|
||||
|
||||
- [Architecture Policy](../policy/architecture.md)
|
||||
- [Testing Policy](../policy/testing.md)
|
||||
- [Documentation Policy](../policy/documentation.md)
|
||||
- the D&D conventions in [D&D Module Internals](../internal/dnd.md)
|
||||
- the durable policy decisions in [the feature roadmap](location.md)
|
||||
|
||||
Use behavior-level tests. Do not add tests that merely freeze source layout,
|
||||
exact prompt wording, message counts, shared-prefix length, or other incidental
|
||||
implementation details. Keep tests deterministic, offline, and owned by the
|
||||
component whose behavior they exercise.
|
||||
|
||||
## Stage 1: Define Location Domain Types And Identity
|
||||
|
||||
### Goal
|
||||
|
||||
Establish the in-process contracts and deterministic identity policy on which
|
||||
both lanes depend.
|
||||
|
||||
### Work
|
||||
|
||||
- Extend `internal/modules/dnd/types.go` with:
|
||||
- `LocationListKind` = `dnd/location-list`;
|
||||
- `LocationOccurrenceListKind` = `dnd/location-occurrence-list`;
|
||||
- `LocationList`, `Location`, `LocationOccurrenceList`,
|
||||
`LocationOccurrence`, and `LocationOccurrenceKind`;
|
||||
- exact JSON members and the four occurrence constants specified in
|
||||
`location.md`.
|
||||
- Add `internal/modules/dnd/locations/identity`.
|
||||
- Implement display normalization and comparison normalization consistently
|
||||
with the existing NPC identity policy. Share a lower-level comparison helper
|
||||
only if doing so preserves NPC behavior exactly; otherwise keep the small
|
||||
policy-specific function explicit.
|
||||
- Implement the versioned compact-JSON-array ID derivation contract from
|
||||
`location.md`, including ID syntax checks and immutable list validation.
|
||||
- Identity validation must require correctly derived, unique IDs while allowing
|
||||
two records to have the same comparison name when their evidence anchors
|
||||
differ.
|
||||
- Add focused tests for Unicode normalization, whitespace, apostrophes,
|
||||
deterministic encoding, evidence ordering, same-name/different-anchor IDs,
|
||||
malformed IDs, missing evidence, and non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The types compile without production registration.
|
||||
- ID derivation exactly follows the documented five-element compact JSON input.
|
||||
- Same normalized name plus different earliest evidence yields different IDs.
|
||||
- Validation does not reject same-name records solely because their names
|
||||
match, and it reports duplicate or mismatched IDs deterministically.
|
||||
- `go test ./internal/modules/dnd/locations/... ./internal/modules/dnd/...` passes
|
||||
for the packages available at this stage.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Small enough for one implementation prompt.
|
||||
|
||||
## Stage 2: Add Durable Codecs And Schemas
|
||||
|
||||
### Goal
|
||||
|
||||
Create strict durable JSON ownership for both artifact kinds without exposing
|
||||
either lane as a selectable pipeline yet.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/codec/locations` and
|
||||
`internal/modules/dnd/codec/locationoccurrences`, following the existing D&D
|
||||
candidate/approved codec pattern.
|
||||
- Add embedded Draft 2020-12 schemas with the IDs, names, root members, required
|
||||
fields, enums, source-reference shape, and `additionalProperties: false`
|
||||
contracts in `location.md`.
|
||||
- Keep both schemas at `v1`.
|
||||
- Support strict candidate decoding before semantic approval and strict durable
|
||||
encoding/decoding after approval.
|
||||
- Add representative valid fixtures and tests for schema metadata, defensive
|
||||
schema bytes, empty arrays, unknown fields, missing fields, invalid types,
|
||||
invalid enum values, malformed source references, invalid ID syntax, and
|
||||
round trips.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Each codec advertises the correct artifact kind and metadata count.
|
||||
- Candidate decoding preserves semantic mistakes for validators while
|
||||
rejecting structurally invalid JSON.
|
||||
- Approved encoding and decoding enforce the durable shape.
|
||||
- `go test ./internal/modules/dnd/codec/locations/... ./internal/modules/dnd/codec/locationoccurrences/...`
|
||||
passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Small enough for one implementation prompt.
|
||||
|
||||
## Stage 3: Extract Shared Entity-Reconciliation Infrastructure
|
||||
|
||||
### Goal
|
||||
|
||||
Create the D&D-shared, domain-safe proposal machinery needed by both NPC and
|
||||
location normalization, without changing NPC production behavior yet.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/shared/entityreconcile`.
|
||||
- Move or generalize the reusable behavior currently owned by
|
||||
`internal/modules/dnd/normalize/npcs/context_material.go` and `proposal.go`:
|
||||
- assign deterministic opaque candidate keys such as `candidate-000001` in
|
||||
input order;
|
||||
- clone candidate names and source references;
|
||||
- build bounded transcript windows in source-document order;
|
||||
- omit candidates whose references cannot safely produce context;
|
||||
- coalesce overlapping or adjacent windows without mutating the source;
|
||||
- define the private `duplicate_groups` proposal with `members` and
|
||||
`canonical` candidate keys;
|
||||
- reject blank, unknown, repeated, ineligible, overlapping, too-small, or
|
||||
canonical-not-a-member groups; and
|
||||
- return defensive, immutable assessment data identifying only safe groups.
|
||||
- Keep LLM calls, retry decisions, artifact mutation, canonical-name policy,
|
||||
durable ID derivation, and warning wording out of this package.
|
||||
- Add a shared prompt instruction asset that states the key-copying and
|
||||
proposal-safety contract without NPC- or location-specific identity rules.
|
||||
- Add a shared private structured-response schema and loader/registration
|
||||
support with a stable `v1` key, ID, name, and fingerprint. Registering the
|
||||
schema more than once must not be required.
|
||||
- Add table-driven tests for context bounds, ordering, invalid references,
|
||||
coalescing, every unsafe proposal category, non-overlapping safe groups,
|
||||
deterministic keys, defensive copies, and non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The package has no dependency on `dnd.NPC`, `dnd.Location`, either
|
||||
normalizer, or a concrete LLM client.
|
||||
- Proposal values can identify duplicate candidates even when display names
|
||||
are equal.
|
||||
- The shared response contract cannot directly supply replacement records or
|
||||
evidence.
|
||||
- Existing NPC packages still compile before their migration.
|
||||
- `go test ./internal/modules/dnd/shared/...` passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium, but coherent and suitable for one implementation prompt. Do not combine
|
||||
it with the NPC migration.
|
||||
|
||||
## Stage 4: Migrate NPC Normalization To The Shared Helper
|
||||
|
||||
### Goal
|
||||
|
||||
Make the existing NPC normalizer the first production consumer of the shared
|
||||
entity-reconciliation contract while retaining its durable behavior.
|
||||
|
||||
### Work
|
||||
|
||||
- Refactor `internal/modules/dnd/normalize/npcs` to use opaque candidate keys,
|
||||
shared context construction, shared proposal assessment, the shared response
|
||||
schema, and the shared generic reconciliation instruction asset.
|
||||
- Retain NPC-owned responsibilities:
|
||||
- comparison-name preparation and deterministic duplicate handling;
|
||||
- the NPC-specific task and canonical display-name rules;
|
||||
- LLM invocation, bounded retry, fallback, warnings, and diagnostics;
|
||||
- application of safe groups, evidence union, NPC ID derivation, and output
|
||||
ordering.
|
||||
- Remove superseded NPC-private context/proposal code and private schema assets
|
||||
once no longer referenced.
|
||||
- Update NPC prompt metadata and checkpoint fingerprints for the intentional
|
||||
prompt/private-schema contract change.
|
||||
- Preserve public module keys, durable NPC schema, identity policy, validator
|
||||
chains, warning bounds, and fallback semantics.
|
||||
- Test equal display names as distinct keyed candidates, alias consolidation,
|
||||
rejected unsafe groups, retry exhaustion, private input ownership, redacted
|
||||
errors, deterministic fallback, and non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- No durable NPC artifact field or module key changes.
|
||||
- NPC normalization cannot confuse two candidates merely because their display
|
||||
names match.
|
||||
- Unsafe proposals leave a valid deterministic result and follow existing
|
||||
retry/fallback policy.
|
||||
- Obsolete NPC-only reconciliation helpers and schema are removed.
|
||||
- `go test ./internal/modules/dnd/normalize/npcs/... ./internal/modules/dnd/shared/...`
|
||||
passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium-to-large but bounded to one existing module. Suitable for one
|
||||
implementation prompt; do not add location normalization in this stage.
|
||||
|
||||
## Stage 5: Implement Location Extraction
|
||||
|
||||
### Goal
|
||||
|
||||
Add the LLM-backed extractor that produces evidence-grounded location
|
||||
candidates.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/extract/locations` following current D&D extractor
|
||||
conventions: strict empty options, typed builder and registration function,
|
||||
`llm_backed` execution metadata, immutable inputs, redacted errors, prompt
|
||||
and response-schema fingerprints, and bounded diagnostics.
|
||||
- Add a private response schema containing only `name` and source ranges; the
|
||||
model must not produce durable IDs or prose.
|
||||
- Compose the prompt from existing shared D&D system, identity, campaign
|
||||
reference, transcript, and evidence assets plus module-owned task and
|
||||
instructions. Preserve the documented extraction-message ordering and cache
|
||||
controls.
|
||||
- Define physical-place inclusion and conservative omission exactly as in
|
||||
`location.md`, including generic labels, aliases, and nested places.
|
||||
- Map source ranges to the current source ID, canonicalize exact duplicate
|
||||
ranges, derive candidate location IDs in code, preserve semantically invalid
|
||||
candidates for validators where safe, and return deterministic ordering.
|
||||
- Add prompt-asset tests that verify shared asset reuse and rendered inputs by
|
||||
behavior, without asserting exact shared-prefix length or prompt wording.
|
||||
- Add extractor tests for empty output, mapping, evidence ownership, generic
|
||||
same-name locations with different anchors, invalid candidate preservation,
|
||||
client failures, registration metadata, fingerprints, and non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The extractor cannot manufacture source identities or accept campaign
|
||||
references as evidence.
|
||||
- The private model response does not contain a durable ID.
|
||||
- Same-name candidates with different evidence survive extraction as distinct
|
||||
candidates.
|
||||
- The package is testable through its local registration but is not yet added
|
||||
to the production D&D registrar.
|
||||
- `go test ./internal/modules/dnd/extract/locations/...` passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium and suitable for one implementation prompt.
|
||||
|
||||
## Stage 6: Implement Semantic Location Normalization
|
||||
|
||||
### Goal
|
||||
|
||||
Add conservative alias and repeated-place reconciliation without collapsing
|
||||
same-named or nested locations by default.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/normalize/locations` as an `llm_backed` normalizer
|
||||
using the shared entity-reconciliation package and private response schema.
|
||||
- Deterministically clone and prepare the merged candidates first:
|
||||
- normalize display whitespace;
|
||||
- canonicalize and deduplicate source references;
|
||||
- remove only exact duplicates with the same comparison name and exact
|
||||
canonical evidence;
|
||||
- assign opaque reconciliation keys; and
|
||||
- retain same-name records with different evidence.
|
||||
- Use bounded transcript windows and a module-owned location task that permits
|
||||
grouping only when evidence clearly identifies one physical place. Explicitly
|
||||
prohibit grouping solely by equal names, proximity, nesting, or generic
|
||||
labels.
|
||||
- Validate proposals through the shared package. Apply only safe groups in
|
||||
deterministic code, choose the canonical name from the selected existing
|
||||
candidate, union evidence, and derive the final evidence-anchored ID.
|
||||
- Retain the deterministic candidate set on unusable proposals and follow the
|
||||
existing NPC retry/fallback and bounded-warning conventions.
|
||||
- Publish prompt, response-schema, identity-policy, normalization-policy, and
|
||||
semantic-context fingerprints.
|
||||
- Test aliases, repeated appearances, same-name distinct places, parent/child
|
||||
locations, invalid and overlapping proposals, proposal retries, fallback,
|
||||
ordering, ID recomputation, warning bounds, idempotent deterministic
|
||||
application, and non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The model proposes groups but cannot directly replace durable locations.
|
||||
- A failed or ambiguous proposal cannot lose a valid candidate.
|
||||
- Same-name locations remain distinct unless an approved evidence-backed group
|
||||
joins them.
|
||||
- Final IDs are derived only after group evidence is unioned.
|
||||
- `go test ./internal/modules/dnd/normalize/locations/...` passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium-to-large but scoped to one normalizer and suitable for one implementation
|
||||
prompt.
|
||||
|
||||
## Stage 7: Add The Immutable Location Registry
|
||||
|
||||
### Goal
|
||||
|
||||
Provide safe generated-reference resolution and an unambiguous prompt
|
||||
projection for downstream occurrence extraction.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/locations/registry`, modeled on the immutable NPC
|
||||
registry and its operation-time resolver.
|
||||
- Define `ReferenceSlot = "locations"`, a 1,048,576-byte limit, and exactly one
|
||||
accepted `application/json` location-list item when bound.
|
||||
- Validate durable decoding and location identity before constructing a
|
||||
registry.
|
||||
- Store canonical durable bytes and semantic digests without retaining mutable
|
||||
caller-owned content. Return defensive copies from all accessors.
|
||||
- Produce a compact, source-free prompt projection containing ordered
|
||||
`{id, name}` pairs. Do not include source references or generated-reference
|
||||
provenance.
|
||||
- Support exact lookup by ID and verify the matching canonical name; do not
|
||||
provide an ambiguous name-only lookup as the occurrence linkage mechanism.
|
||||
- Preserve the established seeded/operation resolver behavior and concurrency-
|
||||
safe semantic caching.
|
||||
- Test absent, empty, malformed, oversized, wrong-media-type, invalid-identity,
|
||||
and valid registries; projections; ID lookup; defensive copies; raw and
|
||||
semantic cache reuse; and concurrent resolution.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Distinct same-name records are both representable and addressable by ID.
|
||||
- Registry evidence cannot appear in the prompt projection.
|
||||
- Malformed static references fail during construction and malformed generated
|
||||
references fail at operation resolution through existing boundaries.
|
||||
- `go test ./internal/modules/dnd/locations/registry/...` passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium and suitable for one implementation prompt.
|
||||
|
||||
## Stage 8: Implement Location-Occurrence Extraction
|
||||
|
||||
### Goal
|
||||
|
||||
Add the dependent LLM-backed lane that classifies source-grounded location
|
||||
occurrences.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/extract/locationoccurrences` with module key
|
||||
`dnd/location-occurrences`, `llm_backed` execution metadata, strict empty
|
||||
options, typed construction, and a required `locations` reference slot.
|
||||
- Resolve the immutable registry at construction and for each operation using
|
||||
the established generated-reference pattern.
|
||||
- Add a private response schema requiring `location_id`, `name`, `kind`, and
|
||||
source ranges. Restrict kinds to `visited`, `planned`, `recalled`, and
|
||||
`mentioned`.
|
||||
- Reuse the shared D&D extraction prompt assets and ordering. Place the compact
|
||||
location registry after the shared transcript/evidence material and before
|
||||
module task/instructions, consistent with current generated grounding.
|
||||
- Encode the exact classification rules, precedence, multi-fact behavior, and
|
||||
conservative omission policy from `location.md`.
|
||||
- Map evidence only to the current source. Copy candidate IDs and names without
|
||||
silently repairing unknown or mismatched values so deterministic validators
|
||||
retain ownership of those diagnostics.
|
||||
- Canonically order output and exact duplicates without dropping distinct
|
||||
kinds or independent evidence.
|
||||
- Test every kind, precedence, multiple supported facts, no-location and no-
|
||||
occurrence outputs, required registry failures, same-name ID selection,
|
||||
source-free prompt projection, current-transcript evidence, prompt/profile
|
||||
metadata, client failures, non-mutation, and local registration.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Construction and operation specs declare `locations` as required and accept
|
||||
only `dnd/location-list` JSON.
|
||||
- The model sees IDs and names but no registry evidence.
|
||||
- Registry context never becomes occurrence evidence.
|
||||
- The extractor remains locally testable but is not production-selectable yet.
|
||||
- `go test ./internal/modules/dnd/extract/locationoccurrences/...` passes
|
||||
offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium and suitable for one implementation prompt.
|
||||
|
||||
## Stage 9: Implement Deterministic Occurrence Normalization
|
||||
|
||||
### Goal
|
||||
|
||||
Canonicalize occurrence records against the exact location registry without a
|
||||
second LLM call.
|
||||
|
||||
### Work
|
||||
|
||||
- Add `internal/modules/dnd/normalize/locationoccurrences` as a deterministic
|
||||
normalizer with the same required `locations` slot.
|
||||
- Clone all inputs. Normalize source ranges, names, ordering, and exact
|
||||
duplicates.
|
||||
- For a known `location_id`, replace display-name variation with the registry's
|
||||
exact canonical name. Do not perform a name-only guess.
|
||||
- Preserve an unknown ID or otherwise invalid record for validator diagnostics
|
||||
and emit bounded warnings where current D&D normalizer conventions require
|
||||
them.
|
||||
- Sort using the complete order defined in `location.md`.
|
||||
- Publish normalization and registry-projection fingerprints consistent with
|
||||
the other registry-backed normalizers.
|
||||
- Test all kind values, canonical name replacement, same-name distinct IDs,
|
||||
exact-duplicate removal, distinct evidence retention, stable ordering,
|
||||
unknown IDs, malformed registry resolution, warnings, idempotence, and
|
||||
non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- No LLM client or prompt assets are required.
|
||||
- Canonicalization is exclusively ID-based.
|
||||
- Invalid records are not silently redirected to a different location.
|
||||
- `go test ./internal/modules/dnd/normalize/locationoccurrences/...` passes
|
||||
offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Small enough for one implementation prompt.
|
||||
|
||||
## Stage 10: Add Location Registry Validators
|
||||
|
||||
### Goal
|
||||
|
||||
Give `dnd/location-list` the complete validator ownership expected of a
|
||||
production D&D artifact.
|
||||
|
||||
### Work
|
||||
|
||||
- Add location validator packages under
|
||||
`internal/modules/dnd/validate/locations` for:
|
||||
- extraction shape;
|
||||
- source-reference bounds/current-chunk ownership;
|
||||
- advisory source relatedness; and
|
||||
- normalized identity derivation and ID uniqueness.
|
||||
- Use shared D&D citation, unit-reference, diagnostic, and matching helpers
|
||||
where their contracts apply.
|
||||
- The identity validator must allow repeated comparison names and validate the
|
||||
evidence-anchored derivation policy instead of importing NPC uniqueness
|
||||
assumptions.
|
||||
- Keep diagnostics indexed, aggregated, bounded, stable, and free of raw prompt
|
||||
or reference content.
|
||||
- Add tests for accepted values, every owned failure, same-name distinct
|
||||
locations, malformed/unreadable citations, advisory relatedness, bounds,
|
||||
registration metadata, fingerprints, nil safety where applicable, and
|
||||
non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Validator responsibilities do not overlap merely to increase test coverage.
|
||||
- Relatedness remains advisory and uses only cited current-transcript text.
|
||||
- Validators do not repair or mutate artifacts.
|
||||
- `go test ./internal/modules/dnd/validate/locations/...` passes offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium and suitable for one implementation prompt.
|
||||
|
||||
## Stage 11: Add Location-Occurrence Validators
|
||||
|
||||
### Goal
|
||||
|
||||
Give `dnd/location-occurrence-list` complete structural, registry, ordering,
|
||||
evidence, and advisory validation.
|
||||
|
||||
### Work
|
||||
|
||||
- Add validator packages under
|
||||
`internal/modules/dnd/validate/locationoccurrences` for:
|
||||
- extraction shape and supported kinds;
|
||||
- required registry membership and exact `location_id`/`name` pairing;
|
||||
- normalized ordering and exact-duplicate invariants;
|
||||
- source-reference bounds/current-chunk ownership; and
|
||||
- advisory source relatedness.
|
||||
- Reuse the immutable location resolver rather than decoding caller-owned
|
||||
references independently in each validator.
|
||||
- Ensure same-name registry records remain distinguishable by ID.
|
||||
- Keep registry context out of evidence checks.
|
||||
- Add focused tests for each kind, unknown IDs, mismatched names, same-name
|
||||
locations, ordering, duplicates, malformed required references, invalid
|
||||
evidence, advisory diagnostics, registration metadata, fingerprints,
|
||||
diagnostic bounds, and non-mutation.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- An ID/name mismatch is rejected even when another registry record has the
|
||||
supplied name.
|
||||
- Missing or malformed required registry references fail at the established
|
||||
boundary.
|
||||
- Validators remain deterministic and do not alter occurrence records or the
|
||||
registry.
|
||||
- `go test ./internal/modules/dnd/validate/locationoccurrences/...` passes
|
||||
offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium and suitable for one implementation prompt.
|
||||
|
||||
## Stage 12: Compose The Production D&D Family
|
||||
|
||||
### Goal
|
||||
|
||||
Make both lanes selectable as one coherent production addition after all
|
||||
component contracts are present.
|
||||
|
||||
### Work
|
||||
|
||||
- Extend `internal/modules/dnd/register` to register, in dependency-safe order:
|
||||
- both codecs;
|
||||
- both extractors;
|
||||
- typed append-order mergers;
|
||||
- the LLM-backed location normalizer;
|
||||
- the deterministic occurrence normalizer;
|
||||
- all validators;
|
||||
- shared reconciliation schema assets and both new prompt manifests;
|
||||
- evidence projectors; and
|
||||
- extract and normalize default validator chains.
|
||||
- Ensure both LLM-backed modules select the maintained `dnd-extraction`
|
||||
fallback profile and inherit the existing profile policy.
|
||||
- Define chain order consistently with existing D&D artifacts: generic JSON,
|
||||
shape, registry/identity or normalized invariants at the appropriate stage,
|
||||
source references, durable JSON Schema, then advisory relatedness.
|
||||
- Update registrar tests for artifact kinds, keys, execution classes, reference
|
||||
slot requirements, builder construction, assets, profile use, evidence
|
||||
projection, chain contents/order, duplicate registration, and failure
|
||||
propagation.
|
||||
- Update any integration-level artifact-kind allowlists or typed registries
|
||||
required by the framework; do not add module-specific orchestration logic.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- One D&D registration call exposes both complete lanes and no partial
|
||||
registration succeeds silently.
|
||||
- Catalog inspection reports correct artifact kinds, stages, execution classes,
|
||||
reference slots, profiles, and fingerprints.
|
||||
- Both artifact kinds support evidence projection without registry evidence
|
||||
leakage.
|
||||
- `go test ./internal/modules/dnd/register/... ./internal/modules/dnd/...` passes
|
||||
offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium-to-large but limited to composition and suitable for one implementation
|
||||
prompt.
|
||||
|
||||
## Stage 13: Add Maintained Pipeline And Handoff Coverage
|
||||
|
||||
### Goal
|
||||
|
||||
Exercise the feature through real configuration, ordered generated references,
|
||||
chunk operations, acceptance gates, and durable output.
|
||||
|
||||
### Work
|
||||
|
||||
- Update `examples/dnd-complete.config.yml`:
|
||||
- add `locations` to the first descriptive step with extract, append-order
|
||||
merge, and LLM-backed normalize bindings;
|
||||
- add a generated `locations` reference to the next step;
|
||||
- add `location-occurrences` to that step with extract, append-order merge,
|
||||
and deterministic normalize bindings; and
|
||||
- add both lanes to evidence-context output where appropriate.
|
||||
- Keep the minimal example minimal unless its stated purpose requires a
|
||||
location lane; do not turn it into a second complete example.
|
||||
- Extend maintained example-loading/config-validation tests.
|
||||
- Add integration tests that prove:
|
||||
- the normalized accepted registry is handed off in memory;
|
||||
- the occurrence lane cannot run before its producer;
|
||||
- missing, cyclic, wrong-kind, wrong-media-type, rejected, or unaccepted
|
||||
producers are rejected at the existing boundaries;
|
||||
- same-name locations remain distinguishable by ID through the handoff;
|
||||
- registry evidence never becomes occurrence evidence;
|
||||
- retries and checkpoints honor prompt, schema, identity, and generated-
|
||||
reference fingerprints; and
|
||||
- output contains both durable artifact envelopes and evidence context.
|
||||
- Use recording/fake structured clients only; no network-dependent tests.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- The complete example loads through the real config path and exercises all
|
||||
registered D&D lanes.
|
||||
- Ordered handoff failure semantics match the framework's existing fail-whole-
|
||||
run policy.
|
||||
- Integration tests cover behavior rather than duplicating package internals.
|
||||
- `go test ./internal/modules/integration/... ./internal/config/...` and any
|
||||
example-specific test targets pass offline.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium-to-large but coherent as one end-to-end integration prompt.
|
||||
|
||||
## Stage 14: Publish Current Documentation And Perform Final Verification
|
||||
|
||||
### Goal
|
||||
|
||||
Make the implemented feature discoverable and retire fulfilled future-work
|
||||
language without leaving development-history documentation behind.
|
||||
|
||||
### Work
|
||||
|
||||
- Create canonical integration contracts:
|
||||
- `docs/integrations/dnd-location-artifacts.md`;
|
||||
- `docs/integrations/dnd-location-occurrence-artifacts.md`.
|
||||
- Update `docs/config.md` with both selectable keys, the `locations` reference
|
||||
slot and compatibility, execution classes, validators, default chains, and
|
||||
the complete-example link.
|
||||
- Update `docs/internal/dnd.md` with nine-lane composition, shared entity
|
||||
reconciliation, evidence-anchored identity, occurrence grounding, prompt
|
||||
asset reuse, and intentional lane differences. Link to integration contracts
|
||||
instead of duplicating their JSON shapes.
|
||||
- Update `docs/integrations/json-output.md`, `README.md`, and other current
|
||||
canonical inventories only where repository inspection shows that the new
|
||||
artifact kinds or maintained example must be listed.
|
||||
- Remove the fulfilled `Location Extraction` section from
|
||||
`docs/roadmap/future.md`. Keep the generic LLM-assisted deduplication item and
|
||||
clarify only if needed that the new D&D helper does not fulfill that broader
|
||||
feature.
|
||||
- Verify all relative links and search current documentation for stale
|
||||
seven-lane counts, missing keys, obsolete location-planning claims, and
|
||||
accidental claims that references are evidence.
|
||||
- Run formatting, focused tests, the full Go test suite, static analysis, and
|
||||
repository-provided config/example checks. Inspect `git diff --check` and
|
||||
confirm no unrelated files changed.
|
||||
- After implementation and verification are complete, leave `location.md` and
|
||||
this plan in place for the user's separate roadmap-retirement step; do not
|
||||
delete them unless explicitly asked.
|
||||
|
||||
### Acceptance Criteria
|
||||
|
||||
- Current documentation describes the implemented contracts and only
|
||||
implemented behavior outside `docs/roadmap/`.
|
||||
- The roadmap no longer presents completed location tracking as future work.
|
||||
- Links, examples, module inventories, and default-chain tables agree with
|
||||
production registration.
|
||||
- `go test ./internal/modules/dnd/...` passes.
|
||||
- `go test ./internal/modules/integration/...` passes.
|
||||
- `go test ./...` passes.
|
||||
- `go vet ./...` passes.
|
||||
- Repository-provided configuration/example validation passes.
|
||||
- `git diff --check` reports no errors.
|
||||
|
||||
### Prompt Size
|
||||
|
||||
Medium and suitable for one implementation prompt.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap fixes the artifact shapes, identity scope and
|
||||
derivation, occurrence categories, classification precedence, reference
|
||||
dependency, reconciliation safety boundary, pipeline placement, and non-goals
|
||||
needed to implement every stage without an additional product decision.
|
||||
@@ -1,337 +0,0 @@
|
||||
# D&D Location Tracking
|
||||
|
||||
## Purpose
|
||||
|
||||
Add evidence-grounded D&D location tracking without turning a single extractor
|
||||
into both an entity registry and an event classifier. The target design follows
|
||||
the established NPC pattern: one lane identifies canonical location records and
|
||||
a later lane records how the party related to those locations in the transcript.
|
||||
|
||||
This roadmap defines the desired end state and policy choices. The ordered work
|
||||
needed to reach that state is in [the implementation plan](implementation.md).
|
||||
|
||||
## User Intent
|
||||
|
||||
- Record locations the party visits or that the session otherwise discusses.
|
||||
- Distinguish current physical presence from plans, recollections, and ordinary
|
||||
mentions.
|
||||
- Preserve transcript evidence for every durable record.
|
||||
- Reconcile aliases and repeated appearances conservatively.
|
||||
- Keep distinct places separate when they happen to share a generic name.
|
||||
- Keep the schemas minimal. Location description, hierarchy, participants, and
|
||||
narrative analysis belong in other artifacts or deterministic joins.
|
||||
|
||||
## Target Capability
|
||||
|
||||
The D&D module family will have two new lanes:
|
||||
|
||||
1. `dnd/locations` produces a session-scoped registry of physical places.
|
||||
2. `dnd/location-occurrences` consumes the normalized location registry and
|
||||
produces an ordered list of source-grounded relationships between the party
|
||||
and those places.
|
||||
|
||||
The normalized location artifact is handed to the occurrence lane through a
|
||||
required generated reference named `locations`. The occurrence lane must use
|
||||
that registry for identity grounding, but the current transcript remains its
|
||||
only evidence source.
|
||||
|
||||
## Durable Artifact Contracts
|
||||
|
||||
Both contracts remain at `v1`; Notarius is pre-release and does not need a
|
||||
compatibility layer for these new artifacts.
|
||||
|
||||
### Location registry
|
||||
|
||||
The location lane uses:
|
||||
|
||||
- artifact kind: `dnd/location-list`
|
||||
- module key: `dnd/locations`
|
||||
- schema ID: `notarius.dnd.locations`
|
||||
- schema name: `notarius_dnd_locations_v1`
|
||||
- media type: `application/json`
|
||||
- root member: `locations`
|
||||
|
||||
Each location contains exactly:
|
||||
|
||||
| Field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | string | Deterministic, session-scoped canonical location identity. |
|
||||
| `name` | string | Evidence-grounded display name or transcript-established label. |
|
||||
| `source_refs` | non-empty source-reference array | Current-transcript evidence that identifies the place. |
|
||||
|
||||
Locations are physical or spatial places: planes, regions, settlements,
|
||||
districts, buildings, rooms, landmarks, routes, and geographic features. A
|
||||
generic label such as `the tavern` is permitted only when the transcript uses
|
||||
it for a specific place. The extractor must not invent a qualifier merely to
|
||||
distinguish that place from another place with the same label.
|
||||
|
||||
The registry does not contain type, parent, description, summary, coordinates,
|
||||
participants, visit status, or occurrence data. Parent and child places are
|
||||
separate identities when the transcript identifies both; nesting alone is not
|
||||
a reason to merge them.
|
||||
|
||||
### Location occurrences
|
||||
|
||||
The occurrence lane uses:
|
||||
|
||||
- artifact kind: `dnd/location-occurrence-list`
|
||||
- module key: `dnd/location-occurrences`
|
||||
- schema ID: `notarius.dnd.location_occurrences`
|
||||
- schema name: `notarius_dnd_location_occurrences_v1`
|
||||
- media type: `application/json`
|
||||
- root member: `occurrences`
|
||||
|
||||
Each occurrence contains exactly:
|
||||
|
||||
| Field | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `location_id` | string | An exact ID from the consumed normalized location registry. |
|
||||
| `name` | string | The canonical display name associated with `location_id`. |
|
||||
| `kind` | enum | `visited`, `planned`, `recalled`, or `mentioned`. |
|
||||
| `source_refs` | non-empty source-reference array | Current-transcript evidence for both the place and the classified occurrence. |
|
||||
|
||||
`location_id` is required even though existing NPC interactions currently use
|
||||
name-only grounding. Locations can legitimately share the same display name,
|
||||
so a name alone cannot provide an unambiguous cross-artifact link. The name is
|
||||
retained for readable standalone output and must exactly match the registry
|
||||
record selected by the ID after normalization.
|
||||
|
||||
## Identity Policy
|
||||
|
||||
Location identity is conservative and scoped to one source document. It is not
|
||||
a campaign-wide or cross-session world identity.
|
||||
|
||||
Display normalization trims surrounding whitespace and collapses internal
|
||||
Unicode whitespace. Comparison normalization uses the existing D&D entity
|
||||
rules: Unicode NFKC normalization, normalized apostrophes, collapsed
|
||||
whitespace, and Unicode case folding.
|
||||
|
||||
The canonical ID is:
|
||||
|
||||
~~~text
|
||||
location:sha256:<lowercase SHA-256 hex digest>
|
||||
~~~
|
||||
|
||||
The digest input is the UTF-8 encoding of compact JSON for this five-element
|
||||
array:
|
||||
|
||||
~~~text
|
||||
["dnd.locations.identity.v1", comparison_name, source_id, start_unit_id, end_unit_id]
|
||||
~~~
|
||||
|
||||
The source values come from the earliest reference after canonical reference
|
||||
sorting and exact deduplication. Compact JSON array encoding is part of the
|
||||
identity contract: it avoids delimiter ambiguity and must not be replaced
|
||||
without changing the policy version. A blank comparison name or missing valid
|
||||
source reference produces no manufactured ID and remains a validation error.
|
||||
|
||||
Including the evidence anchor prevents two unrelated places called `the
|
||||
tavern` from receiving the same ID. When semantic normalization safely groups
|
||||
aliases or repeated appearances, it first chooses an existing canonical display
|
||||
name and unions the evidence; it then derives the final ID from that name and
|
||||
the earliest unioned reference.
|
||||
|
||||
The normalizer may merge records only when transcript evidence clearly shows
|
||||
that they denote the same physical place. It must not merge records solely
|
||||
because:
|
||||
|
||||
- their comparison names are equal;
|
||||
- they are near one another in the transcript;
|
||||
- one is spatially nested inside the other; or
|
||||
- their labels are both generic.
|
||||
|
||||
Distinct normalized records may therefore have the same comparison name, but
|
||||
their IDs must be unique and correctly derived. Exact duplicates with the same
|
||||
comparison name and canonical evidence may be collapsed deterministically.
|
||||
|
||||
## Occurrence Semantics
|
||||
|
||||
Each occurrence has one kind:
|
||||
|
||||
- `visited`: current-session gameplay establishes that one or more party
|
||||
members are physically present at the location, including an arrival,
|
||||
continuing presence, or departure.
|
||||
- `planned`: the party explicitly proposes, intends, or agrees to future travel
|
||||
to the location. Mere hypotheticals or speculation are not plans.
|
||||
- `recalled`: the transcript explicitly recounts or recaps the party being at
|
||||
the location before the current session's live events.
|
||||
- `mentioned`: the location is explicitly referenced but the occurrence does
|
||||
not meet a stronger definition. This includes lore, directions, third-party
|
||||
activity, non-actionable speculation, and out-of-character discussion.
|
||||
|
||||
An inferred but unstated place produces no location or occurrence. Uncertainty
|
||||
is handled by conservative omission rather than an `uncertain` enum value.
|
||||
|
||||
For one occurrence supported by overlapping evidence, classification precedence
|
||||
is `visited`, then `planned`, then `recalled`, then `mentioned`; `mentioned` is
|
||||
the fallback. A passage may produce multiple records when it independently
|
||||
supports separate facts, such as recalling an earlier visit while planning a
|
||||
return. Exact duplicates with the same ID, kind, and canonical evidence are
|
||||
collapsed. Different kinds or independently supported evidence remain.
|
||||
|
||||
Output is ordered by earliest evidence in source-document order, then by
|
||||
`location_id`, `name`, kind order (`visited`, `planned`, `recalled`,
|
||||
`mentioned`), and the remaining canonical reference sequence.
|
||||
|
||||
## Extraction, Normalization, And Evidence
|
||||
|
||||
### Location registry lane
|
||||
|
||||
The extractor is LLM-backed and follows the shared D&D extraction prompt and
|
||||
input conventions. It emits names and source ranges through a private response
|
||||
schema; deterministic mapping supplies the current source ID and derives
|
||||
candidate IDs. Campaign references may disambiguate terminology but never
|
||||
become durable evidence.
|
||||
|
||||
The merger uses the typed append-order convention. The normalizer is LLM-backed:
|
||||
it deterministically prepares names and evidence, then asks the model only for
|
||||
duplicate groups. The model may identify groups and choose a canonical member,
|
||||
but it may not create, delete, rewrite, or directly replace durable records.
|
||||
Code validates the proposal, applies non-overlapping safe groups, unions
|
||||
evidence, derives final IDs, orders output, and emits bounded warnings.
|
||||
|
||||
Malformed, unknown, overlapping, or ambiguous proposal groups are rejected.
|
||||
The normalizer uses the existing bounded retry behavior and falls back to the
|
||||
safe deterministic candidate set if no usable proposal is obtained.
|
||||
|
||||
### Location occurrence lane
|
||||
|
||||
The extractor is LLM-backed and requires exactly one validated `locations`
|
||||
reference. The prompt projection contains only ordered `{id, name}` pairs; it
|
||||
omits registry evidence and reference provenance. The model must copy both
|
||||
values from one projected record and cite current-transcript source ranges for
|
||||
the occurrence.
|
||||
|
||||
The occurrence normalizer is deterministic. It canonicalizes names by exact
|
||||
registry ID, normalizes evidence and ordering, and removes exact duplicates.
|
||||
Unknown IDs and mismatched ID/name pairs remain inspectable validation failures
|
||||
rather than being guessed or silently reassigned.
|
||||
|
||||
The occurrence lane cannot add a missing location to the registry. If the
|
||||
location extractor omitted a place, the correct behavior is to omit its
|
||||
occurrence and improve the upstream extraction later.
|
||||
|
||||
## Shared Entity Reconciliation
|
||||
|
||||
Adding a second LLM-assisted entity registry demonstrates a concrete shared
|
||||
need in the D&D domain. The existing NPC normalization context-window and
|
||||
proposal-safety logic will move to
|
||||
`internal/modules/dnd/shared/entityreconcile` and serve both NPC and location
|
||||
normalizers.
|
||||
|
||||
The shared package owns:
|
||||
|
||||
- deterministic opaque candidate keys;
|
||||
- source-window construction and canonical prompt materials;
|
||||
- a common private duplicate-group response contract;
|
||||
- validation of unknown, repeated, overlapping, malformed, or ineligible
|
||||
candidate keys; and
|
||||
- immutable assessment results identifying safe groups.
|
||||
|
||||
It does not call the LLM, choose domain-specific canonical names, derive
|
||||
durable IDs, mutate domain artifacts, or format domain warnings. Those
|
||||
responsibilities remain in each normalizer.
|
||||
|
||||
NPC normalization will migrate to the shared key-based proposal contract
|
||||
without changing its durable NPC behavior. Its prompt and private response
|
||||
schema fingerprints are expected to change, so stale NPC normalization
|
||||
checkpoints will invalidate normally.
|
||||
|
||||
The two normalizers will reuse an exactly identical shared reconciliation
|
||||
instruction asset and private response schema. Module-owned task text will
|
||||
continue to define the different NPC and location identity rules. This keeps
|
||||
shared prompt content identical without pretending the two domains have the
|
||||
same semantic merge policy.
|
||||
|
||||
This helper is intentionally D&D-specific. It does not implement the broader
|
||||
domain-neutral replacement-element normalizer still described in
|
||||
[future work](future.md).
|
||||
|
||||
## Reference Contract And Pipeline Placement
|
||||
|
||||
The generated reference slot is named `locations` and accepts exactly one JSON
|
||||
artifact of kind `dnd/location-list`, with the established 1 MiB limit. It is
|
||||
required by both extraction and normalization for
|
||||
`dnd/location-occurrences`. Static file bindings remain valid where the
|
||||
framework permits them, but the maintained complete example uses a generated
|
||||
same-run artifact.
|
||||
|
||||
The complete D&D pipeline places `locations` in the first descriptive step
|
||||
alongside the independent NPC, item-event, and scene-description lanes. It
|
||||
places `location-occurrences` in the next step and binds the accepted normalized
|
||||
location artifact from the first step. The occurrence lane has no mandatory
|
||||
NPC or scene-description dependency.
|
||||
|
||||
No current downstream lane is changed to consume location artifacts. Future
|
||||
narrative reports or joins may use the canonical IDs after defining their own
|
||||
contracts.
|
||||
|
||||
## Validation And Production Defaults
|
||||
|
||||
The location registry receives production validators for:
|
||||
|
||||
- required shape and supported ID syntax;
|
||||
- current-document and current-chunk source ranges;
|
||||
- normalized identity derivation and ID uniqueness; and
|
||||
- advisory source relatedness.
|
||||
|
||||
The location occurrence artifact receives production validators for:
|
||||
|
||||
- required shape and the four supported kinds;
|
||||
- registry membership and exact ID/name correspondence;
|
||||
- normalized ordering and exact-duplicate invariants;
|
||||
- current-document and current-chunk source ranges; and
|
||||
- advisory source relatedness.
|
||||
|
||||
Validators remain immutable and diagnostic. Durable JSON Schema validation
|
||||
stays in the production chains after semantic shape and source-reference
|
||||
checks, consistent with the existing D&D lanes.
|
||||
|
||||
## Documentation End State
|
||||
|
||||
Implementation will add canonical integration documents for both durable
|
||||
artifacts and update current-state documentation to cover:
|
||||
|
||||
- both module and artifact keys;
|
||||
- the `locations` generated-reference slot;
|
||||
- production validators and default chains;
|
||||
- D&D family composition, reconciliation, identity, and grounding behavior;
|
||||
- the complete maintained pipeline example; and
|
||||
- JSON output and evidence-context support.
|
||||
|
||||
After the feature is implemented, the fulfilled Location Extraction section is
|
||||
removed from `future.md`. Historical implementation narration remains in
|
||||
version control rather than current documentation.
|
||||
|
||||
## Out Of Scope
|
||||
|
||||
- Campaign-wide or cross-session canonical location IDs.
|
||||
- A location ontology, hierarchy, map, coordinates, or containment graph.
|
||||
- Location descriptions, summaries, participants, ownership, or encounter
|
||||
analysis.
|
||||
- Inferring a location that the transcript does not identify.
|
||||
- Automatically creating registry records from occurrence output.
|
||||
- Changing NPC-interaction artifacts to use NPC IDs.
|
||||
- Making other lanes consume location references.
|
||||
- A generic domain-neutral LLM deduplication framework.
|
||||
- Long-term artifact-version migration machinery.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- Both durable contracts are minimal, strict, versioned, and registered.
|
||||
- Location IDs are deterministic under the documented policy and do not force
|
||||
same-named places to collapse.
|
||||
- Alias and repeat reconciliation is proposal-only, conservatively validated,
|
||||
and safe on retry exhaustion.
|
||||
- Occurrences use one of the four defined kinds and carry an unambiguous
|
||||
registry ID/name pair plus current-transcript evidence.
|
||||
- Missing, malformed, oversized, or incompatible `locations` references fail
|
||||
through the established configuration or operation boundaries.
|
||||
- Both lanes have typed mergers, normalizers, evidence projectors, validators,
|
||||
default chains, prompt/profile metadata, and registration coverage consistent
|
||||
with the D&D family.
|
||||
- NPC normalization retains its durable behavior after adopting the shared
|
||||
reconciliation helper.
|
||||
- The maintained complete example loads and exercises the generated handoff.
|
||||
- Focused D&D and integration tests pass offline, and current documentation
|
||||
describes only implemented behavior once the work is complete.
|
||||
@@ -36,6 +36,8 @@ pipelines:
|
||||
window_units: 3
|
||||
lanes:
|
||||
- item-events
|
||||
- locations
|
||||
- location-occurrences
|
||||
- npcs
|
||||
- spells
|
||||
- combat-turns
|
||||
@@ -59,6 +61,14 @@ pipelines:
|
||||
normalize:
|
||||
module: dnd/npcs
|
||||
retries: 2
|
||||
locations:
|
||||
extract:
|
||||
module: dnd/locations
|
||||
retries: 2
|
||||
merge: appendorder
|
||||
normalize:
|
||||
module: dnd/locations
|
||||
retries: 2
|
||||
scene-descriptions:
|
||||
extract:
|
||||
module: dnd/scene-descriptions
|
||||
@@ -66,9 +76,13 @@ pipelines:
|
||||
merge: appendorder
|
||||
normalize: dnd/scene-descriptions
|
||||
- id: extract-events
|
||||
# Accepted NPC grounding and scene-description eligibility artifacts are
|
||||
# supplied in memory to their compatible consumers in this step.
|
||||
# Accepted registry artifacts and scene-description eligibility artifacts
|
||||
# are supplied in memory to their compatible consumers in this step.
|
||||
references:
|
||||
locations:
|
||||
artifact:
|
||||
step: describe-session
|
||||
lane: locations
|
||||
npcs:
|
||||
artifact:
|
||||
step: describe-session
|
||||
@@ -102,6 +116,12 @@ pipelines:
|
||||
retries: 2
|
||||
merge: appendorder
|
||||
normalize: dnd/npc-interactions
|
||||
location-occurrences:
|
||||
extract:
|
||||
module: dnd/location-occurrences
|
||||
retries: 2
|
||||
merge: appendorder
|
||||
normalize: dnd/location-occurrences
|
||||
- id: track-enemies
|
||||
references:
|
||||
npcs:
|
||||
|
||||
@@ -20,14 +20,19 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/chunk/scenes"
|
||||
locationoccurrencecodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locationoccurrences"
|
||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locations"
|
||||
combat "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
|
||||
enemyevents "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/enemyevents"
|
||||
itemevents "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/itemevents"
|
||||
locationoccurrences "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locationoccurrences"
|
||||
locations "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locations"
|
||||
npcinteractions "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
|
||||
npcs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
scenedescriptions "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||
enemyeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/enemyevents"
|
||||
locationnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locations"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
)
|
||||
|
||||
@@ -117,10 +122,15 @@ func TestMaintainedCompleteExampleProducesEnemyEventsThroughGeneratedHandoffs(t
|
||||
runRoot := filepath.Join(outputRoot, productionRunID)
|
||||
index := readProductionJSON[exampleOutputIndex](t, filepath.Join(runRoot, "index.json"))
|
||||
var enemyOutput exampleOutputIndexEntry
|
||||
var locationOutput, occurrenceOutput exampleOutputIndexEntry
|
||||
for _, entry := range index.OutputFiles {
|
||||
if entry.LaneID == "enemy-events" {
|
||||
switch entry.LaneID {
|
||||
case "enemy-events":
|
||||
enemyOutput = entry
|
||||
break
|
||||
case "locations":
|
||||
locationOutput = entry
|
||||
case "location-occurrences":
|
||||
occurrenceOutput = entry
|
||||
}
|
||||
}
|
||||
if enemyOutput.File != "lanes/enemy-events.json" || enemyOutput.SchemaID != "notarius.dnd.enemy_events" || enemyOutput.SchemaVersion != "v1" {
|
||||
@@ -130,10 +140,26 @@ func TestMaintainedCompleteExampleProducesEnemyEventsThroughGeneratedHandoffs(t
|
||||
if len(value.Events) != 1 || value.Events[0].Name != "Kesh" || value.Events[0].Kind != dnd.EnemyEventKindFled || len(value.Events[0].SourceRefs) != 1 || value.Events[0].SourceRefs[0].SourceID != "session-ravenfall" || value.Events[0].SourceRefs[0].StartUnitID != 10 {
|
||||
t.Fatalf("enemy event artifact = %#v, want source-linked Kesh fleeing event", value)
|
||||
}
|
||||
if locationOutput.File != "lanes/locations.json" || locationOutput.SchemaID != locationcodec.SchemaID || locationOutput.SchemaVersion != locationcodec.SchemaVersion {
|
||||
t.Fatalf("location output = %#v, want typed location registry JSON", locationOutput)
|
||||
}
|
||||
locationsValue := readProductionJSON[dnd.LocationList](t, filepath.Join(runRoot, locationOutput.File))
|
||||
if len(locationsValue.Locations) != 2 || locationsValue.Locations[0].Name != "Moon Gate" || locationsValue.Locations[1].Name != "Moon Gate" || locationsValue.Locations[0].ID == locationsValue.Locations[1].ID {
|
||||
t.Fatalf("location registry = %#v, want distinct source-grounded identities for same-name locations", locationsValue)
|
||||
}
|
||||
if occurrenceOutput.File != "lanes/location-occurrences.json" || occurrenceOutput.SchemaID != locationoccurrencecodec.SchemaID || occurrenceOutput.SchemaVersion != locationoccurrencecodec.SchemaVersion {
|
||||
t.Fatalf("location occurrence output = %#v, want typed occurrence JSON", occurrenceOutput)
|
||||
}
|
||||
occurrencesValue := readProductionJSON[dnd.LocationOccurrenceList](t, filepath.Join(runRoot, occurrenceOutput.File))
|
||||
if len(occurrencesValue.Occurrences) != 2 || occurrencesValue.Occurrences[0].LocationID == occurrencesValue.Occurrences[1].LocationID || occurrencesValue.Occurrences[0].Name != "Moon Gate" || occurrencesValue.Occurrences[1].Name != "Moon Gate" {
|
||||
t.Fatalf("location occurrences = %#v, want source-grounded references to distinct registry identities", occurrencesValue)
|
||||
}
|
||||
|
||||
evidence := readProductionJSON[evidencecontext.Document](t, filepath.Join(runRoot, "evidence-context.json"))
|
||||
if !containsString(evidence.SelectedLanes, "enemy-events") || !evidenceHasLane(evidence, "enemy-events") {
|
||||
t.Fatalf("evidence context = %#v, want direct enemy-event evidence", evidence)
|
||||
for _, laneID := range []string{"enemy-events", "locations", "location-occurrences"} {
|
||||
if !containsString(evidence.SelectedLanes, laneID) || !evidenceHasLane(evidence, laneID) {
|
||||
t.Fatalf("evidence context = %#v, want direct %s evidence", evidence, laneID)
|
||||
}
|
||||
}
|
||||
|
||||
requests := client.requestsFor(enemyevents.PromptID)
|
||||
@@ -154,6 +180,16 @@ func TestMaintainedCompleteExampleProducesEnemyEventsThroughGeneratedHandoffs(t
|
||||
t.Fatalf("enemy event %s prompt input = %q, want compact source-free grounding", slot, input.Content)
|
||||
}
|
||||
}
|
||||
locationRequests := client.requestsFor(locationoccurrences.PromptID)
|
||||
if len(locationRequests) != 2 {
|
||||
t.Fatalf("location occurrence requests = %#v, want one request per scene", locationRequests)
|
||||
}
|
||||
for _, request := range locationRequests {
|
||||
registryInput := request.Inputs["locations"]
|
||||
if !strings.Contains(string(registryInput.Content), "Moon Gate") || !strings.Contains(string(registryInput.Content), `"id"`) || strings.Contains(string(registryInput.Content), "source_refs") {
|
||||
t.Fatalf("location occurrence registry input = %q, want source-free ID grounding", registryInput.Content)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func completeExampleConfigWithTemporaryCache(t *testing.T) string {
|
||||
@@ -207,6 +243,14 @@ func (client *enemyEventLLMClient) CompleteStructured(ctx context.Context, reque
|
||||
kind, title = "combat", "Raiders attack"
|
||||
}
|
||||
content = []byte(fmt.Sprintf(`{"kind":%q,"title":%q,"summary":"session scene"}`, kind, title))
|
||||
case locations.PromptID:
|
||||
unitID := 1
|
||||
if combatScene {
|
||||
unitID = 7
|
||||
}
|
||||
content = []byte(fmt.Sprintf(`{"locations":[{"name":"Moon Gate","source_refs":[{"start_unit_id":%d,"end_unit_id":%d}]}]}`, unitID, unitID))
|
||||
case locationnormalize.PromptID:
|
||||
content = []byte(`{"duplicate_groups":[]}`)
|
||||
case spells.PromptID:
|
||||
content = []byte(`{"spell_casts":[]}`)
|
||||
case itemevents.PromptID:
|
||||
@@ -219,6 +263,27 @@ func (client *enemyEventLLMClient) CompleteStructured(ctx context.Context, reque
|
||||
} else {
|
||||
content = []byte(`{"interactions":[]}`)
|
||||
}
|
||||
case locationoccurrences.PromptID:
|
||||
var registry struct {
|
||||
Locations []struct {
|
||||
ID string `json:"id"`
|
||||
} `json:"locations"`
|
||||
}
|
||||
if err := json.Unmarshal(request.Inputs["locations"].Content, ®istry); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("decode generated location registry: %w", err)
|
||||
}
|
||||
if len(registry.Locations) == 0 {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("generated location registry has no locations")
|
||||
}
|
||||
unitID := 1
|
||||
locationID := registry.Locations[0].ID
|
||||
if combatScene {
|
||||
unitID = 7
|
||||
if len(registry.Locations) > 1 {
|
||||
locationID = registry.Locations[1].ID
|
||||
}
|
||||
}
|
||||
content = []byte(fmt.Sprintf(`{"occurrences":[{"location_id":%q,"name":"Moon Gate","kind":"visited","source_refs":[{"start_unit_id":%d,"end_unit_id":%d}]}]}`, locationID, unitID, unitID))
|
||||
case enemyevents.PromptID:
|
||||
content = []byte(`{"events":[{"name":"Kesh","kind":"fled","source_refs":[{"start_unit_id":10,"end_unit_id":10}]}]}`)
|
||||
default:
|
||||
|
||||
@@ -15,6 +15,10 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationoccurrenceextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locationoccurrences"
|
||||
locationextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locations"
|
||||
locationoccurrencenormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locationoccurrences"
|
||||
locationnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locations"
|
||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/seriatim/input/transcript"
|
||||
)
|
||||
@@ -54,9 +58,31 @@ func TestMaintainedExamplesLoadResolveAndList(t *testing.T) {
|
||||
t.Fatalf("materialize maintained example references for %q: %v", pipelineID, err)
|
||||
}
|
||||
if example.name == "complete" {
|
||||
if got := exampleStepLaneIDs(materialized); strings.Join(got, "|") != "describe-session:item-events,npcs,scene-descriptions|extract-events:combat-turns,npc-interactions,spells|track-enemies:enemy-events" {
|
||||
if got := exampleStepLaneIDs(materialized); strings.Join(got, "|") != "describe-session:item-events,locations,npcs,scene-descriptions|extract-events:combat-turns,location-occurrences,npc-interactions,spells|track-enemies:enemy-events" {
|
||||
t.Fatalf("complete example steps and lanes = %v, want the documented D&D extractor composition", got)
|
||||
}
|
||||
locationLane := referenceContractLane(t, materialized, "locations")
|
||||
if locationLane.ArtifactKind != dnd.LocationListKind || locationLane.Extract.Module != locationextract.Key || locationLane.Extract.Retries != 2 || locationLane.Merge.Module != pipeline.DefaultMergeModule || locationLane.Normalize.Module != locationnormalize.Key || locationLane.Normalize.Retries != 2 {
|
||||
t.Fatalf("location lane = %#v, want typed registry composition", locationLane)
|
||||
}
|
||||
occurrenceLane := referenceContractLane(t, materialized, "location-occurrences")
|
||||
if occurrenceLane.ArtifactKind != dnd.LocationOccurrenceListKind || occurrenceLane.Extract.Module != locationoccurrenceextract.Key || occurrenceLane.Extract.Retries != 2 || occurrenceLane.Merge.Module != pipeline.DefaultMergeModule || occurrenceLane.Normalize.Module != locationoccurrencenormalize.Key {
|
||||
t.Fatalf("location occurrence lane = %#v, want typed occurrence composition", occurrenceLane)
|
||||
}
|
||||
for _, target := range []pipeline.ResolvedReferenceTarget{occurrenceLane.ExtractReferences, occurrenceLane.NormalizeReferences} {
|
||||
binding, found := generatedReferenceBinding(target.Bindings, "locations")
|
||||
if !found || binding.Artifact.Step != "describe-session" || binding.Artifact.Lane != "locations" {
|
||||
t.Fatalf("location occurrence %s reference = %#v, want generated location registry", target.Stage, binding)
|
||||
}
|
||||
}
|
||||
for _, slot := range []string{"party", "glossary"} {
|
||||
if len(occurrenceLane.ExtractReferences.ReferenceSet.Slots[slot].Items) != 1 {
|
||||
t.Fatalf("location occurrence extractor %s reference was not materialized: %#v", slot, occurrenceLane.ExtractReferences)
|
||||
}
|
||||
if _, found := occurrenceLane.NormalizeReferences.ReferenceSet.Slots[slot]; found {
|
||||
t.Fatalf("location occurrence normalizer unexpectedly consumes %s: %#v", slot, occurrenceLane.NormalizeReferences)
|
||||
}
|
||||
}
|
||||
spellLane := referenceContractLane(t, materialized, "spells")
|
||||
if len(spellLane.ExtractReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 ||
|
||||
len(spellLane.NormalizeReferences.ReferenceSet.Slots["spell_catalog"].Items) != 1 {
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.location_occurrences",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["occurrences"],
|
||||
"properties": {
|
||||
"occurrences": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["location_id", "name", "kind", "source_refs"],
|
||||
"properties": {
|
||||
"location_id": {
|
||||
"type": "string",
|
||||
"pattern": "^location:sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"kind": {
|
||||
"type": "string",
|
||||
"enum": ["visited", "planned", "recalled", "mentioned"]
|
||||
},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["source_id", "start_unit_id", "end_unit_id"],
|
||||
"properties": {
|
||||
"source_id": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"start_unit_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"end_unit_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
126
internal/modules/dnd/codec/locationoccurrences/codec.go
Normal file
126
internal/modules/dnd/codec/locationoccurrences/codec.go
Normal file
@@ -0,0 +1,126 @@
|
||||
// Package locationoccurrences encodes durable D&D location occurrence artifacts.
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/candidatejson"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
const (
|
||||
SchemaID = "notarius.dnd.location_occurrences"
|
||||
SchemaName = "notarius_dnd_location_occurrences_v1"
|
||||
SchemaVersion = "v1"
|
||||
MediaType = "application/json"
|
||||
)
|
||||
|
||||
//go:embed assets/schemas/dnd_location_occurrences.v1.json
|
||||
var schemaAssets embed.FS
|
||||
|
||||
var _ contracts.ArtifactCodec[dnd.LocationOccurrenceList] = (*Codec)(nil)
|
||||
|
||||
type Codec struct{}
|
||||
|
||||
func New() *Codec { return &Codec{} }
|
||||
|
||||
func (c *Codec) Kind() contracts.ArtifactKind { return dnd.LocationOccurrenceListKind }
|
||||
|
||||
func (c *Codec) Schema() contracts.ArtifactSchema {
|
||||
raw, err := schemaAssets.ReadFile("assets/schemas/dnd_location_occurrences.v1.json")
|
||||
if err != nil {
|
||||
return contracts.ArtifactSchema{}
|
||||
}
|
||||
return contracts.ArtifactSchema{
|
||||
ID: SchemaID,
|
||||
Name: SchemaName,
|
||||
Version: SchemaVersion,
|
||||
JSONSchema: append([]byte(nil), raw...),
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Codec) MediaType() string { return MediaType }
|
||||
|
||||
func (c *Codec) Metadata(value dnd.LocationOccurrenceList) map[string]any {
|
||||
return map[string]any{"occurrence_count": len(value.Occurrences)}
|
||||
}
|
||||
|
||||
func (c *Codec) Encode(value dnd.LocationOccurrenceList) ([]byte, error) {
|
||||
if err := validate(value); err != nil {
|
||||
return nil, fmt.Errorf("encode dnd location occurrence list: %w", err)
|
||||
}
|
||||
return c.EncodeCandidate(value)
|
||||
}
|
||||
|
||||
// EncodeCandidate provides the durable representation before semantic
|
||||
// validators have approved a value.
|
||||
func (c *Codec) EncodeCandidate(value dnd.LocationOccurrenceList) ([]byte, error) {
|
||||
return candidatejson.EncodeCandidate("dnd location occurrence list", value)
|
||||
}
|
||||
|
||||
func (c *Codec) Decode(content []byte) (dnd.LocationOccurrenceList, error) {
|
||||
value, err := c.DecodeCandidate(content)
|
||||
if err != nil {
|
||||
return dnd.LocationOccurrenceList{}, err
|
||||
}
|
||||
if err := validate(value); err != nil {
|
||||
return dnd.LocationOccurrenceList{}, fmt.Errorf("decode dnd location occurrence list: %w", err)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// DecodeCandidate reads one strict durable JSON value before semantic
|
||||
// validators have approved it.
|
||||
func (c *Codec) DecodeCandidate(content []byte) (dnd.LocationOccurrenceList, error) {
|
||||
return candidatejson.DecodeCandidate[dnd.LocationOccurrenceList]("dnd location occurrence list", content)
|
||||
}
|
||||
|
||||
func validate(value dnd.LocationOccurrenceList) error {
|
||||
if value.Occurrences == nil {
|
||||
return fmt.Errorf("occurrences must be present")
|
||||
}
|
||||
for index, occurrence := range value.Occurrences {
|
||||
prefix := fmt.Sprintf("occurrences[%d]", index)
|
||||
if !identity.IsValidID(occurrence.LocationID) {
|
||||
return fmt.Errorf("%s.location_id must match location ID pattern", prefix)
|
||||
}
|
||||
if strings.TrimSpace(occurrence.Name) == "" {
|
||||
return fmt.Errorf("%s.name must not be empty", prefix)
|
||||
}
|
||||
if !validOccurrenceKind(occurrence.Kind) {
|
||||
return fmt.Errorf("%s.kind must be supported", prefix)
|
||||
}
|
||||
if len(occurrence.SourceRefs) == 0 {
|
||||
return fmt.Errorf("%s.source_refs must contain at least one reference", prefix)
|
||||
}
|
||||
for refIndex, ref := range occurrence.SourceRefs {
|
||||
refPrefix := fmt.Sprintf("%s.source_refs[%d]", prefix, refIndex)
|
||||
if strings.TrimSpace(ref.SourceID) == "" {
|
||||
return fmt.Errorf("%s.source_id must not be empty", refPrefix)
|
||||
}
|
||||
if ref.StartUnitID <= 0 {
|
||||
return fmt.Errorf("%s.start_unit_id must be positive", refPrefix)
|
||||
}
|
||||
if ref.EndUnitID <= 0 {
|
||||
return fmt.Errorf("%s.end_unit_id must be positive", refPrefix)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validOccurrenceKind(value dnd.LocationOccurrenceKind) bool {
|
||||
switch value {
|
||||
case dnd.LocationOccurrenceKindVisited,
|
||||
dnd.LocationOccurrenceKindPlanned,
|
||||
dnd.LocationOccurrenceKindRecalled,
|
||||
dnd.LocationOccurrenceKindMentioned:
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
163
internal/modules/dnd/codec/locationoccurrences/codec_test.go
Normal file
163
internal/modules/dnd/codec/locationoccurrences/codec_test.go
Normal file
@@ -0,0 +1,163 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
func validList() dnd.LocationOccurrenceList {
|
||||
refs := []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}}
|
||||
return dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{
|
||||
LocationID: identity.DeriveID("The Old Tavern", refs),
|
||||
Name: "The Old Tavern",
|
||||
Kind: dnd.LocationOccurrenceKindVisited,
|
||||
SourceRefs: refs,
|
||||
}}}
|
||||
}
|
||||
|
||||
func TestCodecMatchesMaintainedDurableFixture(t *testing.T) {
|
||||
raw, err := os.ReadFile("testdata/dnd_location_occurrences.v1.json")
|
||||
if err != nil {
|
||||
t.Fatalf("read durable fixture: %v", err)
|
||||
}
|
||||
codec := New()
|
||||
value, err := codec.Decode(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode() error = %v, want nil", err)
|
||||
}
|
||||
if want := validList(); !reflect.DeepEqual(value, want) {
|
||||
t.Fatalf("Decode() = %#v, want %#v", value, want)
|
||||
}
|
||||
encoded, err := codec.Encode(value)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
}
|
||||
var compact bytes.Buffer
|
||||
if err := json.Compact(&compact, raw); err != nil {
|
||||
t.Fatalf("compact durable fixture: %v", err)
|
||||
}
|
||||
if !bytes.Equal(encoded, compact.Bytes()) {
|
||||
t.Fatalf("Encode() = %s, want %s", encoded, compact.Bytes())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecOwnsDurableSchemaAndMetadata(t *testing.T) {
|
||||
codec := New()
|
||||
schema := codec.Schema()
|
||||
if codec.Kind() != dnd.LocationOccurrenceListKind || codec.MediaType() != MediaType {
|
||||
t.Fatalf("codec identity = %q/%q", codec.Kind(), codec.MediaType())
|
||||
}
|
||||
if schema.ID != SchemaID || schema.Name != SchemaName || schema.Version != SchemaVersion || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v, want durable location occurrence schema", schema)
|
||||
}
|
||||
var document map[string]any
|
||||
if err := json.Unmarshal(schema.JSONSchema, &document); err != nil || document["$id"] != SchemaID {
|
||||
t.Fatalf("durable schema document = %#v, %v", document, err)
|
||||
}
|
||||
registry := pipeline.NewArtifactCodecRegistry()
|
||||
if err := pipeline.RegisterArtifactCodec(registry, codec); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v", err)
|
||||
}
|
||||
if spec, ok := registry.Spec(dnd.LocationOccurrenceListKind); !ok || spec.SchemaDigest != contracts.DigestArtifactSchema(schema) {
|
||||
t.Fatalf("registered spec = %#v, %t", spec, ok)
|
||||
}
|
||||
first := schema.JSONSchema
|
||||
first[0] = '['
|
||||
if next := codec.Schema().JSONSchema; !json.Valid(next) || next[0] == '[' {
|
||||
t.Fatal("Schema() returned shared bytes")
|
||||
}
|
||||
metadata := codec.Metadata(validList())
|
||||
metadata["other"] = true
|
||||
if next := codec.Metadata(validList()); len(next) != 1 || next["occurrence_count"] != 1 {
|
||||
t.Fatalf("Metadata() = %#v", next)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecSupportsEmptyListsAndCandidateSemanticFailures(t *testing.T) {
|
||||
codec := New()
|
||||
empty := dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{}}
|
||||
if content, err := codec.Encode(empty); err != nil || string(content) != `{"occurrences":[]}` {
|
||||
t.Fatalf("Encode() = %s, %v", content, err)
|
||||
}
|
||||
for _, candidate := range []dnd.LocationOccurrenceList{
|
||||
{},
|
||||
empty,
|
||||
{Occurrences: []dnd.LocationOccurrence{{LocationID: "bad", Name: " ", Kind: "unsupported", SourceRefs: nil}}},
|
||||
{Occurrences: []dnd.LocationOccurrence{{LocationID: "bad", Name: " ", Kind: "unsupported", SourceRefs: []source.SourceRef{}}}},
|
||||
{Occurrences: []dnd.LocationOccurrence{{LocationID: "bad", Name: " ", Kind: "unsupported", SourceRefs: []source.SourceRef{{StartUnitID: 0, EndUnitID: -1}}}}},
|
||||
} {
|
||||
content, err := codec.EncodeCandidate(candidate)
|
||||
if err != nil || !json.Valid(content) {
|
||||
t.Fatalf("EncodeCandidate() = %s, %v", content, err)
|
||||
}
|
||||
decoded, err := codec.DecodeCandidate(content)
|
||||
if err != nil || !reflect.DeepEqual(decoded, candidate) {
|
||||
t.Fatalf("DecodeCandidate() = %#v, %v, want %#v", decoded, err, candidate)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecRejectsStructuralJSONBeforeSemanticApproval(t *testing.T) {
|
||||
valid := `{"occurrences":[{"location_id":"location:sha256:0000000000000000000000000000000000000000000000000000000000000000","name":"The Tavern","kind":"visited","source_refs":[{"source_id":"session","start_unit_id":1,"end_unit_id":1}]}]}`
|
||||
for _, test := range []struct{ name, raw, want string }{
|
||||
{"malformed", `{`, "decode dnd location occurrence list"},
|
||||
{"unknown top-level", `{"occurrences":[],"unexpected":true}`, "unknown field"},
|
||||
{"unknown occurrence field", strings.Replace(valid, `"kind":"visited"`, `"kind":"visited","unexpected":true`, 1), "unknown field"},
|
||||
{"unknown source reference field", strings.Replace(valid, `"end_unit_id":1`, `"end_unit_id":1,"unexpected":true`, 1), "unknown field"},
|
||||
{"invalid type", `{"occurrences":"not-an-array"}`, "cannot unmarshal string"},
|
||||
{"trailing", `{"occurrences":[]} {}`, "multiple JSON values"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := New().DecodeCandidate([]byte(test.raw)); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("DecodeCandidate() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecRejectsApprovedShapeEnumAndReferenceBoundaries(t *testing.T) {
|
||||
base := validList().Occurrences[0]
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
value dnd.LocationOccurrenceList
|
||||
want string
|
||||
}{
|
||||
{"nil occurrences", dnd.LocationOccurrenceList{}, "occurrences must be present"},
|
||||
{"invalid ID", dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{LocationID: "bad", Name: base.Name, Kind: base.Kind, SourceRefs: base.SourceRefs}}}, "location_id must match location ID pattern"},
|
||||
{"blank name", dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{LocationID: base.LocationID, Name: " ", Kind: base.Kind, SourceRefs: base.SourceRefs}}}, "name must not be empty"},
|
||||
{"invalid kind", dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{LocationID: base.LocationID, Name: base.Name, Kind: "unsupported", SourceRefs: base.SourceRefs}}}, "kind must be supported"},
|
||||
{"empty source refs", dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{LocationID: base.LocationID, Name: base.Name, Kind: base.Kind}}}, "source_refs must contain"},
|
||||
{"malformed source reference", dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{LocationID: base.LocationID, Name: base.Name, Kind: base.Kind, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 0}}}}}, "end_unit_id must be positive"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := New().Encode(test.value); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Encode() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecAcceptsEveryOccurrenceKind(t *testing.T) {
|
||||
for _, kind := range []dnd.LocationOccurrenceKind{
|
||||
dnd.LocationOccurrenceKindVisited,
|
||||
dnd.LocationOccurrenceKindPlanned,
|
||||
dnd.LocationOccurrenceKindRecalled,
|
||||
dnd.LocationOccurrenceKindMentioned,
|
||||
} {
|
||||
value := validList()
|
||||
value.Occurrences[0].Kind = kind
|
||||
if _, err := New().Encode(value); err != nil {
|
||||
t.Fatalf("Encode(%q) error = %v", kind, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
12
internal/modules/dnd/codec/locationoccurrences/testdata/dnd_location_occurrences.v1.json
vendored
Normal file
12
internal/modules/dnd/codec/locationoccurrences/testdata/dnd_location_occurrences.v1.json
vendored
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"occurrences": [
|
||||
{
|
||||
"location_id": "location:sha256:cdd2b57615b56a4e92d506cf053cd3985e11d4df4895c5383b30ec2ce9f39ed0",
|
||||
"name": "The Old Tavern",
|
||||
"kind": "visited",
|
||||
"source_refs": [
|
||||
{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 2}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.locations",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["locations"],
|
||||
"properties": {
|
||||
"locations": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "name", "source_refs"],
|
||||
"properties": {
|
||||
"id": {
|
||||
"type": "string",
|
||||
"pattern": "^location:sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"name": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["source_id", "start_unit_id", "end_unit_id"],
|
||||
"properties": {
|
||||
"source_id": {
|
||||
"type": "string",
|
||||
"minLength": 1
|
||||
},
|
||||
"start_unit_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
},
|
||||
"end_unit_id": {
|
||||
"type": "integer",
|
||||
"minimum": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
111
internal/modules/dnd/codec/locations/codec.go
Normal file
111
internal/modules/dnd/codec/locations/codec.go
Normal file
@@ -0,0 +1,111 @@
|
||||
// Package locations encodes durable D&D location artifacts.
|
||||
package locations
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/candidatejson"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
const (
|
||||
SchemaID = "notarius.dnd.locations"
|
||||
SchemaName = "notarius_dnd_locations_v1"
|
||||
SchemaVersion = "v1"
|
||||
MediaType = "application/json"
|
||||
)
|
||||
|
||||
//go:embed assets/schemas/dnd_locations.v1.json
|
||||
var schemaAssets embed.FS
|
||||
|
||||
var _ contracts.ArtifactCodec[dnd.LocationList] = (*Codec)(nil)
|
||||
|
||||
type Codec struct{}
|
||||
|
||||
func New() *Codec { return &Codec{} }
|
||||
|
||||
func (c *Codec) Kind() contracts.ArtifactKind { return dnd.LocationListKind }
|
||||
|
||||
func (c *Codec) Schema() contracts.ArtifactSchema {
|
||||
raw, err := schemaAssets.ReadFile("assets/schemas/dnd_locations.v1.json")
|
||||
if err != nil {
|
||||
return contracts.ArtifactSchema{}
|
||||
}
|
||||
return contracts.ArtifactSchema{
|
||||
ID: SchemaID,
|
||||
Name: SchemaName,
|
||||
Version: SchemaVersion,
|
||||
JSONSchema: append([]byte(nil), raw...),
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Codec) MediaType() string { return MediaType }
|
||||
|
||||
func (c *Codec) Metadata(value dnd.LocationList) map[string]any {
|
||||
return map[string]any{"location_count": len(value.Locations)}
|
||||
}
|
||||
|
||||
func (c *Codec) Encode(value dnd.LocationList) ([]byte, error) {
|
||||
if err := validate(value); err != nil {
|
||||
return nil, fmt.Errorf("encode dnd location list: %w", err)
|
||||
}
|
||||
return c.EncodeCandidate(value)
|
||||
}
|
||||
|
||||
// EncodeCandidate provides the durable representation before semantic
|
||||
// validators have approved a value.
|
||||
func (c *Codec) EncodeCandidate(value dnd.LocationList) ([]byte, error) {
|
||||
return candidatejson.EncodeCandidate("dnd location list", value)
|
||||
}
|
||||
|
||||
func (c *Codec) Decode(content []byte) (dnd.LocationList, error) {
|
||||
value, err := c.DecodeCandidate(content)
|
||||
if err != nil {
|
||||
return dnd.LocationList{}, err
|
||||
}
|
||||
if err := validate(value); err != nil {
|
||||
return dnd.LocationList{}, fmt.Errorf("decode dnd location list: %w", err)
|
||||
}
|
||||
return value, nil
|
||||
}
|
||||
|
||||
// DecodeCandidate reads one strict durable JSON value before semantic
|
||||
// validators have approved it.
|
||||
func (c *Codec) DecodeCandidate(content []byte) (dnd.LocationList, error) {
|
||||
return candidatejson.DecodeCandidate[dnd.LocationList]("dnd location list", content)
|
||||
}
|
||||
|
||||
func validate(value dnd.LocationList) error {
|
||||
if value.Locations == nil {
|
||||
return fmt.Errorf("locations must be present")
|
||||
}
|
||||
for index, location := range value.Locations {
|
||||
prefix := fmt.Sprintf("locations[%d]", index)
|
||||
if !identity.IsValidID(location.ID) {
|
||||
return fmt.Errorf("%s.id must match location ID pattern", prefix)
|
||||
}
|
||||
if strings.TrimSpace(location.Name) == "" {
|
||||
return fmt.Errorf("%s.name must not be empty", prefix)
|
||||
}
|
||||
if len(location.SourceRefs) == 0 {
|
||||
return fmt.Errorf("%s.source_refs must contain at least one reference", prefix)
|
||||
}
|
||||
for refIndex, ref := range location.SourceRefs {
|
||||
refPrefix := fmt.Sprintf("%s.source_refs[%d]", prefix, refIndex)
|
||||
if strings.TrimSpace(ref.SourceID) == "" {
|
||||
return fmt.Errorf("%s.source_id must not be empty", refPrefix)
|
||||
}
|
||||
if ref.StartUnitID <= 0 {
|
||||
return fmt.Errorf("%s.start_unit_id must be positive", refPrefix)
|
||||
}
|
||||
if ref.EndUnitID <= 0 {
|
||||
return fmt.Errorf("%s.end_unit_id must be positive", refPrefix)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
146
internal/modules/dnd/codec/locations/codec_test.go
Normal file
146
internal/modules/dnd/codec/locations/codec_test.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
func validList() dnd.LocationList {
|
||||
refs := []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 2}}
|
||||
return dnd.LocationList{Locations: []dnd.Location{{
|
||||
ID: identity.DeriveID("The Old Tavern", refs),
|
||||
Name: "The Old Tavern",
|
||||
SourceRefs: refs,
|
||||
}}}
|
||||
}
|
||||
|
||||
func TestCodecMatchesMaintainedDurableFixture(t *testing.T) {
|
||||
raw, err := os.ReadFile("testdata/dnd_locations.v1.json")
|
||||
if err != nil {
|
||||
t.Fatalf("read durable fixture: %v", err)
|
||||
}
|
||||
codec := New()
|
||||
value, err := codec.Decode(raw)
|
||||
if err != nil {
|
||||
t.Fatalf("Decode() error = %v, want nil", err)
|
||||
}
|
||||
if want := validList(); !reflect.DeepEqual(value, want) {
|
||||
t.Fatalf("Decode() = %#v, want %#v", value, want)
|
||||
}
|
||||
encoded, err := codec.Encode(value)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v, want nil", err)
|
||||
}
|
||||
var compact bytes.Buffer
|
||||
if err := json.Compact(&compact, raw); err != nil {
|
||||
t.Fatalf("compact durable fixture: %v", err)
|
||||
}
|
||||
if !bytes.Equal(encoded, compact.Bytes()) {
|
||||
t.Fatalf("Encode() = %s, want %s", encoded, compact.Bytes())
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecOwnsDurableSchemaAndMetadata(t *testing.T) {
|
||||
codec := New()
|
||||
schema := codec.Schema()
|
||||
if codec.Kind() != dnd.LocationListKind || codec.MediaType() != MediaType {
|
||||
t.Fatalf("codec identity = %q/%q", codec.Kind(), codec.MediaType())
|
||||
}
|
||||
if schema.ID != SchemaID || schema.Name != SchemaName || schema.Version != SchemaVersion || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v, want durable location schema", schema)
|
||||
}
|
||||
var document map[string]any
|
||||
if err := json.Unmarshal(schema.JSONSchema, &document); err != nil || document["$id"] != SchemaID {
|
||||
t.Fatalf("durable schema document = %#v, %v", document, err)
|
||||
}
|
||||
registry := pipeline.NewArtifactCodecRegistry()
|
||||
if err := pipeline.RegisterArtifactCodec(registry, codec); err != nil {
|
||||
t.Fatalf("RegisterArtifactCodec() error = %v", err)
|
||||
}
|
||||
if spec, ok := registry.Spec(dnd.LocationListKind); !ok || spec.SchemaDigest != contracts.DigestArtifactSchema(schema) {
|
||||
t.Fatalf("registered spec = %#v, %t", spec, ok)
|
||||
}
|
||||
first := schema.JSONSchema
|
||||
first[0] = '['
|
||||
if next := codec.Schema().JSONSchema; !json.Valid(next) || next[0] == '[' {
|
||||
t.Fatal("Schema() returned shared bytes")
|
||||
}
|
||||
metadata := codec.Metadata(validList())
|
||||
metadata["other"] = true
|
||||
if next := codec.Metadata(validList()); len(next) != 1 || next["location_count"] != 1 {
|
||||
t.Fatalf("Metadata() = %#v", next)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecSupportsEmptyListsAndCandidateSemanticFailures(t *testing.T) {
|
||||
codec := New()
|
||||
empty := dnd.LocationList{Locations: []dnd.Location{}}
|
||||
if content, err := codec.Encode(empty); err != nil || string(content) != `{"locations":[]}` {
|
||||
t.Fatalf("Encode() = %s, %v", content, err)
|
||||
}
|
||||
for _, candidate := range []dnd.LocationList{
|
||||
{},
|
||||
empty,
|
||||
{Locations: []dnd.Location{{ID: "bad", Name: " ", SourceRefs: nil}}},
|
||||
{Locations: []dnd.Location{{ID: "bad", Name: " ", SourceRefs: []source.SourceRef{}}}},
|
||||
{Locations: []dnd.Location{{ID: "bad", Name: " ", SourceRefs: []source.SourceRef{{StartUnitID: 0, EndUnitID: -1}}}}},
|
||||
} {
|
||||
content, err := codec.EncodeCandidate(candidate)
|
||||
if err != nil || !json.Valid(content) {
|
||||
t.Fatalf("EncodeCandidate() = %s, %v", content, err)
|
||||
}
|
||||
decoded, err := codec.DecodeCandidate(content)
|
||||
if err != nil || !reflect.DeepEqual(decoded, candidate) {
|
||||
t.Fatalf("DecodeCandidate() = %#v, %v, want %#v", decoded, err, candidate)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecRejectsStructuralJSONBeforeSemanticApproval(t *testing.T) {
|
||||
valid := `{"locations":[{"id":"location:sha256:0000000000000000000000000000000000000000000000000000000000000000","name":"The Tavern","source_refs":[{"source_id":"session","start_unit_id":1,"end_unit_id":1}]}]}`
|
||||
for _, test := range []struct{ name, raw, want string }{
|
||||
{"malformed", `{`, "decode dnd location list"},
|
||||
{"unknown top-level", `{"locations":[],"unexpected":true}`, "unknown field"},
|
||||
{"unknown location field", strings.Replace(valid, `"name":"The Tavern"`, `"name":"The Tavern","unexpected":true`, 1), "unknown field"},
|
||||
{"unknown source reference field", strings.Replace(valid, `"end_unit_id":1`, `"end_unit_id":1,"unexpected":true`, 1), "unknown field"},
|
||||
{"invalid type", `{"locations":"not-an-array"}`, "cannot unmarshal string"},
|
||||
{"trailing", `{"locations":[]} {}`, "multiple JSON values"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := New().DecodeCandidate([]byte(test.raw)); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("DecodeCandidate() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestCodecRejectsApprovedShapeBoundaries(t *testing.T) {
|
||||
base := validList().Locations[0]
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
value dnd.LocationList
|
||||
want string
|
||||
}{
|
||||
{"nil locations", dnd.LocationList{}, "locations must be present"},
|
||||
{"invalid ID", dnd.LocationList{Locations: []dnd.Location{{ID: "bad", Name: base.Name, SourceRefs: base.SourceRefs}}}, "id must match location ID pattern"},
|
||||
{"blank name", dnd.LocationList{Locations: []dnd.Location{{ID: base.ID, Name: " ", SourceRefs: base.SourceRefs}}}, "name must not be empty"},
|
||||
{"empty source refs", dnd.LocationList{Locations: []dnd.Location{{ID: base.ID, Name: base.Name, SourceRefs: nil}}}, "source_refs must contain"},
|
||||
{"malformed source reference", dnd.LocationList{Locations: []dnd.Location{{ID: base.ID, Name: base.Name, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 0, EndUnitID: 1}}}}}, "start_unit_id must be positive"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := New().Encode(test.value); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Encode() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
11
internal/modules/dnd/codec/locations/testdata/dnd_locations.v1.json
vendored
Normal file
11
internal/modules/dnd/codec/locations/testdata/dnd_locations.v1.json
vendored
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"locations": [
|
||||
{
|
||||
"id": "location:sha256:cdd2b57615b56a4e92d506cf053cd3985e11d4df4895c5383b30ec2ce9f39ed0",
|
||||
"name": "The Old Tavern",
|
||||
"source_refs": [
|
||||
{"source_id": "session-alpha", "start_unit_id": 1, "end_unit_id": 2}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
package locationoccurrences
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/prompts/*.yaml assets/prompts/*.md assets/schemas/*.json
|
||||
var embeddedAssets embed.FS
|
||||
@@ -0,0 +1,47 @@
|
||||
id: dnd.location_occurrences
|
||||
version: "v1"
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
- name: players
|
||||
required: false
|
||||
content_type: text/plain
|
||||
- name: party
|
||||
required: false
|
||||
content_type: text/plain
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/plain
|
||||
- name: locations
|
||||
required: true
|
||||
content_type: application/json
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./sharedassets/common-dnd-system.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-identity.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-references.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-transcript.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-extraction-evidence.md
|
||||
- role: user
|
||||
content_file: ./locations.md
|
||||
- role: user
|
||||
content_file: ./task.md
|
||||
- role: user
|
||||
content_file: ./instructions.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: dnd_location_occurrences_llm.v1.json
|
||||
repair_attempts: 0
|
||||
@@ -0,0 +1,8 @@
|
||||
Return the occurrences array even when no occurrence is established. Every
|
||||
record must contain location_id, name, kind, and source_refs. Copy location_id
|
||||
and name from one supplied registry record, and cite only narrow transcript
|
||||
ranges that support both that location and its classified occurrence.
|
||||
|
||||
Do not summarize location descriptions, infer a missing registry record, or
|
||||
use registry context as evidence. Omit source_id; Notarius assigns the current
|
||||
transcript source identity.
|
||||
@@ -0,0 +1,9 @@
|
||||
A normalized location registry is provided below for identity grounding. It may
|
||||
be empty. Each record contains the exact location ID and canonical display name
|
||||
to copy when the transcript establishes an occurrence of that place.
|
||||
|
||||
Registry content is context, not occurrence evidence. Do not derive an
|
||||
occurrence or a source range from the registry, and do not infer a location
|
||||
that is absent from it.
|
||||
|
||||
{{ input "locations" }}
|
||||
@@ -0,0 +1,29 @@
|
||||
Extract Dungeons & Dragons location occurrences from the supplied transcript.
|
||||
|
||||
Include an occurrence only when the transcript establishes one supplied
|
||||
location, one occurrence kind, and a coherent passage supporting both. Use
|
||||
only the exact ID and name pair from the supplied location registry. Return an
|
||||
empty occurrences array when no supplied location has an evidenced occurrence
|
||||
in this transcript passage.
|
||||
|
||||
Use exactly one kind per occurrence:
|
||||
|
||||
- visited: party members are physically present, arrive, remain, or depart;
|
||||
- planned: the party explicitly proposes, intends, or agrees to future travel;
|
||||
- recalled: the transcript explicitly recounts an earlier party visit; or
|
||||
- mentioned: the location is explicitly referenced without stronger support,
|
||||
including non-actionable speculation or a mere hypothetical reference.
|
||||
|
||||
A mere hypothetical or speculative reference is not planned unless the
|
||||
transcript also establishes an actual proposal, intention, or agreement to
|
||||
travel. When the hypothetical itself explicitly names a supplied registry
|
||||
location, it may be mentioned using the narrow passage that supports that
|
||||
reference.
|
||||
|
||||
For overlapping support, visited outranks planned, recalled, and mentioned;
|
||||
planned outranks recalled and mentioned; recalled outranks mentioned. A passage
|
||||
may produce multiple records when it independently establishes separate facts,
|
||||
such as recalling an earlier visit while planning a return. Omit inferred,
|
||||
unstated, uncertain, or unsupported places and occurrences. Do not infer a
|
||||
location or occurrence from surrounding events when the transcript does not
|
||||
state it.
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.location_occurrences.llm",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["occurrences"],
|
||||
"properties": {
|
||||
"occurrences": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["location_id", "name", "kind", "source_refs"],
|
||||
"properties": {
|
||||
"location_id": {"type": "string"},
|
||||
"name": {"type": "string"},
|
||||
"kind": {"enum": ["visited", "planned", "recalled", "mentioned"]},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["start_unit_id", "end_unit_id"],
|
||||
"properties": {
|
||||
"start_unit_id": {"type": "integer"},
|
||||
"end_unit_id": {"type": "integer"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
106
internal/modules/dnd/extract/locationoccurrences/canonicalize.go
Normal file
106
internal/modules/dnd/extract/locationoccurrences/canonicalize.go
Normal file
@@ -0,0 +1,106 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
type orderedOccurrence struct {
|
||||
value dnd.LocationOccurrence
|
||||
earliest int
|
||||
hasEvidence bool
|
||||
}
|
||||
|
||||
func canonicalOccurrenceList(response extractionResponse, order shared.SourceRefOrder, sourceID string) dnd.LocationOccurrenceList {
|
||||
if response.Occurrences == nil {
|
||||
return dnd.LocationOccurrenceList{}
|
||||
}
|
||||
ordered := make([]orderedOccurrence, len(response.Occurrences))
|
||||
for index, occurrence := range response.Occurrences {
|
||||
refs := order.Canonicalize(canonicalSourceRefs(occurrence.SourceRefs, sourceID))
|
||||
earliest, hasEvidence := order.EarliestValid(refs)
|
||||
ordered[index] = orderedOccurrence{value: dnd.LocationOccurrence{
|
||||
LocationID: occurrence.LocationID,
|
||||
Name: occurrence.Name,
|
||||
Kind: dnd.LocationOccurrenceKind(occurrence.Kind),
|
||||
SourceRefs: refs,
|
||||
}, earliest: earliest, hasEvidence: hasEvidence}
|
||||
}
|
||||
sort.SliceStable(ordered, func(left, right int) bool {
|
||||
return lessOccurrence(ordered[left], ordered[right], order)
|
||||
})
|
||||
occurrences := make([]dnd.LocationOccurrence, 0, len(ordered))
|
||||
for _, occurrence := range ordered {
|
||||
if len(occurrences) == 0 || !sameOccurrence(occurrences[len(occurrences)-1], occurrence.value) {
|
||||
occurrences = append(occurrences, occurrence.value)
|
||||
}
|
||||
}
|
||||
return dnd.LocationOccurrenceList{Occurrences: occurrences}
|
||||
}
|
||||
|
||||
func lessOccurrence(left, right orderedOccurrence, order shared.SourceRefOrder) bool {
|
||||
if left.hasEvidence != right.hasEvidence {
|
||||
return left.hasEvidence
|
||||
}
|
||||
if left.hasEvidence && left.earliest != right.earliest {
|
||||
return left.earliest < right.earliest
|
||||
}
|
||||
if left.value.LocationID != right.value.LocationID {
|
||||
return left.value.LocationID < right.value.LocationID
|
||||
}
|
||||
if left.value.Name != right.value.Name {
|
||||
return left.value.Name < right.value.Name
|
||||
}
|
||||
if kindOrder(left.value.Kind) != kindOrder(right.value.Kind) {
|
||||
return kindOrder(left.value.Kind) < kindOrder(right.value.Kind)
|
||||
}
|
||||
return lessReferences(left.value.SourceRefs, right.value.SourceRefs, order)
|
||||
}
|
||||
|
||||
func kindOrder(kind dnd.LocationOccurrenceKind) int {
|
||||
switch kind {
|
||||
case dnd.LocationOccurrenceKindVisited:
|
||||
return 0
|
||||
case dnd.LocationOccurrenceKindPlanned:
|
||||
return 1
|
||||
case dnd.LocationOccurrenceKindRecalled:
|
||||
return 2
|
||||
case dnd.LocationOccurrenceKindMentioned:
|
||||
return 3
|
||||
default:
|
||||
return 4
|
||||
}
|
||||
}
|
||||
|
||||
func lessReferences(left, right []source.SourceRef, order shared.SourceRefOrder) bool {
|
||||
limit := len(left)
|
||||
if len(right) < limit {
|
||||
limit = len(right)
|
||||
}
|
||||
for index := 0; index < limit; index++ {
|
||||
if left[index] == right[index] {
|
||||
continue
|
||||
}
|
||||
return order.Less(left[index], right[index])
|
||||
}
|
||||
return len(left) < len(right)
|
||||
}
|
||||
|
||||
func sameOccurrence(left, right dnd.LocationOccurrence) bool {
|
||||
return left.LocationID == right.LocationID && left.Name == right.Name && left.Kind == right.Kind && reflect.DeepEqual(left.SourceRefs, right.SourceRefs)
|
||||
}
|
||||
|
||||
func canonicalSourceRefs(values []occurrenceSourceRefResponse, sourceID string) []source.SourceRef {
|
||||
if values == nil {
|
||||
return nil
|
||||
}
|
||||
refs := make([]source.SourceRef, len(values))
|
||||
for index, value := range values {
|
||||
refs[index] = source.SourceRef{SourceID: sourceID, StartUnitID: value.StartUnitID, EndUnitID: value.EndUnitID}
|
||||
}
|
||||
return refs
|
||||
}
|
||||
188
internal/modules/dnd/extract/locationoccurrences/extractor.go
Normal file
188
internal/modules/dnd/extract/locationoccurrences/extractor.go
Normal file
@@ -0,0 +1,188 @@
|
||||
// Package locationoccurrences extracts source-grounded D&D location occurrences.
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationregistry "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/registry"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const (
|
||||
Key = "dnd/location-occurrences"
|
||||
mappingPolicy = "dnd.location_occurrences.extract_mapping.v1"
|
||||
)
|
||||
|
||||
const (
|
||||
LocationRegistryReferenceSlot = locationregistry.ReferenceSlot
|
||||
LocationRegistryMaxBytes = locationregistry.MaxBytes
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"chunks", "source.transcript"}
|
||||
var providedCapabilities = []string{"dnd.location_occurrences"}
|
||||
|
||||
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
||||
Glossary: "Optional campaign glossary reference material used only for location-occurrence disambiguation.",
|
||||
Party: "Optional party roster reference material used only for location-occurrence disambiguation.",
|
||||
Players: "Optional player list reference material used only for location-occurrence disambiguation.",
|
||||
Roster: "Deprecated alias for party roster reference material used only for location-occurrence disambiguation.",
|
||||
}
|
||||
|
||||
func referenceSlots() []contracts.ReferenceSlot {
|
||||
slots := shared.ReferenceSlots(referenceSlotDescriptions)
|
||||
slots = append(slots, contracts.ReferenceSlot{
|
||||
Name: LocationRegistryReferenceSlot,
|
||||
Description: "Required normalized location registry used only for location identity grounding, never as occurrence evidence.",
|
||||
Required: true,
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.LocationListKind},
|
||||
MaxBytes: LocationRegistryMaxBytes,
|
||||
})
|
||||
sort.Slice(slots, func(left, right int) bool { return slots[left].Name < slots[right].Name })
|
||||
return slots
|
||||
}
|
||||
|
||||
var _ contracts.Extractor[dnd.LocationOccurrenceList] = (*Extractor)(nil)
|
||||
var _ contracts.ManifestMetadataProvider = (*Extractor)(nil)
|
||||
var _ pipeline.CheckpointFingerprintProvider = (*Extractor)(nil)
|
||||
|
||||
type Options struct{}
|
||||
|
||||
type Extractor struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
locationResolver *locationregistry.Resolver
|
||||
promptSHA string
|
||||
responseSchemaSHA string
|
||||
}
|
||||
|
||||
func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contracts.ReferenceSet) (*Extractor, error) {
|
||||
if llmClient == nil {
|
||||
return nil, extractorErrorf("LLM client must not be nil")
|
||||
}
|
||||
if len(references) > 1 {
|
||||
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||
}
|
||||
var referenceSet contracts.ReferenceSet
|
||||
if len(references) == 1 {
|
||||
referenceSet = references[0]
|
||||
}
|
||||
locationResolver, err := locationregistry.NewResolver(referenceSet)
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("prepare location registry prompt input: %w", err)
|
||||
}
|
||||
promptSHA, err := promptAssetMetadata()
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||
}
|
||||
responseSchema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("load response schema: %w", err)
|
||||
}
|
||||
return &Extractor{llm: llmClient, locationResolver: locationResolver, promptSHA: promptSHA, responseSchemaSHA: responseSchema.SHA256}, nil
|
||||
}
|
||||
|
||||
func (e *Extractor) Key() string { return Key }
|
||||
|
||||
func (e *Extractor) ReferenceSlots() []contracts.ReferenceSlot { return referenceSlots() }
|
||||
|
||||
func (e *Extractor) ManifestMetadata() map[string]any {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
metadata := map[string]any{
|
||||
"prompt_id": PromptID,
|
||||
"prompt_version": SchemaVersion,
|
||||
"prompt_sha256": e.promptSHA,
|
||||
"mapping_policy": mappingPolicy,
|
||||
"response_schema_key": string(ResponseSchemaKey),
|
||||
"response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName,
|
||||
"response_schema_version": SchemaVersion,
|
||||
"response_schema_sha256": e.responseSchemaSHA,
|
||||
}
|
||||
seeded := e.locationResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
metadata["location_registry_digest"] = seeded.Digest()
|
||||
metadata["location_count"] = seeded.Count()
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return []pipeline.CheckpointFingerprint{
|
||||
{Name: "prompt", Value: e.promptSHA},
|
||||
{Name: "response_schema", Value: e.responseSchemaSHA},
|
||||
{Name: "mapping_policy", Value: mappingPolicy},
|
||||
{Name: "location_registry", Value: e.locationResolver.Seeded().ProjectionDigest()},
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.LocationOccurrenceList], error) {
|
||||
if e == nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{}, extractorErrorf("extractor must not be nil")
|
||||
}
|
||||
if e.llm == nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{}, extractorErrorf("LLM client must not be nil")
|
||||
}
|
||||
sourceInput, err := shared.PrepareChunkExtraction(ctx, req)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{}, extractorErrorf("%w", err)
|
||||
}
|
||||
registry, err := e.locationResolver.Resolve(req.References)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{}, extractorErrorf("resolve location registry: %w", err)
|
||||
}
|
||||
if !registry.Bound() {
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{}, extractorErrorf("location registry reference is required")
|
||||
}
|
||||
|
||||
var response extractionResponse
|
||||
inputs := shared.PromptInputs(sourceInput, req.References)
|
||||
inputs[LocationRegistryReferenceSlot] = registry.PromptInput()
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID, Inputs: inputs,
|
||||
}, &response); err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{}, extractorErrorf("complete structured output: %w", err)
|
||||
}
|
||||
return contracts.TypedExtractionResult[dnd.LocationOccurrenceList]{Value: canonicalOccurrenceList(response, shared.NewSourceRefOrder(req.Source), req.Source.ID)}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.LocationOccurrenceListKind, ReferenceSlots: referenceSlots(),
|
||||
}
|
||||
}
|
||||
|
||||
func Register(registry *pipeline.ExtractorRegistry) error {
|
||||
return pipeline.RegisterExtractorBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Extractor[dnd.LocationOccurrenceList], error) {
|
||||
options, err := DecodeOptions(request.Options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(request.Dependencies.LLM, options, request.References)
|
||||
})
|
||||
}
|
||||
|
||||
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||
|
||||
func DecodeOptions(options map[string]any) (Options, error) {
|
||||
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||
return Options{}, extractorErrorf("%w", err)
|
||||
}
|
||||
return Options{}, nil
|
||||
}
|
||||
|
||||
func extractorErrorf(format string, args ...any) error {
|
||||
return fmt.Errorf("dnd location occurrences extractor: "+format, args...)
|
||||
}
|
||||
@@ -0,0 +1,243 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locations"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
func TestExtractMapsKindsOrdersOccurrencesAndPreservesIndependentFacts(t *testing.T) {
|
||||
locations := locationRegistry(t, "The Tavern", "The Tavern")
|
||||
first, second := locations.Locations[0], locations.Locations[1]
|
||||
client := &fakeOccurrencesLLMClient{response: extractionResponse{Occurrences: []occurrenceResponse{
|
||||
{LocationID: second.ID, Name: second.Name, Kind: "mentioned", SourceRefs: occurrenceRefs(30, 30)},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: "mentioned", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: "recalled", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: "planned", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: "visited", SourceRefs: append(occurrenceRefs(10, 10), occurrenceRefs(10, 10)...)},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: "visited", SourceRefs: occurrenceRefs(20, 20)},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: "visited", SourceRefs: occurrenceRefs(10, 10)},
|
||||
}}}
|
||||
references := registryReferences(t, locations)
|
||||
req := extractionRequest()
|
||||
req.References = references
|
||||
result, err := newExtractor(t, client, references).Extract(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(result.Value.Occurrences) != 6 {
|
||||
t.Fatalf("occurrences = %#v, want exact duplicate removed", result.Value.Occurrences)
|
||||
}
|
||||
got := result.Value.Occurrences
|
||||
if kinds := []dnd.LocationOccurrenceKind{got[0].Kind, got[1].Kind, got[2].Kind, got[3].Kind}; !reflect.DeepEqual(kinds, []dnd.LocationOccurrenceKind{dnd.LocationOccurrenceKindVisited, dnd.LocationOccurrenceKindPlanned, dnd.LocationOccurrenceKindRecalled, dnd.LocationOccurrenceKindMentioned}) {
|
||||
t.Fatalf("same-evidence kind order = %#v", kinds)
|
||||
}
|
||||
if got[4].Kind != dnd.LocationOccurrenceKindVisited || got[4].SourceRefs[0].StartUnitID != 20 || got[5].LocationID != second.ID || got[5].SourceRefs[0].StartUnitID != 30 {
|
||||
t.Fatalf("occurrence order = %#v", got)
|
||||
}
|
||||
if !reflect.DeepEqual(got[0].SourceRefs, []source.SourceRef{{SourceID: req.Source.ID, StartUnitID: 10, EndUnitID: 10}}) {
|
||||
t.Fatalf("canonical evidence = %#v", got[0].SourceRefs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractUsesIDsNamesAndCurrentTranscriptEvidenceOnly(t *testing.T) {
|
||||
locations := locationRegistry(t, "The Tavern", "The Tavern")
|
||||
first, second := locations.Locations[0], locations.Locations[1]
|
||||
client := &fakeOccurrencesLLMClient{response: extractionResponse{Occurrences: []occurrenceResponse{{
|
||||
LocationID: second.ID, Name: second.Name, Kind: "visited", SourceRefs: occurrenceRefs(10, 10),
|
||||
}}}}
|
||||
references := registryReferences(t, locations)
|
||||
req := extractionRequest()
|
||||
req.References = references
|
||||
result, err := newExtractor(t, client, references).Extract(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if occurrence := result.Value.Occurrences[0]; occurrence.LocationID != second.ID || occurrence.Name != second.Name || occurrence.SourceRefs[0].SourceID != req.Source.ID {
|
||||
t.Fatalf("occurrence = %#v", occurrence)
|
||||
}
|
||||
input := client.requests[0].Inputs[LocationRegistryReferenceSlot]
|
||||
if input.Name != LocationRegistryReferenceSlot || !strings.Contains(string(input.Content), first.ID) || !strings.Contains(string(input.Content), second.ID) {
|
||||
t.Fatalf("location prompt input = %#v", input)
|
||||
}
|
||||
for _, forbidden := range []string{"source_refs", "source_id", "other-session"} {
|
||||
if strings.Contains(string(input.Content), forbidden) {
|
||||
t.Fatalf("location prompt leaked %q: %s", forbidden, input.Content)
|
||||
}
|
||||
}
|
||||
if strings.Contains(string(client.requests[0].Inputs["transcript"].Content), "other-session") {
|
||||
t.Fatal("transcript input contains registry evidence")
|
||||
}
|
||||
metadata, err := json.Marshal(newExtractor(t, &fakeOccurrencesLLMClient{}, references).ManifestMetadata())
|
||||
if err != nil || strings.Contains(string(metadata), "other-session") || strings.Contains(string(metadata), first.ID) {
|
||||
t.Fatalf("manifest metadata = %s, %v", metadata, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractPreservesUnknownOrMismatchedGroundingForValidators(t *testing.T) {
|
||||
locations := locationRegistry(t, "The Mill")
|
||||
known := locations.Locations[0]
|
||||
client := &fakeOccurrencesLLMClient{response: extractionResponse{Occurrences: []occurrenceResponse{
|
||||
{LocationID: "location:sha256:unknown", Name: "The Mill", Kind: "mentioned", SourceRefs: occurrenceRefs(10, 10)},
|
||||
{LocationID: known.ID, Name: "A Different Mill", Kind: "mentioned", SourceRefs: occurrenceRefs(20, 20)},
|
||||
}}}
|
||||
references := registryReferences(t, locations)
|
||||
req := extractionRequest()
|
||||
req.References = references
|
||||
result, err := newExtractor(t, client, references).Extract(context.Background(), req)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if result.Value.Occurrences[0].LocationID != "location:sha256:unknown" || result.Value.Occurrences[1].Name != "A Different Mill" {
|
||||
t.Fatalf("extractor repaired validator-owned grounding errors: %#v", result.Value.Occurrences)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractRequiresRegistryAndAcceptsEmptyRegistryWithNoOccurrences(t *testing.T) {
|
||||
client := &fakeOccurrencesLLMClient{response: extractionResponse{Occurrences: []occurrenceResponse{}}}
|
||||
if _, err := newExtractor(t, client).Extract(context.Background(), extractionRequest()); err == nil || !strings.Contains(err.Error(), "location registry reference is required") {
|
||||
t.Fatalf("Extract() error = %v", err)
|
||||
}
|
||||
if len(client.requests) != 0 {
|
||||
t.Fatalf("LLM calls = %d", len(client.requests))
|
||||
}
|
||||
empty := dnd.LocationList{Locations: []dnd.Location{}}
|
||||
references := registryReferences(t, empty)
|
||||
req := extractionRequest()
|
||||
req.References = references
|
||||
result, err := newExtractor(t, client, references).Extract(context.Background(), req)
|
||||
if err != nil || result.Value.Occurrences == nil || len(result.Value.Occurrences) != 0 {
|
||||
t.Fatalf("empty registry result = %#v, %v", result, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractResolvesGeneratedRegistryAtOperationTimeAndDoesNotMutateResponse(t *testing.T) {
|
||||
locations := locationRegistry(t, "The Mill")
|
||||
location := locations.Locations[0]
|
||||
client := &fakeOccurrencesLLMClient{response: extractionResponse{Occurrences: []occurrenceResponse{{
|
||||
LocationID: location.ID, Name: location.Name, Kind: "mentioned", SourceRefs: occurrenceRefs(30, 30),
|
||||
}}}}
|
||||
references := registryReferences(t, locations)
|
||||
req := extractionRequest()
|
||||
req.References = references
|
||||
extractor := newExtractor(t, client)
|
||||
result, err := extractor.Extract(context.Background(), req)
|
||||
if err != nil || result.Value.Occurrences[0].Name != "The Mill" {
|
||||
t.Fatalf("Extract() = %#v, %v", result, err)
|
||||
}
|
||||
if input := client.requests[0].Inputs[LocationRegistryReferenceSlot]; !strings.Contains(string(input.Content), location.ID) || input.OriginURI != "" {
|
||||
t.Fatalf("generated registry prompt input = %#v", input)
|
||||
}
|
||||
if _, ok := extractor.ManifestMetadata()["location_registry_digest"]; ok {
|
||||
t.Fatalf("operation registry leaked into static metadata: %#v", extractor.ManifestMetadata())
|
||||
}
|
||||
if client.response.Occurrences[0].SourceRefs[0].StartUnitID != 30 {
|
||||
t.Fatalf("model response mutated: %#v", client.response)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractorContractsMetadataAndFailures(t *testing.T) {
|
||||
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
|
||||
t.Fatalf("New(nil) error = %v", err)
|
||||
}
|
||||
if _, err := New(&fakeOccurrencesLLMClient{}, Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one reference set") {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
malformed := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{LocationRegistryReferenceSlot: {
|
||||
Items: []contracts.ReferenceItem{{SlotName: LocationRegistryReferenceSlot, MediaType: "application/json", Content: []byte(`{"secret":"registry evidence"}`)}},
|
||||
}}}
|
||||
if _, err := New(&fakeOccurrencesLLMClient{}, Options{}, malformed); err == nil || !strings.Contains(err.Error(), "prepare location registry") || strings.Contains(err.Error(), "registry evidence") {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
|
||||
locations := locationRegistry(t, "The Mill")
|
||||
references := registryReferences(t, locations)
|
||||
req := extractionRequest()
|
||||
req.References = references
|
||||
extractor := newExtractor(t, &fakeOccurrencesLLMClient{}, references)
|
||||
var nilExtractor *Extractor
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
extractor *Extractor
|
||||
req contracts.TypedExtractionRequest
|
||||
want string
|
||||
}{
|
||||
{"nil extractor", nilExtractor, req, "extractor"},
|
||||
{"nil client", &Extractor{}, req, "LLM client"},
|
||||
{"invalid request", extractor, mismatchedSourceInputRequest(req), "must match chunk"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := test.extractor.Extract(context.Background(), test.req); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Extract() error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
if _, err := newExtractor(t, &fakeOccurrencesLLMClient{err: errors.New("provider unavailable")}, references).Extract(context.Background(), req); err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
||||
t.Fatalf("provider error = %v", err)
|
||||
}
|
||||
|
||||
spec := ModuleSpec()
|
||||
if spec.Key != Key || spec.Stage != pipeline.StageExtract || spec.ExecutionClass != contracts.ExecutionClassLLMBacked || spec.ArtifactKind != dnd.LocationOccurrenceListKind {
|
||||
t.Fatalf("ModuleSpec() = %#v", spec)
|
||||
}
|
||||
var slot contracts.ReferenceSlot
|
||||
for _, candidate := range spec.ReferenceSlots {
|
||||
if candidate.Name == LocationRegistryReferenceSlot {
|
||||
slot = candidate
|
||||
}
|
||||
}
|
||||
if !slot.Required || !reflect.DeepEqual(slot.AcceptedArtifactKinds, []contracts.ArtifactKind{dnd.LocationListKind}) || slot.MaxBytes != LocationRegistryMaxBytes {
|
||||
t.Fatalf("location registry slot = %#v", slot)
|
||||
}
|
||||
registry := pipeline.NewExtractorRegistry()
|
||||
if err := Register(registry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, ok := registry.Spec(Key); !ok {
|
||||
t.Fatalf("registration missing %q", Key)
|
||||
}
|
||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||
t.Fatal("DecodeOptions() accepted unknown options")
|
||||
}
|
||||
metadata := newExtractor(t, &fakeOccurrencesLLMClient{}, references).ManifestMetadata()
|
||||
for _, key := range []string{"prompt_sha256", "response_schema_sha256", "location_registry_digest"} {
|
||||
if value, ok := metadata[key].(string); !ok || !strings.HasPrefix(value, "sha256:") {
|
||||
t.Fatalf("metadata[%q] = %#v", key, metadata[key])
|
||||
}
|
||||
}
|
||||
if got := newExtractor(t, &fakeOccurrencesLLMClient{}, references).CheckpointFingerprints(); len(got) != 4 || got[3].Name != "location_registry" {
|
||||
t.Fatalf("fingerprints = %#v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func locationRegistry(t *testing.T, names ...string) dnd.LocationList {
|
||||
t.Helper()
|
||||
locations := make([]dnd.Location, len(names))
|
||||
for index, name := range names {
|
||||
refs := []source.SourceRef{{SourceID: "other-session", StartUnitID: index + 1, EndUnitID: index + 1}}
|
||||
locations[index] = dnd.Location{ID: identity.DeriveID(name, refs), Name: name, SourceRefs: refs}
|
||||
}
|
||||
return dnd.LocationList{Locations: locations}
|
||||
}
|
||||
|
||||
func registryReferences(t *testing.T, locations dnd.LocationList) contracts.ReferenceSet {
|
||||
t.Helper()
|
||||
content, err := locationcodec.New().Encode(locations)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{LocationRegistryReferenceSlot: {
|
||||
Slot: contracts.ReferenceSlot{Name: LocationRegistryReferenceSlot},
|
||||
Items: []contracts.ReferenceItem{{SlotName: LocationRegistryReferenceSlot, MediaType: locationcodec.MediaType, Content: content, Origin: contracts.ReferenceOrigin{Type: "generated"}}},
|
||||
}}}
|
||||
}
|
||||
17
internal/modules/dnd/extract/locationoccurrences/model.go
Normal file
17
internal/modules/dnd/extract/locationoccurrences/model.go
Normal file
@@ -0,0 +1,17 @@
|
||||
package locationoccurrences
|
||||
|
||||
type extractionResponse struct {
|
||||
Occurrences []occurrenceResponse `json:"occurrences"`
|
||||
}
|
||||
|
||||
type occurrenceResponse struct {
|
||||
LocationID string `json:"location_id"`
|
||||
Name string `json:"name"`
|
||||
Kind string `json:"kind"`
|
||||
SourceRefs []occurrenceSourceRefResponse `json:"source_refs"`
|
||||
}
|
||||
|
||||
type occurrenceSourceRefResponse struct {
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const promptAssetRoot = "assets/prompts"
|
||||
|
||||
var promptAssetManifest = shared.PromptAssetManifest{
|
||||
ModuleDir: PromptID,
|
||||
ModuleFiles: []promptfs.ModulePromptFile{
|
||||
{Name: "dnd.location_occurrences.yaml", Path: "assets/prompts/dnd.location_occurrences.yaml"},
|
||||
{Name: "locations.md", Path: "assets/prompts/locations.md"},
|
||||
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
||||
},
|
||||
SharedFiles: []string{
|
||||
"common-dnd-system.md",
|
||||
"common-dnd-extraction-evidence.md",
|
||||
"common-dnd-identity.md",
|
||||
"common-dnd-transcript.md",
|
||||
"common-dnd-references.md",
|
||||
},
|
||||
}
|
||||
|
||||
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
||||
promptFS, err := promptAssetManifest.PromptFS(embeddedAssets)
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare location-occurrence prompt assets: %w", err)
|
||||
}
|
||||
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||
}
|
||||
|
||||
func promptAssetMetadata() (string, error) {
|
||||
promptAssetHashOnce.Do(func() {
|
||||
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||
})
|
||||
return promptAssetHash, promptAssetHashErr
|
||||
}
|
||||
|
||||
var (
|
||||
promptAssetHashOnce sync.Once
|
||||
promptAssetHash string
|
||||
promptAssetHashErr error
|
||||
)
|
||||
@@ -0,0 +1,64 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestRegisterPromptAssetsPreparesLocationOccurrencePrompt(t *testing.T) {
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := RegisterPromptAssets(registry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
options, err := registry.PromptKitOptions()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{
|
||||
ID: "location-occurrences-test", Endpoint: "http://127.0.0.1:1/v1", Model: "test",
|
||||
})))
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "location-occurrences-test",
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"transcript": promptkit.Inline(`{"units":[1]}`), "players": promptkit.Inline(" "), "party": promptkit.Inline(" "), "glossary": promptkit.Inline(" "),
|
||||
"locations": promptkit.Inline(`{"locations":[{"id":"location:sha256:test","name":"The Mill"}]}`),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_location_occurrences_llm.v1.json" {
|
||||
t.Fatalf("prepared prompt = %#v", prepared)
|
||||
}
|
||||
var registryMessage string
|
||||
registryIndex := -1
|
||||
evidenceIndex := -1
|
||||
taskIndex := -1
|
||||
for index, message := range prepared.Messages {
|
||||
if strings.Contains(message.Content, "normalized location registry") {
|
||||
registryMessage = message.Content
|
||||
registryIndex = index
|
||||
}
|
||||
if strings.Contains(message.Content, "Transcript units are the only evidence") {
|
||||
evidenceIndex = index
|
||||
}
|
||||
if strings.Contains(message.Content, "Extract Dungeons & Dragons location occurrences") {
|
||||
taskIndex = index
|
||||
}
|
||||
}
|
||||
if !strings.Contains(registryMessage, "location:sha256:test") || !strings.Contains(registryMessage, "The Mill") || strings.Contains(registryMessage, "source_refs") {
|
||||
t.Fatalf("rendered prompt did not preserve source-free registry grounding: %s", registryMessage)
|
||||
}
|
||||
if evidenceIndex < 0 || taskIndex < 0 || registryIndex <= evidenceIndex || registryIndex >= taskIndex {
|
||||
t.Fatalf("registry prompt placement = evidence %d, registry %d, task %d", evidenceIndex, registryIndex, taskIndex)
|
||||
}
|
||||
}
|
||||
21
internal/modules/dnd/extract/locationoccurrences/schema.go
Normal file
21
internal/modules/dnd/extract/locationoccurrences/schema.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package locationoccurrences
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
|
||||
const (
|
||||
PromptID = "dnd.location_occurrences"
|
||||
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_location_occurrences_llm")
|
||||
ResponseSchemaID = "notarius.dnd.location_occurrences.llm"
|
||||
ResponseSchemaName = "notarius_dnd_location_occurrences_llm_v1"
|
||||
SchemaVersion = "v1"
|
||||
)
|
||||
|
||||
func loadResponseSchema() (llm.ResponseSchema, error) {
|
||||
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
|
||||
Key: ResponseSchemaKey,
|
||||
ID: ResponseSchemaID,
|
||||
Version: SchemaVersion,
|
||||
Name: ResponseSchemaName,
|
||||
AssetPath: "assets/schemas/dnd_location_occurrences_llm.v1.json",
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,96 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
)
|
||||
|
||||
func TestResponseSchemaRestrictsPrivateOccurrenceStructureAndKinds(t *testing.T) {
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Name != ResponseSchemaName || schema.Version != SchemaVersion || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v", schema)
|
||||
}
|
||||
valid := map[string]any{"occurrences": []any{map[string]any{
|
||||
"location_id": "location:sha256:test", "name": "The Mill", "kind": "visited",
|
||||
"source_refs": []any{map[string]any{"start_unit_id": 1, "end_unit_id": 2}},
|
||||
}}}
|
||||
if err := validateSchema(t, valid, schema.JSONSchema); err != nil {
|
||||
t.Fatalf("valid response rejected: %v", err)
|
||||
}
|
||||
for _, mutate := range []func(map[string]any){
|
||||
func(value map[string]any) { delete(value, "location_id") },
|
||||
func(value map[string]any) { value["kind"] = "other" },
|
||||
func(value map[string]any) { value["unexpected"] = true },
|
||||
func(value map[string]any) {
|
||||
value["source_refs"].([]any)[0].(map[string]any)["source_id"] = "assigned later"
|
||||
},
|
||||
} {
|
||||
candidate := cloneCandidate(t, valid)
|
||||
mutate(candidate["occurrences"].([]any)[0].(map[string]any))
|
||||
if err := validateSchema(t, candidate, schema.JSONSchema); err == nil {
|
||||
t.Fatal("schema accepted structurally invalid response")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestResponseSchemaIsDefensiveAndContentSafe(t *testing.T) {
|
||||
first, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
first.JSONSchema[0] = '['
|
||||
second, err := loadResponseSchema()
|
||||
if err != nil || !json.Valid(second.JSONSchema) || bytes.Equal(first.JSONSchema, second.JSONSchema) {
|
||||
t.Fatalf("schema = %s, %v", second.JSONSchema, err)
|
||||
}
|
||||
if diagnostics := second.DiagnosticsMap(); diagnostics["key"] != ResponseSchemaKey || diagnostics["id"] != ResponseSchemaID {
|
||||
t.Fatalf("diagnostics = %#v", diagnostics)
|
||||
} else if _, ok := diagnostics["json_schema"]; ok {
|
||||
t.Fatalf("diagnostics leaked schema content: %#v", diagnostics)
|
||||
}
|
||||
}
|
||||
|
||||
func cloneCandidate(t *testing.T, value map[string]any) map[string]any {
|
||||
t.Helper()
|
||||
content, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
var clone map[string]any
|
||||
if err := json.Unmarshal(content, &clone); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return clone
|
||||
}
|
||||
|
||||
func validateSchema(t *testing.T, value map[string]any, schemaContent []byte) error {
|
||||
t.Helper()
|
||||
content, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(content))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
document, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := compiler.AddResource("schema.json", document); err != nil {
|
||||
return err
|
||||
}
|
||||
schema, err := compiler.Compile("schema.json")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return schema.Validate(instance)
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func extractionRequest() contracts.TypedExtractionRequest {
|
||||
doc := sourceDocument()
|
||||
chunk := &source.Chunk{
|
||||
ID: "session-occurrences:chunk:0", SourceID: doc.ID, Index: 0,
|
||||
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 30},
|
||||
Content: []byte(`{"units":[10,20,30]}`), MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), doc.Units...),
|
||||
}
|
||||
return contracts.TypedExtractionRequest{
|
||||
Source: doc, Chunk: chunk,
|
||||
SourceInput: contracts.NewLLMInputMaterial("source", chunk.MediaType, chunk.Content, "sha256:chunk", "file:///session-occurrences.json"),
|
||||
SessionID: "occurrence-session", LLMProfile: "occurrence-profile",
|
||||
}
|
||||
}
|
||||
|
||||
func sourceDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{ID: "session-occurrences", Kind: "transcript", Format: "application/json", Digest: "sha256:test", Units: []source.SourceUnit{
|
||||
{ID: 10, Kind: "transcript_segment", Text: "The party returns to the tavern."},
|
||||
{ID: 20, Kind: "transcript_segment", Text: "They plan to travel to the tavern tomorrow."},
|
||||
{ID: 30, Kind: "transcript_segment", Text: "They recall their first visit to the tavern."},
|
||||
}}
|
||||
}
|
||||
|
||||
func occurrenceRefs(start, end int) []occurrenceSourceRefResponse {
|
||||
return []occurrenceSourceRefResponse{{StartUnitID: start, EndUnitID: end}}
|
||||
}
|
||||
|
||||
func newExtractor(t *testing.T, client contracts.StructuredLLMClient, references ...contracts.ReferenceSet) *Extractor {
|
||||
t.Helper()
|
||||
extractor, err := New(client, Options{}, references...)
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
return extractor
|
||||
}
|
||||
|
||||
func mismatchedSourceInputRequest(req contracts.TypedExtractionRequest) contracts.TypedExtractionRequest {
|
||||
req.SourceInput = contracts.NewLLMInputMaterial("source", "application/json", []byte(`{"different":true}`), "sha256:other", "")
|
||||
return req
|
||||
}
|
||||
|
||||
type fakeOccurrencesLLMClient struct {
|
||||
response extractionResponse
|
||||
err error
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (client *fakeOccurrencesLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.requests = append(client.requests, cloneStructuredCompletionRequest(req))
|
||||
if client.err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, client.err
|
||||
}
|
||||
target, ok := out.(*extractionResponse)
|
||||
if !ok {
|
||||
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
|
||||
}
|
||||
content, err := json.Marshal(client.response)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
if err := json.Unmarshal(content, target); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||
}
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
return req
|
||||
}
|
||||
6
internal/modules/dnd/extract/locations/assets.go
Normal file
6
internal/modules/dnd/extract/locations/assets.go
Normal file
@@ -0,0 +1,6 @@
|
||||
package locations
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/prompts/*.yaml assets/prompts/*.md assets/schemas/*.json
|
||||
var embeddedAssets embed.FS
|
||||
@@ -0,0 +1,42 @@
|
||||
id: dnd.locations
|
||||
version: "v1"
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
- name: players
|
||||
required: false
|
||||
content_type: text/plain
|
||||
- name: party
|
||||
required: false
|
||||
content_type: text/plain
|
||||
- name: glossary
|
||||
required: false
|
||||
content_type: text/plain
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./sharedassets/common-dnd-system.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-identity.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-references.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-transcript.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-extraction-evidence.md
|
||||
- role: user
|
||||
content_file: ./task.md
|
||||
- role: user
|
||||
content_file: ./instructions.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: dnd_locations_llm.v1.json
|
||||
repair_attempts: 0
|
||||
@@ -0,0 +1,6 @@
|
||||
Return only observed location display names and narrow transcript source ranges.
|
||||
|
||||
Exclude people, creatures, objects, organizations, abstract concepts, and
|
||||
places merely inferred from an event. Omit uncertain or unsupported places.
|
||||
Campaign references may clarify terms already present in the transcript, but
|
||||
they are not evidence and must never supply a source range.
|
||||
@@ -0,0 +1,8 @@
|
||||
Extract physical places established by the provided Dungeons & Dragons
|
||||
transcript and cite where each place is identified.
|
||||
|
||||
Include planes, regions, settlements, districts, buildings, rooms, landmarks,
|
||||
routes, and geographic features. A generic label such as "the tavern" is
|
||||
allowed only when the transcript uses it for a specific place. Keep aliases and
|
||||
nested places when the transcript identifies them; do not merge or invent
|
||||
qualifiers for similarly named places.
|
||||
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.locations.llm",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["locations"],
|
||||
"properties": {
|
||||
"locations": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name", "source_refs"],
|
||||
"properties": {
|
||||
"name": {"type": "string"},
|
||||
"source_refs": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["start_unit_id", "end_unit_id"],
|
||||
"properties": {
|
||||
"start_unit_id": {"type": "integer"},
|
||||
"end_unit_id": {"type": "integer"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
86
internal/modules/dnd/extract/locations/canonicalize.go
Normal file
86
internal/modules/dnd/extract/locations/canonicalize.go
Normal file
@@ -0,0 +1,86 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
type orderedLocationResponse struct {
|
||||
value locationResponse
|
||||
earliest int
|
||||
hasEvidence bool
|
||||
}
|
||||
|
||||
func canonicalizeResponse(response *extractionResponse, order shared.SourceRefOrder, sourceID string) {
|
||||
if response == nil {
|
||||
return
|
||||
}
|
||||
ordered := make([]orderedLocationResponse, len(response.Locations))
|
||||
for index := range response.Locations {
|
||||
earliest, hasEvidence := canonicalizeLocation(&response.Locations[index], order, sourceID)
|
||||
ordered[index] = orderedLocationResponse{value: response.Locations[index], earliest: earliest, hasEvidence: hasEvidence}
|
||||
}
|
||||
sort.SliceStable(ordered, func(left, right int) bool {
|
||||
if ordered[left].hasEvidence != ordered[right].hasEvidence {
|
||||
return ordered[left].hasEvidence
|
||||
}
|
||||
if !ordered[left].hasEvidence {
|
||||
return false
|
||||
}
|
||||
return ordered[left].earliest < ordered[right].earliest
|
||||
})
|
||||
for index := range ordered {
|
||||
response.Locations[index] = ordered[index].value
|
||||
}
|
||||
}
|
||||
|
||||
func canonicalizeLocation(location *locationResponse, order shared.SourceRefOrder, sourceID string) (int, bool) {
|
||||
if location == nil {
|
||||
return 0, false
|
||||
}
|
||||
refs := order.Canonicalize(canonicalSourceRefs(location.SourceRefs, sourceID))
|
||||
location.SourceRefs = locationResponseRefs(refs)
|
||||
return order.EarliestValid(refs)
|
||||
}
|
||||
|
||||
func canonicalLocationList(response extractionResponse, sourceID string) dnd.LocationList {
|
||||
if response.Locations == nil {
|
||||
return dnd.LocationList{Locations: nil}
|
||||
}
|
||||
locations := make([]dnd.Location, len(response.Locations))
|
||||
for index, location := range response.Locations {
|
||||
refs := canonicalSourceRefs(location.SourceRefs, sourceID)
|
||||
locations[index] = dnd.Location{
|
||||
ID: identity.DeriveID(location.Name, refs),
|
||||
Name: location.Name,
|
||||
SourceRefs: refs,
|
||||
}
|
||||
}
|
||||
return dnd.LocationList{Locations: locations}
|
||||
}
|
||||
|
||||
func canonicalSourceRefs(values []locationSourceRefResponse, sourceID string) []source.SourceRef {
|
||||
if values == nil {
|
||||
return nil
|
||||
}
|
||||
refs := make([]source.SourceRef, len(values))
|
||||
for index, value := range values {
|
||||
refs[index] = source.SourceRef{SourceID: sourceID, StartUnitID: value.StartUnitID, EndUnitID: value.EndUnitID}
|
||||
}
|
||||
return refs
|
||||
}
|
||||
|
||||
func locationResponseRefs(values []source.SourceRef) []locationSourceRefResponse {
|
||||
if values == nil {
|
||||
return nil
|
||||
}
|
||||
refs := make([]locationSourceRefResponse, len(values))
|
||||
for index, value := range values {
|
||||
refs[index] = locationSourceRefResponse{StartUnitID: value.StartUnitID, EndUnitID: value.EndUnitID}
|
||||
}
|
||||
return refs
|
||||
}
|
||||
152
internal/modules/dnd/extract/locations/extractor.go
Normal file
152
internal/modules/dnd/extract/locations/extractor.go
Normal file
@@ -0,0 +1,152 @@
|
||||
// Package locations extracts source-grounded D&D physical location candidates.
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const (
|
||||
Key = "dnd/locations"
|
||||
mappingPolicy = "dnd.locations.extract_mapping.v1"
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"chunks", "source.transcript"}
|
||||
var providedCapabilities = []string{"dnd.locations"}
|
||||
|
||||
var referenceSlotDescriptions = shared.ReferenceSlotDescriptions{
|
||||
Glossary: "Optional campaign glossary reference material used only for location disambiguation.",
|
||||
Party: "Optional party roster reference material used only for location disambiguation.",
|
||||
Players: "Optional player list reference material used only for location disambiguation.",
|
||||
Roster: "Deprecated alias for party roster reference material used only for location disambiguation.",
|
||||
}
|
||||
|
||||
func referenceSlots() []contracts.ReferenceSlot {
|
||||
return shared.ReferenceSlots(referenceSlotDescriptions)
|
||||
}
|
||||
|
||||
var _ contracts.Extractor[dnd.LocationList] = (*Extractor)(nil)
|
||||
var _ contracts.ManifestMetadataProvider = (*Extractor)(nil)
|
||||
var _ pipeline.CheckpointFingerprintProvider = (*Extractor)(nil)
|
||||
|
||||
type Options struct{}
|
||||
|
||||
type Extractor struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
promptSHA string
|
||||
responseSchemaSHA string
|
||||
}
|
||||
|
||||
func New(llmClient contracts.StructuredLLMClient, _ Options, references ...contracts.ReferenceSet) (*Extractor, error) {
|
||||
if llmClient == nil {
|
||||
return nil, extractorErrorf("LLM client must not be nil")
|
||||
}
|
||||
if len(references) > 1 {
|
||||
return nil, extractorErrorf("at most one reference set may be supplied")
|
||||
}
|
||||
promptSHA, err := promptAssetMetadata()
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("load prompt metadata: %w", err)
|
||||
}
|
||||
responseSchema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
return nil, extractorErrorf("load response schema: %w", err)
|
||||
}
|
||||
return &Extractor{llm: llmClient, promptSHA: promptSHA, responseSchemaSHA: responseSchema.SHA256}, nil
|
||||
}
|
||||
|
||||
func (e *Extractor) Key() string { return Key }
|
||||
|
||||
func (e *Extractor) ReferenceSlots() []contracts.ReferenceSlot { return referenceSlots() }
|
||||
|
||||
func (e *Extractor) ManifestMetadata() map[string]any {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return map[string]any{
|
||||
"prompt_id": PromptID,
|
||||
"prompt_version": SchemaVersion,
|
||||
"prompt_sha256": e.promptSHA,
|
||||
"response_schema_key": string(ResponseSchemaKey),
|
||||
"response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName,
|
||||
"response_schema_version": SchemaVersion,
|
||||
"response_schema_sha256": e.responseSchemaSHA,
|
||||
"identity_policy": identity.Policy,
|
||||
"mapping_policy": mappingPolicy,
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Extractor) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return []pipeline.CheckpointFingerprint{
|
||||
{Name: "prompt", Value: e.promptSHA},
|
||||
{Name: "response_schema", Value: e.responseSchemaSHA},
|
||||
{Name: "identity_policy", Value: identity.Policy},
|
||||
{Name: "mapping_policy", Value: mappingPolicy},
|
||||
}
|
||||
}
|
||||
|
||||
func (e *Extractor) Extract(ctx context.Context, req contracts.TypedExtractionRequest) (contracts.TypedExtractionResult[dnd.LocationList], error) {
|
||||
if e == nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationList]{}, extractorErrorf("extractor must not be nil")
|
||||
}
|
||||
if e.llm == nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationList]{}, extractorErrorf("LLM client must not be nil")
|
||||
}
|
||||
sourceInput, err := shared.PrepareChunkExtraction(ctx, req)
|
||||
if err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationList]{}, extractorErrorf("%w", err)
|
||||
}
|
||||
order := shared.NewSourceRefOrder(req.Source)
|
||||
|
||||
var response extractionResponse
|
||||
if _, err := e.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID,
|
||||
Inputs: shared.PromptInputs(sourceInput, req.References),
|
||||
}, &response); err != nil {
|
||||
return contracts.TypedExtractionResult[dnd.LocationList]{}, extractorErrorf("complete structured output: %w", err)
|
||||
}
|
||||
canonicalizeResponse(&response, order, req.Source.ID)
|
||||
return contracts.TypedExtractionResult[dnd.LocationList]{Value: canonicalLocationList(response, req.Source.ID)}, nil
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{
|
||||
Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked,
|
||||
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.LocationListKind, ReferenceSlots: referenceSlots(),
|
||||
}
|
||||
}
|
||||
|
||||
func Register(registry *pipeline.ExtractorRegistry) error {
|
||||
return pipeline.RegisterExtractorBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Extractor[dnd.LocationList], error) {
|
||||
options, err := DecodeOptions(request.Options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(request.Dependencies.LLM, options, request.References)
|
||||
})
|
||||
}
|
||||
|
||||
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||
|
||||
func DecodeOptions(options map[string]any) (Options, error) {
|
||||
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||
return Options{}, extractorErrorf("%w", err)
|
||||
}
|
||||
return Options{}, nil
|
||||
}
|
||||
|
||||
func extractorErrorf(format string, args ...any) error {
|
||||
return fmt.Errorf("dnd locations extractor: "+format, args...)
|
||||
}
|
||||
125
internal/modules/dnd/extract/locations/extractor_test.go
Normal file
125
internal/modules/dnd/extract/locations/extractor_test.go
Normal file
@@ -0,0 +1,125 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
func TestExtractMapsLocationsWithOwnedEvidenceAndDeterministicOrder(t *testing.T) {
|
||||
client := &fakeLocationsLLMClient{response: extractionResponse{Locations: []locationResponse{
|
||||
{Name: "The Tavern", SourceRefs: responseSourceRefs(3, 3)},
|
||||
{Name: "Old Mill", SourceRefs: []locationSourceRefResponse{{StartUnitID: 2, EndUnitID: 2}, {StartUnitID: 1, EndUnitID: 1}, {StartUnitID: 1, EndUnitID: 1}}},
|
||||
}}}
|
||||
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||
if err != nil {
|
||||
t.Fatalf("Extract() error = %v, want nil", err)
|
||||
}
|
||||
refs := []source.SourceRef{{SourceID: "session-locations", StartUnitID: 1, EndUnitID: 1}, {SourceID: "session-locations", StartUnitID: 2, EndUnitID: 2}}
|
||||
want := dnd.LocationList{Locations: []dnd.Location{
|
||||
{ID: identity.DeriveID("Old Mill", refs), Name: "Old Mill", SourceRefs: refs},
|
||||
{ID: identity.DeriveID("The Tavern", []source.SourceRef{{SourceID: "session-locations", StartUnitID: 3, EndUnitID: 3}}), Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: "session-locations", StartUnitID: 3, EndUnitID: 3}}},
|
||||
}}
|
||||
if !reflect.DeepEqual(result.Value, want) {
|
||||
t.Fatalf("Value = %#v, want %#v", result.Value, want)
|
||||
}
|
||||
result.Value.Locations[0].SourceRefs[0].StartUnitID = 99
|
||||
for _, location := range client.response.Locations {
|
||||
for _, ref := range location.SourceRefs {
|
||||
if ref.StartUnitID == 99 {
|
||||
t.Fatal("result source references alias the model response")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractRetainsSameNameLocationsAtDifferentAnchors(t *testing.T) {
|
||||
client := &fakeLocationsLLMClient{response: extractionResponse{Locations: []locationResponse{
|
||||
{Name: "the tavern", SourceRefs: responseSourceRefs(1, 1)},
|
||||
{Name: "the tavern", SourceRefs: responseSourceRefs(3, 3)},
|
||||
}}}
|
||||
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||
if err != nil || len(result.Value.Locations) != 2 {
|
||||
t.Fatalf("Extract() = %#v, %v; want both same-name candidates", result, err)
|
||||
}
|
||||
if result.Value.Locations[0].ID == result.Value.Locations[1].ID || result.Value.Locations[0].Name != result.Value.Locations[1].Name {
|
||||
t.Fatalf("locations = %#v, want distinct evidence-anchored IDs", result.Value.Locations)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractPreservesInvalidCandidatesForValidators(t *testing.T) {
|
||||
client := &fakeLocationsLLMClient{content: []byte(`{"locations":[{"name":"","source_refs":[{"start_unit_id":0,"end_unit_id":-1}]}]}`)}
|
||||
result, err := newExtractor(t, client).Extract(context.Background(), extractionRequest())
|
||||
if err != nil {
|
||||
t.Fatalf("Extract() error = %v, want nil", err)
|
||||
}
|
||||
location := result.Value.Locations[0]
|
||||
if location.ID != "" || location.Name != "" || !reflect.DeepEqual(location.SourceRefs, []source.SourceRef{{SourceID: "session-locations", StartUnitID: 0, EndUnitID: -1}}) {
|
||||
t.Fatalf("location = %#v, want invalid candidate preserved", location)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractPassesReferencesWithoutTreatingThemAsEvidence(t *testing.T) {
|
||||
client := &fakeLocationsLLMClient{response: extractionResponse{Locations: []locationResponse{}}}
|
||||
req := extractionRequest()
|
||||
req.References = contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
"glossary": {Slot: contracts.ReferenceSlot{Name: "glossary"}, Items: []contracts.ReferenceItem{{SlotName: "glossary", Content: []byte("Old Mill: abandoned granary")}}},
|
||||
}}
|
||||
if _, err := newExtractor(t, client).Extract(context.Background(), req); err != nil {
|
||||
t.Fatalf("Extract() error = %v", err)
|
||||
}
|
||||
inputs := client.requests[0].Inputs
|
||||
if string(inputs["glossary"].Content) != "Old Mill: abandoned granary" || strings.Contains(string(inputs["transcript"].Content), "abandoned granary") {
|
||||
t.Fatalf("prompt inputs = %#v, want separated reference material", inputs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractDoesNotMutateRequestMaterials(t *testing.T) {
|
||||
client := &fakeLocationsLLMClient{response: extractionResponse{Locations: []locationResponse{{Name: "Old Mill", SourceRefs: responseSourceRefs(1, 1)}}}}
|
||||
req := extractionRequest()
|
||||
beforeUnits := append([]source.SourceUnit(nil), req.Source.Units...)
|
||||
beforeChunkUnits := append([]source.SourceUnit(nil), req.Chunk.Units...)
|
||||
beforeContent := append([]byte(nil), req.Chunk.Content...)
|
||||
if _, err := newExtractor(t, client).Extract(context.Background(), req); err != nil {
|
||||
t.Fatalf("Extract() error = %v", err)
|
||||
}
|
||||
if !reflect.DeepEqual(req.Source.Units, beforeUnits) || !reflect.DeepEqual(req.Chunk.Units, beforeChunkUnits) || !reflect.DeepEqual(req.Chunk.Content, beforeContent) {
|
||||
t.Fatalf("Extract() mutated request: %#v", req)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtractHandlesEmptyOutputAndLocalFailures(t *testing.T) {
|
||||
empty, err := newExtractor(t, &fakeLocationsLLMClient{response: extractionResponse{Locations: []locationResponse{}}}).Extract(context.Background(), extractionRequest())
|
||||
if err != nil || empty.Value.Locations == nil || len(empty.Value.Locations) != 0 {
|
||||
t.Fatalf("empty Extract() = %#v, %v; want empty list", empty, err)
|
||||
}
|
||||
request := extractionRequest()
|
||||
var nilExtractor *Extractor
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
extractor *Extractor
|
||||
req contracts.TypedExtractionRequest
|
||||
want string
|
||||
}{
|
||||
{name: "nil extractor", extractor: nilExtractor, req: request, want: "extractor"},
|
||||
{name: "nil client", extractor: &Extractor{}, req: request, want: "LLM client"},
|
||||
{name: "preflight", extractor: newExtractor(t, &fakeLocationsLLMClient{}), req: mismatchedSourceInputRequest(request), want: "must match chunk"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := test.extractor.Extract(context.Background(), test.req); err == nil || !strings.Contains(err.Error(), "dnd locations") || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Extract() error = %v, want local context", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
_, err = newExtractor(t, &fakeLocationsLLMClient{err: errors.New("provider unavailable")}).Extract(context.Background(), request)
|
||||
if err == nil || !strings.Contains(err.Error(), "dnd locations") || !strings.Contains(err.Error(), "provider unavailable") {
|
||||
t.Fatalf("provider error = %v, want contextual provider error", err)
|
||||
}
|
||||
}
|
||||
15
internal/modules/dnd/extract/locations/model.go
Normal file
15
internal/modules/dnd/extract/locations/model.go
Normal file
@@ -0,0 +1,15 @@
|
||||
package locations
|
||||
|
||||
type extractionResponse struct {
|
||||
Locations []locationResponse `json:"locations"`
|
||||
}
|
||||
|
||||
type locationResponse struct {
|
||||
Name string `json:"name"`
|
||||
SourceRefs []locationSourceRefResponse `json:"source_refs"`
|
||||
}
|
||||
|
||||
type locationSourceRefResponse struct {
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
}
|
||||
52
internal/modules/dnd/extract/locations/prompt_assets.go
Normal file
52
internal/modules/dnd/extract/locations/prompt_assets.go
Normal file
@@ -0,0 +1,52 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const promptAssetRoot = "assets/prompts"
|
||||
|
||||
var promptAssetManifest = shared.PromptAssetManifest{
|
||||
ModuleDir: PromptID,
|
||||
ModuleFiles: []promptfs.ModulePromptFile{
|
||||
{Name: "dnd.locations.yaml", Path: "assets/prompts/dnd.locations.yaml"},
|
||||
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
||||
},
|
||||
SharedFiles: []string{
|
||||
"common-dnd-system.md",
|
||||
"common-dnd-extraction-evidence.md",
|
||||
"common-dnd-identity.md",
|
||||
"common-dnd-transcript.md",
|
||||
"common-dnd-references.md",
|
||||
},
|
||||
}
|
||||
|
||||
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
||||
promptFS, err := promptAssetManifest.PromptFS(embeddedAssets)
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare location prompt assets: %w", err)
|
||||
}
|
||||
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||
}
|
||||
|
||||
func promptAssetMetadata() (string, error) {
|
||||
promptAssetHashOnce.Do(func() {
|
||||
promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||
})
|
||||
return promptAssetHash, promptAssetHashErr
|
||||
}
|
||||
|
||||
var (
|
||||
promptAssetHashOnce sync.Once
|
||||
promptAssetHash string
|
||||
promptAssetHashErr error
|
||||
)
|
||||
50
internal/modules/dnd/extract/locations/prompt_assets_test.go
Normal file
50
internal/modules/dnd/extract/locations/prompt_assets_test.go
Normal file
@@ -0,0 +1,50 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestRegisterPromptAssetsPreparesLocationPrompt(t *testing.T) {
|
||||
for _, name := range []string{"common-dnd-system.md", "common-dnd-identity.md", "common-dnd-references.md", "common-dnd-transcript.md", "common-dnd-extraction-evidence.md"} {
|
||||
if !slices.Contains(promptAssetManifest.SharedFiles, name) {
|
||||
t.Fatalf("shared prompt assets = %#v, missing %q", promptAssetManifest.SharedFiles, name)
|
||||
}
|
||||
}
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := RegisterPromptAssets(registry); err != nil {
|
||||
t.Fatalf("RegisterPromptAssets() error = %v", err)
|
||||
}
|
||||
options, err := registry.PromptKitOptions()
|
||||
if err != nil {
|
||||
t.Fatalf("PromptKitOptions() error = %v", err)
|
||||
}
|
||||
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{ID: "location-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "location-test-model"})))
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "location-test-profile", Inputs: map[string]promptkit.ArtifactRef{
|
||||
"transcript": promptkit.Inline(`{"units":[1]}`), "players": promptkit.Inline(" "), "party": promptkit.Inline(" "), "glossary": promptkit.Inline(" "),
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
if prepared.OutputContract.SchemaPath != "dnd_locations_llm.v1.json" {
|
||||
t.Fatalf("output contract = %#v", prepared.OutputContract)
|
||||
}
|
||||
if len(prepared.Messages) != 7 || !strings.Contains(prepared.Messages[3].Content, `"units"`) || strings.Contains(prepared.Messages[3].Content, "location-test") {
|
||||
t.Fatalf("prepared messages = %#v, want rendered transcript only in transcript message", prepared.Messages)
|
||||
}
|
||||
for _, index := range []int{2, 3, 6} {
|
||||
if prepared.Messages[index].CacheControl == nil || prepared.Messages[index].CacheControl.Type != promptkit.CacheControlEphemeral {
|
||||
t.Fatalf("message %d cache control = %#v, want ephemeral", index, prepared.Messages[index].CacheControl)
|
||||
}
|
||||
}
|
||||
}
|
||||
53
internal/modules/dnd/extract/locations/registry_test.go
Normal file
53
internal/modules/dnd/extract/locations/registry_test.go
Normal file
@@ -0,0 +1,53 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
func TestModuleRegistrationAndMetadata(t *testing.T) {
|
||||
if _, err := New(nil, Options{}); err == nil || !strings.Contains(err.Error(), "LLM client") {
|
||||
t.Fatalf("New(nil) error = %v, want client rejection", err)
|
||||
}
|
||||
if _, err := New(&fakeLocationsLLMClient{}, Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one") {
|
||||
t.Fatalf("New() error = %v, want reference-set rejection", err)
|
||||
}
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageExtract, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"chunks", "source.transcript"}, Provides: []string{"dnd.locations"}, ArtifactKind: dnd.LocationListKind, ReferenceSlots: referenceSlots()}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
}
|
||||
registry := pipeline.NewExtractorRegistry()
|
||||
if err := Register(registry); err != nil {
|
||||
t.Fatalf("Register() error = %v", err)
|
||||
}
|
||||
if got, ok := registry.Spec(Key); !ok || !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("registry spec = %#v, present = %t", got, ok)
|
||||
}
|
||||
if err := Register(nil); err == nil || !strings.Contains(err.Error(), "extractor registry") {
|
||||
t.Fatalf("Register(nil) error = %v, want registry rejection", err)
|
||||
}
|
||||
if _, err := DecodeOptions(map[string]any{"unknown": true}); err == nil {
|
||||
t.Fatal("DecodeOptions() accepted unknown option")
|
||||
}
|
||||
extractor := newExtractor(t, &fakeLocationsLLMClient{})
|
||||
metadata := extractor.ManifestMetadata()
|
||||
for key, value := range map[string]string{
|
||||
"prompt_id": PromptID, "prompt_version": SchemaVersion,
|
||||
"response_schema_key": string(ResponseSchemaKey), "response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName, "response_schema_version": SchemaVersion,
|
||||
"identity_policy": identity.Policy, "mapping_policy": mappingPolicy,
|
||||
} {
|
||||
if metadata[key] != value {
|
||||
t.Fatalf("metadata[%q] = %#v, want %q", key, metadata[key], value)
|
||||
}
|
||||
}
|
||||
if got := extractor.CheckpointFingerprints(); len(got) != 4 || got[0].Name != "prompt" || got[1].Name != "response_schema" || got[2].Value != identity.Policy || got[3].Value != mappingPolicy {
|
||||
t.Fatalf("CheckpointFingerprints() = %#v", got)
|
||||
}
|
||||
}
|
||||
@@ -1,12 +1,12 @@
|
||||
package npcs
|
||||
package locations
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
|
||||
const (
|
||||
PromptID = "dnd.npcs.normalize"
|
||||
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_npcs_normalize_llm")
|
||||
ResponseSchemaID = "notarius.dnd.npcs.normalize.llm"
|
||||
ResponseSchemaName = "notarius_dnd_npcs_normalize_llm_v1"
|
||||
PromptID = "dnd.locations"
|
||||
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_locations_llm")
|
||||
ResponseSchemaID = "notarius.dnd.locations.llm"
|
||||
ResponseSchemaName = "notarius_dnd_locations_llm_v1"
|
||||
SchemaVersion = "v1"
|
||||
)
|
||||
|
||||
@@ -16,6 +16,6 @@ func loadResponseSchema() (llm.ResponseSchema, error) {
|
||||
ID: ResponseSchemaID,
|
||||
Version: SchemaVersion,
|
||||
Name: ResponseSchemaName,
|
||||
AssetPath: "assets/schemas/dnd_npcs_normalize_llm.v1.json",
|
||||
AssetPath: "assets/schemas/dnd_locations_llm.v1.json",
|
||||
})
|
||||
}
|
||||
25
internal/modules/dnd/extract/locations/schema_test.go
Normal file
25
internal/modules/dnd/extract/locations/schema_test.go
Normal file
@@ -0,0 +1,25 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestLocationResponseSchemaIsPrivateAndStructural(t *testing.T) {
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v", err)
|
||||
}
|
||||
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Name != ResponseSchemaName || schema.Version != SchemaVersion || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v, want location response schema", schema)
|
||||
}
|
||||
var document map[string]any
|
||||
if err := json.Unmarshal(schema.JSONSchema, &document); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
encoded, err := json.Marshal(document)
|
||||
if err != nil || strings.Contains(string(encoded), `"id"`) {
|
||||
t.Fatalf("schema = %s, want no durable ID field", encoded)
|
||||
}
|
||||
}
|
||||
89
internal/modules/dnd/extract/locations/test_helpers_test.go
Normal file
89
internal/modules/dnd/extract/locations/test_helpers_test.go
Normal file
@@ -0,0 +1,89 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func extractionRequest() contracts.TypedExtractionRequest {
|
||||
doc := sourceDocument()
|
||||
chunk := &source.Chunk{
|
||||
ID: "session-locations:chunk:0", SourceID: doc.ID, Index: 0,
|
||||
Ref: source.SourceRef{SourceID: doc.ID, StartUnitID: 1, EndUnitID: 3},
|
||||
Content: []byte(`{"units":[1,2,3]}`), MediaType: "application/json",
|
||||
Units: append([]source.SourceUnit(nil), doc.Units...),
|
||||
}
|
||||
return contracts.TypedExtractionRequest{
|
||||
Source: doc, Chunk: chunk,
|
||||
SourceInput: contracts.NewLLMInputMaterial("source", chunk.MediaType, chunk.Content, "sha256:chunk", "file:///session-locations.json"),
|
||||
SessionID: "location-session", LLMProfile: "location-profile",
|
||||
}
|
||||
}
|
||||
|
||||
func sourceDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{ID: "session-locations", Kind: "transcript", Format: "application/json", Digest: "sha256:test", Units: []source.SourceUnit{
|
||||
{ID: 1, Kind: "transcript_segment", Text: "The party enters the Old Mill."},
|
||||
{ID: 2, Kind: "transcript_segment", Text: "They leave the old road behind."},
|
||||
{ID: 3, Kind: "transcript_segment", Text: "The tavern is quiet."},
|
||||
}}
|
||||
}
|
||||
|
||||
func responseSourceRefs(startUnitID, endUnitID int) []locationSourceRefResponse {
|
||||
return []locationSourceRefResponse{{StartUnitID: startUnitID, EndUnitID: endUnitID}}
|
||||
}
|
||||
|
||||
func newExtractor(t *testing.T, client contracts.StructuredLLMClient, references ...contracts.ReferenceSet) *Extractor {
|
||||
t.Helper()
|
||||
extractor, err := New(client, Options{}, references...)
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v, want nil", err)
|
||||
}
|
||||
return extractor
|
||||
}
|
||||
|
||||
func mismatchedSourceInputRequest(req contracts.TypedExtractionRequest) contracts.TypedExtractionRequest {
|
||||
req.SourceInput = contracts.NewLLMInputMaterial("source", "application/json", []byte(`{"different":true}`), "sha256:other", "file:///other.json")
|
||||
return req
|
||||
}
|
||||
|
||||
type fakeLocationsLLMClient struct {
|
||||
response extractionResponse
|
||||
content []byte
|
||||
err error
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (client *fakeLocationsLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.requests = append(client.requests, cloneStructuredCompletionRequest(req))
|
||||
if client.err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, client.err
|
||||
}
|
||||
target, ok := out.(*extractionResponse)
|
||||
if !ok {
|
||||
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
|
||||
}
|
||||
content := append([]byte(nil), client.content...)
|
||||
if len(content) != 0 {
|
||||
if err := json.Unmarshal(content, target); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
} else {
|
||||
*target = client.response
|
||||
var err error
|
||||
content, err = json.Marshal(client.response)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||
}
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
return req
|
||||
}
|
||||
183
internal/modules/dnd/locations/identity/identity.go
Normal file
183
internal/modules/dnd/locations/identity/identity.go
Normal file
@@ -0,0 +1,183 @@
|
||||
// Package identity implements deterministic session-scoped location identity.
|
||||
package identity
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"golang.org/x/text/cases"
|
||||
"golang.org/x/text/unicode/norm"
|
||||
)
|
||||
|
||||
const (
|
||||
// IdentityPolicy identifies the durable location ID derivation policy.
|
||||
IdentityPolicy = "dnd.locations.identity.v1"
|
||||
// Policy is an alias for IdentityPolicy.
|
||||
Policy = IdentityPolicy
|
||||
|
||||
idPrefix = "location:sha256:"
|
||||
)
|
||||
|
||||
// IssueCode identifies one deterministic registry identity problem.
|
||||
type IssueCode string
|
||||
|
||||
const (
|
||||
IssueEmptyCanonicalName IssueCode = "empty_canonical_name"
|
||||
IssueMissingEvidence IssueCode = "missing_evidence"
|
||||
IssueInvalidID IssueCode = "invalid_id"
|
||||
IssueIDMismatch IssueCode = "id_mismatch"
|
||||
IssueDuplicateID IssueCode = "duplicate_id"
|
||||
)
|
||||
|
||||
// Issue is an inspectable identity validation problem.
|
||||
type Issue struct {
|
||||
Code IssueCode
|
||||
RecordIndex int
|
||||
Value string
|
||||
}
|
||||
|
||||
func (i Issue) Error() string { return string(i.Code) }
|
||||
|
||||
// NormalizeDisplay returns the durable display form without changing spelling
|
||||
// or punctuation.
|
||||
func NormalizeDisplay(value string) string {
|
||||
return strings.Join(strings.Fields(value), " ")
|
||||
}
|
||||
|
||||
// ComparisonKey returns the stable key used for location identity comparisons.
|
||||
func ComparisonKey(value string) string {
|
||||
value = norm.NFKC.String(value)
|
||||
value = strings.Map(func(r rune) rune {
|
||||
switch r {
|
||||
case '\u2018', '\u2019', '\u02bc':
|
||||
return '\''
|
||||
default:
|
||||
return r
|
||||
}
|
||||
}, value)
|
||||
value = strings.Join(strings.Fields(value), " ")
|
||||
return cases.Fold().String(value)
|
||||
}
|
||||
|
||||
// DeriveID derives a location ID from name and the earliest canonical evidence
|
||||
// reference. It returns an empty string when either identity component is not
|
||||
// available, leaving validation to report the problem instead of manufacturing
|
||||
// an ID.
|
||||
func DeriveID(name string, refs []source.SourceRef) string {
|
||||
comparisonName := ComparisonKey(name)
|
||||
anchor, ok := earliestReference(refs)
|
||||
if comparisonName == "" || !ok {
|
||||
return ""
|
||||
}
|
||||
|
||||
input, err := json.Marshal([]any{
|
||||
IdentityPolicy,
|
||||
comparisonName,
|
||||
anchor.SourceID,
|
||||
anchor.StartUnitID,
|
||||
anchor.EndUnitID,
|
||||
})
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
digest := sha256.Sum256(input)
|
||||
return idPrefix + hex.EncodeToString(digest[:])
|
||||
}
|
||||
|
||||
// IDFor is an alias for DeriveID.
|
||||
func IDFor(name string, refs []source.SourceRef) string { return DeriveID(name, refs) }
|
||||
|
||||
// IsValidID reports whether value has the exact durable location ID syntax.
|
||||
func IsValidID(value string) bool {
|
||||
if len(value) != len(idPrefix)+sha256.Size*2 || !strings.HasPrefix(value, idPrefix) {
|
||||
return false
|
||||
}
|
||||
for _, r := range value[len(idPrefix):] {
|
||||
if !((r >= '0' && r <= '9') || (r >= 'a' && r <= 'f')) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// ValidID is an alias for IsValidID.
|
||||
func ValidID(value string) bool { return IsValidID(value) }
|
||||
|
||||
// ValidateList validates the identity members of list.
|
||||
func ValidateList(list dnd.LocationList) []Issue { return ValidateRegistry(list.Locations) }
|
||||
|
||||
// ValidateRegistry validates location IDs without modifying records or their
|
||||
// source references. Equal comparison names are allowed because their evidence
|
||||
// anchors are part of the identity policy.
|
||||
func ValidateRegistry(locations []dnd.Location) []Issue {
|
||||
issues := make([]Issue, 0)
|
||||
idOwners := make(map[string][]int)
|
||||
|
||||
for recordIndex, location := range locations {
|
||||
if ComparisonKey(location.Name) == "" {
|
||||
issues = append(issues, Issue{Code: IssueEmptyCanonicalName, RecordIndex: recordIndex, Value: location.Name})
|
||||
}
|
||||
if _, ok := earliestReference(location.SourceRefs); !ok {
|
||||
issues = append(issues, Issue{Code: IssueMissingEvidence, RecordIndex: recordIndex})
|
||||
}
|
||||
|
||||
if !IsValidID(location.ID) {
|
||||
issues = append(issues, Issue{Code: IssueInvalidID, RecordIndex: recordIndex, Value: location.ID})
|
||||
} else if expected := DeriveID(location.Name, location.SourceRefs); location.ID != expected {
|
||||
issues = append(issues, Issue{Code: IssueIDMismatch, RecordIndex: recordIndex, Value: location.ID})
|
||||
}
|
||||
if location.ID != "" {
|
||||
idOwners[location.ID] = append(idOwners[location.ID], recordIndex)
|
||||
}
|
||||
}
|
||||
|
||||
for recordIndex, location := range locations {
|
||||
if location.ID != "" && len(idOwners[location.ID]) > 1 && idOwners[location.ID][0] != recordIndex {
|
||||
issues = append(issues, Issue{Code: IssueDuplicateID, RecordIndex: recordIndex, Value: location.ID})
|
||||
}
|
||||
}
|
||||
|
||||
return issues
|
||||
}
|
||||
|
||||
func earliestReference(refs []source.SourceRef) (source.SourceRef, bool) {
|
||||
canonical := canonicalReferences(refs)
|
||||
if len(canonical) == 0 {
|
||||
return source.SourceRef{}, false
|
||||
}
|
||||
return canonical[0], true
|
||||
}
|
||||
|
||||
func canonicalReferences(refs []source.SourceRef) []source.SourceRef {
|
||||
canonical := make([]source.SourceRef, 0, len(refs))
|
||||
for _, ref := range refs {
|
||||
if validIdentityReference(ref) {
|
||||
canonical = append(canonical, ref)
|
||||
}
|
||||
}
|
||||
sort.Slice(canonical, func(left, right int) bool {
|
||||
if canonical[left].SourceID != canonical[right].SourceID {
|
||||
return canonical[left].SourceID < canonical[right].SourceID
|
||||
}
|
||||
if canonical[left].StartUnitID != canonical[right].StartUnitID {
|
||||
return canonical[left].StartUnitID < canonical[right].StartUnitID
|
||||
}
|
||||
return canonical[left].EndUnitID < canonical[right].EndUnitID
|
||||
})
|
||||
unique := canonical[:0]
|
||||
for _, ref := range canonical {
|
||||
if len(unique) == 0 || unique[len(unique)-1] != ref {
|
||||
unique = append(unique, ref)
|
||||
}
|
||||
}
|
||||
return unique
|
||||
}
|
||||
|
||||
func validIdentityReference(ref source.SourceRef) bool {
|
||||
return strings.TrimSpace(ref.SourceID) == ref.SourceID && ref.SourceID != "" && ref.StartUnitID > 0 && ref.EndUnitID > 0
|
||||
}
|
||||
146
internal/modules/dnd/locations/identity/identity_test.go
Normal file
146
internal/modules/dnd/locations/identity/identity_test.go
Normal file
@@ -0,0 +1,146 @@
|
||||
package identity
|
||||
|
||||
import (
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
)
|
||||
|
||||
func TestNormalizeDisplayOnlyChangesWhitespace(t *testing.T) {
|
||||
if got, want := NormalizeDisplay(" The\u2003Old\nTavern "), "The Old Tavern"; got != want {
|
||||
t.Fatalf("NormalizeDisplay() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestComparisonKeyNormalizesUnicodeWhitespaceAndApostrophes(t *testing.T) {
|
||||
for _, pair := range [][2]string{
|
||||
{" Caf\u00e9\u2003d\u2019Or ", "cafe\u0301 d'Or"},
|
||||
{"\uff34\uff48\uff45\u00a0\uff34\uff41\uff56\uff45\uff52\uff4e", "the tavern"},
|
||||
} {
|
||||
if left, right := ComparisonKey(pair[0]), ComparisonKey(pair[1]); left != right {
|
||||
t.Fatalf("ComparisonKey(%q) = %q, want value equal to %q", pair[0], left, pair[1])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeriveIDUsesDocumentedCompactJSONInput(t *testing.T) {
|
||||
refs := []source.SourceRef{
|
||||
{SourceID: "session-alpha", StartUnitID: 9, EndUnitID: 9},
|
||||
{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 9},
|
||||
{SourceID: "session-alpha", StartUnitID: 3, EndUnitID: 9},
|
||||
}
|
||||
const want = "location:sha256:379bd996e754e143b47e6b613a95166e58c111c7451750df7066ca816bb1866c"
|
||||
|
||||
if got := DeriveID(" The\u2003Tavern ", refs); got != want {
|
||||
t.Fatalf("DeriveID() = %q, want %q", got, want)
|
||||
}
|
||||
if got := IDFor("The Tavern", refs); got != want {
|
||||
t.Fatalf("IDFor() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeriveIDUsesEarliestCanonicalEvidenceWithoutMutatingInput(t *testing.T) {
|
||||
refs := []source.SourceRef{
|
||||
{SourceID: "zeta", StartUnitID: 1, EndUnitID: 1},
|
||||
{SourceID: "alpha", StartUnitID: 8, EndUnitID: 8},
|
||||
{SourceID: "alpha", StartUnitID: 2, EndUnitID: 3},
|
||||
}
|
||||
wantRefs := append([]source.SourceRef(nil), refs...)
|
||||
|
||||
first := DeriveID("The Tavern", refs)
|
||||
second := DeriveID("The Tavern", []source.SourceRef{refs[2], refs[0], refs[1]})
|
||||
if first != second {
|
||||
t.Fatalf("DeriveID() changed when evidence order changed: %q != %q", first, second)
|
||||
}
|
||||
if !reflect.DeepEqual(refs, wantRefs) {
|
||||
t.Fatalf("DeriveID() mutated refs: got %#v, want %#v", refs, wantRefs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeriveIDDistinguishesSameNameAtDifferentEvidenceAnchors(t *testing.T) {
|
||||
first := DeriveID("The Tavern", []source.SourceRef{{SourceID: "session", StartUnitID: 3, EndUnitID: 3}})
|
||||
second := DeriveID("the\u00a0tavern", []source.SourceRef{{SourceID: "session", StartUnitID: 18, EndUnitID: 18}})
|
||||
if first == "" || second == "" || first == second {
|
||||
t.Fatalf("same-name locations have IDs %q and %q, want distinct nonempty IDs", first, second)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDeriveIDRejectsMissingIdentityComponents(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
name string
|
||||
nameValue string
|
||||
refs []source.SourceRef
|
||||
}{
|
||||
{name: "blank name", nameValue: " \u2003 ", refs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
|
||||
{name: "missing refs", nameValue: "The Tavern"},
|
||||
{name: "malformed ref", nameValue: "The Tavern", refs: []source.SourceRef{{SourceID: " ", StartUnitID: 1, EndUnitID: 1}}},
|
||||
} {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
if got := DeriveID(tt.nameValue, tt.refs); got != "" {
|
||||
t.Fatalf("DeriveID() = %q, want empty ID", got)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsValidID(t *testing.T) {
|
||||
valid := DeriveID("The Tavern", []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}})
|
||||
if !IsValidID(valid) || !ValidID(valid) {
|
||||
t.Fatalf("derived ID %q is not valid", valid)
|
||||
}
|
||||
for _, value := range []string{"", "location:sha256:", "location:sha256:" + strings.Repeat("A", 64), "location:sha256:" + strings.Repeat("0", 63)} {
|
||||
if IsValidID(value) {
|
||||
t.Fatalf("IsValidID(%q) = true, want false", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRegistryAllowsSameComparisonNameWithDifferentAnchors(t *testing.T) {
|
||||
firstRefs := []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}
|
||||
secondRefs := []source.SourceRef{{SourceID: "session", StartUnitID: 8, EndUnitID: 8}}
|
||||
locations := []dnd.Location{
|
||||
{ID: DeriveID("The Tavern", firstRefs), Name: "The Tavern", SourceRefs: firstRefs},
|
||||
{ID: DeriveID("the\u00a0tavern", secondRefs), Name: "the\u00a0tavern", SourceRefs: secondRefs},
|
||||
}
|
||||
if issues := ValidateRegistry(locations); len(issues) != 0 {
|
||||
t.Fatalf("ValidateRegistry() = %#v, want no issues", issues)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRegistryReportsIdentityProblemsDeterministicallyWithoutMutation(t *testing.T) {
|
||||
refs := []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}
|
||||
validID := DeriveID("The Tavern", refs)
|
||||
locations := []dnd.Location{
|
||||
{ID: validID, Name: "The Tavern", SourceRefs: refs},
|
||||
{ID: validID, Name: "Elsewhere", SourceRefs: refs},
|
||||
{ID: "bad", Name: "", SourceRefs: nil},
|
||||
}
|
||||
wantLocations := cloneLocations(locations)
|
||||
|
||||
issues := ValidateList(dnd.LocationList{Locations: locations})
|
||||
want := []Issue{
|
||||
{Code: IssueIDMismatch, RecordIndex: 1, Value: validID},
|
||||
{Code: IssueEmptyCanonicalName, RecordIndex: 2, Value: ""},
|
||||
{Code: IssueMissingEvidence, RecordIndex: 2},
|
||||
{Code: IssueInvalidID, RecordIndex: 2, Value: "bad"},
|
||||
{Code: IssueDuplicateID, RecordIndex: 1, Value: validID},
|
||||
}
|
||||
if !reflect.DeepEqual(issues, want) {
|
||||
t.Fatalf("ValidateList() = %#v, want %#v", issues, want)
|
||||
}
|
||||
if !reflect.DeepEqual(locations, wantLocations) {
|
||||
t.Fatalf("ValidateList() mutated locations: got %#v, want %#v", locations, wantLocations)
|
||||
}
|
||||
}
|
||||
|
||||
func cloneLocations(input []dnd.Location) []dnd.Location {
|
||||
output := make([]dnd.Location, len(input))
|
||||
copy(output, input)
|
||||
for index := range output {
|
||||
output[index].SourceRefs = append([]source.SourceRef(nil), input[index].SourceRefs...)
|
||||
}
|
||||
return output
|
||||
}
|
||||
277
internal/modules/dnd/locations/registry/registry.go
Normal file
277
internal/modules/dnd/locations/registry/registry.go
Normal file
@@ -0,0 +1,277 @@
|
||||
// Package registry resolves normalized location artifacts into immutable
|
||||
// ID-grounding data for D&D extraction modules.
|
||||
package registry
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locations"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/registryresolver"
|
||||
)
|
||||
|
||||
const (
|
||||
ReferenceSlot = "locations"
|
||||
MaxBytes = 1048576
|
||||
emptyPrompt = `{"locations":[]}`
|
||||
)
|
||||
|
||||
// Registry is an immutable, validated location registry prepared for prompt
|
||||
// grounding. All accessors return defensive copies.
|
||||
type Registry struct {
|
||||
bound bool
|
||||
list dnd.LocationList
|
||||
canonical []byte
|
||||
digest string
|
||||
projectionDigest string
|
||||
promptInput contracts.LLMInputMaterial
|
||||
lookupByID map[string]int
|
||||
}
|
||||
|
||||
// Resolver selects and memoizes immutable location registry views.
|
||||
type Resolver struct {
|
||||
resolver *registryresolver.Resolver[*Registry]
|
||||
}
|
||||
|
||||
// NewResolver validates a materialized construction-time location reference.
|
||||
// An empty slot is permitted because a generated reference is supplied only at
|
||||
// operation time.
|
||||
func NewResolver(references contracts.ReferenceSet) (*Resolver, error) {
|
||||
resolver, err := registryresolver.New(registryResolverConfig(), references)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Resolver{resolver: resolver}, nil
|
||||
}
|
||||
|
||||
// Seeded returns the validated construction-time registry.
|
||||
func (r *Resolver) Seeded() *Registry {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
return r.resolver.Seeded()
|
||||
}
|
||||
|
||||
// Resolve returns the generated operation-time registry when the locations
|
||||
// slot is present, otherwise it returns the construction-time registry.
|
||||
func (r *Resolver) Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
if r == nil || r.resolver == nil {
|
||||
return Resolve(references)
|
||||
}
|
||||
return r.resolver.Resolve(references)
|
||||
}
|
||||
|
||||
// Resolve validates an optional location-list reference. An absent reference
|
||||
// uses the canonical empty projection and has no durable registry identity.
|
||||
func Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
item, present, err := registryresolver.ResolveOptionalSingleItem(references, locationReferenceSpec())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !present {
|
||||
return emptyRegistry(), nil
|
||||
}
|
||||
return loadRegistry(item.Content)
|
||||
}
|
||||
|
||||
func registryResolverConfig() registryresolver.Config[*Registry] {
|
||||
return registryresolver.Config[*Registry]{
|
||||
Reference: locationReferenceSpec(),
|
||||
Absent: func() (*Registry, error) {
|
||||
return emptyRegistry(), nil
|
||||
},
|
||||
Load: loadRegistry,
|
||||
SemanticIdentity: func(registry *Registry) string {
|
||||
return registry.Digest()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func locationReferenceSpec() registryresolver.ReferenceSpec {
|
||||
return registryresolver.ReferenceSpec{SlotName: ReferenceSlot, AcceptedMediaType: locationcodec.MediaType, MaxBytes: MaxBytes}
|
||||
}
|
||||
|
||||
func emptyRegistry() *Registry {
|
||||
content := []byte(emptyPrompt)
|
||||
projectionDigest := semanticDigest(content)
|
||||
return &Registry{
|
||||
list: dnd.LocationList{Locations: []dnd.Location{}},
|
||||
canonical: append([]byte(nil), content...),
|
||||
projectionDigest: projectionDigest,
|
||||
promptInput: contracts.NewLLMInputMaterial(ReferenceSlot, locationcodec.MediaType, content, projectionDigest, ""),
|
||||
lookupByID: map[string]int{},
|
||||
}
|
||||
}
|
||||
|
||||
func loadRegistry(referenceContent []byte) (*Registry, error) {
|
||||
codec := locationcodec.New()
|
||||
value, err := codec.Decode(referenceContent)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("decode location registry: invalid approved location JSON")
|
||||
}
|
||||
if issues := identity.ValidateList(value); len(issues) > 0 {
|
||||
return nil, fmt.Errorf("%s", formatIdentityIssues(issues))
|
||||
}
|
||||
content, err := codec.Encode(value)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encode canonical location registry: approved location value could not be encoded")
|
||||
}
|
||||
|
||||
list := cloneLocationList(value)
|
||||
lookupByID := make(map[string]int, len(list.Locations))
|
||||
for index, location := range list.Locations {
|
||||
lookupByID[location.ID] = index
|
||||
}
|
||||
projection, err := promptProjection(list)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("encode location prompt projection: %w", err)
|
||||
}
|
||||
digest := semanticDigest(content)
|
||||
projectionDigest := semanticDigest(projection)
|
||||
return &Registry{
|
||||
bound: true,
|
||||
list: list,
|
||||
canonical: append([]byte(nil), content...),
|
||||
digest: digest,
|
||||
projectionDigest: projectionDigest,
|
||||
promptInput: contracts.NewLLMInputMaterial(ReferenceSlot, locationcodec.MediaType, projection, projectionDigest, ""),
|
||||
lookupByID: lookupByID,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Bound reports whether a location reference was supplied and validated.
|
||||
func (r *Registry) Bound() bool { return r != nil && r.bound }
|
||||
|
||||
// Locations returns a defensive copy of the validated location records.
|
||||
func (r *Registry) Locations() []dnd.Location {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
return cloneLocations(r.list.Locations)
|
||||
}
|
||||
|
||||
// List returns a defensive copy of the validated location list.
|
||||
func (r *Registry) List() dnd.LocationList {
|
||||
if r == nil {
|
||||
return dnd.LocationList{}
|
||||
}
|
||||
return cloneLocationList(r.list)
|
||||
}
|
||||
|
||||
// CanonicalBytes returns a defensive copy of the canonical durable JSON.
|
||||
func (r *Registry) CanonicalBytes() []byte {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
return append([]byte(nil), r.canonical...)
|
||||
}
|
||||
|
||||
// Digest returns the semantic digest of the canonical JSON, or an empty string
|
||||
// when the registry is unbound.
|
||||
func (r *Registry) Digest() string {
|
||||
if r == nil {
|
||||
return ""
|
||||
}
|
||||
return r.digest
|
||||
}
|
||||
|
||||
// ProjectionDigest returns the digest of the exact source-free prompt
|
||||
// projection, including for an unbound or empty registry.
|
||||
func (r *Registry) ProjectionDigest() string {
|
||||
if r == nil {
|
||||
return ""
|
||||
}
|
||||
return r.projectionDigest
|
||||
}
|
||||
|
||||
// Count returns the number of validated location records.
|
||||
func (r *Registry) Count() int {
|
||||
if r == nil {
|
||||
return 0
|
||||
}
|
||||
return len(r.list.Locations)
|
||||
}
|
||||
|
||||
// PromptInput returns the ordered ID-and-name projection without evidence or
|
||||
// reference provenance.
|
||||
func (r *Registry) PromptInput() contracts.LLMInputMaterial {
|
||||
if r == nil {
|
||||
return contracts.LLMInputMaterial{}
|
||||
}
|
||||
return r.promptInput.Clone()
|
||||
}
|
||||
|
||||
// Lookup returns the canonical location for an exact durable location ID.
|
||||
func (r *Registry) Lookup(id string) (dnd.Location, bool) {
|
||||
if r == nil {
|
||||
return dnd.Location{}, false
|
||||
}
|
||||
index, ok := r.lookupByID[id]
|
||||
if !ok {
|
||||
return dnd.Location{}, false
|
||||
}
|
||||
return cloneLocation(r.list.Locations[index]), true
|
||||
}
|
||||
|
||||
// Matches reports whether id resolves to exactly the supplied canonical name.
|
||||
func (r *Registry) Matches(id, name string) bool {
|
||||
location, ok := r.Lookup(id)
|
||||
return ok && location.Name == name
|
||||
}
|
||||
|
||||
func semanticDigest(content []byte) string {
|
||||
sum := sha256.Sum256(content)
|
||||
return "sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
type projectedLocation struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
}
|
||||
|
||||
type projectedLocationList struct {
|
||||
Locations []projectedLocation `json:"locations"`
|
||||
}
|
||||
|
||||
func promptProjection(list dnd.LocationList) ([]byte, error) {
|
||||
projection := projectedLocationList{Locations: make([]projectedLocation, len(list.Locations))}
|
||||
for index, location := range list.Locations {
|
||||
projection.Locations[index] = projectedLocation{ID: location.ID, Name: location.Name}
|
||||
}
|
||||
return json.Marshal(projection)
|
||||
}
|
||||
|
||||
func formatIdentityIssues(issues []identity.Issue) string {
|
||||
parts := make([]string, len(issues))
|
||||
for index, issue := range issues {
|
||||
parts[index] = fmt.Sprintf("%s at record %d", issue.Code, issue.RecordIndex)
|
||||
}
|
||||
return diagnostics.Aggregate("validate location registry identity", parts)
|
||||
}
|
||||
|
||||
func cloneLocationList(value dnd.LocationList) dnd.LocationList {
|
||||
return dnd.LocationList{Locations: cloneLocations(value.Locations)}
|
||||
}
|
||||
|
||||
func cloneLocations(values []dnd.Location) []dnd.Location {
|
||||
if values == nil {
|
||||
return nil
|
||||
}
|
||||
cloned := make([]dnd.Location, len(values))
|
||||
for index, value := range values {
|
||||
cloned[index] = cloneLocation(value)
|
||||
}
|
||||
return cloned
|
||||
}
|
||||
|
||||
func cloneLocation(value dnd.Location) dnd.Location {
|
||||
value.SourceRefs = append([]source.SourceRef(nil), value.SourceRefs...)
|
||||
return value
|
||||
}
|
||||
191
internal/modules/dnd/locations/registry/registry_test.go
Normal file
191
internal/modules/dnd/locations/registry/registry_test.go
Normal file
@@ -0,0 +1,191 @@
|
||||
package registry
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locations"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
)
|
||||
|
||||
func TestResolveUnboundRegistryHasEmptyProjection(t *testing.T) {
|
||||
registry, err := Resolve(contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
input := registry.PromptInput()
|
||||
if registry.Bound() || registry.Digest() != "" || registry.Count() != 0 || string(input.Content) != emptyPrompt {
|
||||
t.Fatalf("unbound registry = %#v, input = %#v", registry, input)
|
||||
}
|
||||
if registry.ProjectionDigest() == "" || input.Digest != registry.ProjectionDigest() || input.OriginURI != "" {
|
||||
t.Fatalf("projection digest/input = %q/%#v", registry.ProjectionDigest(), input)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveProjectsOrderedLocationsWithoutEvidence(t *testing.T) {
|
||||
registry := resolveList(t, registryFixture())
|
||||
if !registry.Bound() || registry.Count() != 2 || registry.Digest() == "" {
|
||||
t.Fatalf("registry identity = bound %t count %d digest %q", registry.Bound(), registry.Count(), registry.Digest())
|
||||
}
|
||||
projection := string(registry.PromptInput().Content)
|
||||
if !strings.Contains(projection, `"locations":[{"id":`) || !strings.Contains(projection, `"name":"The Tavern"`) || !strings.Contains(projection, `"name":"The Tavern"},{"id":`) {
|
||||
t.Fatalf("projection ordering = %s", projection)
|
||||
}
|
||||
for _, forbidden := range []string{"source_refs", "source_id", "session-alpha"} {
|
||||
if strings.Contains(projection, forbidden) {
|
||||
t.Fatalf("projection leaked %q: %s", forbidden, projection)
|
||||
}
|
||||
}
|
||||
if registry.PromptInput().Digest != registry.ProjectionDigest() || registry.Digest() == registry.ProjectionDigest() {
|
||||
t.Fatalf("full/projection digests = %q/%q", registry.Digest(), registry.ProjectionDigest())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRegistryLookupUsesIDAndReturnsDefensiveCopies(t *testing.T) {
|
||||
registry := resolveList(t, registryFixture())
|
||||
first := registry.Locations()[0]
|
||||
if got, ok := registry.Lookup(first.ID); !ok || got.Name != first.Name || !registry.Matches(first.ID, first.Name) || registry.Matches(first.ID, "Other Tavern") {
|
||||
t.Fatalf("ID lookup/match = %#v, %t", got, ok)
|
||||
}
|
||||
if _, ok := registry.Lookup("The Tavern"); ok {
|
||||
t.Fatal("Lookup accepted a name as an ID")
|
||||
}
|
||||
|
||||
locations := registry.Locations()
|
||||
locations[0].Name = "changed"
|
||||
locations[0].SourceRefs[0].SourceID = "changed"
|
||||
canonical := registry.CanonicalBytes()
|
||||
canonical[0] = '['
|
||||
input := registry.PromptInput()
|
||||
input.Content[0] = '['
|
||||
if next, ok := registry.Lookup(first.ID); !ok || next.Name != first.Name || next.SourceRefs[0].SourceID != "session-alpha" {
|
||||
t.Fatalf("registry mutated through accessor: %#v, %t", next, ok)
|
||||
}
|
||||
if registry.CanonicalBytes()[0] != '{' || registry.PromptInput().Content[0] != '{' {
|
||||
t.Fatal("registry bytes mutated through accessor")
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveRejectsInvalidReferenceInputs(t *testing.T) {
|
||||
valid := registryFixture()
|
||||
content, err := locationcodec.New().Encode(valid)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
invalidIdentity := append([]byte(nil), content...)
|
||||
invalidIdentity = bytes.Replace(invalidIdentity, []byte(valid.Locations[0].ID), []byte("location:sha256:0000000000000000000000000000000000000000000000000000000000000000"), 1)
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
set contracts.ReferenceSet
|
||||
want string
|
||||
}{
|
||||
{"empty bound slot", referenceSet(), "exactly one item"},
|
||||
{"multiple items", referenceSet(item(content), item(content)), "exactly one item"},
|
||||
{"malformed JSON", referenceSet(item([]byte(`{"locations":[`))), "invalid approved"},
|
||||
{"wrong media type", referenceSet(contracts.ReferenceItem{MediaType: "text/plain", Content: content}), "media type must be"},
|
||||
{"oversized", referenceSet(item(make([]byte, MaxBytes+1))), "limit"},
|
||||
{"invalid identity", referenceSet(item(invalidIdentity)), "id_mismatch"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
if _, err := Resolve(test.set); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("Resolve() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverHandlesConstructionAndOperationReferences(t *testing.T) {
|
||||
placeholder, err := NewResolver(referenceSet())
|
||||
if err != nil || placeholder.Seeded().Bound() {
|
||||
t.Fatalf("generated placeholder = %#v, %v; want unbound seed", placeholder, err)
|
||||
}
|
||||
|
||||
validContent := encodeList(t, registryFixture())
|
||||
malformed := referenceSet(item([]byte(`{"locations":[`)))
|
||||
if _, err := NewResolver(malformed); err == nil {
|
||||
t.Fatal("NewResolver(malformed) error = nil")
|
||||
}
|
||||
if _, err := placeholder.Resolve(malformed); err == nil {
|
||||
t.Fatal("Resolve(malformed) error = nil")
|
||||
}
|
||||
|
||||
staticContent := append([]byte(nil), validContent...)
|
||||
staticReferences := referenceSet(item(staticContent))
|
||||
seeded, err := NewResolver(staticReferences)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, err := seeded.Resolve(staticReferences)
|
||||
if err != nil || resolved != seeded.Seeded() {
|
||||
t.Fatalf("seed reuse = %p / %p, %v", resolved, seeded.Seeded(), err)
|
||||
}
|
||||
staticContent[0] = '['
|
||||
delete(staticReferences.Slots, ReferenceSlot)
|
||||
if seeded.Seeded().Count() != 2 || seeded.Seeded().CanonicalBytes()[0] != '{' {
|
||||
t.Fatalf("seeded registry retained construction references: %#v", seeded.Seeded())
|
||||
}
|
||||
if fallback, err := seeded.Resolve(contracts.ReferenceSet{}); err != nil || fallback != seeded.Seeded() {
|
||||
t.Fatalf("fallback = %#v, %v; want seeded registry", fallback, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverReusesEquivalentCanonicalRegistries(t *testing.T) {
|
||||
resolver, err := NewResolver(referenceSet())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
content, err := locationcodec.New().Encode(registryFixture())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
firstSet := referenceSet(item(content))
|
||||
first, err := resolver.Resolve(firstSet)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
spaced := append([]byte("\n "), content...)
|
||||
spaced = append(spaced, '\n')
|
||||
second, err := resolver.Resolve(referenceSet(contracts.ReferenceItem{MediaType: "APPLICATION/JSON; charset=utf-8", Content: spaced}))
|
||||
if err != nil || second != first {
|
||||
t.Fatalf("equivalent canonical registry = %p / %p, %v", first, second, err)
|
||||
}
|
||||
}
|
||||
|
||||
func registryFixture() dnd.LocationList {
|
||||
firstRefs := []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 1, EndUnitID: 1}}
|
||||
secondRefs := []source.SourceRef{{SourceID: "session-alpha", StartUnitID: 2, EndUnitID: 2}}
|
||||
return dnd.LocationList{Locations: []dnd.Location{
|
||||
{ID: identity.DeriveID("The Tavern", firstRefs), Name: "The Tavern", SourceRefs: firstRefs},
|
||||
{ID: identity.DeriveID("The Tavern", secondRefs), Name: "The Tavern", SourceRefs: secondRefs},
|
||||
}}
|
||||
}
|
||||
|
||||
func resolveList(t *testing.T, list dnd.LocationList) *Registry {
|
||||
t.Helper()
|
||||
registry, err := Resolve(referenceSet(item(encodeList(t, list))))
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve() error = %v", err)
|
||||
}
|
||||
return registry
|
||||
}
|
||||
|
||||
func encodeList(t *testing.T, list dnd.LocationList) []byte {
|
||||
t.Helper()
|
||||
content, err := locationcodec.New().Encode(list)
|
||||
if err != nil {
|
||||
t.Fatalf("Encode() error = %v", err)
|
||||
}
|
||||
return content
|
||||
}
|
||||
|
||||
func item(content []byte) contracts.ReferenceItem {
|
||||
return contracts.ReferenceItem{SlotName: ReferenceSlot, MediaType: locationcodec.MediaType, Content: content}
|
||||
}
|
||||
|
||||
func referenceSet(items ...contracts.ReferenceItem) contracts.ReferenceSet {
|
||||
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{ReferenceSlot: {Items: items}}}
|
||||
}
|
||||
361
internal/modules/dnd/normalize/locationoccurrences/normalizer.go
Normal file
361
internal/modules/dnd/normalize/locationoccurrences/normalizer.go
Normal file
@@ -0,0 +1,361 @@
|
||||
// Package locationoccurrences normalizes merged D&D location occurrence candidates.
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationregistry "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/registry"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
)
|
||||
|
||||
const (
|
||||
Key = "dnd/location-occurrences"
|
||||
normalizationPolicy = "dnd.location_occurrences.normalize.v1"
|
||||
NormalizationPolicy = normalizationPolicy
|
||||
|
||||
ReasonCodeNameCanonicalized = "location_occurrence_name_canonicalized"
|
||||
ReasonCodeUnknownLocationID = "location_occurrence_unknown_location_id"
|
||||
ReasonCodeSourceRefsNormalized = "source_references_normalized"
|
||||
ReasonCodeOccurrencesReordered = "location_occurrences_reordered"
|
||||
ReasonCodeDuplicateCollapsed = "duplicate_location_occurrence_collapsed"
|
||||
ReasonCodeWarningsOmitted = "location_occurrence_normalization_warnings_omitted"
|
||||
)
|
||||
|
||||
const (
|
||||
LocationRegistryReferenceSlot = locationregistry.ReferenceSlot
|
||||
LocationRegistryMaxBytes = locationregistry.MaxBytes
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"merged"}
|
||||
var providedCapabilities = []string{"normalized"}
|
||||
|
||||
var _ contracts.Normalizer[dnd.LocationOccurrenceList] = (*Normalizer)(nil)
|
||||
var _ contracts.ManifestMetadataProvider = (*Normalizer)(nil)
|
||||
var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
|
||||
|
||||
type Options struct{}
|
||||
|
||||
type Normalizer struct {
|
||||
locationResolver *locationregistry.Resolver
|
||||
}
|
||||
|
||||
func New(_ Options, references ...contracts.ReferenceSet) (*Normalizer, error) {
|
||||
if len(references) > 1 {
|
||||
return nil, normalizerErrorf("at most one reference set may be supplied")
|
||||
}
|
||||
var referenceSet contracts.ReferenceSet
|
||||
if len(references) == 1 {
|
||||
referenceSet = references[0]
|
||||
}
|
||||
resolver, err := locationregistry.NewResolver(referenceSet)
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("prepare location registry: %w", err)
|
||||
}
|
||||
return &Normalizer{locationResolver: resolver}, nil
|
||||
}
|
||||
|
||||
func (n *Normalizer) Key() string { return Key }
|
||||
|
||||
func (n *Normalizer) ReferenceSlots() []contracts.ReferenceSlot { return referenceSlots() }
|
||||
|
||||
func (n *Normalizer) ManifestMetadata() map[string]any {
|
||||
if n == nil || n.locationResolver == nil {
|
||||
return nil
|
||||
}
|
||||
metadata := map[string]any{"normalization_policy": normalizationPolicy}
|
||||
seeded := n.locationResolver.Seeded()
|
||||
if seeded.Bound() {
|
||||
metadata["location_registry_digest"] = seeded.Digest()
|
||||
metadata["location_count"] = seeded.Count()
|
||||
}
|
||||
return metadata
|
||||
}
|
||||
|
||||
func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
if n == nil || n.locationResolver == nil {
|
||||
return nil
|
||||
}
|
||||
return []pipeline.CheckpointFingerprint{
|
||||
{Name: "normalization_policy", Value: normalizationPolicy},
|
||||
{Name: "location_registry", Value: n.locationResolver.Seeded().ProjectionDigest()},
|
||||
}
|
||||
}
|
||||
|
||||
func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[dnd.LocationOccurrenceList]) (contracts.TypedNormalizeResult[dnd.LocationOccurrenceList], error) {
|
||||
if n == nil || n.locationResolver == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("normalizer must not be nil")
|
||||
}
|
||||
if ctx == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("context must not be nil")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("context error before normalize: %w", err)
|
||||
}
|
||||
registry, err := n.locationResolver.Resolve(req.References)
|
||||
if err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("resolve location registry: %w", err)
|
||||
}
|
||||
if !registry.Bound() {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{}, normalizerErrorf("location registry reference is required")
|
||||
}
|
||||
index := source.NewDocumentIndex(req.Source)
|
||||
value, warnings := normalizeList(req.MergeOutput.Value, index, shared.NewSourceRefOrderFromIndex(index), registry)
|
||||
return contracts.TypedNormalizeResult[dnd.LocationOccurrenceList]{Value: value, Warnings: warnings}, nil
|
||||
}
|
||||
|
||||
type normalizedRecord struct {
|
||||
occurrence dnd.LocationOccurrence
|
||||
inputIndex int
|
||||
}
|
||||
|
||||
type nameCanonicalization struct{ from, to string }
|
||||
|
||||
func normalizeList(input dnd.LocationOccurrenceList, documentIndex source.DocumentIndex, order shared.SourceRefOrder, registry *locationregistry.Registry) (dnd.LocationOccurrenceList, []contracts.Warning) {
|
||||
if input.Occurrences == nil {
|
||||
return dnd.LocationOccurrenceList{}, nil
|
||||
}
|
||||
records := make([]normalizedRecord, len(input.Occurrences))
|
||||
warnings := make([]contracts.Warning, 0)
|
||||
for index, inputOccurrence := range input.Occurrences {
|
||||
occurrence, change, found, refsChanged := normalizeOccurrence(inputOccurrence, order, registry)
|
||||
records[index] = normalizedRecord{occurrence: occurrence, inputIndex: index}
|
||||
if change != nil {
|
||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeNameCanonicalized,
|
||||
Message: fmt.Sprintf("input index %d: location name canonicalized from %s to %s", index, diagnostics.Quote(change.from), diagnostics.Quote(change.to))})
|
||||
}
|
||||
if !found {
|
||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeUnknownLocationID,
|
||||
Message: fmt.Sprintf("input index %d: location ID %s is not in the supplied registry", index, diagnostics.Quote(inputOccurrence.LocationID))})
|
||||
}
|
||||
if refsChanged {
|
||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(index), ReasonCode: ReasonCodeSourceRefsNormalized,
|
||||
Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputOccurrence.SourceRefs), len(occurrence.SourceRefs))})
|
||||
}
|
||||
}
|
||||
sort.SliceStable(records, func(left, right int) bool {
|
||||
return lessOccurrence(order, records[left].occurrence, records[right].occurrence)
|
||||
})
|
||||
for position, record := range records {
|
||||
if position != record.inputIndex {
|
||||
warnings = append(warnings, contracts.Warning{Scope: occurrenceScope(record.inputIndex), ReasonCode: ReasonCodeOccurrencesReordered,
|
||||
Message: fmt.Sprintf("input index %d moved to normalized position %d by canonical occurrence order", record.inputIndex, position)})
|
||||
}
|
||||
}
|
||||
output, duplicateWarnings := collapseDuplicates(records, documentIndex)
|
||||
warnings = append(warnings, duplicateWarnings...)
|
||||
return dnd.LocationOccurrenceList{Occurrences: output}, diagnostics.LimitWarnings(warnings, "location_occurrences", ReasonCodeWarningsOmitted)
|
||||
}
|
||||
|
||||
func normalizeOccurrence(input dnd.LocationOccurrence, order shared.SourceRefOrder, registry *locationregistry.Registry) (dnd.LocationOccurrence, *nameCanonicalization, bool, bool) {
|
||||
output := cloneOccurrence(input)
|
||||
canonical, found := registry.Lookup(input.LocationID)
|
||||
if found {
|
||||
output.Name = canonical.Name
|
||||
}
|
||||
var change *nameCanonicalization
|
||||
if input.Name != output.Name {
|
||||
change = &nameCanonicalization{from: input.Name, to: output.Name}
|
||||
}
|
||||
output.SourceRefs = order.Canonicalize(input.SourceRefs)
|
||||
return output, change, found, !sourceRefsEqual(input.SourceRefs, output.SourceRefs)
|
||||
}
|
||||
|
||||
func cloneOccurrence(input dnd.LocationOccurrence) dnd.LocationOccurrence {
|
||||
input.SourceRefs = append([]source.SourceRef(nil), input.SourceRefs...)
|
||||
return input
|
||||
}
|
||||
|
||||
func sourceRefsEqual(left, right []source.SourceRef) bool {
|
||||
if (left == nil) != (right == nil) || len(left) != len(right) {
|
||||
return false
|
||||
}
|
||||
for index := range left {
|
||||
if left[index] != right[index] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type duplicateGroup struct {
|
||||
retainedIndex int
|
||||
removed []int
|
||||
}
|
||||
|
||||
func collapseDuplicates(records []normalizedRecord, documentIndex source.DocumentIndex) ([]dnd.LocationOccurrence, []contracts.Warning) {
|
||||
if len(records) == 0 {
|
||||
return make([]dnd.LocationOccurrence, 0), nil
|
||||
}
|
||||
keep := make([]bool, len(records))
|
||||
groups := make([]duplicateGroup, 0)
|
||||
groupByKey := make(map[string]int)
|
||||
for index, record := range records {
|
||||
if !validSourceRefs(documentIndex, record.occurrence.SourceRefs) {
|
||||
keep[index] = true
|
||||
continue
|
||||
}
|
||||
key := exactIdentity(record.occurrence)
|
||||
groupIndex, exists := groupByKey[key]
|
||||
if !exists {
|
||||
groupByKey[key] = len(groups)
|
||||
groups = append(groups, duplicateGroup{retainedIndex: record.inputIndex})
|
||||
keep[index] = true
|
||||
continue
|
||||
}
|
||||
groups[groupIndex].removed = append(groups[groupIndex].removed, record.inputIndex)
|
||||
}
|
||||
output := make([]dnd.LocationOccurrence, 0, len(records))
|
||||
for index, record := range records {
|
||||
if keep[index] {
|
||||
output = append(output, cloneOccurrence(record.occurrence))
|
||||
}
|
||||
}
|
||||
warnings := make([]contracts.Warning, 0)
|
||||
for _, group := range groups {
|
||||
if len(group.removed) > 0 {
|
||||
warnings = append(warnings, duplicateWarning(group.retainedIndex, group.removed))
|
||||
}
|
||||
}
|
||||
return output, warnings
|
||||
}
|
||||
|
||||
func validSourceRefs(index source.DocumentIndex, refs []source.SourceRef) bool {
|
||||
if len(refs) == 0 {
|
||||
return false
|
||||
}
|
||||
for _, ref := range refs {
|
||||
if index.ValidateRef(ref) != nil {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func exactIdentity(occurrence dnd.LocationOccurrence) string {
|
||||
var key strings.Builder
|
||||
writeKeyString(&key, occurrence.LocationID)
|
||||
writeKeyString(&key, occurrence.Name)
|
||||
writeKeyString(&key, string(occurrence.Kind))
|
||||
for _, ref := range occurrence.SourceRefs {
|
||||
writeKeyString(&key, ref.SourceID)
|
||||
writeKeyInt(&key, ref.StartUnitID)
|
||||
writeKeyInt(&key, ref.EndUnitID)
|
||||
}
|
||||
return key.String()
|
||||
}
|
||||
|
||||
func writeKeyString(builder *strings.Builder, value string) {
|
||||
builder.WriteString(strconv.Itoa(len(value)))
|
||||
builder.WriteByte(':')
|
||||
builder.WriteString(value)
|
||||
}
|
||||
|
||||
func writeKeyInt(builder *strings.Builder, value int) {
|
||||
builder.WriteString(strconv.Itoa(value))
|
||||
builder.WriteByte(';')
|
||||
}
|
||||
|
||||
func lessOccurrence(order shared.SourceRefOrder, left, right dnd.LocationOccurrence) bool {
|
||||
leftPosition, leftHasEvidence := order.EarliestValid(left.SourceRefs)
|
||||
rightPosition, rightHasEvidence := order.EarliestValid(right.SourceRefs)
|
||||
if leftHasEvidence != rightHasEvidence {
|
||||
return leftHasEvidence
|
||||
}
|
||||
if leftHasEvidence && leftPosition != rightPosition {
|
||||
return leftPosition < rightPosition
|
||||
}
|
||||
if left.LocationID != right.LocationID {
|
||||
return left.LocationID < right.LocationID
|
||||
}
|
||||
if left.Name != right.Name {
|
||||
return left.Name < right.Name
|
||||
}
|
||||
if kindOrder(left.Kind) != kindOrder(right.Kind) {
|
||||
return kindOrder(left.Kind) < kindOrder(right.Kind)
|
||||
}
|
||||
if left.Kind != right.Kind {
|
||||
return left.Kind < right.Kind
|
||||
}
|
||||
return sourceRefsLess(order, left.SourceRefs, right.SourceRefs)
|
||||
}
|
||||
|
||||
func kindOrder(kind dnd.LocationOccurrenceKind) int {
|
||||
switch kind {
|
||||
case dnd.LocationOccurrenceKindVisited:
|
||||
return 0
|
||||
case dnd.LocationOccurrenceKindPlanned:
|
||||
return 1
|
||||
case dnd.LocationOccurrenceKindRecalled:
|
||||
return 2
|
||||
case dnd.LocationOccurrenceKindMentioned:
|
||||
return 3
|
||||
default:
|
||||
return 4
|
||||
}
|
||||
}
|
||||
|
||||
func sourceRefsLess(order shared.SourceRefOrder, left, right []source.SourceRef) bool {
|
||||
for index := 0; index < len(left) && index < len(right); index++ {
|
||||
if left[index] == right[index] {
|
||||
continue
|
||||
}
|
||||
return order.Less(left[index], right[index])
|
||||
}
|
||||
return len(left) < len(right)
|
||||
}
|
||||
|
||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
||||
issues := make([]string, len(removed))
|
||||
for index, removedIndex := range removed {
|
||||
issues[index] = fmt.Sprintf("removed input index %d", removedIndex)
|
||||
}
|
||||
return contracts.Warning{Scope: occurrenceScope(retainedIndex), ReasonCode: ReasonCodeDuplicateCollapsed,
|
||||
Message: diagnostics.Aggregate(fmt.Sprintf("duplicate location occurrence collapsed; retained input index %d", retainedIndex), issues)}
|
||||
}
|
||||
|
||||
func occurrenceScope(index int) string { return fmt.Sprintf("occurrences[%d]", index) }
|
||||
|
||||
func referenceSlots() []contracts.ReferenceSlot {
|
||||
return []contracts.ReferenceSlot{{
|
||||
Name: LocationRegistryReferenceSlot, Description: "Required normalized location registry used only for location identity grounding, never as occurrence evidence.",
|
||||
Required: true, AcceptedMediaTypes: []string{"application/json"}, AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.LocationListKind}, MaxBytes: LocationRegistryMaxBytes,
|
||||
}}
|
||||
}
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassDeterministic,
|
||||
Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...),
|
||||
ArtifactKind: dnd.LocationOccurrenceListKind, ReferenceSlots: referenceSlots()}
|
||||
}
|
||||
|
||||
func Register(registry *pipeline.NormalizerRegistry) error {
|
||||
return pipeline.RegisterNormalizerBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Normalizer[dnd.LocationOccurrenceList], error) {
|
||||
options, err := DecodeOptions(request.Options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(options, request.References)
|
||||
})
|
||||
}
|
||||
|
||||
func DecodeOptions(options map[string]any) (Options, error) {
|
||||
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||
return Options{}, normalizerErrorf("%w", err)
|
||||
}
|
||||
return Options{}, nil
|
||||
}
|
||||
|
||||
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||
|
||||
func normalizerErrorf(format string, args ...any) error {
|
||||
return fmt.Errorf("dnd location occurrences normalizer: "+format, args...)
|
||||
}
|
||||
@@ -0,0 +1,214 @@
|
||||
package locationoccurrences
|
||||
|
||||
import (
|
||||
"context"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locations"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
)
|
||||
|
||||
func TestNormalizeCanonicalizesNamesByIDAndClonesInputs(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 30}, {ID: 10}}}
|
||||
locations := registryLocations("The Tavern", "The Tavern")
|
||||
normalizer := newNormalizer(t, registryReferences(t, locations))
|
||||
input := dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{
|
||||
LocationID: locations.Locations[1].ID, Name: " a tavern ", Kind: dnd.LocationOccurrenceKindVisited,
|
||||
SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}, {SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}, {SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}},
|
||||
}}}
|
||||
before := cloneList(input)
|
||||
result, err := normalizer.Normalize(context.Background(), normalizeRequest(input, doc, contracts.ReferenceSet{}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
occurrence := result.Value.Occurrences[0]
|
||||
if occurrence.Name != locations.Locations[1].Name || !reflect.DeepEqual(occurrence.SourceRefs, []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}, {SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}) {
|
||||
t.Fatalf("normalized occurrence = %#v", occurrence)
|
||||
}
|
||||
if !hasWarning(result.Warnings, ReasonCodeNameCanonicalized) || !hasWarning(result.Warnings, ReasonCodeSourceRefsNormalized) || !reflect.DeepEqual(input, before) {
|
||||
t.Fatalf("warnings/input = %#v/%#v", result.Warnings, input)
|
||||
}
|
||||
second, err := normalizer.Normalize(context.Background(), normalizeRequest(result.Value, doc, contracts.ReferenceSet{}))
|
||||
if err != nil || !reflect.DeepEqual(second.Value, result.Value) || len(second.Warnings) != 0 {
|
||||
t.Fatalf("second normalization = %#v, %v", second, err)
|
||||
}
|
||||
result.Value.Occurrences[0].SourceRefs[0].StartUnitID = 999
|
||||
if input.Occurrences[0].SourceRefs[0].StartUnitID == 999 {
|
||||
t.Fatal("normalized source references share input storage")
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeKeepsSameNamedIDsAndDistinctEvidence(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 50}, {ID: 10}, {ID: 90}}}
|
||||
locations := registryLocations("The Tavern", "The Tavern")
|
||||
first, second := locations.Locations[0], locations.Locations[1]
|
||||
ref := func(unit int) source.SourceRef {
|
||||
return source.SourceRef{SourceID: doc.ID, StartUnitID: unit, EndUnitID: unit}
|
||||
}
|
||||
firstMention := dnd.LocationOccurrence{LocationID: first.ID, Name: first.Name, Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{ref(50)}}
|
||||
input := dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{
|
||||
{LocationID: second.ID, Name: second.Name, Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{ref(90)}},
|
||||
firstMention,
|
||||
firstMention,
|
||||
{LocationID: first.ID, Name: first.Name, Kind: dnd.LocationOccurrenceKindVisited, SourceRefs: []source.SourceRef{ref(50)}},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: dnd.LocationOccurrenceKindPlanned, SourceRefs: []source.SourceRef{ref(50)}},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: dnd.LocationOccurrenceKindRecalled, SourceRefs: []source.SourceRef{ref(50)}},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{ref(10)}},
|
||||
{LocationID: first.ID, Name: first.Name, Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{ref(999)}},
|
||||
}}
|
||||
result, err := newNormalizer(t, registryReferences(t, locations)).Normalize(context.Background(), normalizeRequest(input, doc, contracts.ReferenceSet{}))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got := result.Value.Occurrences
|
||||
if len(got) != 7 || got[0].Kind != dnd.LocationOccurrenceKindVisited || got[1].Kind != dnd.LocationOccurrenceKindPlanned || got[2].Kind != dnd.LocationOccurrenceKindRecalled || got[3].Kind != dnd.LocationOccurrenceKindMentioned || got[3].SourceRefs[0].StartUnitID != 50 || got[4].SourceRefs[0].StartUnitID != 10 || got[5].LocationID != second.ID || got[6].SourceRefs[0].StartUnitID != 999 {
|
||||
t.Fatalf("canonical occurrences = %#v", got)
|
||||
}
|
||||
if !hasWarning(result.Warnings, ReasonCodeDuplicateCollapsed) || !hasWarning(result.Warnings, ReasonCodeOccurrencesReordered) {
|
||||
t.Fatalf("warnings = %#v", result.Warnings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizePreservesUnknownIDsAndMalformedOperationRegistry(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 10}}}
|
||||
locations := registryLocations("The Mill")
|
||||
unknown := dnd.LocationOccurrence{LocationID: "location:sha256:unknown", Name: "The Mill", Kind: "unexpected", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}
|
||||
result, err := newNormalizer(t, registryReferences(t, locations)).Normalize(context.Background(), normalizeRequest(dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{unknown}}, doc, contracts.ReferenceSet{}))
|
||||
if err != nil || !reflect.DeepEqual(result.Value.Occurrences[0], unknown) || !hasWarning(result.Warnings, ReasonCodeUnknownLocationID) {
|
||||
t.Fatalf("unknown normalization = %#v, %v", result, err)
|
||||
}
|
||||
malformed := contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{LocationRegistryReferenceSlot: {
|
||||
Items: []contracts.ReferenceItem{{SlotName: LocationRegistryReferenceSlot, MediaType: "application/json", Content: []byte(`{"private":"registry evidence"}`)}},
|
||||
}}}
|
||||
if _, err := newNormalizer(t).Normalize(context.Background(), normalizeRequest(dnd.LocationOccurrenceList{}, doc, malformed)); err == nil || !strings.Contains(err.Error(), "resolve location registry") || strings.Contains(err.Error(), "registry evidence") {
|
||||
t.Fatalf("operation registry error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizerContractsRequiredRegistryAndWarningBounds(t *testing.T) {
|
||||
if _, err := New(Options{}, contracts.ReferenceSet{}, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), "at most one reference set") {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
normalizer := newNormalizer(t, registryReferences(t, registryLocations("The Mill")))
|
||||
spec := ModuleSpec()
|
||||
if spec.Key != Key || spec.Stage != pipeline.StageNormalize || spec.ExecutionClass != contracts.ExecutionClassDeterministic || spec.ArtifactKind != dnd.LocationOccurrenceListKind {
|
||||
t.Fatalf("ModuleSpec() = %#v", spec)
|
||||
}
|
||||
assertLocationRegistryReferenceSlots(t, "ModuleSpec", spec.ReferenceSlots)
|
||||
assertLocationRegistryReferenceSlots(t, "Normalizer", normalizer.ReferenceSlots())
|
||||
registry := pipeline.NewNormalizerRegistry()
|
||||
if err := Register(registry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
registeredSpec, ok := registry.Spec(Key)
|
||||
if !ok {
|
||||
t.Fatalf("registry missing %q", Key)
|
||||
}
|
||||
assertLocationRegistryReferenceSlots(t, "registered ModuleSpec", registeredSpec.ReferenceSlots)
|
||||
if _, err := DecodeOptions(map[string]any{"unexpected": true}); err == nil {
|
||||
t.Fatal("DecodeOptions() accepted unknown options")
|
||||
}
|
||||
if metadata := normalizer.ManifestMetadata(); metadata["normalization_policy"] != normalizationPolicy || metadata["location_registry_digest"] == "" || metadata["location_count"] != 1 {
|
||||
t.Fatalf("metadata = %#v", metadata)
|
||||
}
|
||||
if fingerprints := normalizer.CheckpointFingerprints(); len(fingerprints) != 2 || fingerprints[1].Name != "location_registry" || fingerprints[1].Value == "" {
|
||||
t.Fatalf("fingerprints = %#v", fingerprints)
|
||||
}
|
||||
if _, err := newNormalizer(t).Normalize(context.Background(), normalizeRequest(dnd.LocationOccurrenceList{}, nil, contracts.ReferenceSet{})); err == nil || !strings.Contains(err.Error(), "required") {
|
||||
t.Fatalf("unbound registry error = %v", err)
|
||||
}
|
||||
|
||||
count := diagnostics.MaxWarnings + 5
|
||||
doc := &source.SourceDocument{ID: "session", Units: make([]source.SourceUnit, count)}
|
||||
input := dnd.LocationOccurrenceList{Occurrences: make([]dnd.LocationOccurrence, count)}
|
||||
location := registryLocations("The Mill").Locations[0]
|
||||
for index := range doc.Units {
|
||||
doc.Units[index].ID = index + 1
|
||||
input.Occurrences[index] = dnd.LocationOccurrence{LocationID: location.ID, Name: "not canonical", Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: count - index, EndUnitID: count - index}}}
|
||||
}
|
||||
bounded, err := newNormalizer(t, registryReferences(t, dnd.LocationList{Locations: []dnd.Location{location}})).Normalize(context.Background(), normalizeRequest(input, doc, contracts.ReferenceSet{}))
|
||||
if err != nil || len(bounded.Warnings) != diagnostics.MaxWarnings || bounded.Warnings[len(bounded.Warnings)-1].ReasonCode != ReasonCodeWarningsOmitted {
|
||||
t.Fatalf("bounded warnings = %#v, %v", bounded.Warnings, err)
|
||||
}
|
||||
}
|
||||
|
||||
func assertLocationRegistryReferenceSlots(t *testing.T, owner string, slots []contracts.ReferenceSlot) {
|
||||
t.Helper()
|
||||
if len(slots) != 1 {
|
||||
t.Fatalf("%s reference slots = %#v, want one location registry", owner, slots)
|
||||
}
|
||||
got := slots[0]
|
||||
description := got.Description
|
||||
got.Description = ""
|
||||
want := contracts.ReferenceSlot{
|
||||
Name: LocationRegistryReferenceSlot,
|
||||
Required: true,
|
||||
AcceptedMediaTypes: []string{"application/json"},
|
||||
AcceptedArtifactKinds: []contracts.ArtifactKind{dnd.LocationListKind},
|
||||
MaxBytes: LocationRegistryMaxBytes,
|
||||
}
|
||||
if !reflect.DeepEqual(got, want) || strings.TrimSpace(description) == "" {
|
||||
t.Fatalf("%s location registry slot = %#v, want contract %#v with a nonempty description", owner, slots[0], want)
|
||||
}
|
||||
}
|
||||
|
||||
func newNormalizer(t *testing.T, references ...contracts.ReferenceSet) *Normalizer {
|
||||
t.Helper()
|
||||
normalizer, err := New(Options{}, references...)
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
return normalizer
|
||||
}
|
||||
|
||||
func normalizeRequest(value dnd.LocationOccurrenceList, doc *source.SourceDocument, references contracts.ReferenceSet) contracts.TypedNormalizeRequest[dnd.LocationOccurrenceList] {
|
||||
return contracts.TypedNormalizeRequest[dnd.LocationOccurrenceList]{Source: doc, MergeOutput: contracts.MergeArtifact[dnd.LocationOccurrenceList]{Value: value}, References: references}
|
||||
}
|
||||
|
||||
func registryLocations(names ...string) dnd.LocationList {
|
||||
locations := make([]dnd.Location, len(names))
|
||||
for index, name := range names {
|
||||
refs := []source.SourceRef{{SourceID: "registry", StartUnitID: index + 1, EndUnitID: index + 1}}
|
||||
locations[index] = dnd.Location{ID: identity.DeriveID(name, refs), Name: name, SourceRefs: refs}
|
||||
}
|
||||
return dnd.LocationList{Locations: locations}
|
||||
}
|
||||
|
||||
func registryReferences(t *testing.T, locations dnd.LocationList) contracts.ReferenceSet {
|
||||
t.Helper()
|
||||
content, err := locationcodec.New().Encode(locations)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{LocationRegistryReferenceSlot: {
|
||||
Items: []contracts.ReferenceItem{{SlotName: LocationRegistryReferenceSlot, MediaType: locationcodec.MediaType, Content: content}},
|
||||
}}}
|
||||
}
|
||||
|
||||
func cloneList(input dnd.LocationOccurrenceList) dnd.LocationOccurrenceList {
|
||||
output := dnd.LocationOccurrenceList{Occurrences: make([]dnd.LocationOccurrence, len(input.Occurrences))}
|
||||
for index, occurrence := range input.Occurrences {
|
||||
output.Occurrences[index] = occurrence
|
||||
output.Occurrences[index].SourceRefs = append([]source.SourceRef(nil), occurrence.SourceRefs...)
|
||||
}
|
||||
if input.Occurrences == nil {
|
||||
output.Occurrences = nil
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
func hasWarning(warnings []contracts.Warning, code string) bool {
|
||||
for _, warning := range warnings {
|
||||
if warning.ReasonCode == code {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
6
internal/modules/dnd/normalize/locations/assets.go
Normal file
6
internal/modules/dnd/normalize/locations/assets.go
Normal file
@@ -0,0 +1,6 @@
|
||||
package locations
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/prompts/*.yaml assets/prompts/*.md
|
||||
var embeddedAssets embed.FS
|
||||
@@ -0,0 +1,2 @@
|
||||
Location candidates:
|
||||
{{ input "candidates" }}
|
||||
@@ -0,0 +1,30 @@
|
||||
id: dnd.locations.normalize
|
||||
version: "v1"
|
||||
default_profile: dnd-extraction
|
||||
inputs:
|
||||
- name: candidates
|
||||
required: true
|
||||
content_type: application/json
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./sharedassets/common-dnd-system.md
|
||||
- role: user
|
||||
content_file: ./task.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-entity-reconciliation.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./candidates.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-transcript.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: dnd_entity_reconcile_llm.v1.json
|
||||
repair_attempts: 0
|
||||
@@ -0,0 +1,8 @@
|
||||
Review location candidates and cited transcript context. Group candidates only
|
||||
when the evidence clearly identifies one physical place.
|
||||
|
||||
Do not group candidates solely because their names match, their evidence is
|
||||
nearby, one place is nested inside another, or their labels are generic. Keep
|
||||
parent and child places, similarly named places, and uncertain aliases
|
||||
separate. For an accepted group, select the supplied candidate with the
|
||||
clearest established display name as canonical.
|
||||
343
internal/modules/dnd/normalize/locations/normalizer.go
Normal file
343
internal/modules/dnd/normalize/locations/normalizer.go
Normal file
@@ -0,0 +1,343 @@
|
||||
// Package locations normalizes merged D&D location candidates conservatively.
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
const (
|
||||
Key = "dnd/locations"
|
||||
PromptID = "dnd.locations.normalize"
|
||||
normalizationPolicy = "dnd.locations.normalize.v1"
|
||||
semanticContextPolicy = "dnd.entity_reconcile.context.v1"
|
||||
semanticContextRadius = 2
|
||||
NormalizationPolicy = normalizationPolicy
|
||||
|
||||
ReasonCodeLocationFieldsNormalized = "location_fields_normalized"
|
||||
ReasonCodeLocationIDRecomputed = "location_id_recomputed"
|
||||
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
|
||||
ReasonCodeDuplicateLocationCollapsed = "duplicate_location_collapsed"
|
||||
ReasonCodeLocationSemanticProposalInvalid = "location_semantic_proposal_invalid"
|
||||
ReasonCodeLocationSemanticReconciliationExhausted = "location_semantic_reconciliation_exhausted"
|
||||
ReasonCodeLocationNormalizationWarningsOmitted = "location_normalization_warnings_omitted"
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"merged"}
|
||||
var providedCapabilities = []string{"normalized"}
|
||||
|
||||
var _ contracts.Normalizer[dnd.LocationList] = (*Normalizer)(nil)
|
||||
var _ contracts.ManifestMetadataProvider = (*Normalizer)(nil)
|
||||
var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
|
||||
|
||||
type Options struct{}
|
||||
|
||||
type Normalizer struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
promptSHA string
|
||||
responseSchemaSHA string
|
||||
}
|
||||
|
||||
func New(llmClient contracts.StructuredLLMClient, _ Options) (*Normalizer, error) {
|
||||
if llmClient == nil {
|
||||
return nil, normalizerErrorf("LLM client must not be nil")
|
||||
}
|
||||
promptSHA, err := promptAssetMetadata()
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("load prompt metadata: %w", err)
|
||||
}
|
||||
responseSchema, err := entityreconcile.LoadResponseSchema()
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("load response schema: %w", err)
|
||||
}
|
||||
return &Normalizer{llm: llmClient, promptSHA: promptSHA, responseSchemaSHA: responseSchema.SHA256}, nil
|
||||
}
|
||||
|
||||
func (n *Normalizer) Key() string { return Key }
|
||||
func (n *Normalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
|
||||
func (n *Normalizer) ManifestMetadata() map[string]any {
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
return map[string]any{
|
||||
"prompt_id": PromptID, "prompt_version": entityreconcile.SchemaVersion, "prompt_sha256": n.promptSHA,
|
||||
"response_schema_key": string(entityreconcile.ResponseSchemaKey), "response_schema_id": entityreconcile.ResponseSchemaID,
|
||||
"response_schema_name": entityreconcile.ResponseSchemaName, "response_schema_version": entityreconcile.SchemaVersion,
|
||||
"response_schema_sha256": n.responseSchemaSHA, "identity_policy": identity.Policy,
|
||||
"normalization_policy": normalizationPolicy, "semantic_context_policy": semanticContextPolicy, "semantic_context_radius": semanticContextRadius,
|
||||
}
|
||||
}
|
||||
|
||||
func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
return []pipeline.CheckpointFingerprint{
|
||||
{Name: "prompt", Value: n.promptSHA}, {Name: "response_schema", Value: n.responseSchemaSHA},
|
||||
{Name: "identity_policy", Value: identity.Policy}, {Name: "normalization_policy", Value: normalizationPolicy},
|
||||
{Name: "semantic_context_policy", Value: fmt.Sprintf("%s:%d", semanticContextPolicy, semanticContextRadius)},
|
||||
}
|
||||
}
|
||||
|
||||
func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalizeRequest[dnd.LocationList]) (contracts.TypedNormalizeResult[dnd.LocationList], error) {
|
||||
if n == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{}, normalizerErrorf("normalizer must not be nil")
|
||||
}
|
||||
if n.llm == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{}, normalizerErrorf("LLM client must not be nil")
|
||||
}
|
||||
if ctx == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{}, normalizerErrorf("context must not be nil")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{}, normalizerErrorf("context error before normalize: %w", err)
|
||||
}
|
||||
|
||||
order := shared.NewSourceRefOrder(req.Source)
|
||||
records, warnings := preprocessRecords(req.MergeOutput.Value, order)
|
||||
deterministic := recordList(records)
|
||||
materials, ready, err := entityreconcile.BuildContext(req.Source, reconciliationCandidates(records), semanticContextRadius)
|
||||
if err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{}, normalizerErrorf("build semantic context: %w", err)
|
||||
}
|
||||
if !ready {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
||||
}
|
||||
|
||||
var response entityreconcile.ProposalResponse
|
||||
if _, err := n.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: entityreconcile.SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID,
|
||||
Inputs: contracts.LLMInputSet{"candidates": materials.Candidates, "transcript": materials.Transcript},
|
||||
}, &response); err != nil {
|
||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
return n.invalidStructuredResult(deterministic, warnings), nil
|
||||
}
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{}, normalizerErrorf("complete structured output: %w", err)
|
||||
}
|
||||
assessment := materials.Assess(response)
|
||||
applied, semanticWarnings := applySafeGroups(records, reconciliationGroups(assessment, materials.CandidateKeys()), order)
|
||||
warnings = append(warnings, semanticWarnings...)
|
||||
if assessment.DiscardedGroups() == 0 {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{Value: recordList(applied), Warnings: limitWarnings(warnings)}, nil
|
||||
}
|
||||
return retryResult(recordList(applied), warnings, assessment), nil
|
||||
}
|
||||
|
||||
func (n *Normalizer) invalidStructuredResult(value dnd.LocationList, warnings []contracts.Warning) contracts.TypedNormalizeResult[dnd.LocationList] {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{Value: value, Warnings: limitWarningsForRetry(warnings), Retry: &contracts.NormalizeRetry{
|
||||
ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: "semantic proposal requires retry: invalid structured output",
|
||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(-1)},
|
||||
}}
|
||||
}
|
||||
|
||||
func retryResult(value dnd.LocationList, warnings []contracts.Warning, assessment entityreconcile.Assessment) contracts.TypedNormalizeResult[dnd.LocationList] {
|
||||
return contracts.TypedNormalizeResult[dnd.LocationList]{Value: value, Warnings: limitWarningsForRetry(warnings), Retry: &contracts.NormalizeRetry{
|
||||
ReasonCode: ReasonCodeLocationSemanticProposalInvalid, Message: diagnostics.Aggregate("semantic proposal requires retry", reconciliationIssues(assessment)),
|
||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(assessment.DiscardedGroups())},
|
||||
}}
|
||||
}
|
||||
|
||||
func semanticFallbackWarning(discarded int) contracts.Warning {
|
||||
message := "semantic proposal could not be applied"
|
||||
if discarded >= 0 {
|
||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discarded)
|
||||
}
|
||||
return contracts.Warning{Scope: "locations", ReasonCode: ReasonCodeLocationSemanticReconciliationExhausted, Message: message}
|
||||
}
|
||||
|
||||
func limitWarnings(warnings []contracts.Warning) []contracts.Warning {
|
||||
return diagnostics.LimitWarnings(warnings, "locations", ReasonCodeLocationNormalizationWarningsOmitted)
|
||||
}
|
||||
|
||||
func limitWarningsForRetry(warnings []contracts.Warning) []contracts.Warning {
|
||||
if warnings == nil {
|
||||
return nil
|
||||
}
|
||||
if len(warnings) < diagnostics.MaxWarnings {
|
||||
return append([]contracts.Warning(nil), warnings...)
|
||||
}
|
||||
displayed := diagnostics.MaxWarnings - 2
|
||||
bounded := append([]contracts.Warning(nil), warnings[:displayed]...)
|
||||
return append(bounded, contracts.Warning{Scope: "locations", ReasonCode: ReasonCodeLocationNormalizationWarningsOmitted, Message: fmt.Sprintf("%d additional warning(s) omitted", len(warnings)-displayed)})
|
||||
}
|
||||
|
||||
type normalizedRecord struct {
|
||||
location dnd.Location
|
||||
inputIndexes []int
|
||||
earliest int
|
||||
}
|
||||
|
||||
func preprocessRecords(input dnd.LocationList, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
||||
if input.Locations == nil {
|
||||
return nil, nil
|
||||
}
|
||||
records := make([]normalizedRecord, len(input.Locations))
|
||||
warnings := make([]contracts.Warning, 0)
|
||||
for index, inputLocation := range input.Locations {
|
||||
location, fieldsChanged, refsChanged := normalizeRecord(inputLocation, order)
|
||||
records[index] = normalizedRecord{location: location, inputIndexes: []int{index}, earliest: index}
|
||||
if fieldsChanged {
|
||||
warnings = append(warnings, contracts.Warning{Scope: locationScope(index), ReasonCode: ReasonCodeLocationFieldsNormalized, Message: fmt.Sprintf("input index %d: location name normalized for %s", index, diagnostics.Quote(inputLocation.Name))})
|
||||
}
|
||||
if refsChanged {
|
||||
warnings = append(warnings, contracts.Warning{Scope: locationScope(index), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized (original count %d, final count %d)", index, len(inputLocation.SourceRefs), len(location.SourceRefs))})
|
||||
}
|
||||
if inputLocation.ID != location.ID {
|
||||
warnings = append(warnings, contracts.Warning{Scope: locationScope(index), ReasonCode: ReasonCodeLocationIDRecomputed, Message: fmt.Sprintf("input index %d: location ID recomputed from %s", index, diagnostics.Quote(location.Name))})
|
||||
}
|
||||
}
|
||||
groups := exactDuplicateGroups(records)
|
||||
output := make([]normalizedRecord, 0, len(groups))
|
||||
for _, members := range groups {
|
||||
retained := cloneRecord(records[members[0]])
|
||||
for _, member := range members[1:] {
|
||||
retained.inputIndexes = append(retained.inputIndexes, records[member].inputIndexes...)
|
||||
}
|
||||
retained.inputIndexes = sortedUniqueIndexes(retained.inputIndexes)
|
||||
output = append(output, retained)
|
||||
if len(members) > 1 {
|
||||
warnings = append(warnings, duplicateWarning(retained.earliest, memberInputIndexes(records, members[1:])))
|
||||
}
|
||||
}
|
||||
return output, warnings
|
||||
}
|
||||
|
||||
func normalizeRecord(input dnd.Location, order shared.SourceRefOrder) (dnd.Location, bool, bool) {
|
||||
output := cloneLocation(input)
|
||||
output.Name = identity.NormalizeDisplay(input.Name)
|
||||
output.SourceRefs = order.Canonicalize(input.SourceRefs)
|
||||
output.ID = identity.DeriveID(output.Name, output.SourceRefs)
|
||||
return output, input.Name != output.Name, !reflect.DeepEqual(input.SourceRefs, output.SourceRefs)
|
||||
}
|
||||
|
||||
func exactDuplicateGroups(records []normalizedRecord) [][]int {
|
||||
groups := make([][]int, 0, len(records))
|
||||
for index, record := range records {
|
||||
key := identity.ComparisonKey(record.location.Name)
|
||||
found := false
|
||||
for groupIndex, members := range groups {
|
||||
first := records[members[0]]
|
||||
if identity.ComparisonKey(first.location.Name) == key && reflect.DeepEqual(first.location.SourceRefs, record.location.SourceRefs) {
|
||||
groups[groupIndex] = append(groups[groupIndex], index)
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
groups = append(groups, []int{index})
|
||||
}
|
||||
}
|
||||
return groups
|
||||
}
|
||||
|
||||
func cloneLocation(input dnd.Location) dnd.Location {
|
||||
input.SourceRefs = cloneSourceRefs(input.SourceRefs)
|
||||
return input
|
||||
}
|
||||
func cloneRecord(input normalizedRecord) normalizedRecord {
|
||||
input.location = cloneLocation(input.location)
|
||||
input.inputIndexes = append([]int(nil), input.inputIndexes...)
|
||||
return input
|
||||
}
|
||||
func cloneSourceRefs(input []source.SourceRef) []source.SourceRef {
|
||||
if input == nil {
|
||||
return nil
|
||||
}
|
||||
return append([]source.SourceRef(nil), input...)
|
||||
}
|
||||
|
||||
func sortedUniqueIndexes(indexes []int) []int {
|
||||
if len(indexes) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := append([]int(nil), indexes...)
|
||||
sort.Ints(out)
|
||||
write := 1
|
||||
for _, index := range out[1:] {
|
||||
if index != out[write-1] {
|
||||
out[write] = index
|
||||
write++
|
||||
}
|
||||
}
|
||||
return out[:write]
|
||||
}
|
||||
func memberInputIndexes(records []normalizedRecord, members []int) []int {
|
||||
indexes := make([]int, 0, len(members))
|
||||
for _, member := range members {
|
||||
indexes = append(indexes, records[member].inputIndexes...)
|
||||
}
|
||||
return sortedUniqueIndexes(indexes)
|
||||
}
|
||||
func recordValues(records []normalizedRecord) []dnd.Location {
|
||||
if records == nil {
|
||||
return nil
|
||||
}
|
||||
values := make([]dnd.Location, len(records))
|
||||
for index, record := range records {
|
||||
values[index] = cloneLocation(record.location)
|
||||
}
|
||||
return values
|
||||
}
|
||||
func recordList(records []normalizedRecord) dnd.LocationList {
|
||||
if records == nil {
|
||||
return dnd.LocationList{}
|
||||
}
|
||||
return dnd.LocationList{Locations: recordValues(records)}
|
||||
}
|
||||
|
||||
func duplicateWarning(retainedIndex int, removed []int) contracts.Warning {
|
||||
const maxDisplayedIndices = 20
|
||||
displayed := removed
|
||||
if len(displayed) > maxDisplayedIndices {
|
||||
displayed = displayed[:maxDisplayedIndices]
|
||||
}
|
||||
indices := make([]string, len(displayed))
|
||||
for index, removedIndex := range displayed {
|
||||
indices[index] = strconv.Itoa(removedIndex)
|
||||
}
|
||||
message := fmt.Sprintf("retained input index %d; removed input indices [%s]", retainedIndex, strings.Join(indices, ", "))
|
||||
if omitted := len(removed) - len(displayed); omitted > 0 {
|
||||
message += fmt.Sprintf("; %d additional removed input indices omitted", omitted)
|
||||
}
|
||||
return contracts.Warning{Scope: locationScope(retainedIndex), ReasonCode: ReasonCodeDuplicateLocationCollapsed, Message: message}
|
||||
}
|
||||
func locationScope(index int) string { return fmt.Sprintf("locations[%d]", index) }
|
||||
|
||||
func ModuleSpec() pipeline.ModuleSpec {
|
||||
return pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: append([]string(nil), requiredCapabilities...), Provides: append([]string(nil), providedCapabilities...), ArtifactKind: dnd.LocationListKind}
|
||||
}
|
||||
func Register(registry *pipeline.NormalizerRegistry) error {
|
||||
return pipeline.RegisterNormalizerBuilder(registry, ModuleSpec(), validateOptions, func(request pipeline.BuildRequest) (contracts.Normalizer[dnd.LocationList], error) {
|
||||
options, err := DecodeOptions(request.Options)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(request.Dependencies.LLM, options)
|
||||
})
|
||||
}
|
||||
func validateOptions(options map[string]any) error { _, err := DecodeOptions(options); return err }
|
||||
func DecodeOptions(options map[string]any) (Options, error) {
|
||||
if err := pipeline.RejectUnknownOptions(options); err != nil {
|
||||
return Options{}, normalizerErrorf("%w", err)
|
||||
}
|
||||
return Options{}, nil
|
||||
}
|
||||
func normalizerErrorf(format string, args ...any) error {
|
||||
return fmt.Errorf("dnd locations normalizer: "+format, args...)
|
||||
}
|
||||
136
internal/modules/dnd/normalize/locations/normalizer_test.go
Normal file
136
internal/modules/dnd/normalize/locations/normalizer_test.go
Normal file
@@ -0,0 +1,136 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
func TestModuleContractAndMetadata(t *testing.T) {
|
||||
want := pipeline.ModuleSpec{Key: Key, Stage: pipeline.StageNormalize, ExecutionClass: contracts.ExecutionClassLLMBacked, Requires: []string{"merged"}, Provides: []string{"normalized"}, ArtifactKind: dnd.LocationListKind}
|
||||
if got := ModuleSpec(); !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||
}
|
||||
registry := pipeline.NewNormalizerRegistry()
|
||||
if err := Register(registry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, err := New(nil, Options{}); err == nil {
|
||||
t.Fatal("New() accepted nil client")
|
||||
}
|
||||
normalizer := newNormalizer(t, &recordingLocationNormalizerClient{})
|
||||
metadata := normalizer.ManifestMetadata()
|
||||
if metadata["identity_policy"] != identity.Policy || metadata["response_schema_id"] != entityreconcile.ResponseSchemaID || metadata["normalization_policy"] != normalizationPolicy || metadata["semantic_context_radius"] != semanticContextRadius {
|
||||
t.Fatalf("metadata = %#v", metadata)
|
||||
}
|
||||
if got := normalizer.CheckpointFingerprints(); len(got) != 5 || got[2].Value != identity.Policy || got[3].Value != normalizationPolicy || got[4].Value != semanticContextPolicy+":2" {
|
||||
t.Fatalf("fingerprints = %#v", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizePreparesOnlyExactDuplicatesAndRetainsSameNameAndNestedPlaces(t *testing.T) {
|
||||
input := dnd.LocationList{Locations: []dnd.Location{
|
||||
{Name: " The Tavern ", SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "the tavern", SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}},
|
||||
{Name: "The Tavern Cellar", SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}},
|
||||
}}
|
||||
before := dnd.LocationList{Locations: append([]dnd.Location(nil), input.Locations...)}
|
||||
result, err := newNormalizer(t, &recordingLocationNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
if err != nil || len(result.Value.Locations) != 3 {
|
||||
t.Fatalf("Normalize() = %#v, %v; want one exact duplicate removed", result, err)
|
||||
}
|
||||
if !reflect.DeepEqual(input, before) {
|
||||
t.Fatalf("Normalize() mutated input: %#v", input)
|
||||
}
|
||||
if got := []string{result.Value.Locations[0].Name, result.Value.Locations[1].Name, result.Value.Locations[2].Name}; !reflect.DeepEqual(got, []string{"The Tavern", "The Tavern", "The Tavern Cellar"}) {
|
||||
t.Fatalf("locations = %#v, want same names and nested place retained", got)
|
||||
}
|
||||
if result.Value.Locations[0].ID == result.Value.Locations[1].ID || !hasWarning(result.Warnings, ReasonCodeDuplicateLocationCollapsed) {
|
||||
t.Fatalf("result = %#v, want evidence-anchored IDs and exact duplicate warning", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeAppliesSafeAliasGroupAndUsesOpaqueInputs(t *testing.T) {
|
||||
client := &recordingLocationNormalizerClient{response: `{"duplicate_groups":[{"members":["candidate-000001","candidate-000002"],"canonical":"candidate-000002"}]}`}
|
||||
doc := semanticDocument()
|
||||
input := dnd.LocationList{Locations: []dnd.Location{
|
||||
{Name: "Old Mill", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "the Greencloak's refuge", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
{Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
|
||||
}}
|
||||
result, err := newNormalizer(t, client).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || result.Retry != nil || len(result.Value.Locations) != 2 {
|
||||
t.Fatalf("Normalize() = %#v, %v", result, err)
|
||||
}
|
||||
merged := result.Value.Locations[0]
|
||||
if merged.Name != "the Greencloak's refuge" || merged.ID != identity.DeriveID(merged.Name, merged.SourceRefs) || len(merged.SourceRefs) != 2 || !hasWarning(result.Warnings, ReasonCodeDuplicateLocationCollapsed) {
|
||||
t.Fatalf("merged location = %#v, warnings = %#v", merged, result.Warnings)
|
||||
}
|
||||
encoded := string(client.requests[0].Inputs["candidates"].Content) + string(client.requests[0].Inputs["transcript"].Content)
|
||||
if strings.Contains(encoded, doc.ID) || !strings.Contains(encoded, "candidate-000001") || strings.Contains(encoded, merged.ID) {
|
||||
t.Fatalf("private inputs = %s", encoded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRejectsUnsafeAndOverlappingGroupsWithoutLosingCandidates(t *testing.T) {
|
||||
doc := semanticDocument()
|
||||
input := dnd.LocationList{Locations: []dnd.Location{
|
||||
{Name: "Old Mill", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "Mill", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
{Name: "Tavern", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
|
||||
}}
|
||||
client := &recordingLocationNormalizerClient{response: `{"duplicate_groups":[{"members":["candidate-000001","candidate-000002"],"canonical":"candidate-000001"},{"members":["candidate-000002","candidate-000003"],"canonical":"candidate-000003"}]}`}
|
||||
result, err := newNormalizer(t, client).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || result.Retry == nil || len(result.Value.Locations) != 3 || !strings.Contains(result.Retry.Message, "overlapping_member") {
|
||||
t.Fatalf("Normalize() = %#v, %v; want safe retry fallback", result, err)
|
||||
}
|
||||
if len(result.Retry.FallbackWarnings) != 1 || !strings.Contains(result.Retry.FallbackWarnings[0].Message, "2 proposal group") {
|
||||
t.Fatalf("fallback warnings = %#v", result.Retry.FallbackWarnings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeHandlesRetryFallbackAndErrors(t *testing.T) {
|
||||
doc := semanticDocument()
|
||||
input := dnd.LocationList{Locations: []dnd.Location{{Name: "Old Mill", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}, {Name: "Mill", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}}}}
|
||||
invalid, err := newNormalizer(t, &recordingLocationNormalizerClient{err: contracts.ErrInvalidStructuredOutput}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || invalid.Retry == nil || invalid.Retry.ReasonCode != ReasonCodeLocationSemanticProposalInvalid {
|
||||
t.Fatalf("invalid result = %#v, %v", invalid, err)
|
||||
}
|
||||
_, err = newNormalizer(t, &recordingLocationNormalizerClient{err: errors.New("provider unavailable")}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err == nil || !strings.Contains(err.Error(), "provider unavailable") {
|
||||
t.Fatalf("provider error = %v", err)
|
||||
}
|
||||
warnings := make([]contracts.Warning, 25)
|
||||
bounded := limitWarningsForRetry(warnings)
|
||||
if len(bounded) != 19 || bounded[len(bounded)-1].ReasonCode != ReasonCodeLocationNormalizationWarningsOmitted {
|
||||
t.Fatalf("retry warning limit = %#v", bounded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeOrdersEvidenceAndIsIdempotent(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "ordered", Units: []source.SourceUnit{{ID: 30}, {ID: 10}}}
|
||||
input := dnd.LocationList{Locations: []dnd.Location{{Name: "Old Mill", SourceRefs: []source.SourceRef{
|
||||
{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10},
|
||||
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30},
|
||||
}}}}
|
||||
normalizer := newNormalizer(t, &recordingLocationNormalizerClient{})
|
||||
first, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || !reflect.DeepEqual([]int{first.Value.Locations[0].SourceRefs[0].StartUnitID, first.Value.Locations[0].SourceRefs[1].StartUnitID}, []int{30, 10}) {
|
||||
t.Fatalf("first Normalize() = %#v, %v; want document-ordered evidence", first, err)
|
||||
}
|
||||
second, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(first.Value, doc))
|
||||
if err != nil || !reflect.DeepEqual(second.Value, first.Value) {
|
||||
t.Fatalf("second Normalize() = %#v, %v; want idempotent value %#v", second, err, first.Value)
|
||||
}
|
||||
}
|
||||
41
internal/modules/dnd/normalize/locations/prompt_assets.go
Normal file
41
internal/modules/dnd/normalize/locations/prompt_assets.go
Normal file
@@ -0,0 +1,41 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const promptAssetRoot = "assets/prompts"
|
||||
|
||||
var promptAssetManifest = shared.PromptAssetManifest{
|
||||
ModuleDir: PromptID,
|
||||
ModuleFiles: []promptfs.ModulePromptFile{
|
||||
{Name: "dnd.locations.normalize.yaml", Path: "assets/prompts/dnd.locations.normalize.yaml"},
|
||||
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||
{Name: "candidates.md", Path: "assets/prompts/candidates.md"},
|
||||
},
|
||||
SharedFiles: []string{"common-dnd-system.md", "common-dnd-entity-reconciliation.md", "common-dnd-transcript.md"},
|
||||
}
|
||||
|
||||
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
||||
promptFS, err := promptAssetManifest.PromptFS(embeddedAssets)
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare location normalization prompt assets: %w", err)
|
||||
}
|
||||
return registry.RegisterPromptFS(promptFS, promptAssetRoot)
|
||||
}
|
||||
|
||||
func promptAssetMetadata() (string, error) {
|
||||
promptAssetHashOnce.Do(func() { promptAssetHash, promptAssetHashErr = promptAssetManifest.Hash(embeddedAssets) })
|
||||
return promptAssetHash, promptAssetHashErr
|
||||
}
|
||||
|
||||
var (
|
||||
promptAssetHashOnce sync.Once
|
||||
promptAssetHash string
|
||||
promptAssetHashErr error
|
||||
)
|
||||
@@ -0,0 +1,46 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestRegisterPromptAssetsPreparesLocationNormalizationPrompt(t *testing.T) {
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := entityreconcile.RegisterSchemaAssets(registry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := RegisterPromptAssets(registry); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
options, err := registry.PromptKitOptions()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
options = append(options, promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{ID: "location-normalize-test", Endpoint: "http://127.0.0.1:1/v1", Model: "test"})))
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{Timeout: time.Second}, options...)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{PromptID: PromptID, PromptVersion: entityreconcile.SchemaVersion, ProfileID: "location-normalize-test", Inputs: map[string]promptkit.ArtifactRef{"candidates": promptkit.Inline(`{"candidates":[{"key":"candidate-000001","name":"The Tavern","source_refs":[]}]}`), "transcript": promptkit.Inline(`{"windows":[{"units":[]}]}`)}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if prepared.OutputContract.SchemaPath != "dnd_entity_reconcile_llm.v1.json" || len(prepared.Messages) != 5 {
|
||||
t.Fatalf("prepared prompt = %#v", prepared)
|
||||
}
|
||||
for _, index := range []int{2, 4} {
|
||||
if prepared.Messages[index].CacheControl == nil {
|
||||
t.Fatalf("message %d cache = %#v", index, prepared.Messages[index].CacheControl)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(prepared.Messages[3].Content, "candidate-000001") || strings.Contains(prepared.Messages[3].Content, `"windows"`) {
|
||||
t.Fatalf("candidate message = %q", prepared.Messages[3].Content)
|
||||
}
|
||||
}
|
||||
112
internal/modules/dnd/normalize/locations/reconciliation.go
Normal file
112
internal/modules/dnd/normalize/locations/reconciliation.go
Normal file
@@ -0,0 +1,112 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/locations/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
type safeReconciliationGroup struct {
|
||||
members []int
|
||||
canonical int
|
||||
}
|
||||
|
||||
func reconciliationCandidates(records []normalizedRecord) []entityreconcile.Candidate {
|
||||
candidates := make([]entityreconcile.Candidate, len(records))
|
||||
for index, record := range records {
|
||||
candidates[index] = entityreconcile.Candidate{Name: record.location.Name, SourceRefs: cloneSourceRefs(record.location.SourceRefs)}
|
||||
}
|
||||
return candidates
|
||||
}
|
||||
|
||||
func reconciliationGroups(assessment entityreconcile.Assessment, candidateKeys []string) []safeReconciliationGroup {
|
||||
positions := make(map[string]int, len(candidateKeys))
|
||||
for index, key := range candidateKeys {
|
||||
positions[key] = index
|
||||
}
|
||||
safeGroups := assessment.SafeGroups()
|
||||
groups := make([]safeReconciliationGroup, 0, len(safeGroups))
|
||||
for _, group := range safeGroups {
|
||||
members := group.Members()
|
||||
memberPositions := make([]int, len(members))
|
||||
valid := true
|
||||
for index, key := range members {
|
||||
position, ok := positions[key]
|
||||
if !ok {
|
||||
valid = false
|
||||
break
|
||||
}
|
||||
memberPositions[index] = position
|
||||
}
|
||||
canonical, ok := positions[group.Canonical()]
|
||||
if valid && ok {
|
||||
groups = append(groups, safeReconciliationGroup{members: memberPositions, canonical: canonical})
|
||||
}
|
||||
}
|
||||
return groups
|
||||
}
|
||||
|
||||
func reconciliationIssues(assessment entityreconcile.Assessment) []string {
|
||||
issues := assessment.Issues()
|
||||
details := make([]string, len(issues))
|
||||
for index, issue := range issues {
|
||||
details[index] = fmt.Sprintf("group %d: %s", issue.GroupIndex, issue.Category)
|
||||
}
|
||||
return details
|
||||
}
|
||||
|
||||
func applySafeGroups(records []normalizedRecord, groups []safeReconciliationGroup, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
||||
byMember := make(map[int]safeReconciliationGroup, len(groups)*2)
|
||||
for _, group := range groups {
|
||||
for _, member := range group.members {
|
||||
byMember[member] = group
|
||||
}
|
||||
}
|
||||
output := make([]normalizedRecord, 0, len(records)-len(groups))
|
||||
warnings := make([]contracts.Warning, 0, len(groups))
|
||||
for index, record := range records {
|
||||
group, grouped := byMember[index]
|
||||
if !grouped {
|
||||
output = append(output, cloneRecord(record))
|
||||
continue
|
||||
}
|
||||
if group.members[0] != index {
|
||||
continue
|
||||
}
|
||||
consolidated := consolidateSemanticGroup(records, group, order)
|
||||
output = append(output, consolidated)
|
||||
warnings = append(warnings, semanticDuplicateWarning(consolidated, records[group.canonical]))
|
||||
}
|
||||
return output, warnings
|
||||
}
|
||||
|
||||
func consolidateSemanticGroup(records []normalizedRecord, group safeReconciliationGroup, order shared.SourceRefOrder) normalizedRecord {
|
||||
output := cloneRecord(records[group.members[0]])
|
||||
output.location.Name = records[group.canonical].location.Name
|
||||
for _, member := range group.members[1:] {
|
||||
output.location.SourceRefs = append(output.location.SourceRefs, records[member].location.SourceRefs...)
|
||||
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
|
||||
if records[member].earliest < output.earliest {
|
||||
output.earliest = records[member].earliest
|
||||
}
|
||||
}
|
||||
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
|
||||
output.location.SourceRefs = order.Canonicalize(output.location.SourceRefs)
|
||||
output.location.ID = identity.DeriveID(output.location.Name, output.location.SourceRefs)
|
||||
return output
|
||||
}
|
||||
|
||||
func semanticDuplicateWarning(record normalizedRecord, canonical normalizedRecord) contracts.Warning {
|
||||
details := make([]string, 0, len(record.inputIndexes)+1)
|
||||
for _, inputIndex := range record.inputIndexes {
|
||||
details = append(details, fmt.Sprintf("input index %d", inputIndex))
|
||||
}
|
||||
if canonical.earliest != record.earliest {
|
||||
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
|
||||
}
|
||||
return contracts.Warning{Scope: locationScope(record.earliest), ReasonCode: ReasonCodeDuplicateLocationCollapsed, Message: diagnostics.Aggregate("semantic duplicate consolidation", details)}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
package locations
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
)
|
||||
|
||||
type recordingLocationNormalizerClient struct {
|
||||
response string
|
||||
err error
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (c *recordingLocationNormalizerClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, output any) (contracts.StructuredCompletionResponse, error) {
|
||||
c.requests = append(c.requests, request)
|
||||
if c.err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, c.err
|
||||
}
|
||||
response := c.response
|
||||
if response == "" {
|
||||
response = `{"duplicate_groups":[]}`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(response), output); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: json.RawMessage(response)}, nil
|
||||
}
|
||||
|
||||
func newNormalizer(t *testing.T, client contracts.StructuredLLMClient) *Normalizer {
|
||||
t.Helper()
|
||||
normalizer, err := New(client, Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
return normalizer
|
||||
}
|
||||
func normalizeRequest(value dnd.LocationList) contracts.TypedNormalizeRequest[dnd.LocationList] {
|
||||
return contracts.TypedNormalizeRequest[dnd.LocationList]{MergeOutput: contracts.MergeArtifact[dnd.LocationList]{Value: value}}
|
||||
}
|
||||
func normalizeRequestWithSource(value dnd.LocationList, doc *source.SourceDocument) contracts.TypedNormalizeRequest[dnd.LocationList] {
|
||||
request := normalizeRequest(value)
|
||||
request.Source = doc
|
||||
return request
|
||||
}
|
||||
func semanticDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{ID: "location-session", Units: []source.SourceUnit{{ID: 10, Kind: "speech", Text: "The old mill is the Greencloak's refuge."}, {ID: 20, Kind: "speech", Text: "The mill stands on the northern road."}, {ID: 30, Kind: "speech", Text: "The tavern is beside the mill."}, {ID: 40, Kind: "speech", Text: "The mill's cellar is flooded."}}}
|
||||
}
|
||||
func hasWarning(warnings []contracts.Warning, reason string) bool {
|
||||
for _, warning := range warnings {
|
||||
if warning.ReasonCode == reason {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -2,5 +2,5 @@ package npcs
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/schemas/dnd_npcs_normalize_llm.v1.json assets/prompts/*.yaml assets/prompts/*.md
|
||||
//go:embed assets/prompts/*.yaml assets/prompts/*.md
|
||||
var embeddedAssets embed.FS
|
||||
|
||||
@@ -14,7 +14,7 @@ messages:
|
||||
- role: user
|
||||
content_file: ./task.md
|
||||
- role: user
|
||||
content_file: ./instructions.md
|
||||
content_file: ./sharedassets/common-dnd-entity-reconciliation.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
@@ -26,5 +26,5 @@ messages:
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: dnd_npcs_normalize_llm.v1.json
|
||||
schema_path: dnd_entity_reconcile_llm.v1.json
|
||||
repair_attempts: 0
|
||||
|
||||
@@ -1,10 +0,0 @@
|
||||
Return duplicate groups only when the transcript context clearly establishes a
|
||||
single individual. Prefer no group when identity is ambiguous.
|
||||
|
||||
Copy supplied display names into each group's members. Choose canonical_name
|
||||
from that same group's supplied members. Prefer a complete stable proper name
|
||||
over an abbreviation, but prefer an unadorned proper name over that name plus a
|
||||
contextual class, role, title, or relationship descriptor unless the descriptor
|
||||
is established as part of the name.
|
||||
|
||||
Do not invent names, source references, replacement records, or explanations.
|
||||
@@ -1,2 +1,13 @@
|
||||
Identify only supplied NPC display names that clearly refer to the same
|
||||
individual in the supplied transcript context.
|
||||
Review NPC candidates and their cited transcript context to identify aliases
|
||||
that refer to the same individual. Propose only groups supported by the
|
||||
transcript, and preserve distinct individuals even when their names are
|
||||
similar.
|
||||
|
||||
For every accepted group, choose as canonical only a supplied candidate from
|
||||
that evidence-supported duplicate group. Prefer a complete, stable proper name
|
||||
over an abbreviation. Prefer an unadorned proper name over that name plus a
|
||||
contextual class, role, title, or relationship descriptor unless the transcript
|
||||
establishes the descriptor as part of the person's name. A longer display name
|
||||
is not inherently more canonical; for example, do not prefer `Captain Aria`
|
||||
over `Aria` solely because it includes the contextual title `Captain`. Do not
|
||||
invent, edit, or combine display names.
|
||||
|
||||
@@ -1,214 +0,0 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
)
|
||||
|
||||
const semanticContextRadius = 2
|
||||
|
||||
type normalizeContextMaterials struct {
|
||||
Candidates contracts.LLMInputMaterial
|
||||
Transcript contracts.LLMInputMaterial
|
||||
candidatePositions []int
|
||||
}
|
||||
|
||||
type normalizeCandidateInput struct {
|
||||
NPCs []normalizeCandidate `json:"npcs"`
|
||||
}
|
||||
|
||||
type normalizeCandidate struct {
|
||||
Name string `json:"name"`
|
||||
SourceRefs []normalizeCandidateSourceRef `json:"source_refs"`
|
||||
}
|
||||
|
||||
type normalizeCandidateSourceRef struct {
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
}
|
||||
|
||||
type normalizeTranscriptInput struct {
|
||||
Windows []normalizeTranscriptWindow `json:"windows"`
|
||||
}
|
||||
|
||||
type normalizeTranscriptWindow struct {
|
||||
Units []normalizeTranscriptUnit `json:"units"`
|
||||
}
|
||||
|
||||
type normalizeTranscriptUnit struct {
|
||||
ID int `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Text string `json:"text"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Cited bool `json:"cited"`
|
||||
}
|
||||
|
||||
type normalizeProposalResponse struct {
|
||||
DuplicateGroups []normalizeProposalGroup `json:"duplicate_groups"`
|
||||
}
|
||||
|
||||
type normalizeProposalGroup struct {
|
||||
Members []string `json:"members"`
|
||||
CanonicalName string `json:"canonical_name"`
|
||||
}
|
||||
|
||||
type sourceInterval struct {
|
||||
start int
|
||||
end int
|
||||
}
|
||||
|
||||
func buildDefaultNormalizeContextMaterials(doc *source.SourceDocument, records []dnd.NPC) (normalizeContextMaterials, bool, error) {
|
||||
return buildNormalizeContextMaterials(doc, records, semanticContextRadius)
|
||||
}
|
||||
|
||||
// buildNormalizeContextMaterials prepares the owned prompt inputs for a
|
||||
// document-level normalization proposal. A false ready value means semantic
|
||||
// normalization has no comparison-distinct eligible candidates to consider.
|
||||
func buildNormalizeContextMaterials(doc *source.SourceDocument, records []dnd.NPC, radius int) (materials normalizeContextMaterials, ready bool, err error) {
|
||||
if doc == nil {
|
||||
return normalizeContextMaterials{}, false, nil
|
||||
}
|
||||
if radius < 0 {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: radius must not be negative")
|
||||
}
|
||||
|
||||
index := source.NewDocumentIndex(doc)
|
||||
candidates := make([]normalizeCandidate, 0, len(records))
|
||||
candidatePositions := make([]int, 0, len(records))
|
||||
intervals := make([]sourceInterval, 0)
|
||||
cited := make([]bool, len(doc.Units))
|
||||
seenKeys := make(map[string]struct{}, len(records))
|
||||
|
||||
for position, record := range records {
|
||||
key := identity.ComparisonKey(record.Name)
|
||||
if key == "" || len(record.SourceRefs) == 0 {
|
||||
continue
|
||||
}
|
||||
if _, exists := seenKeys[key]; exists {
|
||||
continue
|
||||
}
|
||||
|
||||
references, recordIntervals, valid := normalizeRecordReferences(index, record.SourceRefs)
|
||||
if !valid {
|
||||
continue
|
||||
}
|
||||
seenKeys[key] = struct{}{}
|
||||
candidates = append(candidates, normalizeCandidate{Name: record.Name, SourceRefs: references})
|
||||
candidatePositions = append(candidatePositions, position)
|
||||
for _, interval := range recordIntervals {
|
||||
for position := interval.start; position <= interval.end; position++ {
|
||||
cited[position] = true
|
||||
}
|
||||
intervals = append(intervals, sourceInterval{
|
||||
start: maxInt(0, interval.start-radius),
|
||||
end: minInt(len(doc.Units)-1, interval.end+radius),
|
||||
})
|
||||
}
|
||||
}
|
||||
if len(candidates) < 2 {
|
||||
return normalizeContextMaterials{}, false, nil
|
||||
}
|
||||
|
||||
windows, err := normalizeContextWindows(doc.Units, coalesceIntervals(intervals), cited)
|
||||
if err != nil {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: invalid source metadata")
|
||||
}
|
||||
candidateContent, err := json.Marshal(normalizeCandidateInput{NPCs: candidates})
|
||||
if err != nil {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: invalid candidate material")
|
||||
}
|
||||
transcriptContent, err := json.Marshal(normalizeTranscriptInput{Windows: windows})
|
||||
if err != nil {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: invalid transcript material")
|
||||
}
|
||||
return normalizeContextMaterials{
|
||||
Candidates: newNormalizeInputMaterial("candidates", candidateContent),
|
||||
Transcript: newNormalizeInputMaterial("transcript", transcriptContent),
|
||||
candidatePositions: candidatePositions,
|
||||
}, true, nil
|
||||
}
|
||||
|
||||
func normalizeRecordReferences(index source.DocumentIndex, refs []source.SourceRef) ([]normalizeCandidateSourceRef, []sourceInterval, bool) {
|
||||
references := make([]normalizeCandidateSourceRef, 0, len(refs))
|
||||
intervals := make([]sourceInterval, 0, len(refs))
|
||||
for _, ref := range refs {
|
||||
if err := index.ValidateRef(ref); err != nil {
|
||||
return nil, nil, false
|
||||
}
|
||||
start, _ := index.Position(ref.StartUnitID)
|
||||
end, _ := index.Position(ref.EndUnitID)
|
||||
references = append(references, normalizeCandidateSourceRef{StartUnitID: ref.StartUnitID, EndUnitID: ref.EndUnitID})
|
||||
intervals = append(intervals, sourceInterval{start: start, end: end})
|
||||
}
|
||||
return references, intervals, true
|
||||
}
|
||||
|
||||
func coalesceIntervals(intervals []sourceInterval) []sourceInterval {
|
||||
if len(intervals) == 0 {
|
||||
return nil
|
||||
}
|
||||
ordered := append([]sourceInterval(nil), intervals...)
|
||||
sort.Slice(ordered, func(i, j int) bool {
|
||||
if ordered[i].start != ordered[j].start {
|
||||
return ordered[i].start < ordered[j].start
|
||||
}
|
||||
return ordered[i].end < ordered[j].end
|
||||
})
|
||||
coalesced := make([]sourceInterval, 0, len(ordered))
|
||||
for _, interval := range ordered {
|
||||
if len(coalesced) == 0 || interval.start > coalesced[len(coalesced)-1].end+1 {
|
||||
coalesced = append(coalesced, interval)
|
||||
continue
|
||||
}
|
||||
if interval.end > coalesced[len(coalesced)-1].end {
|
||||
coalesced[len(coalesced)-1].end = interval.end
|
||||
}
|
||||
}
|
||||
return coalesced
|
||||
}
|
||||
|
||||
func normalizeContextWindows(units []source.SourceUnit, intervals []sourceInterval, cited []bool) ([]normalizeTranscriptWindow, error) {
|
||||
windows := make([]normalizeTranscriptWindow, 0, len(intervals))
|
||||
for _, interval := range intervals {
|
||||
window := normalizeTranscriptWindow{Units: make([]normalizeTranscriptUnit, 0, interval.end-interval.start+1)}
|
||||
for position := interval.start; position <= interval.end; position++ {
|
||||
unit := units[position]
|
||||
metadata, err := source.CloneMetadata(unit.Metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
window.Units = append(window.Units, normalizeTranscriptUnit{
|
||||
ID: unit.ID, Kind: unit.Kind, Text: unit.Text, Metadata: metadata, Cited: cited[position],
|
||||
})
|
||||
}
|
||||
windows = append(windows, window)
|
||||
}
|
||||
return windows, nil
|
||||
}
|
||||
|
||||
func newNormalizeInputMaterial(name string, content []byte) contracts.LLMInputMaterial {
|
||||
digest := sha256.Sum256(content)
|
||||
return contracts.NewLLMInputMaterial(name, "application/json", content, "sha256:"+hex.EncodeToString(digest[:]), "")
|
||||
}
|
||||
|
||||
func minInt(left, right int) int {
|
||||
if left < right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
|
||||
func maxInt(left, right int) int {
|
||||
if left > right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
@@ -1,132 +0,0 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
)
|
||||
|
||||
func TestBuildNormalizeContextMaterialsUsesDocumentOrderAndOwnedInputs(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
|
||||
{ID: 40, Kind: "narration", Text: "zero"},
|
||||
{ID: 10, Kind: "speech", Text: "one", Metadata: map[string]any{"speaker": map[string]any{"name": "Mira"}}},
|
||||
{ID: 70, Kind: "speech", Text: "two"},
|
||||
{ID: 20, Kind: "narration", Text: "three"},
|
||||
{ID: 90, Kind: "speech", Text: "four"},
|
||||
{ID: 30, Kind: "narration", Text: "five"},
|
||||
}}
|
||||
records := []dnd.NPC{
|
||||
{Name: "Mira Thorn", ID: "npc:sha256:internal", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 20}}},
|
||||
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 90, EndUnitID: 90}}},
|
||||
{Name: "Broken", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 10}}},
|
||||
}
|
||||
before := append([]dnd.NPC(nil), records...)
|
||||
materials, ready, err := buildNormalizeContextMaterials(doc, records, 1)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("buildNormalizeContextMaterials() = %#v, %t, %v; want ready materials", materials, ready, err)
|
||||
}
|
||||
if !reflect.DeepEqual(records, before) {
|
||||
t.Fatalf("records mutated to %#v", records)
|
||||
}
|
||||
for _, material := range []struct {
|
||||
name string
|
||||
data []byte
|
||||
}{
|
||||
{name: "candidates", data: materials.Candidates.Content},
|
||||
{name: "transcript", data: materials.Transcript.Content},
|
||||
} {
|
||||
if !json.Valid(material.data) || string(material.data) == "" {
|
||||
t.Fatalf("%s content = %q, want JSON", material.name, material.data)
|
||||
}
|
||||
digest := sha256.Sum256(material.data)
|
||||
wantDigest := "sha256:" + hex.EncodeToString(digest[:])
|
||||
got := materials.Candidates
|
||||
if material.name == "transcript" {
|
||||
got = materials.Transcript
|
||||
}
|
||||
if got.Name != material.name || got.MediaType != "application/json" || got.OriginURI != "" || got.Digest != wantDigest {
|
||||
t.Fatalf("%s material = %#v, want owned JSON material", material.name, got)
|
||||
}
|
||||
}
|
||||
encoded := string(materials.Candidates.Content) + string(materials.Transcript.Content)
|
||||
if strings.Contains(encoded, "npc:sha256:internal") || strings.Contains(encoded, doc.ID) {
|
||||
t.Fatalf("model material leaked private identifier or source id: %s", encoded)
|
||||
}
|
||||
|
||||
var candidates normalizeCandidateInput
|
||||
if err := json.Unmarshal(materials.Candidates.Content, &candidates); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(candidates.NPCs) != 2 || candidates.NPCs[0].Name != "Mira Thorn" || candidates.NPCs[0].SourceRefs[0] != (normalizeCandidateSourceRef{StartUnitID: 10, EndUnitID: 20}) {
|
||||
t.Fatalf("candidates = %#v, want two valid current-order candidates", candidates)
|
||||
}
|
||||
var transcript normalizeTranscriptInput
|
||||
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 6 {
|
||||
t.Fatalf("transcript = %#v, want one coalesced window", transcript)
|
||||
}
|
||||
units := transcript.Windows[0].Units
|
||||
for index, wantID := range []int{40, 10, 70, 20, 90, 30} {
|
||||
if units[index].ID != wantID {
|
||||
t.Fatalf("unit %d id = %d, want source-order id %d", index, units[index].ID, wantID)
|
||||
}
|
||||
}
|
||||
if units[0].Cited || !units[1].Cited || !units[2].Cited || !units[3].Cited || !units[4].Cited || units[5].Cited {
|
||||
t.Fatalf("citation markers = %#v, want original ranges only", units)
|
||||
}
|
||||
if units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
|
||||
t.Fatalf("metadata = %#v, want copied generic metadata", units[1].Metadata)
|
||||
}
|
||||
|
||||
windows, err := normalizeContextWindows(doc.Units, []sourceInterval{{start: 1, end: 1}}, make([]bool, len(doc.Units)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
windows[0].Units[0].Metadata["speaker"].(map[string]any)["name"] = "changed"
|
||||
if doc.Units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
|
||||
t.Fatal("copied metadata aliases source document")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildNormalizeContextMaterialsExcludesInvalidReferencesAndCoalescesAdjacentWindows(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
|
||||
{ID: 9}, {ID: 3}, {ID: 8}, {ID: 1}, {ID: 7}, {ID: 2}, {ID: 6},
|
||||
}}
|
||||
records := []dnd.NPC{
|
||||
{Name: "One", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 3, EndUnitID: 3}}},
|
||||
{Name: "Two", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 8, EndUnitID: 8}}},
|
||||
{Name: "Blank"},
|
||||
{Name: "Missing", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 99, EndUnitID: 99}}},
|
||||
{Name: "Foreign", SourceRefs: []source.SourceRef{{SourceID: "other", StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "Reversed", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 6, EndUnitID: 3}}},
|
||||
}
|
||||
materials, ready, err := buildNormalizeContextMaterials(doc, records, 0)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("buildNormalizeContextMaterials() error = %v, ready = %t", err, ready)
|
||||
}
|
||||
var candidates normalizeCandidateInput
|
||||
if err := json.Unmarshal(materials.Candidates.Content, &candidates); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := []string{candidates.NPCs[0].Name, candidates.NPCs[1].Name}; !reflect.DeepEqual(got, []string{"One", "Two"}) {
|
||||
t.Fatalf("candidate names = %#v, want only valid records", got)
|
||||
}
|
||||
var transcript normalizeTranscriptInput
|
||||
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 2 {
|
||||
t.Fatalf("windows = %#v, want adjacent cited units coalesced", transcript.Windows)
|
||||
}
|
||||
if transcript.Windows[0].Units[0].ID != 3 || transcript.Windows[0].Units[1].ID != 8 {
|
||||
t.Fatalf("window units = %#v, want document-order adjacent units", transcript.Windows[0].Units)
|
||||
}
|
||||
}
|
||||
@@ -17,12 +17,14 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
const (
|
||||
Key = "dnd/npcs"
|
||||
PromptID = "dnd.npcs.normalize"
|
||||
normalizationPolicy = "dnd.npcs.normalize.v3"
|
||||
semanticContextPolicy = "dnd.npcs.semantic_context.v1"
|
||||
semanticContextPolicy = "dnd.entity_reconcile.context.v1"
|
||||
NormalizationPolicy = normalizationPolicy
|
||||
|
||||
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
|
||||
@@ -57,7 +59,7 @@ func New(llmClient contracts.StructuredLLMClient, _ Options) (*Normalizer, error
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("load prompt metadata: %w", err)
|
||||
}
|
||||
responseSchema, err := loadResponseSchema()
|
||||
responseSchema, err := entityreconcile.LoadResponseSchema()
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("load response schema: %w", err)
|
||||
}
|
||||
@@ -73,12 +75,12 @@ func (n *Normalizer) ManifestMetadata() map[string]any {
|
||||
}
|
||||
return map[string]any{
|
||||
"prompt_id": PromptID,
|
||||
"prompt_version": SchemaVersion,
|
||||
"prompt_version": entityreconcile.SchemaVersion,
|
||||
"prompt_sha256": n.promptSHA,
|
||||
"response_schema_key": string(ResponseSchemaKey),
|
||||
"response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName,
|
||||
"response_schema_version": SchemaVersion,
|
||||
"response_schema_key": string(entityreconcile.ResponseSchemaKey),
|
||||
"response_schema_id": entityreconcile.ResponseSchemaID,
|
||||
"response_schema_name": entityreconcile.ResponseSchemaName,
|
||||
"response_schema_version": entityreconcile.SchemaVersion,
|
||||
"response_schema_sha256": n.responseSchemaSHA,
|
||||
"identity_policy": identity.Policy,
|
||||
"normalization_policy": normalizationPolicy,
|
||||
@@ -117,7 +119,7 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
||||
order := shared.NewSourceRefOrder(req.Source)
|
||||
records, warnings := preprocessRecords(req.MergeOutput.Value, order)
|
||||
deterministic := recordList(records)
|
||||
materials, ready, err := buildDefaultNormalizeContextMaterials(req.Source, recordValues(records))
|
||||
materials, ready, err := entityreconcile.BuildContext(req.Source, reconciliationCandidates(records), semanticContextRadius)
|
||||
if err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("build semantic context: %w", err)
|
||||
}
|
||||
@@ -125,9 +127,9 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
||||
}
|
||||
|
||||
var response normalizeProposalResponse
|
||||
var response entityreconcile.ProposalResponse
|
||||
if _, err := n.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: entityreconcile.SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID,
|
||||
Inputs: contracts.LLMInputSet{"candidates": materials.Candidates, "transcript": materials.Transcript},
|
||||
}, &response); err != nil {
|
||||
@@ -137,10 +139,10 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("complete structured output: %w", err)
|
||||
}
|
||||
|
||||
assessment := assessProposal(response, records, materials.candidatePositions)
|
||||
applied, semanticWarnings := applySafeGroups(records, assessment.safeGroups, order)
|
||||
assessment := materials.Assess(response)
|
||||
applied, semanticWarnings := applySafeGroups(records, reconciliationGroups(assessment, materials.CandidateKeys()), order)
|
||||
warnings = append(warnings, semanticWarnings...)
|
||||
if assessment.discardedGroups == 0 {
|
||||
if assessment.DiscardedGroups() == 0 {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: recordList(applied), Warnings: limitWarnings(warnings)}, nil
|
||||
}
|
||||
return retryResult(recordList(applied), warnings, assessment), nil
|
||||
@@ -158,14 +160,14 @@ func (n *Normalizer) invalidStructuredResult(value dnd.NPCList, warnings []contr
|
||||
}
|
||||
}
|
||||
|
||||
func retryResult(value dnd.NPCList, warnings []contracts.Warning, assessment proposalAssessment) contracts.TypedNormalizeResult[dnd.NPCList] {
|
||||
func retryResult(value dnd.NPCList, warnings []contracts.Warning, assessment entityreconcile.Assessment) contracts.TypedNormalizeResult[dnd.NPCList] {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{
|
||||
Value: value,
|
||||
Warnings: limitWarningsForRetry(warnings),
|
||||
Retry: &contracts.NormalizeRetry{
|
||||
ReasonCode: ReasonCodeNPCSemanticProposalInvalid,
|
||||
Message: diagnostics.Aggregate("semantic proposal requires retry", assessment.issues),
|
||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(assessment.discardedGroups)},
|
||||
Message: diagnostics.Aggregate("semantic proposal requires retry", reconciliationIssues(assessment)),
|
||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(assessment.DiscardedGroups())},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
func TestModuleContractAndIdentity(t *testing.T) {
|
||||
@@ -33,7 +34,7 @@ func TestModuleContractAndIdentity(t *testing.T) {
|
||||
t.Fatal("New(nil, Options{}) error = nil, want nil client rejection")
|
||||
}
|
||||
normalizer := newNormalizer(t, &recordingNPCNormalizerClient{})
|
||||
if metadata := normalizer.ManifestMetadata(); metadata["identity_policy"] != identity.Policy || metadata["normalization_policy"] != normalizationPolicy || metadata["prompt_id"] != PromptID || metadata["semantic_context_policy"] != semanticContextPolicy || metadata["semantic_context_radius"] != semanticContextRadius {
|
||||
if metadata := normalizer.ManifestMetadata(); metadata["identity_policy"] != identity.Policy || metadata["normalization_policy"] != normalizationPolicy || metadata["prompt_id"] != PromptID || metadata["response_schema_key"] != string(entityreconcile.ResponseSchemaKey) || metadata["response_schema_id"] != entityreconcile.ResponseSchemaID || metadata["response_schema_name"] != entityreconcile.ResponseSchemaName || metadata["semantic_context_policy"] != semanticContextPolicy || metadata["semantic_context_radius"] != semanticContextRadius {
|
||||
t.Fatalf("metadata = %#v", metadata)
|
||||
}
|
||||
wantFingerprints := []pipeline.CheckpointFingerprint{{Name: "prompt", Value: normalizer.promptSHA}, {Name: "response_schema", Value: normalizer.responseSchemaSHA}, {Name: "identity_policy", Value: identity.Policy}, {Name: "normalization_policy", Value: normalizationPolicy}, {Name: "semantic_context_policy", Value: semanticContextPolicy + ":2"}}
|
||||
|
||||
@@ -16,11 +16,11 @@ var promptAssetManifest = shared.PromptAssetManifest{
|
||||
ModuleFiles: []promptfs.ModulePromptFile{
|
||||
{Name: "dnd.npcs.normalize.yaml", Path: "assets/prompts/dnd.npcs.normalize.yaml"},
|
||||
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
||||
{Name: "candidates.md", Path: "assets/prompts/candidates.md"},
|
||||
},
|
||||
SharedFiles: []string{
|
||||
"common-dnd-system.md",
|
||||
"common-dnd-entity-reconciliation.md",
|
||||
"common-dnd-transcript.md",
|
||||
},
|
||||
}
|
||||
@@ -30,10 +30,7 @@ func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare NPC normalization prompt assets: %w", err)
|
||||
}
|
||||
if err := registry.RegisterPromptFS(promptFS, promptAssetRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||
return registry.RegisterPromptFS(promptFS, promptAssetRoot)
|
||||
}
|
||||
|
||||
func promptAssetMetadata() (string, error) {
|
||||
|
||||
@@ -8,17 +8,21 @@ import (
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
"gitea.maximumdirect.net/eric/promptkit"
|
||||
)
|
||||
|
||||
func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
|
||||
if want := []string{"common-dnd-system.md", "common-dnd-transcript.md"}; !reflect.DeepEqual(promptAssetManifest.SharedFiles, want) {
|
||||
if want := []string{"common-dnd-system.md", "common-dnd-entity-reconciliation.md", "common-dnd-transcript.md"}; !reflect.DeepEqual(promptAssetManifest.SharedFiles, want) {
|
||||
t.Fatalf("shared prompt assets = %#v, want %#v", promptAssetManifest.SharedFiles, want)
|
||||
}
|
||||
if promptHash, err := promptAssetMetadata(); err != nil || promptHash == "" {
|
||||
t.Fatalf("promptAssetMetadata() = %q, %v; want prompt fingerprint", promptHash, err)
|
||||
}
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := entityreconcile.RegisterSchemaAssets(registry); err != nil {
|
||||
t.Fatalf("RegisterSchemaAssets() error = %v", err)
|
||||
}
|
||||
if err := RegisterPromptAssets(registry); err != nil {
|
||||
t.Fatalf("RegisterPromptAssets() error = %v", err)
|
||||
}
|
||||
@@ -34,16 +38,16 @@ func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), promptkit.RunRequest{
|
||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "normalize-test-profile",
|
||||
PromptID: PromptID, PromptVersion: entityreconcile.SchemaVersion, ProfileID: "normalize-test-profile",
|
||||
Inputs: map[string]promptkit.ArtifactRef{
|
||||
"candidates": promptkit.Inline(`{"npcs":[{"name":"Mira","source_refs":[]}]}`),
|
||||
"candidates": promptkit.Inline(`{"candidates":[{"key":"candidate-000001","name":"Mira","source_refs":[]}]}`),
|
||||
"transcript": promptkit.Inline(`{"windows":[{"units":[]}]}`),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_npcs_normalize_llm.v1.json" {
|
||||
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_entity_reconcile_llm.v1.json" {
|
||||
t.Fatalf("prepared prompt = %#v, want normalization prompt identity and schema", prepared)
|
||||
}
|
||||
if len(prepared.Messages) != 5 {
|
||||
|
||||
@@ -1,195 +0,0 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
)
|
||||
|
||||
type proposalAssessment struct {
|
||||
safeGroups []safeProposalGroup
|
||||
discardedGroups int
|
||||
issues []string
|
||||
}
|
||||
|
||||
type safeProposalGroup struct {
|
||||
members []int
|
||||
canonical int
|
||||
}
|
||||
|
||||
type assessedProposalGroup struct {
|
||||
members []int
|
||||
canonical int
|
||||
locallyValid bool
|
||||
conflicting bool
|
||||
}
|
||||
|
||||
func assessProposal(response normalizeProposalResponse, records []normalizedRecord, candidatePositions []int) proposalAssessment {
|
||||
positionsByKey := make(map[string][]int, len(candidatePositions))
|
||||
for _, position := range candidatePositions {
|
||||
if position < 0 || position >= len(records) {
|
||||
continue
|
||||
}
|
||||
key := identity.ComparisonKey(records[position].npc.Name)
|
||||
if key != "" {
|
||||
positionsByKey[key] = append(positionsByKey[key], position)
|
||||
}
|
||||
}
|
||||
groups := make([]assessedProposalGroup, len(response.DuplicateGroups))
|
||||
issues := make([]string, 0)
|
||||
owners := make(map[int][]int)
|
||||
|
||||
for groupIndex, proposal := range response.DuplicateGroups {
|
||||
group, groupIssues := assessProposalGroup(proposal, positionsByKey)
|
||||
groups[groupIndex] = group
|
||||
for _, position := range group.members {
|
||||
owners[position] = append(owners[position], groupIndex)
|
||||
}
|
||||
for _, issue := range groupIssues {
|
||||
issues = append(issues, proposalIssue(groupIndex, issue))
|
||||
}
|
||||
}
|
||||
for groupIndex := range groups {
|
||||
for _, position := range groups[groupIndex].members {
|
||||
if len(owners[position]) > 1 {
|
||||
groups[groupIndex].conflicting = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if groups[groupIndex].conflicting {
|
||||
issues = append(issues, proposalIssue(groupIndex, "overlapping_member"))
|
||||
}
|
||||
}
|
||||
|
||||
assessment := proposalAssessment{issues: issues}
|
||||
for _, group := range groups {
|
||||
if !group.locallyValid || group.conflicting {
|
||||
assessment.discardedGroups++
|
||||
continue
|
||||
}
|
||||
assessment.safeGroups = append(assessment.safeGroups, safeProposalGroup{members: group.members, canonical: group.canonical})
|
||||
}
|
||||
return assessment
|
||||
}
|
||||
|
||||
func assessProposalGroup(proposal normalizeProposalGroup, positionsByKey map[string][]int) (assessedProposalGroup, []string) {
|
||||
issues := make([]string, 0)
|
||||
members := make([]int, 0, len(proposal.Members))
|
||||
seenMembers := make(map[int]struct{}, len(proposal.Members))
|
||||
for _, name := range proposal.Members {
|
||||
position, issue := resolveCandidate(name, positionsByKey)
|
||||
if issue != "" {
|
||||
issues = append(issues, "member_"+issue)
|
||||
continue
|
||||
}
|
||||
if _, exists := seenMembers[position]; exists {
|
||||
issues = append(issues, "repeated_member")
|
||||
continue
|
||||
}
|
||||
seenMembers[position] = struct{}{}
|
||||
members = append(members, position)
|
||||
}
|
||||
canonical, canonicalIssue := resolveCandidate(proposal.CanonicalName, positionsByKey)
|
||||
if canonicalIssue != "" {
|
||||
issues = append(issues, "canonical_"+canonicalIssue)
|
||||
}
|
||||
if len(members) < 2 {
|
||||
issues = append(issues, "fewer_than_two_members")
|
||||
}
|
||||
if canonicalIssue == "" && !containsPosition(members, canonical) {
|
||||
issues = append(issues, "canonical_not_member")
|
||||
}
|
||||
sort.Ints(members)
|
||||
return assessedProposalGroup{
|
||||
members: members, canonical: canonical, locallyValid: len(issues) == 0,
|
||||
}, issues
|
||||
}
|
||||
|
||||
func resolveCandidate(name string, positionsByKey map[string][]int) (position int, issue string) {
|
||||
key := identity.ComparisonKey(name)
|
||||
if key == "" {
|
||||
return 0, "blank"
|
||||
}
|
||||
positions := positionsByKey[key]
|
||||
if len(positions) == 0 {
|
||||
return 0, "unknown"
|
||||
}
|
||||
if len(positions) != 1 {
|
||||
return 0, "ambiguous"
|
||||
}
|
||||
return positions[0], ""
|
||||
}
|
||||
|
||||
func containsPosition(positions []int, want int) bool {
|
||||
for _, position := range positions {
|
||||
if position == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func proposalIssue(groupIndex int, category string) string {
|
||||
return "group " + strconv.Itoa(groupIndex) + ": " + category
|
||||
}
|
||||
|
||||
func applySafeGroups(records []normalizedRecord, groups []safeProposalGroup, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
||||
byMember := make(map[int]safeProposalGroup, len(groups)*2)
|
||||
for _, group := range groups {
|
||||
for _, member := range group.members {
|
||||
byMember[member] = group
|
||||
}
|
||||
}
|
||||
output := make([]normalizedRecord, 0, len(records)-len(groups))
|
||||
warnings := make([]contracts.Warning, 0, len(groups))
|
||||
for index, record := range records {
|
||||
group, grouped := byMember[index]
|
||||
if !grouped {
|
||||
output = append(output, cloneRecord(record))
|
||||
continue
|
||||
}
|
||||
if group.members[0] != index {
|
||||
continue
|
||||
}
|
||||
consolidated := consolidateSemanticGroup(records, group, order)
|
||||
output = append(output, consolidated)
|
||||
warnings = append(warnings, semanticDuplicateWarning(consolidated, records[group.canonical]))
|
||||
}
|
||||
return output, warnings
|
||||
}
|
||||
|
||||
func consolidateSemanticGroup(records []normalizedRecord, group safeProposalGroup, order shared.SourceRefOrder) normalizedRecord {
|
||||
output := cloneRecord(records[group.members[0]])
|
||||
output.npc.Name = records[group.canonical].npc.Name
|
||||
for _, member := range group.members[1:] {
|
||||
output.npc.SourceRefs = append(output.npc.SourceRefs, records[member].npc.SourceRefs...)
|
||||
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
|
||||
if records[member].earliest < output.earliest {
|
||||
output.earliest = records[member].earliest
|
||||
}
|
||||
}
|
||||
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
|
||||
output.npc.SourceRefs = order.Canonicalize(output.npc.SourceRefs)
|
||||
output.npc.ID = identity.DeriveID(output.npc.Name)
|
||||
return output
|
||||
}
|
||||
|
||||
func semanticDuplicateWarning(record normalizedRecord, canonical normalizedRecord) contracts.Warning {
|
||||
details := make([]string, 0, len(record.inputIndexes)+1)
|
||||
for _, inputIndex := range record.inputIndexes {
|
||||
details = append(details, fmt.Sprintf("input index %d", inputIndex))
|
||||
}
|
||||
if canonical.earliest != record.earliest {
|
||||
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
|
||||
}
|
||||
return contracts.Warning{
|
||||
Scope: npcScope(record.earliest),
|
||||
ReasonCode: ReasonCodeDuplicateNPCCollapsed,
|
||||
Message: diagnostics.Aggregate("semantic duplicate consolidation", details),
|
||||
}
|
||||
}
|
||||
122
internal/modules/dnd/normalize/npcs/reconciliation.go
Normal file
122
internal/modules/dnd/normalize/npcs/reconciliation.go
Normal file
@@ -0,0 +1,122 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
const semanticContextRadius = 2
|
||||
|
||||
type safeReconciliationGroup struct {
|
||||
members []int
|
||||
canonical int
|
||||
}
|
||||
|
||||
func reconciliationCandidates(records []normalizedRecord) []entityreconcile.Candidate {
|
||||
candidates := make([]entityreconcile.Candidate, len(records))
|
||||
for index, record := range records {
|
||||
candidates[index] = entityreconcile.Candidate{
|
||||
Name: record.npc.Name,
|
||||
SourceRefs: cloneSourceRefs(record.npc.SourceRefs),
|
||||
}
|
||||
}
|
||||
return candidates
|
||||
}
|
||||
|
||||
func reconciliationGroups(assessment entityreconcile.Assessment, candidateKeys []string) []safeReconciliationGroup {
|
||||
positions := make(map[string]int, len(candidateKeys))
|
||||
for index, key := range candidateKeys {
|
||||
positions[key] = index
|
||||
}
|
||||
safeGroups := assessment.SafeGroups()
|
||||
groups := make([]safeReconciliationGroup, 0, len(safeGroups))
|
||||
for _, group := range safeGroups {
|
||||
members := group.Members()
|
||||
memberPositions := make([]int, len(members))
|
||||
valid := true
|
||||
for index, key := range members {
|
||||
position, ok := positions[key]
|
||||
if !ok {
|
||||
valid = false
|
||||
break
|
||||
}
|
||||
memberPositions[index] = position
|
||||
}
|
||||
canonical, ok := positions[group.Canonical()]
|
||||
if !valid || !ok {
|
||||
continue
|
||||
}
|
||||
groups = append(groups, safeReconciliationGroup{members: memberPositions, canonical: canonical})
|
||||
}
|
||||
return groups
|
||||
}
|
||||
|
||||
func reconciliationIssues(assessment entityreconcile.Assessment) []string {
|
||||
issues := assessment.Issues()
|
||||
details := make([]string, len(issues))
|
||||
for index, issue := range issues {
|
||||
details[index] = fmt.Sprintf("group %d: %s", issue.GroupIndex, issue.Category)
|
||||
}
|
||||
return details
|
||||
}
|
||||
|
||||
func applySafeGroups(records []normalizedRecord, groups []safeReconciliationGroup, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
||||
byMember := make(map[int]safeReconciliationGroup, len(groups)*2)
|
||||
for _, group := range groups {
|
||||
for _, member := range group.members {
|
||||
byMember[member] = group
|
||||
}
|
||||
}
|
||||
output := make([]normalizedRecord, 0, len(records)-len(groups))
|
||||
warnings := make([]contracts.Warning, 0, len(groups))
|
||||
for index, record := range records {
|
||||
group, grouped := byMember[index]
|
||||
if !grouped {
|
||||
output = append(output, cloneRecord(record))
|
||||
continue
|
||||
}
|
||||
if group.members[0] != index {
|
||||
continue
|
||||
}
|
||||
consolidated := consolidateSemanticGroup(records, group, order)
|
||||
output = append(output, consolidated)
|
||||
warnings = append(warnings, semanticDuplicateWarning(consolidated, records[group.canonical]))
|
||||
}
|
||||
return output, warnings
|
||||
}
|
||||
|
||||
func consolidateSemanticGroup(records []normalizedRecord, group safeReconciliationGroup, order shared.SourceRefOrder) normalizedRecord {
|
||||
output := cloneRecord(records[group.members[0]])
|
||||
output.npc.Name = records[group.canonical].npc.Name
|
||||
for _, member := range group.members[1:] {
|
||||
output.npc.SourceRefs = append(output.npc.SourceRefs, records[member].npc.SourceRefs...)
|
||||
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
|
||||
if records[member].earliest < output.earliest {
|
||||
output.earliest = records[member].earliest
|
||||
}
|
||||
}
|
||||
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
|
||||
output.npc.SourceRefs = order.Canonicalize(output.npc.SourceRefs)
|
||||
output.npc.ID = identity.DeriveID(output.npc.Name)
|
||||
return output
|
||||
}
|
||||
|
||||
func semanticDuplicateWarning(record normalizedRecord, canonical normalizedRecord) contracts.Warning {
|
||||
details := make([]string, 0, len(record.inputIndexes)+1)
|
||||
for _, inputIndex := range record.inputIndexes {
|
||||
details = append(details, fmt.Sprintf("input index %d", inputIndex))
|
||||
}
|
||||
if canonical.earliest != record.earliest {
|
||||
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
|
||||
}
|
||||
return contracts.Warning{
|
||||
Scope: npcScope(record.earliest),
|
||||
ReasonCode: ReasonCodeDuplicateNPCCollapsed,
|
||||
Message: diagnostics.Aggregate("semantic duplicate consolidation", details),
|
||||
}
|
||||
}
|
||||
@@ -1,65 +0,0 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
)
|
||||
|
||||
func TestNormalizeResponseSchemaIsStrictlyStructural(t *testing.T) {
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v", err)
|
||||
}
|
||||
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Name != ResponseSchemaName || schema.Version != SchemaVersion || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v, want private normalization schema identity", schema)
|
||||
}
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
value any
|
||||
valid bool
|
||||
}{
|
||||
{name: "empty groups", value: map[string]any{"duplicate_groups": []any{}}, valid: true},
|
||||
{name: "semantically invalid group", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{"", "unknown"}, "canonical_name": ""}}}, valid: true},
|
||||
{name: "missing groups", value: map[string]any{}},
|
||||
{name: "unknown top level field", value: map[string]any{"duplicate_groups": []any{}, "extra": true}},
|
||||
{name: "unknown group field", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{}, "canonical_name": "Mira", "extra": true}}}},
|
||||
{name: "wrong groups type", value: map[string]any{"duplicate_groups": "no"}},
|
||||
{name: "wrong member type", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{1}, "canonical_name": "Mira"}}}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
content, err := json.Marshal(test.value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err = validateNormalizeSchema(content, schema.JSONSchema)
|
||||
if (err == nil) != test.valid {
|
||||
t.Fatalf("validateNormalizeSchema() error = %v, want valid=%t", err, test.valid)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func validateNormalizeSchema(instanceContent, schemaContent []byte) error {
|
||||
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(instanceContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
document, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := compiler.AddResource("schema.json", document); err != nil {
|
||||
return err
|
||||
}
|
||||
compiled, err := compiler.Compile("schema.json")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return compiled.Validate(instance)
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
)
|
||||
|
||||
func TestNormalizeSkipsSemanticCompletionWithoutTwoEligibleCandidates(t *testing.T) {
|
||||
@@ -29,7 +30,7 @@ func TestNormalizeSkipsSemanticCompletionWithoutTwoEligibleCandidates(t *testing
|
||||
}
|
||||
|
||||
func TestNormalizeAppliesSafeProposalAndUsesPrivateInputs(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":[" MIRA THORN ","Mira"],"canonical_name":"Mira Thorn"}]}`}
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["candidate-000001","candidate-000002"],"canonical":"candidate-000002"}]}`}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
@@ -62,7 +63,7 @@ func TestNormalizeAppliesSafeProposalAndUsesPrivateInputs(t *testing.T) {
|
||||
t.Fatalf("completion calls = %d, want one", len(client.requests))
|
||||
}
|
||||
completion := client.requests[0]
|
||||
if completion.StageName != Key || completion.PromptID != PromptID || completion.PromptVersion != SchemaVersion || completion.ProfileID != request.LLMProfile || completion.SessionID != request.SessionID || len(completion.Inputs) != 2 {
|
||||
if completion.StageName != Key || completion.PromptID != PromptID || completion.PromptVersion != entityreconcile.SchemaVersion || completion.ProfileID != request.LLMProfile || completion.SessionID != request.SessionID || len(completion.Inputs) != 2 {
|
||||
t.Fatalf("completion request = %#v, want normalize request identity and exactly two inputs", completion)
|
||||
}
|
||||
encoded := string(completion.Inputs["candidates"].Content) + string(completion.Inputs["transcript"].Content)
|
||||
@@ -72,7 +73,7 @@ func TestNormalizeAppliesSafeProposalAndUsesPrivateInputs(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestNormalizeUnsafeProposalReturnsSafeRetryFallback(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Captain Vale","Unknown"],"canonical_name":"Captain Vale"}]}`}
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["candidate-000001","candidate-000002"],"canonical":"candidate-000002"},{"members":["candidate-000003","unknown"],"canonical":"candidate-000003"}]}`}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
@@ -95,30 +96,6 @@ func TestNormalizeUnsafeProposalReturnsSafeRetryFallback(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRejectsOverlapsWithoutResponseOrderDependence(t *testing.T) {
|
||||
records := []normalizedRecord{
|
||||
{npc: dnd.NPC{Name: "Alpha"}}, {npc: dnd.NPC{Name: "Bravo"}}, {npc: dnd.NPC{Name: "Charlie"}}, {npc: dnd.NPC{Name: "Delta"}},
|
||||
}
|
||||
for index := range records {
|
||||
records[index].inputIndexes = []int{index}
|
||||
records[index].earliest = index
|
||||
}
|
||||
response := normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{
|
||||
{Members: []string{"Alpha", "Bravo"}, CanonicalName: "Alpha"},
|
||||
{Members: []string{"Bravo", "Charlie"}, CanonicalName: "Bravo"},
|
||||
{Members: []string{"Charlie", "Delta"}, CanonicalName: "Charlie"},
|
||||
}}
|
||||
assessment := assessProposal(response, records, []int{0, 1, 2, 3})
|
||||
if assessment.discardedGroups != 3 || len(assessment.safeGroups) != 0 {
|
||||
t.Fatalf("assessment = %#v, want chained conflicts all discarded", assessment)
|
||||
}
|
||||
for _, issue := range []string{"group 0: overlapping_member", "group 1: overlapping_member", "group 2: overlapping_member"} {
|
||||
if !containsString(assessment.issues, issue) {
|
||||
t.Fatalf("issues = %#v, want %q", assessment.issues, issue)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeInvalidStructuredOutputAndOperationalErrorsRemainDistinct(t *testing.T) {
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
@@ -159,7 +136,7 @@ func TestNormalizeRedactsContextMaterialFailures(t *testing.T) {
|
||||
request := normalizeRequestWithSource(input, doc)
|
||||
request.SourceInput = contracts.NewLLMInputMaterial("source", "application/json", []byte(transcript), "sha256:test", originPath)
|
||||
_, err := normalizer.Normalize(context.Background(), request)
|
||||
if err == nil || !strings.Contains(err.Error(), "build NPC normalization context: invalid source metadata") {
|
||||
if err == nil || !strings.Contains(err.Error(), "build entity reconciliation context: invalid source metadata") {
|
||||
t.Fatalf("Normalize() error = %v; want content-safe context-material failure", err)
|
||||
}
|
||||
for _, forbidden := range []string{metadataKey, metadataValue, transcript, firstName, secondName, sourceID, originPath, "float64", "non-finite"} {
|
||||
@@ -174,8 +151,8 @@ func TestNormalizeRedactsContextMaterialFailures(t *testing.T) {
|
||||
|
||||
func TestNormalizeDoesNotAccumulateSafeGroupsAcrossAttempts(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{responses: []string{
|
||||
`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Captain Vale","Unknown"],"canonical_name":"Captain Vale"}]}`,
|
||||
`{"duplicate_groups":[{"members":["Mira","Captain Vale"],"canonical_name":"Captain Vale"}]}`,
|
||||
`{"duplicate_groups":[{"members":["candidate-000001","candidate-000002"],"canonical":"candidate-000002"},{"members":["candidate-000003","unknown"],"canonical":"candidate-000003"}]}`,
|
||||
`{"duplicate_groups":[{"members":["candidate-000001","candidate-000003"],"canonical":"candidate-000003"}]}`,
|
||||
}}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
@@ -198,7 +175,7 @@ func TestNormalizeDoesNotAccumulateSafeGroupsAcrossAttempts(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestNormalizeExcludesIneligibleRecordsFromSemanticGroups(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["Mira","Broken"],"canonical_name":"Mira"}]}`}
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["candidate-000001","candidate-000002"],"canonical":"candidate-000001"}]}`}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
@@ -210,42 +187,31 @@ func TestNormalizeExcludesIneligibleRecordsFromSemanticGroups(t *testing.T) {
|
||||
if err != nil || result.Retry == nil || len(result.Value.NPCs) != 3 {
|
||||
t.Fatalf("Normalize() = %#v, %v; want unchanged retry fallback", result, err)
|
||||
}
|
||||
if !strings.Contains(result.Retry.Message, "member_unknown") || result.Value.NPCs[1].Name != "Broken" {
|
||||
if !strings.Contains(result.Retry.Message, "member_ineligible") || result.Value.NPCs[1].Name != "Broken" {
|
||||
t.Fatalf("result = %#v, want ineligible record excluded but preserved", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProposalValidationRejectsUnsafeCategories(t *testing.T) {
|
||||
func TestReconciliationCandidatesKeepEqualDisplayNamesDistinct(t *testing.T) {
|
||||
doc := semanticDocument()
|
||||
records := []normalizedRecord{
|
||||
{npc: dnd.NPC{Name: "Mira"}, inputIndexes: []int{0}, earliest: 0},
|
||||
{npc: dnd.NPC{Name: "Mira Thorn"}, inputIndexes: []int{1}, earliest: 1},
|
||||
{npc: dnd.NPC{Name: "Captain Vale"}, inputIndexes: []int{2}, earliest: 2},
|
||||
{npc: dnd.NPC{Name: "The Guard", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}},
|
||||
{npc: dnd.NPC{Name: "The Guard", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}}},
|
||||
}
|
||||
for _, proposal := range []normalizeProposalGroup{
|
||||
{Members: []string{"Mira"}, CanonicalName: "Mira"},
|
||||
{Members: []string{"Mira", "Mira"}, CanonicalName: "Mira"},
|
||||
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: "Unknown"},
|
||||
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: " "},
|
||||
{Members: []string{" ", "Mira Thorn"}, CanonicalName: "Mira Thorn"},
|
||||
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: "Captain Vale"},
|
||||
} {
|
||||
assessment := assessProposal(normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{proposal}}, records, []int{0, 1, 2})
|
||||
if assessment.discardedGroups != 1 || len(assessment.safeGroups) != 0 {
|
||||
t.Fatalf("assessment for %#v = %#v, want discarded unsafe group", proposal, assessment)
|
||||
materials, ready, err := entityreconcile.BuildContext(doc, reconciliationCandidates(records), semanticContextRadius)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("BuildContext() = %#v, %t, %v; want ready keyed candidates", materials, ready, err)
|
||||
}
|
||||
keys := materials.CandidateKeys()
|
||||
if !reflect.DeepEqual(keys, []string{"candidate-000001", "candidate-000002"}) || strings.Count(string(materials.Candidates.Content), `"The Guard"`) != 2 {
|
||||
t.Fatalf("candidate keys and inputs = %#v, %s; want distinct equal-display candidates", keys, materials.Candidates.Content)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProposalResolutionUsesOnlyExistingComparisonKeyEquivalences(t *testing.T) {
|
||||
records := []normalizedRecord{
|
||||
{npc: dnd.NPC{Name: "O'Neill"}, inputIndexes: []int{0}, earliest: 0},
|
||||
{npc: dnd.NPC{Name: "Mira Thorn"}, inputIndexes: []int{1}, earliest: 1},
|
||||
}
|
||||
assessment := assessProposal(normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{{
|
||||
Members: []string{" O’NEILL ", "Ｍｉｒａ Ｔｈｏｒｎ"}, CanonicalName: "Ｍｉｒａ Ｔｈｏｒｎ",
|
||||
}}}, records, []int{0, 1})
|
||||
if assessment.discardedGroups != 0 || len(assessment.safeGroups) != 1 || assessment.safeGroups[0].canonical != 1 {
|
||||
t.Fatalf("assessment = %#v, want comparison-key-only resolution", assessment)
|
||||
assessment := materials.Assess(entityreconcile.ProposalResponse{DuplicateGroups: []entityreconcile.DuplicateGroup{{
|
||||
Members: keys, Canonical: keys[1],
|
||||
}}})
|
||||
groups := reconciliationGroups(assessment, keys)
|
||||
if assessment.DiscardedGroups() != 0 || len(groups) != 1 || groups[0].canonical != 1 {
|
||||
t.Fatalf("reconciliation = %#v, %#v; want distinct keyed group", assessment, groups)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -276,12 +242,3 @@ func cloneNPCList(input dnd.NPCList) dnd.NPCList {
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
func containsString(values []string, want string) bool {
|
||||
for _, value := range values {
|
||||
if value == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -7,9 +7,6 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"mime"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
@@ -17,6 +14,7 @@ import (
|
||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/registryresolver"
|
||||
)
|
||||
|
||||
const (
|
||||
@@ -37,39 +35,20 @@ type Registry struct {
|
||||
lookupByKey map[string]int
|
||||
}
|
||||
|
||||
// Resolver retains only the validated construction-time registry and immutable
|
||||
// canonical registries keyed by their semantic digest. Operation references
|
||||
// are resolved on demand; caller-owned reference bytes are never retained.
|
||||
// Resolver selects and memoizes immutable NPC registry views.
|
||||
type Resolver struct {
|
||||
seeded *Registry
|
||||
|
||||
mu sync.Mutex
|
||||
cache map[string]*Registry
|
||||
rawCache map[string]*Registry
|
||||
resolver *registryresolver.Resolver[*Registry]
|
||||
}
|
||||
|
||||
// NewResolver validates the optional construction-time NPC reference and
|
||||
// prepares the operation-time registry cache. A malformed static reference
|
||||
// therefore fails before any operation starts.
|
||||
func NewResolver(references contracts.ReferenceSet) (*Resolver, error) {
|
||||
seeded, err := Resolve(constructionReferences(references))
|
||||
resolver, err := registryresolver.New(registryResolverConfig(), references)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Resolver{seeded: seeded, cache: make(map[string]*Registry), rawCache: make(map[string]*Registry)}, nil
|
||||
}
|
||||
|
||||
func constructionReferences(references contracts.ReferenceSet) contracts.ReferenceSet {
|
||||
slot, ok := references.Slots[ReferenceSlot]
|
||||
if !ok || len(slot.Items) > 0 {
|
||||
return references
|
||||
}
|
||||
cloned := contracts.ReferenceSet{Slots: make(map[string]contracts.ResolvedReferenceSlot, len(references.Slots))}
|
||||
for name, value := range references.Slots {
|
||||
cloned.Slots[name] = value
|
||||
}
|
||||
delete(cloned.Slots, ReferenceSlot)
|
||||
return cloned
|
||||
return &Resolver{resolver: resolver}, nil
|
||||
}
|
||||
|
||||
// Seeded returns the immutable construction-time registry. Its accessors are
|
||||
@@ -78,7 +57,7 @@ func (r *Resolver) Seeded() *Registry {
|
||||
if r == nil {
|
||||
return nil
|
||||
}
|
||||
return r.seeded
|
||||
return r.resolver.Seeded()
|
||||
}
|
||||
|
||||
// Resolve returns the effective registry for one operation. An operation
|
||||
@@ -86,62 +65,43 @@ func (r *Resolver) Seeded() *Registry {
|
||||
// matching that registry reuses it; other canonical registries are cached by
|
||||
// digest for concurrent chunk operations.
|
||||
func (r *Resolver) Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
if r == nil {
|
||||
if r == nil || r.resolver == nil {
|
||||
return Resolve(references)
|
||||
}
|
||||
if _, ok := references.Slots[ReferenceSlot]; !ok {
|
||||
return r.seeded, nil
|
||||
}
|
||||
|
||||
slot := references.Slots[ReferenceSlot]
|
||||
rawKey := ""
|
||||
if len(slot.Items) == 1 {
|
||||
rawKey = strings.ToLower(strings.TrimSpace(slot.Items[0].MediaType)) + "\x00" + semanticDigest(slot.Items[0].Content)
|
||||
}
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if rawKey != "" {
|
||||
if cached, ok := r.rawCache[rawKey]; ok {
|
||||
return cached, nil
|
||||
}
|
||||
}
|
||||
resolved, err := Resolve(references)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if sameRegistryIdentity(r.seeded, resolved) {
|
||||
if rawKey != "" {
|
||||
r.rawCache[rawKey] = r.seeded
|
||||
}
|
||||
return r.seeded, nil
|
||||
}
|
||||
|
||||
if cached, ok := r.cache[resolved.Digest()]; ok {
|
||||
if rawKey != "" {
|
||||
r.rawCache[rawKey] = cached
|
||||
}
|
||||
return cached, nil
|
||||
}
|
||||
r.cache[resolved.Digest()] = resolved
|
||||
if rawKey != "" {
|
||||
r.rawCache[rawKey] = resolved
|
||||
}
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
func sameRegistryIdentity(first, second *Registry) bool {
|
||||
if first == nil || second == nil {
|
||||
return first == second
|
||||
}
|
||||
return first.bound == second.bound && first.digest == second.digest
|
||||
return r.resolver.Resolve(references)
|
||||
}
|
||||
|
||||
// Resolve prepares the optional NPC registry reference. An absent slot
|
||||
// produces the exact empty prompt input and no semantic registry identity.
|
||||
func Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
slot, ok := references.Slots[ReferenceSlot]
|
||||
if !ok {
|
||||
item, present, err := registryresolver.ResolveOptionalSingleItem(references, npcReferenceSpec())
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !present {
|
||||
return emptyRegistry(), nil
|
||||
}
|
||||
return loadRegistry(item.Content)
|
||||
}
|
||||
|
||||
func registryResolverConfig() registryresolver.Config[*Registry] {
|
||||
return registryresolver.Config[*Registry]{
|
||||
Reference: npcReferenceSpec(),
|
||||
Absent: func() (*Registry, error) {
|
||||
return emptyRegistry(), nil
|
||||
},
|
||||
Load: loadRegistry,
|
||||
SemanticIdentity: func(registry *Registry) string {
|
||||
return registry.Digest()
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func npcReferenceSpec() registryresolver.ReferenceSpec {
|
||||
return registryresolver.ReferenceSpec{SlotName: ReferenceSlot, AcceptedMediaType: npccodec.MediaType, MaxBytes: MaxBytes}
|
||||
}
|
||||
|
||||
func emptyRegistry() *Registry {
|
||||
content := []byte(emptyPrompt)
|
||||
projectionDigest := semanticDigest(content)
|
||||
return &Registry{
|
||||
@@ -150,26 +110,12 @@ func Resolve(references contracts.ReferenceSet) (*Registry, error) {
|
||||
projectionDigest: projectionDigest,
|
||||
promptInput: contracts.NewLLMInputMaterial(ReferenceSlot, npccodec.MediaType, content, projectionDigest, ""),
|
||||
lookupByKey: map[string]int{},
|
||||
}, nil
|
||||
}
|
||||
if len(slot.Items) != 1 {
|
||||
return nil, fmt.Errorf("reference slot %q must contain exactly one item", ReferenceSlot)
|
||||
}
|
||||
|
||||
item := slot.Items[0]
|
||||
mediaType, _, err := mime.ParseMediaType(item.MediaType)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("reference slot %q item media type is invalid", ReferenceSlot)
|
||||
}
|
||||
if !strings.EqualFold(mediaType, npccodec.MediaType) {
|
||||
return nil, fmt.Errorf("reference slot %q item media type must be %s", ReferenceSlot, npccodec.MediaType)
|
||||
}
|
||||
if len(item.Content) > MaxBytes {
|
||||
return nil, fmt.Errorf("reference slot %q item is %d bytes, limit %d", ReferenceSlot, len(item.Content), MaxBytes)
|
||||
}
|
||||
}
|
||||
|
||||
func loadRegistry(referenceContent []byte) (*Registry, error) {
|
||||
codec := npccodec.New()
|
||||
value, err := codec.Decode(item.Content)
|
||||
value, err := codec.Decode(referenceContent)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("decode NPC registry: invalid approved NPC JSON")
|
||||
}
|
||||
|
||||
@@ -112,13 +112,55 @@ func TestResolveRejectsMalformedOrUnsupportedRegistryInput(t *testing.T) {
|
||||
|
||||
func TestResolverReusesEquivalentCanonicalRegistries(t *testing.T) {
|
||||
set := listReferenceSet(t, registryFixture())
|
||||
resolver, err := NewResolver(set)
|
||||
resolver, err := NewResolver(contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, err := resolver.Resolve(set)
|
||||
if err != nil || resolved != resolver.Seeded() {
|
||||
t.Fatalf("Resolve() = %p, %v; seeded %p", resolved, err, resolver.Seeded())
|
||||
first, err := resolver.Resolve(set)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
content := set.Slots[ReferenceSlot].Items[0].Content
|
||||
equivalent := append([]byte("\n "), content...)
|
||||
equivalent = append(equivalent, '\n')
|
||||
second, err := resolver.Resolve(referenceSet(contracts.ReferenceItem{
|
||||
MediaType: "APPLICATION/JSON; charset=utf-8",
|
||||
Content: equivalent,
|
||||
}))
|
||||
if err != nil || second != first {
|
||||
t.Fatalf("equivalent canonical registry = %p / %p, %v", first, second, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverHandlesConstructionAndOperationReferences(t *testing.T) {
|
||||
placeholder, err := NewResolver(referenceSet())
|
||||
if err != nil || placeholder.Seeded().Bound() {
|
||||
t.Fatalf("generated placeholder = %#v, %v; want unbound seed", placeholder, err)
|
||||
}
|
||||
|
||||
valid := listReferenceSet(t, registryFixture())
|
||||
validContent := valid.Slots[ReferenceSlot].Items[0].Content
|
||||
malformed := referenceSet(contracts.ReferenceItem{MediaType: npccodec.MediaType, Content: []byte(`{"npcs":[`)})
|
||||
if _, err := NewResolver(malformed); err == nil {
|
||||
t.Fatal("NewResolver(malformed) error = nil")
|
||||
}
|
||||
if _, err := placeholder.Resolve(malformed); err == nil {
|
||||
t.Fatal("Resolve(malformed) error = nil")
|
||||
}
|
||||
|
||||
staticContent := append([]byte(nil), validContent...)
|
||||
staticReferences := referenceSet(contracts.ReferenceItem{MediaType: npccodec.MediaType, Content: staticContent})
|
||||
seeded, err := NewResolver(staticReferences)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
staticContent[0] = '['
|
||||
delete(staticReferences.Slots, ReferenceSlot)
|
||||
if seeded.Seeded().Count() != 2 || seeded.Seeded().CanonicalBytes()[0] != '{' {
|
||||
t.Fatalf("seeded registry retained construction references: %#v", seeded.Seeded())
|
||||
}
|
||||
if fallback, err := seeded.Resolve(contracts.ReferenceSet{}); err != nil || fallback != seeded.Seeded() {
|
||||
t.Fatalf("fallback = %#v, %v; want seeded registry", fallback, err)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -147,8 +189,8 @@ func listReferenceSet(t *testing.T, list dnd.NPCList) contracts.ReferenceSet {
|
||||
return referenceSet(contracts.ReferenceItem{SlotName: ReferenceSlot, MediaType: npccodec.MediaType, Content: content})
|
||||
}
|
||||
|
||||
func referenceSet(item contracts.ReferenceItem) contracts.ReferenceSet {
|
||||
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{ReferenceSlot: {Items: []contracts.ReferenceItem{item}}}}
|
||||
func referenceSet(items ...contracts.ReferenceItem) contracts.ReferenceSet {
|
||||
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{ReferenceSlot: {Items: items}}}
|
||||
}
|
||||
|
||||
func TestProjectionIsStableForEquivalentNormalizedRegistries(t *testing.T) {
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
combatextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
|
||||
enemyeventextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/enemyevents"
|
||||
itemeventextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/itemevents"
|
||||
locationoccurrenceextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locationoccurrences"
|
||||
locationextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locations"
|
||||
interactionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
|
||||
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
scenedescriptionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
||||
@@ -12,6 +14,8 @@ import (
|
||||
combatnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/combatturns"
|
||||
enemyeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/enemyevents"
|
||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||
locationoccurrencenormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locationoccurrences"
|
||||
locationnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locations"
|
||||
interactionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcinteractions"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
scenedescriptionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/scenedescriptions"
|
||||
@@ -29,6 +33,15 @@ import (
|
||||
itemeventshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemevents/shape"
|
||||
itemeventrefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemevents/source_refs"
|
||||
itemeventrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemevents/source_relatedness"
|
||||
occurrenceinvariants "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/invariants"
|
||||
occurrenceregistry "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/registry"
|
||||
occurrenceshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/shape"
|
||||
occurrencerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/source_refs"
|
||||
occurrencerelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/source_relatedness"
|
||||
locationidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/identity"
|
||||
locationshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/shape"
|
||||
locationrefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/source_refs"
|
||||
locationrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/source_relatedness"
|
||||
interactioninvariants "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcinteractions/invariants"
|
||||
interactionregistry "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcinteractions/registry"
|
||||
interactionshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcinteractions/shape"
|
||||
@@ -245,5 +258,42 @@ func registerDefaultChains(registry *pipeline.ValidatorChainRegistry) error {
|
||||
},
|
||||
})
|
||||
}},
|
||||
{name: "locations validator chain", register: func() error {
|
||||
return registry.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageExtract, Module: locationextract.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key), pipeline.Binding(locationshape.Key), pipeline.Binding(locationrefs.Key),
|
||||
pipeline.Binding(validjsonschema.Key), pipeline.Binding(locationrelatedness.Key),
|
||||
},
|
||||
})
|
||||
}},
|
||||
{name: "locations normalize validator chain", register: func() error {
|
||||
return registry.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageNormalize, Module: locationnormalize.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key), pipeline.Binding(locationshape.Key), pipeline.Binding(locationidentity.Key),
|
||||
pipeline.Binding(locationrefs.Key), pipeline.Binding(validjsonschema.Key), pipeline.Binding(locationrelatedness.Key),
|
||||
},
|
||||
})
|
||||
}},
|
||||
{name: "location occurrences validator chain", register: func() error {
|
||||
return registry.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageExtract, Module: locationoccurrenceextract.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key), pipeline.Binding(occurrenceshape.Key), pipeline.Binding(occurrenceregistry.Key),
|
||||
pipeline.Binding(occurrencerefs.Key), pipeline.Binding(validjsonschema.Key), pipeline.Binding(occurrencerelatedness.Key),
|
||||
},
|
||||
})
|
||||
}},
|
||||
{name: "location occurrences normalize validator chain", register: func() error {
|
||||
return registry.Register(pipeline.ValidatorChainMapping{
|
||||
Stage: pipeline.StageNormalize, Module: locationoccurrencenormalize.Key,
|
||||
Validators: []pipeline.ModuleBinding{
|
||||
pipeline.Binding(validjson.Key), pipeline.Binding(occurrenceshape.Key), pipeline.Binding(occurrenceregistry.Key),
|
||||
pipeline.Binding(occurrenceinvariants.Key), pipeline.Binding(occurrencerefs.Key), pipeline.Binding(validjsonschema.Key),
|
||||
pipeline.Binding(occurrencerelatedness.Key),
|
||||
},
|
||||
})
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -25,6 +25,12 @@ func registerEvidence(registry *pipeline.ArtifactEvidenceRegistry) error {
|
||||
{name: "scene descriptions evidence", register: func() error {
|
||||
return pipeline.RegisterArtifactEvidence(registry, dnd.SceneDescriptionListKind, sceneDescriptionEvidence)
|
||||
}},
|
||||
{name: "locations evidence", register: func() error {
|
||||
return pipeline.RegisterArtifactEvidence(registry, dnd.LocationListKind, locationEvidence)
|
||||
}},
|
||||
{name: "location occurrences evidence", register: func() error {
|
||||
return pipeline.RegisterArtifactEvidence(registry, dnd.LocationOccurrenceListKind, locationOccurrenceEvidence)
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -83,3 +89,19 @@ func sceneDescriptionEvidence(value dnd.SceneDescriptionList) []source.SourceRef
|
||||
}
|
||||
return refs
|
||||
}
|
||||
|
||||
func locationEvidence(value dnd.LocationList) []source.SourceRef {
|
||||
var refs []source.SourceRef
|
||||
for _, location := range value.Locations {
|
||||
refs = append(refs, location.SourceRefs...)
|
||||
}
|
||||
return append([]source.SourceRef(nil), refs...)
|
||||
}
|
||||
|
||||
func locationOccurrenceEvidence(value dnd.LocationOccurrenceList) []source.SourceRef {
|
||||
var refs []source.SourceRef
|
||||
for _, occurrence := range value.Occurrences {
|
||||
refs = append(refs, occurrence.SourceRefs...)
|
||||
}
|
||||
return append([]source.SourceRef(nil), refs...)
|
||||
}
|
||||
|
||||
@@ -152,6 +152,48 @@ func appendSceneDescriptionLists(values []dnd.SceneDescriptionList) (dnd.SceneDe
|
||||
return combined, nil
|
||||
}
|
||||
|
||||
func appendLocationLists(values []dnd.LocationList) (dnd.LocationList, error) {
|
||||
count := 0
|
||||
present := false
|
||||
for _, value := range values {
|
||||
if value.Locations != nil {
|
||||
present = true
|
||||
}
|
||||
count += len(value.Locations)
|
||||
}
|
||||
if !present {
|
||||
return dnd.LocationList{}, nil
|
||||
}
|
||||
combined := dnd.LocationList{Locations: make([]dnd.Location, 0, count)}
|
||||
for _, value := range values {
|
||||
for _, location := range value.Locations {
|
||||
combined.Locations = append(combined.Locations, cloneLocation(location))
|
||||
}
|
||||
}
|
||||
return combined, nil
|
||||
}
|
||||
|
||||
func appendLocationOccurrenceLists(values []dnd.LocationOccurrenceList) (dnd.LocationOccurrenceList, error) {
|
||||
count := 0
|
||||
present := false
|
||||
for _, value := range values {
|
||||
if value.Occurrences != nil {
|
||||
present = true
|
||||
}
|
||||
count += len(value.Occurrences)
|
||||
}
|
||||
if !present {
|
||||
return dnd.LocationOccurrenceList{}, nil
|
||||
}
|
||||
combined := dnd.LocationOccurrenceList{Occurrences: make([]dnd.LocationOccurrence, 0, count)}
|
||||
for _, value := range values {
|
||||
for _, occurrence := range value.Occurrences {
|
||||
combined.Occurrences = append(combined.Occurrences, cloneLocationOccurrence(occurrence))
|
||||
}
|
||||
}
|
||||
return combined, nil
|
||||
}
|
||||
|
||||
func cloneCombatTurn(value dnd.CombatTurn) dnd.CombatTurn {
|
||||
clone := value
|
||||
clone.SourceRefs = cloneSourceRefs(value.SourceRefs)
|
||||
@@ -192,6 +234,18 @@ func cloneNPCInteraction(value dnd.NPCInteraction) dnd.NPCInteraction {
|
||||
return clone
|
||||
}
|
||||
|
||||
func cloneLocation(value dnd.Location) dnd.Location {
|
||||
clone := value
|
||||
clone.SourceRefs = cloneSourceRefs(value.SourceRefs)
|
||||
return clone
|
||||
}
|
||||
|
||||
func cloneLocationOccurrence(value dnd.LocationOccurrence) dnd.LocationOccurrence {
|
||||
clone := value
|
||||
clone.SourceRefs = cloneSourceRefs(value.SourceRefs)
|
||||
return clone
|
||||
}
|
||||
|
||||
func cloneSourceRefs(refs []source.SourceRef) []source.SourceRef {
|
||||
return slices.Clone(refs)
|
||||
}
|
||||
|
||||
@@ -8,6 +8,8 @@ import (
|
||||
combatcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/combatturns"
|
||||
enemyeventcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/enemyevents"
|
||||
itemeventcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/itemevents"
|
||||
locationoccurrencecodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locationoccurrences"
|
||||
locationcodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/locations"
|
||||
interactioncodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcinteractions"
|
||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||
scenedescriptioncodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/scenedescriptions"
|
||||
@@ -15,6 +17,8 @@ import (
|
||||
combatextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
|
||||
enemyeventextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/enemyevents"
|
||||
itemeventextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/itemevents"
|
||||
locationoccurrenceextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locationoccurrences"
|
||||
locationextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locations"
|
||||
interactionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
|
||||
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
scenedescriptionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
||||
@@ -22,10 +26,13 @@ import (
|
||||
combatnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/combatturns"
|
||||
enemyeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/enemyevents"
|
||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||
locationoccurrencenormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locationoccurrences"
|
||||
locationnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locations"
|
||||
interactionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcinteractions"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
scenedescriptionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/scenedescriptions"
|
||||
spellnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/spells"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/entityreconcile"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/merge/appendorder"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/generic/normalize/noop"
|
||||
)
|
||||
@@ -42,6 +49,10 @@ func registerModules(registries pipeline.Registries) error {
|
||||
{name: "scene descriptions codec", register: func() error {
|
||||
return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, scenedescriptioncodec.New())
|
||||
}},
|
||||
{name: "locations codec", register: func() error { return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, locationcodec.New()) }},
|
||||
{name: "location occurrences codec", register: func() error {
|
||||
return pipeline.RegisterArtifactCodec(registries.ArtifactCodecs, locationoccurrencecodec.New())
|
||||
}},
|
||||
{name: "scenes chunker", register: func() error { return scenes.Register(registries.Chunkers) }},
|
||||
{name: "spells extractor", register: func() error { return spellextract.Register(registries.Extractors) }},
|
||||
{name: "npcs extractor", register: func() error { return npcextract.Register(registries.Extractors) }},
|
||||
@@ -50,6 +61,8 @@ func registerModules(registries pipeline.Registries) error {
|
||||
{name: "item events extractor", register: func() error { return itemeventextract.Register(registries.Extractors) }},
|
||||
{name: "npc interactions extractor", register: func() error { return interactionextract.Register(registries.Extractors) }},
|
||||
{name: "scene descriptions extractor", register: func() error { return scenedescriptionextract.Register(registries.Extractors) }},
|
||||
{name: "locations extractor", register: func() error { return locationextract.Register(registries.Extractors) }},
|
||||
{name: "location occurrences extractor", register: func() error { return locationoccurrenceextract.Register(registries.Extractors) }},
|
||||
{name: "spell-list appendorder merger", register: func() error {
|
||||
return appendorder.RegisterTyped(registries.Mergers, dnd.SpellListKind, appendSpellLists)
|
||||
}},
|
||||
@@ -71,6 +84,12 @@ func registerModules(registries pipeline.Registries) error {
|
||||
{name: "scene-description-list appendorder merger", register: func() error {
|
||||
return appendorder.RegisterTyped(registries.Mergers, dnd.SceneDescriptionListKind, appendSceneDescriptionLists)
|
||||
}},
|
||||
{name: "location-list appendorder merger", register: func() error {
|
||||
return appendorder.RegisterTyped(registries.Mergers, dnd.LocationListKind, appendLocationLists)
|
||||
}},
|
||||
{name: "location-occurrence-list appendorder merger", register: func() error {
|
||||
return appendorder.RegisterTyped(registries.Mergers, dnd.LocationOccurrenceListKind, appendLocationOccurrenceLists)
|
||||
}},
|
||||
{name: "spells normalizer", register: func() error { return spellnormalize.Register(registries.Normalizers) }},
|
||||
{name: "npcs normalizer", register: func() error { return npcnormalize.Register(registries.Normalizers) }},
|
||||
{name: "combat turns normalizer", register: func() error { return combatnormalize.Register(registries.Normalizers) }},
|
||||
@@ -78,6 +97,8 @@ func registerModules(registries pipeline.Registries) error {
|
||||
{name: "item events normalizer", register: func() error { return itemeventnormalize.Register(registries.Normalizers) }},
|
||||
{name: "npc interactions normalizer", register: func() error { return interactionnormalize.Register(registries.Normalizers) }},
|
||||
{name: "scene descriptions normalizer", register: func() error { return scenedescriptionnormalize.Register(registries.Normalizers) }},
|
||||
{name: "locations normalizer", register: func() error { return locationnormalize.Register(registries.Normalizers) }},
|
||||
{name: "location occurrences normalizer", register: func() error { return locationoccurrencenormalize.Register(registries.Normalizers) }},
|
||||
{name: "spell-list noop normalizer", register: func() error {
|
||||
return noop.RegisterTyped[dnd.SpellList](registries.Normalizers, dnd.SpellListKind)
|
||||
}},
|
||||
@@ -99,11 +120,18 @@ func registerModules(registries pipeline.Registries) error {
|
||||
{name: "scene-description-list noop normalizer", register: func() error {
|
||||
return noop.RegisterTyped[dnd.SceneDescriptionList](registries.Normalizers, dnd.SceneDescriptionListKind)
|
||||
}},
|
||||
{name: "location-list noop normalizer", register: func() error {
|
||||
return noop.RegisterTyped[dnd.LocationList](registries.Normalizers, dnd.LocationListKind)
|
||||
}},
|
||||
{name: "location-occurrence-list noop normalizer", register: func() error {
|
||||
return noop.RegisterTyped[dnd.LocationOccurrenceList](registries.Normalizers, dnd.LocationOccurrenceListKind)
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
func registerPromptAssets(assets *llm.AssetRegistry) error {
|
||||
return runRegistrations([]registration{
|
||||
{name: "entity reconciliation schema assets", register: func() error { return entityreconcile.RegisterSchemaAssets(assets) }},
|
||||
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
|
||||
{name: "spells prompt assets", register: func() error { return spellextract.RegisterPromptAssets(assets) }},
|
||||
{name: "npcs prompt assets", register: func() error { return npcextract.RegisterPromptAssets(assets) }},
|
||||
@@ -113,5 +141,8 @@ func registerPromptAssets(assets *llm.AssetRegistry) error {
|
||||
{name: "item events prompt assets", register: func() error { return itemeventextract.RegisterPromptAssets(assets) }},
|
||||
{name: "npc interactions prompt assets", register: func() error { return interactionextract.RegisterPromptAssets(assets) }},
|
||||
{name: "scene descriptions prompt assets", register: func() error { return scenedescriptionextract.RegisterPromptAssets(assets) }},
|
||||
{name: "locations prompt assets", register: func() error { return locationextract.RegisterPromptAssets(assets) }},
|
||||
{name: "location normalization prompt assets", register: func() error { return locationnormalize.RegisterPromptAssets(assets) }},
|
||||
{name: "location occurrences prompt assets", register: func() error { return locationoccurrenceextract.RegisterPromptAssets(assets) }},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -15,6 +15,8 @@ import (
|
||||
combatextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/combatturns"
|
||||
enemyeventextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/enemyevents"
|
||||
itemeventextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/itemevents"
|
||||
locationoccurrenceextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locationoccurrences"
|
||||
locationextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/locations"
|
||||
interactionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
|
||||
npcextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
scenedescriptionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
||||
@@ -22,6 +24,8 @@ import (
|
||||
combatnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/combatturns"
|
||||
enemyeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/enemyevents"
|
||||
itemeventnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/itemevents"
|
||||
locationoccurrencenormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locationoccurrences"
|
||||
locationnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/locations"
|
||||
interactionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcinteractions"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
scenedescriptionnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/scenedescriptions"
|
||||
@@ -51,6 +55,9 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
"dnd.npc_interactions/dnd.npc_interactions.yaml",
|
||||
"dnd.scene_descriptions/dnd.scene_descriptions.yaml",
|
||||
"dnd.npcs.normalize/dnd.npcs.normalize.yaml",
|
||||
"dnd.locations/dnd.locations.yaml",
|
||||
"dnd.locations.normalize/dnd.locations.normalize.yaml",
|
||||
"dnd.location_occurrences/dnd.location_occurrences.yaml",
|
||||
} {
|
||||
content, err := fs.ReadFile(promptFS, name)
|
||||
if err != nil {
|
||||
@@ -71,16 +78,16 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("SchemaFS() error = %v", err)
|
||||
}
|
||||
if _, err := fs.ReadFile(schemaFS, "dnd_npcs_normalize_llm.v1.json"); err != nil {
|
||||
t.Fatalf("normalization schema asset = %v, want registered private schema", err)
|
||||
if _, err := fs.ReadFile(schemaFS, "dnd_entity_reconcile_llm.v1.json"); err != nil {
|
||||
t.Fatalf("entity reconciliation schema asset = %v, want registered shared schema", err)
|
||||
}
|
||||
assertContainsKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
|
||||
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells", npcextract.Key, combatextract.Key, enemyeventextract.Key, itemeventextract.Key, interactionextract.Key, scenedescriptionextract.Key})
|
||||
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, npcnormalize.Key, combatnormalize.Key, enemyeventnormalize.Key, itemeventnormalize.Key, interactionnormalize.Key, scenedescriptionnormalize.Key, pipeline.DefaultNormalizeModule})
|
||||
assertContainsArtifactKinds(t, registries.ArtifactCodecs.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind})
|
||||
assertContainsArtifactKinds(t, registries.ArtifactEvidence.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind})
|
||||
assertContainsArtifactKinds(t, registries.Mergers.RegisteredArtifactKinds(pipeline.DefaultMergeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind})
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(pipeline.DefaultNormalizeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind})
|
||||
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells", npcextract.Key, combatextract.Key, enemyeventextract.Key, itemeventextract.Key, interactionextract.Key, scenedescriptionextract.Key, locationextract.Key, locationoccurrenceextract.Key})
|
||||
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, npcnormalize.Key, combatnormalize.Key, enemyeventnormalize.Key, itemeventnormalize.Key, interactionnormalize.Key, scenedescriptionnormalize.Key, locationnormalize.Key, locationoccurrencenormalize.Key, pipeline.DefaultNormalizeModule})
|
||||
assertContainsArtifactKinds(t, registries.ArtifactCodecs.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind, dnd.LocationListKind, dnd.LocationOccurrenceListKind})
|
||||
assertContainsArtifactKinds(t, registries.ArtifactEvidence.RegisteredKinds(), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind, dnd.LocationListKind, dnd.LocationOccurrenceListKind})
|
||||
assertContainsArtifactKinds(t, registries.Mergers.RegisteredArtifactKinds(pipeline.DefaultMergeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind, dnd.LocationListKind, dnd.LocationOccurrenceListKind})
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(pipeline.DefaultNormalizeModule), []contracts.ArtifactKind{dnd.SpellListKind, dnd.NPCListKind, dnd.CombatTurnListKind, dnd.EnemyEventListKind, dnd.ItemEventListKind, dnd.NPCInteractionListKind, dnd.SceneDescriptionListKind, dnd.LocationListKind, dnd.LocationOccurrenceListKind})
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(npcnormalize.Key), []contracts.ArtifactKind{dnd.NPCListKind})
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(combatnormalize.Key), []contracts.ArtifactKind{dnd.CombatTurnListKind})
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(enemyeventnormalize.Key), []contracts.ArtifactKind{dnd.EnemyEventListKind})
|
||||
@@ -88,6 +95,8 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(interactionnormalize.Key), []contracts.ArtifactKind{dnd.NPCInteractionListKind})
|
||||
assertContainsArtifactKinds(t, registries.Normalizers.RegisteredArtifactKinds(scenedescriptionnormalize.Key), []contracts.ArtifactKind{dnd.SceneDescriptionListKind})
|
||||
assertContainsKeys(t, "validators", registries.Validators.RegisteredKeys(), []string{
|
||||
"extract/dnd/locations/shape", "normalize/dnd/locations/identity", "extract/dnd/locations/source_refs", "extract/dnd/locations/source_relatedness",
|
||||
"extract/dnd/location-occurrences/shape", "extract/dnd/location-occurrences/registry", "normalize/dnd/location-occurrences/invariants", "extract/dnd/location-occurrences/source_refs", "extract/dnd/location-occurrences/source_relatedness",
|
||||
"extract/dnd/npcs/shape",
|
||||
"extract/dnd/npcs/source_refs",
|
||||
"extract/dnd/npcs/source_relatedness",
|
||||
@@ -121,6 +130,22 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
"generic/always_accept",
|
||||
"generic/always_reject",
|
||||
})
|
||||
locationExtractChain := []pipeline.ModuleBinding{pipeline.Binding("generic/valid_json"), pipeline.Binding("extract/dnd/locations/shape"), pipeline.Binding("extract/dnd/locations/source_refs"), pipeline.Binding("generic/valid_json_schema"), pipeline.Binding("extract/dnd/locations/source_relatedness")}
|
||||
locationNormalizeChain := []pipeline.ModuleBinding{pipeline.Binding("generic/valid_json"), pipeline.Binding("extract/dnd/locations/shape"), pipeline.Binding("normalize/dnd/locations/identity"), pipeline.Binding("extract/dnd/locations/source_refs"), pipeline.Binding("generic/valid_json_schema"), pipeline.Binding("extract/dnd/locations/source_relatedness")}
|
||||
occurrenceExtractChain := []pipeline.ModuleBinding{pipeline.Binding("generic/valid_json"), pipeline.Binding("extract/dnd/location-occurrences/shape"), pipeline.Binding("extract/dnd/location-occurrences/registry"), pipeline.Binding("extract/dnd/location-occurrences/source_refs"), pipeline.Binding("generic/valid_json_schema"), pipeline.Binding("extract/dnd/location-occurrences/source_relatedness")}
|
||||
occurrenceNormalizeChain := []pipeline.ModuleBinding{pipeline.Binding("generic/valid_json"), pipeline.Binding("extract/dnd/location-occurrences/shape"), pipeline.Binding("extract/dnd/location-occurrences/registry"), pipeline.Binding("normalize/dnd/location-occurrences/invariants"), pipeline.Binding("extract/dnd/location-occurrences/source_refs"), pipeline.Binding("generic/valid_json_schema"), pipeline.Binding("extract/dnd/location-occurrences/source_relatedness")}
|
||||
for _, test := range []struct {
|
||||
stage pipeline.ModuleStage
|
||||
key string
|
||||
want []pipeline.ModuleBinding
|
||||
}{
|
||||
{pipeline.StageExtract, locationextract.Key, locationExtractChain}, {pipeline.StageNormalize, locationnormalize.Key, locationNormalizeChain},
|
||||
{pipeline.StageExtract, locationoccurrenceextract.Key, occurrenceExtractChain}, {pipeline.StageNormalize, locationoccurrencenormalize.Key, occurrenceNormalizeChain},
|
||||
} {
|
||||
if got := registries.ValidatorChains.Validators(test.stage, test.key); !reflect.DeepEqual(got, test.want) {
|
||||
t.Fatalf("validator chain for %s/%s = %#v, want %#v", test.stage, test.key, got, test.want)
|
||||
}
|
||||
}
|
||||
wantChain := []pipeline.ModuleBinding{
|
||||
pipeline.Binding("generic/valid_json"),
|
||||
pipeline.Binding("extract/dnd/spells/shape"),
|
||||
@@ -289,6 +314,12 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
"dnd.npcs/sharedassets/common-dnd-system.md",
|
||||
"dnd.npcs/sharedassets/common-dnd-transcript.md",
|
||||
"dnd.npcs/task.md",
|
||||
"dnd.npcs.normalize/candidates.md",
|
||||
"dnd.npcs.normalize/dnd.npcs.normalize.yaml",
|
||||
"dnd.npcs.normalize/sharedassets/common-dnd-entity-reconciliation.md",
|
||||
"dnd.npcs.normalize/sharedassets/common-dnd-system.md",
|
||||
"dnd.npcs.normalize/sharedassets/common-dnd-transcript.md",
|
||||
"dnd.npcs.normalize/task.md",
|
||||
"dnd.combat_turns/dnd.combat_turns.yaml",
|
||||
"dnd.combat_turns/instructions.md",
|
||||
"dnd.combat_turns/sharedassets/common-dnd-references.md",
|
||||
@@ -328,6 +359,8 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
"dnd_item_events_llm.v1.json",
|
||||
"dnd_npc_interactions_llm.v1.json",
|
||||
"dnd_scene_descriptions_llm.v1.json",
|
||||
"dnd_locations_llm.v1.json",
|
||||
"dnd_location_occurrences_llm.v1.json",
|
||||
})
|
||||
if spec, ok := registries.Chunkers.Spec("dnd/scenes"); !ok || spec.Key != "dnd/scenes" {
|
||||
t.Fatalf("scene chunker spec = %#v, present = %t; want family-owned spec", spec, ok)
|
||||
@@ -378,6 +411,30 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
if !extractOK || interactionExtractSpec.ArtifactKind != dnd.NPCInteractionListKind || !normalizeOK || interactionNormalizeSpec.ArtifactKind != dnd.NPCInteractionListKind || interactionNormalizeSpec.Stage != pipeline.StageNormalize {
|
||||
t.Fatalf("NPC interaction specs = %#v / %#v, present = %t / %t", interactionExtractSpec, interactionNormalizeSpec, extractOK, normalizeOK)
|
||||
}
|
||||
locationExtractSpec, locationExtractOK := registries.Extractors.Spec(locationextract.Key)
|
||||
locationNormalizeSpec, locationNormalizeOK := registries.Normalizers.Spec(locationnormalize.Key)
|
||||
if !locationExtractOK || locationExtractSpec.ArtifactKind != dnd.LocationListKind || locationExtractSpec.ExecutionClass != contracts.ExecutionClassLLMBacked || !locationNormalizeOK || locationNormalizeSpec.ArtifactKind != dnd.LocationListKind || locationNormalizeSpec.ExecutionClass != contracts.ExecutionClassLLMBacked {
|
||||
t.Fatalf("location specs = %#v / %#v", locationExtractSpec, locationNormalizeSpec)
|
||||
}
|
||||
occurrenceExtractSpec, occurrenceExtractOK := registries.Extractors.Spec(locationoccurrenceextract.Key)
|
||||
occurrenceNormalizeSpec, occurrenceNormalizeOK := registries.Normalizers.Spec(locationoccurrencenormalize.Key)
|
||||
if !occurrenceExtractOK || occurrenceExtractSpec.ArtifactKind != dnd.LocationOccurrenceListKind || occurrenceExtractSpec.ExecutionClass != contracts.ExecutionClassLLMBacked || !occurrenceNormalizeOK || occurrenceNormalizeSpec.ArtifactKind != dnd.LocationOccurrenceListKind || occurrenceNormalizeSpec.ExecutionClass != contracts.ExecutionClassDeterministic {
|
||||
t.Fatalf("location occurrence specs = %#v / %#v", occurrenceExtractSpec, occurrenceNormalizeSpec)
|
||||
}
|
||||
locationRegistrySlot := referenceSlot(occurrenceExtractSpec.ReferenceSlots, "locations")
|
||||
occurrenceNormalizeRegistrySlot := referenceSlot(occurrenceNormalizeSpec.ReferenceSlots, "locations")
|
||||
if len(occurrenceExtractSpec.ReferenceSlots) != 5 || len(occurrenceNormalizeSpec.ReferenceSlots) != 1 {
|
||||
t.Fatalf("location occurrence reference slots = %#v / %#v, want extractor campaign context and normalizer registry only", occurrenceExtractSpec.ReferenceSlots, occurrenceNormalizeSpec.ReferenceSlots)
|
||||
}
|
||||
if !locationRegistrySlot.Required || !reflect.DeepEqual(locationRegistrySlot.AcceptedMediaTypes, []string{"application/json"}) || !reflect.DeepEqual(locationRegistrySlot.AcceptedArtifactKinds, []contracts.ArtifactKind{dnd.LocationListKind}) || locationRegistrySlot.MaxBytes != 1048576 || !sameReferenceSlotContract(locationRegistrySlot, occurrenceNormalizeRegistrySlot) {
|
||||
t.Fatalf("location registry slots disagree: %#v / %#v", occurrenceExtractSpec.ReferenceSlots, occurrenceNormalizeSpec.ReferenceSlots)
|
||||
}
|
||||
for _, name := range []string{"party", "roster", "players", "glossary"} {
|
||||
slot := referenceSlot(occurrenceExtractSpec.ReferenceSlots, name)
|
||||
if slot.Name != name || slot.Required || len(slot.AcceptedArtifactKinds) != 0 {
|
||||
t.Fatalf("location occurrence extractor campaign slot %q = %#v, want optional text context", name, slot)
|
||||
}
|
||||
}
|
||||
sceneExtractSpec, sceneExtractOK := registries.Extractors.Spec(scenedescriptionextract.Key)
|
||||
sceneNormalizeSpec, sceneNormalizeOK := registries.Normalizers.Spec(scenedescriptionnormalize.Key)
|
||||
if !sceneExtractOK || sceneExtractSpec.ArtifactKind != dnd.SceneDescriptionListKind || !sceneNormalizeOK || sceneNormalizeSpec.ArtifactKind != dnd.SceneDescriptionListKind || sceneNormalizeSpec.Stage != pipeline.StageNormalize {
|
||||
@@ -388,7 +445,7 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
}
|
||||
extractRegistrySlot := referenceSlot(interactionExtractSpec.ReferenceSlots, "npcs")
|
||||
normalizeRegistrySlot := referenceSlot(interactionNormalizeSpec.ReferenceSlots, "npcs")
|
||||
if !extractRegistrySlot.Required || !reflect.DeepEqual(extractRegistrySlot.AcceptedArtifactKinds, []contracts.ArtifactKind{dnd.NPCListKind}) || !reflect.DeepEqual(extractRegistrySlot, normalizeRegistrySlot) {
|
||||
if !extractRegistrySlot.Required || !reflect.DeepEqual(extractRegistrySlot.AcceptedArtifactKinds, []contracts.ArtifactKind{dnd.NPCListKind}) || !sameReferenceSlotContract(extractRegistrySlot, normalizeRegistrySlot) {
|
||||
t.Fatalf("NPC interaction registry slots disagree: %#v / %#v", interactionExtractSpec.ReferenceSlots, interactionNormalizeSpec.ReferenceSlots)
|
||||
}
|
||||
}
|
||||
@@ -422,6 +479,12 @@ func TestEvidenceProjectorsPreserveDirectReferencesWithIndependentStorage(t *tes
|
||||
{name: "scene descriptions", project: func() []source.SourceRef {
|
||||
return sceneDescriptionEvidence(dnd.SceneDescriptionList{Scenes: []dnd.SceneDescription{{SourceRef: first}, {SourceRef: second}}})
|
||||
}, want: []source.SourceRef{first, second}},
|
||||
{name: "locations", project: func() []source.SourceRef {
|
||||
return locationEvidence(dnd.LocationList{Locations: []dnd.Location{{SourceRefs: []source.SourceRef{first, second}}}})
|
||||
}, want: []source.SourceRef{first, second}},
|
||||
{name: "location occurrences", project: func() []source.SourceRef {
|
||||
return locationOccurrenceEvidence(dnd.LocationOccurrenceList{Occurrences: []dnd.LocationOccurrence{{SourceRefs: []source.SourceRef{first, second}}}})
|
||||
}, want: []source.SourceRef{first, second}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got := test.project()
|
||||
@@ -445,6 +508,12 @@ func referenceSlot(slots []contracts.ReferenceSlot, name string) contracts.Refer
|
||||
return contracts.ReferenceSlot{}
|
||||
}
|
||||
|
||||
func sameReferenceSlotContract(first, second contracts.ReferenceSlot) bool {
|
||||
first.Description = ""
|
||||
second.Description = ""
|
||||
return reflect.DeepEqual(first, second)
|
||||
}
|
||||
|
||||
func TestAppendNPCListsPreservesOrderAndArrayPresence(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
@@ -476,6 +545,44 @@ func TestAppendNPCListsPreservesOrderAndArrayPresence(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestAppendLocationListsPreserveOrderPresenceAndOwnership(t *testing.T) {
|
||||
refs := []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}
|
||||
input := []dnd.LocationList{{Locations: []dnd.Location{{ID: "one", Name: "First", SourceRefs: refs}}}, {Locations: []dnd.Location{{ID: "two", Name: "Second", SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}}}}}
|
||||
got, err := appendLocationLists(input)
|
||||
if err != nil || !reflect.DeepEqual([]string{got.Locations[0].Name, got.Locations[1].Name}, []string{"First", "Second"}) {
|
||||
t.Fatalf("appendLocationLists() = %#v, %v", got, err)
|
||||
}
|
||||
got.Locations[0].SourceRefs[0].StartUnitID = 99
|
||||
if input[0].Locations[0].SourceRefs[0].StartUnitID != 1 {
|
||||
t.Fatal("merged locations share source-reference storage")
|
||||
}
|
||||
for _, values := range [][]dnd.LocationList{nil, {{}, {}}} {
|
||||
result, err := appendLocationLists(values)
|
||||
if err != nil || result.Locations != nil {
|
||||
t.Fatalf("nil-only merge = %#v, %v", result, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAppendLocationOccurrenceListsPreserveOrderPresenceAndOwnership(t *testing.T) {
|
||||
refs := []source.SourceRef{{SourceID: "session", StartUnitID: 1, EndUnitID: 1}}
|
||||
input := []dnd.LocationOccurrenceList{{Occurrences: []dnd.LocationOccurrence{{LocationID: "one", Name: "First", Kind: dnd.LocationOccurrenceKindVisited, SourceRefs: refs}}}, {Occurrences: []dnd.LocationOccurrence{{LocationID: "two", Name: "Second", Kind: dnd.LocationOccurrenceKindMentioned, SourceRefs: []source.SourceRef{{SourceID: "session", StartUnitID: 2, EndUnitID: 2}}}}}}
|
||||
got, err := appendLocationOccurrenceLists(input)
|
||||
if err != nil || !reflect.DeepEqual([]string{got.Occurrences[0].Name, got.Occurrences[1].Name}, []string{"First", "Second"}) {
|
||||
t.Fatalf("appendLocationOccurrenceLists() = %#v, %v", got, err)
|
||||
}
|
||||
got.Occurrences[0].SourceRefs[0].StartUnitID = 99
|
||||
if input[0].Occurrences[0].SourceRefs[0].StartUnitID != 1 {
|
||||
t.Fatal("merged location occurrences share source-reference storage")
|
||||
}
|
||||
for _, values := range [][]dnd.LocationOccurrenceList{nil, {{}, {}}} {
|
||||
result, err := appendLocationOccurrenceLists(values)
|
||||
if err != nil || result.Occurrences != nil {
|
||||
t.Fatalf("nil-only merge = %#v, %v", result, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestAppendSpellListsPreservesOrderPresenceAndOwnership(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -16,6 +16,15 @@ import (
|
||||
itemeventshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemevents/shape"
|
||||
itemeventrefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemevents/source_refs"
|
||||
itemeventrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/itemevents/source_relatedness"
|
||||
occurrenceinvariants "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/invariants"
|
||||
occurrenceregistry "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/registry"
|
||||
occurrenceshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/shape"
|
||||
occurrencerefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/source_refs"
|
||||
occurrencerelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locationoccurrences/source_relatedness"
|
||||
locationidentity "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/identity"
|
||||
locationshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/shape"
|
||||
locationrefs "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/source_refs"
|
||||
locationrelatedness "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/locations/source_relatedness"
|
||||
interactioninvariants "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcinteractions/invariants"
|
||||
interactionregistry "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcinteractions/registry"
|
||||
interactionshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcinteractions/shape"
|
||||
@@ -69,6 +78,15 @@ func registerValidators(registries pipeline.Registries) error {
|
||||
{name: "scene description source references validator", register: func() error { return scenerefs.Register(registries.Validators) }},
|
||||
{name: "scene description source relatedness validator", register: func() error { return scenerelatedness.Register(registries.Validators) }},
|
||||
{name: "scene description normalized invariants validator", register: func() error { return sceneinvariants.Register(registries.Validators) }},
|
||||
{name: "location shape validator", register: func() error { return locationshape.Register(registries.Validators) }},
|
||||
{name: "location identity validator", register: func() error { return locationidentity.Register(registries.Validators) }},
|
||||
{name: "location source references validator", register: func() error { return locationrefs.Register(registries.Validators) }},
|
||||
{name: "location source relatedness validator", register: func() error { return locationrelatedness.Register(registries.Validators) }},
|
||||
{name: "location occurrence shape validator", register: func() error { return occurrenceshape.Register(registries.Validators) }},
|
||||
{name: "location occurrence registry validator", register: func() error { return occurrenceregistry.Register(registries.Validators) }},
|
||||
{name: "location occurrence normalized invariants validator", register: func() error { return occurrenceinvariants.Register(registries.Validators) }},
|
||||
{name: "location occurrence source references validator", register: func() error { return occurrencerefs.Register(registries.Validators) }},
|
||||
{name: "location occurrence source relatedness validator", register: func() error { return occurrencerelatedness.Register(registries.Validators) }},
|
||||
{name: "spell-list always accept validator", register: func() error {
|
||||
return alwaysaccept.RegisterTyped[dnd.SpellList](registries.Validators, dnd.SpellListKind)
|
||||
}},
|
||||
@@ -111,5 +129,17 @@ func registerValidators(registries pipeline.Registries) error {
|
||||
{name: "scene-description-list always reject validator", register: func() error {
|
||||
return alwaysreject.RegisterTyped[dnd.SceneDescriptionList](registries.Validators, dnd.SceneDescriptionListKind)
|
||||
}},
|
||||
{name: "location-list always accept validator", register: func() error {
|
||||
return alwaysaccept.RegisterTyped[dnd.LocationList](registries.Validators, dnd.LocationListKind)
|
||||
}},
|
||||
{name: "location-list always reject validator", register: func() error {
|
||||
return alwaysreject.RegisterTyped[dnd.LocationList](registries.Validators, dnd.LocationListKind)
|
||||
}},
|
||||
{name: "location-occurrence-list always accept validator", register: func() error {
|
||||
return alwaysaccept.RegisterTyped[dnd.LocationOccurrenceList](registries.Validators, dnd.LocationOccurrenceListKind)
|
||||
}},
|
||||
{name: "location-occurrence-list always reject validator", register: func() error {
|
||||
return alwaysreject.RegisterTyped[dnd.LocationOccurrenceList](registries.Validators, dnd.LocationOccurrenceListKind)
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -29,6 +29,7 @@ var sharedPromptPaths = map[string]string{
|
||||
"common-dnd-transcript.md": "assets/prompts/common-dnd-transcript.md",
|
||||
"common-dnd-references.md": "assets/prompts/common-dnd-references.md",
|
||||
"common-dnd-npcs.md": "assets/prompts/common-dnd-npcs.md",
|
||||
"common-dnd-entity-reconciliation.md": "assets/prompts/common-dnd-entity-reconciliation.md",
|
||||
}
|
||||
|
||||
func (manifest PromptAssetManifest) PromptFS(moduleFS fs.FS) (fs.FS, error) {
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
Identify only well-supported duplicate groups among the supplied candidates.
|
||||
|
||||
Candidate keys are opaque identifiers. Copy each selected key exactly. A group
|
||||
must contain at least two supplied keys, and its `canonical` key must be one of
|
||||
its members. Do not create keys, records, names, source references, evidence,
|
||||
or replacement values. Omit any uncertain or unsafe group.
|
||||
6
internal/modules/dnd/shared/entityreconcile/assets.go
Normal file
6
internal/modules/dnd/shared/entityreconcile/assets.go
Normal file
@@ -0,0 +1,6 @@
|
||||
package entityreconcile
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/schemas/dnd_entity_reconcile_llm.v1.json
|
||||
var embeddedAssets embed.FS
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.npcs.normalize.llm",
|
||||
"$id": "notarius.dnd.entity_reconcile.llm",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["duplicate_groups"],
|
||||
@@ -10,13 +10,13 @@
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["members", "canonical_name"],
|
||||
"required": ["members", "canonical"],
|
||||
"properties": {
|
||||
"members": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
},
|
||||
"canonical_name": {"type": "string"}
|
||||
"canonical": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
226
internal/modules/dnd/shared/entityreconcile/context.go
Normal file
226
internal/modules/dnd/shared/entityreconcile/context.go
Normal file
@@ -0,0 +1,226 @@
|
||||
// Package entityreconcile provides safe, D&D-specific duplicate proposal
|
||||
// materials shared by entity normalizers.
|
||||
package entityreconcile
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
const candidateKeyFormat = "candidate-%06d"
|
||||
|
||||
// Candidate is one domain-neutral entity candidate supplied by a normalizer.
|
||||
// BuildContext never retains or mutates its source references.
|
||||
type Candidate struct {
|
||||
Name string
|
||||
SourceRefs []source.SourceRef
|
||||
}
|
||||
|
||||
// Materials contains owned prompt inputs and opaque candidate-key mappings.
|
||||
type Materials struct {
|
||||
Candidates contracts.LLMInputMaterial
|
||||
Transcript contracts.LLMInputMaterial
|
||||
|
||||
candidateKeys []string
|
||||
eligible map[string]struct{}
|
||||
}
|
||||
|
||||
// CandidateKeys returns all deterministic keys in candidate input order.
|
||||
func (m Materials) CandidateKeys() []string {
|
||||
return append([]string(nil), m.candidateKeys...)
|
||||
}
|
||||
|
||||
// EligibleCandidateKeys returns only candidates whose evidence safely produced
|
||||
// transcript context, preserving candidate input order.
|
||||
func (m Materials) EligibleCandidateKeys() []string {
|
||||
keys := make([]string, 0, len(m.eligible))
|
||||
for _, key := range m.candidateKeys {
|
||||
if _, ok := m.eligible[key]; ok {
|
||||
keys = append(keys, key)
|
||||
}
|
||||
}
|
||||
return keys
|
||||
}
|
||||
|
||||
type candidateInput struct {
|
||||
Candidates []candidateView `json:"candidates"`
|
||||
}
|
||||
|
||||
type candidateView struct {
|
||||
Key string `json:"key"`
|
||||
Name string `json:"name"`
|
||||
SourceRefs []candidateSourceRef `json:"source_refs"`
|
||||
}
|
||||
|
||||
type candidateSourceRef struct {
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
}
|
||||
|
||||
type transcriptInput struct {
|
||||
Windows []transcriptWindow `json:"windows"`
|
||||
}
|
||||
|
||||
type transcriptWindow struct {
|
||||
Units []transcriptUnit `json:"units"`
|
||||
}
|
||||
|
||||
type transcriptUnit struct {
|
||||
ID int `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Text string `json:"text"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Cited bool `json:"cited"`
|
||||
}
|
||||
|
||||
type sourceInterval struct {
|
||||
start int
|
||||
end int
|
||||
}
|
||||
|
||||
// BuildContext constructs bounded, source-ordered prompt inputs. It returns
|
||||
// ready=false when fewer than two candidates have safe context.
|
||||
func BuildContext(doc *source.SourceDocument, candidates []Candidate, radius int) (Materials, bool, error) {
|
||||
if radius < 0 {
|
||||
return Materials{}, false, fmt.Errorf("build entity reconciliation context: radius must not be negative")
|
||||
}
|
||||
materials := Materials{
|
||||
candidateKeys: make([]string, len(candidates)),
|
||||
eligible: make(map[string]struct{}),
|
||||
}
|
||||
for index := range candidates {
|
||||
key := fmt.Sprintf(candidateKeyFormat, index+1)
|
||||
materials.candidateKeys[index] = key
|
||||
}
|
||||
if doc == nil {
|
||||
return materials, false, nil
|
||||
}
|
||||
|
||||
index := source.NewDocumentIndex(doc)
|
||||
views := make([]candidateView, 0, len(candidates))
|
||||
intervals := make([]sourceInterval, 0)
|
||||
cited := make([]bool, len(doc.Units))
|
||||
for candidateIndex, candidate := range candidates {
|
||||
references, candidateIntervals, valid := candidateReferences(index, candidate.SourceRefs)
|
||||
if !valid {
|
||||
continue
|
||||
}
|
||||
key := materials.candidateKeys[candidateIndex]
|
||||
materials.eligible[key] = struct{}{}
|
||||
views = append(views, candidateView{Key: key, Name: candidate.Name, SourceRefs: references})
|
||||
for _, interval := range candidateIntervals {
|
||||
for position := interval.start; position <= interval.end; position++ {
|
||||
cited[position] = true
|
||||
}
|
||||
intervals = append(intervals, sourceInterval{
|
||||
start: maxInt(0, interval.start-radius),
|
||||
end: minInt(len(doc.Units)-1, interval.end+radius),
|
||||
})
|
||||
}
|
||||
}
|
||||
if len(views) < 2 {
|
||||
return materials, false, nil
|
||||
}
|
||||
|
||||
windows, err := contextWindows(doc.Units, coalesceIntervals(intervals), cited)
|
||||
if err != nil {
|
||||
return Materials{}, false, fmt.Errorf("build entity reconciliation context: invalid source metadata")
|
||||
}
|
||||
candidateContent, err := json.Marshal(candidateInput{Candidates: views})
|
||||
if err != nil {
|
||||
return Materials{}, false, fmt.Errorf("build entity reconciliation context: invalid candidate material")
|
||||
}
|
||||
transcriptContent, err := json.Marshal(transcriptInput{Windows: windows})
|
||||
if err != nil {
|
||||
return Materials{}, false, fmt.Errorf("build entity reconciliation context: invalid transcript material")
|
||||
}
|
||||
materials.Candidates = newInputMaterial("candidates", candidateContent)
|
||||
materials.Transcript = newInputMaterial("transcript", transcriptContent)
|
||||
return materials, true, nil
|
||||
}
|
||||
|
||||
func candidateReferences(index source.DocumentIndex, refs []source.SourceRef) ([]candidateSourceRef, []sourceInterval, bool) {
|
||||
if len(refs) == 0 {
|
||||
return nil, nil, false
|
||||
}
|
||||
references := make([]candidateSourceRef, 0, len(refs))
|
||||
intervals := make([]sourceInterval, 0, len(refs))
|
||||
for _, ref := range refs {
|
||||
if err := index.ValidateRef(ref); err != nil {
|
||||
return nil, nil, false
|
||||
}
|
||||
start, _ := index.Position(ref.StartUnitID)
|
||||
end, _ := index.Position(ref.EndUnitID)
|
||||
references = append(references, candidateSourceRef{StartUnitID: ref.StartUnitID, EndUnitID: ref.EndUnitID})
|
||||
intervals = append(intervals, sourceInterval{start: start, end: end})
|
||||
}
|
||||
return references, intervals, true
|
||||
}
|
||||
|
||||
func coalesceIntervals(intervals []sourceInterval) []sourceInterval {
|
||||
if len(intervals) == 0 {
|
||||
return nil
|
||||
}
|
||||
ordered := append([]sourceInterval(nil), intervals...)
|
||||
sort.Slice(ordered, func(left, right int) bool {
|
||||
if ordered[left].start != ordered[right].start {
|
||||
return ordered[left].start < ordered[right].start
|
||||
}
|
||||
return ordered[left].end < ordered[right].end
|
||||
})
|
||||
coalesced := make([]sourceInterval, 0, len(ordered))
|
||||
for _, interval := range ordered {
|
||||
if len(coalesced) == 0 || interval.start > coalesced[len(coalesced)-1].end+1 {
|
||||
coalesced = append(coalesced, interval)
|
||||
continue
|
||||
}
|
||||
if interval.end > coalesced[len(coalesced)-1].end {
|
||||
coalesced[len(coalesced)-1].end = interval.end
|
||||
}
|
||||
}
|
||||
return coalesced
|
||||
}
|
||||
|
||||
func contextWindows(units []source.SourceUnit, intervals []sourceInterval, cited []bool) ([]transcriptWindow, error) {
|
||||
windows := make([]transcriptWindow, 0, len(intervals))
|
||||
for _, interval := range intervals {
|
||||
window := transcriptWindow{Units: make([]transcriptUnit, 0, interval.end-interval.start+1)}
|
||||
for position := interval.start; position <= interval.end; position++ {
|
||||
unit := units[position]
|
||||
metadata, err := source.CloneMetadata(unit.Metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
window.Units = append(window.Units, transcriptUnit{
|
||||
ID: unit.ID, Kind: unit.Kind, Text: unit.Text, Metadata: metadata, Cited: cited[position],
|
||||
})
|
||||
}
|
||||
windows = append(windows, window)
|
||||
}
|
||||
return windows, nil
|
||||
}
|
||||
|
||||
func newInputMaterial(name string, content []byte) contracts.LLMInputMaterial {
|
||||
digest := sha256.Sum256(content)
|
||||
return contracts.NewLLMInputMaterial(name, "application/json", content, "sha256:"+hex.EncodeToString(digest[:]), "")
|
||||
}
|
||||
|
||||
func minInt(left, right int) int {
|
||||
if left < right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
|
||||
func maxInt(left, right int) int {
|
||||
if left > right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
@@ -0,0 +1,275 @@
|
||||
package entityreconcile
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io/fs"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
)
|
||||
|
||||
func TestBuildContextUsesOpaqueKeysSourceOrderAndOwnedData(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
|
||||
{ID: 40, Kind: "narration", Text: "zero"},
|
||||
{ID: 10, Kind: "speech", Text: "one", Metadata: map[string]any{"speaker": map[string]any{"name": "Mira"}}},
|
||||
{ID: 70, Kind: "speech", Text: "two"},
|
||||
{ID: 20, Kind: "narration", Text: "three"},
|
||||
{ID: 90, Kind: "speech", Text: "four"},
|
||||
}}
|
||||
candidates := []Candidate{
|
||||
{Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 20}}},
|
||||
{Name: "The Tavern", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 90, EndUnitID: 90}}},
|
||||
{Name: "Broken", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 10}}},
|
||||
}
|
||||
before := cloneCandidates(candidates)
|
||||
|
||||
materials, ready, err := BuildContext(doc, candidates, 1)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("BuildContext() = %#v, %t, %v; want ready materials", materials, ready, err)
|
||||
}
|
||||
if !reflect.DeepEqual(candidates, before) {
|
||||
t.Fatalf("BuildContext() mutated candidates: %#v", candidates)
|
||||
}
|
||||
if got, want := materials.CandidateKeys(), []string{"candidate-000001", "candidate-000002", "candidate-000003"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("CandidateKeys() = %#v, want %#v", got, want)
|
||||
}
|
||||
if got, want := materials.EligibleCandidateKeys(), []string{"candidate-000001", "candidate-000002"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("EligibleCandidateKeys() = %#v, want %#v", got, want)
|
||||
}
|
||||
if !json.Valid(materials.Candidates.Content) || !json.Valid(materials.Transcript.Content) {
|
||||
t.Fatalf("prompt materials are not JSON: %#v", materials)
|
||||
}
|
||||
if strings.Contains(string(materials.Candidates.Content), doc.ID) {
|
||||
t.Fatalf("candidate material leaked source identity: %s", materials.Candidates.Content)
|
||||
}
|
||||
|
||||
var candidatePayload candidateInput
|
||||
if err := json.Unmarshal(materials.Candidates.Content, &candidatePayload); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(candidatePayload.Candidates) != 2 || candidatePayload.Candidates[0].Key != "candidate-000001" || candidatePayload.Candidates[1].Key != "candidate-000002" || candidatePayload.Candidates[0].Name != candidatePayload.Candidates[1].Name {
|
||||
t.Fatalf("candidate payload = %#v, want distinct opaque keys for equal names", candidatePayload)
|
||||
}
|
||||
if got := candidatePayload.Candidates[0].SourceRefs[0]; got != (candidateSourceRef{StartUnitID: 10, EndUnitID: 20}) {
|
||||
t.Fatalf("candidate reference = %#v", got)
|
||||
}
|
||||
|
||||
var transcript transcriptInput
|
||||
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 5 {
|
||||
t.Fatalf("windows = %#v, want one bounded coalesced window", transcript.Windows)
|
||||
}
|
||||
units := transcript.Windows[0].Units
|
||||
for index, wantID := range []int{40, 10, 70, 20, 90} {
|
||||
if units[index].ID != wantID {
|
||||
t.Fatalf("window unit %d = %d, want source-order %d", index, units[index].ID, wantID)
|
||||
}
|
||||
}
|
||||
if units[0].Cited || !units[1].Cited || !units[2].Cited || !units[3].Cited || !units[4].Cited {
|
||||
t.Fatalf("citation flags = %#v", units)
|
||||
}
|
||||
|
||||
windows, err := contextWindows(doc.Units, []sourceInterval{{start: 1, end: 1}}, make([]bool, len(doc.Units)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
windows[0].Units[0].Metadata["speaker"].(map[string]any)["name"] = "changed"
|
||||
if doc.Units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
|
||||
t.Fatal("context metadata aliases source document")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildContextExcludesUnsafeReferencesAndCoalescesAdjacentWindows(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{{ID: 9}, {ID: 3}, {ID: 8}, {ID: 1}, {ID: 7}}}
|
||||
candidates := []Candidate{
|
||||
{Name: "One", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 3, EndUnitID: 3}}},
|
||||
{Name: "Two", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 8, EndUnitID: 8}}},
|
||||
{Name: "Missing", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 99, EndUnitID: 99}}},
|
||||
{Name: "Foreign", SourceRefs: []source.SourceRef{{SourceID: "other", StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "Blank"},
|
||||
}
|
||||
materials, ready, err := BuildContext(doc, candidates, 0)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("BuildContext() error = %v, ready = %t", err, ready)
|
||||
}
|
||||
if got, want := materials.EligibleCandidateKeys(), []string{"candidate-000001", "candidate-000002"}; !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("EligibleCandidateKeys() = %#v, want %#v", got, want)
|
||||
}
|
||||
var transcript transcriptInput
|
||||
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 2 || transcript.Windows[0].Units[0].ID != 3 || transcript.Windows[0].Units[1].ID != 8 {
|
||||
t.Fatalf("windows = %#v, want adjacent document-order units coalesced", transcript.Windows)
|
||||
}
|
||||
if _, ready, err := BuildContext(doc, candidates, -1); err == nil || ready {
|
||||
t.Fatalf("BuildContext(radius=-1) = ready %t, err %v", ready, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessmentRejectsEveryUnsafeProposalCategory(t *testing.T) {
|
||||
materials := preparedMaterials(t, 4, true)
|
||||
keys := materials.CandidateKeys()
|
||||
unsafe := []struct {
|
||||
name string
|
||||
response ProposalResponse
|
||||
category string
|
||||
}{
|
||||
{"blank member", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{"", keys[1]}, Canonical: keys[1]}}}, "member_blank"},
|
||||
{"unknown member", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{"candidate-999999", keys[1]}, Canonical: keys[1]}}}, "member_unknown"},
|
||||
{"ineligible member", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{keys[0], keys[3]}, Canonical: keys[0]}}}, "member_ineligible"},
|
||||
{"repeated member", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{keys[0], keys[0]}, Canonical: keys[0]}}}, "repeated_member"},
|
||||
{"too small", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{keys[0]}, Canonical: keys[0]}}}, "fewer_than_two_members"},
|
||||
{"canonical blank", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{keys[0], keys[1]}, Canonical: ""}}}, "canonical_blank"},
|
||||
{"canonical not member", ProposalResponse{DuplicateGroups: []DuplicateGroup{{Members: []string{keys[0], keys[1]}, Canonical: keys[2]}}}, "canonical_not_member"},
|
||||
{"overlapping", ProposalResponse{DuplicateGroups: []DuplicateGroup{
|
||||
{Members: []string{keys[0], keys[1]}, Canonical: keys[0]},
|
||||
{Members: []string{keys[1], keys[2]}, Canonical: keys[2]},
|
||||
}}, "overlapping_member"},
|
||||
}
|
||||
for _, test := range unsafe {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
assessment := materials.Assess(test.response)
|
||||
if len(assessment.SafeGroups()) != 0 || assessment.DiscardedGroups() != len(test.response.DuplicateGroups) || !hasIssue(assessment.Issues(), test.category) {
|
||||
t.Fatalf("Assess() = groups %#v discarded %d issues %#v; want %q rejection", assessment.SafeGroups(), assessment.DiscardedGroups(), assessment.Issues(), test.category)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssessmentReturnsNonOverlappingSafeGroupsAndDefensiveCopies(t *testing.T) {
|
||||
materials := preparedMaterials(t, 4, false)
|
||||
keys := materials.CandidateKeys()
|
||||
assessment := materials.Assess(ProposalResponse{DuplicateGroups: []DuplicateGroup{
|
||||
{Members: []string{keys[1], keys[0]}, Canonical: keys[1]},
|
||||
{Members: []string{keys[3], keys[2]}, Canonical: keys[2]},
|
||||
}})
|
||||
groups := assessment.SafeGroups()
|
||||
if assessment.DiscardedGroups() != 0 || len(assessment.Issues()) != 0 || len(groups) != 2 {
|
||||
t.Fatalf("assessment = %#v, %d, %#v", groups, assessment.DiscardedGroups(), assessment.Issues())
|
||||
}
|
||||
if got, want := groups[0].Members(), []string{keys[0], keys[1]}; !reflect.DeepEqual(got, want) || groups[0].Canonical() != keys[1] {
|
||||
t.Fatalf("first safe group = %#v / %q", got, groups[0].Canonical())
|
||||
}
|
||||
keys[0] = "changed"
|
||||
if materials.CandidateKeys()[0] == "changed" {
|
||||
t.Fatal("CandidateKeys() exposed retained keys")
|
||||
}
|
||||
members := groups[0].Members()
|
||||
members[0] = "changed"
|
||||
if groups[0].Members()[0] == "changed" || assessment.SafeGroups()[0].Members()[0] == "changed" {
|
||||
t.Fatal("SafeGroups() exposed retained members")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSharedResponseSchemaIsPrivateStrictAndRegisterableOnce(t *testing.T) {
|
||||
schema, err := LoadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Name != ResponseSchemaName || schema.Version != SchemaVersion || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v", schema)
|
||||
}
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
value any
|
||||
valid bool
|
||||
}{
|
||||
{"empty groups", map[string]any{"duplicate_groups": []any{}}, true},
|
||||
{"semantic proposal problem", map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{""}, "canonical": ""}}}, true},
|
||||
{"missing groups", map[string]any{}, false},
|
||||
{"unknown top level", map[string]any{"duplicate_groups": []any{}, "extra": true}, false},
|
||||
{"replacement name", map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{}, "canonical": "candidate-000001", "name": "replacement"}}}, false},
|
||||
{"replacement evidence", map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{}, "canonical": "candidate-000001", "source_refs": []any{}}}}, false},
|
||||
{"wrong key type", map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{1}, "canonical": "candidate-000001"}}}, false},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
content, err := json.Marshal(test.value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err = validateSchema(content, schema.JSONSchema)
|
||||
if (err == nil) != test.valid {
|
||||
t.Fatalf("validateSchema() error = %v, want valid=%t", err, test.valid)
|
||||
}
|
||||
})
|
||||
}
|
||||
first := schema.JSONSchema
|
||||
first[0] = '['
|
||||
second, err := LoadResponseSchema()
|
||||
if err != nil || !json.Valid(second.JSONSchema) || bytes.Equal(first, second.JSONSchema) {
|
||||
t.Fatalf("LoadResponseSchema() returned shared content: %s, %v", second.JSONSchema, err)
|
||||
}
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := RegisterSchemaAssets(registry); err != nil {
|
||||
t.Fatalf("RegisterSchemaAssets() error = %v", err)
|
||||
}
|
||||
if schemaFS, err := registry.SchemaFS(); err != nil {
|
||||
t.Fatalf("SchemaFS() error = %v", err)
|
||||
} else if content, err := fs.ReadFile(schemaFS, "dnd_entity_reconcile_llm.v1.json"); err != nil || !json.Valid(content) {
|
||||
t.Fatalf("shared schema asset = %s, %v", content, err)
|
||||
}
|
||||
}
|
||||
|
||||
func preparedMaterials(t *testing.T, count int, includeIneligible bool) Materials {
|
||||
t.Helper()
|
||||
doc := &source.SourceDocument{ID: "session", Units: make([]source.SourceUnit, count)}
|
||||
candidates := make([]Candidate, count)
|
||||
for index := range candidates {
|
||||
doc.Units[index] = source.SourceUnit{ID: index + 1, Text: "unit"}
|
||||
candidates[index] = Candidate{Name: "same display name", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: index + 1, EndUnitID: index + 1}}}
|
||||
}
|
||||
if includeIneligible && count > 3 {
|
||||
candidates[3].SourceRefs = []source.SourceRef{{SourceID: "other", StartUnitID: 1, EndUnitID: 1}}
|
||||
}
|
||||
materials, ready, err := BuildContext(doc, candidates, 0)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("BuildContext() = %#v, %t, %v", materials, ready, err)
|
||||
}
|
||||
return materials
|
||||
}
|
||||
|
||||
func cloneCandidates(input []Candidate) []Candidate {
|
||||
output := make([]Candidate, len(input))
|
||||
copy(output, input)
|
||||
for index := range output {
|
||||
output[index].SourceRefs = append([]source.SourceRef(nil), input[index].SourceRefs...)
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
func hasIssue(issues []Issue, want string) bool {
|
||||
for _, issue := range issues {
|
||||
if issue.Category == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func validateSchema(instanceContent, schemaContent []byte) error {
|
||||
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(instanceContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
document, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := compiler.AddResource("schema.json", document); err != nil {
|
||||
return err
|
||||
}
|
||||
compiled, err := compiler.Compile("schema.json")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return compiled.Validate(instance)
|
||||
}
|
||||
168
internal/modules/dnd/shared/entityreconcile/proposal.go
Normal file
168
internal/modules/dnd/shared/entityreconcile/proposal.go
Normal file
@@ -0,0 +1,168 @@
|
||||
package entityreconcile
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// ProposalResponse is the private structured response exchanged with the
|
||||
// reconciliation prompt. It identifies candidates only by opaque keys.
|
||||
type ProposalResponse struct {
|
||||
DuplicateGroups []DuplicateGroup `json:"duplicate_groups"`
|
||||
}
|
||||
|
||||
// DuplicateGroup proposes candidate keys that might denote one entity.
|
||||
type DuplicateGroup struct {
|
||||
Members []string `json:"members"`
|
||||
Canonical string `json:"canonical"`
|
||||
}
|
||||
|
||||
// Issue identifies one unsafe proposal category without prescribing a warning
|
||||
// message or retry policy to a consuming normalizer.
|
||||
type Issue struct {
|
||||
GroupIndex int
|
||||
Category string
|
||||
}
|
||||
|
||||
// SafeGroup identifies one validated, non-overlapping duplicate group.
|
||||
type SafeGroup struct {
|
||||
members []string
|
||||
canonical string
|
||||
}
|
||||
|
||||
// Members returns an owned copy of the group's candidate keys.
|
||||
func (g SafeGroup) Members() []string { return append([]string(nil), g.members...) }
|
||||
|
||||
// Canonical returns the selected canonical candidate key.
|
||||
func (g SafeGroup) Canonical() string { return g.canonical }
|
||||
|
||||
// Assessment contains only safe groups. Its accessors return owned copies so
|
||||
// callers cannot mutate retained assessment data.
|
||||
type Assessment struct {
|
||||
safeGroups []SafeGroup
|
||||
discardedGroups int
|
||||
issues []Issue
|
||||
}
|
||||
|
||||
// SafeGroups returns validated non-overlapping groups in proposal order.
|
||||
func (a Assessment) SafeGroups() []SafeGroup {
|
||||
groups := make([]SafeGroup, len(a.safeGroups))
|
||||
for index, group := range a.safeGroups {
|
||||
groups[index] = SafeGroup{members: append([]string(nil), group.members...), canonical: group.canonical}
|
||||
}
|
||||
return groups
|
||||
}
|
||||
|
||||
// DiscardedGroups returns the number of rejected proposal groups.
|
||||
func (a Assessment) DiscardedGroups() int { return a.discardedGroups }
|
||||
|
||||
// Issues returns the deterministic rejection categories in proposal order.
|
||||
func (a Assessment) Issues() []Issue { return append([]Issue(nil), a.issues...) }
|
||||
|
||||
// Assess validates a proposal against the opaque keys created by BuildContext.
|
||||
func (m Materials) Assess(response ProposalResponse) Assessment {
|
||||
all := make(map[string]struct{}, len(m.candidateKeys))
|
||||
for _, key := range m.candidateKeys {
|
||||
all[key] = struct{}{}
|
||||
}
|
||||
groups := make([]assessedGroup, len(response.DuplicateGroups))
|
||||
issues := make([]Issue, 0)
|
||||
for groupIndex, proposal := range response.DuplicateGroups {
|
||||
groups[groupIndex] = assessGroup(proposal, all, m.eligible)
|
||||
for _, category := range groups[groupIndex].issues {
|
||||
issues = append(issues, Issue{GroupIndex: groupIndex, Category: category})
|
||||
}
|
||||
}
|
||||
|
||||
owners := make(map[string][]int)
|
||||
for groupIndex, group := range groups {
|
||||
if !group.locallyValid {
|
||||
continue
|
||||
}
|
||||
for _, key := range group.members {
|
||||
owners[key] = append(owners[key], groupIndex)
|
||||
}
|
||||
}
|
||||
for groupIndex := range groups {
|
||||
if !groups[groupIndex].locallyValid {
|
||||
continue
|
||||
}
|
||||
for _, key := range groups[groupIndex].members {
|
||||
if len(owners[key]) > 1 {
|
||||
groups[groupIndex].conflicting = true
|
||||
issues = append(issues, Issue{GroupIndex: groupIndex, Category: "overlapping_member"})
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
assessment := Assessment{issues: issues}
|
||||
for _, group := range groups {
|
||||
if !group.locallyValid || group.conflicting {
|
||||
assessment.discardedGroups++
|
||||
continue
|
||||
}
|
||||
assessment.safeGroups = append(assessment.safeGroups, SafeGroup{members: append([]string(nil), group.members...), canonical: group.canonical})
|
||||
}
|
||||
return assessment
|
||||
}
|
||||
|
||||
type assessedGroup struct {
|
||||
members []string
|
||||
canonical string
|
||||
issues []string
|
||||
locallyValid bool
|
||||
conflicting bool
|
||||
}
|
||||
|
||||
func assessGroup(proposal DuplicateGroup, all, eligible map[string]struct{}) assessedGroup {
|
||||
issues := make([]string, 0)
|
||||
members := make([]string, 0, len(proposal.Members))
|
||||
seen := make(map[string]struct{}, len(proposal.Members))
|
||||
for _, key := range proposal.Members {
|
||||
if category := keyCategory(key, all, eligible); category != "" {
|
||||
issues = append(issues, "member_"+category)
|
||||
continue
|
||||
}
|
||||
if _, exists := seen[key]; exists {
|
||||
issues = append(issues, "repeated_member")
|
||||
continue
|
||||
}
|
||||
seen[key] = struct{}{}
|
||||
members = append(members, key)
|
||||
}
|
||||
canonicalCategory := keyCategory(proposal.Canonical, all, eligible)
|
||||
if canonicalCategory != "" {
|
||||
issues = append(issues, "canonical_"+canonicalCategory)
|
||||
}
|
||||
if len(members) < 2 {
|
||||
issues = append(issues, "fewer_than_two_members")
|
||||
}
|
||||
if canonicalCategory == "" && !contains(members, proposal.Canonical) {
|
||||
issues = append(issues, "canonical_not_member")
|
||||
}
|
||||
sort.Strings(members)
|
||||
return assessedGroup{members: members, canonical: proposal.Canonical, issues: issues, locallyValid: len(issues) == 0}
|
||||
}
|
||||
|
||||
func keyCategory(key string, all, eligible map[string]struct{}) string {
|
||||
if strings.TrimSpace(key) == "" {
|
||||
return "blank"
|
||||
}
|
||||
if _, ok := all[key]; !ok {
|
||||
return "unknown"
|
||||
}
|
||||
if _, ok := eligible[key]; !ok {
|
||||
return "ineligible"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func contains(values []string, want string) bool {
|
||||
for _, value := range values {
|
||||
if value == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
31
internal/modules/dnd/shared/entityreconcile/schema.go
Normal file
31
internal/modules/dnd/shared/entityreconcile/schema.go
Normal file
@@ -0,0 +1,31 @@
|
||||
package entityreconcile
|
||||
|
||||
import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
)
|
||||
|
||||
const (
|
||||
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_entity_reconcile_llm")
|
||||
ResponseSchemaID = "notarius.dnd.entity_reconcile.llm"
|
||||
ResponseSchemaName = "notarius_dnd_entity_reconcile_llm_v1"
|
||||
SchemaVersion = "v1"
|
||||
SchemaAssetPath = "assets/schemas/dnd_entity_reconcile_llm.v1.json"
|
||||
)
|
||||
|
||||
// LoadResponseSchema returns the shared private duplicate-group response
|
||||
// contract. It is intentionally separate from durable artifact schemas.
|
||||
func LoadResponseSchema() (llm.ResponseSchema, error) {
|
||||
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
|
||||
Key: ResponseSchemaKey,
|
||||
ID: ResponseSchemaID,
|
||||
Version: SchemaVersion,
|
||||
Name: ResponseSchemaName,
|
||||
AssetPath: SchemaAssetPath,
|
||||
})
|
||||
}
|
||||
|
||||
// RegisterSchemaAssets makes the shared private response schema available to
|
||||
// prompt preparation. A family registrar can register it once for all consumers.
|
||||
func RegisterSchemaAssets(registry *llm.AssetRegistry) error {
|
||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||
}
|
||||
268
internal/modules/dnd/shared/registryresolver/resolver.go
Normal file
268
internal/modules/dnd/shared/registryresolver/resolver.go
Normal file
@@ -0,0 +1,268 @@
|
||||
// Package registryresolver provides the shared reference-selection and caching
|
||||
// mechanics used by immutable D&D registry views.
|
||||
package registryresolver
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"mime"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
// ReferenceSpec describes one optional, single-item registry reference.
|
||||
type ReferenceSpec struct {
|
||||
SlotName string
|
||||
AcceptedMediaType string
|
||||
MaxBytes int64
|
||||
}
|
||||
|
||||
// Item is an owned reference payload. MediaType is the parsed, normalized
|
||||
// media type without parameters. Reference provenance and caller digests are
|
||||
// intentionally excluded.
|
||||
type Item struct {
|
||||
MediaType string
|
||||
Content []byte
|
||||
}
|
||||
|
||||
// validatedItem borrows its content from the supplied reference set. Callers
|
||||
// must copy content before passing it to a callback that may retain it.
|
||||
type validatedItem struct {
|
||||
mediaType string
|
||||
content []byte
|
||||
}
|
||||
|
||||
// Config supplies the domain-owned operations needed to prepare immutable
|
||||
// registry views. Absent and Load must return values whose mutable state is not
|
||||
// exposed to callers. Load receives owned bytes and may retain them. Errors
|
||||
// returned by either callback must be bounded and must not contain input data.
|
||||
type Config[V any] struct {
|
||||
Reference ReferenceSpec
|
||||
Absent func() (V, error)
|
||||
Load func([]byte) (V, error)
|
||||
SemanticIdentity func(V) string
|
||||
}
|
||||
|
||||
type preparedView[V any] struct {
|
||||
value V
|
||||
bound bool
|
||||
identity string
|
||||
}
|
||||
|
||||
type semanticKey struct {
|
||||
bound bool
|
||||
identity string
|
||||
}
|
||||
|
||||
// Resolver retains one immutable construction-time view and memoizes valid
|
||||
// operation-time views by raw reference content and semantic identity.
|
||||
type Resolver[V any] struct {
|
||||
config Config[V]
|
||||
seeded preparedView[V]
|
||||
mediaType string
|
||||
slotName string
|
||||
|
||||
mu sync.Mutex
|
||||
rawCache map[string]preparedView[V]
|
||||
semanticCache map[semanticKey]preparedView[V]
|
||||
}
|
||||
|
||||
// New validates the configuration and construction-time reference. A present
|
||||
// slot with no items is treated as an absent generated-reference placeholder
|
||||
// only at this construction boundary.
|
||||
func New[V any](config Config[V], references contracts.ReferenceSet) (*Resolver[V], error) {
|
||||
normalized, mediaType, err := normalizeConfig(config)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
resolver := &Resolver[V]{
|
||||
config: normalized,
|
||||
mediaType: mediaType,
|
||||
slotName: normalized.Reference.SlotName,
|
||||
rawCache: make(map[string]preparedView[V]),
|
||||
semanticCache: make(map[semanticKey]preparedView[V]),
|
||||
}
|
||||
|
||||
if slot, present := references.Slots[resolver.slotName]; present && len(slot.Items) == 0 {
|
||||
resolver.seeded, err = resolver.absent()
|
||||
} else {
|
||||
resolver.seeded, err = resolver.resolveUncached(references)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return resolver, nil
|
||||
}
|
||||
|
||||
// Seeded returns the immutable construction-time view.
|
||||
func (r *Resolver[V]) Seeded() V {
|
||||
if r == nil {
|
||||
var zero V
|
||||
return zero
|
||||
}
|
||||
return r.seeded.value
|
||||
}
|
||||
|
||||
// Resolve returns the construction-time view when the operation does not
|
||||
// supply the configured slot. A present slot is validated and loaded as an
|
||||
// operation-time override.
|
||||
func (r *Resolver[V]) Resolve(references contracts.ReferenceSet) (V, error) {
|
||||
if r == nil {
|
||||
var zero V
|
||||
return zero, fmt.Errorf("registry resolver must not be nil")
|
||||
}
|
||||
if _, present := references.Slots[r.slotName]; !present {
|
||||
return r.seeded.value, nil
|
||||
}
|
||||
|
||||
item, _, err := validateOptionalSingleItem(references, r.config.Reference, r.mediaType)
|
||||
if err != nil {
|
||||
var zero V
|
||||
return zero, err
|
||||
}
|
||||
rawKey := rawReferenceKey(item)
|
||||
|
||||
r.mu.Lock()
|
||||
defer r.mu.Unlock()
|
||||
if cached, ok := r.rawCache[rawKey]; ok {
|
||||
return cached.value, nil
|
||||
}
|
||||
|
||||
resolved, err := r.load(append([]byte(nil), item.content...))
|
||||
if err != nil {
|
||||
var zero V
|
||||
return zero, err
|
||||
}
|
||||
if sameIdentity(r.seeded, resolved) {
|
||||
r.rawCache[rawKey] = r.seeded
|
||||
return r.seeded.value, nil
|
||||
}
|
||||
|
||||
key := semanticKey{bound: resolved.bound, identity: resolved.identity}
|
||||
if cached, ok := r.semanticCache[key]; ok {
|
||||
r.rawCache[rawKey] = cached
|
||||
return cached.value, nil
|
||||
}
|
||||
r.semanticCache[key] = resolved
|
||||
r.rawCache[rawKey] = resolved
|
||||
return resolved.value, nil
|
||||
}
|
||||
|
||||
// ResolveOptionalSingleItem validates and copies one optional registry item.
|
||||
// A missing slot returns present=false. A present slot must contain exactly one
|
||||
// item, even when it represents an operation-time generated reference.
|
||||
func ResolveOptionalSingleItem(references contracts.ReferenceSet, spec ReferenceSpec) (Item, bool, error) {
|
||||
normalized, mediaType, err := normalizeReferenceSpec(spec)
|
||||
if err != nil {
|
||||
return Item{}, false, err
|
||||
}
|
||||
item, present, err := validateOptionalSingleItem(references, normalized, mediaType)
|
||||
if err != nil || !present {
|
||||
return Item{}, present, err
|
||||
}
|
||||
return Item{
|
||||
MediaType: item.mediaType,
|
||||
Content: append([]byte(nil), item.content...),
|
||||
}, true, nil
|
||||
}
|
||||
|
||||
func (r *Resolver[V]) resolveUncached(references contracts.ReferenceSet) (preparedView[V], error) {
|
||||
item, present, err := validateOptionalSingleItem(references, r.config.Reference, r.mediaType)
|
||||
if err != nil {
|
||||
return preparedView[V]{}, err
|
||||
}
|
||||
if !present {
|
||||
return r.absent()
|
||||
}
|
||||
return r.load(append([]byte(nil), item.content...))
|
||||
}
|
||||
|
||||
func (r *Resolver[V]) absent() (preparedView[V], error) {
|
||||
value, err := r.config.Absent()
|
||||
if err != nil {
|
||||
return preparedView[V]{}, err
|
||||
}
|
||||
return preparedView[V]{value: value, identity: r.config.SemanticIdentity(value)}, nil
|
||||
}
|
||||
|
||||
func (r *Resolver[V]) load(content []byte) (preparedView[V], error) {
|
||||
value, err := r.config.Load(content)
|
||||
if err != nil {
|
||||
return preparedView[V]{}, err
|
||||
}
|
||||
identity := strings.TrimSpace(r.config.SemanticIdentity(value))
|
||||
if identity == "" {
|
||||
return preparedView[V]{}, fmt.Errorf("load reference slot %q: semantic identity must not be empty", r.slotName)
|
||||
}
|
||||
return preparedView[V]{value: value, bound: true, identity: identity}, nil
|
||||
}
|
||||
|
||||
func normalizeConfig[V any](config Config[V]) (Config[V], string, error) {
|
||||
reference, mediaType, err := normalizeReferenceSpec(config.Reference)
|
||||
if err != nil {
|
||||
return Config[V]{}, "", err
|
||||
}
|
||||
if config.Absent == nil {
|
||||
return Config[V]{}, "", fmt.Errorf("registry resolver absent-view callback must not be nil")
|
||||
}
|
||||
if config.Load == nil {
|
||||
return Config[V]{}, "", fmt.Errorf("registry resolver loader must not be nil")
|
||||
}
|
||||
if config.SemanticIdentity == nil {
|
||||
return Config[V]{}, "", fmt.Errorf("registry resolver semantic-identity callback must not be nil")
|
||||
}
|
||||
config.Reference = reference
|
||||
return config, mediaType, nil
|
||||
}
|
||||
|
||||
func normalizeReferenceSpec(spec ReferenceSpec) (ReferenceSpec, string, error) {
|
||||
spec.SlotName = strings.TrimSpace(spec.SlotName)
|
||||
if spec.SlotName == "" {
|
||||
return ReferenceSpec{}, "", fmt.Errorf("registry reference slot name must not be empty")
|
||||
}
|
||||
mediaType, _, err := mime.ParseMediaType(spec.AcceptedMediaType)
|
||||
if err != nil || strings.TrimSpace(mediaType) == "" {
|
||||
return ReferenceSpec{}, "", fmt.Errorf("registry reference slot %q accepted media type is invalid", spec.SlotName)
|
||||
}
|
||||
mediaType = strings.ToLower(mediaType)
|
||||
if spec.MaxBytes <= 0 {
|
||||
return ReferenceSpec{}, "", fmt.Errorf("registry reference slot %q maximum size must be positive", spec.SlotName)
|
||||
}
|
||||
spec.AcceptedMediaType = mediaType
|
||||
return spec, mediaType, nil
|
||||
}
|
||||
|
||||
func validateOptionalSingleItem(references contracts.ReferenceSet, spec ReferenceSpec, acceptedMediaType string) (validatedItem, bool, error) {
|
||||
slot, present := references.Slots[spec.SlotName]
|
||||
if !present {
|
||||
return validatedItem{}, false, nil
|
||||
}
|
||||
if len(slot.Items) != 1 {
|
||||
return validatedItem{}, true, fmt.Errorf("reference slot %q must contain exactly one item", spec.SlotName)
|
||||
}
|
||||
item := slot.Items[0]
|
||||
mediaType, _, err := mime.ParseMediaType(item.MediaType)
|
||||
if err != nil {
|
||||
return validatedItem{}, true, fmt.Errorf("reference slot %q item media type is invalid", spec.SlotName)
|
||||
}
|
||||
mediaType = strings.ToLower(mediaType)
|
||||
if !strings.EqualFold(mediaType, acceptedMediaType) {
|
||||
return validatedItem{}, true, fmt.Errorf("reference slot %q item media type must be %s", spec.SlotName, acceptedMediaType)
|
||||
}
|
||||
if int64(len(item.Content)) > spec.MaxBytes {
|
||||
return validatedItem{}, true, fmt.Errorf("reference slot %q item is %d bytes, limit %d", spec.SlotName, len(item.Content), spec.MaxBytes)
|
||||
}
|
||||
return validatedItem{mediaType: mediaType, content: item.Content}, true, nil
|
||||
}
|
||||
|
||||
func rawReferenceKey(item validatedItem) string {
|
||||
sum := sha256.Sum256(item.content)
|
||||
return item.mediaType + "\x00sha256:" + hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func sameIdentity[V any](first, second preparedView[V]) bool {
|
||||
return first.bound == second.bound && first.identity == second.identity
|
||||
}
|
||||
356
internal/modules/dnd/shared/registryresolver/resolver_test.go
Normal file
356
internal/modules/dnd/shared/registryresolver/resolver_test.go
Normal file
@@ -0,0 +1,356 @@
|
||||
package registryresolver_test
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/registryresolver"
|
||||
)
|
||||
|
||||
const (
|
||||
testSlot = "registry"
|
||||
testMediaType = "application/json"
|
||||
)
|
||||
|
||||
type immutableView struct {
|
||||
value string
|
||||
identity string
|
||||
}
|
||||
|
||||
func TestResolverHandlesConstructionStateFallbackAndOverrides(t *testing.T) {
|
||||
config := resolverConfig(nil)
|
||||
absent, err := registryresolver.New(config, contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
placeholder, err := registryresolver.New(config, referenceSet())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if absent.Seeded().value != "absent" || placeholder.Seeded().value != "absent" {
|
||||
t.Fatalf("seeded views = %#v / %#v, want absent", absent.Seeded(), placeholder.Seeded())
|
||||
}
|
||||
|
||||
seedContent := []byte("seeded")
|
||||
seeded, err := registryresolver.New(config, referenceSet(referenceItem(seedContent)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
fallback, err := seeded.Resolve(contracts.ReferenceSet{})
|
||||
if err != nil || fallback != seeded.Seeded() {
|
||||
t.Fatalf("fallback = %p, %v; seeded = %p", fallback, err, seeded.Seeded())
|
||||
}
|
||||
override, err := seeded.Resolve(referenceSet(referenceItem([]byte("override"))))
|
||||
if err != nil || override == seeded.Seeded() || override.value != "override" {
|
||||
t.Fatalf("override = %#v, %v", override, err)
|
||||
}
|
||||
if _, err := seeded.Resolve(referenceSet()); err == nil || !strings.Contains(err.Error(), "exactly one item") {
|
||||
t.Fatalf("operation placeholder error = %v", err)
|
||||
}
|
||||
if _, err := registryresolver.New(config, referenceSet(referenceItem([]byte("one")), referenceItem([]byte("two")))); err == nil || !strings.Contains(err.Error(), "exactly one item") {
|
||||
t.Fatalf("construction cardinality error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveOptionalSingleItemValidatesAndOwnsContent(t *testing.T) {
|
||||
spec := referenceSpec()
|
||||
if item, present, err := registryresolver.ResolveOptionalSingleItem(contracts.ReferenceSet{}, spec); err != nil || present || item.Content != nil {
|
||||
t.Fatalf("absent item = %#v, %t, %v", item, present, err)
|
||||
}
|
||||
|
||||
content := []byte("approved")
|
||||
set := referenceSet(contracts.ReferenceItem{
|
||||
SlotName: testSlot,
|
||||
MediaType: "Application/JSON; Charset=UTF-8",
|
||||
Content: content,
|
||||
Digest: "sha256:caller-supplied",
|
||||
Origin: contracts.ReferenceOrigin{Type: "file", URI: "/private/campaign.json"},
|
||||
})
|
||||
item, present, err := registryresolver.ResolveOptionalSingleItem(set, spec)
|
||||
if err != nil || !present || item.MediaType != testMediaType || string(item.Content) != "approved" {
|
||||
t.Fatalf("resolved item = %#v, %t, %v", item, present, err)
|
||||
}
|
||||
content[0] = 'X'
|
||||
if string(item.Content) != "approved" {
|
||||
t.Fatalf("resolved item retained caller bytes: %q", item.Content)
|
||||
}
|
||||
|
||||
secret := "private campaign material"
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
spec registryresolver.ReferenceSpec
|
||||
set contracts.ReferenceSet
|
||||
want string
|
||||
}{
|
||||
{"blank slot", registryresolver.ReferenceSpec{AcceptedMediaType: testMediaType, MaxBytes: 64}, contracts.ReferenceSet{}, "slot name"},
|
||||
{"invalid accepted media", registryresolver.ReferenceSpec{SlotName: testSlot, AcceptedMediaType: "not a type", MaxBytes: 64}, contracts.ReferenceSet{}, "accepted media type"},
|
||||
{"invalid maximum", registryresolver.ReferenceSpec{SlotName: testSlot, AcceptedMediaType: testMediaType}, contracts.ReferenceSet{}, "maximum size"},
|
||||
{"empty slot", spec, referenceSet(), "exactly one item"},
|
||||
{"multiple items", spec, referenceSet(referenceItem([]byte("one")), referenceItem([]byte("two"))), "exactly one item"},
|
||||
{"invalid item media", spec, referenceSet(contracts.ReferenceItem{MediaType: secret, Content: []byte(secret)}), "media type is invalid"},
|
||||
{"wrong item media", spec, referenceSet(contracts.ReferenceItem{MediaType: "text/plain", Content: []byte(secret)}), "must be application/json"},
|
||||
{"oversized", spec, referenceSet(referenceItem([]byte(strings.Repeat("x", 65)))), "limit 64"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
_, _, err := registryresolver.ResolveOptionalSingleItem(test.set, test.spec)
|
||||
if err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("error = %v, want %q", err, test.want)
|
||||
}
|
||||
if strings.Contains(err.Error(), secret) || strings.Contains(err.Error(), "/private/") {
|
||||
t.Fatalf("error exposed reference content or provenance: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverReusesRawSemanticAndSeededViewsByActualContent(t *testing.T) {
|
||||
var loads atomic.Int32
|
||||
resolver, err := registryresolver.New(resolverConfig(&loads), referenceSet(referenceItem([]byte("alpha"))))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if loads.Load() != 1 {
|
||||
t.Fatalf("construction loads = %d, want 1", loads.Load())
|
||||
}
|
||||
|
||||
seedEquivalent := referenceSet(referenceItem([]byte(" alpha ")))
|
||||
resolvedSeed, err := resolver.Resolve(seedEquivalent)
|
||||
if err != nil || resolvedSeed != resolver.Seeded() {
|
||||
t.Fatalf("semantic seeded reuse = %p, %v; want %p", resolvedSeed, err, resolver.Seeded())
|
||||
}
|
||||
if _, err := resolver.Resolve(seedEquivalent); err != nil || loads.Load() != 2 {
|
||||
t.Fatalf("raw seeded reuse loads = %d, err = %v; want 2", loads.Load(), err)
|
||||
}
|
||||
|
||||
betaSet := referenceSet(referenceItem([]byte("beta")))
|
||||
beta, err := resolver.Resolve(betaSet)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
betaAgain, err := resolver.Resolve(betaSet)
|
||||
if err != nil || betaAgain != beta {
|
||||
t.Fatalf("raw reuse = %p / %p, %v", beta, betaAgain, err)
|
||||
}
|
||||
betaEquivalent, err := resolver.Resolve(referenceSet(referenceItem([]byte("\n beta \t"))))
|
||||
if err != nil || betaEquivalent != beta {
|
||||
t.Fatalf("semantic reuse = %p / %p, %v", beta, betaEquivalent, err)
|
||||
}
|
||||
mediaVariant := referenceItem([]byte("beta"))
|
||||
mediaVariant.MediaType = "APPLICATION/JSON; charset=utf-8"
|
||||
if got, err := resolver.Resolve(referenceSet(mediaVariant)); err != nil || got != beta {
|
||||
t.Fatalf("normalized media raw reuse = %p / %p, %v", beta, got, err)
|
||||
}
|
||||
if loads.Load() != 4 {
|
||||
t.Fatalf("loads after raw and semantic reuse = %d, want 4", loads.Load())
|
||||
}
|
||||
|
||||
first := referenceItem([]byte("gamma"))
|
||||
second := referenceItem([]byte("delta"))
|
||||
first.Digest = "sha256:" + strings.Repeat("0", 64)
|
||||
second.Digest = first.Digest
|
||||
gamma, err := resolver.Resolve(referenceSet(first))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
delta, err := resolver.Resolve(referenceSet(second))
|
||||
if err != nil || gamma == delta || gamma.value == delta.value {
|
||||
t.Fatalf("caller digest aliased different bytes: %#v / %#v, %v", gamma, delta, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverSerializesConcurrentLoads(t *testing.T) {
|
||||
var loads atomic.Int32
|
||||
resolver, err := registryresolver.New(resolverConfig(&loads), contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
loads.Store(0)
|
||||
references := referenceSet(referenceItem([]byte("shared")))
|
||||
|
||||
const callers = 32
|
||||
results := make(chan *immutableView, callers)
|
||||
errors := make(chan error, callers)
|
||||
var group sync.WaitGroup
|
||||
for range callers {
|
||||
group.Add(1)
|
||||
go func() {
|
||||
defer group.Done()
|
||||
view, err := resolver.Resolve(references)
|
||||
if err != nil {
|
||||
errors <- err
|
||||
return
|
||||
}
|
||||
results <- view
|
||||
}()
|
||||
}
|
||||
group.Wait()
|
||||
close(results)
|
||||
close(errors)
|
||||
for err := range errors {
|
||||
t.Error(err)
|
||||
}
|
||||
var first *immutableView
|
||||
for result := range results {
|
||||
if first == nil {
|
||||
first = result
|
||||
}
|
||||
if result != first {
|
||||
t.Fatalf("concurrent result = %p, want %p", result, first)
|
||||
}
|
||||
}
|
||||
if loads.Load() != 1 {
|
||||
t.Fatalf("concurrent loads = %d, want 1", loads.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolverDoesNotCacheFailuresOrRetainCallerState(t *testing.T) {
|
||||
loaderFailure := errors.New("loader unavailable")
|
||||
var loads atomic.Int32
|
||||
var loadedBytes []byte
|
||||
config := resolverConfig(&loads)
|
||||
config.Load = func(content []byte) (*immutableView, error) {
|
||||
loads.Add(1)
|
||||
loadedBytes = content
|
||||
if string(content) == "fail" {
|
||||
return nil, loaderFailure
|
||||
}
|
||||
value := strings.TrimSpace(string(content))
|
||||
return &immutableView{value: value, identity: value}, nil
|
||||
}
|
||||
|
||||
seedBytes := []byte("seed")
|
||||
seedReferences := referenceSet(referenceItem(seedBytes))
|
||||
resolver, err := registryresolver.New(config, seedReferences)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
seedBytes[0] = 'X'
|
||||
delete(seedReferences.Slots, testSlot)
|
||||
if resolver.Seeded().value != "seed" || string(loadedBytes) != "seed" {
|
||||
t.Fatalf("construction retained caller state: %#v / %q", resolver.Seeded(), loadedBytes)
|
||||
}
|
||||
|
||||
failing := referenceSet(referenceItem([]byte("fail")))
|
||||
for range 2 {
|
||||
if _, err := resolver.Resolve(failing); !errors.Is(err, loaderFailure) {
|
||||
t.Fatalf("loader error = %v", err)
|
||||
}
|
||||
}
|
||||
if loads.Load() != 3 {
|
||||
t.Fatalf("loads after repeated failure = %d, want 3", loads.Load())
|
||||
}
|
||||
|
||||
operationBytes := []byte("operation")
|
||||
operationReferences := referenceSet(referenceItem(operationBytes))
|
||||
view, err := resolver.Resolve(operationReferences)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
operationBytes[0] = 'X'
|
||||
operationReferences.Slots[testSlot] = contracts.ResolvedReferenceSlot{}
|
||||
if view.value != "operation" || string(loadedBytes) != "operation" {
|
||||
t.Fatalf("operation retained caller state: %#v / %q", view, loadedBytes)
|
||||
}
|
||||
|
||||
var invalidLoads atomic.Int32
|
||||
config.Load = func([]byte) (*immutableView, error) {
|
||||
invalidLoads.Add(1)
|
||||
return &immutableView{value: "invalid"}, nil
|
||||
}
|
||||
missingIdentity, err := registryresolver.New(config, contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for range 2 {
|
||||
if _, err := missingIdentity.Resolve(referenceSet(referenceItem([]byte("invalid")))); err == nil || !strings.Contains(err.Error(), "semantic identity") {
|
||||
t.Fatalf("missing identity error = %v", err)
|
||||
}
|
||||
}
|
||||
if invalidLoads.Load() != 2 {
|
||||
t.Fatalf("invalid view loads = %d, want 2", invalidLoads.Load())
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewValidatesCallbacksAndDoesNotCacheAbsentFailure(t *testing.T) {
|
||||
base := resolverConfig(nil)
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
mutate func(*registryresolver.Config[*immutableView])
|
||||
want string
|
||||
}{
|
||||
{"absent", func(config *registryresolver.Config[*immutableView]) { config.Absent = nil }, "absent-view"},
|
||||
{"loader", func(config *registryresolver.Config[*immutableView]) { config.Load = nil }, "loader"},
|
||||
{"identity", func(config *registryresolver.Config[*immutableView]) { config.SemanticIdentity = nil }, "semantic-identity"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
config := base
|
||||
test.mutate(&config)
|
||||
if _, err := registryresolver.New(config, contracts.ReferenceSet{}); err == nil || !strings.Contains(err.Error(), test.want) {
|
||||
t.Fatalf("New() error = %v, want %q", err, test.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
absentFailure := errors.New("absent unavailable")
|
||||
base.Absent = func() (*immutableView, error) { return nil, absentFailure }
|
||||
if _, err := registryresolver.New(base, contracts.ReferenceSet{}); !errors.Is(err, absentFailure) {
|
||||
t.Fatalf("absent error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func resolverConfig(loads *atomic.Int32) registryresolver.Config[*immutableView] {
|
||||
return registryresolver.Config[*immutableView]{
|
||||
Reference: referenceSpec(),
|
||||
Absent: func() (*immutableView, error) {
|
||||
return &immutableView{value: "absent", identity: "absent"}, nil
|
||||
},
|
||||
Load: func(content []byte) (*immutableView, error) {
|
||||
if loads != nil {
|
||||
loads.Add(1)
|
||||
}
|
||||
value := strings.TrimSpace(string(content))
|
||||
return &immutableView{value: value, identity: "identity:" + value}, nil
|
||||
},
|
||||
SemanticIdentity: func(view *immutableView) string {
|
||||
if view == nil {
|
||||
return ""
|
||||
}
|
||||
return view.identity
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func referenceSpec() registryresolver.ReferenceSpec {
|
||||
return registryresolver.ReferenceSpec{SlotName: testSlot, AcceptedMediaType: testMediaType, MaxBytes: 64}
|
||||
}
|
||||
|
||||
func referenceItem(content []byte) contracts.ReferenceItem {
|
||||
return contracts.ReferenceItem{SlotName: testSlot, MediaType: testMediaType, Content: content}
|
||||
}
|
||||
|
||||
func referenceSet(items ...contracts.ReferenceItem) contracts.ReferenceSet {
|
||||
return contracts.ReferenceSet{Slots: map[string]contracts.ResolvedReferenceSlot{
|
||||
testSlot: {Slot: contracts.ReferenceSlot{Name: testSlot}, Items: items},
|
||||
}}
|
||||
}
|
||||
|
||||
func TestErrorsDoNotFormatReferenceData(t *testing.T) {
|
||||
secret := "hidden campaign notes"
|
||||
config := resolverConfig(nil)
|
||||
config.Load = func([]byte) (*immutableView, error) { return nil, fmt.Errorf("safe loader failure") }
|
||||
resolver, err := registryresolver.New(config, contracts.ReferenceSet{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
item := referenceItem([]byte(secret))
|
||||
item.Origin.URI = "/private/campaign.json"
|
||||
_, err = resolver.Resolve(referenceSet(item))
|
||||
if err == nil || strings.Contains(err.Error(), secret) || strings.Contains(err.Error(), item.Origin.URI) {
|
||||
t.Fatalf("error exposed reference data: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -20,6 +20,10 @@ const ItemEventListKind contracts.ArtifactKind = "dnd/item-event-list"
|
||||
|
||||
const EnemyEventListKind contracts.ArtifactKind = "dnd/enemy-event-list"
|
||||
|
||||
const LocationListKind contracts.ArtifactKind = "dnd/location-list"
|
||||
|
||||
const LocationOccurrenceListKind contracts.ArtifactKind = "dnd/location-occurrence-list"
|
||||
|
||||
type SpellList struct {
|
||||
SpellCasts []SpellCast `json:"spell_casts"`
|
||||
}
|
||||
@@ -144,3 +148,33 @@ type EnemyEvent struct {
|
||||
Kind EnemyEventKind `json:"kind"`
|
||||
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||
}
|
||||
|
||||
type LocationList struct {
|
||||
Locations []Location `json:"locations"`
|
||||
}
|
||||
|
||||
type Location struct {
|
||||
ID string `json:"id"`
|
||||
Name string `json:"name"`
|
||||
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||
}
|
||||
|
||||
type LocationOccurrenceKind string
|
||||
|
||||
const (
|
||||
LocationOccurrenceKindVisited LocationOccurrenceKind = "visited"
|
||||
LocationOccurrenceKindPlanned LocationOccurrenceKind = "planned"
|
||||
LocationOccurrenceKindRecalled LocationOccurrenceKind = "recalled"
|
||||
LocationOccurrenceKindMentioned LocationOccurrenceKind = "mentioned"
|
||||
)
|
||||
|
||||
type LocationOccurrenceList struct {
|
||||
Occurrences []LocationOccurrence `json:"occurrences"`
|
||||
}
|
||||
|
||||
type LocationOccurrence struct {
|
||||
LocationID string `json:"location_id"`
|
||||
Name string `json:"name"`
|
||||
Kind LocationOccurrenceKind `json:"kind"`
|
||||
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user