Compare commits
7 Commits
a4d64f6f16
...
0a4a29a2df
| Author | SHA1 | Date | |
|---|---|---|---|
| 0a4a29a2df | |||
| 373ba13562 | |||
| c51c2d7fe5 | |||
| b2012b9b2c | |||
| 404bfc8da5 | |||
| 382ca3fb6c | |||
| d9b5c25cfe |
116
docs/integrations/dnd-spells.md
Normal file
116
docs/integrations/dnd-spells.md
Normal file
@@ -0,0 +1,116 @@
|
|||||||
|
# D&D Spell Cast Extraction
|
||||||
|
|
||||||
|
This document describes the D&D spell-cast extractor currently implemented in
|
||||||
|
Notarius.
|
||||||
|
|
||||||
|
## Module
|
||||||
|
|
||||||
|
- Module key: `dnd/spells`
|
||||||
|
- Artifact type: `dnd.spell_cast`
|
||||||
|
- Schema version: `v1`
|
||||||
|
- Prompt ID: `dnd.spells`
|
||||||
|
- Response schema key: `dnd_spells`
|
||||||
|
- Response schema ID: `notarius.dnd.spells`
|
||||||
|
- Response schema name: `notarius_dnd_spells_v1`
|
||||||
|
|
||||||
|
The module is an extract module. It reads a generic source chunk, renders the
|
||||||
|
`dnd.spells` prompt, calls the configured structured LLM client, and returns
|
||||||
|
spell-cast artifact candidates.
|
||||||
|
|
||||||
|
## Source Expectations
|
||||||
|
|
||||||
|
The extractor expects a generic `SourceDocument` and active source chunk. It
|
||||||
|
does not depend on concrete Seriatim package types.
|
||||||
|
|
||||||
|
Pipeline resolution must provide these capabilities before the extractor runs:
|
||||||
|
|
||||||
|
- `chunks`
|
||||||
|
- `source.transcript`
|
||||||
|
|
||||||
|
Source units may include transcript metadata such as speaker and timestamps.
|
||||||
|
That metadata is optional prompt context. It is not part of the durable spell
|
||||||
|
payload.
|
||||||
|
|
||||||
|
## Artifact Payload
|
||||||
|
|
||||||
|
Each approved artifact payload is a JSON object with these fields:
|
||||||
|
|
||||||
|
- `caster`: in-world character or creature casting the spell;
|
||||||
|
- `spell`: spell name;
|
||||||
|
- `effect`: concise spell effect in the scene;
|
||||||
|
- `narrative_description`: short description of the spell cast in context.
|
||||||
|
|
||||||
|
`caster` is not the table speaker. NPCs, monsters, and other DM-voiced
|
||||||
|
characters can be casters.
|
||||||
|
|
||||||
|
## Source References
|
||||||
|
|
||||||
|
The structured LLM response must include `source_refs` for each spell cast.
|
||||||
|
Each source reference uses the generic source-reference shape:
|
||||||
|
|
||||||
|
- `source_id`
|
||||||
|
- `start_unit_id`
|
||||||
|
- `end_unit_id`
|
||||||
|
|
||||||
|
The extractor copies those references into the generic artifact envelope
|
||||||
|
`source_refs` field. The durable `dnd.spell_cast` payload does not duplicate
|
||||||
|
source references.
|
||||||
|
|
||||||
|
Source-reference IDs must match the source document and source-unit IDs
|
||||||
|
exactly. The validator chain rejects unknown source IDs, unknown unit IDs, and
|
||||||
|
reversed unit ranges.
|
||||||
|
|
||||||
|
## Validators
|
||||||
|
|
||||||
|
The extractor provides these deterministic validators by default, in order:
|
||||||
|
|
||||||
|
- `dnd/spells/shape`
|
||||||
|
- `dnd/spells/source_refs`
|
||||||
|
|
||||||
|
`dnd/spells/shape` rejects:
|
||||||
|
|
||||||
|
- malformed JSON payloads with reason code `invalid_payload`;
|
||||||
|
- blank `caster`, `spell`, `effect`, or `narrative_description` fields with
|
||||||
|
reason code `missing_required_field`.
|
||||||
|
|
||||||
|
`dnd/spells/source_refs` rejects:
|
||||||
|
|
||||||
|
- candidates with no source references using reason code `missing_source_ref`;
|
||||||
|
- invalid source references using reason code `invalid_source_ref`.
|
||||||
|
|
||||||
|
The source-reference validator uses the core `source.ValidateRef` behavior, so
|
||||||
|
its rejection message includes the underlying source-reference validation
|
||||||
|
error.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
The module declares these required capabilities:
|
||||||
|
|
||||||
|
- `chunks`
|
||||||
|
- `source.transcript`
|
||||||
|
|
||||||
|
The module declares this provided capability:
|
||||||
|
|
||||||
|
- `dnd.spell_casts`
|
||||||
|
|
||||||
|
A pipeline artifact lane can reference the extractor with:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
merge: appendorder
|
||||||
|
normalize: noop
|
||||||
|
```
|
||||||
|
|
||||||
|
## Limits
|
||||||
|
|
||||||
|
This checkpoint implements only D&D spell-cast extraction. It does not
|
||||||
|
implement:
|
||||||
|
|
||||||
|
- item extraction;
|
||||||
|
- NPC extraction;
|
||||||
|
- combat extraction;
|
||||||
|
- encounter extraction;
|
||||||
|
- broad D&D rules validation;
|
||||||
|
- a CLI `run` workflow.
|
||||||
@@ -2,7 +2,9 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
This document describes planned work, not implemented behavior.
|
This document records the target scope for checkpoint 6. The implemented
|
||||||
|
integration contract is documented in
|
||||||
|
[`docs/integrations/dnd-spells.md`](../integrations/dnd-spells.md).
|
||||||
|
|
||||||
## Goal
|
## Goal
|
||||||
|
|
||||||
|
|||||||
64
internal/framework/llm/assets/schemas/dnd_spells.v1.json
Normal file
64
internal/framework/llm/assets/schemas/dnd_spells.v1.json
Normal file
@@ -0,0 +1,64 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "notarius.dnd.spells",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["spell_casts"],
|
||||||
|
"properties": {
|
||||||
|
"spell_casts": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"caster",
|
||||||
|
"spell",
|
||||||
|
"effect",
|
||||||
|
"narrative_description",
|
||||||
|
"source_refs"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"caster": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"spell": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"effect": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"narrative_description": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"source_refs": {
|
||||||
|
"type": "array",
|
||||||
|
"minItems": 1,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["source_id", "start_unit_id", "end_unit_id"],
|
||||||
|
"properties": {
|
||||||
|
"source_id": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"start_unit_id": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
},
|
||||||
|
"end_unit_id": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -17,6 +17,7 @@ var schemaAssets embed.FS
|
|||||||
type ResponseSchemaKey string
|
type ResponseSchemaKey string
|
||||||
|
|
||||||
const (
|
const (
|
||||||
|
DNDSpellsSchemaKey ResponseSchemaKey = "dnd_spells"
|
||||||
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
|
TestArtifactSchemaKey ResponseSchemaKey = "test_artifact"
|
||||||
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
|
TestValidatorDecisionSchemaKey ResponseSchemaKey = "test_validator_decision"
|
||||||
|
|
||||||
@@ -34,6 +35,13 @@ type ResponseSchema struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
var responseSchemaRegistry = map[ResponseSchemaKey]ResponseSchema{
|
var responseSchemaRegistry = map[ResponseSchemaKey]ResponseSchema{
|
||||||
|
DNDSpellsSchemaKey: mustLoadResponseSchema(
|
||||||
|
DNDSpellsSchemaKey,
|
||||||
|
"notarius.dnd.spells",
|
||||||
|
schemaVersionV1,
|
||||||
|
"notarius_dnd_spells_v1",
|
||||||
|
"assets/schemas/dnd_spells.v1.json",
|
||||||
|
),
|
||||||
TestArtifactSchemaKey: mustLoadResponseSchema(
|
TestArtifactSchemaKey: mustLoadResponseSchema(
|
||||||
TestArtifactSchemaKey,
|
TestArtifactSchemaKey,
|
||||||
"notarius.test_artifact",
|
"notarius.test_artifact",
|
||||||
|
|||||||
@@ -7,8 +7,9 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestLookupResponseSchemaSucceedsForTestSchemas(t *testing.T) {
|
func TestLookupResponseSchemaSucceedsForRegisteredSchemas(t *testing.T) {
|
||||||
tests := []ResponseSchemaKey{
|
tests := []ResponseSchemaKey{
|
||||||
|
DNDSpellsSchemaKey,
|
||||||
TestArtifactSchemaKey,
|
TestArtifactSchemaKey,
|
||||||
TestValidatorDecisionSchemaKey,
|
TestValidatorDecisionSchemaKey,
|
||||||
}
|
}
|
||||||
@@ -50,17 +51,22 @@ func TestMustLookupResponseSchemaPanicsForUnknownKey(t *testing.T) {
|
|||||||
|
|
||||||
func TestRegisteredResponseSchemasSortedByKey(t *testing.T) {
|
func TestRegisteredResponseSchemasSortedByKey(t *testing.T) {
|
||||||
schemas := RegisteredResponseSchemas()
|
schemas := RegisteredResponseSchemas()
|
||||||
if len(schemas) != 2 {
|
if len(schemas) != 3 {
|
||||||
t.Fatalf("expected two schemas, got %d", len(schemas))
|
t.Fatalf("expected three schemas, got %d", len(schemas))
|
||||||
}
|
}
|
||||||
|
|
||||||
keys := make([]string, len(schemas))
|
keys := make([]string, len(schemas))
|
||||||
|
seen := make(map[ResponseSchemaKey]bool, len(schemas))
|
||||||
for i, schema := range schemas {
|
for i, schema := range schemas {
|
||||||
keys[i] = string(schema.Key)
|
keys[i] = string(schema.Key)
|
||||||
|
seen[schema.Key] = true
|
||||||
}
|
}
|
||||||
if !sort.StringsAreSorted(keys) {
|
if !sort.StringsAreSorted(keys) {
|
||||||
t.Fatalf("expected sorted keys, got %v", keys)
|
t.Fatalf("expected sorted keys, got %v", keys)
|
||||||
}
|
}
|
||||||
|
if !seen[DNDSpellsSchemaKey] {
|
||||||
|
t.Fatalf("registered schemas = %v, want %q", keys, DNDSpellsSchemaKey)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestResponseSchemaContentIsValidJSON(t *testing.T) {
|
func TestResponseSchemaContentIsValidJSON(t *testing.T) {
|
||||||
@@ -72,26 +78,30 @@ func TestResponseSchemaContentIsValidJSON(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
|
func TestResponseSchemaJSONIsMutationSafe(t *testing.T) {
|
||||||
first := MustLookupResponseSchema(TestArtifactSchemaKey)
|
for _, key := range []ResponseSchemaKey{DNDSpellsSchemaKey, TestArtifactSchemaKey} {
|
||||||
first.JSONSchema[0] = '['
|
t.Run(string(key), func(t *testing.T) {
|
||||||
|
first := MustLookupResponseSchema(key)
|
||||||
|
first.JSONSchema[0] = '['
|
||||||
|
|
||||||
second := MustLookupResponseSchema(TestArtifactSchemaKey)
|
second := MustLookupResponseSchema(key)
|
||||||
if !json.Valid(second.JSONSchema) {
|
if !json.Valid(second.JSONSchema) {
|
||||||
t.Fatalf("schema JSON was mutated: %s", second.JSONSchema)
|
t.Fatalf("schema JSON was mutated: %s", second.JSONSchema)
|
||||||
}
|
}
|
||||||
if len(second.JSONSchema) > 0 && second.JSONSchema[0] == '[' {
|
if len(second.JSONSchema) > 0 && second.JSONSchema[0] == '[' {
|
||||||
t.Fatalf("schema JSON did not use defensive copy")
|
t.Fatalf("schema JSON did not use defensive copy")
|
||||||
}
|
}
|
||||||
|
|
||||||
registered := RegisteredResponseSchemas()
|
registered := RegisteredResponseSchemas()
|
||||||
for i := range registered {
|
for i := range registered {
|
||||||
if registered[i].Key == TestArtifactSchemaKey {
|
if registered[i].Key == key {
|
||||||
registered[i].JSONSchema[0] = '['
|
registered[i].JSONSchema[0] = '['
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
again := MustLookupResponseSchema(TestArtifactSchemaKey)
|
again := MustLookupResponseSchema(key)
|
||||||
if !json.Valid(again.JSONSchema) || again.JSONSchema[0] == '[' {
|
if !json.Valid(again.JSONSchema) || again.JSONSchema[0] == '[' {
|
||||||
t.Fatalf("registered schema JSON did not use defensive copy")
|
t.Fatalf("registered schema JSON did not use defensive copy")
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
9
internal/framework/prompt/assets/dnd/spells/system.md
Normal file
9
internal/framework/prompt/assets/dnd/spells/system.md
Normal file
@@ -0,0 +1,9 @@
|
|||||||
|
You extract D&D spell-cast artifacts from source units.
|
||||||
|
|
||||||
|
{{ hardening }}
|
||||||
|
|
||||||
|
Extract only spell casts that are supported by the provided source text. Do not
|
||||||
|
infer spells from general D&D knowledge or from table chatter that does not
|
||||||
|
identify a spell being cast.
|
||||||
|
|
||||||
|
Source references must use the source-unit IDs exactly as provided.
|
||||||
21
internal/framework/prompt/assets/dnd/spells/user.md
Normal file
21
internal/framework/prompt/assets/dnd/spells/user.md
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
Source document ID: {{ .SourceID }}
|
||||||
|
{{ if .HasChunk }}
|
||||||
|
Chunk ID: {{ .ChunkID }}
|
||||||
|
Chunk index: {{ .ChunkIndex }}
|
||||||
|
{{ end }}
|
||||||
|
|
||||||
|
Source units:
|
||||||
|
{{ range .Units }}
|
||||||
|
- Unit ID: {{ .ID }}
|
||||||
|
Text: {{ .Text }}
|
||||||
|
{{ if .Metadata }}
|
||||||
|
Metadata:
|
||||||
|
{{ range .Metadata }}
|
||||||
|
- {{ .Key }}: {{ .Value }}
|
||||||
|
{{ end }}
|
||||||
|
{{ end }}
|
||||||
|
{{ end }}
|
||||||
|
|
||||||
|
Return only D&D spell-cast artifacts. For each spell cast, identify the in-world
|
||||||
|
caster, spell name, effect, narrative description, and source references using
|
||||||
|
source_id, start_unit_id, and end_unit_id.
|
||||||
@@ -17,6 +17,7 @@ var embeddedAssets embed.FS
|
|||||||
const (
|
const (
|
||||||
SourceBuiltin = "builtin"
|
SourceBuiltin = "builtin"
|
||||||
VersionV1 = "v1"
|
VersionV1 = "v1"
|
||||||
|
DNDSpellsPromptID = "dnd.spells"
|
||||||
TestGenericPromptID = "test.generic"
|
TestGenericPromptID = "test.generic"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -65,6 +66,13 @@ func init() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
defs := []definition{
|
defs := []definition{
|
||||||
|
{
|
||||||
|
id: DNDSpellsPromptID,
|
||||||
|
version: VersionV1,
|
||||||
|
embeddedDir: "assets/dnd/spells",
|
||||||
|
systemPath: "assets/dnd/spells/system.md",
|
||||||
|
userPath: "assets/dnd/spells/user.md",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
id: TestGenericPromptID,
|
id: TestGenericPromptID,
|
||||||
version: VersionV1,
|
version: VersionV1,
|
||||||
|
|||||||
@@ -6,26 +6,38 @@ import (
|
|||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
func TestLookupMetadataSucceedsForGenericPrompt(t *testing.T) {
|
func TestLookupMetadataSucceedsForRegisteredPrompts(t *testing.T) {
|
||||||
metadata, ok := LookupMetadata(TestGenericPromptID)
|
tests := []struct {
|
||||||
if !ok {
|
promptID string
|
||||||
t.Fatalf("expected metadata for %q", TestGenericPromptID)
|
embeddedPath string
|
||||||
|
}{
|
||||||
|
{promptID: DNDSpellsPromptID, embeddedPath: "assets/dnd/spells"},
|
||||||
|
{promptID: TestGenericPromptID, embeddedPath: "assets/test/generic"},
|
||||||
}
|
}
|
||||||
|
|
||||||
if metadata.PromptID != TestGenericPromptID {
|
for _, tc := range tests {
|
||||||
t.Fatalf("unexpected prompt ID: %q", metadata.PromptID)
|
t.Run(tc.promptID, func(t *testing.T) {
|
||||||
}
|
metadata, ok := LookupMetadata(tc.promptID)
|
||||||
if metadata.PromptVersion != VersionV1 {
|
if !ok {
|
||||||
t.Fatalf("unexpected prompt version: %q", metadata.PromptVersion)
|
t.Fatalf("expected metadata for %q", tc.promptID)
|
||||||
}
|
}
|
||||||
if metadata.PromptSource != SourceBuiltin {
|
|
||||||
t.Fatalf("unexpected prompt source: %q", metadata.PromptSource)
|
if metadata.PromptID != tc.promptID {
|
||||||
}
|
t.Fatalf("unexpected prompt ID: %q", metadata.PromptID)
|
||||||
if metadata.EmbeddedPath != "assets/test/generic" {
|
}
|
||||||
t.Fatalf("unexpected embedded path: %q", metadata.EmbeddedPath)
|
if metadata.PromptVersion != VersionV1 {
|
||||||
}
|
t.Fatalf("unexpected prompt version: %q", metadata.PromptVersion)
|
||||||
if !strings.HasPrefix(metadata.SHA256, "sha256:") {
|
}
|
||||||
t.Fatalf("expected prefixed hash, got %q", metadata.SHA256)
|
if metadata.PromptSource != SourceBuiltin {
|
||||||
|
t.Fatalf("unexpected prompt source: %q", metadata.PromptSource)
|
||||||
|
}
|
||||||
|
if metadata.EmbeddedPath != tc.embeddedPath {
|
||||||
|
t.Fatalf("unexpected embedded path: %q", metadata.EmbeddedPath)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(metadata.SHA256, "sha256:") {
|
||||||
|
t.Fatalf("expected prefixed hash, got %q", metadata.SHA256)
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -47,17 +59,22 @@ func TestMustLookupMetadataPanicsForUnknownPromptID(t *testing.T) {
|
|||||||
|
|
||||||
func TestRegisteredMetadataSortedByPromptID(t *testing.T) {
|
func TestRegisteredMetadataSortedByPromptID(t *testing.T) {
|
||||||
registered := RegisteredMetadata()
|
registered := RegisteredMetadata()
|
||||||
if len(registered) != 1 {
|
if len(registered) != 2 {
|
||||||
t.Fatalf("expected one registered prompt, got %d", len(registered))
|
t.Fatalf("expected two registered prompts, got %d", len(registered))
|
||||||
}
|
}
|
||||||
|
|
||||||
ids := make([]string, len(registered))
|
ids := make([]string, len(registered))
|
||||||
|
seen := make(map[string]bool, len(registered))
|
||||||
for i, metadata := range registered {
|
for i, metadata := range registered {
|
||||||
ids[i] = metadata.PromptID
|
ids[i] = metadata.PromptID
|
||||||
|
seen[metadata.PromptID] = true
|
||||||
}
|
}
|
||||||
if !sort.StringsAreSorted(ids) {
|
if !sort.StringsAreSorted(ids) {
|
||||||
t.Fatalf("expected sorted prompt IDs, got %v", ids)
|
t.Fatalf("expected sorted prompt IDs, got %v", ids)
|
||||||
}
|
}
|
||||||
|
if !seen[DNDSpellsPromptID] {
|
||||||
|
t.Fatalf("registered prompt IDs = %v, want %q", ids, DNDSpellsPromptID)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestHardeningTextAvailable(t *testing.T) {
|
func TestHardeningTextAvailable(t *testing.T) {
|
||||||
|
|||||||
@@ -64,3 +64,47 @@ func TestRenderUserSystemIncludesHardeningText(t *testing.T) {
|
|||||||
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
|
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRenderDNDSpellsPromptIncludesHardeningText(t *testing.T) {
|
||||||
|
system, user, metadata, err := RenderUserSystem(DNDSpellsPromptID, map[string]any{
|
||||||
|
"SourceID": "session-alpha",
|
||||||
|
"HasChunk": true,
|
||||||
|
"ChunkID": "session-alpha:chunk:0",
|
||||||
|
"ChunkIndex": 0,
|
||||||
|
"Units": []map[string]any{
|
||||||
|
{
|
||||||
|
"ID": "seg-001",
|
||||||
|
"Text": "Aria casts Cure Wounds.",
|
||||||
|
"Metadata": []map[string]string{
|
||||||
|
{"Key": "speaker", "Value": "Alice"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("RenderUserSystem: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
hardening := strings.TrimSpace(HardeningText())
|
||||||
|
if hardening == "" {
|
||||||
|
t.Fatalf("expected hardening text")
|
||||||
|
}
|
||||||
|
if !strings.Contains(system, hardening) {
|
||||||
|
t.Fatalf("expected rendered system prompt to include hardening text: %q", system)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"session-alpha", "session-alpha:chunk:0", "seg-001", "Aria casts Cure Wounds.", "speaker: Alice"} {
|
||||||
|
if !strings.Contains(user, want) {
|
||||||
|
t.Fatalf("rendered user prompt = %q, want substring %q", user, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if metadata.PromptID != DNDSpellsPromptID {
|
||||||
|
t.Fatalf("unexpected metadata: %+v", metadata)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderDNDSpellsPromptMissingTemplateDataReturnsError(t *testing.T) {
|
||||||
|
_, _, _, err := RenderUserSystem(DNDSpellsPromptID, map[string]any{})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "SourceID") {
|
||||||
|
t.Fatalf("expected missing SourceID error, got %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|||||||
276
internal/modules/extract/dnd/spells/config_test.go
Normal file
276
internal/modules/extract/dnd/spells/config_test.go
Normal file
@@ -0,0 +1,276 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestPipelineConfigLoadsAndResolvesWithDNDSpellsExtractor(t *testing.T) {
|
||||||
|
data, err := os.ReadFile("testdata/pipeline.yml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(pipeline.yml) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
fileCfg, err := config.ParseFileConfigYAML(data)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg := config.Default()
|
||||||
|
if err := cfg.ApplyFileConfig(fileCfg); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
resolved, err := cfg.Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "dnd-spells-fixture",
|
||||||
|
Catalog: dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{}),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(resolved.ResolvedPipeline.ArtifactLanes) != 1 {
|
||||||
|
t.Fatalf("len(ArtifactLanes) = %d, want 1", len(resolved.ResolvedPipeline.ArtifactLanes))
|
||||||
|
}
|
||||||
|
lane := resolved.ResolvedPipeline.ArtifactLanes[0]
|
||||||
|
if lane.ID != "spells" {
|
||||||
|
t.Fatalf("lane ID = %q, want spells", lane.ID)
|
||||||
|
}
|
||||||
|
if lane.Extract.Module != Key {
|
||||||
|
t.Fatalf("extract module = %q, want %q", lane.Extract.Module, Key)
|
||||||
|
}
|
||||||
|
if resolved.ResolvedPipeline.Digest == "" {
|
||||||
|
t.Fatal("resolved digest is empty")
|
||||||
|
}
|
||||||
|
|
||||||
|
again, err := cfg.Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "dnd-spells-fixture",
|
||||||
|
Catalog: dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{}),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("second Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if resolved.ResolvedPipeline.Digest != again.ResolvedPipeline.Digest {
|
||||||
|
t.Fatalf("resolved digest = %q, second digest = %q; want stable digest", resolved.ResolvedPipeline.Digest, again.ResolvedPipeline.Digest)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPipelineConfigRejectsMissingTranscriptCapabilityForDNDSpells(t *testing.T) {
|
||||||
|
inputSpec := seriatim.ModuleSpec()
|
||||||
|
inputSpec.Provides = withoutCapability(inputSpec.Provides, "source.transcript")
|
||||||
|
chunkSpec := dndSpellsChunkerSpec()
|
||||||
|
chunkSpec.Requires = nil
|
||||||
|
|
||||||
|
_, err := loadDNDSpellsPipelineConfig(t).Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "dnd-spells-fixture",
|
||||||
|
Catalog: dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{
|
||||||
|
input: inputSpec,
|
||||||
|
chunk: chunkSpec,
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Resolve() error = nil, want missing capability error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "missing capability") ||
|
||||||
|
!strings.Contains(err.Error(), "source.transcript") ||
|
||||||
|
!strings.Contains(err.Error(), Key) {
|
||||||
|
t.Fatalf("Resolve() error = %q, want dnd/spells missing source.transcript capability", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPipelineConfigRejectsMissingSpellCastsCapabilityForAppendOrder(t *testing.T) {
|
||||||
|
extractorSpec := ModuleSpec()
|
||||||
|
extractorSpec.Provides = withoutCapability(extractorSpec.Provides, "dnd.spell_casts")
|
||||||
|
|
||||||
|
_, err := loadDNDSpellsPipelineConfig(t).Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "dnd-spells-fixture",
|
||||||
|
Catalog: dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{
|
||||||
|
extractor: extractorSpec,
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Resolve() error = nil, want missing capability error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "missing capability") ||
|
||||||
|
!strings.Contains(err.Error(), "dnd.spell_casts") ||
|
||||||
|
!strings.Contains(err.Error(), pipeline.DefaultMergeModule) {
|
||||||
|
t.Fatalf("Resolve() error = %q, want appendorder missing dnd.spell_casts capability", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPipelineConfigRejectsUnknownLaneSelection(t *testing.T) {
|
||||||
|
_, err := loadDNDSpellsPipelineConfig(t).Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "dnd-spells-fixture",
|
||||||
|
Only: []string{"missing"},
|
||||||
|
Catalog: dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{}),
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Resolve() error = nil, want unknown lane error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "selected artifact lane") || !strings.Contains(err.Error(), "missing") {
|
||||||
|
t.Fatalf("Resolve() error = %q, want unknown lane context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func loadDNDSpellsPipelineConfig(t *testing.T) config.Config {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
data, err := os.ReadFile("testdata/pipeline.yml")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(pipeline.yml) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
fileCfg, err := config.ParseFileConfigYAML(data)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ParseFileConfigYAML() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
cfg := config.Default()
|
||||||
|
if err := cfg.ApplyFileConfig(fileCfg); err != nil {
|
||||||
|
t.Fatalf("ApplyFileConfig() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return cfg
|
||||||
|
}
|
||||||
|
|
||||||
|
type dndSpellsCatalogSpecs struct {
|
||||||
|
input pipeline.ModuleSpec
|
||||||
|
chunk pipeline.ModuleSpec
|
||||||
|
extractor pipeline.ModuleSpec
|
||||||
|
}
|
||||||
|
|
||||||
|
func dndSpellsTestCatalog(t *testing.T, specs dndSpellsCatalogSpecs) pipeline.ModuleCatalog {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
inputs := pipeline.NewInputAdapterRegistry()
|
||||||
|
chunkers := pipeline.NewChunkerRegistry()
|
||||||
|
extractors := pipeline.NewExtractorRegistry()
|
||||||
|
mergers := pipeline.NewMergerRegistry()
|
||||||
|
normalizers := pipeline.NewNormalizerRegistry()
|
||||||
|
outputs := pipeline.NewOutputEncoderRegistry()
|
||||||
|
|
||||||
|
if specs.input.Key == "" {
|
||||||
|
if err := seriatim.Register(inputs); err != nil {
|
||||||
|
t.Fatalf("register seriatim input: %v", err)
|
||||||
|
}
|
||||||
|
} else if err := inputs.RegisterWithSpec(specs.input, func() (contracts.InputAdapter, error) {
|
||||||
|
return seriatim.New(), nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register seriatim input override: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
chunkSpec := specs.chunk
|
||||||
|
if chunkSpec.Key == "" {
|
||||||
|
chunkSpec = dndSpellsChunkerSpec()
|
||||||
|
}
|
||||||
|
if err := chunkers.RegisterWithSpec(chunkSpec, func() (contracts.Chunker, error) {
|
||||||
|
return dndSpellsChunker{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register chunker: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if specs.extractor.Key == "" {
|
||||||
|
if err := Register(extractors); err != nil {
|
||||||
|
t.Fatalf("register dnd spells extractor: %v", err)
|
||||||
|
}
|
||||||
|
} else if err := extractors.RegisterWithSpec(specs.extractor, func() (contracts.Extractor, error) {
|
||||||
|
return New(), nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register dnd spells extractor override: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := mergers.RegisterWithSpec(pipeline.ModuleSpec{
|
||||||
|
Key: pipeline.DefaultMergeModule,
|
||||||
|
Stage: pipeline.StageMerge,
|
||||||
|
Requires: []string{"dnd.spell_casts"},
|
||||||
|
}, func() (contracts.Merger, error) {
|
||||||
|
return pipeline.AppendOrderMerger{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register merger: %v", err)
|
||||||
|
}
|
||||||
|
if err := normalizers.RegisterWithSpec(pipeline.ModuleSpec{
|
||||||
|
Key: pipeline.DefaultNormalizeModule,
|
||||||
|
Stage: pipeline.StageNormalize,
|
||||||
|
}, func() (contracts.Normalizer, error) {
|
||||||
|
return pipeline.NoopNormalizer{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register normalizer: %v", err)
|
||||||
|
}
|
||||||
|
if err := outputs.RegisterWithSpec(pipeline.ModuleSpec{
|
||||||
|
Key: pipeline.DefaultOutputModule,
|
||||||
|
Stage: pipeline.StageOutput,
|
||||||
|
}, func() (contracts.OutputEncoder, error) {
|
||||||
|
return dndSpellsOutput{}, nil
|
||||||
|
}); err != nil {
|
||||||
|
t.Fatalf("register output: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return pipeline.ModuleCatalog{
|
||||||
|
Inputs: inputs,
|
||||||
|
Chunkers: chunkers,
|
||||||
|
Extractors: extractors,
|
||||||
|
Mergers: mergers,
|
||||||
|
Normalizers: normalizers,
|
||||||
|
Outputs: outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func dndSpellsChunkerSpec() pipeline.ModuleSpec {
|
||||||
|
return pipeline.ModuleSpec{
|
||||||
|
Key: "fake/chunk",
|
||||||
|
Stage: pipeline.StageChunk,
|
||||||
|
Requires: []string{"source.transcript"},
|
||||||
|
Provides: []string{"chunks"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
type dndSpellsChunker struct{}
|
||||||
|
|
||||||
|
func (dndSpellsChunker) Key() string {
|
||||||
|
return "fake/chunk"
|
||||||
|
}
|
||||||
|
|
||||||
|
func (dndSpellsChunker) Chunk(ctx context.Context, req contracts.ChunkRequest) (contracts.ChunkResult, error) {
|
||||||
|
return contracts.ChunkResult{
|
||||||
|
Chunks: []contracts.SourceChunk{
|
||||||
|
{
|
||||||
|
ID: req.Source.ID + ":chunk:0",
|
||||||
|
SourceID: req.Source.ID,
|
||||||
|
Index: 0,
|
||||||
|
Units: append([]source.SourceUnit(nil), req.Source.Units...),
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type dndSpellsOutput struct{}
|
||||||
|
|
||||||
|
func (dndSpellsOutput) Key() string {
|
||||||
|
return pipeline.DefaultOutputModule
|
||||||
|
}
|
||||||
|
|
||||||
|
func (dndSpellsOutput) Encode(ctx context.Context, req contracts.OutputRequest) (contracts.OutputResult, error) {
|
||||||
|
return contracts.OutputResult{
|
||||||
|
Bytes: []byte(`{"encoded":true}`),
|
||||||
|
ContentType: "application/json",
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func withoutCapability(capabilities []string, capability string) []string {
|
||||||
|
filtered := make([]string, 0, len(capabilities))
|
||||||
|
for _, candidate := range capabilities {
|
||||||
|
if candidate != capability {
|
||||||
|
filtered = append(filtered, candidate)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return filtered
|
||||||
|
}
|
||||||
|
|
||||||
|
var (
|
||||||
|
_ contracts.Chunker = dndSpellsChunker{}
|
||||||
|
_ contracts.OutputEncoder = dndSpellsOutput{}
|
||||||
|
)
|
||||||
147
internal/modules/extract/dnd/spells/extractor.go
Normal file
147
internal/modules/extract/dnd/spells/extractor.go
Normal file
@@ -0,0 +1,147 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Key = "dnd/spells"
|
||||||
|
const ArtifactType = "dnd.spell_cast"
|
||||||
|
const SchemaVersion = "v1"
|
||||||
|
|
||||||
|
var requiredCapabilities = []string{
|
||||||
|
"chunks",
|
||||||
|
"source.transcript",
|
||||||
|
}
|
||||||
|
|
||||||
|
var providedCapabilities = []string{
|
||||||
|
"dnd.spell_casts",
|
||||||
|
}
|
||||||
|
|
||||||
|
var _ contracts.Extractor = (*Extractor)(nil)
|
||||||
|
|
||||||
|
type Extractor struct{}
|
||||||
|
|
||||||
|
func New() *Extractor {
|
||||||
|
return &Extractor{}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) Key() string {
|
||||||
|
return Key
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) ArtifactType() string {
|
||||||
|
return ArtifactType
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) SchemaVersion() string {
|
||||||
|
return SchemaVersion
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) Validators() []contracts.Validator {
|
||||||
|
return []contracts.Validator{
|
||||||
|
ShapeValidator{},
|
||||||
|
SourceRefValidator{},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e *Extractor) Extract(ctx context.Context, req contracts.ExtractionRequest) (contracts.ExtractionResult, error) {
|
||||||
|
if e == nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("extractor must not be nil")
|
||||||
|
}
|
||||||
|
if ctx == nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("context must not be nil")
|
||||||
|
}
|
||||||
|
if err := ctx.Err(); err != nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("context error before extraction: %w", err)
|
||||||
|
}
|
||||||
|
if req.Source == nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("source must not be nil")
|
||||||
|
}
|
||||||
|
if req.Chunk == nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("chunk must not be nil")
|
||||||
|
}
|
||||||
|
if len(req.Chunk.Units) == 0 {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("chunk %q units must not be empty", req.Chunk.ID)
|
||||||
|
}
|
||||||
|
if req.LLMClient == nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("LLM client must not be nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
system, user, _, err := renderPrompt(req)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("render prompt: %w", err)
|
||||||
|
}
|
||||||
|
schema, ok := llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)
|
||||||
|
if !ok {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("lookup response schema %q", llm.DNDSpellsSchemaKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
var response extractionResponse
|
||||||
|
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||||
|
StageName: Key,
|
||||||
|
Messages: []contracts.LLMMessage{
|
||||||
|
{Role: "system", Content: system},
|
||||||
|
{Role: "user", Content: user},
|
||||||
|
},
|
||||||
|
ResponseSchemaName: schema.Name,
|
||||||
|
ResponseSchema: schema.JSONSchema,
|
||||||
|
}, &response); err != nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("complete structured output: %w", err)
|
||||||
|
}
|
||||||
|
if response.SpellCasts == nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("malformed structured output: spell_casts must be present")
|
||||||
|
}
|
||||||
|
if len(response.SpellCasts) == 0 {
|
||||||
|
return contracts.ExtractionResult{}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
candidates := make([]artifacts.ArtifactCandidate, 0, len(response.SpellCasts))
|
||||||
|
for i, spellCast := range response.SpellCasts {
|
||||||
|
payload, err := spellCastPayload(spellCast)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.ExtractionResult{}, extractorErrorf("marshal spell cast[%d]: %w", i, err)
|
||||||
|
}
|
||||||
|
candidates = append(candidates, artifacts.ArtifactCandidate{
|
||||||
|
Payload: payload,
|
||||||
|
SourceRefs: append([]source.SourceRef(nil), spellCast.SourceRefs...),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return contracts.ExtractionResult{Candidates: candidates}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ModuleSpec() pipeline.ModuleSpec {
|
||||||
|
return pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
Requires: append([]string(nil), requiredCapabilities...),
|
||||||
|
Provides: append([]string(nil), providedCapabilities...),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func Register(registry *pipeline.ExtractorRegistry) error {
|
||||||
|
return registry.RegisterWithSpec(ModuleSpec(), func() (contracts.Extractor, error) {
|
||||||
|
return New(), nil
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func spellCastPayload(spellCast spellCastResponse) (json.RawMessage, error) {
|
||||||
|
return json.Marshal(SpellCast{
|
||||||
|
Caster: strings.TrimSpace(spellCast.Caster),
|
||||||
|
Spell: strings.TrimSpace(spellCast.Spell),
|
||||||
|
Effect: strings.TrimSpace(spellCast.Effect),
|
||||||
|
NarrativeDescription: strings.TrimSpace(spellCast.NarrativeDescription),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractorErrorf(format string, args ...any) error {
|
||||||
|
return fmt.Errorf("dnd spells extractor: "+format, args...)
|
||||||
|
}
|
||||||
275
internal/modules/extract/dnd/spells/extractor_test.go
Normal file
275
internal/modules/extract/dnd/spells/extractor_test.go
Normal file
@@ -0,0 +1,275 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{
|
||||||
|
response: extractionResponse{
|
||||||
|
SpellCasts: []spellCastResponse{
|
||||||
|
{
|
||||||
|
Caster: " Aria ",
|
||||||
|
Spell: " Cure Wounds ",
|
||||||
|
Effect: " Heals an injured ally. ",
|
||||||
|
NarrativeDescription: " Aria restores the fighter after the fight. ",
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(client.requests) != 1 {
|
||||||
|
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
||||||
|
}
|
||||||
|
req := client.requests[0]
|
||||||
|
if req.StageName != Key {
|
||||||
|
t.Fatalf("StageName = %q, want %q", req.StageName, Key)
|
||||||
|
}
|
||||||
|
schema := llm.MustLookupResponseSchema(llm.DNDSpellsSchemaKey)
|
||||||
|
if req.ResponseSchemaName != schema.Name {
|
||||||
|
t.Fatalf("ResponseSchemaName = %q, want %q", req.ResponseSchemaName, schema.Name)
|
||||||
|
}
|
||||||
|
if !bytes.Equal(req.ResponseSchema, schema.JSONSchema) {
|
||||||
|
t.Fatal("ResponseSchema does not match registered D&D spells schema")
|
||||||
|
}
|
||||||
|
if len(req.Messages) != 2 {
|
||||||
|
t.Fatalf("len(Messages) = %d, want 2", len(req.Messages))
|
||||||
|
}
|
||||||
|
if req.Messages[0].Role != "system" || req.Messages[1].Role != "user" {
|
||||||
|
t.Fatalf("Messages roles = %#v, want system then user", req.Messages)
|
||||||
|
}
|
||||||
|
if !strings.Contains(req.Messages[0].Content, "D&D spell-cast") {
|
||||||
|
t.Fatalf("system message = %q, want D&D spell context", req.Messages[0].Content)
|
||||||
|
}
|
||||||
|
for _, want := range []string{"session-alpha", "session-alpha:chunk:0", "seg-001", "Cure Wounds"} {
|
||||||
|
if !strings.Contains(req.Messages[1].Content, want) {
|
||||||
|
t.Fatalf("user message = %q, want substring %q", req.Messages[1].Content, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(result.Candidates) != 1 {
|
||||||
|
t.Fatalf("len(Candidates) = %d, want 1", len(result.Candidates))
|
||||||
|
}
|
||||||
|
candidate := result.Candidates[0]
|
||||||
|
if candidate.Index != 0 || candidate.ExtractorKey != "" || candidate.ArtifactType != "" || candidate.SchemaVersion != "" {
|
||||||
|
t.Fatalf("candidate envelope fields = %#v, want runner-normalized zero values", candidate)
|
||||||
|
}
|
||||||
|
var payload SpellCast
|
||||||
|
if err := json.Unmarshal(candidate.Payload, &payload); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(Payload) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
wantPayload := SpellCast{
|
||||||
|
Caster: "Aria",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Heals an injured ally.",
|
||||||
|
NarrativeDescription: "Aria restores the fighter after the fight.",
|
||||||
|
}
|
||||||
|
if payload != wantPayload {
|
||||||
|
t.Fatalf("payload = %#v, want %#v", payload, wantPayload)
|
||||||
|
}
|
||||||
|
wantRef := source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"}
|
||||||
|
if len(candidate.SourceRefs) != 1 || candidate.SourceRefs[0] != wantRef {
|
||||||
|
t.Fatalf("SourceRefs = %#v, want %#v", candidate.SourceRefs, []source.SourceRef{wantRef})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractReturnsNoCandidatesForEmptyResponse(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
|
||||||
|
|
||||||
|
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(result.Candidates) != 0 {
|
||||||
|
t.Fatalf("Candidates = %#v, want none", result.Candidates)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractRejectsMissingSpellCasts(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{response: extractionResponse{}}
|
||||||
|
|
||||||
|
_, err := New().Extract(context.Background(), extractionRequestWithClient(client))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Extract() error = nil, want malformed output error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "dnd spells") || !strings.Contains(err.Error(), "spell_casts") {
|
||||||
|
t.Fatalf("Extract() error = %q, want spell_casts context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractWrapsLLMClientError(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{err: errors.New("provider unavailable")}
|
||||||
|
|
||||||
|
_, err := New().Extract(context.Background(), extractionRequestWithClient(client))
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Extract() error = nil, want LLM error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "dnd spells") || !strings.Contains(err.Error(), "provider unavailable") {
|
||||||
|
t.Fatalf("Extract() error = %q, want wrapped LLM context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractRejectsInvalidRequests(t *testing.T) {
|
||||||
|
validClient := &fakeSpellsLLMClient{response: extractionResponse{SpellCasts: []spellCastResponse{}}}
|
||||||
|
validReq := extractionRequestWithClient(validClient)
|
||||||
|
canceledCtx, cancel := context.WithCancel(context.Background())
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
extractor *Extractor
|
||||||
|
ctx context.Context
|
||||||
|
req contracts.ExtractionRequest
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{name: "nil extractor", extractor: nil, ctx: context.Background(), req: validReq, want: "extractor"},
|
||||||
|
{name: "nil context", extractor: New(), ctx: nil, req: validReq, want: "context"},
|
||||||
|
{name: "canceled context", extractor: New(), ctx: canceledCtx, req: validReq, want: "context"},
|
||||||
|
{name: "nil source", extractor: New(), ctx: context.Background(), req: contracts.ExtractionRequest{Chunk: validReq.Chunk, LLMClient: validReq.LLMClient}, want: "source"},
|
||||||
|
{name: "nil chunk", extractor: New(), ctx: context.Background(), req: contracts.ExtractionRequest{Source: validReq.Source, LLMClient: validReq.LLMClient}, want: "chunk"},
|
||||||
|
{name: "empty chunk units", extractor: New(), ctx: context.Background(), req: emptyChunkRequest(validReq), want: "units"},
|
||||||
|
{name: "nil LLM client", extractor: New(), ctx: context.Background(), req: contracts.ExtractionRequest{Source: validReq.Source, Chunk: validReq.Chunk}, want: "LLM client"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
_, err := tt.extractor.Extract(tt.ctx, tt.req)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Extract() error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "dnd spells") || !strings.Contains(err.Error(), tt.want) {
|
||||||
|
t.Fatalf("Extract() error = %q, want %q context", err.Error(), tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractPreservesResponseOrder(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{
|
||||||
|
response: extractionResponse{
|
||||||
|
SpellCasts: []spellCastResponse{
|
||||||
|
{
|
||||||
|
Caster: "Aria",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Heals.",
|
||||||
|
NarrativeDescription: "First spell.",
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-001"}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Caster: "Bandit Shaman",
|
||||||
|
Spell: "Fire Bolt",
|
||||||
|
Effect: "Burns.",
|
||||||
|
NarrativeDescription: "Second spell.",
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: "seg-002", EndUnitID: "seg-002"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(result.Candidates) != 2 {
|
||||||
|
t.Fatalf("len(Candidates) = %d, want 2", len(result.Candidates))
|
||||||
|
}
|
||||||
|
|
||||||
|
var first, second SpellCast
|
||||||
|
if err := json.Unmarshal(result.Candidates[0].Payload, &first); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(first) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(result.Candidates[1].Payload, &second); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(second) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if first.Spell != "Cure Wounds" || second.Spell != "Fire Bolt" {
|
||||||
|
t.Fatalf("candidate order = %q, %q; want response order", first.Spell, second.Spell)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExtractCopiesCandidateSourceRefs(t *testing.T) {
|
||||||
|
client := &fakeSpellsLLMClient{
|
||||||
|
response: extractionResponse{
|
||||||
|
SpellCasts: []spellCastResponse{
|
||||||
|
{
|
||||||
|
Caster: "Aria",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Heals.",
|
||||||
|
NarrativeDescription: "Aria heals.",
|
||||||
|
SourceRefs: []source.SourceRef{{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"}},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := New().Extract(context.Background(), extractionRequestWithClient(client))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Extract() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
client.response.SpellCasts[0].SourceRefs[0].StartUnitID = "mutated"
|
||||||
|
|
||||||
|
if got := result.Candidates[0].SourceRefs[0].StartUnitID; got != "seg-001" {
|
||||||
|
t.Fatalf("candidate source ref start = %q, want copied seg-001", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func extractionRequestWithClient(client contracts.StructuredLLMClient) contracts.ExtractionRequest {
|
||||||
|
req := promptExtractionRequest()
|
||||||
|
req.LLMClient = client
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
func emptyChunkRequest(req contracts.ExtractionRequest) contracts.ExtractionRequest {
|
||||||
|
req.Chunk = &contracts.SourceChunk{
|
||||||
|
ID: req.Chunk.ID,
|
||||||
|
SourceID: req.Chunk.SourceID,
|
||||||
|
Index: req.Chunk.Index,
|
||||||
|
}
|
||||||
|
return req
|
||||||
|
}
|
||||||
|
|
||||||
|
type fakeSpellsLLMClient struct {
|
||||||
|
response extractionResponse
|
||||||
|
err error
|
||||||
|
requests []contracts.StructuredCompletionRequest
|
||||||
|
}
|
||||||
|
|
||||||
|
func (client *fakeSpellsLLMClient) CompleteStructured(ctx context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||||
|
client.requests = append(client.requests, contracts.StructuredCompletionRequest{
|
||||||
|
StageName: req.StageName,
|
||||||
|
Messages: append([]contracts.LLMMessage(nil), req.Messages...),
|
||||||
|
Model: req.Model,
|
||||||
|
ResponseSchemaName: req.ResponseSchemaName,
|
||||||
|
ResponseSchema: append(json.RawMessage(nil), req.ResponseSchema...),
|
||||||
|
})
|
||||||
|
if client.err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, client.err
|
||||||
|
}
|
||||||
|
|
||||||
|
target, ok := out.(*extractionResponse)
|
||||||
|
if !ok {
|
||||||
|
return contracts.StructuredCompletionResponse{}, errors.New("unexpected output target")
|
||||||
|
}
|
||||||
|
*target = client.response
|
||||||
|
content, err := json.Marshal(client.response)
|
||||||
|
if err != nil {
|
||||||
|
return contracts.StructuredCompletionResponse{}, err
|
||||||
|
}
|
||||||
|
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||||
|
}
|
||||||
22
internal/modules/extract/dnd/spells/model.go
Normal file
22
internal/modules/extract/dnd/spells/model.go
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import "gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
|
||||||
|
type SpellCast struct {
|
||||||
|
Caster string `json:"caster"`
|
||||||
|
Spell string `json:"spell"`
|
||||||
|
Effect string `json:"effect"`
|
||||||
|
NarrativeDescription string `json:"narrative_description"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type extractionResponse struct {
|
||||||
|
SpellCasts []spellCastResponse `json:"spell_casts"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type spellCastResponse struct {
|
||||||
|
Caster string `json:"caster"`
|
||||||
|
Spell string `json:"spell"`
|
||||||
|
Effect string `json:"effect"`
|
||||||
|
NarrativeDescription string `json:"narrative_description"`
|
||||||
|
SourceRefs []source.SourceRef `json:"source_refs"`
|
||||||
|
}
|
||||||
90
internal/modules/extract/dnd/spells/prompt.go
Normal file
90
internal/modules/extract/dnd/spells/prompt.go
Normal file
@@ -0,0 +1,90 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/prompt"
|
||||||
|
)
|
||||||
|
|
||||||
|
type promptData struct {
|
||||||
|
SourceID string
|
||||||
|
HasChunk bool
|
||||||
|
ChunkID string
|
||||||
|
ChunkIndex int
|
||||||
|
Units []promptUnit
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptUnit struct {
|
||||||
|
ID string
|
||||||
|
Text string
|
||||||
|
Metadata []promptMetadata
|
||||||
|
}
|
||||||
|
|
||||||
|
type promptMetadata struct {
|
||||||
|
Key string
|
||||||
|
Value string
|
||||||
|
}
|
||||||
|
|
||||||
|
func buildPromptData(req contracts.ExtractionRequest) (promptData, error) {
|
||||||
|
if req.Source == nil {
|
||||||
|
return promptData{}, fmt.Errorf("dnd spells prompt: source must not be nil")
|
||||||
|
}
|
||||||
|
if req.Chunk == nil {
|
||||||
|
return promptData{}, fmt.Errorf("dnd spells prompt: chunk must not be nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
data := promptData{
|
||||||
|
SourceID: req.Source.ID,
|
||||||
|
HasChunk: true,
|
||||||
|
ChunkID: req.Chunk.ID,
|
||||||
|
ChunkIndex: req.Chunk.Index,
|
||||||
|
Units: make([]promptUnit, 0, len(req.Chunk.Units)),
|
||||||
|
}
|
||||||
|
for _, unit := range req.Chunk.Units {
|
||||||
|
data.Units = append(data.Units, promptUnit{
|
||||||
|
ID: unit.ID,
|
||||||
|
Text: unit.Text,
|
||||||
|
Metadata: selectedMetadata(unit),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return data, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func renderPrompt(req contracts.ExtractionRequest) (system string, user string, metadata prompt.Metadata, err error) {
|
||||||
|
data, err := buildPromptData(req)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", prompt.Metadata{}, err
|
||||||
|
}
|
||||||
|
system, user, metadata, err = prompt.RenderUserSystem(prompt.DNDSpellsPromptID, data)
|
||||||
|
if err != nil {
|
||||||
|
return "", "", prompt.Metadata{}, fmt.Errorf("dnd spells prompt: %w", err)
|
||||||
|
}
|
||||||
|
return system, user, metadata, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func selectedMetadata(unit source.SourceUnit) []promptMetadata {
|
||||||
|
if len(unit.Metadata) == 0 {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
keys := []string{"speaker", "start", "end"}
|
||||||
|
metadata := make([]promptMetadata, 0, len(keys))
|
||||||
|
for _, key := range keys {
|
||||||
|
value, ok := unit.Metadata[key]
|
||||||
|
if !ok {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
rendered := strings.TrimSpace(fmt.Sprint(value))
|
||||||
|
if rendered == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
metadata = append(metadata, promptMetadata{
|
||||||
|
Key: key,
|
||||||
|
Value: rendered,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
return metadata
|
||||||
|
}
|
||||||
159
internal/modules/extract/dnd/spells/prompt_test.go
Normal file
159
internal/modules/extract/dnd/spells/prompt_test.go
Normal file
@@ -0,0 +1,159 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/prompt"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestBuildPromptDataFromGenericSourceChunk(t *testing.T) {
|
||||||
|
req := promptExtractionRequest()
|
||||||
|
|
||||||
|
data, err := buildPromptData(req)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("buildPromptData() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if data.SourceID != "session-alpha" {
|
||||||
|
t.Fatalf("SourceID = %q, want session-alpha", data.SourceID)
|
||||||
|
}
|
||||||
|
if !data.HasChunk || data.ChunkID != "session-alpha:chunk:0" || data.ChunkIndex != 0 {
|
||||||
|
t.Fatalf("chunk data = %#v, want fixture chunk", data)
|
||||||
|
}
|
||||||
|
if len(data.Units) != 2 {
|
||||||
|
t.Fatalf("len(Units) = %d, want 2", len(data.Units))
|
||||||
|
}
|
||||||
|
first := data.Units[0]
|
||||||
|
if first.ID != "seg-001" || first.Text != "Aria raises her hand and casts Cure Wounds." {
|
||||||
|
t.Fatalf("first unit = %#v, want source unit data", first)
|
||||||
|
}
|
||||||
|
wantMetadata := []promptMetadata{
|
||||||
|
{Key: "speaker", Value: "Alice"},
|
||||||
|
{Key: "start", Value: "1.25"},
|
||||||
|
{Key: "end", Value: "3.5"},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(first.Metadata, wantMetadata) {
|
||||||
|
t.Fatalf("first.Metadata = %#v, want %#v", first.Metadata, wantMetadata)
|
||||||
|
}
|
||||||
|
if len(data.Units[1].Metadata) != 0 {
|
||||||
|
t.Fatalf("second.Metadata = %#v, want no selected metadata", data.Units[1].Metadata)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPromptDataDoesNotMutateRequest(t *testing.T) {
|
||||||
|
req := promptExtractionRequest()
|
||||||
|
beforeSource := mustJSON(t, req.Source)
|
||||||
|
beforeChunk := mustJSON(t, req.Chunk)
|
||||||
|
beforeRequest := mustJSON(t, req)
|
||||||
|
|
||||||
|
if _, err := buildPromptData(req); err != nil {
|
||||||
|
t.Fatalf("buildPromptData() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
afterSource := mustJSON(t, req.Source)
|
||||||
|
afterChunk := mustJSON(t, req.Chunk)
|
||||||
|
afterRequest := mustJSON(t, req)
|
||||||
|
if beforeSource != afterSource || beforeChunk != afterChunk || beforeRequest != afterRequest {
|
||||||
|
t.Fatalf(
|
||||||
|
"request mutated:\nsource before: %s\nsource after: %s\nchunk before: %s\nchunk after: %s\nrequest before: %s\nrequest after: %s",
|
||||||
|
beforeSource,
|
||||||
|
afterSource,
|
||||||
|
beforeChunk,
|
||||||
|
afterChunk,
|
||||||
|
beforeRequest,
|
||||||
|
afterRequest,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRenderPromptIncludesSourceContext(t *testing.T) {
|
||||||
|
system, user, metadata, err := renderPrompt(promptExtractionRequest())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("renderPrompt() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(system, prompt.HardeningText()) {
|
||||||
|
t.Fatalf("system prompt = %q, want hardening text", system)
|
||||||
|
}
|
||||||
|
for _, want := range []string{
|
||||||
|
"session-alpha",
|
||||||
|
"session-alpha:chunk:0",
|
||||||
|
"seg-001",
|
||||||
|
"Aria raises her hand and casts Cure Wounds.",
|
||||||
|
"speaker: Alice",
|
||||||
|
"start: 1.25",
|
||||||
|
"end: 3.5",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(user, want) {
|
||||||
|
t.Fatalf("user prompt = %q, want substring %q", user, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if metadata.PromptID != prompt.DNDSpellsPromptID {
|
||||||
|
t.Fatalf("metadata.PromptID = %q, want %q", metadata.PromptID, prompt.DNDSpellsPromptID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestBuildPromptDataRejectsMissingSourceContext(t *testing.T) {
|
||||||
|
if _, err := buildPromptData(contracts.ExtractionRequest{}); err == nil || !strings.Contains(err.Error(), "source") {
|
||||||
|
t.Fatalf("buildPromptData() error = %v, want source error", err)
|
||||||
|
}
|
||||||
|
if _, err := buildPromptData(contracts.ExtractionRequest{Source: promptSourceDocument()}); err == nil || !strings.Contains(err.Error(), "chunk") {
|
||||||
|
t.Fatalf("buildPromptData() error = %v, want chunk error", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptExtractionRequest() contracts.ExtractionRequest {
|
||||||
|
doc := promptSourceDocument()
|
||||||
|
chunk := &contracts.SourceChunk{
|
||||||
|
ID: "session-alpha:chunk:0",
|
||||||
|
SourceID: doc.ID,
|
||||||
|
Index: 0,
|
||||||
|
Units: append([]source.SourceUnit(nil), doc.Units...),
|
||||||
|
Metadata: map[string]any{"ignored": "chunk metadata"},
|
||||||
|
}
|
||||||
|
return contracts.ExtractionRequest{
|
||||||
|
Source: doc,
|
||||||
|
Chunk: chunk,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func promptSourceDocument() *source.SourceDocument {
|
||||||
|
return &source.SourceDocument{
|
||||||
|
ID: "session-alpha",
|
||||||
|
Kind: "transcript",
|
||||||
|
Format: "application/vnd.seriatim.minimal+json",
|
||||||
|
Digest: "sha256:test",
|
||||||
|
Units: []source.SourceUnit{
|
||||||
|
{
|
||||||
|
ID: "seg-001",
|
||||||
|
Kind: "transcript_segment",
|
||||||
|
Text: "Aria raises her hand and casts Cure Wounds.",
|
||||||
|
Metadata: map[string]any{
|
||||||
|
"speaker": "Alice",
|
||||||
|
"start": json.Number("1.25"),
|
||||||
|
"end": json.Number("3.5"),
|
||||||
|
"ignored": "not rendered",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
ID: "seg-002",
|
||||||
|
Kind: "transcript_segment",
|
||||||
|
Text: "The fighter's wounds begin to close.",
|
||||||
|
Metadata: map[string]any{"ignored": "not rendered"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustJSON(t *testing.T, value any) string {
|
||||||
|
t.Helper()
|
||||||
|
encoded, err := json.Marshal(value)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Marshal() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return string(encoded)
|
||||||
|
}
|
||||||
93
internal/modules/extract/dnd/spells/registry_test.go
Normal file
93
internal/modules/extract/dnd/spells/registry_test.go
Normal file
@@ -0,0 +1,93 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestNewReturnsExtractorWithMetadata(t *testing.T) {
|
||||||
|
extractor := New()
|
||||||
|
if extractor == nil {
|
||||||
|
t.Fatal("New() = nil, want extractor")
|
||||||
|
}
|
||||||
|
if extractor.Key() != Key {
|
||||||
|
t.Fatalf("extractor.Key() = %q, want %q", extractor.Key(), Key)
|
||||||
|
}
|
||||||
|
if extractor.ArtifactType() != ArtifactType {
|
||||||
|
t.Fatalf("extractor.ArtifactType() = %q, want %q", extractor.ArtifactType(), ArtifactType)
|
||||||
|
}
|
||||||
|
if extractor.SchemaVersion() != SchemaVersion {
|
||||||
|
t.Fatalf("extractor.SchemaVersion() = %q, want %q", extractor.SchemaVersion(), SchemaVersion)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestModuleSpec(t *testing.T) {
|
||||||
|
got := ModuleSpec()
|
||||||
|
want := pipeline.ModuleSpec{
|
||||||
|
Key: Key,
|
||||||
|
Stage: pipeline.StageExtract,
|
||||||
|
Requires: []string{
|
||||||
|
"chunks",
|
||||||
|
"source.transcript",
|
||||||
|
},
|
||||||
|
Provides: []string{
|
||||||
|
"dnd.spell_casts",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("ModuleSpec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
got.Requires[0] = "changed"
|
||||||
|
got.Provides[0] = "changed"
|
||||||
|
again := ModuleSpec()
|
||||||
|
if !reflect.DeepEqual(again, want) {
|
||||||
|
t.Fatalf("ModuleSpec() after caller mutation = %#v, want %#v", again, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterMakesExtractorBuildable(t *testing.T) {
|
||||||
|
registry := pipeline.NewExtractorRegistry()
|
||||||
|
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
extractor, err := registry.Build(Key)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Build() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if extractor.Key() != Key {
|
||||||
|
t.Fatalf("extractor.Key() = %q, want %q", extractor.Key(), Key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterStoresModuleSpec(t *testing.T) {
|
||||||
|
registry := pipeline.NewExtractorRegistry()
|
||||||
|
|
||||||
|
if err := Register(registry); err != nil {
|
||||||
|
t.Fatalf("Register() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, ok := registry.Spec(Key)
|
||||||
|
if !ok {
|
||||||
|
t.Fatal("Spec() ok = false, want true")
|
||||||
|
}
|
||||||
|
want := ModuleSpec()
|
||||||
|
if !reflect.DeepEqual(got, want) {
|
||||||
|
t.Fatalf("Spec() = %#v, want %#v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRegisterNilRegistryReturnsError(t *testing.T) {
|
||||||
|
err := Register(nil)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Register(nil) error = nil, want error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "extractor registry") {
|
||||||
|
t.Fatalf("Register(nil) error = %q, want registry context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
214
internal/modules/extract/dnd/spells/runner_test.go
Normal file
214
internal/modules/extract/dnd/spells/runner_test.go
Normal file
@@ -0,0 +1,214 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/config"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/modules/input/seriatim"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRunnerProcessesSeriatimInputWithDNDSpellsExtractor(t *testing.T) {
|
||||||
|
raw := readDNDSpellsFixture(t)
|
||||||
|
expectedDoc := parseDNDSpellsFixture(t, raw)
|
||||||
|
resolved := resolveDNDSpellsPipeline(t)
|
||||||
|
llmClient := &fakeSpellsLLMClient{
|
||||||
|
response: extractionResponse{
|
||||||
|
SpellCasts: []spellCastResponse{
|
||||||
|
{
|
||||||
|
Caster: "Aria",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Heals an injured ally.",
|
||||||
|
NarrativeDescription: "Aria restores the fighter after the fight.",
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{SourceID: expectedDoc.ID, StartUnitID: "seg-001", EndUnitID: "seg-001"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Caster: "Borin",
|
||||||
|
Spell: "Fire Bolt",
|
||||||
|
Effect: "Scorches the wight.",
|
||||||
|
NarrativeDescription: "Borin hurls fire at the wight.",
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{SourceID: expectedDoc.ID, StartUnitID: "seg-003", EndUnitID: "seg-003"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := pipeline.New(dndSpellsRunnerRegistries(t)).Run(context.Background(), pipeline.RunInput{
|
||||||
|
Pipeline: resolved.ResolvedPipeline,
|
||||||
|
RawInput: raw,
|
||||||
|
LLMClient: llmClient,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(output.Approved) != 2 {
|
||||||
|
t.Fatalf("len(Approved) = %d, want 2", len(output.Approved))
|
||||||
|
}
|
||||||
|
var first, second SpellCast
|
||||||
|
if err := json.Unmarshal(output.Approved[0].Payload, &first); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(first payload) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if err := json.Unmarshal(output.Approved[1].Payload, &second); err != nil {
|
||||||
|
t.Fatalf("Unmarshal(second payload) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if first.Spell != "Cure Wounds" || second.Spell != "Fire Bolt" {
|
||||||
|
t.Fatalf("approved spell order = %q, %q; want response order", first.Spell, second.Spell)
|
||||||
|
}
|
||||||
|
if first.Caster != "Aria" || second.Caster != "Borin" {
|
||||||
|
t.Fatalf("approved casters = %q, %q; want spell data", first.Caster, second.Caster)
|
||||||
|
}
|
||||||
|
for _, artifact := range output.Approved {
|
||||||
|
if artifact.ExtractorKey != Key || artifact.ArtifactType != ArtifactType || artifact.SchemaVersion != SchemaVersion {
|
||||||
|
t.Fatalf("approved artifact envelope = %#v, want dnd spells envelope", artifact)
|
||||||
|
}
|
||||||
|
if len(artifact.SourceRefs) != 1 {
|
||||||
|
t.Fatalf("len(SourceRefs) = %d, want 1", len(artifact.SourceRefs))
|
||||||
|
}
|
||||||
|
if err := source.ValidateRef(expectedDoc, artifact.SourceRefs[0]); err != nil {
|
||||||
|
t.Fatalf("ValidateRef() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if output.Manifest.InputModule != seriatim.Key {
|
||||||
|
t.Fatalf("manifest input module = %q, want %q", output.Manifest.InputModule, seriatim.Key)
|
||||||
|
}
|
||||||
|
if output.Manifest.ValidationStatus != "approved" {
|
||||||
|
t.Fatalf("ValidationStatus = %q, want approved", output.Manifest.ValidationStatus)
|
||||||
|
}
|
||||||
|
if len(output.Manifest.ArtifactLanes) != 1 {
|
||||||
|
t.Fatalf("len(ArtifactLanes) = %d, want 1", len(output.Manifest.ArtifactLanes))
|
||||||
|
}
|
||||||
|
lane := output.Manifest.ArtifactLanes[0]
|
||||||
|
if lane.ID != "spells" || lane.Extractor != Key {
|
||||||
|
t.Fatalf("manifest lane = %#v, want spells lane with dnd/spells extractor", lane)
|
||||||
|
}
|
||||||
|
if output.ContentType != "application/json" {
|
||||||
|
t.Fatalf("ContentType = %q, want application/json", output.ContentType)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerRejectsDNDSpellCastWithInvalidSourceRef(t *testing.T) {
|
||||||
|
raw := readDNDSpellsFixture(t)
|
||||||
|
resolved := resolveDNDSpellsPipeline(t)
|
||||||
|
llmClient := &fakeSpellsLLMClient{
|
||||||
|
response: extractionResponse{
|
||||||
|
SpellCasts: []spellCastResponse{
|
||||||
|
{
|
||||||
|
Caster: "Aria",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Heals an injured ally.",
|
||||||
|
NarrativeDescription: "Aria restores the fighter after the fight.",
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{SourceID: "spell-session", StartUnitID: "seg-999", EndUnitID: "seg-999"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
output, err := pipeline.New(dndSpellsRunnerRegistries(t)).Run(context.Background(), pipeline.RunInput{
|
||||||
|
Pipeline: resolved.ResolvedPipeline,
|
||||||
|
RawInput: raw,
|
||||||
|
LLMClient: llmClient,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Run() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
if len(output.Approved) != 0 {
|
||||||
|
t.Fatalf("len(Approved) = %d, want 0", len(output.Approved))
|
||||||
|
}
|
||||||
|
if len(output.Rejected) != 1 {
|
||||||
|
t.Fatalf("len(Rejected) = %d, want 1", len(output.Rejected))
|
||||||
|
}
|
||||||
|
rejected := output.Rejected[0]
|
||||||
|
if rejected.ValidatorName != sourceRefValidatorName {
|
||||||
|
t.Fatalf("ValidatorName = %q, want %q", rejected.ValidatorName, sourceRefValidatorName)
|
||||||
|
}
|
||||||
|
if rejected.ReasonCode != reasonInvalidSourceRef {
|
||||||
|
t.Fatalf("ReasonCode = %q, want %q", rejected.ReasonCode, reasonInvalidSourceRef)
|
||||||
|
}
|
||||||
|
if output.Manifest.ValidationStatus != "rejected" {
|
||||||
|
t.Fatalf("ValidationStatus = %q, want rejected", output.Manifest.ValidationStatus)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunnerFailsWhenDNDSpellsExtractorReturnsMalformedOutput(t *testing.T) {
|
||||||
|
raw := readDNDSpellsFixture(t)
|
||||||
|
resolved := resolveDNDSpellsPipeline(t)
|
||||||
|
llmClient := &fakeSpellsLLMClient{response: extractionResponse{}}
|
||||||
|
|
||||||
|
output, err := pipeline.New(dndSpellsRunnerRegistries(t)).Run(context.Background(), pipeline.RunInput{
|
||||||
|
Pipeline: resolved.ResolvedPipeline,
|
||||||
|
RawInput: raw,
|
||||||
|
LLMClient: llmClient,
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("Run() error = nil, want malformed extraction error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "extract lane") ||
|
||||||
|
!strings.Contains(err.Error(), "dnd spells") ||
|
||||||
|
!strings.Contains(err.Error(), "spell_casts") {
|
||||||
|
t.Fatalf("Run() error = %q, want D&D spells extraction context", err.Error())
|
||||||
|
}
|
||||||
|
if output.Manifest.ValidationStatus != "failed" {
|
||||||
|
t.Fatalf("ValidationStatus = %q, want failed", output.Manifest.ValidationStatus)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func resolveDNDSpellsPipeline(t *testing.T) config.EffectiveConfig {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
resolved, err := loadDNDSpellsPipelineConfig(t).Resolve(config.ResolveInput{
|
||||||
|
PipelineID: "dnd-spells-fixture",
|
||||||
|
Catalog: dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{}),
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Resolve() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return resolved
|
||||||
|
}
|
||||||
|
|
||||||
|
func dndSpellsRunnerRegistries(t *testing.T) pipeline.Registries {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
catalog := dndSpellsTestCatalog(t, dndSpellsCatalogSpecs{})
|
||||||
|
return pipeline.Registries{
|
||||||
|
Inputs: catalog.Inputs,
|
||||||
|
Chunkers: catalog.Chunkers,
|
||||||
|
Extractors: catalog.Extractors,
|
||||||
|
Mergers: catalog.Mergers,
|
||||||
|
Normalizers: catalog.Normalizers,
|
||||||
|
Outputs: catalog.Outputs,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func readDNDSpellsFixture(t *testing.T) []byte {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
raw, err := os.ReadFile("testdata/seriatim_spell_session.json")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ReadFile(seriatim_spell_session.json) error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
}
|
||||||
|
|
||||||
|
func parseDNDSpellsFixture(t *testing.T, raw []byte) *source.SourceDocument {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
doc, err := seriatim.New().Parse(context.Background(), contracts.ParseRequest{Raw: raw})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Parse() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
return doc
|
||||||
|
}
|
||||||
48
internal/modules/extract/dnd/spells/schema_test.go
Normal file
48
internal/modules/extract/dnd/spells/schema_test.go
Normal file
@@ -0,0 +1,48 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLookupResponseSchemaForSpells(t *testing.T) {
|
||||||
|
schema, ok := llm.LookupResponseSchema(llm.DNDSpellsSchemaKey)
|
||||||
|
if !ok {
|
||||||
|
t.Fatalf("LookupResponseSchema(%q) ok = false, want true", llm.DNDSpellsSchemaKey)
|
||||||
|
}
|
||||||
|
if schema.ID != "notarius.dnd.spells" {
|
||||||
|
t.Fatalf("schema.ID = %q, want notarius.dnd.spells", schema.ID)
|
||||||
|
}
|
||||||
|
if schema.Version != SchemaVersion {
|
||||||
|
t.Fatalf("schema.Version = %q, want %q", schema.Version, SchemaVersion)
|
||||||
|
}
|
||||||
|
if schema.Name != "notarius_dnd_spells_v1" {
|
||||||
|
t.Fatalf("schema.Name = %q, want notarius_dnd_spells_v1", schema.Name)
|
||||||
|
}
|
||||||
|
if !strings.HasPrefix(schema.SHA256, "sha256:") {
|
||||||
|
t.Fatalf("schema.SHA256 = %q, want sha256 prefix", schema.SHA256)
|
||||||
|
}
|
||||||
|
if !json.Valid(schema.JSONSchema) {
|
||||||
|
t.Fatalf("schema.JSONSchema is invalid JSON: %s", schema.JSONSchema)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestResponseSchemaDiagnosticsOmitRawSchema(t *testing.T) {
|
||||||
|
schema := llm.MustLookupResponseSchema(llm.DNDSpellsSchemaKey)
|
||||||
|
diagnostics := schema.DiagnosticsMap()
|
||||||
|
|
||||||
|
if diagnostics["key"] != llm.DNDSpellsSchemaKey {
|
||||||
|
t.Fatalf("diagnostics[key] = %#v, want %q", diagnostics["key"], llm.DNDSpellsSchemaKey)
|
||||||
|
}
|
||||||
|
for _, key := range []string{"id", "version", "name", "sha256"} {
|
||||||
|
if diagnostics[key] == "" {
|
||||||
|
t.Fatalf("diagnostics[%q] = %#v, want value", key, diagnostics[key])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if _, ok := diagnostics["json_schema"]; ok {
|
||||||
|
t.Fatalf("diagnostics should omit raw schema content: %#v", diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
11
internal/modules/extract/dnd/spells/testdata/pipeline.yml
vendored
Normal file
11
internal/modules/extract/dnd/spells/testdata/pipeline.yml
vendored
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
version: 1
|
||||||
|
pipelines:
|
||||||
|
dnd-spells-fixture:
|
||||||
|
input: seriatim
|
||||||
|
chunk: fake/chunk
|
||||||
|
artifacts:
|
||||||
|
spells:
|
||||||
|
extract: dnd/spells
|
||||||
|
merge: appendorder
|
||||||
|
normalize: noop
|
||||||
|
output: json
|
||||||
29
internal/modules/extract/dnd/spells/testdata/seriatim_spell_session.json
vendored
Normal file
29
internal/modules/extract/dnd/spells/testdata/seriatim_spell_session.json
vendored
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"metadata": {
|
||||||
|
"id": "spell-session",
|
||||||
|
"title": "Synthetic D&D spell session"
|
||||||
|
},
|
||||||
|
"segments": [
|
||||||
|
{
|
||||||
|
"id": "seg-001",
|
||||||
|
"start": 0,
|
||||||
|
"end": 4,
|
||||||
|
"speaker": "Alice",
|
||||||
|
"text": "Aria raises her holy symbol and casts Cure Wounds."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "seg-002",
|
||||||
|
"start": 4,
|
||||||
|
"end": 8,
|
||||||
|
"speaker": "DM",
|
||||||
|
"text": "The bandit mage casts Shield as the blow lands."
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "seg-003",
|
||||||
|
"start": 8,
|
||||||
|
"end": 12,
|
||||||
|
"speaker": "Bob",
|
||||||
|
"text": "Borin points at the wight and casts Fire Bolt."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
104
internal/modules/extract/dnd/spells/validator.go
Normal file
104
internal/modules/extract/dnd/spells/validator.go
Normal file
@@ -0,0 +1,104 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"fmt"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/validate"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
shapeValidatorName = "dnd/spells/shape"
|
||||||
|
sourceRefValidatorName = "dnd/spells/source_refs"
|
||||||
|
|
||||||
|
reasonInvalidPayload = "invalid_payload"
|
||||||
|
reasonMissingRequiredField = "missing_required_field"
|
||||||
|
reasonMissingSourceRef = "missing_source_ref"
|
||||||
|
reasonInvalidSourceRef = "invalid_source_ref"
|
||||||
|
)
|
||||||
|
|
||||||
|
var _ contracts.Validator = ShapeValidator{}
|
||||||
|
var _ contracts.Validator = SourceRefValidator{}
|
||||||
|
|
||||||
|
type ShapeValidator struct{}
|
||||||
|
|
||||||
|
func (validator ShapeValidator) Name() string {
|
||||||
|
return shapeValidatorName
|
||||||
|
}
|
||||||
|
|
||||||
|
func (validator ShapeValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
||||||
|
decisions := make([]contracts.ValidationDecision, 0, len(req.Candidates))
|
||||||
|
for _, candidate := range req.Candidates {
|
||||||
|
decisions = append(decisions, validateShape(candidate))
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{
|
||||||
|
ValidatorName: validator.Name(),
|
||||||
|
Decisions: decisions,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type SourceRefValidator struct{}
|
||||||
|
|
||||||
|
func (validator SourceRefValidator) Name() string {
|
||||||
|
return sourceRefValidatorName
|
||||||
|
}
|
||||||
|
|
||||||
|
func (validator SourceRefValidator) Validate(ctx context.Context, req contracts.ValidationRequest) (contracts.ValidationResult, error) {
|
||||||
|
if req.Source == nil {
|
||||||
|
return contracts.ValidationResult{}, fmt.Errorf("dnd spells source refs validator: source must not be nil")
|
||||||
|
}
|
||||||
|
|
||||||
|
decisions := make([]contracts.ValidationDecision, 0, len(req.Candidates))
|
||||||
|
for _, candidate := range req.Candidates {
|
||||||
|
decisions = append(decisions, validateSourceRefs(req.Source, candidate))
|
||||||
|
}
|
||||||
|
return contracts.ValidationResult{
|
||||||
|
ValidatorName: validator.Name(),
|
||||||
|
Decisions: decisions,
|
||||||
|
}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateShape(candidate artifacts.ArtifactCandidate) contracts.ValidationDecision {
|
||||||
|
var payload SpellCast
|
||||||
|
if err := json.Unmarshal(candidate.Payload, &payload); err != nil {
|
||||||
|
return validate.Rejected(candidate.Index, reasonInvalidPayload, fmt.Sprintf("invalid spell cast payload: %v", err))
|
||||||
|
}
|
||||||
|
for _, field := range requiredSpellCastFields(payload) {
|
||||||
|
if strings.TrimSpace(field.value) == "" {
|
||||||
|
return validate.Rejected(candidate.Index, reasonMissingRequiredField, fmt.Sprintf("missing required field %q", field.name))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return validate.Approved(candidate.Index)
|
||||||
|
}
|
||||||
|
|
||||||
|
func validateSourceRefs(doc *source.SourceDocument, candidate artifacts.ArtifactCandidate) contracts.ValidationDecision {
|
||||||
|
if len(candidate.SourceRefs) == 0 {
|
||||||
|
return validate.Rejected(candidate.Index, reasonMissingSourceRef, "spell cast candidate must include at least one source ref")
|
||||||
|
}
|
||||||
|
for _, ref := range candidate.SourceRefs {
|
||||||
|
if err := source.ValidateRef(doc, ref); err != nil {
|
||||||
|
return validate.Rejected(candidate.Index, reasonInvalidSourceRef, err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return validate.Approved(candidate.Index)
|
||||||
|
}
|
||||||
|
|
||||||
|
func requiredSpellCastFields(payload SpellCast) []struct {
|
||||||
|
name string
|
||||||
|
value string
|
||||||
|
} {
|
||||||
|
return []struct {
|
||||||
|
name string
|
||||||
|
value string
|
||||||
|
}{
|
||||||
|
{name: "caster", value: payload.Caster},
|
||||||
|
{name: "spell", value: payload.Spell},
|
||||||
|
{name: "effect", value: payload.Effect},
|
||||||
|
{name: "narrative_description", value: payload.NarrativeDescription},
|
||||||
|
}
|
||||||
|
}
|
||||||
271
internal/modules/extract/dnd/spells/validator_test.go
Normal file
271
internal/modules/extract/dnd/spells/validator_test.go
Normal file
@@ -0,0 +1,271 @@
|
|||||||
|
package spells
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/artifacts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||||
|
"gitea.maximumdirect.net/eric/notarius/internal/framework/validate"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExtractorValidatorsReturnsExpectedChain(t *testing.T) {
|
||||||
|
validators := New().Validators()
|
||||||
|
|
||||||
|
if len(validators) != 2 {
|
||||||
|
t.Fatalf("len(Validators()) = %d, want 2", len(validators))
|
||||||
|
}
|
||||||
|
if validators[0].Name() != shapeValidatorName {
|
||||||
|
t.Fatalf("Validators()[0].Name() = %q, want %q", validators[0].Name(), shapeValidatorName)
|
||||||
|
}
|
||||||
|
if validators[1].Name() != sourceRefValidatorName {
|
||||||
|
t.Fatalf("Validators()[1].Name() = %q, want %q", validators[1].Name(), sourceRefValidatorName)
|
||||||
|
}
|
||||||
|
|
||||||
|
validators[0] = nil
|
||||||
|
again := New().Validators()
|
||||||
|
if len(again) != 2 || again[0] == nil || again[0].Name() != shapeValidatorName {
|
||||||
|
t.Fatalf("Validators() after caller mutation = %#v, want fresh validators", again)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorsApproveValidCandidate(t *testing.T) {
|
||||||
|
candidate := validSpellCandidate(7)
|
||||||
|
|
||||||
|
shapeResult, err := ShapeValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{candidate},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ShapeValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, shapeResult, shapeValidatorName, 7, true, validate.ReasonApproved)
|
||||||
|
|
||||||
|
sourceRefResult, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Source: promptSourceDocument(),
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{candidate},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SourceRefValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, sourceRefResult, sourceRefValidatorName, 7, true, validate.ReasonApproved)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestShapeValidatorRejectsMalformedPayload(t *testing.T) {
|
||||||
|
candidate := validSpellCandidate(3)
|
||||||
|
candidate.Payload = json.RawMessage(`{"caster":`)
|
||||||
|
|
||||||
|
result, err := ShapeValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{candidate},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ShapeValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, result, shapeValidatorName, 3, false, reasonInvalidPayload)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestShapeValidatorRejectsBlankRequiredFields(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
mutate func(*SpellCast)
|
||||||
|
}{
|
||||||
|
{name: "caster", mutate: func(payload *SpellCast) { payload.Caster = " \t" }},
|
||||||
|
{name: "spell", mutate: func(payload *SpellCast) { payload.Spell = "" }},
|
||||||
|
{name: "effect", mutate: func(payload *SpellCast) { payload.Effect = "\n" }},
|
||||||
|
{name: "narrative description", mutate: func(payload *SpellCast) { payload.NarrativeDescription = " " }},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
payload := validSpellPayload()
|
||||||
|
tt.mutate(&payload)
|
||||||
|
candidate := validSpellCandidate(5)
|
||||||
|
candidate.Payload = mustSpellPayload(t, payload)
|
||||||
|
|
||||||
|
result, err := ShapeValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{candidate},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ShapeValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, result, shapeValidatorName, 5, false, reasonMissingRequiredField)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestShapeValidatorDoesNotRequireSourceDocument(t *testing.T) {
|
||||||
|
result, err := ShapeValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{validSpellCandidate(11)},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ShapeValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, result, shapeValidatorName, 11, true, validate.ReasonApproved)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSourceRefValidatorRejectsMissingRefs(t *testing.T) {
|
||||||
|
candidate := validSpellCandidate(13)
|
||||||
|
candidate.SourceRefs = nil
|
||||||
|
|
||||||
|
result, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Source: promptSourceDocument(),
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{candidate},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SourceRefValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, result, sourceRefValidatorName, 13, false, reasonMissingSourceRef)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSourceRefValidatorRejectsInvalidRefs(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
ref source.SourceRef
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "unknown source id",
|
||||||
|
ref: source.SourceRef{SourceID: "session-beta", StartUnitID: "seg-001", EndUnitID: "seg-002"},
|
||||||
|
want: "does not match",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unknown start unit",
|
||||||
|
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-999", EndUnitID: "seg-002"},
|
||||||
|
want: "start_unit_id",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "unknown end unit",
|
||||||
|
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-999"},
|
||||||
|
want: "end_unit_id",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "reversed unit range",
|
||||||
|
ref: source.SourceRef{SourceID: "session-alpha", StartUnitID: "seg-002", EndUnitID: "seg-001"},
|
||||||
|
want: "appears after",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tt := range tests {
|
||||||
|
t.Run(tt.name, func(t *testing.T) {
|
||||||
|
candidate := validSpellCandidate(17)
|
||||||
|
candidate.SourceRefs = []source.SourceRef{tt.ref}
|
||||||
|
|
||||||
|
result, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Source: promptSourceDocument(),
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{candidate},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SourceRefValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertSingleDecision(t, result, sourceRefValidatorName, 17, false, reasonInvalidSourceRef)
|
||||||
|
if !strings.Contains(result.Decisions[0].Message, tt.want) {
|
||||||
|
t.Fatalf("Message = %q, want substring %q", result.Decisions[0].Message, tt.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSourceRefValidatorRequiresSourceDocument(t *testing.T) {
|
||||||
|
_, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Candidates: []artifacts.ArtifactCandidate{validSpellCandidate(19)},
|
||||||
|
})
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("SourceRefValidator.Validate() error = nil, want source error")
|
||||||
|
}
|
||||||
|
if !strings.Contains(err.Error(), "dnd spells") || !strings.Contains(err.Error(), "source") {
|
||||||
|
t.Fatalf("SourceRefValidator.Validate() error = %q, want source context", err.Error())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidatorsPreserveCandidateIndexes(t *testing.T) {
|
||||||
|
candidates := []artifacts.ArtifactCandidate{
|
||||||
|
validSpellCandidate(23),
|
||||||
|
validSpellCandidate(29),
|
||||||
|
}
|
||||||
|
|
||||||
|
shapeResult, err := ShapeValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Candidates: candidates,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("ShapeValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertDecisionIndexes(t, shapeResult.Decisions, []int{23, 29})
|
||||||
|
|
||||||
|
sourceRefResult, err := SourceRefValidator{}.Validate(context.Background(), contracts.ValidationRequest{
|
||||||
|
Source: promptSourceDocument(),
|
||||||
|
Candidates: candidates,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("SourceRefValidator.Validate() error = %v, want nil", err)
|
||||||
|
}
|
||||||
|
assertDecisionIndexes(t, sourceRefResult.Decisions, []int{23, 29})
|
||||||
|
}
|
||||||
|
|
||||||
|
func validSpellCandidate(index int) artifacts.ArtifactCandidate {
|
||||||
|
return artifacts.ArtifactCandidate{
|
||||||
|
Index: index,
|
||||||
|
Payload: spellPayload(validSpellPayload()),
|
||||||
|
SourceRefs: []source.SourceRef{
|
||||||
|
{SourceID: "session-alpha", StartUnitID: "seg-001", EndUnitID: "seg-002"},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func validSpellPayload() SpellCast {
|
||||||
|
return SpellCast{
|
||||||
|
Caster: "Aria",
|
||||||
|
Spell: "Cure Wounds",
|
||||||
|
Effect: "Heals an injured ally.",
|
||||||
|
NarrativeDescription: "Aria restores the fighter after the fight.",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func mustSpellPayload(t *testing.T, payload SpellCast) json.RawMessage {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
return spellPayload(payload)
|
||||||
|
}
|
||||||
|
|
||||||
|
func spellPayload(payload SpellCast) json.RawMessage {
|
||||||
|
encoded, err := json.Marshal(payload)
|
||||||
|
if err != nil {
|
||||||
|
panic(err)
|
||||||
|
}
|
||||||
|
return encoded
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertSingleDecision(t *testing.T, result contracts.ValidationResult, wantName string, wantIndex int, wantApproved bool, wantReason string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if result.ValidatorName != wantName {
|
||||||
|
t.Fatalf("ValidatorName = %q, want %q", result.ValidatorName, wantName)
|
||||||
|
}
|
||||||
|
if len(result.Decisions) != 1 {
|
||||||
|
t.Fatalf("len(Decisions) = %d, want 1", len(result.Decisions))
|
||||||
|
}
|
||||||
|
decision := result.Decisions[0]
|
||||||
|
if decision.CandidateIndex != wantIndex {
|
||||||
|
t.Fatalf("CandidateIndex = %d, want %d", decision.CandidateIndex, wantIndex)
|
||||||
|
}
|
||||||
|
if decision.Approved != wantApproved {
|
||||||
|
t.Fatalf("Approved = %t, want %t", decision.Approved, wantApproved)
|
||||||
|
}
|
||||||
|
if decision.ReasonCode != wantReason {
|
||||||
|
t.Fatalf("ReasonCode = %q, want %q", decision.ReasonCode, wantReason)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func assertDecisionIndexes(t *testing.T, decisions []contracts.ValidationDecision, want []int) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if len(decisions) != len(want) {
|
||||||
|
t.Fatalf("len(Decisions) = %d, want %d", len(decisions), len(want))
|
||||||
|
}
|
||||||
|
for i := range want {
|
||||||
|
if decisions[i].CandidateIndex != want[i] {
|
||||||
|
t.Fatalf("Decisions[%d].CandidateIndex = %d, want %d", i, decisions[i].CandidateIndex, want[i])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user