Compare commits

...

5 Commits

40 changed files with 1987 additions and 85 deletions

View File

@@ -61,9 +61,9 @@ No pipelines are built in. A run requires a configured pipeline.
If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
catalog. Prompt definitions may also name default profile IDs. The current D&D
scene and spell prompts default to the built-in `mistral-small-3` profile when a
module binding does not set `llm_profile`. That built-in profile reads its
credential from `OPENROUTER_API_KEY`.
scene, extraction, and NPC-normalization prompts default to `gemini-2-flash`
when a module binding does not set `llm_profile`. The selected built-in profile
reads its credentials from its Scriptorium profile definition.
## Scriptorium Profiles
@@ -322,7 +322,8 @@ Binding fields:
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
Scriptorium prompt default select the profile.
- `retries`: non-negative retry count for extra runtime attempts after the
first attempt. Default: `0`. Supported on `chunk`, `extract`, `merge`, and
first attempt. Default: `0`, which permits one initial attempt and no
additional attempts. Supported on `chunk`, `extract`, `merge`, and
`normalize` bindings.
- `options`: optional module-specific settings.
- `references`: optional reference bindings. Supported only for `chunk`,
@@ -338,6 +339,15 @@ During resolution, each selected module's registered option validator runs.
Production input, chunk, and output bindings reject unknown or invalid options
with the affected binding context.
### NPC Semantic Normalization
The `dnd/npcs` normalizer combines deterministic canonical-name consolidation
with a document-level LLM identity decision for eligible differently named NPC
records. Its `llm_profile` and `retries` use the ordinary `normalize` binding
fields above; `retries: 0` therefore still makes one normalization attempt.
The normalizer declares no references and accepts no options. Context-window
selection is current module policy, not configuration.
### JSON Output Options
The `json` output module accepts only `include_chunk_map`, a boolean that
@@ -388,7 +398,7 @@ production validators do not call the LLM and must not set `llm_profile`.
| merge | `appendorder` | Combines typed artifacts in chunk order. |
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical name and unions exact evidence. |
| normalize | `dnd/npcs` | Deterministically prepares typed D&D NPC-list artifacts, then uses an LLM-assisted document-level identity decision to consolidate safe name-based matches and union exact evidence. |
| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
| normalize | `dnd/item-events` | Deterministically trims, source-orders, and removes only exact D&D item-event duplicates. |
| normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. |

View File

@@ -75,10 +75,17 @@ The production identities are:
The extractor maps private model records to the current source identity and
assigns deterministic IDs. Extraction validation checks shape, source
references, and source relatedness. The normalizer then consolidates records
only when their normalized canonical names match, preserves the first record's
display and output position, unions exact evidence, and validates the retained
registry's identity. No LLM is used for consolidation.
references, and source relatedness. The normalizer first consolidates equal
canonical-name matches, then may make one document-level LLM-assisted identity
decision per configured normalize attempt for eligible distinctly named
records. Consolidation is name-based: it retains a supplied canonical display
name, keeps the earliest affected output position, derives its ID again, and
unions exact evidence in canonical order. Ambiguous, invalid, or conflicting
proposals are not applied; independently safe matches may still be retained.
After the retry budget is exhausted, the safe result is accepted with bounded
normalization warnings and the usual validation. The durable v1 artifact shape
does not add aliases, proposal fields, or any other semantic-normalization
representation.
The extraction prompt asks only for individually identifiable NPC names backed
by source evidence. Groups, generic roles, invented labels, and descriptive or

View File

@@ -154,6 +154,22 @@ identity, campaign-reference, and instruction messages are ephemeral cache
boundaries; the transcript is last and has no cache control. Compatible shared
messages remain canonical shared assets rather than copied package text.
### D&D NPC Normalization Prompt Ordering And Cache Boundaries
NPC normalization has a distinct prompt and response-schema identity from NPC
extraction. Its stable message tiers are the common D&D system and identity
assets, followed by package-owned task and normalization instructions. Cache
boundaries follow the shared identity tier and the package instructions. The
variable tail contains the private candidate-name-and-range input and a
windowed transcript input whose cited units provide local context; neither has
a cache boundary because it changes with the document.
This prompt intentionally omits extraction-evidence and campaign-reference
assets: it reconciles existing records rather than extracting events or adding
evidence. Its package-owned manifest and schema identity are fingerprinted
separately, so a normalization prompt or schema change cannot reuse a prior
normalization checkpoint.
Shared wording belongs in the canonical assets under
`internal/modules/dnd/shared`; extraction packages reference those assets in
their manifests instead of copying similar text into package-local files.

View File

@@ -395,11 +395,27 @@ omission-summary warning when truncated.
### `internal/modules/dnd/normalize/npcs`
The NPC normalizer performs deterministic identity-aware consolidation in
merged input order. It consolidates only equal canonical-name comparison keys,
retains the first display record, and unions exact source references. It exposes
the identity policy as its local checkpoint fingerprint and emits bounded
normalization warnings.
The NPC normalizer deterministically trims display names, recomputes IDs,
canonicalizes evidence, and consolidates equal comparison keys before semantic
work. Records are eligible for the document-level identity call only when they
have a non-empty comparison key and wholly valid current-document references.
It sends private candidate names and source ranges plus coalesced, cited
transcript windows to its own prompt; stable NPC IDs and the durable artifact
shape are not prompt inputs.
The private structured response proposes groups of supplied names and a
canonical supplied name. Deterministic comparison-key resolution validates each
group, discards unsafe or overlapping groups, and independently applies safe
ones. Application preserves earliest record order, unions canonical evidence,
and derives the final canonical ID. Invalid structured output and discarded
groups request framework retry with a safe fallback; bounded diagnostics become
durable only on final fallback exhaustion.
The normalizer records prompt and response-schema identities and digests,
identity and normalization policies, and semantic-context policy and radius as
manifest metadata. Its local checkpoint fingerprints cover the prompt, response
schema, identity policy, normalization policy, and semantic-context policy so a
meaningful behavior change invalidates prior normalize reuse.
## Merger And Normalizer

View File

@@ -119,7 +119,7 @@ Configuration. The implemented module packages are:
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
| `internal/modules/dnd/normalize/npcs` | Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics. |
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,

View File

@@ -322,9 +322,24 @@ error when attempts are exhausted. A rejection becomes a recorded
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
processing immediately.
Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
from discarded attempts are not promoted. Configuration owns retry counts and
validator overrides; see [Module Bindings](../config.md#module-bindings).
Structured-completion adapters classify malformed or undecodable provider
output with the provider-neutral `contracts.ErrInvalidStructuredOutput` error.
A typed normalizer may turn that condition, or another unsafe proposal, into a
normalize retry directive with a deterministic safe candidate, stable
diagnostic, and fallback warnings. The directive consumes the same configured
normalize retry budget: `retries` permits that many additional attempts after
the initial attempt. It neither creates a normalizer-local retry loop nor
records an accepted checkpoint for the discarded attempt.
When a later normalize attempt succeeds, its candidate alone proceeds through
the usual validation and checkpoint path. When the final attempt still returns
a directive, the runner validates its supplied safe fallback through that same
normalizer validator chain before accepting or rejecting it. Ordinary
attempt-local warnings and fallback warnings remain unpromoted while another
attempt is available; only final exhaustion promotes the supplied fallback
warnings. Rejected output is a non-fatal pipeline outcome and does not advance.
Configuration owns retry counts and validator overrides; see
[Module Bindings](../config.md#module-bindings).
## Checkpoint And Debug Hooks
@@ -389,7 +404,9 @@ boundaries. Every executed chunk, extract, merge, and normalize attempt writes
one terminal envelope for acceptance, validator rejection, module or validator
error, or applicable candidate or final serialization error. The envelope
contains its attempt-local warnings, any available candidate and rejection,
and terminal error text; failures before a candidate exists omit that payload.
and terminal error text; normalize retry directives retain their attempt-local
candidate and diagnostic, while only the final safe fallback reaches validation.
Failures before a candidate exists omit that payload.
Only LLM calls made by the module operation belong to the module attempt.
Validator calls retain independent scopes under `validate/` and are not
duplicated into the module envelope. A failed terminal-envelope write is a

View File

@@ -86,6 +86,22 @@ workflow intentionally crosses a process or session boundary. Those files are
validated against the consumer slot and must be protected as sensitive
campaign data. They are not part of the maintained ordered handoff workflow.
### NPC Semantic Normalization
Before the first step can release its accepted NPC artifact across the ordered
generated-reference barrier, `dnd/npcs` performs one document-level semantic
normalization call for each configured normalize attempt when eligible distinct
names remain. The normalize binding's `retries` setting controls additional
attempts. If an invalid or unsafe identity proposal exhausts that budget, the
run safely accepts the deterministic and any independently safe partial
consolidation, with a bounded warning; ordinary validation still applies before
the artifact can cross the barrier.
An accepted normalized NPC checkpoint can be reused on `--resume` just like
other accepted normalize work. A changed normalization prompt, response schema,
or policy identity produces a cold cache miss, so the current reconciliation is
recomputed rather than silently reusing incompatible state.
## Chunk-Plan Cache
Chunk plans are stored at:

View File

@@ -2,7 +2,7 @@
## Status
Accepted scope; not implemented.
Implemented.
## Purpose

View File

@@ -2,7 +2,7 @@
## Status
Ready for implementation.
Completed.
## Objective

View File

@@ -43,7 +43,10 @@ pipelines:
module: dnd/npcs
retries: 2
merge: appendorder
normalize: dnd/npcs
normalize:
module: dnd/npcs
llm_profile: gemini-2-flash
retries: 2
scene-descriptions:
extract:
module: dnd/scene-descriptions

View File

@@ -0,0 +1,7 @@
package contracts
import "errors"
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
// the caller's declared structured-output contract.
var ErrInvalidStructuredOutput = errors.New("invalid structured output")

View File

@@ -90,6 +90,15 @@ type TypedNormalizeRequest[T any] struct {
type TypedNormalizeResult[T any] struct {
Value T
Warnings []Warning
Retry *NormalizeRetry
}
// NormalizeRetry asks the framework to retry normalization while retaining a
// safe candidate for acceptance if the retry budget is exhausted.
type NormalizeRetry struct {
ReasonCode string
Message string
FallbackWarnings []Warning
}
type Normalizer[T any] interface {

View File

@@ -113,17 +113,17 @@ func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contract
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
}
if result == nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: empty result", promptID)
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
}
response := c.responseFromResult(result, prepared)
if result.Validation.Status == scriptorium.ValidationFailed || !result.Validation.IsValid {
return response, fmt.Errorf("run Scriptorium prompt %q: validation failed: %s", promptID, strings.Join(result.Validation.Errors, "; "))
return response, fmt.Errorf("run Scriptorium prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
}
if len(strings.TrimSpace(string(response.Content))) == 0 {
return response, fmt.Errorf("run Scriptorium prompt %q: empty structured output", promptID)
return response, fmt.Errorf("run Scriptorium prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
}
if err := json.Unmarshal(response.Content, out); err != nil {
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w", promptID, err)
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
}
return response, nil
}

View File

@@ -113,7 +113,7 @@ func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
if err == nil || !strings.Contains(err.Error(), "validation failed") {
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "validation failed") {
t.Fatalf("CompleteStructured() error = %v, want validation failure", err)
}
if got := string(resp.Content); got != `{"bad":true}` {
@@ -138,7 +138,7 @@ func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
if err == nil || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
}
if got := string(resp.Content); got != `{"ok":true}` {
@@ -163,6 +163,9 @@ func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t
if err == nil {
t.Fatalf("CompleteStructured() error = nil, want provider error")
}
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
t.Fatalf("provider error = %v, must not be classified as invalid structured output", err)
}
if !strings.Contains(err.Error(), `run Scriptorium prompt "adapter.test"`) {
t.Fatalf("error = %q, want operation context", err.Error())
}
@@ -186,11 +189,27 @@ func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
if !errors.Is(err, context.Canceled) {
if !errors.Is(err, context.Canceled) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
}
}
func TestScriptoriumClientClassifiesEmptyStructuredCompletion(t *testing.T) {
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{allowEmpty: true})
var out map[string]any
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
PromptID: "adapter.test",
SessionID: "session-123",
Inputs: contracts.LLMInputSet{
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
},
}, &out)
if !errors.Is(err, contracts.ErrInvalidStructuredOutput) {
t.Fatalf("CompleteStructured() error = %v, want invalid structured output", err)
}
}
func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
fake := &fakeScriptoriumLLM{
content: `{"ok":true}`,
@@ -296,6 +315,7 @@ output:
type fakeScriptoriumLLM struct {
content string
allowEmpty bool
err error
block chan struct{}
mu sync.Mutex
@@ -329,10 +349,10 @@ func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.Gener
return nil, f.err
}
content := f.content
if content == "" {
if content == "" && !f.allowEmpty {
content = `{"ok":true}`
}
if !json.Valid([]byte(content)) {
if !f.allowEmpty && !json.Valid([]byte(content)) {
return nil, errors.New("test fake must return JSON content")
}
return &scriptorium.GenerateResponse{

View File

@@ -80,12 +80,23 @@ func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleS
if err != nil {
return erasedTypedResult{}, err
}
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry)}, nil
},
}
return nil
}
func cloneNormalizeRetry(retry *contracts.NormalizeRetry) *contracts.NormalizeRetry {
if retry == nil {
return nil
}
return &contracts.NormalizeRetry{
ReasonCode: retry.ReasonCode,
Message: retry.Message,
FallbackWarnings: cloneWarnings(retry.FallbackWarnings),
}
}
func (r *NormalizerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
if r == nil {
return fmt.Errorf("normalizer registry must not be nil")

View File

@@ -0,0 +1,52 @@
package pipeline
import (
"context"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
type retryingNotesNormalizer struct {
warnings []contracts.Warning
retry *contracts.NormalizeRetry
}
func (retryingNotesNormalizer) Key() string { return "test/retry-normalize" }
func (retryingNotesNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
func (n retryingNotesNormalizer) Normalize(_ context.Context, req contracts.TypedNormalizeRequest[codecNotes]) (contracts.TypedNormalizeResult[codecNotes], error) {
return contracts.TypedNormalizeResult[codecNotes]{Value: req.MergeOutput.Value, Warnings: n.warnings, Retry: n.retry}, nil
}
func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
warnings := []contracts.Warning{{Scope: "attempt", ReasonCode: "ordinary", Message: "ordinary warning"}}
retry := &contracts.NormalizeRetry{
ReasonCode: "retryable",
Message: "safe fallback available",
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
}
registry := NewNormalizerRegistry()
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
}); err != nil {
t.Fatalf("RegisterNormalizer() error = %v", err)
}
entry, ok := registry.typedEntry("test/retry-normalize", "test/notes")
if !ok {
t.Fatal("typed normalizer entry missing")
}
implementation, err := entry.builder(BuildRequest{})
if err != nil {
t.Fatalf("builder() error = %v", err)
}
result, err := entry.normalize(context.Background(), implementation, contracts.TypedNormalizeRequest[any]{MergeOutput: contracts.MergeArtifact[any]{Value: codecNotes{Items: []string{"safe"}}}})
if err != nil {
t.Fatalf("normalize() error = %v", err)
}
warnings[0].Message = "mutated"
retry.Message = "mutated"
retry.FallbackWarnings[0].Message = "mutated"
if result.Retry == nil || result.Warnings[0].Message != "ordinary warning" || result.Retry.Message != "safe fallback available" || result.Retry.FallbackWarnings[0].Message != "fallback warning" {
t.Fatalf("erased retry result = %#v, want independent warning data", result)
}
}

View File

@@ -0,0 +1,174 @@
package pipeline
import (
"context"
"fmt"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
)
func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
tests := []struct {
name string
retries int
operation func(int) erasedTypedResult
validator *preparedValidator
wantCalls int
wantItem string
wantWarnings []string
wantRejected int
wantDebug []string
wantCheckpoint int
}{
{
name: "accepts zero-retry fallback",
retries: 0,
operation: func(int) erasedTypedResult {
return retryableNormalizeResult("fallback", "ordinary", "fallback-warning")
},
wantCalls: 1,
wantItem: "fallback",
wantWarnings: []string{"ordinary", "fallback-warning"},
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
wantCheckpoint: 1,
},
{
name: "retries before accepting ordinary result",
retries: 1,
operation: func(attempt int) erasedTypedResult {
if attempt == 1 {
return retryableNormalizeResult("discarded", "discarded-ordinary", "discarded-fallback")
}
return erasedTypedResult{Value: codecNotes{Items: []string{"accepted"}}, Warnings: []contracts.Warning{{Scope: "accepted", ReasonCode: "ordinary", Message: "accepted-warning"}}}
},
wantCalls: 2,
wantItem: "accepted",
wantWarnings: []string{"accepted-warning"},
wantDebug: []string{`"another_attempt":true`, `"fallback_accepted":false`},
wantCheckpoint: 1,
},
{
name: "accepts final fallback after exhaustion",
retries: 1,
operation: func(attempt int) erasedTypedResult {
return retryableNormalizeResult(fmt.Sprintf("fallback-%d", attempt), fmt.Sprintf("ordinary-%d", attempt), fmt.Sprintf("fallback-warning-%d", attempt))
},
wantCalls: 2,
wantItem: "fallback-2",
wantWarnings: []string{"ordinary-2", "fallback-warning-2"},
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
wantCheckpoint: 1,
},
{
name: "keeps final fallback rejection terminal",
retries: 1,
operation: func(int) erasedTypedResult {
return retryableNormalizeResult("rejected", "ordinary", "fallback-warning")
},
validator: &preparedValidator{
resolved: ResolvedValidator{Binding: Binding("reject-final-fallback"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "reject fallback"}, nil
},
},
wantCalls: 2,
wantRejected: 1,
wantDebug: []string{`"fallback_accepted":true`, `"rejection"`},
wantCheckpoint: 0,
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
lane := &prepared.Steps[0].lanes[0]
lane.resolved.Normalize.Retries = tc.retries
if tc.validator != nil {
lane.normalizeValidators.validators = []preparedValidator{*tc.validator}
}
calls := 0
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
calls++
return tc.operation(calls), nil
}
debug := newCapturedDebugRecorder()
checkpoints := &candidateCheckpointRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug, Checkpoints: checkpoints})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if calls != tc.wantCalls {
t.Fatalf("normalize calls = %d, want %d", calls, tc.wantCalls)
}
if checkpoints.normalizeSucceeded != tc.wantCheckpoint {
t.Fatalf("normalize checkpoints = %d, want %d", checkpoints.normalizeSucceeded, tc.wantCheckpoint)
}
if len(output.Rejected) != tc.wantRejected {
t.Fatalf("rejected outputs = %#v, want %d", output.Rejected, tc.wantRejected)
}
var retryDebug strings.Builder
for _, name := range debug.names() {
if strings.HasPrefix(name, "normalize/notes/attempt-") && strings.HasSuffix(name, ".json") {
retryDebug.Write(debug.json[name])
}
}
for _, fragment := range tc.wantDebug {
if !strings.Contains(retryDebug.String(), fragment) {
t.Fatalf("retry debug = %s, want %q", retryDebug.String(), fragment)
}
}
if tc.wantRejected != 0 {
if output.Rejected[0].AttemptCount != tc.wantCalls {
t.Fatalf("rejection attempt count = %d, want %d", output.Rejected[0].AttemptCount, tc.wantCalls)
}
return
}
if len(output.NormalizeOutputs) != 1 {
t.Fatalf("normalize outputs = %#v, want one", output.NormalizeOutputs)
}
decoded, err := lane.typed.codec.decode(output.NormalizeOutputs[0].Artifact.Content)
if err != nil {
t.Fatalf("decode normalized output: %v", err)
}
normalized, ok := decoded.(codecNotes)
if !ok {
t.Fatalf("decoded normalized output = %T, want codecNotes", decoded)
}
if got := firstNote(normalized); got != tc.wantItem {
t.Fatalf("normalized item = %q, want %q", got, tc.wantItem)
}
gotWarnings := make([]string, len(output.Warnings))
for index, warning := range output.Warnings {
gotWarnings[index] = warning.Message
}
if strings.Join(gotWarnings, "|") != strings.Join(tc.wantWarnings, "|") {
t.Fatalf("durable warnings = %#v, want %#v", gotWarnings, tc.wantWarnings)
}
})
}
}
func TestRunnerRejectsBlankNormalizeRetryDiagnostic(t *testing.T) {
prepared := preparedAttemptDebugPipeline(t)
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
return erasedTypedResult{Value: codecNotes{Items: []string{"safe"}}, Retry: &contracts.NormalizeRetry{ReasonCode: " ", Message: "missing reason"}}, nil
}
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: newCapturedDebugRecorder()})
if err == nil || !strings.Contains(err.Error(), "blank reason code or message") {
t.Fatalf("Run() error = %v, want retry diagnostic contract error", err)
}
}
func retryableNormalizeResult(item, ordinary, fallback string) erasedTypedResult {
return erasedTypedResult{
Value: codecNotes{Items: []string{item}},
Warnings: []contracts.Warning{{Scope: "attempt", ReasonCode: "ordinary", Message: ordinary}},
Retry: &contracts.NormalizeRetry{
ReasonCode: "retryable_normalization",
Message: "safe fallback is available",
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "fallback", Message: fallback}},
},
}
}

View File

@@ -8,6 +8,7 @@ import (
"errors"
"fmt"
"path"
"strings"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
@@ -362,9 +363,30 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr)
return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
}
var retryPayload map[string]any
if result.Retry != nil {
if strings.TrimSpace(result.Retry.ReasonCode) == "" || strings.TrimSpace(result.Retry.Message) == "" {
return false, nil, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings)}, fmt.Errorf("normalize lane %q returned retry directive with blank reason code or message", lane.ID))
}
retryRemaining := attempt <= lane.Normalize.Retries
retryPayload = map[string]any{
"reason_code": result.Retry.ReasonCode,
"message": result.Retry.Message,
"another_attempt": retryRemaining,
"fallback_accepted": !retryRemaining,
}
if retryRemaining {
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "retry": retryPayload}
return false, nil, terminal.record(payload, nil)
}
attemptWarnings = append(attemptWarnings, cloneWarnings(result.Retry.FallbackWarnings)...)
}
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
attemptWarnings = append(attemptWarnings, warnings...)
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
if retryPayload != nil {
payload["retry"] = retryPayload
}
if validateErr != nil || rejected != nil {
return false, rejected, terminal.record(payload, validateErr)
}

View File

@@ -24,6 +24,7 @@ type erasedMergeArtifact struct {
type erasedTypedResult struct {
Value any
Warnings []contracts.Warning
Retry *contracts.NormalizeRetry
}
type typedValidationTarget struct {

View File

@@ -0,0 +1,6 @@
package npcs
import "embed"
//go:embed assets/schemas/dnd_npcs_normalize_llm.v1.json assets/prompts/*.yaml assets/prompts/*.md
var embeddedAssets embed.FS

View File

@@ -0,0 +1,4 @@
The supplied NPC candidates are below. Use only these display names in the
response.
{{ input "candidates" }}

View File

@@ -0,0 +1,32 @@
id: dnd.npcs.normalize
version: "v1"
default_profile: gemini-2-flash
inputs:
- name: candidates
required: true
content_type: application/json
- name: transcript
required: true
content_type: application/json
messages:
- role: system
content_file: ./sharedassets/common-dnd-system.md
- role: user
content_file: ./sharedassets/common-dnd-identity.md
cache_control:
type: ephemeral
- role: user
content_file: ./task.md
- role: user
content_file: ./instructions.md
cache_control:
type: ephemeral
- role: user
content_file: ./candidates.md
- role: user
content_file: ./sharedassets/common-dnd-transcript.md
output:
format: json
validation_mode: json_schema
schema_path: dnd_npcs_normalize_llm.v1.json
repair_attempts: 0

View File

@@ -0,0 +1,10 @@
Return duplicate groups only when the transcript context clearly establishes a
single individual. Prefer no group when identity is ambiguous.
Copy supplied display names into each group's members. Choose canonical_name
from that same group's supplied members. Prefer a complete stable proper name
over an abbreviation, but prefer an unadorned proper name over that name plus a
contextual class, role, title, or relationship descriptor unless the descriptor
is established as part of the name.
Do not invent names, source references, replacement records, or explanations.

View File

@@ -0,0 +1,2 @@
Identify only supplied NPC display names that clearly refer to the same
individual in the supplied transcript context.

View File

@@ -0,0 +1,24 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "notarius.dnd.npcs.normalize.llm",
"type": "object",
"additionalProperties": false,
"required": ["duplicate_groups"],
"properties": {
"duplicate_groups": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["members", "canonical_name"],
"properties": {
"members": {
"type": "array",
"items": {"type": "string"}
},
"canonical_name": {"type": "string"}
}
}
}
}
}

View File

@@ -0,0 +1,214 @@
package npcs
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"sort"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
const semanticContextRadius = 2
type normalizeContextMaterials struct {
Candidates contracts.LLMInputMaterial
Transcript contracts.LLMInputMaterial
candidatePositions []int
}
type normalizeCandidateInput struct {
NPCs []normalizeCandidate `json:"npcs"`
}
type normalizeCandidate struct {
Name string `json:"name"`
SourceRefs []normalizeCandidateSourceRef `json:"source_refs"`
}
type normalizeCandidateSourceRef struct {
StartUnitID int `json:"start_unit_id"`
EndUnitID int `json:"end_unit_id"`
}
type normalizeTranscriptInput struct {
Windows []normalizeTranscriptWindow `json:"windows"`
}
type normalizeTranscriptWindow struct {
Units []normalizeTranscriptUnit `json:"units"`
}
type normalizeTranscriptUnit struct {
ID int `json:"id"`
Kind string `json:"kind"`
Text string `json:"text"`
Metadata map[string]any `json:"metadata,omitempty"`
Cited bool `json:"cited"`
}
type normalizeProposalResponse struct {
DuplicateGroups []normalizeProposalGroup `json:"duplicate_groups"`
}
type normalizeProposalGroup struct {
Members []string `json:"members"`
CanonicalName string `json:"canonical_name"`
}
type sourceInterval struct {
start int
end int
}
func buildDefaultNormalizeContextMaterials(doc *source.SourceDocument, records []dnd.NPC) (normalizeContextMaterials, bool, error) {
return buildNormalizeContextMaterials(doc, records, semanticContextRadius)
}
// buildNormalizeContextMaterials prepares the owned prompt inputs for a
// document-level normalization proposal. A false ready value means semantic
// normalization has no comparison-distinct eligible candidates to consider.
func buildNormalizeContextMaterials(doc *source.SourceDocument, records []dnd.NPC, radius int) (materials normalizeContextMaterials, ready bool, err error) {
if doc == nil {
return normalizeContextMaterials{}, false, nil
}
if radius < 0 {
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: radius must not be negative")
}
index := source.NewDocumentIndex(doc)
candidates := make([]normalizeCandidate, 0, len(records))
candidatePositions := make([]int, 0, len(records))
intervals := make([]sourceInterval, 0)
cited := make([]bool, len(doc.Units))
seenKeys := make(map[string]struct{}, len(records))
for position, record := range records {
key := identity.ComparisonKey(record.Name)
if key == "" || len(record.SourceRefs) == 0 {
continue
}
if _, exists := seenKeys[key]; exists {
continue
}
references, recordIntervals, valid := normalizeRecordReferences(index, record.SourceRefs)
if !valid {
continue
}
seenKeys[key] = struct{}{}
candidates = append(candidates, normalizeCandidate{Name: record.Name, SourceRefs: references})
candidatePositions = append(candidatePositions, position)
for _, interval := range recordIntervals {
for position := interval.start; position <= interval.end; position++ {
cited[position] = true
}
intervals = append(intervals, sourceInterval{
start: maxInt(0, interval.start-radius),
end: minInt(len(doc.Units)-1, interval.end+radius),
})
}
}
if len(candidates) < 2 {
return normalizeContextMaterials{}, false, nil
}
windows, err := normalizeContextWindows(doc.Units, coalesceIntervals(intervals), cited)
if err != nil {
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: copy source metadata: %w", err)
}
candidateContent, err := json.Marshal(normalizeCandidateInput{NPCs: candidates})
if err != nil {
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: encode candidates: %w", err)
}
transcriptContent, err := json.Marshal(normalizeTranscriptInput{Windows: windows})
if err != nil {
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: encode transcript: %w", err)
}
return normalizeContextMaterials{
Candidates: newNormalizeInputMaterial("candidates", candidateContent),
Transcript: newNormalizeInputMaterial("transcript", transcriptContent),
candidatePositions: candidatePositions,
}, true, nil
}
func normalizeRecordReferences(index source.DocumentIndex, refs []source.SourceRef) ([]normalizeCandidateSourceRef, []sourceInterval, bool) {
references := make([]normalizeCandidateSourceRef, 0, len(refs))
intervals := make([]sourceInterval, 0, len(refs))
for _, ref := range refs {
if err := index.ValidateRef(ref); err != nil {
return nil, nil, false
}
start, _ := index.Position(ref.StartUnitID)
end, _ := index.Position(ref.EndUnitID)
references = append(references, normalizeCandidateSourceRef{StartUnitID: ref.StartUnitID, EndUnitID: ref.EndUnitID})
intervals = append(intervals, sourceInterval{start: start, end: end})
}
return references, intervals, true
}
func coalesceIntervals(intervals []sourceInterval) []sourceInterval {
if len(intervals) == 0 {
return nil
}
ordered := append([]sourceInterval(nil), intervals...)
sort.Slice(ordered, func(i, j int) bool {
if ordered[i].start != ordered[j].start {
return ordered[i].start < ordered[j].start
}
return ordered[i].end < ordered[j].end
})
coalesced := make([]sourceInterval, 0, len(ordered))
for _, interval := range ordered {
if len(coalesced) == 0 || interval.start > coalesced[len(coalesced)-1].end+1 {
coalesced = append(coalesced, interval)
continue
}
if interval.end > coalesced[len(coalesced)-1].end {
coalesced[len(coalesced)-1].end = interval.end
}
}
return coalesced
}
func normalizeContextWindows(units []source.SourceUnit, intervals []sourceInterval, cited []bool) ([]normalizeTranscriptWindow, error) {
windows := make([]normalizeTranscriptWindow, 0, len(intervals))
for _, interval := range intervals {
window := normalizeTranscriptWindow{Units: make([]normalizeTranscriptUnit, 0, interval.end-interval.start+1)}
for position := interval.start; position <= interval.end; position++ {
unit := units[position]
metadata, err := source.CloneMetadata(unit.Metadata)
if err != nil {
return nil, err
}
window.Units = append(window.Units, normalizeTranscriptUnit{
ID: unit.ID, Kind: unit.Kind, Text: unit.Text, Metadata: metadata, Cited: cited[position],
})
}
windows = append(windows, window)
}
return windows, nil
}
func newNormalizeInputMaterial(name string, content []byte) contracts.LLMInputMaterial {
digest := sha256.Sum256(content)
return contracts.NewLLMInputMaterial(name, "application/json", content, "sha256:"+hex.EncodeToString(digest[:]), "")
}
func minInt(left, right int) int {
if left < right {
return left
}
return right
}
func maxInt(left, right int) int {
if left > right {
return left
}
return right
}

View File

@@ -0,0 +1,132 @@
package npcs
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
)
func TestBuildNormalizeContextMaterialsUsesDocumentOrderAndOwnedInputs(t *testing.T) {
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
{ID: 40, Kind: "narration", Text: "zero"},
{ID: 10, Kind: "speech", Text: "one", Metadata: map[string]any{"speaker": map[string]any{"name": "Mira"}}},
{ID: 70, Kind: "speech", Text: "two"},
{ID: 20, Kind: "narration", Text: "three"},
{ID: 90, Kind: "speech", Text: "four"},
{ID: 30, Kind: "narration", Text: "five"},
}}
records := []dnd.NPC{
{Name: "Mira Thorn", ID: "npc:sha256:internal", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 20}}},
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 90, EndUnitID: 90}}},
{Name: "Broken", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 10}}},
}
before := append([]dnd.NPC(nil), records...)
materials, ready, err := buildNormalizeContextMaterials(doc, records, 1)
if err != nil || !ready {
t.Fatalf("buildNormalizeContextMaterials() = %#v, %t, %v; want ready materials", materials, ready, err)
}
if !reflect.DeepEqual(records, before) {
t.Fatalf("records mutated to %#v", records)
}
for _, material := range []struct {
name string
data []byte
}{
{name: "candidates", data: materials.Candidates.Content},
{name: "transcript", data: materials.Transcript.Content},
} {
if !json.Valid(material.data) || string(material.data) == "" {
t.Fatalf("%s content = %q, want JSON", material.name, material.data)
}
digest := sha256.Sum256(material.data)
wantDigest := "sha256:" + hex.EncodeToString(digest[:])
got := materials.Candidates
if material.name == "transcript" {
got = materials.Transcript
}
if got.Name != material.name || got.MediaType != "application/json" || got.OriginURI != "" || got.Digest != wantDigest {
t.Fatalf("%s material = %#v, want owned JSON material", material.name, got)
}
}
encoded := string(materials.Candidates.Content) + string(materials.Transcript.Content)
if strings.Contains(encoded, "npc:sha256:internal") || strings.Contains(encoded, doc.ID) {
t.Fatalf("model material leaked private identifier or source id: %s", encoded)
}
var candidates normalizeCandidateInput
if err := json.Unmarshal(materials.Candidates.Content, &candidates); err != nil {
t.Fatal(err)
}
if len(candidates.NPCs) != 2 || candidates.NPCs[0].Name != "Mira Thorn" || candidates.NPCs[0].SourceRefs[0] != (normalizeCandidateSourceRef{StartUnitID: 10, EndUnitID: 20}) {
t.Fatalf("candidates = %#v, want two valid current-order candidates", candidates)
}
var transcript normalizeTranscriptInput
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
t.Fatal(err)
}
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 6 {
t.Fatalf("transcript = %#v, want one coalesced window", transcript)
}
units := transcript.Windows[0].Units
for index, wantID := range []int{40, 10, 70, 20, 90, 30} {
if units[index].ID != wantID {
t.Fatalf("unit %d id = %d, want source-order id %d", index, units[index].ID, wantID)
}
}
if units[0].Cited || !units[1].Cited || !units[2].Cited || !units[3].Cited || !units[4].Cited || units[5].Cited {
t.Fatalf("citation markers = %#v, want original ranges only", units)
}
if units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
t.Fatalf("metadata = %#v, want copied generic metadata", units[1].Metadata)
}
windows, err := normalizeContextWindows(doc.Units, []sourceInterval{{start: 1, end: 1}}, make([]bool, len(doc.Units)))
if err != nil {
t.Fatal(err)
}
windows[0].Units[0].Metadata["speaker"].(map[string]any)["name"] = "changed"
if doc.Units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
t.Fatal("copied metadata aliases source document")
}
}
func TestBuildNormalizeContextMaterialsExcludesInvalidReferencesAndCoalescesAdjacentWindows(t *testing.T) {
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
{ID: 9}, {ID: 3}, {ID: 8}, {ID: 1}, {ID: 7}, {ID: 2}, {ID: 6},
}}
records := []dnd.NPC{
{Name: "One", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 3, EndUnitID: 3}}},
{Name: "Two", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 8, EndUnitID: 8}}},
{Name: "Blank"},
{Name: "Missing", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 99, EndUnitID: 99}}},
{Name: "Foreign", SourceRefs: []source.SourceRef{{SourceID: "other", StartUnitID: 1, EndUnitID: 1}}},
{Name: "Reversed", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 6, EndUnitID: 3}}},
}
materials, ready, err := buildNormalizeContextMaterials(doc, records, 0)
if err != nil || !ready {
t.Fatalf("buildNormalizeContextMaterials() error = %v, ready = %t", err, ready)
}
var candidates normalizeCandidateInput
if err := json.Unmarshal(materials.Candidates.Content, &candidates); err != nil {
t.Fatal(err)
}
if got := []string{candidates.NPCs[0].Name, candidates.NPCs[1].Name}; !reflect.DeepEqual(got, []string{"One", "Two"}) {
t.Fatalf("candidate names = %#v, want only valid records", got)
}
var transcript normalizeTranscriptInput
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
t.Fatal(err)
}
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 2 {
t.Fatalf("windows = %#v, want adjacent cited units coalesced", transcript.Windows)
}
if transcript.Windows[0].Units[0].ID != 3 || transcript.Windows[0].Units[1].ID != 8 {
t.Fatalf("window units = %#v, want document-order adjacent units", transcript.Windows[0].Units)
}
}

View File

@@ -3,8 +3,10 @@ package npcs
import (
"context"
"errors"
"fmt"
"reflect"
"sort"
"strconv"
"strings"
@@ -18,14 +20,18 @@ import (
)
const (
Key = "dnd/npcs"
normalizationPolicy = "dnd.npcs.normalize.v2"
NormalizationPolicy = normalizationPolicy
Key = "dnd/npcs"
normalizationPolicy = "dnd.npcs.normalize.v3"
semanticContextPolicy = "dnd.npcs.semantic_context.v1"
NormalizationPolicy = normalizationPolicy
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
ReasonCodeNPCIDRecomputed = "npc_id_recomputed"
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
ReasonCodeDuplicateNPCCollapsed = "duplicate_npc_collapsed"
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
ReasonCodeNPCIDRecomputed = "npc_id_recomputed"
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
ReasonCodeDuplicateNPCCollapsed = "duplicate_npc_collapsed"
ReasonCodeNPCSemanticProposalInvalid = "npc_semantic_proposal_invalid"
ReasonCodeNPCSemanticReconciliationExhausted = "npc_semantic_reconciliation_exhausted"
ReasonCodeNPCNormalizationWarningsOmitted = "npc_normalization_warnings_omitted"
)
var requiredCapabilities = []string{"merged"}
@@ -36,9 +42,28 @@ var _ contracts.ManifestMetadataProvider = (*Normalizer)(nil)
var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
type Options struct{}
type Normalizer struct{}
func New(Options) *Normalizer { return &Normalizer{} }
type Normalizer struct {
llm contracts.StructuredLLMClient
promptSHA string
responseSchemaSHA string
}
func New(llmClient contracts.StructuredLLMClient, _ Options) (*Normalizer, error) {
if llmClient == nil {
return nil, normalizerErrorf("LLM client must not be nil")
}
promptSHA, err := scriptoriumPromptMetadata()
if err != nil {
return nil, normalizerErrorf("load prompt metadata: %w", err)
}
responseSchema, err := loadResponseSchema()
if err != nil {
return nil, normalizerErrorf("load response schema: %w", err)
}
return &Normalizer{llm: llmClient, promptSHA: promptSHA, responseSchemaSHA: responseSchema.SHA256}, nil
}
func (n *Normalizer) Key() string { return Key }
func (n *Normalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
@@ -46,7 +71,20 @@ func (n *Normalizer) ManifestMetadata() map[string]any {
if n == nil {
return nil
}
return map[string]any{"identity_policy": identity.Policy, "normalization_policy": normalizationPolicy}
return map[string]any{
"prompt_id": PromptID,
"prompt_version": SchemaVersion,
"prompt_sha256": n.promptSHA,
"response_schema_key": string(ResponseSchemaKey),
"response_schema_id": ResponseSchemaID,
"response_schema_name": ResponseSchemaName,
"response_schema_version": SchemaVersion,
"response_schema_sha256": n.responseSchemaSHA,
"identity_policy": identity.Policy,
"normalization_policy": normalizationPolicy,
"semantic_context_policy": semanticContextPolicy,
"semantic_context_radius": semanticContextRadius,
}
}
func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
@@ -54,8 +92,11 @@ func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
return nil
}
return []pipeline.CheckpointFingerprint{
{Name: "prompt", Value: n.promptSHA},
{Name: "response_schema", Value: n.responseSchemaSHA},
{Name: "identity_policy", Value: identity.Policy},
{Name: "normalization_policy", Value: normalizationPolicy},
{Name: "semantic_context_policy", Value: fmt.Sprintf("%s:%d", semanticContextPolicy, semanticContextRadius)},
}
}
@@ -63,30 +104,114 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
if n == nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("normalizer must not be nil")
}
if n.llm == nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("LLM client must not be nil")
}
if ctx == nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context must not be nil")
}
if err := ctx.Err(); err != nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context error before normalize: %w", err)
}
order := shared.NewSourceRefOrder(req.Source)
value, warnings := normalizeList(req.MergeOutput.Value, order)
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: value, Warnings: warnings}, nil
records, warnings := preprocessRecords(req.MergeOutput.Value, order)
deterministic := recordList(records)
materials, ready, err := buildDefaultNormalizeContextMaterials(req.Source, recordValues(records))
if err != nil {
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("build semantic context: %w", err)
}
if !ready {
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
}
var response normalizeProposalResponse
if _, err := n.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
ProfileID: req.LLMProfile, SessionID: req.SessionID,
Inputs: contracts.LLMInputSet{"candidates": materials.Candidates, "transcript": materials.Transcript},
}, &response); err != nil {
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
return n.invalidStructuredResult(deterministic, warnings), nil
}
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("complete structured output: %w", err)
}
assessment := assessProposal(response, records, materials.candidatePositions)
applied, semanticWarnings := applySafeGroups(records, assessment.safeGroups, order)
warnings = append(warnings, semanticWarnings...)
if assessment.discardedGroups == 0 {
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: recordList(applied), Warnings: limitWarnings(warnings)}, nil
}
return retryResult(recordList(applied), warnings, assessment), nil
}
func (n *Normalizer) invalidStructuredResult(value dnd.NPCList, warnings []contracts.Warning) contracts.TypedNormalizeResult[dnd.NPCList] {
return contracts.TypedNormalizeResult[dnd.NPCList]{
Value: value,
Warnings: limitWarningsForRetry(warnings),
Retry: &contracts.NormalizeRetry{
ReasonCode: ReasonCodeNPCSemanticProposalInvalid,
Message: "semantic proposal requires retry: invalid structured output",
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(-1)},
},
}
}
func retryResult(value dnd.NPCList, warnings []contracts.Warning, assessment proposalAssessment) contracts.TypedNormalizeResult[dnd.NPCList] {
return contracts.TypedNormalizeResult[dnd.NPCList]{
Value: value,
Warnings: limitWarningsForRetry(warnings),
Retry: &contracts.NormalizeRetry{
ReasonCode: ReasonCodeNPCSemanticProposalInvalid,
Message: diagnostics.Aggregate("semantic proposal requires retry", assessment.issues),
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(assessment.discardedGroups)},
},
}
}
func semanticFallbackWarning(discardedGroups int) contracts.Warning {
message := "semantic proposal could not be applied"
if discardedGroups >= 0 {
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discardedGroups)
}
return contracts.Warning{Scope: "npcs", ReasonCode: ReasonCodeNPCSemanticReconciliationExhausted, Message: message}
}
func limitWarnings(warnings []contracts.Warning) []contracts.Warning {
return diagnostics.LimitWarnings(warnings, "npcs", ReasonCodeNPCNormalizationWarningsOmitted)
}
func limitWarningsForRetry(warnings []contracts.Warning) []contracts.Warning {
if warnings == nil {
return nil
}
if len(warnings) < diagnostics.MaxWarnings {
return append([]contracts.Warning(nil), warnings...)
}
displayed := diagnostics.MaxWarnings - 2
bounded := append([]contracts.Warning(nil), warnings[:displayed]...)
return append(bounded, contracts.Warning{
Scope: "npcs", ReasonCode: ReasonCodeNPCNormalizationWarningsOmitted,
Message: fmt.Sprintf("%d additional warning(s) omitted", len(warnings)-displayed),
})
}
type normalizedRecord struct {
npc dnd.NPC
npc dnd.NPC
inputIndexes []int
earliest int
}
func normalizeList(input dnd.NPCList, order shared.SourceRefOrder) (dnd.NPCList, []contracts.Warning) {
func preprocessRecords(input dnd.NPCList, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
if input.NPCs == nil {
return dnd.NPCList{}, nil
return nil, nil
}
records := make([]normalizedRecord, len(input.NPCs))
warnings := make([]contracts.Warning, 0)
for index, inputNPC := range input.NPCs {
npc, fieldsChanged, referencesChanged := normalizeRecord(inputNPC, order)
records[index] = normalizedRecord{npc: npc}
records[index] = normalizedRecord{npc: npc, inputIndexes: []int{index}, earliest: index}
if fieldsChanged {
warnings = append(warnings, contracts.Warning{Scope: npcScope(index), ReasonCode: ReasonCodeNPCFieldsNormalized, Message: fmt.Sprintf("input index %d: NPC name normalized for %s", index, diagnostics.Quote(inputNPC.Name))})
}
@@ -99,16 +224,16 @@ func normalizeList(input dnd.NPCList, order shared.SourceRefOrder) (dnd.NPCList,
}
groups := canonicalNameGroups(records)
output := dnd.NPCList{NPCs: make([]dnd.NPC, 0, len(groups))}
output := make([]normalizedRecord, 0, len(groups))
for _, members := range groups {
consolidated, referencesChanged := consolidate(records, members, order)
retainedIndex := members[0]
output.NPCs = append(output.NPCs, consolidated)
output = append(output, consolidated)
retainedIndex := consolidated.earliest
if referencesChanged {
warnings = append(warnings, contracts.Warning{Scope: npcScope(retainedIndex), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized during identity consolidation (final count %d)", retainedIndex, len(consolidated.SourceRefs))})
warnings = append(warnings, contracts.Warning{Scope: npcScope(retainedIndex), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized during identity consolidation (final count %d)", retainedIndex, len(consolidated.npc.SourceRefs))})
}
if len(members) > 1 {
warnings = append(warnings, duplicateWarning(retainedIndex, members[1:]))
warnings = append(warnings, duplicateWarning(retainedIndex, memberInputIndexes(records, members[1:])))
}
}
return output, warnings
@@ -144,15 +269,68 @@ func canonicalNameGroups(records []normalizedRecord) [][]int {
return groups
}
func consolidate(records []normalizedRecord, members []int, order shared.SourceRefOrder) (dnd.NPC, bool) {
output := cloneNPC(records[members[0]].npc)
originalRefs := cloneSourceRefs(output.SourceRefs)
func consolidate(records []normalizedRecord, members []int, order shared.SourceRefOrder) (normalizedRecord, bool) {
output := cloneRecord(records[members[0]])
originalRefs := cloneSourceRefs(output.npc.SourceRefs)
for _, member := range members[1:] {
output.SourceRefs = append(output.SourceRefs, records[member].npc.SourceRefs...)
output.npc.SourceRefs = append(output.npc.SourceRefs, records[member].npc.SourceRefs...)
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
if records[member].earliest < output.earliest {
output.earliest = records[member].earliest
}
}
output.SourceRefs = order.Canonicalize(output.SourceRefs)
output.ID = identity.DeriveID(output.Name)
return output, !reflect.DeepEqual(originalRefs, output.SourceRefs)
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
output.npc.SourceRefs = order.Canonicalize(output.npc.SourceRefs)
output.npc.ID = identity.DeriveID(output.npc.Name)
return output, !reflect.DeepEqual(originalRefs, output.npc.SourceRefs)
}
func cloneRecord(input normalizedRecord) normalizedRecord {
input.npc = cloneNPC(input.npc)
input.inputIndexes = append([]int(nil), input.inputIndexes...)
return input
}
func memberInputIndexes(records []normalizedRecord, members []int) []int {
indexes := make([]int, 0, len(members))
for _, member := range members {
indexes = append(indexes, records[member].inputIndexes...)
}
return sortedUniqueIndexes(indexes)
}
func sortedUniqueIndexes(indexes []int) []int {
if len(indexes) == 0 {
return nil
}
out := append([]int(nil), indexes...)
sort.Ints(out)
write := 1
for _, index := range out[1:] {
if index != out[write-1] {
out[write] = index
write++
}
}
return out[:write]
}
func recordValues(records []normalizedRecord) []dnd.NPC {
if records == nil {
return nil
}
values := make([]dnd.NPC, len(records))
for index, record := range records {
values[index] = cloneNPC(record.npc)
}
return values
}
func recordList(records []normalizedRecord) dnd.NPCList {
if records == nil {
return dnd.NPCList{}
}
return dnd.NPCList{NPCs: recordValues(records)}
}
func cloneSourceRefs(input []source.SourceRef) []source.SourceRef {
@@ -191,7 +369,7 @@ func Register(registry *pipeline.NormalizerRegistry) error {
if err != nil {
return nil, err
}
return New(options), nil
return New(request.Dependencies.LLM, options)
})
}

View File

@@ -2,6 +2,7 @@ package npcs
import (
"context"
"encoding/json"
"reflect"
"strings"
"testing"
@@ -28,11 +29,14 @@ func TestModuleContractAndIdentity(t *testing.T) {
if err := Register(registry); err != nil {
t.Fatalf("Register() error = %v", err)
}
normalizer := New(Options{})
if metadata := normalizer.ManifestMetadata(); metadata["identity_policy"] != identity.Policy || metadata["normalization_policy"] != normalizationPolicy {
if _, err := New(nil, Options{}); err == nil {
t.Fatal("New(nil, Options{}) error = nil, want nil client rejection")
}
normalizer := newNormalizer(t, &recordingNPCNormalizerClient{})
if metadata := normalizer.ManifestMetadata(); metadata["identity_policy"] != identity.Policy || metadata["normalization_policy"] != normalizationPolicy || metadata["prompt_id"] != PromptID || metadata["semantic_context_policy"] != semanticContextPolicy || metadata["semantic_context_radius"] != semanticContextRadius {
t.Fatalf("metadata = %#v", metadata)
}
wantFingerprints := []pipeline.CheckpointFingerprint{{Name: "identity_policy", Value: identity.Policy}, {Name: "normalization_policy", Value: normalizationPolicy}}
wantFingerprints := []pipeline.CheckpointFingerprint{{Name: "prompt", Value: normalizer.promptSHA}, {Name: "response_schema", Value: normalizer.responseSchemaSHA}, {Name: "identity_policy", Value: identity.Policy}, {Name: "normalization_policy", Value: normalizationPolicy}, {Name: "semantic_context_policy", Value: semanticContextPolicy + ":2"}}
if got := normalizer.CheckpointFingerprints(); !reflect.DeepEqual(got, wantFingerprints) {
t.Fatalf("fingerprints = %#v, want %#v", got, wantFingerprints)
}
@@ -46,7 +50,7 @@ func TestNormalizeNamesEvidenceAndIDs(t *testing.T) {
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
},
}}}
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
@@ -68,7 +72,7 @@ func TestNormalizeOrdersEvidenceBySourceDocumentPosition(t *testing.T) {
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30},
{SourceID: doc.ID, StartUnitID: 999, EndUnitID: 999},
}}}}
result, err := New(Options{}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
@@ -84,7 +88,7 @@ func TestNormalizeConsolidatesCanonicalNamesOnlyAndUnionsEvidence(t *testing.T)
{Name: "captain vale", SourceRefs: []source.SourceRef{{SourceID: "b", StartUnitID: 2, EndUnitID: 2}}},
{Name: "The Captain", SourceRefs: []source.SourceRef{{SourceID: "c", StartUnitID: 3, EndUnitID: 3}}},
}}
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
if err != nil {
t.Fatalf("Normalize() error = %v", err)
}
@@ -102,7 +106,7 @@ func TestNormalizeConsolidatesCanonicalNamesOnlyAndUnionsEvidence(t *testing.T)
func TestNormalizePreservesInvalidCandidatesAndDoesNotAliasInput(t *testing.T) {
input := dnd.NPCList{NPCs: []dnd.NPC{{Name: " ", SourceRefs: []source.SourceRef{{SourceID: "source", StartUnitID: 0, EndUnitID: -1}}}}}
before := dnd.NPCList{NPCs: []dnd.NPC{{Name: " ", SourceRefs: []source.SourceRef{{SourceID: "source", StartUnitID: 0, EndUnitID: -1}}}}}
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
if err != nil || !reflect.DeepEqual(input, before) {
t.Fatalf("Normalize() = %#v, %v; input mutated to %#v", result, err, input)
}
@@ -116,7 +120,7 @@ func TestNormalizePreservesInvalidCandidatesAndDoesNotAliasInput(t *testing.T) {
}
func TestNormalizeHandlesNilAndCanceledCalls(t *testing.T) {
normalizer := New(Options{})
normalizer := newNormalizer(t, &recordingNPCNormalizerClient{})
result, err := normalizer.Normalize(context.Background(), normalizeRequest(dnd.NPCList{NPCs: nil}))
if err != nil || result.Value.NPCs != nil {
t.Fatalf("nil list result = %#v, error = %v", result.Value, err)
@@ -128,6 +132,44 @@ func TestNormalizeHandlesNilAndCanceledCalls(t *testing.T) {
}
}
type recordingNPCNormalizerClient struct {
response string
responses []string
err error
requests []contracts.StructuredCompletionRequest
}
func (c *recordingNPCNormalizerClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, output any) (contracts.StructuredCompletionResponse, error) {
c.requests = append(c.requests, request)
if c.err != nil {
return contracts.StructuredCompletionResponse{}, c.err
}
response := c.response
if len(c.responses) > 0 {
responseIndex := len(c.requests) - 1
if responseIndex >= len(c.responses) {
responseIndex = len(c.responses) - 1
}
response = c.responses[responseIndex]
}
if response == "" {
response = `{"duplicate_groups":[]}`
}
if err := json.Unmarshal([]byte(response), output); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return contracts.StructuredCompletionResponse{Content: json.RawMessage(response)}, nil
}
func newNormalizer(t *testing.T, client contracts.StructuredLLMClient) *Normalizer {
t.Helper()
normalizer, err := New(client, Options{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
return normalizer
}
func normalizeRequest(value dnd.NPCList) contracts.TypedNormalizeRequest[dnd.NPCList] {
return contracts.TypedNormalizeRequest[dnd.NPCList]{MergeOutput: contracts.MergeArtifact[dnd.NPCList]{Value: value}}
}

View File

@@ -0,0 +1,195 @@
package npcs
import (
"fmt"
"sort"
"strconv"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
)
type proposalAssessment struct {
safeGroups []safeProposalGroup
discardedGroups int
issues []string
}
type safeProposalGroup struct {
members []int
canonical int
}
type assessedProposalGroup struct {
members []int
canonical int
locallyValid bool
conflicting bool
}
func assessProposal(response normalizeProposalResponse, records []normalizedRecord, candidatePositions []int) proposalAssessment {
positionsByKey := make(map[string][]int, len(candidatePositions))
for _, position := range candidatePositions {
if position < 0 || position >= len(records) {
continue
}
key := identity.ComparisonKey(records[position].npc.Name)
if key != "" {
positionsByKey[key] = append(positionsByKey[key], position)
}
}
groups := make([]assessedProposalGroup, len(response.DuplicateGroups))
issues := make([]string, 0)
owners := make(map[int][]int)
for groupIndex, proposal := range response.DuplicateGroups {
group, groupIssues := assessProposalGroup(proposal, positionsByKey)
groups[groupIndex] = group
for _, position := range group.members {
owners[position] = append(owners[position], groupIndex)
}
for _, issue := range groupIssues {
issues = append(issues, proposalIssue(groupIndex, issue))
}
}
for groupIndex := range groups {
for _, position := range groups[groupIndex].members {
if len(owners[position]) > 1 {
groups[groupIndex].conflicting = true
break
}
}
if groups[groupIndex].conflicting {
issues = append(issues, proposalIssue(groupIndex, "overlapping_member"))
}
}
assessment := proposalAssessment{issues: issues}
for _, group := range groups {
if !group.locallyValid || group.conflicting {
assessment.discardedGroups++
continue
}
assessment.safeGroups = append(assessment.safeGroups, safeProposalGroup{members: group.members, canonical: group.canonical})
}
return assessment
}
func assessProposalGroup(proposal normalizeProposalGroup, positionsByKey map[string][]int) (assessedProposalGroup, []string) {
issues := make([]string, 0)
members := make([]int, 0, len(proposal.Members))
seenMembers := make(map[int]struct{}, len(proposal.Members))
for _, name := range proposal.Members {
position, issue := resolveCandidate(name, positionsByKey)
if issue != "" {
issues = append(issues, "member_"+issue)
continue
}
if _, exists := seenMembers[position]; exists {
issues = append(issues, "repeated_member")
continue
}
seenMembers[position] = struct{}{}
members = append(members, position)
}
canonical, canonicalIssue := resolveCandidate(proposal.CanonicalName, positionsByKey)
if canonicalIssue != "" {
issues = append(issues, "canonical_"+canonicalIssue)
}
if len(members) < 2 {
issues = append(issues, "fewer_than_two_members")
}
if canonicalIssue == "" && !containsPosition(members, canonical) {
issues = append(issues, "canonical_not_member")
}
sort.Ints(members)
return assessedProposalGroup{
members: members, canonical: canonical, locallyValid: len(issues) == 0,
}, issues
}
func resolveCandidate(name string, positionsByKey map[string][]int) (position int, issue string) {
key := identity.ComparisonKey(name)
if key == "" {
return 0, "blank"
}
positions := positionsByKey[key]
if len(positions) == 0 {
return 0, "unknown"
}
if len(positions) != 1 {
return 0, "ambiguous"
}
return positions[0], ""
}
func containsPosition(positions []int, want int) bool {
for _, position := range positions {
if position == want {
return true
}
}
return false
}
func proposalIssue(groupIndex int, category string) string {
return "group " + strconv.Itoa(groupIndex) + ": " + category
}
func applySafeGroups(records []normalizedRecord, groups []safeProposalGroup, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
byMember := make(map[int]safeProposalGroup, len(groups)*2)
for _, group := range groups {
for _, member := range group.members {
byMember[member] = group
}
}
output := make([]normalizedRecord, 0, len(records)-len(groups))
warnings := make([]contracts.Warning, 0, len(groups))
for index, record := range records {
group, grouped := byMember[index]
if !grouped {
output = append(output, cloneRecord(record))
continue
}
if group.members[0] != index {
continue
}
consolidated := consolidateSemanticGroup(records, group, order)
output = append(output, consolidated)
warnings = append(warnings, semanticDuplicateWarning(consolidated, records[group.canonical]))
}
return output, warnings
}
func consolidateSemanticGroup(records []normalizedRecord, group safeProposalGroup, order shared.SourceRefOrder) normalizedRecord {
output := cloneRecord(records[group.members[0]])
output.npc.Name = records[group.canonical].npc.Name
for _, member := range group.members[1:] {
output.npc.SourceRefs = append(output.npc.SourceRefs, records[member].npc.SourceRefs...)
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
if records[member].earliest < output.earliest {
output.earliest = records[member].earliest
}
}
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
output.npc.SourceRefs = order.Canonicalize(output.npc.SourceRefs)
output.npc.ID = identity.DeriveID(output.npc.Name)
return output
}
func semanticDuplicateWarning(record normalizedRecord, canonical normalizedRecord) contracts.Warning {
details := make([]string, 0, len(record.inputIndexes)+1)
for _, inputIndex := range record.inputIndexes {
details = append(details, fmt.Sprintf("input index %d", inputIndex))
}
if canonical.earliest != record.earliest {
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
}
return contracts.Warning{
Scope: npcScope(record.earliest),
ReasonCode: ReasonCodeDuplicateNPCCollapsed,
Message: diagnostics.Aggregate("semantic duplicate consolidation", details),
}
}

View File

@@ -0,0 +1,21 @@
package npcs
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
const (
PromptID = "dnd.npcs.normalize"
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_npcs_normalize_llm")
ResponseSchemaID = "notarius.dnd.npcs.normalize.llm"
ResponseSchemaName = "notarius_dnd_npcs_normalize_llm_v1"
SchemaVersion = "v1"
)
func loadResponseSchema() (llm.ResponseSchema, error) {
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
Key: ResponseSchemaKey,
ID: ResponseSchemaID,
Version: SchemaVersion,
Name: ResponseSchemaName,
AssetPath: "assets/schemas/dnd_npcs_normalize_llm.v1.json",
})
}

View File

@@ -0,0 +1,65 @@
package npcs
import (
"bytes"
"encoding/json"
"strings"
"testing"
"github.com/santhosh-tekuri/jsonschema/v6"
)
func TestNormalizeResponseSchemaIsStrictlyStructural(t *testing.T) {
schema, err := loadResponseSchema()
if err != nil {
t.Fatalf("loadResponseSchema() error = %v", err)
}
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Name != ResponseSchemaName || schema.Version != SchemaVersion || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
t.Fatalf("schema = %#v, want private normalization schema identity", schema)
}
for _, test := range []struct {
name string
value any
valid bool
}{
{name: "empty groups", value: map[string]any{"duplicate_groups": []any{}}, valid: true},
{name: "semantically invalid group", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{"", "unknown"}, "canonical_name": ""}}}, valid: true},
{name: "missing groups", value: map[string]any{}},
{name: "unknown top level field", value: map[string]any{"duplicate_groups": []any{}, "extra": true}},
{name: "unknown group field", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{}, "canonical_name": "Mira", "extra": true}}}},
{name: "wrong groups type", value: map[string]any{"duplicate_groups": "no"}},
{name: "wrong member type", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{1}, "canonical_name": "Mira"}}}},
} {
t.Run(test.name, func(t *testing.T) {
content, err := json.Marshal(test.value)
if err != nil {
t.Fatal(err)
}
err = validateNormalizeSchema(content, schema.JSONSchema)
if (err == nil) != test.valid {
t.Fatalf("validateNormalizeSchema() error = %v, want valid=%t", err, test.valid)
}
})
}
}
func validateNormalizeSchema(instanceContent, schemaContent []byte) error {
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(instanceContent))
if err != nil {
return err
}
document, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
if err != nil {
return err
}
compiler := jsonschema.NewCompiler()
if err := compiler.AddResource("schema.json", document); err != nil {
return err
}
compiled, err := compiler.Compile("schema.json")
if err != nil {
return err
}
return compiled.Validate(instance)
}

View File

@@ -0,0 +1,51 @@
package npcs
import (
"fmt"
"sync"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
)
const scriptoriumPromptRoot = "assets/prompts"
var promptAssetManifest = shared.PromptAssetManifest{
ModuleDir: PromptID,
ModuleFiles: []promptfs.ModulePromptFile{
{Name: "dnd.npcs.normalize.yaml", Path: "assets/prompts/dnd.npcs.normalize.yaml"},
{Name: "task.md", Path: "assets/prompts/task.md"},
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
{Name: "candidates.md", Path: "assets/prompts/candidates.md"},
},
SharedFiles: []string{
"common-dnd-system.md",
"common-dnd-identity.md",
"common-dnd-transcript.md",
},
}
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
promptFS, err := promptAssetManifest.PromptFS(embeddedAssets)
if err != nil {
return fmt.Errorf("prepare NPC normalization prompt assets: %w", err)
}
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
return err
}
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
}
func scriptoriumPromptMetadata() (string, error) {
scriptoriumPromptHashOnce.Do(func() {
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
})
return scriptoriumPromptHash, scriptoriumPromptHashErr
}
var (
scriptoriumPromptHashOnce sync.Once
scriptoriumPromptHash string
scriptoriumPromptHashErr error
)

View File

@@ -0,0 +1,61 @@
package npcs
import (
"context"
"strings"
"testing"
"time"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
"gitea.maximumdirect.net/eric/scriptorium"
)
func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
registry := llm.NewAssetRegistry()
if err := RegisterPromptAssets(registry); err != nil {
t.Fatalf("RegisterPromptAssets() error = %v", err)
}
options, err := registry.ScriptoriumOptions()
if err != nil {
t.Fatalf("ScriptoriumOptions() error = %v", err)
}
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
ID: "normalize-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "normalize-test-model",
})))
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
if err != nil {
t.Fatalf("NewEngine() error = %v", err)
}
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "normalize-test-profile",
Inputs: map[string]scriptorium.ArtifactRef{
"candidates": scriptorium.Inline(`{"npcs":[{"name":"Mira","source_refs":[]}]}`),
"transcript": scriptorium.Inline(`{"windows":[{"units":[]}]}`),
},
})
if err != nil {
t.Fatalf("Prepare() error = %v", err)
}
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_npcs_normalize_llm.v1.json" {
t.Fatalf("prepared prompt = %#v, want normalization prompt identity and schema", prepared)
}
if len(prepared.Messages) != 6 {
t.Fatalf("prepared messages = %d, want 6", len(prepared.Messages))
}
for _, index := range []int{1, 3} {
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != scriptorium.CacheControlEphemeral {
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
}
}
for _, index := range []int{0, 2, 4, 5} {
if cache := prepared.Messages[index].CacheControl; cache != nil {
t.Errorf("message %d cache control = %#v, want nil", index, cache)
}
}
if !strings.Contains(prepared.Messages[4].Content, `"Mira"`) || strings.Contains(prepared.Messages[4].Content, `"windows"`) {
t.Fatalf("candidate message = %q, want only rendered candidates", prepared.Messages[4].Content)
}
if !strings.Contains(prepared.Messages[5].Content, `"windows"`) || strings.Contains(prepared.Messages[5].Content, `"Mira"`) {
t.Fatalf("transcript message = %q, want only rendered transcript", prepared.Messages[5].Content)
}
}

View File

@@ -0,0 +1,265 @@
package npcs
import (
"context"
"errors"
"math"
"reflect"
"strings"
"testing"
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
func TestNormalizeSkipsSemanticCompletionWithoutTwoEligibleCandidates(t *testing.T) {
client := &recordingNPCNormalizerClient{}
normalizer := newNormalizer(t, client)
doc := semanticDocument()
input := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}}}
result, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil || len(client.requests) != 0 || result.Retry != nil {
t.Fatalf("Normalize() = %#v, %v; calls = %d, want deterministic no-call result", result, err, len(client.requests))
}
if result.Value.NPCs[0].Name != "Mira Thorn" {
t.Fatalf("NPCs = %#v, want deterministic record", result.Value.NPCs)
}
}
func TestNormalizeAppliesSafeProposalAndUsesPrivateInputs(t *testing.T) {
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":[" MIRA THORN ","Mira"],"canonical_name":"Mira Thorn"}]}`}
normalizer := newNormalizer(t, client)
doc := semanticDocument()
input := dnd.NPCList{NPCs: []dnd.NPC{
{ID: "npc:sha256:short", Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
{ID: "npc:sha256:long", Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
{ID: "npc:sha256:captain", Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
}}
before := cloneNPCList(input)
request := normalizeRequestWithSource(input, doc)
request.LLMProfile = "normalizer-profile"
request.SessionID = "normalizer-session"
result, err := normalizer.Normalize(context.Background(), request)
if err != nil || result.Retry != nil {
t.Fatalf("Normalize() = %#v, %v; want accepted semantic result", result, err)
}
if !reflect.DeepEqual(input, before) {
t.Fatalf("Normalize() mutated input to %#v", input)
}
if len(result.Value.NPCs) != 2 || result.Value.NPCs[0].Name != "Mira Thorn" || result.Value.NPCs[1].Name != "Captain Vale" {
t.Fatalf("NPCs = %#v, want canonical record at earliest position", result.Value.NPCs)
}
merged := result.Value.NPCs[0]
if merged.ID != identity.DeriveID("Mira Thorn") || !reflect.DeepEqual(merged.SourceRefs, []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}, {SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}) {
t.Fatalf("merged NPC = %#v, want canonical ID and original evidence union", merged)
}
if !hasWarning(result.Warnings, ReasonCodeDuplicateNPCCollapsed, "npcs[0]") {
t.Fatalf("warnings = %#v, want semantic collapse warning", result.Warnings)
}
if len(client.requests) != 1 {
t.Fatalf("completion calls = %d, want one", len(client.requests))
}
completion := client.requests[0]
if completion.StageName != Key || completion.PromptID != PromptID || completion.PromptVersion != SchemaVersion || completion.ProfileID != request.LLMProfile || completion.SessionID != request.SessionID || len(completion.Inputs) != 2 {
t.Fatalf("completion request = %#v, want normalize request identity and exactly two inputs", completion)
}
encoded := string(completion.Inputs["candidates"].Content) + string(completion.Inputs["transcript"].Content)
if strings.Contains(encoded, "npc:sha256:") || strings.Contains(encoded, doc.ID) {
t.Fatalf("completion inputs leaked private identifiers: %s", encoded)
}
}
func TestNormalizeUnsafeProposalReturnsSafeRetryFallback(t *testing.T) {
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Captain Vale","Unknown"],"canonical_name":"Captain Vale"}]}`}
normalizer := newNormalizer(t, client)
doc := semanticDocument()
input := dnd.NPCList{NPCs: []dnd.NPC{
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
}}
result, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil || result.Retry == nil {
t.Fatalf("Normalize() = %#v, %v; want retryable safe fallback", result, err)
}
if result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || !strings.Contains(result.Retry.Message, "group 1: member_unknown") {
t.Fatalf("retry = %#v, want bounded invalid-proposal diagnostics", result.Retry)
}
if len(result.Value.NPCs) != 2 || result.Value.NPCs[0].Name != "Mira Thorn" || result.Value.NPCs[1].Name != "Captain Vale" {
t.Fatalf("fallback NPCs = %#v, want independently safe group applied", result.Value.NPCs)
}
if len(result.Retry.FallbackWarnings) != 1 || result.Retry.FallbackWarnings[0].ReasonCode != ReasonCodeNPCSemanticReconciliationExhausted || !strings.Contains(result.Retry.FallbackWarnings[0].Message, "1 proposal group") {
t.Fatalf("fallback warnings = %#v, want exact omitted-group warning", result.Retry.FallbackWarnings)
}
}
func TestNormalizeRejectsOverlapsWithoutResponseOrderDependence(t *testing.T) {
records := []normalizedRecord{
{npc: dnd.NPC{Name: "Alpha"}}, {npc: dnd.NPC{Name: "Bravo"}}, {npc: dnd.NPC{Name: "Charlie"}}, {npc: dnd.NPC{Name: "Delta"}},
}
for index := range records {
records[index].inputIndexes = []int{index}
records[index].earliest = index
}
response := normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{
{Members: []string{"Alpha", "Bravo"}, CanonicalName: "Alpha"},
{Members: []string{"Bravo", "Charlie"}, CanonicalName: "Bravo"},
{Members: []string{"Charlie", "Delta"}, CanonicalName: "Charlie"},
}}
assessment := assessProposal(response, records, []int{0, 1, 2, 3})
if assessment.discardedGroups != 3 || len(assessment.safeGroups) != 0 {
t.Fatalf("assessment = %#v, want chained conflicts all discarded", assessment)
}
for _, issue := range []string{"group 0: overlapping_member", "group 1: overlapping_member", "group 2: overlapping_member"} {
if !containsString(assessment.issues, issue) {
t.Fatalf("issues = %#v, want %q", assessment.issues, issue)
}
}
}
func TestNormalizeInvalidStructuredOutputAndOperationalErrorsRemainDistinct(t *testing.T) {
doc := semanticDocument()
input := dnd.NPCList{NPCs: []dnd.NPC{
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
}}
invalid := newNormalizer(t, &recordingNPCNormalizerClient{err: contracts.ErrInvalidStructuredOutput})
result, err := invalid.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil || result.Retry == nil || result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || len(result.Value.NPCs) != 2 {
t.Fatalf("invalid structured result = %#v, %v; want deterministic retry fallback", result, err)
}
operational := errors.New("provider unavailable")
if _, err := newNormalizer(t, &recordingNPCNormalizerClient{err: operational}).Normalize(context.Background(), normalizeRequestWithSource(input, doc)); !errors.Is(err, operational) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
t.Fatalf("operational completion error = %v, want ordinary error", err)
}
}
func TestNormalizeRejectsContextEncodingFailuresWithoutLeakingContent(t *testing.T) {
client := &recordingNPCNormalizerClient{}
normalizer := newNormalizer(t, client)
doc := semanticDocument()
doc.Units[0].Metadata = map[string]any{"invalid": math.NaN()}
input := dnd.NPCList{NPCs: []dnd.NPC{
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
}}
if _, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc)); err == nil || !strings.Contains(err.Error(), "build semantic context") || strings.Contains(err.Error(), doc.Units[0].Text) || len(client.requests) != 0 {
t.Fatalf("Normalize() error = %v, calls = %d; want safe preparation error before completion", err, len(client.requests))
}
}
func TestNormalizeDoesNotAccumulateSafeGroupsAcrossAttempts(t *testing.T) {
client := &recordingNPCNormalizerClient{responses: []string{
`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Captain Vale","Unknown"],"canonical_name":"Captain Vale"}]}`,
`{"duplicate_groups":[{"members":["Mira","Captain Vale"],"canonical_name":"Captain Vale"}]}`,
}}
normalizer := newNormalizer(t, client)
doc := semanticDocument()
input := dnd.NPCList{NPCs: []dnd.NPC{
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
}}
first, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil || first.Retry == nil || len(first.Value.NPCs) != 2 {
t.Fatalf("first Normalize() = %#v, %v; want partial retry fallback", first, err)
}
second, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil || second.Retry != nil {
t.Fatalf("second Normalize() = %#v, %v; want accepted independent retry result", second, err)
}
if got := []string{second.Value.NPCs[0].Name, second.Value.NPCs[1].Name}; !reflect.DeepEqual(got, []string{"Captain Vale", "Mira Thorn"}) {
t.Fatalf("second NPCs = %#v, want proposal applied to original merge output", second.Value.NPCs)
}
}
func TestNormalizeExcludesIneligibleRecordsFromSemanticGroups(t *testing.T) {
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["Mira","Broken"],"canonical_name":"Mira"}]}`}
normalizer := newNormalizer(t, client)
doc := semanticDocument()
input := dnd.NPCList{NPCs: []dnd.NPC{
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
{Name: "Broken", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 999, EndUnitID: 999}}},
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
}}
result, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
if err != nil || result.Retry == nil || len(result.Value.NPCs) != 3 {
t.Fatalf("Normalize() = %#v, %v; want unchanged retry fallback", result, err)
}
if !strings.Contains(result.Retry.Message, "member_unknown") || result.Value.NPCs[1].Name != "Broken" {
t.Fatalf("result = %#v, want ineligible record excluded but preserved", result)
}
}
func TestProposalValidationRejectsUnsafeCategories(t *testing.T) {
records := []normalizedRecord{
{npc: dnd.NPC{Name: "Mira"}, inputIndexes: []int{0}, earliest: 0},
{npc: dnd.NPC{Name: "Mira Thorn"}, inputIndexes: []int{1}, earliest: 1},
{npc: dnd.NPC{Name: "Captain Vale"}, inputIndexes: []int{2}, earliest: 2},
}
for _, proposal := range []normalizeProposalGroup{
{Members: []string{"Mira"}, CanonicalName: "Mira"},
{Members: []string{"Mira", "Mira"}, CanonicalName: "Mira"},
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: "Unknown"},
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: " "},
{Members: []string{" ", "Mira Thorn"}, CanonicalName: "Mira Thorn"},
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: "Captain Vale"},
} {
assessment := assessProposal(normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{proposal}}, records, []int{0, 1, 2})
if assessment.discardedGroups != 1 || len(assessment.safeGroups) != 0 {
t.Fatalf("assessment for %#v = %#v, want discarded unsafe group", proposal, assessment)
}
}
}
func TestProposalResolutionUsesOnlyExistingComparisonKeyEquivalences(t *testing.T) {
records := []normalizedRecord{
{npc: dnd.NPC{Name: "O'Neill"}, inputIndexes: []int{0}, earliest: 0},
{npc: dnd.NPC{Name: "Mira Thorn"}, inputIndexes: []int{1}, earliest: 1},
}
assessment := assessProposal(normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{{
Members: []string{" ONEILL ", " "}, CanonicalName: " ",
}}}, records, []int{0, 1})
if assessment.discardedGroups != 0 || len(assessment.safeGroups) != 1 || assessment.safeGroups[0].canonical != 1 {
t.Fatalf("assessment = %#v, want comparison-key-only resolution", assessment)
}
}
func TestRetryWarningLimitReservesExhaustionWarningPosition(t *testing.T) {
warnings := make([]contracts.Warning, 0, 25)
for index := 0; index < 25; index++ {
warnings = append(warnings, contracts.Warning{Scope: "npcs", ReasonCode: "test", Message: "warning"})
}
bounded := limitWarningsForRetry(warnings)
if len(bounded) != 19 || bounded[len(bounded)-1].ReasonCode != ReasonCodeNPCNormalizationWarningsOmitted || !strings.Contains(bounded[len(bounded)-1].Message, "7 additional") {
t.Fatalf("retry warnings = %#v, want 18 warnings plus accurate omission summary", bounded)
}
}
func semanticDocument() *source.SourceDocument {
return &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
{ID: 10, Kind: "speech", Text: "Mira speaks"},
{ID: 15, Kind: "narration", Text: "surrounding context"},
{ID: 20, Kind: "speech", Text: "Mira Thorn replies"},
{ID: 30, Kind: "speech", Text: "Captain Vale watches"},
}}
}
func cloneNPCList(input dnd.NPCList) dnd.NPCList {
output := dnd.NPCList{NPCs: make([]dnd.NPC, len(input.NPCs))}
for index, npc := range input.NPCs {
output.NPCs[index] = cloneNPC(npc)
}
return output
}
func containsString(values []string, want string) bool {
for _, value := range values {
if value == want {
return true
}
}
return false
}

View File

@@ -95,6 +95,7 @@ func registerPromptAssets(assets *llm.AssetRegistry) error {
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
{name: "spells prompt assets", register: func() error { return spellextract.RegisterPromptAssets(assets) }},
{name: "npcs prompt assets", register: func() error { return npcextract.RegisterPromptAssets(assets) }},
{name: "npc normalization prompt assets", register: func() error { return npcnormalize.RegisterPromptAssets(assets) }},
{name: "combat turns prompt assets", register: func() error { return combatextract.RegisterPromptAssets(assets) }},
{name: "item events prompt assets", register: func() error { return itemeventextract.RegisterPromptAssets(assets) }},
{name: "npc interactions prompt assets", register: func() error { return interactionextract.RegisterPromptAssets(assets) }},

View File

@@ -32,6 +32,20 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
if err := Register(registries, assets); err != nil {
t.Fatalf("Register() error = %v, want nil", err)
}
promptFS, err := assets.PromptFS()
if err != nil {
t.Fatalf("PromptFS() error = %v", err)
}
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
}
schemaFS, err := assets.SchemaFS()
if err != nil {
t.Fatalf("SchemaFS() error = %v", err)
}
if _, err := fs.ReadFile(schemaFS, "dnd_npcs_normalize_llm.v1.json"); err != nil {
t.Fatalf("normalization schema asset = %v, want registered private schema", err)
}
assertContainsKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells", npcextract.Key, combatextract.Key, itemeventextract.Key, interactionextract.Key, scenedescriptionextract.Key})
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, npcnormalize.Key, combatnormalize.Key, itemeventnormalize.Key, interactionnormalize.Key, scenedescriptionnormalize.Key, pipeline.DefaultNormalizeModule})

View File

@@ -24,6 +24,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
sceneextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
)
func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T) {
@@ -47,6 +48,9 @@ func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T
}
for name, value := range map[string]string{
"extract:npcs:dnd/npcs:mapping_policy": "dnd.npcs.extract_mapping.v2",
"normalize:npcs:dnd/npcs:identity_policy": "dnd.npcs.identity.v1",
"normalize:npcs:dnd/npcs:normalization_policy": "dnd.npcs.normalize.v3",
"normalize:npcs:dnd/npcs:semantic_context_policy": "dnd.npcs.semantic_context.v1:2",
"extract:spells:dnd/spells:mapping_policy": "dnd.spells.extract_mapping.v2",
"extract:combat:dnd/combat-turns:scene_gate_policy": "dnd.combat_turns.scene_gate.v1",
} {
@@ -55,6 +59,8 @@ func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T
for _, name := range []string{
"extract:npcs:dnd/npcs:prompt",
"extract:npcs:dnd/npcs:response_schema",
"normalize:npcs:dnd/npcs:prompt",
"normalize:npcs:dnd/npcs:response_schema",
"extract:spells:dnd/spells:prompt",
"extract:spells:dnd/spells:response_schema",
"extract:spells:dnd/spells:npc_registry",
@@ -455,6 +461,8 @@ func (client *groundedDNDLLMClient) CompleteStructured(ctx context.Context, requ
"name": "Hooded Guard", "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}},
},
}}
case npcnormalize.PromptID:
payload = map[string]any{"duplicate_groups": []any{}}
case sceneextract.PromptID:
kind := client.sceneKind
if kind == "" {

View File

@@ -5,6 +5,8 @@ import (
"encoding/json"
"fmt"
"os"
"reflect"
"strings"
"sync"
"testing"
@@ -16,6 +18,8 @@ import (
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
interactionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
)
func TestNPCInteractionPipelineUsesAcceptedRegistryAndCurrentEvidence(t *testing.T) {
@@ -70,6 +74,59 @@ func TestNPCInteractionPipelineUsesAcceptedRegistryAndCurrentEvidence(t *testing
}
}
func TestSemanticNPCNormalizationCrossesOrderedRegistryHandoff(t *testing.T) {
registries := productionNPCRegistries(t)
cfg := loadNPCInteractionPipelineConfig(t)
profile := cfg.Pipelines["dnd-npc-interactions-fixture"]
profile.Chunk = pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}}
cfg.Pipelines["dnd-npc-interactions-fixture"] = profile
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-npc-interactions-fixture", Catalog: moduleCatalog(registries)})
if err != nil {
t.Fatal(err)
}
resolved, warnings, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, moduleCatalog(registries), pipeline.ReferenceMaterializationOptions{})
if err != nil || len(warnings) != 0 {
t.Fatalf("MaterializeReferences() error = %v warnings = %#v", err, warnings)
}
client := &semanticNPCInteractionClient{}
raw := []byte(`{"metadata":{"id":"semantic-session","title":"Semantic NPC session"},"segments":[{"id":1,"start":0,"end":1,"speaker":"DM","text":"Mira Thorn enters."},{"id":2,"start":1,"end":2,"speaker":"DM","text":"Mira Thorn, the Greencloak, waves."}]}`)
output, err := runPreparedPipeline(t, registries, resolved, client, pipeline.RunInput{RawInput: raw})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if client.requestCount(npcs.PromptID) != 2 || client.requestCount(npcnormalize.PromptID) != 1 {
t.Fatalf("prompt requests = %#v, want two extraction calls and one document normalization call", client.requests)
}
npcOutput := normalizedLane(t, output, "npcs")
npcsValue, err := npccodec.New().Decode(npcOutput.Artifact.Content)
if err != nil {
t.Fatal(err)
}
if npcOutput.StepID != "identify-npcs" || len(npcsValue.NPCs) != 1 || npcsValue.NPCs[0].Name != "Mira Thorn" || npcsValue.NPCs[0].ID != identity.DeriveID("Mira Thorn") {
t.Fatalf("NPC output = %#v / %#v, want canonical ordered producer artifact", npcOutput, npcsValue)
}
if refs := npcsValue.NPCs[0].SourceRefs; !reflect.DeepEqual(refs, []source.SourceRef{{SourceID: "semantic-session", StartUnitID: 1, EndUnitID: 1}, {SourceID: "semantic-session", StartUnitID: 2, EndUnitID: 2}}) {
t.Fatalf("NPC evidence = %#v, want original extraction evidence union", refs)
}
interactionOutput := normalizedLane(t, output, "interactions")
if interactionOutput.StepID != "extract-interactions" {
t.Fatalf("interaction output step = %q, want ordered downstream step", interactionOutput.StepID)
}
registryRequest := client.requestFor(t, interactionextract.PromptID)
if got := string(registryRequest.Inputs["npcs"].Content); got != `{"npcs":[{"name":"Mira Thorn"}]}` {
t.Fatalf("downstream registry = %s, want one canonical names-only NPC", got)
}
manifestContent, err := json.Marshal(output.Manifest)
if err != nil {
t.Fatal(err)
}
for _, forbidden := range []string{"npc:sha256:", "Mira Thorn enters.", "Mira Thorn, the Greencloak, waves."} {
if strings.Contains(string(manifestContent), forbidden) {
t.Fatalf("manifest leaked private identity or transcript content %q: %s", forbidden, manifestContent)
}
}
}
func TestNPCInteractionPipelineSkipsConsumerWhenNPCProducerIsRejected(t *testing.T) {
registries := productionNPCRegistries(t)
resolved := resolveNPCInteractionPipeline(t, registries)
@@ -168,6 +225,8 @@ func (client *npcInteractionLLMClient) CompleteStructured(ctx context.Context, r
map[string]any{"name": "Hooded Guard", "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}}},
}}
}
case npcnormalize.PromptID:
payload = map[string]any{"duplicate_groups": []any{}}
case interactionextract.PromptID:
payload = map[string]any{"interactions": []any{
map[string]any{"name": "Hooded Guard", "kind": "noncombat_presence", "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}}},
@@ -212,3 +271,59 @@ func (client *npcInteractionLLMClient) requestCount(promptID string) int {
}
var _ contracts.StructuredLLMClient = (*npcInteractionLLMClient)(nil)
type semanticNPCInteractionClient struct {
requests []contracts.StructuredCompletionRequest
npcCalls int
}
func (client *semanticNPCInteractionClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
client.requests = append(client.requests, cloneStructuredCompletionRequest(request))
var payload any
switch request.PromptID {
case npcs.PromptID:
client.npcCalls++
name := "Mira Thorn"
if client.npcCalls == 2 {
name = "Mira Thorn, the Greencloak"
}
payload = map[string]any{"npcs": []any{map[string]any{"name": name, "source_refs": []any{map[string]int{"start_unit_id": client.npcCalls, "end_unit_id": client.npcCalls}}}}}
case npcnormalize.PromptID:
payload = map[string]any{"duplicate_groups": []any{map[string]any{"members": []string{"Mira Thorn", "Mira Thorn, the Greencloak"}, "canonical_name": "Mira Thorn"}}}
case interactionextract.PromptID:
payload = map[string]any{"interactions": []any{map[string]any{"name": "Mira Thorn", "kind": "dialogue", "source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}}}}}
default:
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected semantic pipeline prompt %q", request.PromptID)
}
content, err := json.Marshal(payload)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
if err := json.Unmarshal(content, out); err != nil {
return contracts.StructuredCompletionResponse{}, err
}
return contracts.StructuredCompletionResponse{Content: content}, nil
}
func (client *semanticNPCInteractionClient) requestCount(promptID string) int {
count := 0
for _, request := range client.requests {
if request.PromptID == promptID {
count++
}
}
return count
}
func (client *semanticNPCInteractionClient) requestFor(t *testing.T, promptID string) contracts.StructuredCompletionRequest {
t.Helper()
for _, request := range client.requests {
if request.PromptID == promptID {
return request
}
}
t.Fatalf("requests = %#v, missing %q", client.requests, promptID)
return contracts.StructuredCompletionRequest{}
}
var _ contracts.StructuredLLMClient = (*semanticNPCInteractionClient)(nil)

View File

@@ -14,6 +14,7 @@ import (
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
@@ -95,7 +96,7 @@ func TestRunnerProcessesSeriatimInputWithProductionDNDNPCPipeline(t *testing.T)
t.Fatalf("manifest lane = %#v, want NPC production composition", lane)
}
normalizerMetadata, ok := lane.Metadata["normalizer"].(map[string]any)
if !ok || normalizerMetadata["identity_policy"] != identity.Policy || normalizerMetadata["normalization_policy"] != "dnd.npcs.normalize.v2" {
if !ok || normalizerMetadata["identity_policy"] != identity.Policy || normalizerMetadata["normalization_policy"] != npcnormalize.NormalizationPolicy || normalizerMetadata["prompt_id"] != npcnormalize.PromptID || normalizerMetadata["response_schema_id"] != npcnormalize.ResponseSchemaID {
t.Fatalf("normalizer metadata = %#v, want identity and normalization policies", lane.Metadata)
}
var npcOutputFile *contracts.OutputFile
@@ -108,8 +109,8 @@ func TestRunnerProcessesSeriatimInputWithProductionDNDNPCPipeline(t *testing.T)
if npcOutputFile == nil || npcOutputFile.ContentType != npccodec.MediaType {
t.Fatalf("output files = %#v, want JSON NPC lane file", output.OutputFiles)
}
if len(client.requests) != 1 || client.requests[0].PromptID != npcs.PromptID {
t.Fatalf("LLM requests = %#v, want one NPC prompt request", client.requests)
if len(client.requests) != 2 || client.requests[0].PromptID != npcs.PromptID || client.requests[1].PromptID != npcnormalize.PromptID {
t.Fatalf("LLM requests = %#v, want extraction and normalization prompt requests", client.requests)
}
}
@@ -170,6 +171,52 @@ func TestProductionNPCPipelineRoutesSemanticCandidatesToDeterministicValidators(
}
}
func TestProductionNPCNormalizationRetryUsesFinalSafeProposal(t *testing.T) {
registries := productionNPCRegistries(t)
baseResponse := npcProductionResponse{NPCs: []npcProductionRecord{
{Name: "Mira", SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 1}}},
{Name: "Mira Thorn", SourceRefs: []npcProductionSourceRef{{StartUnitID: 2, EndUnitID: 2}}},
{Name: "Hooded Guard", SourceRefs: []npcProductionSourceRef{{StartUnitID: 3, EndUnitID: 3}}},
}}
partial := []byte(`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Hooded Guard","Unknown"],"canonical_name":"Hooded Guard"}]}`)
safe := []byte(`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"}]}`)
for _, test := range []struct {
name string
retries int
responses [][]byte
wantCalls int
wantExhaustion bool
}{
{name: "default retry budget", responses: [][]byte{partial}, wantCalls: 1, wantExhaustion: true},
{name: "later complete proposal", retries: 1, responses: [][]byte{partial, safe}, wantCalls: 2},
} {
t.Run(test.name, func(t *testing.T) {
cfg := loadNPCPipelineConfig(t)
profile := cfg.Pipelines["dnd-npcs-fixture"]
lane := profile.Artifacts["npcs"]
lane.Normalize.Retries = test.retries
profile.Artifacts["npcs"] = lane
cfg.Pipelines["dnd-npcs-fixture"] = profile
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-npcs-fixture", Catalog: moduleCatalog(registries)})
if err != nil {
t.Fatal(err)
}
client := &fakeNPCProductionLLMClient{response: baseResponse, normalizeResponses: test.responses}
output, err := runPreparedPipeline(t, registries, effective.ResolvedPipeline, client, pipeline.RunInput{RawInput: readNPCFixture(t)})
if err != nil {
t.Fatalf("Run() error = %v", err)
}
if client.requestCount(npcnormalize.PromptID) != test.wantCalls || len(output.NormalizeOutputs) != 1 {
t.Fatalf("normalization calls = %d output = %#v, want %d accepted result", client.requestCount(npcnormalize.PromptID), output.NormalizeOutputs, test.wantCalls)
}
if got := hasWarningReason(output.Warnings, npcnormalize.ReasonCodeNPCSemanticReconciliationExhausted); got != test.wantExhaustion {
t.Fatalf("warnings = %#v, exhaustion = %t, want %t", output.Warnings, got, test.wantExhaustion)
}
})
}
}
type npcProductionResponse struct {
NPCs []npcProductionRecord `json:"npcs"`
}
@@ -185,26 +232,41 @@ type npcProductionSourceRef struct {
}
type fakeNPCProductionLLMClient struct {
response npcProductionResponse
rawResponses [][]byte
requests []contracts.StructuredCompletionRequest
response npcProductionResponse
rawResponses [][]byte
normalizeResponses [][]byte
requests []contracts.StructuredCompletionRequest
}
func (client *fakeNPCProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
client.requests = append(client.requests, req)
var content []byte
if client.rawResponses != nil {
index := len(client.requests) - 1
if index >= len(client.rawResponses) {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("missing fake NPC response %d", index)
switch req.PromptID {
case npcs.PromptID:
if client.rawResponses != nil {
index := client.requestCount(npcs.PromptID) - 1
if index >= len(client.rawResponses) {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("missing fake NPC response %d", index)
}
content = append([]byte(nil), client.rawResponses[index]...)
} else {
var err error
content, err = json.Marshal(client.response)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
}
}
content = append([]byte(nil), client.rawResponses[index]...)
} else {
var err error
content, err = json.Marshal(client.response)
if err != nil {
return contracts.StructuredCompletionResponse{}, err
case npcnormalize.PromptID:
content = []byte(`{"duplicate_groups":[]}`)
if client.normalizeResponses != nil {
index := client.requestCount(npcnormalize.PromptID) - 1
if index >= len(client.normalizeResponses) {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("missing fake NPC normalization response %d", index)
}
content = append([]byte(nil), client.normalizeResponses[index]...)
}
default:
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected fake NPC prompt %q", req.PromptID)
}
if err := json.Unmarshal(content, out); err != nil {
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate NPC structured target: %w", err)
@@ -212,6 +274,16 @@ func (client *fakeNPCProductionLLMClient) CompleteStructured(_ context.Context,
return contracts.StructuredCompletionResponse{Content: content}, nil
}
func (client *fakeNPCProductionLLMClient) requestCount(promptID string) int {
count := 0
for _, request := range client.requests {
if request.PromptID == promptID {
count++
}
}
return count
}
func productionNPCRegistries(t *testing.T) pipeline.Registries {
t.Helper()
registries := pipeline.Registries{
@@ -283,3 +355,12 @@ func hasNPCWarning(warnings []contracts.Warning, reason string) bool {
}
return false
}
func hasWarningReason(warnings []contracts.Warning, reason string) bool {
for _, warning := range warnings {
if warning.ReasonCode == reason {
return true
}
}
return false
}