Compare commits
5 Commits
26bd59a5a2
...
fbb8e0d241
| Author | SHA1 | Date | |
|---|---|---|---|
| fbb8e0d241 | |||
| 8d9a496935 | |||
| d1c48db4bc | |||
| 6bd781d344 | |||
| 8a12c56971 |
@@ -61,9 +61,9 @@ No pipelines are built in. A run requires a configured pipeline.
|
||||
|
||||
If `scriptorium` is omitted, Notarius uses Scriptorium's built-in profile
|
||||
catalog. Prompt definitions may also name default profile IDs. The current D&D
|
||||
scene and spell prompts default to the built-in `mistral-small-3` profile when a
|
||||
module binding does not set `llm_profile`. That built-in profile reads its
|
||||
credential from `OPENROUTER_API_KEY`.
|
||||
scene, extraction, and NPC-normalization prompts default to `gemini-2-flash`
|
||||
when a module binding does not set `llm_profile`. The selected built-in profile
|
||||
reads its credentials from its Scriptorium profile definition.
|
||||
|
||||
## Scriptorium Profiles
|
||||
|
||||
@@ -322,7 +322,8 @@ Binding fields:
|
||||
- `llm_profile`: optional Scriptorium profile ID. Empty or omitted lets the
|
||||
Scriptorium prompt default select the profile.
|
||||
- `retries`: non-negative retry count for extra runtime attempts after the
|
||||
first attempt. Default: `0`. Supported on `chunk`, `extract`, `merge`, and
|
||||
first attempt. Default: `0`, which permits one initial attempt and no
|
||||
additional attempts. Supported on `chunk`, `extract`, `merge`, and
|
||||
`normalize` bindings.
|
||||
- `options`: optional module-specific settings.
|
||||
- `references`: optional reference bindings. Supported only for `chunk`,
|
||||
@@ -338,6 +339,15 @@ During resolution, each selected module's registered option validator runs.
|
||||
Production input, chunk, and output bindings reject unknown or invalid options
|
||||
with the affected binding context.
|
||||
|
||||
### NPC Semantic Normalization
|
||||
|
||||
The `dnd/npcs` normalizer combines deterministic canonical-name consolidation
|
||||
with a document-level LLM identity decision for eligible differently named NPC
|
||||
records. Its `llm_profile` and `retries` use the ordinary `normalize` binding
|
||||
fields above; `retries: 0` therefore still makes one normalization attempt.
|
||||
The normalizer declares no references and accepts no options. Context-window
|
||||
selection is current module policy, not configuration.
|
||||
|
||||
### JSON Output Options
|
||||
|
||||
The `json` output module accepts only `include_chunk_map`, a boolean that
|
||||
@@ -388,7 +398,7 @@ production validators do not call the LLM and must not set `llm_profile`.
|
||||
| merge | `appendorder` | Combines typed artifacts in chunk order. |
|
||||
| normalize | `noop` | Passes merged typed artifacts through unchanged. |
|
||||
| normalize | `dnd/spells` | Deterministically canonicalizes and de-duplicates typed D&D spell-list artifacts. |
|
||||
| normalize | `dnd/npcs` | Deterministically consolidates typed D&D NPC-list artifacts by canonical name and unions exact evidence. |
|
||||
| normalize | `dnd/npcs` | Deterministically prepares typed D&D NPC-list artifacts, then uses an LLM-assisted document-level identity decision to consolidate safe name-based matches and union exact evidence. |
|
||||
| normalize | `dnd/combat-turns` | Deterministically canonicalizes, orders, and de-duplicates typed D&D combat-turn artifacts. |
|
||||
| normalize | `dnd/item-events` | Deterministically trims, source-orders, and removes only exact D&D item-event duplicates. |
|
||||
| normalize | `dnd/npc-interactions` | Canonicalizes registry NPC names, orders interaction occurrences, and removes only exact duplicates. |
|
||||
|
||||
@@ -75,10 +75,17 @@ The production identities are:
|
||||
|
||||
The extractor maps private model records to the current source identity and
|
||||
assigns deterministic IDs. Extraction validation checks shape, source
|
||||
references, and source relatedness. The normalizer then consolidates records
|
||||
only when their normalized canonical names match, preserves the first record's
|
||||
display and output position, unions exact evidence, and validates the retained
|
||||
registry's identity. No LLM is used for consolidation.
|
||||
references, and source relatedness. The normalizer first consolidates equal
|
||||
canonical-name matches, then may make one document-level LLM-assisted identity
|
||||
decision per configured normalize attempt for eligible distinctly named
|
||||
records. Consolidation is name-based: it retains a supplied canonical display
|
||||
name, keeps the earliest affected output position, derives its ID again, and
|
||||
unions exact evidence in canonical order. Ambiguous, invalid, or conflicting
|
||||
proposals are not applied; independently safe matches may still be retained.
|
||||
After the retry budget is exhausted, the safe result is accepted with bounded
|
||||
normalization warnings and the usual validation. The durable v1 artifact shape
|
||||
does not add aliases, proposal fields, or any other semantic-normalization
|
||||
representation.
|
||||
|
||||
The extraction prompt asks only for individually identifiable NPC names backed
|
||||
by source evidence. Groups, generic roles, invented labels, and descriptive or
|
||||
|
||||
@@ -154,6 +154,22 @@ identity, campaign-reference, and instruction messages are ephemeral cache
|
||||
boundaries; the transcript is last and has no cache control. Compatible shared
|
||||
messages remain canonical shared assets rather than copied package text.
|
||||
|
||||
### D&D NPC Normalization Prompt Ordering And Cache Boundaries
|
||||
|
||||
NPC normalization has a distinct prompt and response-schema identity from NPC
|
||||
extraction. Its stable message tiers are the common D&D system and identity
|
||||
assets, followed by package-owned task and normalization instructions. Cache
|
||||
boundaries follow the shared identity tier and the package instructions. The
|
||||
variable tail contains the private candidate-name-and-range input and a
|
||||
windowed transcript input whose cited units provide local context; neither has
|
||||
a cache boundary because it changes with the document.
|
||||
|
||||
This prompt intentionally omits extraction-evidence and campaign-reference
|
||||
assets: it reconciles existing records rather than extracting events or adding
|
||||
evidence. Its package-owned manifest and schema identity are fingerprinted
|
||||
separately, so a normalization prompt or schema change cannot reuse a prior
|
||||
normalization checkpoint.
|
||||
|
||||
Shared wording belongs in the canonical assets under
|
||||
`internal/modules/dnd/shared`; extraction packages reference those assets in
|
||||
their manifests instead of copying similar text into package-local files.
|
||||
|
||||
@@ -395,11 +395,27 @@ omission-summary warning when truncated.
|
||||
|
||||
### `internal/modules/dnd/normalize/npcs`
|
||||
|
||||
The NPC normalizer performs deterministic identity-aware consolidation in
|
||||
merged input order. It consolidates only equal canonical-name comparison keys,
|
||||
retains the first display record, and unions exact source references. It exposes
|
||||
the identity policy as its local checkpoint fingerprint and emits bounded
|
||||
normalization warnings.
|
||||
The NPC normalizer deterministically trims display names, recomputes IDs,
|
||||
canonicalizes evidence, and consolidates equal comparison keys before semantic
|
||||
work. Records are eligible for the document-level identity call only when they
|
||||
have a non-empty comparison key and wholly valid current-document references.
|
||||
It sends private candidate names and source ranges plus coalesced, cited
|
||||
transcript windows to its own prompt; stable NPC IDs and the durable artifact
|
||||
shape are not prompt inputs.
|
||||
|
||||
The private structured response proposes groups of supplied names and a
|
||||
canonical supplied name. Deterministic comparison-key resolution validates each
|
||||
group, discards unsafe or overlapping groups, and independently applies safe
|
||||
ones. Application preserves earliest record order, unions canonical evidence,
|
||||
and derives the final canonical ID. Invalid structured output and discarded
|
||||
groups request framework retry with a safe fallback; bounded diagnostics become
|
||||
durable only on final fallback exhaustion.
|
||||
|
||||
The normalizer records prompt and response-schema identities and digests,
|
||||
identity and normalization policies, and semantic-context policy and radius as
|
||||
manifest metadata. Its local checkpoint fingerprints cover the prompt, response
|
||||
schema, identity policy, normalization policy, and semantic-context policy so a
|
||||
meaningful behavior change invalidates prior normalize reuse.
|
||||
|
||||
## Merger And Normalizer
|
||||
|
||||
|
||||
@@ -119,7 +119,7 @@ Configuration. The implemented module packages are:
|
||||
| `internal/modules/generic/merge/appendorder` | Combines accepted extraction results in chunk order. |
|
||||
| `internal/modules/generic/normalize/noop` | Preserves accepted merged output. |
|
||||
| `internal/modules/dnd/normalize/spells` | Canonicalizes catalog-backed spell names and exact source references, conservatively collapses duplicate casts, and reports deterministic warnings and independently scoped catalog checkpoint identity. |
|
||||
| `internal/modules/dnd/normalize/npcs` | Consolidates NPC records deterministically by canonical name, unions exact evidence, and reports bounded warnings. |
|
||||
| `internal/modules/dnd/normalize/npcs` | Deterministically prepares and safely applies document-level LLM-assisted NPC identity consolidation, preserving canonical evidence, order, and diagnostics. |
|
||||
| `internal/modules/generic/output/json` | Encodes manifests, lane payloads, warnings, rejections, and an explicitly enabled accepted chunk map as logical JSON files. |
|
||||
|
||||
`internal/modules/dnd/shared` owns reusable D&D prompt fragments,
|
||||
|
||||
@@ -322,9 +322,24 @@ error when attempts are exhausted. A rejection becomes a recorded
|
||||
`RejectedOutput` when attempts are exhausted. Cancellation stops retry
|
||||
processing immediately.
|
||||
|
||||
Rejected output is a non-fatal pipeline outcome and does not advance. Warnings
|
||||
from discarded attempts are not promoted. Configuration owns retry counts and
|
||||
validator overrides; see [Module Bindings](../config.md#module-bindings).
|
||||
Structured-completion adapters classify malformed or undecodable provider
|
||||
output with the provider-neutral `contracts.ErrInvalidStructuredOutput` error.
|
||||
A typed normalizer may turn that condition, or another unsafe proposal, into a
|
||||
normalize retry directive with a deterministic safe candidate, stable
|
||||
diagnostic, and fallback warnings. The directive consumes the same configured
|
||||
normalize retry budget: `retries` permits that many additional attempts after
|
||||
the initial attempt. It neither creates a normalizer-local retry loop nor
|
||||
records an accepted checkpoint for the discarded attempt.
|
||||
|
||||
When a later normalize attempt succeeds, its candidate alone proceeds through
|
||||
the usual validation and checkpoint path. When the final attempt still returns
|
||||
a directive, the runner validates its supplied safe fallback through that same
|
||||
normalizer validator chain before accepting or rejecting it. Ordinary
|
||||
attempt-local warnings and fallback warnings remain unpromoted while another
|
||||
attempt is available; only final exhaustion promotes the supplied fallback
|
||||
warnings. Rejected output is a non-fatal pipeline outcome and does not advance.
|
||||
Configuration owns retry counts and validator overrides; see
|
||||
[Module Bindings](../config.md#module-bindings).
|
||||
|
||||
## Checkpoint And Debug Hooks
|
||||
|
||||
@@ -389,7 +404,9 @@ boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
||||
one terminal envelope for acceptance, validator rejection, module or validator
|
||||
error, or applicable candidate or final serialization error. The envelope
|
||||
contains its attempt-local warnings, any available candidate and rejection,
|
||||
and terminal error text; failures before a candidate exists omit that payload.
|
||||
and terminal error text; normalize retry directives retain their attempt-local
|
||||
candidate and diagnostic, while only the final safe fallback reaches validation.
|
||||
Failures before a candidate exists omit that payload.
|
||||
Only LLM calls made by the module operation belong to the module attempt.
|
||||
Validator calls retain independent scopes under `validate/` and are not
|
||||
duplicated into the module envelope. A failed terminal-envelope write is a
|
||||
|
||||
@@ -86,6 +86,22 @@ workflow intentionally crosses a process or session boundary. Those files are
|
||||
validated against the consumer slot and must be protected as sensitive
|
||||
campaign data. They are not part of the maintained ordered handoff workflow.
|
||||
|
||||
### NPC Semantic Normalization
|
||||
|
||||
Before the first step can release its accepted NPC artifact across the ordered
|
||||
generated-reference barrier, `dnd/npcs` performs one document-level semantic
|
||||
normalization call for each configured normalize attempt when eligible distinct
|
||||
names remain. The normalize binding's `retries` setting controls additional
|
||||
attempts. If an invalid or unsafe identity proposal exhausts that budget, the
|
||||
run safely accepts the deterministic and any independently safe partial
|
||||
consolidation, with a bounded warning; ordinary validation still applies before
|
||||
the artifact can cross the barrier.
|
||||
|
||||
An accepted normalized NPC checkpoint can be reused on `--resume` just like
|
||||
other accepted normalize work. A changed normalization prompt, response schema,
|
||||
or policy identity produces a cold cache miss, so the current reconciliation is
|
||||
recomputed rather than silently reusing incompatible state.
|
||||
|
||||
## Chunk-Plan Cache
|
||||
|
||||
Chunk plans are stored at:
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Status
|
||||
|
||||
Accepted scope; not implemented.
|
||||
Implemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
## Status
|
||||
|
||||
Ready for implementation.
|
||||
Completed.
|
||||
|
||||
## Objective
|
||||
|
||||
|
||||
@@ -43,7 +43,10 @@ pipelines:
|
||||
module: dnd/npcs
|
||||
retries: 2
|
||||
merge: appendorder
|
||||
normalize: dnd/npcs
|
||||
normalize:
|
||||
module: dnd/npcs
|
||||
llm_profile: gemini-2-flash
|
||||
retries: 2
|
||||
scene-descriptions:
|
||||
extract:
|
||||
module: dnd/scene-descriptions
|
||||
|
||||
7
internal/framework/contracts/errors.go
Normal file
7
internal/framework/contracts/errors.go
Normal file
@@ -0,0 +1,7 @@
|
||||
package contracts
|
||||
|
||||
import "errors"
|
||||
|
||||
// ErrInvalidStructuredOutput identifies a provider response that cannot satisfy
|
||||
// the caller's declared structured-output contract.
|
||||
var ErrInvalidStructuredOutput = errors.New("invalid structured output")
|
||||
@@ -90,6 +90,15 @@ type TypedNormalizeRequest[T any] struct {
|
||||
type TypedNormalizeResult[T any] struct {
|
||||
Value T
|
||||
Warnings []Warning
|
||||
Retry *NormalizeRetry
|
||||
}
|
||||
|
||||
// NormalizeRetry asks the framework to retry normalization while retaining a
|
||||
// safe candidate for acceptance if the retry budget is exhausted.
|
||||
type NormalizeRetry struct {
|
||||
ReasonCode string
|
||||
Message string
|
||||
FallbackWarnings []Warning
|
||||
}
|
||||
|
||||
type Normalizer[T any] interface {
|
||||
|
||||
@@ -113,17 +113,17 @@ func (c *ScriptoriumClient) CompleteStructured(ctx context.Context, req contract
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w", promptID, redactScriptoriumError(err))
|
||||
}
|
||||
if result == nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: empty result", promptID)
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("run Scriptorium prompt %q: %w: empty result", promptID, contracts.ErrInvalidStructuredOutput)
|
||||
}
|
||||
response := c.responseFromResult(result, prepared)
|
||||
if result.Validation.Status == scriptorium.ValidationFailed || !result.Validation.IsValid {
|
||||
return response, fmt.Errorf("run Scriptorium prompt %q: validation failed: %s", promptID, strings.Join(result.Validation.Errors, "; "))
|
||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: validation failed: %s", promptID, contracts.ErrInvalidStructuredOutput, strings.Join(result.Validation.Errors, "; "))
|
||||
}
|
||||
if len(strings.TrimSpace(string(response.Content))) == 0 {
|
||||
return response, fmt.Errorf("run Scriptorium prompt %q: empty structured output", promptID)
|
||||
return response, fmt.Errorf("run Scriptorium prompt %q: %w: empty structured output", promptID, contracts.ErrInvalidStructuredOutput)
|
||||
}
|
||||
if err := json.Unmarshal(response.Content, out); err != nil {
|
||||
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w", promptID, err)
|
||||
return response, fmt.Errorf("decode Scriptorium structured output for prompt %q: %w: %w", promptID, contracts.ErrInvalidStructuredOutput, err)
|
||||
}
|
||||
return response, nil
|
||||
}
|
||||
|
||||
@@ -113,7 +113,7 @@ func TestScriptoriumClientValidationFailureReturnsError(t *testing.T) {
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
if err == nil || !strings.Contains(err.Error(), "validation failed") {
|
||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "validation failed") {
|
||||
t.Fatalf("CompleteStructured() error = %v, want validation failure", err)
|
||||
}
|
||||
if got := string(resp.Content); got != `{"bad":true}` {
|
||||
@@ -138,7 +138,7 @@ func TestScriptoriumClientDecodeFailureReturnsRawResponse(t *testing.T) {
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
if err == nil || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
|
||||
if err == nil || !errors.Is(err, contracts.ErrInvalidStructuredOutput) || !strings.Contains(err.Error(), "decode Scriptorium structured output") {
|
||||
t.Fatalf("CompleteStructured() error = %v, want decode failure", err)
|
||||
}
|
||||
if got := string(resp.Content); got != `{"ok":true}` {
|
||||
@@ -163,6 +163,9 @@ func TestScriptoriumClientProviderFailureIncludesContextAndRedactsBearerToken(t
|
||||
if err == nil {
|
||||
t.Fatalf("CompleteStructured() error = nil, want provider error")
|
||||
}
|
||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
t.Fatalf("provider error = %v, must not be classified as invalid structured output", err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), `run Scriptorium prompt "adapter.test"`) {
|
||||
t.Fatalf("error = %q, want operation context", err.Error())
|
||||
}
|
||||
@@ -186,11 +189,27 @@ func TestScriptoriumClientContextCancellationIsRespected(t *testing.T) {
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
if !errors.Is(err, context.Canceled) {
|
||||
if !errors.Is(err, context.Canceled) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
t.Fatalf("CompleteStructured() error = %v, want context canceled", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScriptoriumClientClassifiesEmptyStructuredCompletion(t *testing.T) {
|
||||
client := newTestScriptoriumClient(t, &fakeScriptoriumLLM{allowEmpty: true})
|
||||
|
||||
var out map[string]any
|
||||
_, err := client.CompleteStructured(context.Background(), contracts.StructuredCompletionRequest{
|
||||
PromptID: "adapter.test",
|
||||
SessionID: "session-123",
|
||||
Inputs: contracts.LLMInputSet{
|
||||
"transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""),
|
||||
},
|
||||
}, &out)
|
||||
if !errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
t.Fatalf("CompleteStructured() error = %v, want invalid structured output", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestScheduledScriptoriumClientBoundsConcurrentCalls(t *testing.T) {
|
||||
fake := &fakeScriptoriumLLM{
|
||||
content: `{"ok":true}`,
|
||||
@@ -296,6 +315,7 @@ output:
|
||||
|
||||
type fakeScriptoriumLLM struct {
|
||||
content string
|
||||
allowEmpty bool
|
||||
err error
|
||||
block chan struct{}
|
||||
mu sync.Mutex
|
||||
@@ -329,10 +349,10 @@ func (f *fakeScriptoriumLLM) Generate(ctx context.Context, req scriptorium.Gener
|
||||
return nil, f.err
|
||||
}
|
||||
content := f.content
|
||||
if content == "" {
|
||||
if content == "" && !f.allowEmpty {
|
||||
content = `{"ok":true}`
|
||||
}
|
||||
if !json.Valid([]byte(content)) {
|
||||
if !f.allowEmpty && !json.Valid([]byte(content)) {
|
||||
return nil, errors.New("test fake must return JSON content")
|
||||
}
|
||||
return &scriptorium.GenerateResponse{
|
||||
|
||||
@@ -80,12 +80,23 @@ func RegisterNormalizerBuilder[T any](registry *NormalizerRegistry, spec ModuleS
|
||||
if err != nil {
|
||||
return erasedTypedResult{}, err
|
||||
}
|
||||
return erasedTypedResult{Value: result.Value, Warnings: result.Warnings}, nil
|
||||
return erasedTypedResult{Value: result.Value, Warnings: cloneWarnings(result.Warnings), Retry: cloneNormalizeRetry(result.Retry)}, nil
|
||||
},
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func cloneNormalizeRetry(retry *contracts.NormalizeRetry) *contracts.NormalizeRetry {
|
||||
if retry == nil {
|
||||
return nil
|
||||
}
|
||||
return &contracts.NormalizeRetry{
|
||||
ReasonCode: retry.ReasonCode,
|
||||
Message: retry.Message,
|
||||
FallbackWarnings: cloneWarnings(retry.FallbackWarnings),
|
||||
}
|
||||
}
|
||||
|
||||
func (r *NormalizerRegistry) validateOptions(key string, kind contracts.ArtifactKind, options map[string]any) error {
|
||||
if r == nil {
|
||||
return fmt.Errorf("normalizer registry must not be nil")
|
||||
|
||||
52
internal/framework/pipeline/normalizer_registry_test.go
Normal file
52
internal/framework/pipeline/normalizer_registry_test.go
Normal file
@@ -0,0 +1,52 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
type retryingNotesNormalizer struct {
|
||||
warnings []contracts.Warning
|
||||
retry *contracts.NormalizeRetry
|
||||
}
|
||||
|
||||
func (retryingNotesNormalizer) Key() string { return "test/retry-normalize" }
|
||||
func (retryingNotesNormalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
func (n retryingNotesNormalizer) Normalize(_ context.Context, req contracts.TypedNormalizeRequest[codecNotes]) (contracts.TypedNormalizeResult[codecNotes], error) {
|
||||
return contracts.TypedNormalizeResult[codecNotes]{Value: req.MergeOutput.Value, Warnings: n.warnings, Retry: n.retry}, nil
|
||||
}
|
||||
|
||||
func TestNormalizerRegistryErasureClonesRetryDirective(t *testing.T) {
|
||||
warnings := []contracts.Warning{{Scope: "attempt", ReasonCode: "ordinary", Message: "ordinary warning"}}
|
||||
retry := &contracts.NormalizeRetry{
|
||||
ReasonCode: "retryable",
|
||||
Message: "safe fallback available",
|
||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "omitted", Message: "fallback warning"}},
|
||||
}
|
||||
registry := NewNormalizerRegistry()
|
||||
if err := RegisterNormalizer(registry, ModuleSpec{Key: "test/retry-normalize", Stage: StageNormalize, ArtifactKind: "test/notes"}, func() (contracts.Normalizer[codecNotes], error) {
|
||||
return retryingNotesNormalizer{warnings: warnings, retry: retry}, nil
|
||||
}); err != nil {
|
||||
t.Fatalf("RegisterNormalizer() error = %v", err)
|
||||
}
|
||||
entry, ok := registry.typedEntry("test/retry-normalize", "test/notes")
|
||||
if !ok {
|
||||
t.Fatal("typed normalizer entry missing")
|
||||
}
|
||||
implementation, err := entry.builder(BuildRequest{})
|
||||
if err != nil {
|
||||
t.Fatalf("builder() error = %v", err)
|
||||
}
|
||||
result, err := entry.normalize(context.Background(), implementation, contracts.TypedNormalizeRequest[any]{MergeOutput: contracts.MergeArtifact[any]{Value: codecNotes{Items: []string{"safe"}}}})
|
||||
if err != nil {
|
||||
t.Fatalf("normalize() error = %v", err)
|
||||
}
|
||||
warnings[0].Message = "mutated"
|
||||
retry.Message = "mutated"
|
||||
retry.FallbackWarnings[0].Message = "mutated"
|
||||
if result.Retry == nil || result.Warnings[0].Message != "ordinary warning" || result.Retry.Message != "safe fallback available" || result.Retry.FallbackWarnings[0].Message != "fallback warning" {
|
||||
t.Fatalf("erased retry result = %#v, want independent warning data", result)
|
||||
}
|
||||
}
|
||||
174
internal/framework/pipeline/runner_normalize_retry_test.go
Normal file
174
internal/framework/pipeline/runner_normalize_retry_test.go
Normal file
@@ -0,0 +1,174 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
)
|
||||
|
||||
func TestRunnerHandlesRetryableNormalizeFallbacks(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
retries int
|
||||
operation func(int) erasedTypedResult
|
||||
validator *preparedValidator
|
||||
wantCalls int
|
||||
wantItem string
|
||||
wantWarnings []string
|
||||
wantRejected int
|
||||
wantDebug []string
|
||||
wantCheckpoint int
|
||||
}{
|
||||
{
|
||||
name: "accepts zero-retry fallback",
|
||||
retries: 0,
|
||||
operation: func(int) erasedTypedResult {
|
||||
return retryableNormalizeResult("fallback", "ordinary", "fallback-warning")
|
||||
},
|
||||
wantCalls: 1,
|
||||
wantItem: "fallback",
|
||||
wantWarnings: []string{"ordinary", "fallback-warning"},
|
||||
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
|
||||
wantCheckpoint: 1,
|
||||
},
|
||||
{
|
||||
name: "retries before accepting ordinary result",
|
||||
retries: 1,
|
||||
operation: func(attempt int) erasedTypedResult {
|
||||
if attempt == 1 {
|
||||
return retryableNormalizeResult("discarded", "discarded-ordinary", "discarded-fallback")
|
||||
}
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"accepted"}}, Warnings: []contracts.Warning{{Scope: "accepted", ReasonCode: "ordinary", Message: "accepted-warning"}}}
|
||||
},
|
||||
wantCalls: 2,
|
||||
wantItem: "accepted",
|
||||
wantWarnings: []string{"accepted-warning"},
|
||||
wantDebug: []string{`"another_attempt":true`, `"fallback_accepted":false`},
|
||||
wantCheckpoint: 1,
|
||||
},
|
||||
{
|
||||
name: "accepts final fallback after exhaustion",
|
||||
retries: 1,
|
||||
operation: func(attempt int) erasedTypedResult {
|
||||
return retryableNormalizeResult(fmt.Sprintf("fallback-%d", attempt), fmt.Sprintf("ordinary-%d", attempt), fmt.Sprintf("fallback-warning-%d", attempt))
|
||||
},
|
||||
wantCalls: 2,
|
||||
wantItem: "fallback-2",
|
||||
wantWarnings: []string{"ordinary-2", "fallback-warning-2"},
|
||||
wantDebug: []string{`"another_attempt":false`, `"fallback_accepted":true`},
|
||||
wantCheckpoint: 1,
|
||||
},
|
||||
{
|
||||
name: "keeps final fallback rejection terminal",
|
||||
retries: 1,
|
||||
operation: func(int) erasedTypedResult {
|
||||
return retryableNormalizeResult("rejected", "ordinary", "fallback-warning")
|
||||
},
|
||||
validator: &preparedValidator{
|
||||
resolved: ResolvedValidator{Binding: Binding("reject-final-fallback"), Target: ValidatorTargetTyped, ArtifactKind: "test/notes"},
|
||||
typedValidate: func(context.Context, any, typedValidationTarget) (contracts.ValidationResult, error) {
|
||||
return contracts.ValidationResult{Approved: false, ReasonCode: "rejected", Message: "reject fallback"}, nil
|
||||
},
|
||||
},
|
||||
wantCalls: 2,
|
||||
wantRejected: 1,
|
||||
wantDebug: []string{`"fallback_accepted":true`, `"rejection"`},
|
||||
wantCheckpoint: 0,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
lane := &prepared.Steps[0].lanes[0]
|
||||
lane.resolved.Normalize.Retries = tc.retries
|
||||
if tc.validator != nil {
|
||||
lane.normalizeValidators.validators = []preparedValidator{*tc.validator}
|
||||
}
|
||||
calls := 0
|
||||
lane.typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
calls++
|
||||
return tc.operation(calls), nil
|
||||
}
|
||||
debug := newCapturedDebugRecorder()
|
||||
checkpoints := &candidateCheckpointRecorder{CheckpointRecorder: NoopCheckpointRecorder()}
|
||||
output, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: debug, Checkpoints: checkpoints})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if calls != tc.wantCalls {
|
||||
t.Fatalf("normalize calls = %d, want %d", calls, tc.wantCalls)
|
||||
}
|
||||
if checkpoints.normalizeSucceeded != tc.wantCheckpoint {
|
||||
t.Fatalf("normalize checkpoints = %d, want %d", checkpoints.normalizeSucceeded, tc.wantCheckpoint)
|
||||
}
|
||||
if len(output.Rejected) != tc.wantRejected {
|
||||
t.Fatalf("rejected outputs = %#v, want %d", output.Rejected, tc.wantRejected)
|
||||
}
|
||||
var retryDebug strings.Builder
|
||||
for _, name := range debug.names() {
|
||||
if strings.HasPrefix(name, "normalize/notes/attempt-") && strings.HasSuffix(name, ".json") {
|
||||
retryDebug.Write(debug.json[name])
|
||||
}
|
||||
}
|
||||
for _, fragment := range tc.wantDebug {
|
||||
if !strings.Contains(retryDebug.String(), fragment) {
|
||||
t.Fatalf("retry debug = %s, want %q", retryDebug.String(), fragment)
|
||||
}
|
||||
}
|
||||
if tc.wantRejected != 0 {
|
||||
if output.Rejected[0].AttemptCount != tc.wantCalls {
|
||||
t.Fatalf("rejection attempt count = %d, want %d", output.Rejected[0].AttemptCount, tc.wantCalls)
|
||||
}
|
||||
return
|
||||
}
|
||||
if len(output.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("normalize outputs = %#v, want one", output.NormalizeOutputs)
|
||||
}
|
||||
decoded, err := lane.typed.codec.decode(output.NormalizeOutputs[0].Artifact.Content)
|
||||
if err != nil {
|
||||
t.Fatalf("decode normalized output: %v", err)
|
||||
}
|
||||
normalized, ok := decoded.(codecNotes)
|
||||
if !ok {
|
||||
t.Fatalf("decoded normalized output = %T, want codecNotes", decoded)
|
||||
}
|
||||
if got := firstNote(normalized); got != tc.wantItem {
|
||||
t.Fatalf("normalized item = %q, want %q", got, tc.wantItem)
|
||||
}
|
||||
gotWarnings := make([]string, len(output.Warnings))
|
||||
for index, warning := range output.Warnings {
|
||||
gotWarnings[index] = warning.Message
|
||||
}
|
||||
if strings.Join(gotWarnings, "|") != strings.Join(tc.wantWarnings, "|") {
|
||||
t.Fatalf("durable warnings = %#v, want %#v", gotWarnings, tc.wantWarnings)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunnerRejectsBlankNormalizeRetryDiagnostic(t *testing.T) {
|
||||
prepared := preparedAttemptDebugPipeline(t)
|
||||
prepared.Steps[0].lanes[0].typed.normalize = func(context.Context, any, contracts.TypedNormalizeRequest[any]) (erasedTypedResult, error) {
|
||||
return erasedTypedResult{Value: codecNotes{Items: []string{"safe"}}, Retry: &contracts.NormalizeRetry{ReasonCode: " ", Message: "missing reason"}}, nil
|
||||
}
|
||||
_, err := New().Run(context.Background(), RunInput{Prepared: prepared, RawInput: []byte("input"), Debug: newCapturedDebugRecorder()})
|
||||
if err == nil || !strings.Contains(err.Error(), "blank reason code or message") {
|
||||
t.Fatalf("Run() error = %v, want retry diagnostic contract error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func retryableNormalizeResult(item, ordinary, fallback string) erasedTypedResult {
|
||||
return erasedTypedResult{
|
||||
Value: codecNotes{Items: []string{item}},
|
||||
Warnings: []contracts.Warning{{Scope: "attempt", ReasonCode: "ordinary", Message: ordinary}},
|
||||
Retry: &contracts.NormalizeRetry{
|
||||
ReasonCode: "retryable_normalization",
|
||||
Message: "safe fallback is available",
|
||||
FallbackWarnings: []contracts.Warning{{Scope: "fallback", ReasonCode: "fallback", Message: fallback}},
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -8,6 +8,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"path"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
@@ -362,9 +363,30 @@ func (r *Runner) runNormalizeStage(ctx context.Context, input RunInput, checkpoi
|
||||
attemptErr := fmt.Errorf("serialize normalize candidate for lane %q: %w", lane.ID, encodeErr)
|
||||
return false, nil, terminal.record(map[string]any{"warnings": debugWarningEnvelopes(attemptWarnings)}, attemptErr)
|
||||
}
|
||||
var retryPayload map[string]any
|
||||
if result.Retry != nil {
|
||||
if strings.TrimSpace(result.Retry.ReasonCode) == "" || strings.TrimSpace(result.Retry.Message) == "" {
|
||||
return false, nil, terminal.record(map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings)}, fmt.Errorf("normalize lane %q returned retry directive with blank reason code or message", lane.ID))
|
||||
}
|
||||
retryRemaining := attempt <= lane.Normalize.Retries
|
||||
retryPayload = map[string]any{
|
||||
"reason_code": result.Retry.ReasonCode,
|
||||
"message": result.Retry.Message,
|
||||
"another_attempt": retryRemaining,
|
||||
"fallback_accepted": !retryRemaining,
|
||||
}
|
||||
if retryRemaining {
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "retry": retryPayload}
|
||||
return false, nil, terminal.record(payload, nil)
|
||||
}
|
||||
attemptWarnings = append(attemptWarnings, cloneWarnings(result.Retry.FallbackWarnings)...)
|
||||
}
|
||||
warnings, rejected, validateErr := r.validateTypedArtifact(attemptCtx, typed.codec, typedValidationTarget{stage: StageNormalize, stepID: input.stepID, laneID: lane.ID, moduleKey: lane.Normalize.Module, source: doc, sourceID: doc.ID, sourceInput: sourceInput.Clone(), sessionID: sessionID, references: normalizeReferences, metadata: input.Metadata, value: result.Value, candidate: &serializedCandidate}, prepared.normalizeValidators, attempt, input.Debug)
|
||||
attemptWarnings = append(attemptWarnings, warnings...)
|
||||
payload := map[string]any{"output": debugCheckpointArtifact(serializedCandidate), "warnings": debugWarningEnvelopes(attemptWarnings), "rejection": debugRejectedOutputPtr(rejected)}
|
||||
if retryPayload != nil {
|
||||
payload["retry"] = retryPayload
|
||||
}
|
||||
if validateErr != nil || rejected != nil {
|
||||
return false, rejected, terminal.record(payload, validateErr)
|
||||
}
|
||||
|
||||
@@ -24,6 +24,7 @@ type erasedMergeArtifact struct {
|
||||
type erasedTypedResult struct {
|
||||
Value any
|
||||
Warnings []contracts.Warning
|
||||
Retry *contracts.NormalizeRetry
|
||||
}
|
||||
|
||||
type typedValidationTarget struct {
|
||||
|
||||
6
internal/modules/dnd/normalize/npcs/assets.go
Normal file
6
internal/modules/dnd/normalize/npcs/assets.go
Normal file
@@ -0,0 +1,6 @@
|
||||
package npcs
|
||||
|
||||
import "embed"
|
||||
|
||||
//go:embed assets/schemas/dnd_npcs_normalize_llm.v1.json assets/prompts/*.yaml assets/prompts/*.md
|
||||
var embeddedAssets embed.FS
|
||||
@@ -0,0 +1,4 @@
|
||||
The supplied NPC candidates are below. Use only these display names in the
|
||||
response.
|
||||
|
||||
{{ input "candidates" }}
|
||||
@@ -0,0 +1,32 @@
|
||||
id: dnd.npcs.normalize
|
||||
version: "v1"
|
||||
default_profile: gemini-2-flash
|
||||
inputs:
|
||||
- name: candidates
|
||||
required: true
|
||||
content_type: application/json
|
||||
- name: transcript
|
||||
required: true
|
||||
content_type: application/json
|
||||
messages:
|
||||
- role: system
|
||||
content_file: ./sharedassets/common-dnd-system.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-identity.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./task.md
|
||||
- role: user
|
||||
content_file: ./instructions.md
|
||||
cache_control:
|
||||
type: ephemeral
|
||||
- role: user
|
||||
content_file: ./candidates.md
|
||||
- role: user
|
||||
content_file: ./sharedassets/common-dnd-transcript.md
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: dnd_npcs_normalize_llm.v1.json
|
||||
repair_attempts: 0
|
||||
@@ -0,0 +1,10 @@
|
||||
Return duplicate groups only when the transcript context clearly establishes a
|
||||
single individual. Prefer no group when identity is ambiguous.
|
||||
|
||||
Copy supplied display names into each group's members. Choose canonical_name
|
||||
from that same group's supplied members. Prefer a complete stable proper name
|
||||
over an abbreviation, but prefer an unadorned proper name over that name plus a
|
||||
contextual class, role, title, or relationship descriptor unless the descriptor
|
||||
is established as part of the name.
|
||||
|
||||
Do not invent names, source references, replacement records, or explanations.
|
||||
@@ -0,0 +1,2 @@
|
||||
Identify only supplied NPC display names that clearly refer to the same
|
||||
individual in the supplied transcript context.
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "notarius.dnd.npcs.normalize.llm",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["duplicate_groups"],
|
||||
"properties": {
|
||||
"duplicate_groups": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["members", "canonical_name"],
|
||||
"properties": {
|
||||
"members": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
},
|
||||
"canonical_name": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
214
internal/modules/dnd/normalize/npcs/context_material.go
Normal file
214
internal/modules/dnd/normalize/npcs/context_material.go
Normal file
@@ -0,0 +1,214 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"sort"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
)
|
||||
|
||||
const semanticContextRadius = 2
|
||||
|
||||
type normalizeContextMaterials struct {
|
||||
Candidates contracts.LLMInputMaterial
|
||||
Transcript contracts.LLMInputMaterial
|
||||
candidatePositions []int
|
||||
}
|
||||
|
||||
type normalizeCandidateInput struct {
|
||||
NPCs []normalizeCandidate `json:"npcs"`
|
||||
}
|
||||
|
||||
type normalizeCandidate struct {
|
||||
Name string `json:"name"`
|
||||
SourceRefs []normalizeCandidateSourceRef `json:"source_refs"`
|
||||
}
|
||||
|
||||
type normalizeCandidateSourceRef struct {
|
||||
StartUnitID int `json:"start_unit_id"`
|
||||
EndUnitID int `json:"end_unit_id"`
|
||||
}
|
||||
|
||||
type normalizeTranscriptInput struct {
|
||||
Windows []normalizeTranscriptWindow `json:"windows"`
|
||||
}
|
||||
|
||||
type normalizeTranscriptWindow struct {
|
||||
Units []normalizeTranscriptUnit `json:"units"`
|
||||
}
|
||||
|
||||
type normalizeTranscriptUnit struct {
|
||||
ID int `json:"id"`
|
||||
Kind string `json:"kind"`
|
||||
Text string `json:"text"`
|
||||
Metadata map[string]any `json:"metadata,omitempty"`
|
||||
Cited bool `json:"cited"`
|
||||
}
|
||||
|
||||
type normalizeProposalResponse struct {
|
||||
DuplicateGroups []normalizeProposalGroup `json:"duplicate_groups"`
|
||||
}
|
||||
|
||||
type normalizeProposalGroup struct {
|
||||
Members []string `json:"members"`
|
||||
CanonicalName string `json:"canonical_name"`
|
||||
}
|
||||
|
||||
type sourceInterval struct {
|
||||
start int
|
||||
end int
|
||||
}
|
||||
|
||||
func buildDefaultNormalizeContextMaterials(doc *source.SourceDocument, records []dnd.NPC) (normalizeContextMaterials, bool, error) {
|
||||
return buildNormalizeContextMaterials(doc, records, semanticContextRadius)
|
||||
}
|
||||
|
||||
// buildNormalizeContextMaterials prepares the owned prompt inputs for a
|
||||
// document-level normalization proposal. A false ready value means semantic
|
||||
// normalization has no comparison-distinct eligible candidates to consider.
|
||||
func buildNormalizeContextMaterials(doc *source.SourceDocument, records []dnd.NPC, radius int) (materials normalizeContextMaterials, ready bool, err error) {
|
||||
if doc == nil {
|
||||
return normalizeContextMaterials{}, false, nil
|
||||
}
|
||||
if radius < 0 {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: radius must not be negative")
|
||||
}
|
||||
|
||||
index := source.NewDocumentIndex(doc)
|
||||
candidates := make([]normalizeCandidate, 0, len(records))
|
||||
candidatePositions := make([]int, 0, len(records))
|
||||
intervals := make([]sourceInterval, 0)
|
||||
cited := make([]bool, len(doc.Units))
|
||||
seenKeys := make(map[string]struct{}, len(records))
|
||||
|
||||
for position, record := range records {
|
||||
key := identity.ComparisonKey(record.Name)
|
||||
if key == "" || len(record.SourceRefs) == 0 {
|
||||
continue
|
||||
}
|
||||
if _, exists := seenKeys[key]; exists {
|
||||
continue
|
||||
}
|
||||
|
||||
references, recordIntervals, valid := normalizeRecordReferences(index, record.SourceRefs)
|
||||
if !valid {
|
||||
continue
|
||||
}
|
||||
seenKeys[key] = struct{}{}
|
||||
candidates = append(candidates, normalizeCandidate{Name: record.Name, SourceRefs: references})
|
||||
candidatePositions = append(candidatePositions, position)
|
||||
for _, interval := range recordIntervals {
|
||||
for position := interval.start; position <= interval.end; position++ {
|
||||
cited[position] = true
|
||||
}
|
||||
intervals = append(intervals, sourceInterval{
|
||||
start: maxInt(0, interval.start-radius),
|
||||
end: minInt(len(doc.Units)-1, interval.end+radius),
|
||||
})
|
||||
}
|
||||
}
|
||||
if len(candidates) < 2 {
|
||||
return normalizeContextMaterials{}, false, nil
|
||||
}
|
||||
|
||||
windows, err := normalizeContextWindows(doc.Units, coalesceIntervals(intervals), cited)
|
||||
if err != nil {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: copy source metadata: %w", err)
|
||||
}
|
||||
candidateContent, err := json.Marshal(normalizeCandidateInput{NPCs: candidates})
|
||||
if err != nil {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: encode candidates: %w", err)
|
||||
}
|
||||
transcriptContent, err := json.Marshal(normalizeTranscriptInput{Windows: windows})
|
||||
if err != nil {
|
||||
return normalizeContextMaterials{}, false, fmt.Errorf("build NPC normalization context: encode transcript: %w", err)
|
||||
}
|
||||
return normalizeContextMaterials{
|
||||
Candidates: newNormalizeInputMaterial("candidates", candidateContent),
|
||||
Transcript: newNormalizeInputMaterial("transcript", transcriptContent),
|
||||
candidatePositions: candidatePositions,
|
||||
}, true, nil
|
||||
}
|
||||
|
||||
func normalizeRecordReferences(index source.DocumentIndex, refs []source.SourceRef) ([]normalizeCandidateSourceRef, []sourceInterval, bool) {
|
||||
references := make([]normalizeCandidateSourceRef, 0, len(refs))
|
||||
intervals := make([]sourceInterval, 0, len(refs))
|
||||
for _, ref := range refs {
|
||||
if err := index.ValidateRef(ref); err != nil {
|
||||
return nil, nil, false
|
||||
}
|
||||
start, _ := index.Position(ref.StartUnitID)
|
||||
end, _ := index.Position(ref.EndUnitID)
|
||||
references = append(references, normalizeCandidateSourceRef{StartUnitID: ref.StartUnitID, EndUnitID: ref.EndUnitID})
|
||||
intervals = append(intervals, sourceInterval{start: start, end: end})
|
||||
}
|
||||
return references, intervals, true
|
||||
}
|
||||
|
||||
func coalesceIntervals(intervals []sourceInterval) []sourceInterval {
|
||||
if len(intervals) == 0 {
|
||||
return nil
|
||||
}
|
||||
ordered := append([]sourceInterval(nil), intervals...)
|
||||
sort.Slice(ordered, func(i, j int) bool {
|
||||
if ordered[i].start != ordered[j].start {
|
||||
return ordered[i].start < ordered[j].start
|
||||
}
|
||||
return ordered[i].end < ordered[j].end
|
||||
})
|
||||
coalesced := make([]sourceInterval, 0, len(ordered))
|
||||
for _, interval := range ordered {
|
||||
if len(coalesced) == 0 || interval.start > coalesced[len(coalesced)-1].end+1 {
|
||||
coalesced = append(coalesced, interval)
|
||||
continue
|
||||
}
|
||||
if interval.end > coalesced[len(coalesced)-1].end {
|
||||
coalesced[len(coalesced)-1].end = interval.end
|
||||
}
|
||||
}
|
||||
return coalesced
|
||||
}
|
||||
|
||||
func normalizeContextWindows(units []source.SourceUnit, intervals []sourceInterval, cited []bool) ([]normalizeTranscriptWindow, error) {
|
||||
windows := make([]normalizeTranscriptWindow, 0, len(intervals))
|
||||
for _, interval := range intervals {
|
||||
window := normalizeTranscriptWindow{Units: make([]normalizeTranscriptUnit, 0, interval.end-interval.start+1)}
|
||||
for position := interval.start; position <= interval.end; position++ {
|
||||
unit := units[position]
|
||||
metadata, err := source.CloneMetadata(unit.Metadata)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
window.Units = append(window.Units, normalizeTranscriptUnit{
|
||||
ID: unit.ID, Kind: unit.Kind, Text: unit.Text, Metadata: metadata, Cited: cited[position],
|
||||
})
|
||||
}
|
||||
windows = append(windows, window)
|
||||
}
|
||||
return windows, nil
|
||||
}
|
||||
|
||||
func newNormalizeInputMaterial(name string, content []byte) contracts.LLMInputMaterial {
|
||||
digest := sha256.Sum256(content)
|
||||
return contracts.NewLLMInputMaterial(name, "application/json", content, "sha256:"+hex.EncodeToString(digest[:]), "")
|
||||
}
|
||||
|
||||
func minInt(left, right int) int {
|
||||
if left < right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
|
||||
func maxInt(left, right int) int {
|
||||
if left > right {
|
||||
return left
|
||||
}
|
||||
return right
|
||||
}
|
||||
132
internal/modules/dnd/normalize/npcs/context_material_test.go
Normal file
132
internal/modules/dnd/normalize/npcs/context_material_test.go
Normal file
@@ -0,0 +1,132 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
)
|
||||
|
||||
func TestBuildNormalizeContextMaterialsUsesDocumentOrderAndOwnedInputs(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
|
||||
{ID: 40, Kind: "narration", Text: "zero"},
|
||||
{ID: 10, Kind: "speech", Text: "one", Metadata: map[string]any{"speaker": map[string]any{"name": "Mira"}}},
|
||||
{ID: 70, Kind: "speech", Text: "two"},
|
||||
{ID: 20, Kind: "narration", Text: "three"},
|
||||
{ID: 90, Kind: "speech", Text: "four"},
|
||||
{ID: 30, Kind: "narration", Text: "five"},
|
||||
}}
|
||||
records := []dnd.NPC{
|
||||
{Name: "Mira Thorn", ID: "npc:sha256:internal", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 20}}},
|
||||
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 90, EndUnitID: 90}}},
|
||||
{Name: "Broken", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 10}}},
|
||||
}
|
||||
before := append([]dnd.NPC(nil), records...)
|
||||
materials, ready, err := buildNormalizeContextMaterials(doc, records, 1)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("buildNormalizeContextMaterials() = %#v, %t, %v; want ready materials", materials, ready, err)
|
||||
}
|
||||
if !reflect.DeepEqual(records, before) {
|
||||
t.Fatalf("records mutated to %#v", records)
|
||||
}
|
||||
for _, material := range []struct {
|
||||
name string
|
||||
data []byte
|
||||
}{
|
||||
{name: "candidates", data: materials.Candidates.Content},
|
||||
{name: "transcript", data: materials.Transcript.Content},
|
||||
} {
|
||||
if !json.Valid(material.data) || string(material.data) == "" {
|
||||
t.Fatalf("%s content = %q, want JSON", material.name, material.data)
|
||||
}
|
||||
digest := sha256.Sum256(material.data)
|
||||
wantDigest := "sha256:" + hex.EncodeToString(digest[:])
|
||||
got := materials.Candidates
|
||||
if material.name == "transcript" {
|
||||
got = materials.Transcript
|
||||
}
|
||||
if got.Name != material.name || got.MediaType != "application/json" || got.OriginURI != "" || got.Digest != wantDigest {
|
||||
t.Fatalf("%s material = %#v, want owned JSON material", material.name, got)
|
||||
}
|
||||
}
|
||||
encoded := string(materials.Candidates.Content) + string(materials.Transcript.Content)
|
||||
if strings.Contains(encoded, "npc:sha256:internal") || strings.Contains(encoded, doc.ID) {
|
||||
t.Fatalf("model material leaked private identifier or source id: %s", encoded)
|
||||
}
|
||||
|
||||
var candidates normalizeCandidateInput
|
||||
if err := json.Unmarshal(materials.Candidates.Content, &candidates); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(candidates.NPCs) != 2 || candidates.NPCs[0].Name != "Mira Thorn" || candidates.NPCs[0].SourceRefs[0] != (normalizeCandidateSourceRef{StartUnitID: 10, EndUnitID: 20}) {
|
||||
t.Fatalf("candidates = %#v, want two valid current-order candidates", candidates)
|
||||
}
|
||||
var transcript normalizeTranscriptInput
|
||||
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 6 {
|
||||
t.Fatalf("transcript = %#v, want one coalesced window", transcript)
|
||||
}
|
||||
units := transcript.Windows[0].Units
|
||||
for index, wantID := range []int{40, 10, 70, 20, 90, 30} {
|
||||
if units[index].ID != wantID {
|
||||
t.Fatalf("unit %d id = %d, want source-order id %d", index, units[index].ID, wantID)
|
||||
}
|
||||
}
|
||||
if units[0].Cited || !units[1].Cited || !units[2].Cited || !units[3].Cited || !units[4].Cited || units[5].Cited {
|
||||
t.Fatalf("citation markers = %#v, want original ranges only", units)
|
||||
}
|
||||
if units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
|
||||
t.Fatalf("metadata = %#v, want copied generic metadata", units[1].Metadata)
|
||||
}
|
||||
|
||||
windows, err := normalizeContextWindows(doc.Units, []sourceInterval{{start: 1, end: 1}}, make([]bool, len(doc.Units)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
windows[0].Units[0].Metadata["speaker"].(map[string]any)["name"] = "changed"
|
||||
if doc.Units[1].Metadata["speaker"].(map[string]any)["name"] != "Mira" {
|
||||
t.Fatal("copied metadata aliases source document")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuildNormalizeContextMaterialsExcludesInvalidReferencesAndCoalescesAdjacentWindows(t *testing.T) {
|
||||
doc := &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
|
||||
{ID: 9}, {ID: 3}, {ID: 8}, {ID: 1}, {ID: 7}, {ID: 2}, {ID: 6},
|
||||
}}
|
||||
records := []dnd.NPC{
|
||||
{Name: "One", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 3, EndUnitID: 3}}},
|
||||
{Name: "Two", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 8, EndUnitID: 8}}},
|
||||
{Name: "Blank"},
|
||||
{Name: "Missing", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 99, EndUnitID: 99}}},
|
||||
{Name: "Foreign", SourceRefs: []source.SourceRef{{SourceID: "other", StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "Reversed", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 6, EndUnitID: 3}}},
|
||||
}
|
||||
materials, ready, err := buildNormalizeContextMaterials(doc, records, 0)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("buildNormalizeContextMaterials() error = %v, ready = %t", err, ready)
|
||||
}
|
||||
var candidates normalizeCandidateInput
|
||||
if err := json.Unmarshal(materials.Candidates.Content, &candidates); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := []string{candidates.NPCs[0].Name, candidates.NPCs[1].Name}; !reflect.DeepEqual(got, []string{"One", "Two"}) {
|
||||
t.Fatalf("candidate names = %#v, want only valid records", got)
|
||||
}
|
||||
var transcript normalizeTranscriptInput
|
||||
if err := json.Unmarshal(materials.Transcript.Content, &transcript); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(transcript.Windows) != 1 || len(transcript.Windows[0].Units) != 2 {
|
||||
t.Fatalf("windows = %#v, want adjacent cited units coalesced", transcript.Windows)
|
||||
}
|
||||
if transcript.Windows[0].Units[0].ID != 3 || transcript.Windows[0].Units[1].ID != 8 {
|
||||
t.Fatalf("window units = %#v, want document-order adjacent units", transcript.Windows[0].Units)
|
||||
}
|
||||
}
|
||||
@@ -3,8 +3,10 @@ package npcs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
|
||||
@@ -19,13 +21,17 @@ import (
|
||||
|
||||
const (
|
||||
Key = "dnd/npcs"
|
||||
normalizationPolicy = "dnd.npcs.normalize.v2"
|
||||
normalizationPolicy = "dnd.npcs.normalize.v3"
|
||||
semanticContextPolicy = "dnd.npcs.semantic_context.v1"
|
||||
NormalizationPolicy = normalizationPolicy
|
||||
|
||||
ReasonCodeNPCFieldsNormalized = "npc_fields_normalized"
|
||||
ReasonCodeNPCIDRecomputed = "npc_id_recomputed"
|
||||
ReasonCodeSourceReferencesNormalized = "source_references_normalized"
|
||||
ReasonCodeDuplicateNPCCollapsed = "duplicate_npc_collapsed"
|
||||
ReasonCodeNPCSemanticProposalInvalid = "npc_semantic_proposal_invalid"
|
||||
ReasonCodeNPCSemanticReconciliationExhausted = "npc_semantic_reconciliation_exhausted"
|
||||
ReasonCodeNPCNormalizationWarningsOmitted = "npc_normalization_warnings_omitted"
|
||||
)
|
||||
|
||||
var requiredCapabilities = []string{"merged"}
|
||||
@@ -36,9 +42,28 @@ var _ contracts.ManifestMetadataProvider = (*Normalizer)(nil)
|
||||
var _ pipeline.CheckpointFingerprintProvider = (*Normalizer)(nil)
|
||||
|
||||
type Options struct{}
|
||||
type Normalizer struct{}
|
||||
|
||||
func New(Options) *Normalizer { return &Normalizer{} }
|
||||
type Normalizer struct {
|
||||
llm contracts.StructuredLLMClient
|
||||
promptSHA string
|
||||
responseSchemaSHA string
|
||||
}
|
||||
|
||||
func New(llmClient contracts.StructuredLLMClient, _ Options) (*Normalizer, error) {
|
||||
if llmClient == nil {
|
||||
return nil, normalizerErrorf("LLM client must not be nil")
|
||||
}
|
||||
promptSHA, err := scriptoriumPromptMetadata()
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("load prompt metadata: %w", err)
|
||||
}
|
||||
responseSchema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
return nil, normalizerErrorf("load response schema: %w", err)
|
||||
}
|
||||
return &Normalizer{llm: llmClient, promptSHA: promptSHA, responseSchemaSHA: responseSchema.SHA256}, nil
|
||||
}
|
||||
|
||||
func (n *Normalizer) Key() string { return Key }
|
||||
func (n *Normalizer) ReferenceSlots() []contracts.ReferenceSlot { return nil }
|
||||
|
||||
@@ -46,7 +71,20 @@ func (n *Normalizer) ManifestMetadata() map[string]any {
|
||||
if n == nil {
|
||||
return nil
|
||||
}
|
||||
return map[string]any{"identity_policy": identity.Policy, "normalization_policy": normalizationPolicy}
|
||||
return map[string]any{
|
||||
"prompt_id": PromptID,
|
||||
"prompt_version": SchemaVersion,
|
||||
"prompt_sha256": n.promptSHA,
|
||||
"response_schema_key": string(ResponseSchemaKey),
|
||||
"response_schema_id": ResponseSchemaID,
|
||||
"response_schema_name": ResponseSchemaName,
|
||||
"response_schema_version": SchemaVersion,
|
||||
"response_schema_sha256": n.responseSchemaSHA,
|
||||
"identity_policy": identity.Policy,
|
||||
"normalization_policy": normalizationPolicy,
|
||||
"semantic_context_policy": semanticContextPolicy,
|
||||
"semantic_context_radius": semanticContextRadius,
|
||||
}
|
||||
}
|
||||
|
||||
func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
@@ -54,8 +92,11 @@ func (n *Normalizer) CheckpointFingerprints() []pipeline.CheckpointFingerprint {
|
||||
return nil
|
||||
}
|
||||
return []pipeline.CheckpointFingerprint{
|
||||
{Name: "prompt", Value: n.promptSHA},
|
||||
{Name: "response_schema", Value: n.responseSchemaSHA},
|
||||
{Name: "identity_policy", Value: identity.Policy},
|
||||
{Name: "normalization_policy", Value: normalizationPolicy},
|
||||
{Name: "semantic_context_policy", Value: fmt.Sprintf("%s:%d", semanticContextPolicy, semanticContextRadius)},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -63,30 +104,114 @@ func (n *Normalizer) Normalize(ctx context.Context, req contracts.TypedNormalize
|
||||
if n == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("normalizer must not be nil")
|
||||
}
|
||||
if n.llm == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("LLM client must not be nil")
|
||||
}
|
||||
if ctx == nil {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context must not be nil")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("context error before normalize: %w", err)
|
||||
}
|
||||
|
||||
order := shared.NewSourceRefOrder(req.Source)
|
||||
value, warnings := normalizeList(req.MergeOutput.Value, order)
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: value, Warnings: warnings}, nil
|
||||
records, warnings := preprocessRecords(req.MergeOutput.Value, order)
|
||||
deterministic := recordList(records)
|
||||
materials, ready, err := buildDefaultNormalizeContextMaterials(req.Source, recordValues(records))
|
||||
if err != nil {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("build semantic context: %w", err)
|
||||
}
|
||||
if !ready {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: deterministic, Warnings: limitWarnings(warnings)}, nil
|
||||
}
|
||||
|
||||
var response normalizeProposalResponse
|
||||
if _, err := n.llm.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: Key, PromptID: PromptID, PromptVersion: SchemaVersion,
|
||||
ProfileID: req.LLMProfile, SessionID: req.SessionID,
|
||||
Inputs: contracts.LLMInputSet{"candidates": materials.Candidates, "transcript": materials.Transcript},
|
||||
}, &response); err != nil {
|
||||
if errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
return n.invalidStructuredResult(deterministic, warnings), nil
|
||||
}
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{}, normalizerErrorf("complete structured output: %w", err)
|
||||
}
|
||||
|
||||
assessment := assessProposal(response, records, materials.candidatePositions)
|
||||
applied, semanticWarnings := applySafeGroups(records, assessment.safeGroups, order)
|
||||
warnings = append(warnings, semanticWarnings...)
|
||||
if assessment.discardedGroups == 0 {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{Value: recordList(applied), Warnings: limitWarnings(warnings)}, nil
|
||||
}
|
||||
return retryResult(recordList(applied), warnings, assessment), nil
|
||||
}
|
||||
|
||||
func (n *Normalizer) invalidStructuredResult(value dnd.NPCList, warnings []contracts.Warning) contracts.TypedNormalizeResult[dnd.NPCList] {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{
|
||||
Value: value,
|
||||
Warnings: limitWarningsForRetry(warnings),
|
||||
Retry: &contracts.NormalizeRetry{
|
||||
ReasonCode: ReasonCodeNPCSemanticProposalInvalid,
|
||||
Message: "semantic proposal requires retry: invalid structured output",
|
||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(-1)},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func retryResult(value dnd.NPCList, warnings []contracts.Warning, assessment proposalAssessment) contracts.TypedNormalizeResult[dnd.NPCList] {
|
||||
return contracts.TypedNormalizeResult[dnd.NPCList]{
|
||||
Value: value,
|
||||
Warnings: limitWarningsForRetry(warnings),
|
||||
Retry: &contracts.NormalizeRetry{
|
||||
ReasonCode: ReasonCodeNPCSemanticProposalInvalid,
|
||||
Message: diagnostics.Aggregate("semantic proposal requires retry", assessment.issues),
|
||||
FallbackWarnings: []contracts.Warning{semanticFallbackWarning(assessment.discardedGroups)},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func semanticFallbackWarning(discardedGroups int) contracts.Warning {
|
||||
message := "semantic proposal could not be applied"
|
||||
if discardedGroups >= 0 {
|
||||
message = fmt.Sprintf("%d proposal group(s) omitted after semantic proposal retry exhaustion", discardedGroups)
|
||||
}
|
||||
return contracts.Warning{Scope: "npcs", ReasonCode: ReasonCodeNPCSemanticReconciliationExhausted, Message: message}
|
||||
}
|
||||
|
||||
func limitWarnings(warnings []contracts.Warning) []contracts.Warning {
|
||||
return diagnostics.LimitWarnings(warnings, "npcs", ReasonCodeNPCNormalizationWarningsOmitted)
|
||||
}
|
||||
|
||||
func limitWarningsForRetry(warnings []contracts.Warning) []contracts.Warning {
|
||||
if warnings == nil {
|
||||
return nil
|
||||
}
|
||||
if len(warnings) < diagnostics.MaxWarnings {
|
||||
return append([]contracts.Warning(nil), warnings...)
|
||||
}
|
||||
displayed := diagnostics.MaxWarnings - 2
|
||||
bounded := append([]contracts.Warning(nil), warnings[:displayed]...)
|
||||
return append(bounded, contracts.Warning{
|
||||
Scope: "npcs", ReasonCode: ReasonCodeNPCNormalizationWarningsOmitted,
|
||||
Message: fmt.Sprintf("%d additional warning(s) omitted", len(warnings)-displayed),
|
||||
})
|
||||
}
|
||||
|
||||
type normalizedRecord struct {
|
||||
npc dnd.NPC
|
||||
inputIndexes []int
|
||||
earliest int
|
||||
}
|
||||
|
||||
func normalizeList(input dnd.NPCList, order shared.SourceRefOrder) (dnd.NPCList, []contracts.Warning) {
|
||||
func preprocessRecords(input dnd.NPCList, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
||||
if input.NPCs == nil {
|
||||
return dnd.NPCList{}, nil
|
||||
return nil, nil
|
||||
}
|
||||
records := make([]normalizedRecord, len(input.NPCs))
|
||||
warnings := make([]contracts.Warning, 0)
|
||||
for index, inputNPC := range input.NPCs {
|
||||
npc, fieldsChanged, referencesChanged := normalizeRecord(inputNPC, order)
|
||||
records[index] = normalizedRecord{npc: npc}
|
||||
records[index] = normalizedRecord{npc: npc, inputIndexes: []int{index}, earliest: index}
|
||||
if fieldsChanged {
|
||||
warnings = append(warnings, contracts.Warning{Scope: npcScope(index), ReasonCode: ReasonCodeNPCFieldsNormalized, Message: fmt.Sprintf("input index %d: NPC name normalized for %s", index, diagnostics.Quote(inputNPC.Name))})
|
||||
}
|
||||
@@ -99,16 +224,16 @@ func normalizeList(input dnd.NPCList, order shared.SourceRefOrder) (dnd.NPCList,
|
||||
}
|
||||
|
||||
groups := canonicalNameGroups(records)
|
||||
output := dnd.NPCList{NPCs: make([]dnd.NPC, 0, len(groups))}
|
||||
output := make([]normalizedRecord, 0, len(groups))
|
||||
for _, members := range groups {
|
||||
consolidated, referencesChanged := consolidate(records, members, order)
|
||||
retainedIndex := members[0]
|
||||
output.NPCs = append(output.NPCs, consolidated)
|
||||
output = append(output, consolidated)
|
||||
retainedIndex := consolidated.earliest
|
||||
if referencesChanged {
|
||||
warnings = append(warnings, contracts.Warning{Scope: npcScope(retainedIndex), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized during identity consolidation (final count %d)", retainedIndex, len(consolidated.SourceRefs))})
|
||||
warnings = append(warnings, contracts.Warning{Scope: npcScope(retainedIndex), ReasonCode: ReasonCodeSourceReferencesNormalized, Message: fmt.Sprintf("input index %d: source references normalized during identity consolidation (final count %d)", retainedIndex, len(consolidated.npc.SourceRefs))})
|
||||
}
|
||||
if len(members) > 1 {
|
||||
warnings = append(warnings, duplicateWarning(retainedIndex, members[1:]))
|
||||
warnings = append(warnings, duplicateWarning(retainedIndex, memberInputIndexes(records, members[1:])))
|
||||
}
|
||||
}
|
||||
return output, warnings
|
||||
@@ -144,15 +269,68 @@ func canonicalNameGroups(records []normalizedRecord) [][]int {
|
||||
return groups
|
||||
}
|
||||
|
||||
func consolidate(records []normalizedRecord, members []int, order shared.SourceRefOrder) (dnd.NPC, bool) {
|
||||
output := cloneNPC(records[members[0]].npc)
|
||||
originalRefs := cloneSourceRefs(output.SourceRefs)
|
||||
func consolidate(records []normalizedRecord, members []int, order shared.SourceRefOrder) (normalizedRecord, bool) {
|
||||
output := cloneRecord(records[members[0]])
|
||||
originalRefs := cloneSourceRefs(output.npc.SourceRefs)
|
||||
for _, member := range members[1:] {
|
||||
output.SourceRefs = append(output.SourceRefs, records[member].npc.SourceRefs...)
|
||||
output.npc.SourceRefs = append(output.npc.SourceRefs, records[member].npc.SourceRefs...)
|
||||
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
|
||||
if records[member].earliest < output.earliest {
|
||||
output.earliest = records[member].earliest
|
||||
}
|
||||
output.SourceRefs = order.Canonicalize(output.SourceRefs)
|
||||
output.ID = identity.DeriveID(output.Name)
|
||||
return output, !reflect.DeepEqual(originalRefs, output.SourceRefs)
|
||||
}
|
||||
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
|
||||
output.npc.SourceRefs = order.Canonicalize(output.npc.SourceRefs)
|
||||
output.npc.ID = identity.DeriveID(output.npc.Name)
|
||||
return output, !reflect.DeepEqual(originalRefs, output.npc.SourceRefs)
|
||||
}
|
||||
|
||||
func cloneRecord(input normalizedRecord) normalizedRecord {
|
||||
input.npc = cloneNPC(input.npc)
|
||||
input.inputIndexes = append([]int(nil), input.inputIndexes...)
|
||||
return input
|
||||
}
|
||||
|
||||
func memberInputIndexes(records []normalizedRecord, members []int) []int {
|
||||
indexes := make([]int, 0, len(members))
|
||||
for _, member := range members {
|
||||
indexes = append(indexes, records[member].inputIndexes...)
|
||||
}
|
||||
return sortedUniqueIndexes(indexes)
|
||||
}
|
||||
|
||||
func sortedUniqueIndexes(indexes []int) []int {
|
||||
if len(indexes) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := append([]int(nil), indexes...)
|
||||
sort.Ints(out)
|
||||
write := 1
|
||||
for _, index := range out[1:] {
|
||||
if index != out[write-1] {
|
||||
out[write] = index
|
||||
write++
|
||||
}
|
||||
}
|
||||
return out[:write]
|
||||
}
|
||||
|
||||
func recordValues(records []normalizedRecord) []dnd.NPC {
|
||||
if records == nil {
|
||||
return nil
|
||||
}
|
||||
values := make([]dnd.NPC, len(records))
|
||||
for index, record := range records {
|
||||
values[index] = cloneNPC(record.npc)
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
func recordList(records []normalizedRecord) dnd.NPCList {
|
||||
if records == nil {
|
||||
return dnd.NPCList{}
|
||||
}
|
||||
return dnd.NPCList{NPCs: recordValues(records)}
|
||||
}
|
||||
|
||||
func cloneSourceRefs(input []source.SourceRef) []source.SourceRef {
|
||||
@@ -191,7 +369,7 @@ func Register(registry *pipeline.NormalizerRegistry) error {
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return New(options), nil
|
||||
return New(request.Dependencies.LLM, options)
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package npcs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
@@ -28,11 +29,14 @@ func TestModuleContractAndIdentity(t *testing.T) {
|
||||
if err := Register(registry); err != nil {
|
||||
t.Fatalf("Register() error = %v", err)
|
||||
}
|
||||
normalizer := New(Options{})
|
||||
if metadata := normalizer.ManifestMetadata(); metadata["identity_policy"] != identity.Policy || metadata["normalization_policy"] != normalizationPolicy {
|
||||
if _, err := New(nil, Options{}); err == nil {
|
||||
t.Fatal("New(nil, Options{}) error = nil, want nil client rejection")
|
||||
}
|
||||
normalizer := newNormalizer(t, &recordingNPCNormalizerClient{})
|
||||
if metadata := normalizer.ManifestMetadata(); metadata["identity_policy"] != identity.Policy || metadata["normalization_policy"] != normalizationPolicy || metadata["prompt_id"] != PromptID || metadata["semantic_context_policy"] != semanticContextPolicy || metadata["semantic_context_radius"] != semanticContextRadius {
|
||||
t.Fatalf("metadata = %#v", metadata)
|
||||
}
|
||||
wantFingerprints := []pipeline.CheckpointFingerprint{{Name: "identity_policy", Value: identity.Policy}, {Name: "normalization_policy", Value: normalizationPolicy}}
|
||||
wantFingerprints := []pipeline.CheckpointFingerprint{{Name: "prompt", Value: normalizer.promptSHA}, {Name: "response_schema", Value: normalizer.responseSchemaSHA}, {Name: "identity_policy", Value: identity.Policy}, {Name: "normalization_policy", Value: normalizationPolicy}, {Name: "semantic_context_policy", Value: semanticContextPolicy + ":2"}}
|
||||
if got := normalizer.CheckpointFingerprints(); !reflect.DeepEqual(got, wantFingerprints) {
|
||||
t.Fatalf("fingerprints = %#v, want %#v", got, wantFingerprints)
|
||||
}
|
||||
@@ -46,7 +50,7 @@ func TestNormalizeNamesEvidenceAndIDs(t *testing.T) {
|
||||
{SourceID: "b", StartUnitID: 2, EndUnitID: 3},
|
||||
},
|
||||
}}}
|
||||
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v", err)
|
||||
}
|
||||
@@ -68,7 +72,7 @@ func TestNormalizeOrdersEvidenceBySourceDocumentPosition(t *testing.T) {
|
||||
{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30},
|
||||
{SourceID: doc.ID, StartUnitID: 999, EndUnitID: 999},
|
||||
}}}}
|
||||
result, err := New(Options{}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v", err)
|
||||
}
|
||||
@@ -84,7 +88,7 @@ func TestNormalizeConsolidatesCanonicalNamesOnlyAndUnionsEvidence(t *testing.T)
|
||||
{Name: "captain vale", SourceRefs: []source.SourceRef{{SourceID: "b", StartUnitID: 2, EndUnitID: 2}}},
|
||||
{Name: "The Captain", SourceRefs: []source.SourceRef{{SourceID: "c", StartUnitID: 3, EndUnitID: 3}}},
|
||||
}}
|
||||
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
if err != nil {
|
||||
t.Fatalf("Normalize() error = %v", err)
|
||||
}
|
||||
@@ -102,7 +106,7 @@ func TestNormalizeConsolidatesCanonicalNamesOnlyAndUnionsEvidence(t *testing.T)
|
||||
func TestNormalizePreservesInvalidCandidatesAndDoesNotAliasInput(t *testing.T) {
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{{Name: " ", SourceRefs: []source.SourceRef{{SourceID: "source", StartUnitID: 0, EndUnitID: -1}}}}}
|
||||
before := dnd.NPCList{NPCs: []dnd.NPC{{Name: " ", SourceRefs: []source.SourceRef{{SourceID: "source", StartUnitID: 0, EndUnitID: -1}}}}}
|
||||
result, err := New(Options{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
result, err := newNormalizer(t, &recordingNPCNormalizerClient{}).Normalize(context.Background(), normalizeRequest(input))
|
||||
if err != nil || !reflect.DeepEqual(input, before) {
|
||||
t.Fatalf("Normalize() = %#v, %v; input mutated to %#v", result, err, input)
|
||||
}
|
||||
@@ -116,7 +120,7 @@ func TestNormalizePreservesInvalidCandidatesAndDoesNotAliasInput(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestNormalizeHandlesNilAndCanceledCalls(t *testing.T) {
|
||||
normalizer := New(Options{})
|
||||
normalizer := newNormalizer(t, &recordingNPCNormalizerClient{})
|
||||
result, err := normalizer.Normalize(context.Background(), normalizeRequest(dnd.NPCList{NPCs: nil}))
|
||||
if err != nil || result.Value.NPCs != nil {
|
||||
t.Fatalf("nil list result = %#v, error = %v", result.Value, err)
|
||||
@@ -128,6 +132,44 @@ func TestNormalizeHandlesNilAndCanceledCalls(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
type recordingNPCNormalizerClient struct {
|
||||
response string
|
||||
responses []string
|
||||
err error
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (c *recordingNPCNormalizerClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, output any) (contracts.StructuredCompletionResponse, error) {
|
||||
c.requests = append(c.requests, request)
|
||||
if c.err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, c.err
|
||||
}
|
||||
response := c.response
|
||||
if len(c.responses) > 0 {
|
||||
responseIndex := len(c.requests) - 1
|
||||
if responseIndex >= len(c.responses) {
|
||||
responseIndex = len(c.responses) - 1
|
||||
}
|
||||
response = c.responses[responseIndex]
|
||||
}
|
||||
if response == "" {
|
||||
response = `{"duplicate_groups":[]}`
|
||||
}
|
||||
if err := json.Unmarshal([]byte(response), output); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: json.RawMessage(response)}, nil
|
||||
}
|
||||
|
||||
func newNormalizer(t *testing.T, client contracts.StructuredLLMClient) *Normalizer {
|
||||
t.Helper()
|
||||
normalizer, err := New(client, Options{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
return normalizer
|
||||
}
|
||||
|
||||
func normalizeRequest(value dnd.NPCList) contracts.TypedNormalizeRequest[dnd.NPCList] {
|
||||
return contracts.TypedNormalizeRequest[dnd.NPCList]{MergeOutput: contracts.MergeArtifact[dnd.NPCList]{Value: value}}
|
||||
}
|
||||
|
||||
195
internal/modules/dnd/normalize/npcs/proposal.go
Normal file
195
internal/modules/dnd/normalize/npcs/proposal.go
Normal file
@@ -0,0 +1,195 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sort"
|
||||
"strconv"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared/diagnostics"
|
||||
)
|
||||
|
||||
type proposalAssessment struct {
|
||||
safeGroups []safeProposalGroup
|
||||
discardedGroups int
|
||||
issues []string
|
||||
}
|
||||
|
||||
type safeProposalGroup struct {
|
||||
members []int
|
||||
canonical int
|
||||
}
|
||||
|
||||
type assessedProposalGroup struct {
|
||||
members []int
|
||||
canonical int
|
||||
locallyValid bool
|
||||
conflicting bool
|
||||
}
|
||||
|
||||
func assessProposal(response normalizeProposalResponse, records []normalizedRecord, candidatePositions []int) proposalAssessment {
|
||||
positionsByKey := make(map[string][]int, len(candidatePositions))
|
||||
for _, position := range candidatePositions {
|
||||
if position < 0 || position >= len(records) {
|
||||
continue
|
||||
}
|
||||
key := identity.ComparisonKey(records[position].npc.Name)
|
||||
if key != "" {
|
||||
positionsByKey[key] = append(positionsByKey[key], position)
|
||||
}
|
||||
}
|
||||
groups := make([]assessedProposalGroup, len(response.DuplicateGroups))
|
||||
issues := make([]string, 0)
|
||||
owners := make(map[int][]int)
|
||||
|
||||
for groupIndex, proposal := range response.DuplicateGroups {
|
||||
group, groupIssues := assessProposalGroup(proposal, positionsByKey)
|
||||
groups[groupIndex] = group
|
||||
for _, position := range group.members {
|
||||
owners[position] = append(owners[position], groupIndex)
|
||||
}
|
||||
for _, issue := range groupIssues {
|
||||
issues = append(issues, proposalIssue(groupIndex, issue))
|
||||
}
|
||||
}
|
||||
for groupIndex := range groups {
|
||||
for _, position := range groups[groupIndex].members {
|
||||
if len(owners[position]) > 1 {
|
||||
groups[groupIndex].conflicting = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if groups[groupIndex].conflicting {
|
||||
issues = append(issues, proposalIssue(groupIndex, "overlapping_member"))
|
||||
}
|
||||
}
|
||||
|
||||
assessment := proposalAssessment{issues: issues}
|
||||
for _, group := range groups {
|
||||
if !group.locallyValid || group.conflicting {
|
||||
assessment.discardedGroups++
|
||||
continue
|
||||
}
|
||||
assessment.safeGroups = append(assessment.safeGroups, safeProposalGroup{members: group.members, canonical: group.canonical})
|
||||
}
|
||||
return assessment
|
||||
}
|
||||
|
||||
func assessProposalGroup(proposal normalizeProposalGroup, positionsByKey map[string][]int) (assessedProposalGroup, []string) {
|
||||
issues := make([]string, 0)
|
||||
members := make([]int, 0, len(proposal.Members))
|
||||
seenMembers := make(map[int]struct{}, len(proposal.Members))
|
||||
for _, name := range proposal.Members {
|
||||
position, issue := resolveCandidate(name, positionsByKey)
|
||||
if issue != "" {
|
||||
issues = append(issues, "member_"+issue)
|
||||
continue
|
||||
}
|
||||
if _, exists := seenMembers[position]; exists {
|
||||
issues = append(issues, "repeated_member")
|
||||
continue
|
||||
}
|
||||
seenMembers[position] = struct{}{}
|
||||
members = append(members, position)
|
||||
}
|
||||
canonical, canonicalIssue := resolveCandidate(proposal.CanonicalName, positionsByKey)
|
||||
if canonicalIssue != "" {
|
||||
issues = append(issues, "canonical_"+canonicalIssue)
|
||||
}
|
||||
if len(members) < 2 {
|
||||
issues = append(issues, "fewer_than_two_members")
|
||||
}
|
||||
if canonicalIssue == "" && !containsPosition(members, canonical) {
|
||||
issues = append(issues, "canonical_not_member")
|
||||
}
|
||||
sort.Ints(members)
|
||||
return assessedProposalGroup{
|
||||
members: members, canonical: canonical, locallyValid: len(issues) == 0,
|
||||
}, issues
|
||||
}
|
||||
|
||||
func resolveCandidate(name string, positionsByKey map[string][]int) (position int, issue string) {
|
||||
key := identity.ComparisonKey(name)
|
||||
if key == "" {
|
||||
return 0, "blank"
|
||||
}
|
||||
positions := positionsByKey[key]
|
||||
if len(positions) == 0 {
|
||||
return 0, "unknown"
|
||||
}
|
||||
if len(positions) != 1 {
|
||||
return 0, "ambiguous"
|
||||
}
|
||||
return positions[0], ""
|
||||
}
|
||||
|
||||
func containsPosition(positions []int, want int) bool {
|
||||
for _, position := range positions {
|
||||
if position == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func proposalIssue(groupIndex int, category string) string {
|
||||
return "group " + strconv.Itoa(groupIndex) + ": " + category
|
||||
}
|
||||
|
||||
func applySafeGroups(records []normalizedRecord, groups []safeProposalGroup, order shared.SourceRefOrder) ([]normalizedRecord, []contracts.Warning) {
|
||||
byMember := make(map[int]safeProposalGroup, len(groups)*2)
|
||||
for _, group := range groups {
|
||||
for _, member := range group.members {
|
||||
byMember[member] = group
|
||||
}
|
||||
}
|
||||
output := make([]normalizedRecord, 0, len(records)-len(groups))
|
||||
warnings := make([]contracts.Warning, 0, len(groups))
|
||||
for index, record := range records {
|
||||
group, grouped := byMember[index]
|
||||
if !grouped {
|
||||
output = append(output, cloneRecord(record))
|
||||
continue
|
||||
}
|
||||
if group.members[0] != index {
|
||||
continue
|
||||
}
|
||||
consolidated := consolidateSemanticGroup(records, group, order)
|
||||
output = append(output, consolidated)
|
||||
warnings = append(warnings, semanticDuplicateWarning(consolidated, records[group.canonical]))
|
||||
}
|
||||
return output, warnings
|
||||
}
|
||||
|
||||
func consolidateSemanticGroup(records []normalizedRecord, group safeProposalGroup, order shared.SourceRefOrder) normalizedRecord {
|
||||
output := cloneRecord(records[group.members[0]])
|
||||
output.npc.Name = records[group.canonical].npc.Name
|
||||
for _, member := range group.members[1:] {
|
||||
output.npc.SourceRefs = append(output.npc.SourceRefs, records[member].npc.SourceRefs...)
|
||||
output.inputIndexes = append(output.inputIndexes, records[member].inputIndexes...)
|
||||
if records[member].earliest < output.earliest {
|
||||
output.earliest = records[member].earliest
|
||||
}
|
||||
}
|
||||
output.inputIndexes = sortedUniqueIndexes(output.inputIndexes)
|
||||
output.npc.SourceRefs = order.Canonicalize(output.npc.SourceRefs)
|
||||
output.npc.ID = identity.DeriveID(output.npc.Name)
|
||||
return output
|
||||
}
|
||||
|
||||
func semanticDuplicateWarning(record normalizedRecord, canonical normalizedRecord) contracts.Warning {
|
||||
details := make([]string, 0, len(record.inputIndexes)+1)
|
||||
for _, inputIndex := range record.inputIndexes {
|
||||
details = append(details, fmt.Sprintf("input index %d", inputIndex))
|
||||
}
|
||||
if canonical.earliest != record.earliest {
|
||||
details = append(details, fmt.Sprintf("canonical display name from input index %d", canonical.earliest))
|
||||
}
|
||||
return contracts.Warning{
|
||||
Scope: npcScope(record.earliest),
|
||||
ReasonCode: ReasonCodeDuplicateNPCCollapsed,
|
||||
Message: diagnostics.Aggregate("semantic duplicate consolidation", details),
|
||||
}
|
||||
}
|
||||
21
internal/modules/dnd/normalize/npcs/schema.go
Normal file
21
internal/modules/dnd/normalize/npcs/schema.go
Normal file
@@ -0,0 +1,21 @@
|
||||
package npcs
|
||||
|
||||
import "gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
|
||||
const (
|
||||
PromptID = "dnd.npcs.normalize"
|
||||
ResponseSchemaKey = llm.ResponseSchemaKey("dnd_npcs_normalize_llm")
|
||||
ResponseSchemaID = "notarius.dnd.npcs.normalize.llm"
|
||||
ResponseSchemaName = "notarius_dnd_npcs_normalize_llm_v1"
|
||||
SchemaVersion = "v1"
|
||||
)
|
||||
|
||||
func loadResponseSchema() (llm.ResponseSchema, error) {
|
||||
return llm.LoadResponseSchema(embeddedAssets, llm.ResponseSchemaDefinition{
|
||||
Key: ResponseSchemaKey,
|
||||
ID: ResponseSchemaID,
|
||||
Version: SchemaVersion,
|
||||
Name: ResponseSchemaName,
|
||||
AssetPath: "assets/schemas/dnd_npcs_normalize_llm.v1.json",
|
||||
})
|
||||
}
|
||||
65
internal/modules/dnd/normalize/npcs/schema_test.go
Normal file
65
internal/modules/dnd/normalize/npcs/schema_test.go
Normal file
@@ -0,0 +1,65 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/santhosh-tekuri/jsonschema/v6"
|
||||
)
|
||||
|
||||
func TestNormalizeResponseSchemaIsStrictlyStructural(t *testing.T) {
|
||||
schema, err := loadResponseSchema()
|
||||
if err != nil {
|
||||
t.Fatalf("loadResponseSchema() error = %v", err)
|
||||
}
|
||||
if schema.Key != ResponseSchemaKey || schema.ID != ResponseSchemaID || schema.Name != ResponseSchemaName || schema.Version != SchemaVersion || !strings.HasPrefix(schema.SHA256, "sha256:") || !json.Valid(schema.JSONSchema) {
|
||||
t.Fatalf("schema = %#v, want private normalization schema identity", schema)
|
||||
}
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
value any
|
||||
valid bool
|
||||
}{
|
||||
{name: "empty groups", value: map[string]any{"duplicate_groups": []any{}}, valid: true},
|
||||
{name: "semantically invalid group", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{"", "unknown"}, "canonical_name": ""}}}, valid: true},
|
||||
{name: "missing groups", value: map[string]any{}},
|
||||
{name: "unknown top level field", value: map[string]any{"duplicate_groups": []any{}, "extra": true}},
|
||||
{name: "unknown group field", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{}, "canonical_name": "Mira", "extra": true}}}},
|
||||
{name: "wrong groups type", value: map[string]any{"duplicate_groups": "no"}},
|
||||
{name: "wrong member type", value: map[string]any{"duplicate_groups": []any{map[string]any{"members": []any{1}, "canonical_name": "Mira"}}}},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
content, err := json.Marshal(test.value)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err = validateNormalizeSchema(content, schema.JSONSchema)
|
||||
if (err == nil) != test.valid {
|
||||
t.Fatalf("validateNormalizeSchema() error = %v, want valid=%t", err, test.valid)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func validateNormalizeSchema(instanceContent, schemaContent []byte) error {
|
||||
instance, err := jsonschema.UnmarshalJSON(bytes.NewReader(instanceContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
document, err := jsonschema.UnmarshalJSON(bytes.NewReader(schemaContent))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
compiler := jsonschema.NewCompiler()
|
||||
if err := compiler.AddResource("schema.json", document); err != nil {
|
||||
return err
|
||||
}
|
||||
compiled, err := compiler.Compile("schema.json")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return compiled.Validate(instance)
|
||||
}
|
||||
51
internal/modules/dnd/normalize/npcs/scriptorium_assets.go
Normal file
51
internal/modules/dnd/normalize/npcs/scriptorium_assets.go
Normal file
@@ -0,0 +1,51 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/promptfs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/shared"
|
||||
)
|
||||
|
||||
const scriptoriumPromptRoot = "assets/prompts"
|
||||
|
||||
var promptAssetManifest = shared.PromptAssetManifest{
|
||||
ModuleDir: PromptID,
|
||||
ModuleFiles: []promptfs.ModulePromptFile{
|
||||
{Name: "dnd.npcs.normalize.yaml", Path: "assets/prompts/dnd.npcs.normalize.yaml"},
|
||||
{Name: "task.md", Path: "assets/prompts/task.md"},
|
||||
{Name: "instructions.md", Path: "assets/prompts/instructions.md"},
|
||||
{Name: "candidates.md", Path: "assets/prompts/candidates.md"},
|
||||
},
|
||||
SharedFiles: []string{
|
||||
"common-dnd-system.md",
|
||||
"common-dnd-identity.md",
|
||||
"common-dnd-transcript.md",
|
||||
},
|
||||
}
|
||||
|
||||
func RegisterPromptAssets(registry *llm.AssetRegistry) error {
|
||||
promptFS, err := promptAssetManifest.PromptFS(embeddedAssets)
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare NPC normalization prompt assets: %w", err)
|
||||
}
|
||||
if err := registry.RegisterPromptFS(promptFS, scriptoriumPromptRoot); err != nil {
|
||||
return err
|
||||
}
|
||||
return registry.RegisterSchemaFS(embeddedAssets, "assets/schemas")
|
||||
}
|
||||
|
||||
func scriptoriumPromptMetadata() (string, error) {
|
||||
scriptoriumPromptHashOnce.Do(func() {
|
||||
scriptoriumPromptHash, scriptoriumPromptHashErr = promptAssetManifest.Hash(embeddedAssets)
|
||||
})
|
||||
return scriptoriumPromptHash, scriptoriumPromptHashErr
|
||||
}
|
||||
|
||||
var (
|
||||
scriptoriumPromptHashOnce sync.Once
|
||||
scriptoriumPromptHash string
|
||||
scriptoriumPromptHashErr error
|
||||
)
|
||||
@@ -0,0 +1,61 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
|
||||
"gitea.maximumdirect.net/eric/scriptorium"
|
||||
)
|
||||
|
||||
func TestRegisterPromptAssetsPreparesNormalizationPrompt(t *testing.T) {
|
||||
registry := llm.NewAssetRegistry()
|
||||
if err := RegisterPromptAssets(registry); err != nil {
|
||||
t.Fatalf("RegisterPromptAssets() error = %v", err)
|
||||
}
|
||||
options, err := registry.ScriptoriumOptions()
|
||||
if err != nil {
|
||||
t.Fatalf("ScriptoriumOptions() error = %v", err)
|
||||
}
|
||||
options = append(options, scriptorium.WithProfiles(scriptorium.OpenAICompatibleProfile(scriptorium.OpenAICompatibleProfileConfig{
|
||||
ID: "normalize-test-profile", Endpoint: "http://127.0.0.1:1/v1", Model: "normalize-test-model",
|
||||
})))
|
||||
engine, err := scriptorium.NewEngine(scriptorium.Config{Timeout: time.Second}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
prepared, err := engine.Prepare(context.Background(), scriptorium.RunRequest{
|
||||
PromptID: PromptID, PromptVersion: SchemaVersion, ProfileID: "normalize-test-profile",
|
||||
Inputs: map[string]scriptorium.ArtifactRef{
|
||||
"candidates": scriptorium.Inline(`{"npcs":[{"name":"Mira","source_refs":[]}]}`),
|
||||
"transcript": scriptorium.Inline(`{"windows":[{"units":[]}]}`),
|
||||
},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Prepare() error = %v", err)
|
||||
}
|
||||
if prepared.PromptID != PromptID || prepared.OutputContract.SchemaPath != "dnd_npcs_normalize_llm.v1.json" {
|
||||
t.Fatalf("prepared prompt = %#v, want normalization prompt identity and schema", prepared)
|
||||
}
|
||||
if len(prepared.Messages) != 6 {
|
||||
t.Fatalf("prepared messages = %d, want 6", len(prepared.Messages))
|
||||
}
|
||||
for _, index := range []int{1, 3} {
|
||||
if cache := prepared.Messages[index].CacheControl; cache == nil || cache.Type != scriptorium.CacheControlEphemeral {
|
||||
t.Errorf("message %d cache control = %#v, want ephemeral", index, cache)
|
||||
}
|
||||
}
|
||||
for _, index := range []int{0, 2, 4, 5} {
|
||||
if cache := prepared.Messages[index].CacheControl; cache != nil {
|
||||
t.Errorf("message %d cache control = %#v, want nil", index, cache)
|
||||
}
|
||||
}
|
||||
if !strings.Contains(prepared.Messages[4].Content, `"Mira"`) || strings.Contains(prepared.Messages[4].Content, `"windows"`) {
|
||||
t.Fatalf("candidate message = %q, want only rendered candidates", prepared.Messages[4].Content)
|
||||
}
|
||||
if !strings.Contains(prepared.Messages[5].Content, `"windows"`) || strings.Contains(prepared.Messages[5].Content, `"Mira"`) {
|
||||
t.Fatalf("transcript message = %q, want only rendered transcript", prepared.Messages[5].Content)
|
||||
}
|
||||
}
|
||||
265
internal/modules/dnd/normalize/npcs/semantic_normalizer_test.go
Normal file
265
internal/modules/dnd/normalize/npcs/semantic_normalizer_test.go
Normal file
@@ -0,0 +1,265 @@
|
||||
package npcs
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"math"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/contracts"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
)
|
||||
|
||||
func TestNormalizeSkipsSemanticCompletionWithoutTwoEligibleCandidates(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}}}}
|
||||
result, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || len(client.requests) != 0 || result.Retry != nil {
|
||||
t.Fatalf("Normalize() = %#v, %v; calls = %d, want deterministic no-call result", result, err, len(client.requests))
|
||||
}
|
||||
if result.Value.NPCs[0].Name != "Mira Thorn" {
|
||||
t.Fatalf("NPCs = %#v, want deterministic record", result.Value.NPCs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeAppliesSafeProposalAndUsesPrivateInputs(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":[" MIRA THORN ","Mira"],"canonical_name":"Mira Thorn"}]}`}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
{ID: "npc:sha256:short", Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{ID: "npc:sha256:long", Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
{ID: "npc:sha256:captain", Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
|
||||
}}
|
||||
before := cloneNPCList(input)
|
||||
request := normalizeRequestWithSource(input, doc)
|
||||
request.LLMProfile = "normalizer-profile"
|
||||
request.SessionID = "normalizer-session"
|
||||
result, err := normalizer.Normalize(context.Background(), request)
|
||||
if err != nil || result.Retry != nil {
|
||||
t.Fatalf("Normalize() = %#v, %v; want accepted semantic result", result, err)
|
||||
}
|
||||
if !reflect.DeepEqual(input, before) {
|
||||
t.Fatalf("Normalize() mutated input to %#v", input)
|
||||
}
|
||||
if len(result.Value.NPCs) != 2 || result.Value.NPCs[0].Name != "Mira Thorn" || result.Value.NPCs[1].Name != "Captain Vale" {
|
||||
t.Fatalf("NPCs = %#v, want canonical record at earliest position", result.Value.NPCs)
|
||||
}
|
||||
merged := result.Value.NPCs[0]
|
||||
if merged.ID != identity.DeriveID("Mira Thorn") || !reflect.DeepEqual(merged.SourceRefs, []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}, {SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}) {
|
||||
t.Fatalf("merged NPC = %#v, want canonical ID and original evidence union", merged)
|
||||
}
|
||||
if !hasWarning(result.Warnings, ReasonCodeDuplicateNPCCollapsed, "npcs[0]") {
|
||||
t.Fatalf("warnings = %#v, want semantic collapse warning", result.Warnings)
|
||||
}
|
||||
if len(client.requests) != 1 {
|
||||
t.Fatalf("completion calls = %d, want one", len(client.requests))
|
||||
}
|
||||
completion := client.requests[0]
|
||||
if completion.StageName != Key || completion.PromptID != PromptID || completion.PromptVersion != SchemaVersion || completion.ProfileID != request.LLMProfile || completion.SessionID != request.SessionID || len(completion.Inputs) != 2 {
|
||||
t.Fatalf("completion request = %#v, want normalize request identity and exactly two inputs", completion)
|
||||
}
|
||||
encoded := string(completion.Inputs["candidates"].Content) + string(completion.Inputs["transcript"].Content)
|
||||
if strings.Contains(encoded, "npc:sha256:") || strings.Contains(encoded, doc.ID) {
|
||||
t.Fatalf("completion inputs leaked private identifiers: %s", encoded)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeUnsafeProposalReturnsSafeRetryFallback(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Captain Vale","Unknown"],"canonical_name":"Captain Vale"}]}`}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
|
||||
}}
|
||||
result, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || result.Retry == nil {
|
||||
t.Fatalf("Normalize() = %#v, %v; want retryable safe fallback", result, err)
|
||||
}
|
||||
if result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || !strings.Contains(result.Retry.Message, "group 1: member_unknown") {
|
||||
t.Fatalf("retry = %#v, want bounded invalid-proposal diagnostics", result.Retry)
|
||||
}
|
||||
if len(result.Value.NPCs) != 2 || result.Value.NPCs[0].Name != "Mira Thorn" || result.Value.NPCs[1].Name != "Captain Vale" {
|
||||
t.Fatalf("fallback NPCs = %#v, want independently safe group applied", result.Value.NPCs)
|
||||
}
|
||||
if len(result.Retry.FallbackWarnings) != 1 || result.Retry.FallbackWarnings[0].ReasonCode != ReasonCodeNPCSemanticReconciliationExhausted || !strings.Contains(result.Retry.FallbackWarnings[0].Message, "1 proposal group") {
|
||||
t.Fatalf("fallback warnings = %#v, want exact omitted-group warning", result.Retry.FallbackWarnings)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRejectsOverlapsWithoutResponseOrderDependence(t *testing.T) {
|
||||
records := []normalizedRecord{
|
||||
{npc: dnd.NPC{Name: "Alpha"}}, {npc: dnd.NPC{Name: "Bravo"}}, {npc: dnd.NPC{Name: "Charlie"}}, {npc: dnd.NPC{Name: "Delta"}},
|
||||
}
|
||||
for index := range records {
|
||||
records[index].inputIndexes = []int{index}
|
||||
records[index].earliest = index
|
||||
}
|
||||
response := normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{
|
||||
{Members: []string{"Alpha", "Bravo"}, CanonicalName: "Alpha"},
|
||||
{Members: []string{"Bravo", "Charlie"}, CanonicalName: "Bravo"},
|
||||
{Members: []string{"Charlie", "Delta"}, CanonicalName: "Charlie"},
|
||||
}}
|
||||
assessment := assessProposal(response, records, []int{0, 1, 2, 3})
|
||||
if assessment.discardedGroups != 3 || len(assessment.safeGroups) != 0 {
|
||||
t.Fatalf("assessment = %#v, want chained conflicts all discarded", assessment)
|
||||
}
|
||||
for _, issue := range []string{"group 0: overlapping_member", "group 1: overlapping_member", "group 2: overlapping_member"} {
|
||||
if !containsString(assessment.issues, issue) {
|
||||
t.Fatalf("issues = %#v, want %q", assessment.issues, issue)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeInvalidStructuredOutputAndOperationalErrorsRemainDistinct(t *testing.T) {
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
}}
|
||||
invalid := newNormalizer(t, &recordingNPCNormalizerClient{err: contracts.ErrInvalidStructuredOutput})
|
||||
result, err := invalid.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || result.Retry == nil || result.Retry.ReasonCode != ReasonCodeNPCSemanticProposalInvalid || len(result.Value.NPCs) != 2 {
|
||||
t.Fatalf("invalid structured result = %#v, %v; want deterministic retry fallback", result, err)
|
||||
}
|
||||
operational := errors.New("provider unavailable")
|
||||
if _, err := newNormalizer(t, &recordingNPCNormalizerClient{err: operational}).Normalize(context.Background(), normalizeRequestWithSource(input, doc)); !errors.Is(err, operational) || errors.Is(err, contracts.ErrInvalidStructuredOutput) {
|
||||
t.Fatalf("operational completion error = %v, want ordinary error", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeRejectsContextEncodingFailuresWithoutLeakingContent(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
doc.Units[0].Metadata = map[string]any{"invalid": math.NaN()}
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
}}
|
||||
if _, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc)); err == nil || !strings.Contains(err.Error(), "build semantic context") || strings.Contains(err.Error(), doc.Units[0].Text) || len(client.requests) != 0 {
|
||||
t.Fatalf("Normalize() error = %v, calls = %d; want safe preparation error before completion", err, len(client.requests))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeDoesNotAccumulateSafeGroupsAcrossAttempts(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{responses: []string{
|
||||
`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Captain Vale","Unknown"],"canonical_name":"Captain Vale"}]}`,
|
||||
`{"duplicate_groups":[{"members":["Mira","Captain Vale"],"canonical_name":"Captain Vale"}]}`,
|
||||
}}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "Mira Thorn", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 20, EndUnitID: 20}}},
|
||||
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
|
||||
}}
|
||||
first, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || first.Retry == nil || len(first.Value.NPCs) != 2 {
|
||||
t.Fatalf("first Normalize() = %#v, %v; want partial retry fallback", first, err)
|
||||
}
|
||||
second, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || second.Retry != nil {
|
||||
t.Fatalf("second Normalize() = %#v, %v; want accepted independent retry result", second, err)
|
||||
}
|
||||
if got := []string{second.Value.NPCs[0].Name, second.Value.NPCs[1].Name}; !reflect.DeepEqual(got, []string{"Captain Vale", "Mira Thorn"}) {
|
||||
t.Fatalf("second NPCs = %#v, want proposal applied to original merge output", second.Value.NPCs)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNormalizeExcludesIneligibleRecordsFromSemanticGroups(t *testing.T) {
|
||||
client := &recordingNPCNormalizerClient{response: `{"duplicate_groups":[{"members":["Mira","Broken"],"canonical_name":"Mira"}]}`}
|
||||
normalizer := newNormalizer(t, client)
|
||||
doc := semanticDocument()
|
||||
input := dnd.NPCList{NPCs: []dnd.NPC{
|
||||
{Name: "Mira", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 10, EndUnitID: 10}}},
|
||||
{Name: "Broken", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 999, EndUnitID: 999}}},
|
||||
{Name: "Captain Vale", SourceRefs: []source.SourceRef{{SourceID: doc.ID, StartUnitID: 30, EndUnitID: 30}}},
|
||||
}}
|
||||
result, err := normalizer.Normalize(context.Background(), normalizeRequestWithSource(input, doc))
|
||||
if err != nil || result.Retry == nil || len(result.Value.NPCs) != 3 {
|
||||
t.Fatalf("Normalize() = %#v, %v; want unchanged retry fallback", result, err)
|
||||
}
|
||||
if !strings.Contains(result.Retry.Message, "member_unknown") || result.Value.NPCs[1].Name != "Broken" {
|
||||
t.Fatalf("result = %#v, want ineligible record excluded but preserved", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProposalValidationRejectsUnsafeCategories(t *testing.T) {
|
||||
records := []normalizedRecord{
|
||||
{npc: dnd.NPC{Name: "Mira"}, inputIndexes: []int{0}, earliest: 0},
|
||||
{npc: dnd.NPC{Name: "Mira Thorn"}, inputIndexes: []int{1}, earliest: 1},
|
||||
{npc: dnd.NPC{Name: "Captain Vale"}, inputIndexes: []int{2}, earliest: 2},
|
||||
}
|
||||
for _, proposal := range []normalizeProposalGroup{
|
||||
{Members: []string{"Mira"}, CanonicalName: "Mira"},
|
||||
{Members: []string{"Mira", "Mira"}, CanonicalName: "Mira"},
|
||||
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: "Unknown"},
|
||||
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: " "},
|
||||
{Members: []string{" ", "Mira Thorn"}, CanonicalName: "Mira Thorn"},
|
||||
{Members: []string{"Mira", "Mira Thorn"}, CanonicalName: "Captain Vale"},
|
||||
} {
|
||||
assessment := assessProposal(normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{proposal}}, records, []int{0, 1, 2})
|
||||
if assessment.discardedGroups != 1 || len(assessment.safeGroups) != 0 {
|
||||
t.Fatalf("assessment for %#v = %#v, want discarded unsafe group", proposal, assessment)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProposalResolutionUsesOnlyExistingComparisonKeyEquivalences(t *testing.T) {
|
||||
records := []normalizedRecord{
|
||||
{npc: dnd.NPC{Name: "O'Neill"}, inputIndexes: []int{0}, earliest: 0},
|
||||
{npc: dnd.NPC{Name: "Mira Thorn"}, inputIndexes: []int{1}, earliest: 1},
|
||||
}
|
||||
assessment := assessProposal(normalizeProposalResponse{DuplicateGroups: []normalizeProposalGroup{{
|
||||
Members: []string{" O’NEILL ", "Mira Thorn"}, CanonicalName: "Mira Thorn",
|
||||
}}}, records, []int{0, 1})
|
||||
if assessment.discardedGroups != 0 || len(assessment.safeGroups) != 1 || assessment.safeGroups[0].canonical != 1 {
|
||||
t.Fatalf("assessment = %#v, want comparison-key-only resolution", assessment)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRetryWarningLimitReservesExhaustionWarningPosition(t *testing.T) {
|
||||
warnings := make([]contracts.Warning, 0, 25)
|
||||
for index := 0; index < 25; index++ {
|
||||
warnings = append(warnings, contracts.Warning{Scope: "npcs", ReasonCode: "test", Message: "warning"})
|
||||
}
|
||||
bounded := limitWarningsForRetry(warnings)
|
||||
if len(bounded) != 19 || bounded[len(bounded)-1].ReasonCode != ReasonCodeNPCNormalizationWarningsOmitted || !strings.Contains(bounded[len(bounded)-1].Message, "7 additional") {
|
||||
t.Fatalf("retry warnings = %#v, want 18 warnings plus accurate omission summary", bounded)
|
||||
}
|
||||
}
|
||||
|
||||
func semanticDocument() *source.SourceDocument {
|
||||
return &source.SourceDocument{ID: "session", Units: []source.SourceUnit{
|
||||
{ID: 10, Kind: "speech", Text: "Mira speaks"},
|
||||
{ID: 15, Kind: "narration", Text: "surrounding context"},
|
||||
{ID: 20, Kind: "speech", Text: "Mira Thorn replies"},
|
||||
{ID: 30, Kind: "speech", Text: "Captain Vale watches"},
|
||||
}}
|
||||
}
|
||||
|
||||
func cloneNPCList(input dnd.NPCList) dnd.NPCList {
|
||||
output := dnd.NPCList{NPCs: make([]dnd.NPC, len(input.NPCs))}
|
||||
for index, npc := range input.NPCs {
|
||||
output.NPCs[index] = cloneNPC(npc)
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
func containsString(values []string, want string) bool {
|
||||
for _, value := range values {
|
||||
if value == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
@@ -95,6 +95,7 @@ func registerPromptAssets(assets *llm.AssetRegistry) error {
|
||||
{name: "scenes prompt assets", register: func() error { return scenes.RegisterPromptAssets(assets) }},
|
||||
{name: "spells prompt assets", register: func() error { return spellextract.RegisterPromptAssets(assets) }},
|
||||
{name: "npcs prompt assets", register: func() error { return npcextract.RegisterPromptAssets(assets) }},
|
||||
{name: "npc normalization prompt assets", register: func() error { return npcnormalize.RegisterPromptAssets(assets) }},
|
||||
{name: "combat turns prompt assets", register: func() error { return combatextract.RegisterPromptAssets(assets) }},
|
||||
{name: "item events prompt assets", register: func() error { return itemeventextract.RegisterPromptAssets(assets) }},
|
||||
{name: "npc interactions prompt assets", register: func() error { return interactionextract.RegisterPromptAssets(assets) }},
|
||||
|
||||
@@ -32,6 +32,20 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
|
||||
if err := Register(registries, assets); err != nil {
|
||||
t.Fatalf("Register() error = %v, want nil", err)
|
||||
}
|
||||
promptFS, err := assets.PromptFS()
|
||||
if err != nil {
|
||||
t.Fatalf("PromptFS() error = %v", err)
|
||||
}
|
||||
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
|
||||
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
|
||||
}
|
||||
schemaFS, err := assets.SchemaFS()
|
||||
if err != nil {
|
||||
t.Fatalf("SchemaFS() error = %v", err)
|
||||
}
|
||||
if _, err := fs.ReadFile(schemaFS, "dnd_npcs_normalize_llm.v1.json"); err != nil {
|
||||
t.Fatalf("normalization schema asset = %v, want registered private schema", err)
|
||||
}
|
||||
assertContainsKeys(t, "chunkers", registries.Chunkers.RegisteredKeys(), []string{"dnd/scenes"})
|
||||
assertContainsKeys(t, "extractors", registries.Extractors.RegisteredKeys(), []string{"dnd/spells", npcextract.Key, combatextract.Key, itemeventextract.Key, interactionextract.Key, scenedescriptionextract.Key})
|
||||
assertContainsKeys(t, "normalizers", registries.Normalizers.RegisteredKeys(), []string{spellnormalize.Key, npcnormalize.Key, combatnormalize.Key, itemeventnormalize.Key, interactionnormalize.Key, scenedescriptionnormalize.Key, pipeline.DefaultNormalizeModule})
|
||||
|
||||
@@ -24,6 +24,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
sceneextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/scenedescriptions"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/spells"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
)
|
||||
|
||||
func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T) {
|
||||
@@ -47,6 +48,9 @@ func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T
|
||||
}
|
||||
for name, value := range map[string]string{
|
||||
"extract:npcs:dnd/npcs:mapping_policy": "dnd.npcs.extract_mapping.v2",
|
||||
"normalize:npcs:dnd/npcs:identity_policy": "dnd.npcs.identity.v1",
|
||||
"normalize:npcs:dnd/npcs:normalization_policy": "dnd.npcs.normalize.v3",
|
||||
"normalize:npcs:dnd/npcs:semantic_context_policy": "dnd.npcs.semantic_context.v1:2",
|
||||
"extract:spells:dnd/spells:mapping_policy": "dnd.spells.extract_mapping.v2",
|
||||
"extract:combat:dnd/combat-turns:scene_gate_policy": "dnd.combat_turns.scene_gate.v1",
|
||||
} {
|
||||
@@ -55,6 +59,8 @@ func TestNPCOutputGroundsSpellAndCombatConsumersThroughOneOperation(t *testing.T
|
||||
for _, name := range []string{
|
||||
"extract:npcs:dnd/npcs:prompt",
|
||||
"extract:npcs:dnd/npcs:response_schema",
|
||||
"normalize:npcs:dnd/npcs:prompt",
|
||||
"normalize:npcs:dnd/npcs:response_schema",
|
||||
"extract:spells:dnd/spells:prompt",
|
||||
"extract:spells:dnd/spells:response_schema",
|
||||
"extract:spells:dnd/spells:npc_registry",
|
||||
@@ -455,6 +461,8 @@ func (client *groundedDNDLLMClient) CompleteStructured(ctx context.Context, requ
|
||||
"name": "Hooded Guard", "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}},
|
||||
},
|
||||
}}
|
||||
case npcnormalize.PromptID:
|
||||
payload = map[string]any{"duplicate_groups": []any{}}
|
||||
case sceneextract.PromptID:
|
||||
kind := client.sceneKind
|
||||
if kind == "" {
|
||||
|
||||
@@ -5,6 +5,8 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
|
||||
@@ -16,6 +18,8 @@ import (
|
||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||
interactionextract "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcinteractions"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
)
|
||||
|
||||
func TestNPCInteractionPipelineUsesAcceptedRegistryAndCurrentEvidence(t *testing.T) {
|
||||
@@ -70,6 +74,59 @@ func TestNPCInteractionPipelineUsesAcceptedRegistryAndCurrentEvidence(t *testing
|
||||
}
|
||||
}
|
||||
|
||||
func TestSemanticNPCNormalizationCrossesOrderedRegistryHandoff(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
cfg := loadNPCInteractionPipelineConfig(t)
|
||||
profile := cfg.Pipelines["dnd-npc-interactions-fixture"]
|
||||
profile.Chunk = pipeline.ModuleBinding{Module: "generic", Options: map[string]any{"max_units": 1}}
|
||||
cfg.Pipelines["dnd-npc-interactions-fixture"] = profile
|
||||
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-npc-interactions-fixture", Catalog: moduleCatalog(registries)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
resolved, warnings, err := pipeline.MaterializeReferences(effective.ResolvedPipeline, moduleCatalog(registries), pipeline.ReferenceMaterializationOptions{})
|
||||
if err != nil || len(warnings) != 0 {
|
||||
t.Fatalf("MaterializeReferences() error = %v warnings = %#v", err, warnings)
|
||||
}
|
||||
client := &semanticNPCInteractionClient{}
|
||||
raw := []byte(`{"metadata":{"id":"semantic-session","title":"Semantic NPC session"},"segments":[{"id":1,"start":0,"end":1,"speaker":"DM","text":"Mira Thorn enters."},{"id":2,"start":1,"end":2,"speaker":"DM","text":"Mira Thorn, the Greencloak, waves."}]}`)
|
||||
output, err := runPreparedPipeline(t, registries, resolved, client, pipeline.RunInput{RawInput: raw})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if client.requestCount(npcs.PromptID) != 2 || client.requestCount(npcnormalize.PromptID) != 1 {
|
||||
t.Fatalf("prompt requests = %#v, want two extraction calls and one document normalization call", client.requests)
|
||||
}
|
||||
npcOutput := normalizedLane(t, output, "npcs")
|
||||
npcsValue, err := npccodec.New().Decode(npcOutput.Artifact.Content)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if npcOutput.StepID != "identify-npcs" || len(npcsValue.NPCs) != 1 || npcsValue.NPCs[0].Name != "Mira Thorn" || npcsValue.NPCs[0].ID != identity.DeriveID("Mira Thorn") {
|
||||
t.Fatalf("NPC output = %#v / %#v, want canonical ordered producer artifact", npcOutput, npcsValue)
|
||||
}
|
||||
if refs := npcsValue.NPCs[0].SourceRefs; !reflect.DeepEqual(refs, []source.SourceRef{{SourceID: "semantic-session", StartUnitID: 1, EndUnitID: 1}, {SourceID: "semantic-session", StartUnitID: 2, EndUnitID: 2}}) {
|
||||
t.Fatalf("NPC evidence = %#v, want original extraction evidence union", refs)
|
||||
}
|
||||
interactionOutput := normalizedLane(t, output, "interactions")
|
||||
if interactionOutput.StepID != "extract-interactions" {
|
||||
t.Fatalf("interaction output step = %q, want ordered downstream step", interactionOutput.StepID)
|
||||
}
|
||||
registryRequest := client.requestFor(t, interactionextract.PromptID)
|
||||
if got := string(registryRequest.Inputs["npcs"].Content); got != `{"npcs":[{"name":"Mira Thorn"}]}` {
|
||||
t.Fatalf("downstream registry = %s, want one canonical names-only NPC", got)
|
||||
}
|
||||
manifestContent, err := json.Marshal(output.Manifest)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
for _, forbidden := range []string{"npc:sha256:", "Mira Thorn enters.", "Mira Thorn, the Greencloak, waves."} {
|
||||
if strings.Contains(string(manifestContent), forbidden) {
|
||||
t.Fatalf("manifest leaked private identity or transcript content %q: %s", forbidden, manifestContent)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNPCInteractionPipelineSkipsConsumerWhenNPCProducerIsRejected(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
resolved := resolveNPCInteractionPipeline(t, registries)
|
||||
@@ -168,6 +225,8 @@ func (client *npcInteractionLLMClient) CompleteStructured(ctx context.Context, r
|
||||
map[string]any{"name": "Hooded Guard", "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}}},
|
||||
}}
|
||||
}
|
||||
case npcnormalize.PromptID:
|
||||
payload = map[string]any{"duplicate_groups": []any{}}
|
||||
case interactionextract.PromptID:
|
||||
payload = map[string]any{"interactions": []any{
|
||||
map[string]any{"name": "Hooded Guard", "kind": "noncombat_presence", "source_refs": []any{map[string]int{"start_unit_id": 3, "end_unit_id": 3}}},
|
||||
@@ -212,3 +271,59 @@ func (client *npcInteractionLLMClient) requestCount(promptID string) int {
|
||||
}
|
||||
|
||||
var _ contracts.StructuredLLMClient = (*npcInteractionLLMClient)(nil)
|
||||
|
||||
type semanticNPCInteractionClient struct {
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
npcCalls int
|
||||
}
|
||||
|
||||
func (client *semanticNPCInteractionClient) CompleteStructured(_ context.Context, request contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.requests = append(client.requests, cloneStructuredCompletionRequest(request))
|
||||
var payload any
|
||||
switch request.PromptID {
|
||||
case npcs.PromptID:
|
||||
client.npcCalls++
|
||||
name := "Mira Thorn"
|
||||
if client.npcCalls == 2 {
|
||||
name = "Mira Thorn, the Greencloak"
|
||||
}
|
||||
payload = map[string]any{"npcs": []any{map[string]any{"name": name, "source_refs": []any{map[string]int{"start_unit_id": client.npcCalls, "end_unit_id": client.npcCalls}}}}}
|
||||
case npcnormalize.PromptID:
|
||||
payload = map[string]any{"duplicate_groups": []any{map[string]any{"members": []string{"Mira Thorn", "Mira Thorn, the Greencloak"}, "canonical_name": "Mira Thorn"}}}
|
||||
case interactionextract.PromptID:
|
||||
payload = map[string]any{"interactions": []any{map[string]any{"name": "Mira Thorn", "kind": "dialogue", "source_refs": []any{map[string]int{"start_unit_id": 1, "end_unit_id": 1}}}}}
|
||||
default:
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected semantic pipeline prompt %q", request.PromptID)
|
||||
}
|
||||
content, err := json.Marshal(payload)
|
||||
if err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
if err := json.Unmarshal(content, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||
}
|
||||
|
||||
func (client *semanticNPCInteractionClient) requestCount(promptID string) int {
|
||||
count := 0
|
||||
for _, request := range client.requests {
|
||||
if request.PromptID == promptID {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
func (client *semanticNPCInteractionClient) requestFor(t *testing.T, promptID string) contracts.StructuredCompletionRequest {
|
||||
t.Helper()
|
||||
for _, request := range client.requests {
|
||||
if request.PromptID == promptID {
|
||||
return request
|
||||
}
|
||||
}
|
||||
t.Fatalf("requests = %#v, missing %q", client.requests, promptID)
|
||||
return contracts.StructuredCompletionRequest{}
|
||||
}
|
||||
|
||||
var _ contracts.StructuredLLMClient = (*semanticNPCInteractionClient)(nil)
|
||||
|
||||
@@ -14,6 +14,7 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/framework/pipeline"
|
||||
npccodec "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/codec/npcs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/extract/npcs"
|
||||
npcnormalize "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/normalize/npcs"
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/npcs/identity"
|
||||
dndregister "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/register"
|
||||
npcshape "gitea.maximumdirect.net/eric/notarius/internal/modules/dnd/validate/npcs/shape"
|
||||
@@ -95,7 +96,7 @@ func TestRunnerProcessesSeriatimInputWithProductionDNDNPCPipeline(t *testing.T)
|
||||
t.Fatalf("manifest lane = %#v, want NPC production composition", lane)
|
||||
}
|
||||
normalizerMetadata, ok := lane.Metadata["normalizer"].(map[string]any)
|
||||
if !ok || normalizerMetadata["identity_policy"] != identity.Policy || normalizerMetadata["normalization_policy"] != "dnd.npcs.normalize.v2" {
|
||||
if !ok || normalizerMetadata["identity_policy"] != identity.Policy || normalizerMetadata["normalization_policy"] != npcnormalize.NormalizationPolicy || normalizerMetadata["prompt_id"] != npcnormalize.PromptID || normalizerMetadata["response_schema_id"] != npcnormalize.ResponseSchemaID {
|
||||
t.Fatalf("normalizer metadata = %#v, want identity and normalization policies", lane.Metadata)
|
||||
}
|
||||
var npcOutputFile *contracts.OutputFile
|
||||
@@ -108,8 +109,8 @@ func TestRunnerProcessesSeriatimInputWithProductionDNDNPCPipeline(t *testing.T)
|
||||
if npcOutputFile == nil || npcOutputFile.ContentType != npccodec.MediaType {
|
||||
t.Fatalf("output files = %#v, want JSON NPC lane file", output.OutputFiles)
|
||||
}
|
||||
if len(client.requests) != 1 || client.requests[0].PromptID != npcs.PromptID {
|
||||
t.Fatalf("LLM requests = %#v, want one NPC prompt request", client.requests)
|
||||
if len(client.requests) != 2 || client.requests[0].PromptID != npcs.PromptID || client.requests[1].PromptID != npcnormalize.PromptID {
|
||||
t.Fatalf("LLM requests = %#v, want extraction and normalization prompt requests", client.requests)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -170,6 +171,52 @@ func TestProductionNPCPipelineRoutesSemanticCandidatesToDeterministicValidators(
|
||||
}
|
||||
}
|
||||
|
||||
func TestProductionNPCNormalizationRetryUsesFinalSafeProposal(t *testing.T) {
|
||||
registries := productionNPCRegistries(t)
|
||||
baseResponse := npcProductionResponse{NPCs: []npcProductionRecord{
|
||||
{Name: "Mira", SourceRefs: []npcProductionSourceRef{{StartUnitID: 1, EndUnitID: 1}}},
|
||||
{Name: "Mira Thorn", SourceRefs: []npcProductionSourceRef{{StartUnitID: 2, EndUnitID: 2}}},
|
||||
{Name: "Hooded Guard", SourceRefs: []npcProductionSourceRef{{StartUnitID: 3, EndUnitID: 3}}},
|
||||
}}
|
||||
partial := []byte(`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"},{"members":["Hooded Guard","Unknown"],"canonical_name":"Hooded Guard"}]}`)
|
||||
safe := []byte(`{"duplicate_groups":[{"members":["Mira","Mira Thorn"],"canonical_name":"Mira Thorn"}]}`)
|
||||
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
retries int
|
||||
responses [][]byte
|
||||
wantCalls int
|
||||
wantExhaustion bool
|
||||
}{
|
||||
{name: "default retry budget", responses: [][]byte{partial}, wantCalls: 1, wantExhaustion: true},
|
||||
{name: "later complete proposal", retries: 1, responses: [][]byte{partial, safe}, wantCalls: 2},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
cfg := loadNPCPipelineConfig(t)
|
||||
profile := cfg.Pipelines["dnd-npcs-fixture"]
|
||||
lane := profile.Artifacts["npcs"]
|
||||
lane.Normalize.Retries = test.retries
|
||||
profile.Artifacts["npcs"] = lane
|
||||
cfg.Pipelines["dnd-npcs-fixture"] = profile
|
||||
effective, err := cfg.Resolve(config.ResolveInput{PipelineID: "dnd-npcs-fixture", Catalog: moduleCatalog(registries)})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
client := &fakeNPCProductionLLMClient{response: baseResponse, normalizeResponses: test.responses}
|
||||
output, err := runPreparedPipeline(t, registries, effective.ResolvedPipeline, client, pipeline.RunInput{RawInput: readNPCFixture(t)})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if client.requestCount(npcnormalize.PromptID) != test.wantCalls || len(output.NormalizeOutputs) != 1 {
|
||||
t.Fatalf("normalization calls = %d output = %#v, want %d accepted result", client.requestCount(npcnormalize.PromptID), output.NormalizeOutputs, test.wantCalls)
|
||||
}
|
||||
if got := hasWarningReason(output.Warnings, npcnormalize.ReasonCodeNPCSemanticReconciliationExhausted); got != test.wantExhaustion {
|
||||
t.Fatalf("warnings = %#v, exhaustion = %t, want %t", output.Warnings, got, test.wantExhaustion)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type npcProductionResponse struct {
|
||||
NPCs []npcProductionRecord `json:"npcs"`
|
||||
}
|
||||
@@ -187,14 +234,17 @@ type npcProductionSourceRef struct {
|
||||
type fakeNPCProductionLLMClient struct {
|
||||
response npcProductionResponse
|
||||
rawResponses [][]byte
|
||||
normalizeResponses [][]byte
|
||||
requests []contracts.StructuredCompletionRequest
|
||||
}
|
||||
|
||||
func (client *fakeNPCProductionLLMClient) CompleteStructured(_ context.Context, req contracts.StructuredCompletionRequest, out any) (contracts.StructuredCompletionResponse, error) {
|
||||
client.requests = append(client.requests, req)
|
||||
var content []byte
|
||||
switch req.PromptID {
|
||||
case npcs.PromptID:
|
||||
if client.rawResponses != nil {
|
||||
index := len(client.requests) - 1
|
||||
index := client.requestCount(npcs.PromptID) - 1
|
||||
if index >= len(client.rawResponses) {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("missing fake NPC response %d", index)
|
||||
}
|
||||
@@ -206,12 +256,34 @@ func (client *fakeNPCProductionLLMClient) CompleteStructured(_ context.Context,
|
||||
return contracts.StructuredCompletionResponse{}, err
|
||||
}
|
||||
}
|
||||
case npcnormalize.PromptID:
|
||||
content = []byte(`{"duplicate_groups":[]}`)
|
||||
if client.normalizeResponses != nil {
|
||||
index := client.requestCount(npcnormalize.PromptID) - 1
|
||||
if index >= len(client.normalizeResponses) {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("missing fake NPC normalization response %d", index)
|
||||
}
|
||||
content = append([]byte(nil), client.normalizeResponses[index]...)
|
||||
}
|
||||
default:
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("unexpected fake NPC prompt %q", req.PromptID)
|
||||
}
|
||||
if err := json.Unmarshal(content, out); err != nil {
|
||||
return contracts.StructuredCompletionResponse{}, fmt.Errorf("populate NPC structured target: %w", err)
|
||||
}
|
||||
return contracts.StructuredCompletionResponse{Content: content}, nil
|
||||
}
|
||||
|
||||
func (client *fakeNPCProductionLLMClient) requestCount(promptID string) int {
|
||||
count := 0
|
||||
for _, request := range client.requests {
|
||||
if request.PromptID == promptID {
|
||||
count++
|
||||
}
|
||||
}
|
||||
return count
|
||||
}
|
||||
|
||||
func productionNPCRegistries(t *testing.T) pipeline.Registries {
|
||||
t.Helper()
|
||||
registries := pipeline.Registries{
|
||||
@@ -283,3 +355,12 @@ func hasNPCWarning(warnings []contracts.Warning, reason string) bool {
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func hasWarningReason(warnings []contracts.Warning, reason string) bool {
|
||||
for _, warning := range warnings {
|
||||
if warning.ReasonCode == reason {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user