Add D&D extraction fallback profile

This commit is contained in:
2026-08-03 16:39:45 +00:00
parent b05634ee86
commit a3bd0c1867
16 changed files with 145 additions and 16 deletions

View File

@@ -13,7 +13,7 @@ owns prompt, profile, and schema file contracts.
Notarius relies on the root `promptkit` package to: Notarius relies on the root `promptkit` package to:
- construct an `Engine` with filesystem-backed prompt, schema, and optional - construct an `Engine` with filesystem-backed prompt, schema, and optional
profile sources; operator and application-fallback profile sources;
- prepare one frozen execution from a `RunRequest` with named inline artifacts, - prepare one frozen execution from a `RunRequest` with named inline artifacts,
variables, a direct session ID, prompt identity, and profile selection, then variables, a direct session ID, prompt identity, and profile selection, then
record credential-redacted details and run that exact execution; record credential-redacted details and run that exact execution;
@@ -69,6 +69,8 @@ error type, and leaves retries to the calling pipeline stage.
module assets, maps its transport-neutral completion contract, prepares and module assets, maps its transport-neutral completion contract, prepares and
executes requests, validates output, records provenance, captures debug executes requests, validates output, records provenance, captures debug
material, redacts errors, and preserves timeout ownership. material, redacts errors, and preserves timeout ownership.
[D&D Module Internals](../internal/dnd.md) owns the embedded
`dnd-extraction` fallback profile and the maintained D&D prompt defaults.
[Configuration](../config.md#promptkit-profiles) defines how a Notarius [Configuration](../config.md#promptkit-profiles) defines how a Notarius
configuration selects one PromptKit profile source and optionally registers configuration selects one PromptKit profile source and optionally registers
the conventional local backend. the conventional local backend.

View File

@@ -22,10 +22,11 @@ does not repeat their JSON shapes or schemas.
## Family Composition ## Family Composition
The D&D registrar registers the familys artifact codecs, extractors, typed The D&D registrar registers the familys artifact codecs, extractors, typed
append-order mergers, normalizers, validators, prompt assets, and default append-order mergers, normalizers, validators, prompt assets, fallback LLM
validator chains. Each extractor and normalizer has a stable module spec, profile asset, and default validator chains. Each extractor and normalizer has
strict option decoding, and a typed builder. Configuration remains the a stable module spec, strict option decoding, and a typed builder.
canonical owner of the exact keys and validator order. Configuration remains the canonical owner of the exact keys and validator
order.
Private structured-LLM response schemas are deliberately minimal. They reject Private structured-LLM response schemas are deliberately minimal. They reject
invalid JSON structure, missing required fields, incompatible types, and invalid JSON structure, missing required fields, incompatible types, and
@@ -41,6 +42,13 @@ reference, and transcript assets instead of copying their text into individual
modules. A manifests declared sequence, including cache-control placement, is modules. A manifests declared sequence, including cache-control placement, is
part of the prompt behavior. part of the prompt behavior.
Every maintained D&D LLM prompt selects `dnd-extraction` as its default
profile. The D&D registrar embeds that fallback profile with the maintained
OpenRouter model, timeout, and service-tier policy. An operator may provide a
complete profile with the same ID through the configured PromptKit source; that
definition replaces the fallback rather than merging with it. The fallback
leaves reasoning and optional sampling controls unspecified.
All extraction prompts share this four-message rendered prefix: the system All extraction prompts share this four-message rendered prefix: the system
message without cache control, the identity message without cache control, the message without cache control, the identity message without cache control, the
campaign-reference message with ephemeral cache control, and the chunk campaign-reference message with ephemeral cache control, and the chunk

View File

@@ -66,9 +66,11 @@ profile-source construction to apply the configured profile directory or file,
the optional registered fallback profile assets, and the optional conventional the optional registered fallback profile assets, and the optional conventional
`local` backend. Preflight therefore resolves the same profile sources and `local` backend. Preflight therefore resolves the same profile sources and
backend membership as runtime without performing generation. Fallback assets backend membership as runtime without performing generation. Fallback assets
are mounted only when at least one source is registered. PromptKit owns source are mounted only when at least one source is registered. The production D&D
precedence and profile parsing: an operator-provided matching profile takes registrar contributes its `dnd-extraction` fallback, and the maintained D&D
precedence over a fallback profile without Notarius merging either document. prompts select that logical ID by default. PromptKit owns source precedence and
profile parsing: an operator-provided matching profile takes precedence over a
fallback profile without Notarius merging either document.
When the registration is absent, a profile selecting `backend: local` fails When the registration is absent, a profile selecting `backend: local` fails
inspection instead of falling back to a built-in or endpoint-only target. inspection instead of falling back to a built-in or endpoint-only target.

View File

@@ -201,6 +201,76 @@ func TestDefaultCLICompositionValidatesRepresentativeConfiguration(t *testing.T)
} }
} }
func TestProductionAssetsResolveDNDExtractionProfile(t *testing.T) {
components := productionTestComponents(t)
newEngine := func(profileFile string) (*promptkit.Engine, error) {
t.Helper()
options, err := components.assets.PromptKitOptions()
if err != nil {
return nil, err
}
if profileFile != "" {
options = append(options, promptkit.WithProfileFile(profileFile))
}
return promptkit.NewEngine(promptkit.Config{}, options...)
}
t.Run("fallback", func(t *testing.T) {
engine, err := newEngine("")
if err != nil {
t.Fatal(err)
}
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
if err != nil {
t.Fatalf("InspectProfile() error = %v, want fallback profile", err)
}
params := inspection.EffectiveModelParams
if params.BackendID != "openrouter" || params.Model != "openai/gpt-5.6-luna" || params.TimeoutSeconds != 240 || params.ServiceTier != "flex" {
t.Fatalf("fallback profile parameters = %#v", params)
}
if params.ReasoningEffort != "" || params.Temperature != 0 || params.MaxTokens != 0 || params.TopP != 0 {
t.Fatalf("fallback profile selected optional provider controls: %#v", params)
}
})
t.Run("valid operator profile wins", func(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
if err := os.WriteFile(profilePath, []byte(`id: dnd-extraction
endpoint: http://operator.example.test/v1
model: operator-model
timeout_seconds: 75
`), 0o600); err != nil {
t.Fatal(err)
}
engine, err := newEngine(profilePath)
if err != nil {
t.Fatal(err)
}
inspection, err := engine.InspectProfile(context.Background(), "dnd-extraction")
if err != nil {
t.Fatalf("InspectProfile() error = %v, want operator profile", err)
}
params := inspection.EffectiveModelParams
if params.BackendID != "" || params.Endpoint != "http://operator.example.test/v1" || params.Model != "operator-model" || params.TimeoutSeconds != 75 || params.ServiceTier != "" {
t.Fatalf("operator profile parameters = %#v, want complete replacement", params)
}
})
t.Run("invalid operator profile does not fall through", func(t *testing.T) {
profilePath := filepath.Join(t.TempDir(), "profiles.yaml")
if err := os.WriteFile(profilePath, []byte("id: dnd-extraction\nendpoint: http://operator.example.test/v1\nmodel: operator-model\nunknown: value\n"), 0o600); err != nil {
t.Fatal(err)
}
engine, err := newEngine(profilePath)
if err == nil {
_, err = engine.InspectProfile(context.Background(), "dnd-extraction")
}
if err == nil {
t.Fatal("operator profile error = nil, want failure instead of fallback")
}
})
}
func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) { func TestProductionPromptAssetsPrepareWithoutProviderCredentials(t *testing.T) {
components := productionTestComponents(t) components := productionTestComponents(t)
cfg := config.Default() cfg := config.Default()

View File

@@ -1,6 +1,6 @@
id: dnd.scenes id: dnd.scenes
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.combat_turns id: dnd.combat_turns
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.item_events id: dnd.item_events
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.npc_interactions id: dnd.npc_interactions
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.npcs id: dnd.npcs
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.scene_descriptions id: dnd.scene_descriptions
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.spells id: dnd.spells
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: transcript - name: transcript
required: true required: true

View File

@@ -1,6 +1,6 @@
id: dnd.npcs.normalize id: dnd.npcs.normalize
version: "v1" version: "v1"
default_profile: gemini-2-flash default_profile: dnd-extraction
inputs: inputs:
- name: candidates - name: candidates
required: true required: true

View File

@@ -0,0 +1,5 @@
id: dnd-extraction
backend: openrouter
model: openai/gpt-5.6-luna
timeout_seconds: 240
service_tier: flex

View File

@@ -0,0 +1,14 @@
package register
import (
"embed"
"gitea.maximumdirect.net/eric/notarius/internal/framework/llm"
)
//go:embed assets/profiles/*.yaml
var embeddedProfileAssets embed.FS
func registerFallbackProfiles(assets *llm.AssetRegistry) error {
return assets.RegisterFallbackProfileFS(embeddedProfileAssets, "assets/profiles")
}

View File

@@ -30,6 +30,9 @@ func Register(registries pipeline.Registries, assets *llm.AssetRegistry) error {
if err := registerPromptAssets(assets); err != nil { if err := registerPromptAssets(assets); err != nil {
return err return err
} }
if err := registerFallbackProfiles(assets); err != nil {
return err
}
return registerDefaultChains(registries.ValidatorChains) return registerDefaultChains(registries.ValidatorChains)
} }

View File

@@ -39,6 +39,31 @@ func TestRegisterAddsDNDFamily(t *testing.T) {
if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil { if _, err := fs.ReadFile(promptFS, "dnd.npcs.normalize/dnd.npcs.normalize.yaml"); err != nil {
t.Fatalf("normalization prompt asset = %v, want registered private prompt", err) t.Fatalf("normalization prompt asset = %v, want registered private prompt", err)
} }
for _, name := range []string{
"dnd.scenes/dnd.scenes.yaml",
"dnd.spells/dnd.spells.yaml",
"dnd.npcs/dnd.npcs.yaml",
"dnd.combat_turns/dnd.combat_turns.yaml",
"dnd.item_events/dnd.item_events.yaml",
"dnd.npc_interactions/dnd.npc_interactions.yaml",
"dnd.scene_descriptions/dnd.scene_descriptions.yaml",
"dnd.npcs.normalize/dnd.npcs.normalize.yaml",
} {
content, err := fs.ReadFile(promptFS, name)
if err != nil {
t.Fatalf("read prompt asset %q: %v", name, err)
}
if !strings.Contains(string(content), "default_profile: dnd-extraction") {
t.Fatalf("prompt asset %q does not select dnd-extraction", name)
}
}
fallbackFS, err := assets.FallbackProfileFS()
if err != nil {
t.Fatalf("FallbackProfileFS() error = %v", err)
}
if _, err := fs.ReadFile(fallbackFS, "dnd-extraction.yaml"); err != nil {
t.Fatalf("fallback profile asset = %v, want registered D&D profile", err)
}
schemaFS, err := assets.SchemaFS() schemaFS, err := assets.SchemaFS()
if err != nil { if err != nil {
t.Fatalf("SchemaFS() error = %v", err) t.Fatalf("SchemaFS() error = %v", err)