Finish implementation of the scriptorium migration
This commit is contained in:
@@ -1,295 +1,33 @@
|
||||
# Scriptorium LLM Runtime Roadmap
|
||||
|
||||
This roadmap describes the target state for replacing Notarius' local
|
||||
OpenAI-compatible LLM adapter with an integration based on
|
||||
`gitea.maximumdirect.net/eric/scriptorium`.
|
||||
|
||||
The upgrade is worthwhile only if it preserves Notarius' architectural
|
||||
boundaries:
|
||||
|
||||
- modules own prompt intent, prompt variables, response schemas, validation, and
|
||||
domain interpretation;
|
||||
- provider plumbing stays behind framework LLM contracts;
|
||||
- diagnostics and manifests remain Notarius-owned and secret-free;
|
||||
- raw API keys are resolved operationally and must not be stored in config,
|
||||
diagnostics, manifests, examples, or prompt/profile assets;
|
||||
- Scriptorium types do not leak into chunk, extract, or normalize module
|
||||
contracts unless explicitly chosen as a future public contract.
|
||||
|
||||
## Motivation
|
||||
|
||||
Notarius currently has a local structured-output OpenAI-compatible adapter. That
|
||||
adapter is intentionally narrow, but it makes Notarius responsible for provider
|
||||
request shape, profile support, structured-output retries, model-specific
|
||||
extensions, and future cache-related request fields.
|
||||
|
||||
Scriptorium adds capabilities that are directly useful for Notarius:
|
||||
|
||||
- built-in profiles for many providers and model families, including OpenRouter
|
||||
and local OpenAI-compatible inference endpoints;
|
||||
- prompt definitions with multiple system and user messages;
|
||||
- per-prompt `session_id` support for sticky provider routing and input cache
|
||||
affinity;
|
||||
- per-message `cache_control` support for providers that understand it;
|
||||
- a simple structured-output path where a prompt supplies a schema and
|
||||
Scriptorium sends it upstream and retries non-compliant responses.
|
||||
|
||||
The most important near-term motivation is input-cache reuse. Existing D&D
|
||||
recap prompts already use a cacheable user message shaped as:
|
||||
|
||||
```text
|
||||
A transcript of a Dungeons & Dragons gameplay session is provided below.
|
||||
|
||||
{{ input "transcript" }}
|
||||
```
|
||||
|
||||
The `transcript` input is the same Seriatim JSON transcript that Notarius
|
||||
already accepts as MVP input, including stable numbered segments. Notarius
|
||||
should align chunk, extract, and normalize LLM calls so this transcript message
|
||||
can be byte-identical across recap generation and structured extraction work.
|
||||
For large transcripts, this can materially reduce provider cost when cached
|
||||
input reads are available.
|
||||
|
||||
## Target State
|
||||
|
||||
Notarius uses Scriptorium as the production LLM execution engine behind the
|
||||
existing module-facing LLM boundary.
|
||||
|
||||
Chunk, extract, and normalize modules continue to receive a Notarius
|
||||
`StructuredLLMClient` through their existing request contracts. They do not
|
||||
construct provider clients, read secrets, or depend directly on Scriptorium
|
||||
request/result types.
|
||||
|
||||
The production CLI constructs a Scriptorium-backed client from the effective
|
||||
Notarius configuration, wraps it in Notarius scheduling and diagnostics policy
|
||||
where needed, and returns Notarius-owned non-secret LLM profile manifest
|
||||
metadata.
|
||||
|
||||
Module-owned prompts are rendered as ordered chat messages, not only as one
|
||||
system message and one user message. Prompt rendering must support:
|
||||
|
||||
- repeated roles;
|
||||
- shared input messages;
|
||||
- module-specific task and instruction messages;
|
||||
- optional cache-control metadata;
|
||||
- a stable session ID for calls that should share provider routing/cache
|
||||
affinity;
|
||||
- structured-output schema metadata owned by the module.
|
||||
|
||||
The shared D&D transcript message should be rendered from the original Seriatim
|
||||
input bytes, not reconstructed from normalized source units. The parsed source
|
||||
document remains the canonical source-reference model for validation and
|
||||
artifact grounding, but the cacheable transcript prompt message should preserve
|
||||
the exact transcript payload supplied to the run.
|
||||
|
||||
Other large text inputs, including references such as campaign glossaries,
|
||||
rosters, previous recaps, and future reference producers, should follow the same
|
||||
input-material model. They should be available to prompt rendering as stable
|
||||
input artifacts that can be placed in cacheable messages without being copied
|
||||
into manifests or diagnostics.
|
||||
|
||||
## Boundary Requirements
|
||||
|
||||
### Provider Plumbing
|
||||
|
||||
Provider-specific request fields, profile expansion, and OpenAI-compatible wire
|
||||
details should be isolated in the framework LLM runtime or in the
|
||||
Scriptorium-backed adapter. Stage modules should request structured completion
|
||||
through Notarius contracts and should not know whether the backing client is the
|
||||
legacy local adapter or Scriptorium.
|
||||
|
||||
### Prompt Ownership
|
||||
|
||||
Modules remain responsible for prompt intent, prompt versioning, response schema
|
||||
selection, and semantic validation. Scriptorium provides the prompt definition,
|
||||
rendering, execution, and structured-output workflow, but it should not become
|
||||
the owner of D&D-specific semantics.
|
||||
|
||||
Shared prompt assets are allowed when they express generic reusable context,
|
||||
such as the D&D transcript input message. Module-specific task prompts should
|
||||
remain owned by the relevant module.
|
||||
|
||||
### Diagnostics And Manifests
|
||||
|
||||
Notarius should continue to decide what appears in diagnostics and run
|
||||
manifests. Scriptorium prepared/run metadata may be useful input, but Notarius
|
||||
must filter it through existing policy:
|
||||
|
||||
- no raw prompt payloads by default;
|
||||
- no raw response schema content in manifests;
|
||||
- no source transcript payloads in manifests;
|
||||
- no API keys or bearer tokens;
|
||||
- enough prompt/profile/schema hashes and IDs to audit a run later.
|
||||
|
||||
### Secret Handling
|
||||
|
||||
Raw API keys should stay out of durable config and prompt/profile assets.
|
||||
Notarius may continue resolving API keys from configured environment-variable
|
||||
names, then pass the secret request-scope to Scriptorium. Redacted diagnostics
|
||||
must continue to prove that resolved secrets are not serialized.
|
||||
|
||||
## Prompt And Cache Strategy
|
||||
|
||||
The desired D&D LLM prompt shape is:
|
||||
|
||||
1. stable shared system message;
|
||||
2. stable shared transcript user message containing the original Seriatim JSON;
|
||||
3. optional stable cacheable context messages, such as previous recap,
|
||||
glossary, roster, or other references;
|
||||
4. module-specific task message;
|
||||
5. module-specific instruction message;
|
||||
6. module-owned structured-output schema, when the module expects JSON.
|
||||
|
||||
The transcript message should be byte-identical whenever the same transcript
|
||||
input bytes are used. Avoid reconstructing JSON from parsed source units because
|
||||
formatting, key order, whitespace, or escaping changes would defeat cache reuse.
|
||||
|
||||
The transcript and large reference messages should be cacheable when the active
|
||||
provider path supports cache-control metadata. Providers that ignore cache
|
||||
controls should still receive a valid prompt.
|
||||
|
||||
The session ID should be stable for all LLM calls that operate on the same
|
||||
session transcript and should be explicitly visible in diagnostics or manifest
|
||||
metadata only as a non-secret identifier.
|
||||
|
||||
Notarius should provide an easy CLI UX for setting the session ID. This lets an
|
||||
external D&D pipeline orchestrator pass the same session ID used by recap or
|
||||
other LLM steps, maximizing provider routing affinity and cached input reuse.
|
||||
|
||||
## Configuration Intent
|
||||
|
||||
Notarius configuration should remain the user-facing source of pipeline
|
||||
composition and operational settings. For LLM execution profiles, Notarius
|
||||
should cut over to Scriptorium profile configuration instead of maintaining a
|
||||
separate Notarius-specific profile schema.
|
||||
|
||||
Notarius should allow configuration to select Scriptorium built-in profiles and
|
||||
to point at Scriptorium profile files or directories. This avoids evolving two
|
||||
nearly identical profile systems and gives users immediate access to the
|
||||
provider and model catalog that motivated the migration.
|
||||
|
||||
Notarius still owns validation, redaction, and manifest provenance for the
|
||||
effective pipeline. The integration should wrap Scriptorium profile loading so
|
||||
that errors are actionable, secrets remain environment-based, and emitted
|
||||
metadata stays non-secret.
|
||||
|
||||
## Compatibility Policy
|
||||
|
||||
This feature should be a hard cutover to Scriptorium-backed prompt execution and
|
||||
profile loading. Backward compatibility with the local Notarius prompt renderer,
|
||||
local LLM profile schema, or local OpenAI-compatible adapter is not a product
|
||||
requirement for this migration.
|
||||
|
||||
The final production runtime should have one documented LLM execution path.
|
||||
Module behavior may still be tested with fake Notarius `StructuredLLMClient`
|
||||
implementations, but production runtime behavior should be Scriptorium-backed.
|
||||
|
||||
## Documentation Outcomes
|
||||
|
||||
When this feature is implemented, current-behavior docs should be updated to
|
||||
describe:
|
||||
|
||||
- the implemented production LLM runtime;
|
||||
- supported LLM profile fields and provider/profile selection behavior;
|
||||
- cacheable transcript prompt behavior, if exposed to users;
|
||||
- session ID behavior;
|
||||
- diagnostics and manifest provenance;
|
||||
- troubleshooting for Scriptorium profile, prompt, schema, and validation
|
||||
failures.
|
||||
|
||||
The OpenAI-compatible integration doc should either be retired, narrowed to the
|
||||
legacy fallback, or reframed as an upstream provider contract delegated through
|
||||
Scriptorium, depending on the final runtime shape.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
The Scriptorium migration does not itself require:
|
||||
|
||||
- a general workflow language;
|
||||
- arbitrary per-stage prompt authoring by end users;
|
||||
- token budgeting or context-window planning;
|
||||
- non-file reference producers;
|
||||
- semantic retrieval over transcript or reference content;
|
||||
- multiple effective LLM profiles in one Notarius run.
|
||||
|
||||
Those remain separate future features.
|
||||
|
||||
## Resolved Design Choices
|
||||
|
||||
### Prompt Asset Ownership And Format
|
||||
|
||||
Production LLM prompts should cut over to Scriptorium-compatible prompt
|
||||
definitions without preserving backward compatibility for the current Notarius
|
||||
system/user prompt renderer.
|
||||
|
||||
Module-owned prompt definitions should keep task prompts and schemas near the
|
||||
module that owns the semantic behavior. Shared prompt messages, such as the D&D
|
||||
transcript input message and shared reference/context messages, should live in a
|
||||
shared prompt asset area and be referenced by module-owned prompt definitions.
|
||||
|
||||
This uses Scriptorium's native strengths: ordered messages, cache-control
|
||||
metadata, session IDs, input helpers, profile selection, and schema-backed
|
||||
execution. It also gives Notarius a direct way to reuse the exact transcript and
|
||||
reference messages across D&D recap, chunking, extraction, and normalization
|
||||
work without each module hand-rolling message assembly.
|
||||
|
||||
### Raw Transcript Preservation Location
|
||||
|
||||
Original input bytes should be preserved as run-scoped input material and made
|
||||
available to prompt rendering without storing them in run manifests. The parsed
|
||||
`SourceDocument` should continue to carry normalized source units for framework
|
||||
logic and source-reference validation.
|
||||
|
||||
The same model should apply to all large text inputs, including references.
|
||||
Glossaries, campaign rosters, previous recaps, and other future reference
|
||||
documents may be large enough to deserve their own cacheable messages. They
|
||||
should be treated as prompt input artifacts rather than being copied into source
|
||||
document metadata.
|
||||
|
||||
This keeps the cacheable transcript message byte-identical to the original
|
||||
Seriatim JSON and avoids source-format leakage into chunk, extract, and
|
||||
normalize contracts. It also reduces the risk that raw inputs or references are
|
||||
accidentally serialized into diagnostics or manifests.
|
||||
|
||||
### Session ID Source
|
||||
|
||||
Notarius should add an explicit session ID concept for LLM calls, with a
|
||||
deterministic default derived from the source document identity when no
|
||||
operator-provided value is configured.
|
||||
|
||||
The CLI should provide an easy way to supply this session ID. This lets an
|
||||
external D&D pipeline orchestrator call Notarius with the same session ID used
|
||||
by recap generation or other LLM steps, maximizing provider routing affinity and
|
||||
cached input savings.
|
||||
|
||||
OpenRouter sticky routing and provider cache behavior are most useful when
|
||||
every related call for the same session shares a stable identifier. An explicit
|
||||
concept makes the behavior auditable and avoids each module inventing its own
|
||||
ID. A deterministic default keeps simple local runs ergonomic.
|
||||
|
||||
### Scriptorium Profile Exposure
|
||||
|
||||
Notarius should cut over to Scriptorium profile configuration instead of
|
||||
maintaining a separate Notarius LLM profile schema. Configuration should be able
|
||||
to select Scriptorium built-in profiles and point at Scriptorium profile files
|
||||
or directories.
|
||||
|
||||
The Scriptorium profile format is already close to the desired Notarius target.
|
||||
Maintaining a separate Notarius profile schema would likely create duplicate
|
||||
configuration that eventually converges back toward Scriptorium's model. A hard
|
||||
cutover avoids that churn and gives users immediate access to Scriptorium's
|
||||
provider and model catalog.
|
||||
|
||||
Notarius still needs to wrap this profile loading with Notarius-owned
|
||||
validation, diagnostics, manifest provenance, and secret-redaction policy.
|
||||
|
||||
### Structured-Output Retry Ownership
|
||||
|
||||
Notarius should use Scriptorium's structured-output execution and retry behavior
|
||||
for production calls, while translating results and errors back into Notarius'
|
||||
`StructuredLLMClient` response and error expectations.
|
||||
|
||||
This avoids duplicating structured-output enforcement in Notarius and lets
|
||||
Scriptorium own provider-specific request and repair mechanics. Notarius still
|
||||
retains module-level validation and source-reference checks after decoded
|
||||
structured output is returned.
|
||||
# Completed Scriptorium LLM Runtime Migration
|
||||
|
||||
The Scriptorium-backed LLM runtime migration is complete. Notarius now uses
|
||||
`gitea.maximumdirect.net/eric/scriptorium` as the production LLM execution
|
||||
engine behind Notarius-owned module contracts, diagnostics, manifest
|
||||
provenance, and secret-handling policy.
|
||||
|
||||
## Implemented Outcomes
|
||||
|
||||
- production LLM execution uses Scriptorium prompt assets and profiles;
|
||||
- chunk, extract, and normalize modules call LLMs through the Notarius
|
||||
`StructuredLLMClient` boundary;
|
||||
- modules pass prompt IDs, prompt versions, profile IDs, session IDs, input
|
||||
materials, and prompt variables instead of locally rendered chat messages;
|
||||
- source transcript input is preserved as run-scoped input material for
|
||||
byte-stable cacheable prompt messages;
|
||||
- Scriptorium profile provenance is recorded in run manifests without raw
|
||||
prompts, schemas, source payloads, references, or secrets.
|
||||
|
||||
## Canonical Docs
|
||||
|
||||
Current behavior is documented in:
|
||||
|
||||
- [Configuration](../config.md)
|
||||
- [CLI Reference](../cli.md)
|
||||
- [Internal LLM Runtime](../internal/llm.md)
|
||||
- [Pipeline Internals](../internal/pipeline.md)
|
||||
- [JSON Output Integration](../integrations/json-output.md)
|
||||
- [Troubleshooting](../troubleshooting.md)
|
||||
|
||||
There is no active Scriptorium-specific roadmap work in this file. Deferred
|
||||
work related to broader LLM, reference, retrieval, context-window, or provider
|
||||
strategy belongs in [Future Roadmap](future.md).
|
||||
|
||||
@@ -489,16 +489,10 @@ func effectiveLLMProfileIDs(resolved pipeline.ResolvedPipeline) []string {
|
||||
seen[id] = struct{}{}
|
||||
}
|
||||
}
|
||||
add(resolved.Input)
|
||||
add(resolved.Chunk)
|
||||
add(resolved.Output)
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
add(lane.Extract)
|
||||
add(lane.Merge)
|
||||
add(lane.Normalize)
|
||||
for _, validator := range lane.Validators {
|
||||
add(validator)
|
||||
}
|
||||
}
|
||||
ids := make([]string, 0, len(seen))
|
||||
for id := range seen {
|
||||
|
||||
@@ -707,6 +707,63 @@ pipelines:
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunConfigValidateIgnoresNonLLMStageScriptoriumProfiles(t *testing.T) {
|
||||
profilePath := writeScriptoriumProfileFile(t, "known", "http://profile.test/v1", "test-model")
|
||||
configPath := writeTestConfig(t, `version: 2
|
||||
scriptorium:
|
||||
profile_file: `+profilePath+`
|
||||
pipelines:
|
||||
example:
|
||||
input:
|
||||
module: fake/input
|
||||
llm_profile: missing-input
|
||||
output:
|
||||
module: json
|
||||
llm_profile: missing-output
|
||||
artifacts:
|
||||
events:
|
||||
extract: fake/extract
|
||||
merge:
|
||||
module: appendorder
|
||||
llm_profile: missing-merge
|
||||
`)
|
||||
var stdout bytes.Buffer
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := RunWithOptions([]string{"config", "validate", "--config", configPath, "--pipeline", "example"}, &stdout, &stderr, Options{
|
||||
Catalog: fakeCatalog(t),
|
||||
})
|
||||
|
||||
if code != 0 {
|
||||
t.Fatalf("RunWithOptions() code = %d, stderr=%q, want success", code, stderr.String())
|
||||
}
|
||||
if strings.Contains(stderr.String(), "missing-") {
|
||||
t.Fatalf("stderr = %q, want non-LLM stage profiles ignored", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestEffectiveLLMProfileIDsUsesLLMCapableStagesOnly(t *testing.T) {
|
||||
resolved := pipeline.ResolvedPipeline{
|
||||
Input: pipeline.ModuleBinding{LLMProfile: "input-profile"},
|
||||
Chunk: pipeline.ModuleBinding{LLMProfile: "chunk-profile"},
|
||||
Output: pipeline.ModuleBinding{LLMProfile: "output-profile"},
|
||||
ArtifactLanes: []pipeline.ResolvedArtifactLane{
|
||||
{
|
||||
Extract: pipeline.ModuleBinding{LLMProfile: "extract-profile"},
|
||||
Merge: pipeline.ModuleBinding{LLMProfile: "merge-profile"},
|
||||
Normalize: pipeline.ModuleBinding{LLMProfile: "normalize-profile"},
|
||||
Validators: []pipeline.ModuleBinding{{LLMProfile: "validator-profile"}},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
got := effectiveLLMProfileIDs(resolved)
|
||||
want := []string{"chunk-profile", "extract-profile", "normalize-profile"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Fatalf("effectiveLLMProfileIDs() = %#v, want %#v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunPipelineSessionIDFlagRecordsExplicitTrimmedValue(t *testing.T) {
|
||||
configPath := writeTestConfig(t, mvpConfigYAML("dnd-session", "dnd/spells"))
|
||||
inputPath := writeSeriatimInput(t)
|
||||
|
||||
@@ -65,16 +65,10 @@ func (c Config) Resolve(input ResolveInput) (EffectiveConfig, error) {
|
||||
}
|
||||
|
||||
func applyLLMProfileOverride(profile *pipeline.PipelineProfile, profileID string) {
|
||||
profile.Input.LLMProfile = profileID
|
||||
profile.Chunk.LLMProfile = profileID
|
||||
profile.Output.LLMProfile = profileID
|
||||
for laneID, lane := range profile.Artifacts {
|
||||
lane.Extract.LLMProfile = profileID
|
||||
lane.Merge.LLMProfile = profileID
|
||||
lane.Normalize.LLMProfile = profileID
|
||||
for i := range lane.Validators {
|
||||
lane.Validators[i].LLMProfile = profileID
|
||||
}
|
||||
profile.Artifacts[laneID] = lane
|
||||
}
|
||||
}
|
||||
|
||||
@@ -156,6 +156,14 @@ func TestResolveDigestChangesWhenEffectiveConfigChanges(t *testing.T) {
|
||||
|
||||
func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
|
||||
cfg := validConfig()
|
||||
profile := cfg.Pipelines["example"]
|
||||
profile.Input.LLMProfile = "input-profile"
|
||||
profile.Output.LLMProfile = "output-profile"
|
||||
lane := profile.Artifacts["events"]
|
||||
lane.Merge.LLMProfile = "merge-profile"
|
||||
lane.Validators[0].LLMProfile = "validator-profile"
|
||||
profile.Artifacts["events"] = lane
|
||||
cfg.Pipelines["example"] = profile
|
||||
|
||||
base, err := cfg.Resolve(ResolveInput{PipelineID: "example", Catalog: fakeCatalog(t)})
|
||||
if err != nil {
|
||||
@@ -173,18 +181,30 @@ func TestResolveLLMProfileOverrideAppliesBeforeDigest(t *testing.T) {
|
||||
if base.ResolvedPipeline.Digest == effective.ResolvedPipeline.Digest {
|
||||
t.Fatalf("expected digest to change after LLM profile override")
|
||||
}
|
||||
for _, binding := range resolvedBindings(effective.ResolvedPipeline) {
|
||||
for _, binding := range llmCapableBindings(effective.ResolvedPipeline) {
|
||||
if binding.LLMProfile != "runtime" {
|
||||
t.Fatalf("binding profile = %q, want runtime", binding.LLMProfile)
|
||||
t.Fatalf("LLM-capable binding profile = %q, want runtime", binding.LLMProfile)
|
||||
}
|
||||
}
|
||||
if effective.ResolvedPipeline.Input.LLMProfile != "input-profile" {
|
||||
t.Fatalf("input profile = %q, want original input-profile", effective.ResolvedPipeline.Input.LLMProfile)
|
||||
}
|
||||
if effective.ResolvedPipeline.Output.LLMProfile != "output-profile" {
|
||||
t.Fatalf("output profile = %q, want original output-profile", effective.ResolvedPipeline.Output.LLMProfile)
|
||||
}
|
||||
eventLane := effective.ResolvedPipeline.ArtifactLanes[0]
|
||||
if eventLane.Merge.LLMProfile != "merge-profile" {
|
||||
t.Fatalf("merge profile = %q, want original merge-profile", eventLane.Merge.LLMProfile)
|
||||
}
|
||||
if len(eventLane.Validators) != 1 || eventLane.Validators[0].LLMProfile != "validator-profile" {
|
||||
t.Fatalf("validator profiles = %#v, want original validator-profile", eventLane.Validators)
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedBindings(resolved pipeline.ResolvedPipeline) []pipeline.ModuleBinding {
|
||||
bindings := []pipeline.ModuleBinding{resolved.Input, resolved.Chunk, resolved.Output}
|
||||
func llmCapableBindings(resolved pipeline.ResolvedPipeline) []pipeline.ModuleBinding {
|
||||
bindings := []pipeline.ModuleBinding{resolved.Chunk}
|
||||
for _, lane := range resolved.ArtifactLanes {
|
||||
bindings = append(bindings, lane.Extract, lane.Merge, lane.Normalize)
|
||||
bindings = append(bindings, lane.Validators...)
|
||||
bindings = append(bindings, lane.Extract, lane.Normalize)
|
||||
}
|
||||
return bindings
|
||||
}
|
||||
|
||||
@@ -8,23 +8,14 @@ import (
|
||||
"gitea.maximumdirect.net/eric/notarius/internal/core/source"
|
||||
)
|
||||
|
||||
type LLMMessage struct {
|
||||
Role string `json:"role"`
|
||||
Content string `json:"content"`
|
||||
}
|
||||
|
||||
type StructuredCompletionRequest struct {
|
||||
StageName string `json:"stage_name"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Inputs LLMInputSet `json:"inputs,omitempty"`
|
||||
Vars map[string]any `json:"vars,omitempty"`
|
||||
Messages []LLMMessage `json:"messages"`
|
||||
Model string `json:"model,omitempty"`
|
||||
ResponseSchemaName string `json:"response_schema_name,omitempty"`
|
||||
ResponseSchema json.RawMessage `json:"response_schema,omitempty"`
|
||||
StageName string `json:"stage_name"`
|
||||
PromptID string `json:"prompt_id,omitempty"`
|
||||
PromptVersion string `json:"prompt_version,omitempty"`
|
||||
ProfileID string `json:"profile_id,omitempty"`
|
||||
SessionID string `json:"session_id,omitempty"`
|
||||
Inputs LLMInputSet `json:"inputs,omitempty"`
|
||||
Vars map[string]any `json:"vars,omitempty"`
|
||||
}
|
||||
|
||||
type StructuredCompletionResponse struct {
|
||||
|
||||
@@ -269,8 +269,9 @@ func (extractor walkingSkeletonExtractor) Extract(ctx context.Context, req contr
|
||||
Call int `json:"call"`
|
||||
}
|
||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: "fake/extract",
|
||||
ResponseSchemaName: "fake_event",
|
||||
StageName: "fake/extract",
|
||||
PromptID: "fake.event",
|
||||
PromptVersion: "v1",
|
||||
}, &response); err != nil {
|
||||
return contracts.ExtractionResult{}, err
|
||||
}
|
||||
@@ -349,8 +350,9 @@ func (normalizer walkingSkeletonNormalizer) Normalize(ctx context.Context, req c
|
||||
Call int `json:"call"`
|
||||
}
|
||||
if _, err := req.LLMClient.CompleteStructured(ctx, contracts.StructuredCompletionRequest{
|
||||
StageName: "fake/normalize",
|
||||
ResponseSchemaName: "fake_normalize",
|
||||
StageName: "fake/normalize",
|
||||
PromptID: "fake.normalize",
|
||||
PromptVersion: "v1",
|
||||
}, &response); err != nil {
|
||||
return contracts.NormalizeResult{}, err
|
||||
}
|
||||
|
||||
@@ -118,9 +118,6 @@ func TestChunkReturnsSceneChunksFromStructuredOutput(t *testing.T) {
|
||||
if req.SessionID != "session-123" || req.ProfileID != "profile-scenes" {
|
||||
t.Fatalf("session/profile = %q/%q, want session-123/profile-scenes", req.SessionID, req.ProfileID)
|
||||
}
|
||||
if len(req.Messages) != 0 || req.ResponseSchemaName != "" || len(req.ResponseSchema) != 0 {
|
||||
t.Fatalf("legacy prompt fields set: messages=%#v schema=%q/%s", req.Messages, req.ResponseSchemaName, req.ResponseSchema)
|
||||
}
|
||||
transcript, ok := req.Inputs["transcript"]
|
||||
if !ok {
|
||||
t.Fatalf("transcript input missing from %#v", req.Inputs)
|
||||
@@ -493,8 +490,18 @@ func (client *fakeScenesLLMClient) CompleteStructured(ctx context.Context, req c
|
||||
}
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Messages = append([]contracts.LLMMessage(nil), req.Messages...)
|
||||
req.ResponseSchema = append(json.RawMessage(nil), req.ResponseSchema...)
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
req.Vars = cloneVars(req.Vars)
|
||||
return req
|
||||
}
|
||||
|
||||
func cloneVars(in map[string]any) map[string]any {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(in))
|
||||
for key, value := range in {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -46,9 +46,6 @@ func TestExtractReturnsSpellCandidateFromStructuredOutput(t *testing.T) {
|
||||
if req.SessionID != "session-123" || req.ProfileID != "profile-spells" {
|
||||
t.Fatalf("session/profile = %q/%q, want session-123/profile-spells", req.SessionID, req.ProfileID)
|
||||
}
|
||||
if len(req.Messages) != 0 || req.ResponseSchemaName != "" || len(req.ResponseSchema) != 0 {
|
||||
t.Fatalf("legacy prompt fields set: messages=%#v schema=%q/%s", req.Messages, req.ResponseSchemaName, req.ResponseSchema)
|
||||
}
|
||||
transcript := req.Inputs["transcript"]
|
||||
if transcript.Name != "transcript" || transcript.MediaType != "application/json" || transcript.Digest != "sha256:transcript" || transcript.OriginURI != "file:///session-alpha.json" {
|
||||
t.Fatalf("transcript metadata = %#v", transcript)
|
||||
@@ -135,8 +132,8 @@ func TestExtractPassesReferencesAsPromptInputs(t *testing.T) {
|
||||
t.Fatalf("LLM calls = %d, want 1", len(client.requests))
|
||||
}
|
||||
request := client.requests[0]
|
||||
if len(request.Messages) != 0 {
|
||||
t.Fatalf("Messages = %#v, want no locally rendered prompt", request.Messages)
|
||||
if request.PromptID != PromptID || request.PromptVersion != SchemaVersion {
|
||||
t.Fatalf("prompt = %q/%q, want %q/%q", request.PromptID, request.PromptVersion, PromptID, SchemaVersion)
|
||||
}
|
||||
if got := string(request.Inputs["roster"].Content); got != "Aria Brightmantle: party cleric" {
|
||||
t.Fatalf("roster input = %q, want reference content", got)
|
||||
@@ -337,8 +334,18 @@ func (client *fakeSpellsLLMClient) CompleteStructured(ctx context.Context, req c
|
||||
}
|
||||
|
||||
func cloneStructuredCompletionRequest(req contracts.StructuredCompletionRequest) contracts.StructuredCompletionRequest {
|
||||
req.Messages = append([]contracts.LLMMessage(nil), req.Messages...)
|
||||
req.ResponseSchema = append(json.RawMessage(nil), req.ResponseSchema...)
|
||||
req.Inputs = req.Inputs.Clone()
|
||||
req.Vars = cloneVars(req.Vars)
|
||||
return req
|
||||
}
|
||||
|
||||
func cloneVars(in map[string]any) map[string]any {
|
||||
if len(in) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make(map[string]any, len(in))
|
||||
for key, value := range in {
|
||||
out[key] = value
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -153,8 +153,8 @@ func TestRunnerPassesRosterAndGlossaryReferencesToDNDSpellsPrompt(t *testing.T)
|
||||
t.Fatalf("LLM calls = %d, want 1", len(llmClient.requests))
|
||||
}
|
||||
request := llmClient.requests[0]
|
||||
if len(request.Messages) != 0 {
|
||||
t.Fatalf("Messages = %#v, want no locally rendered prompt", request.Messages)
|
||||
if request.PromptID != PromptID || request.PromptVersion != SchemaVersion {
|
||||
t.Fatalf("prompt = %q/%q, want %q/%q", request.PromptID, request.PromptVersion, PromptID, SchemaVersion)
|
||||
}
|
||||
if got := string(request.Inputs["roster"].Content); got != "Aria: party cleric\nBorin: fighter" {
|
||||
t.Fatalf("roster input = %q, want reference text", got)
|
||||
@@ -191,8 +191,8 @@ func TestRunnerDoesNotExtractSpellMentionedOnlyInRoster(t *testing.T) {
|
||||
t.Fatalf("LLM calls = %d, want 1", len(llmClient.requests))
|
||||
}
|
||||
request := llmClient.requests[0]
|
||||
if len(request.Messages) != 0 {
|
||||
t.Fatalf("Messages = %#v, want no locally rendered prompt", request.Messages)
|
||||
if request.PromptID != PromptID || request.PromptVersion != SchemaVersion {
|
||||
t.Fatalf("prompt = %q/%q, want %q/%q", request.PromptID, request.PromptVersion, PromptID, SchemaVersion)
|
||||
}
|
||||
if got := string(request.Inputs["roster"].Content); !strings.Contains(got, "Lightning Bolt") {
|
||||
t.Fatalf("roster input = %q, want roster-only spell in reference input", got)
|
||||
|
||||
Reference in New Issue
Block a user