Wire backend capacity into engine runtime
This commit is contained in:
11
types.go
11
types.go
@@ -561,10 +561,13 @@ type StructuredOutputJSONSpec struct {
|
||||
|
||||
// LLMClient executes rendered prompts for [Engine.Run].
|
||||
//
|
||||
// Generate may be called concurrently. It must honor context cancellation to
|
||||
// make Run responsive to cancellation. The request and all nested maps,
|
||||
// slices, and pointers are client-owned copies and may be mutated or retained
|
||||
// without affecting engine state.
|
||||
// Generate is scheduled according to the resolved backend's capacity policy.
|
||||
// It may still be called concurrently for different backend pools or unlimited
|
||||
// backends. Cancellation while waiting for capacity can prevent Generate from
|
||||
// being called. Once invoked, it must honor context cancellation to make Run
|
||||
// responsive to cancellation. The request and all nested maps, slices, and
|
||||
// pointers are client-owned copies and may be mutated or retained without
|
||||
// affecting engine state.
|
||||
//
|
||||
// Generate receives rendered messages and may receive a direct API key. A
|
||||
// client must protect those values and any raw output in its logging, storage,
|
||||
|
||||
Reference in New Issue
Block a user