Wire backend capacity into engine runtime

This commit is contained in:
2026-07-29 21:11:08 +00:00
parent 861da355d8
commit d2c4051dd0
5 changed files with 720 additions and 15 deletions

View File

@@ -561,10 +561,13 @@ type StructuredOutputJSONSpec struct {
// LLMClient executes rendered prompts for [Engine.Run].
//
// Generate may be called concurrently. It must honor context cancellation to
// make Run responsive to cancellation. The request and all nested maps,
// slices, and pointers are client-owned copies and may be mutated or retained
// without affecting engine state.
// Generate is scheduled according to the resolved backend's capacity policy.
// It may still be called concurrently for different backend pools or unlimited
// backends. Cancellation while waiting for capacity can prevent Generate from
// being called. Once invoked, it must honor context cancellation to make Run
// responsive to cancellation. The request and all nested maps, slices, and
// pointers are client-owned copies and may be mutated or retained without
// affecting engine state.
//
// Generate receives rendered messages and may receive a direct API key. A
// client must protect those values and any raw output in its logging, storage,