Address backend implementation review findings
This commit is contained in:
@@ -35,9 +35,10 @@ consumers.
|
||||
|
||||
### Backend-specific concurrency management
|
||||
|
||||
Extend the selected
|
||||
[LLM backend registry](backends.md) with optional per-backend concurrency limits
|
||||
and bounded, buffered admission queues. Promptkit could then route simultaneous
|
||||
Extend the
|
||||
[LLM backend registry](../consumers/pkg-promptkit.md#register-a-custom-backend)
|
||||
with optional per-backend concurrency limits and bounded, buffered admission
|
||||
queues. Promptkit could then route simultaneous
|
||||
generation requests according to backend capacity while containing accidental
|
||||
runaway submission. Downstream consumers would continue invoking synchronous
|
||||
`Run` calls, including concurrently from multiple goroutines, and each
|
||||
|
||||
Reference in New Issue
Block a user