Add roadmap and implementation plan for backend concurrency scaling
This commit is contained in:
@@ -33,41 +33,9 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
### Backend-specific concurrency management
|
||||
|
||||
Extend the
|
||||
[LLM backend registry](../consumers/pkg-promptkit.md#register-a-custom-backend)
|
||||
with optional per-backend concurrency limits and bounded, buffered admission
|
||||
queues. Promptkit could then route simultaneous
|
||||
generation requests according to backend capacity while containing accidental
|
||||
runaway submission. Downstream consumers would continue invoking synchronous
|
||||
`Run` calls, including concurrently from multiple goroutines, and each
|
||||
admitted call would wait for and return its ordinary result.
|
||||
|
||||
- Scope limits to an engine instance rather than hidden process-global state.
|
||||
- Give different backend IDs independent capacity pools. A profile endpoint
|
||||
override would remain part of its selected backend's pool.
|
||||
- Configure active concurrency and waiting capacity separately. Concurrency
|
||||
protects the backend, while queue capacity protects the process from
|
||||
admitting an unbounded backlog.
|
||||
- Give queue capacity a generous, configurable bounded default intended as a
|
||||
safety ceiling for bugs or unintended loops rather than a routine
|
||||
application constraint. Select an exact default during implementation
|
||||
planning and measurement.
|
||||
- Reject a call with a recognizable capacity error when its backend queue is
|
||||
full rather than allowing it to wait outside the bounded queue.
|
||||
- Admit requests before expensive preparation and artifact copying where
|
||||
practical so queued work remains lightweight.
|
||||
- Apply a limit to each actual generation request, including repair attempts,
|
||||
without unnecessarily serializing prompt preparation.
|
||||
- Make queued and active waits respect caller cancellation and deadlines.
|
||||
- Treat concurrency as backend policy rather than a profile-level model
|
||||
setting.
|
||||
- Keep the queue ephemeral and in-process, with no survival guarantee across
|
||||
engine or process shutdown.
|
||||
- Preserve the existing execution model as far as practical. Durable jobs,
|
||||
polling, priorities, application worker lifecycle, retries, and
|
||||
cross-process coordination would be separate future capabilities.
|
||||
No ideas are currently cataloged. Backend-specific concurrency management has
|
||||
been selected for active planning in the
|
||||
[focused concurrency roadmap](concurrency.md).
|
||||
|
||||
## Entry Format
|
||||
|
||||
|
||||
Reference in New Issue
Block a user