Add roadmap and implementation plan for backend concurrency scaling

This commit is contained in:
2026-07-29 20:42:45 +00:00
parent dc39562ff7
commit ffe6d261a9
3 changed files with 1067 additions and 35 deletions

View File

@@ -33,41 +33,9 @@ consumers.
## Ideas
### Backend-specific concurrency management
Extend the
[LLM backend registry](../consumers/pkg-promptkit.md#register-a-custom-backend)
with optional per-backend concurrency limits and bounded, buffered admission
queues. Promptkit could then route simultaneous
generation requests according to backend capacity while containing accidental
runaway submission. Downstream consumers would continue invoking synchronous
`Run` calls, including concurrently from multiple goroutines, and each
admitted call would wait for and return its ordinary result.
- Scope limits to an engine instance rather than hidden process-global state.
- Give different backend IDs independent capacity pools. A profile endpoint
override would remain part of its selected backend's pool.
- Configure active concurrency and waiting capacity separately. Concurrency
protects the backend, while queue capacity protects the process from
admitting an unbounded backlog.
- Give queue capacity a generous, configurable bounded default intended as a
safety ceiling for bugs or unintended loops rather than a routine
application constraint. Select an exact default during implementation
planning and measurement.
- Reject a call with a recognizable capacity error when its backend queue is
full rather than allowing it to wait outside the bounded queue.
- Admit requests before expensive preparation and artifact copying where
practical so queued work remains lightweight.
- Apply a limit to each actual generation request, including repair attempts,
without unnecessarily serializing prompt preparation.
- Make queued and active waits respect caller cancellation and deadlines.
- Treat concurrency as backend policy rather than a profile-level model
setting.
- Keep the queue ephemeral and in-process, with no survival guarantee across
engine or process shutdown.
- Preserve the existing execution model as far as practical. Durable jobs,
polling, priorities, application worker lifecycle, retries, and
cross-process coordination would be separate future capabilities.
No ideas are currently cataloged. Backend-specific concurrency management has
been selected for active planning in the
[focused concurrency roadmap](concurrency.md).
## Entry Format