eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare the backend registry implementation roadmap

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-29 16:50:36 +00:00
parent b462153483
commit d0010689f3
4 changed files with 684 additions and 243 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
docs/roadmap/future.md
View File

@@ -33,29 +33,15 @@ consumers.
## Ideas
### Extensible LLM backend registry
Introduce a registry that separates backend-specific connection,
authentication, and limited request defaults from model execution profiles.
Initial support would cover OpenAI-compatible backends and include a small
built-in catalog, potentially starting with OpenRouter. A profile could select
a backend while optionally overriding its default endpoint, and each backend
could name an optional environment variable for its API key without storing
the credential itself. Downstream consumers could register additional,
uniquely named backends, such as OpenAI or unauthenticated local-network
services, but could not replace built-in IDs. Model selection and generation
settings would remain profile concerns, and custom model clients would remain
available for behavior outside the registry's supported protocol.
### Backend-specific concurrency management
Extend the proposed LLM backend registry with optional per-backend concurrency
limits and bounded, buffered admission queues. Promptkit could then route
simultaneous generation requests according to backend capacity while
containing accidental runaway submission. Downstream consumers would continue
invoking synchronous `Run` calls, including concurrently from multiple
goroutines, and each admitted call would wait for and return its ordinary
result.
Extend the selected
[LLM backend registry](backends.md) with optional per-backend concurrency limits
and bounded, buffered admission queues. Promptkit could then route simultaneous
generation requests according to backend capacity while containing accidental
runaway submission. Downstream consumers would continue invoking synchronous
`Run` calls, including concurrently from multiple goroutines, and each
admitted call would wait for and return its ordinary result.
- Scope limits to an engine instance rather than hidden process-global state.
- Give different backend IDs independent capacity pools. A profile endpoint