181 lines
7.7 KiB
Markdown
181 lines
7.7 KiB
Markdown
# Extensible LLM Backend Registry
|
|
|
|
## Purpose
|
|
|
|
This roadmap defines the scope and target end state for formal
|
|
OpenAI-compatible backend support in Promptkit. It establishes the behavioral
|
|
boundary and policy choices for the work.
|
|
|
|
This document is planning material, not a description of current behavior.
|
|
Current public contracts remain owned by Go declarations and GoDoc, framework
|
|
file formats by the [format reference](../formats.md), and outbound HTTP
|
|
behavior by the
|
|
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md).
|
|
|
|
## Motivation
|
|
|
|
Execution profiles currently carry backend connection and authentication
|
|
details alongside model and generation settings. This repeats values such as
|
|
the OpenRouter endpoint and `OPENROUTER_API_KEY` across profiles, leaves the
|
|
built-in catalog implicitly tied to one backend, and gives consumers no
|
|
explicit extension point for naming other OpenAI-compatible services.
|
|
|
|
Promptkit should distinguish:
|
|
|
|
- a **backend**, which identifies reusable connection, authentication, and
|
|
limited backend-wide request defaults; and
|
|
- a **profile**, which selects a model and its reusable generation settings.
|
|
|
|
That distinction should let consumers configure OpenRouter, OpenAI,
|
|
local-network services, or other OpenAI-compatible deployments without
|
|
duplicating backend defaults in every profile or replacing Promptkit's model
|
|
client.
|
|
|
|
## Scope
|
|
|
|
The work introduces an engine-scoped registry of OpenAI-compatible backend
|
|
definitions.
|
|
|
|
Each backend definition will provide:
|
|
|
|
- a stable, non-empty backend ID;
|
|
- a default OpenAI-compatible base endpoint;
|
|
- an optional environment-variable name for its API key; and
|
|
- limited JSON-compatible request defaults when they are genuinely
|
|
backend-wide rather than model-specific.
|
|
|
|
Promptkit will provide a small built-in backend catalog, initially containing
|
|
OpenRouter. Downstream consumers will be able to register additional backend
|
|
IDs during engine construction. Registration will be explicit and local to an
|
|
engine; it will not mutate package-global state.
|
|
|
|
Profiles will be able to name a backend. A profile that selects a backend may
|
|
override its default endpoint and other supported defaults while continuing to
|
|
own its model and generation settings. Existing endpoint-based custom profiles
|
|
will remain supported without requiring a backend ID.
|
|
|
|
The built-in model profile catalog will use the built-in OpenRouter backend
|
|
instead of repeating OpenRouter connection and credential defaults in every
|
|
profile.
|
|
|
|
## Resolution And Precedence
|
|
|
|
Effective execution settings will resolve in this order:
|
|
|
|
1. application-neutral framework defaults;
|
|
2. selected backend defaults, when a backend is named;
|
|
3. selected profile values; and
|
|
4. explicit per-run overrides.
|
|
|
|
A profile endpoint will override its selected backend's endpoint. A per-run
|
|
endpoint override will continue to take precedence over both.
|
|
|
|
Credential selection will preserve direct request credentials as the highest
|
|
precedence. An explicit per-run environment-variable override will take
|
|
precedence over profile credential configuration, which will take precedence
|
|
over the backend's default environment-variable name. Backend definitions and
|
|
profiles will contain credential-source metadata only, never resolved secret
|
|
values.
|
|
|
|
Backend request defaults, profile request values, and per-run request
|
|
overrides will follow one deterministic replacement rule. Resolution must not
|
|
introduce implicit deep merging whose result depends on map iteration or
|
|
incidental representation.
|
|
|
|
## Registration And Validation
|
|
|
|
Backend registration will be deterministic and validated during engine
|
|
construction.
|
|
|
|
- Built-in backend IDs are reserved and cannot be replaced by consumers.
|
|
- Consumer registrations may add only new IDs.
|
|
- Duplicate consumer IDs are invalid, including duplicates introduced through
|
|
repeated configuration.
|
|
- Backend IDs, endpoints, credential-source metadata, and request defaults
|
|
must be validated before an engine is returned.
|
|
- Selecting an unknown backend is an error associated with the profile or
|
|
request boundary that selected it.
|
|
- A profile without a backend must continue to provide the connection details
|
|
required by the current endpoint-based path.
|
|
- A backend may omit credential requirements so unauthenticated local-network
|
|
services remain supported.
|
|
|
|
Failures will retain error identities appropriate to the existing engine
|
|
configuration, profile-loading, and per-run validation boundaries.
|
|
|
|
## Public And Extension Boundaries
|
|
|
|
The effective backend ID will be observable in prepared and completed run
|
|
metadata and will be available to injected model clients. This gives consumers
|
|
and extensions a stable routing identity without requiring them to infer a
|
|
backend from an endpoint URL.
|
|
|
|
The existing public model-client injection boundary will remain supported.
|
|
Because the initial registry supports only OpenAI-compatible backends,
|
|
Promptkit does not need backend-specific transport factories or multiple
|
|
protocol implementations in this scope. The built-in client will continue to
|
|
send the resolved execution target to the selected OpenAI-compatible endpoint.
|
|
|
|
Backend definitions are configuration values, not live service objects. They
|
|
will not own mutable connections, credentials, health state, or process
|
|
lifecycle.
|
|
|
|
## Compatibility
|
|
|
|
The target end state preserves the existing ways consumers configure and run
|
|
Promptkit:
|
|
|
|
- Profiles with explicit endpoints and no backend ID continue to work.
|
|
- Profile endpoint overrides remain supported.
|
|
- Direct request API keys and environment-variable overrides retain their
|
|
precedence.
|
|
- Consumers may continue injecting a custom model client.
|
|
- Model IDs and generation settings remain profile concerns.
|
|
- Prompt selection, rendering, validation, artifacts, and synchronous
|
|
`Prepare` and `Run` behavior remain unchanged except for exposing the
|
|
resolved backend identity.
|
|
|
|
## Non-Goals
|
|
|
|
This scope does not include:
|
|
|
|
- non-OpenAI-compatible protocols or provider-specific SDKs;
|
|
- backend-specific concurrency limits or admission queues;
|
|
- asynchronous task submission or durable jobs;
|
|
- model discovery, live model catalogs, or provider capability probing;
|
|
- retries, rate limiting, failover, load balancing, or health checks;
|
|
- automatic backend selection based on model names or endpoint inspection;
|
|
- global mutable registration;
|
|
- consumer replacement of built-in backend IDs;
|
|
- credential discovery beyond configured direct values and environment
|
|
variables; or
|
|
- provider-specific application, deployment, or security policy.
|
|
|
|
The dependent concurrency and bounded-queue work remains a separate future
|
|
unit after backend identity and resolution are stable.
|
|
|
|
## Target End State
|
|
|
|
This roadmap reaches its target end state when:
|
|
|
|
- Promptkit has an immutable, engine-scoped registry of validated
|
|
OpenAI-compatible backend definitions;
|
|
- OpenRouter is available as a built-in backend with its endpoint and
|
|
credential environment-variable default;
|
|
- consumers can register additional unique backend IDs without modifying
|
|
Promptkit;
|
|
- profiles can select a backend and optionally override its endpoint;
|
|
- existing endpoint-only profiles remain valid;
|
|
- backend, profile, and per-run values resolve through documented,
|
|
deterministic precedence;
|
|
- secret values remain outside backend and profile definitions;
|
|
- the effective backend identity is available in preparation, execution, and
|
|
injected-client values;
|
|
- the built-in model profiles use the OpenRouter backend rather than duplicate
|
|
its connection defaults;
|
|
- the built-in OpenAI-compatible client and injected clients continue to work
|
|
through the existing generation boundary; and
|
|
- current-state GoDoc, format, consumer, integration, architecture, and
|
|
internal documentation describe the implemented behavior without relying on
|
|
this roadmap.
|