Extensible LLM Backend Registry

Purpose

This roadmap defines the scope and target end state for formal OpenAI-compatible backend support in Promptkit. It establishes the behavioral boundary and policy choices for the work.

This document is planning material, not a description of current behavior. Current public contracts remain owned by Go declarations and GoDoc, framework file formats by the format reference, and outbound HTTP behavior by the OpenAI-compatible integration contract.

Motivation

Execution profiles currently carry backend connection and authentication details alongside model and generation settings. This repeats values such as the OpenRouter endpoint and OPENROUTER_API_KEY across profiles, leaves the built-in catalog implicitly tied to one backend, and gives consumers no explicit extension point for naming other OpenAI-compatible services.

Promptkit should distinguish:

a backend, which identifies reusable connection, authentication, and limited backend-wide request defaults; and
a profile, which selects a model and its reusable generation settings.

That distinction should let consumers configure OpenRouter, OpenAI, local-network services, or other OpenAI-compatible deployments without duplicating backend defaults in every profile or replacing Promptkit's model client.

Scope

The work introduces an engine-scoped registry of OpenAI-compatible backend definitions.

Each backend definition will provide:

a stable, non-empty backend ID;
a default OpenAI-compatible base endpoint;
an optional environment-variable name for its API key; and
limited JSON-compatible request defaults when they are genuinely backend-wide rather than model-specific.

Promptkit will provide a small built-in backend catalog, initially containing OpenRouter. Downstream consumers will be able to register additional backend IDs during engine construction. Registration will be explicit and local to an engine; it will not mutate package-global state.

Profiles will be able to name a backend. A profile that selects a backend may override its default endpoint and other supported defaults while continuing to own its model and generation settings. Existing endpoint-based custom profiles will remain supported without requiring a backend ID.

The built-in model profile catalog will use the built-in OpenRouter backend instead of repeating OpenRouter connection and credential defaults in every profile.

Resolution And Precedence

Effective execution settings will resolve in this order:

application-neutral framework defaults;
selected backend defaults, when a backend is named;
selected profile values; and
explicit per-run overrides.

A profile endpoint will override its selected backend's endpoint. A per-run endpoint override will continue to take precedence over both.

Credential selection will preserve direct request credentials as the highest precedence. An explicit per-run environment-variable override will take precedence over profile credential configuration, which will take precedence over the backend's default environment-variable name. Backend definitions and profiles will contain credential-source metadata only, never resolved secret values.

Backend request defaults, profile request values, and per-run request overrides will follow one deterministic replacement rule. Resolution must not introduce implicit deep merging whose result depends on map iteration or incidental representation.

Registration And Validation

Backend registration will be deterministic and validated during engine construction.

Built-in backend IDs are reserved and cannot be replaced by consumers.
Consumer registrations may add only new IDs.
Duplicate consumer IDs are invalid, including duplicates introduced through repeated configuration.
Backend IDs, endpoints, credential-source metadata, and request defaults must be validated before an engine is returned.
Selecting an unknown backend is an error associated with the profile or request boundary that selected it.
A profile without a backend must continue to provide the connection details required by the current endpoint-based path.
A backend may omit credential requirements so unauthenticated local-network services remain supported.

Failures will retain error identities appropriate to the existing engine configuration, profile-loading, and per-run validation boundaries.

Public And Extension Boundaries

The effective backend ID will be observable in prepared and completed run metadata and will be available to injected model clients. This gives consumers and extensions a stable routing identity without requiring them to infer a backend from an endpoint URL.

The existing public model-client injection boundary will remain supported. Because the initial registry supports only OpenAI-compatible backends, Promptkit does not need backend-specific transport factories or multiple protocol implementations in this scope. The built-in client will continue to send the resolved execution target to the selected OpenAI-compatible endpoint.

Backend definitions are configuration values, not live service objects. They will not own mutable connections, credentials, health state, or process lifecycle.

Compatibility

The target end state preserves the existing ways consumers configure and run Promptkit:

Profiles with explicit endpoints and no backend ID continue to work.
Profile endpoint overrides remain supported.
Direct request API keys and environment-variable overrides retain their precedence.
Consumers may continue injecting a custom model client.
Model IDs and generation settings remain profile concerns.
Prompt selection, rendering, validation, artifacts, and synchronous Prepare and Run behavior remain unchanged except for exposing the resolved backend identity.

Non-Goals

This scope does not include:

non-OpenAI-compatible protocols or provider-specific SDKs;
backend-specific concurrency limits or admission queues;
asynchronous task submission or durable jobs;
model discovery, live model catalogs, or provider capability probing;
retries, rate limiting, failover, load balancing, or health checks;
automatic backend selection based on model names or endpoint inspection;
global mutable registration;
consumer replacement of built-in backend IDs;
credential discovery beyond configured direct values and environment variables; or
provider-specific application, deployment, or security policy.

The dependent concurrency and bounded-queue work remains a separate future unit after backend identity and resolution are stable.

Target End State

This roadmap reaches its target end state when:

Promptkit has an immutable, engine-scoped registry of validated OpenAI-compatible backend definitions;
OpenRouter is available as a built-in backend with its endpoint and credential environment-variable default;
consumers can register additional unique backend IDs without modifying Promptkit;
profiles can select a backend and optionally override its endpoint;
existing endpoint-only profiles remain valid;
backend, profile, and per-run values resolve through documented, deterministic precedence;
secret values remain outside backend and profile definitions;
the effective backend identity is available in preparation, execution, and injected-client values;
the built-in model profiles use the OpenRouter backend rather than duplicate its connection defaults;
the built-in OpenAI-compatible client and injected clients continue to work through the existing generation boundary; and
current-state GoDoc, format, consumer, integration, architecture, and internal documentation describe the implemented behavior without relying on this roadmap.

7.7 KiB Raw Blame History