From a10d1a22455e0dbc264905555a00140bdde073a8 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 15:27:12 +0000 Subject: [PATCH] Select provider conventions through backend configuration --- docs/roadmap/invocation-scoped-token-usage.md | 120 +++++++++++++++--- 1 file changed, 102 insertions(+), 18 deletions(-) diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md index 437bcde..d386ec6 100644 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -126,6 +126,81 @@ must not inflate its totals. Terminal usage is an aggregate, not an additional contribution to be added to attempt usage. These rules apply to cost as well as token counts. +## Backend Provider Selection + +Add an optional backend `provider` field identifying the API implementation +whose conventions PromptKit should use. It identifies the service contacted, +not the organization that trained the selected model. Backend IDs remain resource +identities; names, namespaces, and endpoint hostnames do not select provider +behavior. + +```yaml +promptkit: + backends: + - id: research-router + provider: openrouter + endpoint: https://openrouter.ai/api/v1 + api_key_env: OPENROUTER_API_KEY +``` + +An omitted or null provider retains ordinary OpenAI-compatible behavior without +provider-specific defaults or metadata interpretation. Initially, `openrouter` +selects the OpenRouter conventions described here. A vLLM-specific selection can +follow when its relevant response contract and representative fixtures have been +reviewed; this work does not claim vLLM-specific support. + +Resolve the field through ordinary ConfigKit composition and inheritance. Carry +the selected value through configuration capture and prepared execution, expose +it in backend inspection, and declare it in the bundled OpenRouter backend. +It is PromptKit metadata, not an outbound `extra_params` entry. Selecting a +provider does not choose a model or alter the configured endpoint. An endpoint +override retains the provider selection. + +### Smoke-Test Defaults And Explicit Overrides + +When an authentication smoke test is explicitly requested, the selected provider +supplies its default probe convention. `provider: openrouter` supplies the +existing `openrouter_key` convention. Preserve explicit +`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint` +settings as overrides: + +- A supplied kind selects the diagnostic adapter instead of the provider default. + `kind: none` continues to declare authentication testing inapplicable. +- An omitted/null kind uses the provider default, if available. This permits an + endpoint override without repeating the provider's default kind. +- A supplied endpoint overrides the selected adapter's default destination. + An omitted/null endpoint uses that adapter's existing endpoint fallback. + +An absent/null authentication block likewise leaves selection to the provider +default. Clearing the block removes its explicit overrides; `kind: none` is the +explicit way to declare authentication inapplicable despite a provider default. +Without a provider default or explicit adapter, retain existing unsupported +diagnostic behavior. Existing explicit adapters continue to work without a +provider declaration. Reachability and inference-default settings keep their +existing behavior. + +These are conventions for explicitly requested diagnostics. Loading a backend, +constructing an engine, and ordinary generation do not initiate probes. + +### Provider Behavior Ownership + +Centralize provider-specific behavior within `internal/llm`, which already owns +OpenAI-compatible transport and OpenRouter authentication-response recognition. +Keep common request construction, credentials, HTTP execution, framing, and +common response fields shared. Provider-specific code owns additional response +metadata interpretation and authentication-probe conventions. + +Use one provider-selection mechanism inside that package. Generation, +observation, and smoke-test coordination consume provider-neutral facts or invoke +generic diagnostic operations rather than repeating provider dispatch and +response interpretation. Explicit diagnostic adapter selection remains supported +at this boundary. Update the current OpenRouter-specific diagnostic collaborator +method to a generic operation as part of this consolidation. + +Provider-specific files and straightforward dispatch are sufficient initially. +No new `internal/providers` package, registration framework, or broad interface +requiring every provider to implement every capability is needed. + ## Optional Provider-Reported Cost ### Initial Provider Support @@ -137,18 +212,11 @@ and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-t Do not add `cost_details` breakdowns or combine upstream inference cost with the top-level charge; those describe different facts and could double-count cost. -For the built-in client, select this interpretation for the resolved -`openrouter` backend ID. Do not infer USD semantics from an arbitrary backend's -field named `cost`, or add endpoint discovery or verification. An endpoint -override on that backend continues to select its OpenRouter response contract. -Other built-in backend selections report unknown cost initially. Additional -provider adapters or cost-capability configuration are not needed for this scope. - -This baseline selects by exact backend identity: a separately named or -collection-qualified backend pointing to OpenRouter would report unknown cost. -Backend configuration as an alternative to identity-based selection is now an -open discussion; settle that selection before implementing provider cost support. -The contextual-observation stage does not depend on that decision. +For the built-in client, select this interpretation through `provider: openrouter`, +regardless of backend identity or namespace. Do not infer USD semantics from an +arbitrary backend's field named `cost`, or add endpoint discovery or verification. +Without an OpenRouter provider selection, built-in cost is unknown initially. +No separate cost-reporting selector is needed. Use the cost included in the normal completion response. No additional usage lookup, pricing request, or opt-in provider request parameter is needed. @@ -245,6 +313,17 @@ logging infrastructure are outside this scope. - Preserve existing explicit selection and coordinator behavior while applying the composition and lifetime rules above. Preserve existing observer selection when no contextual observer is selected. +- Add `provider` to the backend configuration model, typed source helpers, + capture, inspection, and execution plumbing. Update the bundled OpenRouter + backend to select it explicitly. Retain the selection through prepared runs. +- Consolidate provider-specific response and authentication-probe behavior in + `internal/llm`; keep coordination and accounting provider-neutral. Cover + provider-default probes, explicit kind and endpoint overrides, `kind: none`, + and existing explicit probes without a provider selection. +- Verify provider selection independently of backend names and namespaces, + including renamed and inherited backends, generic behavior when omitted/null, + and retention through endpoint overrides. Confirm provider metadata is not + serialized into the completion request body. - Carry optional OpenRouter cost through response decoding, internal/public conversions, successful run results, attempt and terminal events, and existing group summaries. Keep result cost available without registering an observer. @@ -263,7 +342,7 @@ logging infrastructure are outside this scope. failures or cancellation, and known zero versus unknown or incomplete counts. Use offline fixtures and injected clients; live inference is unnecessary. - Cover fractional USD values, explicit zero, omitted/null and unusable cost, - non-OpenRouter built-in selection, optional injected-client cost, mixed known + generic built-in selection, optional injected-client cost, mixed known and unknown totals, repair/retry aggregation, and duplicate group registration. Use tolerant comparisons where floating-point addition requires them; do not turn telemetry into an exact-decimal guarantee. @@ -272,6 +351,9 @@ logging infrastructure are outside this scope. availability, units, and completeness. Document OpenRouter cost decoding in the [integration contract](../integrations/openai-compatible-chat.md#response-handling), and update internal observation and model-client documentation for changed flows. + Document `provider`, its smoke-test defaults and override semantics in the + framework format reference and relevant public GoDoc; update inspection and + diagnostic documentation with the implemented behavior. ## Implementation Planning Guidance @@ -279,12 +361,14 @@ Plan two separately reviewable stages: 1. Contextual observer selection, including ordinary, prepared, coordinated, and nested execution boundaries and the existing group collector. -2. Provider-reported cost, carried through decoding, accounting, public results, - events, and group summaries, with the corresponding documentation. +2. Backend provider selection and consolidation within `internal/llm`, including + smoke-test defaults and explicit overrides, followed by provider-reported cost + through decoding, accounting, public results, events, and group summaries. + Include the corresponding configuration and API documentation. -The second stage is more extensive than the context helper. Resolve the backend -selection discussion before finalizing that stage; avoid coupling delivery of -contextual observation to provider cost support. +The second stage is more extensive than the context helper and may be divided +into focused implementation units. Backend provider selection is decided above; +avoid coupling delivery of contextual observation to provider cost support. Today, successful results accumulate generation responses in the runner, while optional observation recorders account for dispatched attempts. Adding cost to