From 440d9a2c8814708b25e94d0d25d3e5d80b7842ac Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 15:57:37 +0000 Subject: [PATCH] Add vLLM provider conventions to the observation roadmap --- docs/roadmap/invocation-scoped-token-usage.md | 139 +++++++++++++----- 1 file changed, 106 insertions(+), 33 deletions(-) diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md index 52cfceb..e73fee1 100644 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -1,6 +1,6 @@ # Invocation-Scoped Execution Observation -Status: agreed direction; vLLM scope under discussion; implementation pending. +Status: accepted scope; implementation pending. ## Purpose @@ -9,8 +9,8 @@ operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should participate automatically. Include optional provider-reported cost alongside token usage, with built-in cost support initially limited to OpenRouter. Extend -OpenRouter token reporting with reasoning counts, documented cache reads and -writes, and a derived token cache-hit ratio. +OpenRouter and vLLM token reporting with reasoning counts, documented cache reads +and writes, and a shared derived token cache-hit ratio. The motivating consumer is CommandKit's optional `ckprompt` integration, which could use this facility to report token usage for an entire command or pipeline. @@ -151,19 +151,16 @@ promptkit: Normalize the final effective provider value by trimming whitespace and lowercasing. Omitted, null, empty, and whitespace-only values are equivalent. Reject an unsupported nonempty normalized value as an ordinary configuration -error rather than silently using generic behavior. The agreed provider selections -are `openai-compatible` and `openrouter`; the possible `vllm` addition remains -under discussion. +error rather than silently using generic behavior. The supported selections for +this feature are `openai-compatible`, `openrouter`, and `vllm`. Omitted/null/blank provider values select the same effective behavior as explicit `provider: openai-compatible`. This is a runtime fallback, not an explicitly sourced configuration value. Backend inspection retains the distinction between an explicit provider declaration and an omitted/null/blank declaration. -The `openrouter` provider reuses the shared OpenAI-compatible foundation and adds -the OpenRouter response-metadata and diagnostic conventions below. Before -finalizing this roadmap, discuss the conventions and scope of a `vllm` selection; -this document does not yet select a vLLM contract. +The `openrouter` and `vllm` providers reuse the shared OpenAI-compatible foundation +and add their response-metadata and diagnostic conventions below. Resolve the field through ordinary ConfigKit composition and inheritance. Carry the selected value through configuration capture and prepared execution, expose @@ -201,11 +198,29 @@ Continue sending caller-selected supported settings and return backend errors normally, without capability negotiation or fallback requests. This is the existing text Chat Completions scope, not a claim to support the full OpenAI API. +### vLLM Provider + +`provider: vllm` selects the shared generation path plus vLLM-specific usage +normalization and a default authentication diagnostic. It supplies no default +endpoint, model, API-key environment variable, or sampling settings, and does not +create a backend definition automatically. + +Preserve caller-selected reasoning settings, structured-output requests, and +extra parameters. Do not automatically change chat-template arguments, enable +thinking, or translate model-specific controls based on the provider label. +No separate generation protocol is needed; vLLM documents an +[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/). + +The token mappings are specified below. Monetary cost remains unknown, not zero, +including when vLLM is locally hosted. This provider adds no price estimator, +server administration, metrics collection, or automatic capability/version probe. + ### Smoke-Test Defaults And Explicit Overrides When an authentication smoke test is explicitly requested, the selected provider supplies its default probe convention. `provider: openrouter` supplies the -existing `openrouter_key` convention. Preserve explicit +existing `openrouter_key` convention; `provider: vllm` supplies the `vllm_models` +convention described below. Preserve explicit `smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint` settings as overrides: @@ -233,6 +248,30 @@ values. Configuration inspection preserves the distinction between configured overrides and provider-supplied fallbacks, including their existing source provenance. +### vLLM Authentication Diagnostic + +The `vllm_models` adapter makes one GET request to `/models` relative to the +configured API base, supplying an available credential through the existing +credential rules. A base ending in `/v1` therefore produces `/v1/models`. +Existing explicit diagnostic endpoint overrides remain available. The adapter +can dispatch without a credential; selecting vLLM does not declare authentication +required or inapplicable. + +Recognize a successful models-list response as acceptance of the probe request +and a 401 as rejection. Unrecognized responses leave acceptance unknown under +the existing diagnostic reporting model. Report observed credential presence and +response facts without claiming that authentication is enforced or that models +listing establishes inference authorization. Preserve the shared override rules, +including `kind: none`; explicit `vllm_models` selection also works without a +`vllm` provider declaration, just as explicit OpenRouter adapters do. + +vLLM documents `/v1/models` as protected when its API-key authentication is +configured, while `/health` does not establish authentication acceptance. See its +[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations). +Retain existing reachability and inference diagnostic behavior rather than +adding health monitoring. These requests occur only during explicitly requested +diagnostics, never during ordinary generation or engine construction. + ### Diagnostic Migration This changes the current authentication metadata contract: an authentication @@ -263,9 +302,9 @@ requiring every provider to implement every capability is needed. This is configurable selection among PromptKit-supported provider implementations, not a public API for downstream registration of additional implementations. -## OpenRouter Token And Cache Reporting +## Token And Cache Reporting -### Response Mapping +### OpenRouter Response Mapping Use OpenRouter's documented Chat Completions usage fields. Its [response schema](https://openrouter.ai/docs/api_reference/overview), @@ -290,15 +329,47 @@ nested value; an omitted nested value remains unknown. Add `ReasoningTokens` to the public/domain token values, presence indicators, and observed token usage, including conversions, result serialization, attempt -and terminal events, and group summaries. Initially, OpenRouter supplies this +and terminal events, and group summaries. OpenRouter and vLLM supply this additional built-in response interpretation; the generic provider preserves its existing behavior. Injected clients may supply the canonical count and its presence through the existing model-client boundary. +### vLLM Response Mapping And Availability + +Reuse the ordinary `prompt_tokens`, `completion_tokens`, and `total_tokens` +counts. For `provider: vllm`, map these additional fields relative to `usage`: + +| Response field | PromptKit field | +| --- | --- | +| `completion_tokens_details.reasoning_tokens` | `ReasoningTokens` | +| `prompt_tokens_details.cached_tokens` | `CachedTokens` | +| `prompt_tokens_details.created_cache_tokens` | `CacheWriteTokens` | + +vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/) +uses `created_cache_tokens`, unlike OpenRouter's `cache_write_tokens`. Its +[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/) +describe the creation count as prompt tokens counted as local prefix-cache +writes. Normalize that token activity to `CacheWriteTokens` without implying a +billable write charge. Use the selected provider's documented field, not a sum +of alternative cache-write fields; an absent vLLM creation count stays unknown. + +Detail availability depends on server version and configuration. Current vLLM +gates prompt details on `--enable-prompt-tokens-details` and reasoning details on +a configured reasoning parser. Document these reporting settings, but do not +configure or discover them automatically. Older or differently configured +servers may omit the fields; accept those responses with the affected counts +unknown. Do not infer reasoning counts from generated text or cache writes from +input tokens minus cache reads. + +Once normalized, these counts use the shared presence, aggregation, and +cache-hit rules below. No vLLM-specific result type, ratio, or group accounting +is needed. + ### Count Relationships And Availability Reasoning tokens are included in completion tokens, as explained in OpenRouter's [reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens). +vLLM likewise reports its reasoning detail within the generated-token usage. Cache reads and writes describe input-token breakdowns. Present these as subsets or details rather than additional consumption to add to input/output totals. Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it. @@ -363,8 +434,8 @@ For the built-in client, select this interpretation through `provider: openroute regardless of backend identity or namespace. Do not infer USD semantics from an arbitrary backend's field named `cost`, or add endpoint discovery or verification. Built-in cost interpretation is an extension unique to `openrouter` initially; -`openai-compatible` leaves cost unknown even if a response contains `usage.cost`. -No separate cost-reporting selector is needed. Aggregation remains +`openai-compatible` and `vllm` leave cost unknown even if a response contains +`usage.cost`. No separate cost-reporting selector is needed. Aggregation remains provider-neutral after the model-client boundary supplies interpreted USD facts. Use the cost included in the normal completion response. No additional usage @@ -485,6 +556,17 @@ outside this scope. decoding, including omitted/null/zero values and the nested value taking precedence when a top-level value is also present. Preserve generic-provider token mappings and existing field meanings. +- Add the `vllm` provider selection, reasoning and cache-detail mappings, and + `vllm_models` authentication adapter. Cover supplied/missing credentials, + recognized acceptance, rejection, unrecognized responses, provider defaults, + and explicit kind/endpoint overrides, including `none` and adapter selection + without a provider declaration. Verify that ordinary generation performs no + diagnostic requests and leaves monetary cost unknown. +- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache + details, including responses representative of servers without detail + reporting enabled. Verify `created_cache_tokens` mapping independently of + OpenRouter's cache-write field and carry normalized counts through results, + events, and groups. Do not require a live server or version discovery. - Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and incomplete counts, zero input, inconsistent counts, independence from cache-write availability, and token-weighted aggregate ratios. Verify subset @@ -515,13 +597,16 @@ outside this scope. aggregation example to the consumer observation guide, including cost availability, units, and completeness. Document the reasoning/cache fields and cache-hit helper in their owning GoDoc and illustrate input/output breakdowns - without double-counting. Document OpenRouter token and cost decoding in - the [integration contract](../integrations/openai-compatible-chat.md#response-handling), + without double-counting. Document OpenRouter/vLLM token decoding and OpenRouter + cost decoding in the + [integration contract](../integrations/openai-compatible-chat.md#response-handling), and update internal observation and model-client documentation for changed flows. Document `provider`, its smoke-test defaults and override semantics in the framework format reference and relevant public GoDoc; update inspection and diagnostic documentation with the implemented behavior. Include the diagnostic - migration in release notes. + migration in release notes. Include vLLM usage availability, server reporting + settings, the meaning of cache creation, and the limits of its authentication + diagnostic in the relevant consumer and integration documentation. ## Implementation Planning Guidance @@ -530,7 +615,7 @@ Organize the eventual implementation plan around two workstreams: 1. Contextual observer selection, including ordinary, prepared, coordinated, and nested execution boundaries and the existing group collector. 2. Backend provider selection and consolidation within `internal/llm`, including - smoke-test defaults and explicit overrides, followed by OpenRouter token + smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token extensions and provider-reported cost through decoding, accounting, public results, events, and group summaries, plus the derived cache-hit helper. Include the corresponding configuration and API documentation. @@ -539,8 +624,7 @@ The second area is more extensive than the context helper. Keep backend provider configuration, diagnostic consolidation, token/cache reporting, and cost accounting in separately reviewable implementation units rather than one large implementation prompt. -Complete the remaining provider discussion before preparing that plan; avoid -coupling delivery of contextual observation to provider cost support. +Avoid coupling delivery of contextual observation to provider cost support. Today, successful results accumulate generation responses in the runner, while optional observation recorders account for dispatched attempts. Adding cost to @@ -559,14 +643,3 @@ framing or content validation. This needs no general numeric-conversion layer. Use the completion criteria above to select focused offline checks for each unit. This roadmap owns target behavior; a separate implementation plan owns the detailed work sequence. - -## Topics For Further Discussion - -Before finalizing the roadmap, discuss a `vllm` provider with vLLM-specific -conventions, identifying the response and diagnostic behavior to support from -reviewed upstream contracts and fixtures. - -The `openai-compatible` default, normalization, unsupported-value handling, -override precedence, and configuration-inspection boundaries above are agreed -decisions. The vLLM discussion does not yet add provider behavior or an -implementation commitment.