promptkit/docs/integrations/openai-compatible-chat.md

OpenAI-Compatible Chat Integration

Purpose

This document defines the outbound HTTP behavior implemented by Promptkit's internal OpenAI-compatible model client. The internal model-client document owns implementation flow, errors, and test ownership. The root Promptkit engine uses this client by default unless a consumer injects another implementation. The framework format reference owns the profile and prompt values that produce these outbound settings.

Endpoint And Method

Generation sends an HTTP POST with Content-Type: application/json. Before the client is called, the engine resolves framework, backend, profile, and request values into one execution target. Endpoint configuration is trimmed and must be an absolute HTTP or HTTPS URL with a host and without user information, a query, or a fragment. A non-empty endpoint from the target overrides the client's configured base URL. The final selected endpoint is validated again before transport.

The completion URL is composed through parsed URL path operations. Nested base paths are retained, repeated trailing slashes are normalized, and the result has exactly one appended /chat/completions suffix. Generation fails before sending when neither source supplies a valid endpoint.

The target's backend ID is routing metadata for prepared values, results, and injected clients. The built-in client does not derive the URL from that ID and does not serialize it in the provider request.

Authentication

A usable API key supplied directly on the execution target takes precedence. Otherwise, when an API-key environment-variable name is supplied, the client reads and trims that variable. A bearer header is sent only when the resolved direct or environment credential is non-empty. When neither source is usable, the client omits Authorization and handles the provider response normally. An explicitly required target with no usable source is rejected before transport.

The target contains the already resolved environment-variable name: an explicit request override takes precedence over profile metadata, which takes precedence over the backend default. Only the name reaches prepared metadata; the environment value is read just before the provider call and is never added to the JSON body.

Explicit Smoke Diagnostics

Promptkit's explicit smoke diagnostics are separate consumer operations; they are not part of ordinary Run, the offline examples, or release validation. With the built-in client, inference diagnostics use the same configured client and /chat/completions request as generation. Their content cases are private to Promptkit and are not application prompt, schema, or provider protocol extensions. The public Engine.SmokeTest GoDoc owns request selection, report facts, and injected-client behavior.

HTTP-only diagnostics use the configured built-in HTTP client. Reachability is one unauthenticated GET to the selected diagnostic URL; an observed response counts as reachable even when its status is non-2xx. Authentication behavior is selected from backend metadata, not inferred from the host. The openrouter_key adapter makes one GET to /key relative to the effective backend base, unless configuration supplies a full diagnostic endpoint (see the backend format reference). It may send the available bearer credential, but can dispatch without one. A 401 is recognized as rejection. A 200 is recognized as acceptance only when its JSON object has a data object containing string label, Boolean is_free_tier, and numeric usage; unrelated fields and account limits are ignored. This is only a minimal identifying subset of OpenRouter's documented current-key response, not validation of the full account document. Other statuses and unrecognized 200 responses do not establish acceptance.

Diagnostic HTTP evidence describes the last request observed at Promptkit's RoundTripper boundary and any response associated with it. It includes the selected destination without URL user information, response status, and credential presence when observed. Redirects can change the last observed destination or strip credentials. This evidence does not prove that bytes reached a peer or reveal dispatch hidden inside a custom transport. Diagnostic reports do not retain headers, credentials, provider bodies, or generated text. A consumer-injected LLMClient can support inference diagnostics, but does not establish HTTP status, destination, reachability, or upstream credential-supply facts; Promptkit does not fall back to a built-in transport for HTTP-only diagnostics.

Request Body

The request body always contains model and messages. The execution target's model takes precedence over the client's configured model, and one must be available.

Each ordinary message contains its role and string content. A cache-controlled message instead uses a text content block containing type, text, and cache_control; an empty cache-control TTL is omitted.

Promptkit sends only developer, system, user, and assistant roles and does so without provider-specific translation. Tool and deprecated function payloads are outside this text-message contract. A backend or model that rejects an otherwise supported role or context returns its ordinary provider error, which follows the normal generation-error path.

The effective direct or prompt-rendered session ID is trimmed, limited to 256 Unicode code points, and sent when nonempty as top-level session_id. It is never also sent as a session header.

The client conditionally includes:

• temperature, max_tokens, and top_p only when selected by a profile or runtime override, including an explicit runtime zero; they are absent when unspecified;
• non-empty service_tier and effective reasoning_effort; an explicitly disabled reasoning setting is empty and therefore omitted; and
• response_format for JSON Schema structured output, including its name, strict flag, and schema document. Plain JSON validation does not add an object-only response constraint.

The engine resolves backend, profile, and request extra-parameter maps by whole-map replacement rather than key merging. The resulting effective map is merged into the top-level body. Empty keys, values that cannot be encoded, and collisions with these reserved fields are rejected before any provider call:

• model
• session_id
• messages
• temperature
• max_tokens
• top_p
• service_tier
• reasoning_effort
• response_format

backend_id, api_key_env, and resolved credential values are not provider request fields.

Response Handling

Any 2xx response body is limited to 16 MiB (16,777,216 bytes). A larger declared Content-Length is rejected before the body is read, and streamed, chunked, or underreported bodies are read through the same bound with at most one additional byte used to detect overflow. A body exactly at the limit is allowed. The body is closed on every outcome and an oversized stream is not drained.

The bounded body must contain exactly one OpenAI-compatible JSON response object followed only by JSON whitespace and EOF. The client returns the first choice's explicitly present string message content, including an empty or whitespace-only string, and maps prompt, completion, total, cached, and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace data, a second JSON value, absent choices, missing content, null content, non-string content, and size overflow are malformed responses and return no partial result.

Each of the five token fields retains whether the provider reported a non-null integer, including zero. Omitted and null fields are unknown for observations. A strictly decoded successful envelope can yield observed usage even when a later missing choice or content makes the attempt malformed. Invalid framing and non-2xx error bodies supply no observed usage; provider work may still have occurred. Reported counts are not inferred, reconciled, or proof of complete cost accounting.

For a non-2xx status, Promptkit recognizes one JSON document with a top-level object-valued error member. Its optional message and type fields must be strings, and code may be a string or JSON number. Valid supported fields are handled independently, numeric codes retain their JSON number text, and unknown fields are ignored. Missing, invalid, malformed, or multiply framed envelopes contribute no provider detail.

Non-success bodies have a 65,536-byte limit. A larger declared Content-Length is not read; otherwise the client reads at most one additional byte to detect streamed or underreported overflow. Empty, unreadable, oversized, malformed, and unrecognized bodies retain only the received status. The body is always closed and no oversized stream is drained beyond that probe.

Extracted strings are made valid UTF-8, trimmed, and converted to one line by collapsing Unicode whitespace, control, and format-character runs. Blank values are omitted. Codes and types longer than 256 Unicode code points are omitted; messages longer than 4,096 code points are truncated at a code-point boundary with an ellipsis inside the limit. Promptkit never exposes raw bodies, headers, endpoints, credentials, request data, schemas, generated content, or unsupported provider metadata through this handling.

An outbound http.Client.Do failure retains both Promptkit's request-failure identity and the exact transport error for errors.Is and errors.As checks. The rendered error does not include the selected endpoint, request headers, request content, credentials, or provider body.

Transient Status Retries

The built-in path retries only HTTP 429, 500, 502, 503, and 504. It makes two additional attempts by default; engine configuration or an individual run can choose a different nonnegative count, including zero for single-attempt behavior. The same budget applies separately to initial generation and each structural repair generation. An injected LLMClient is dispatched once by Promptkit regardless of this setting. A custom http.Client on the built-in path remains subject to these retries.

Each retry uses equal jitter between half and all of an exponential local delay, starting at 500 milliseconds and capped at 8 seconds. The first Retry-After header value sets a minimum not-before time measured from response receipt if it is valid unsigned decimal seconds or an HTTP date. Invalid values use local delay; past dates impose no extra delay. The provider minimum is not capped at 8 seconds. Retry waiting is cancellable and holds no active backend permit. A caller without a deadline can wait a long time for provider timing.

The request body, selected URL, and resolved authorization are fixed across attempts of one generation. Each attempt creates a fresh HTTP request and receives its own configured generation and HTTP-client timeout after capacity admission. The caller context covers every attempt and wait without a reset. Transport failures, malformed responses, noneligible statuses, and invalid generated content are not transient retries. A positive budget ending in eligible statuses reports retry exhaustion with the final provider cause.

Timeout And Cancellation

Timeouts are layered:

• the caller context remains the outer cancellation boundary;
• a positive generation timeout adds a request context deadline;
• zero adds no generation-specific deadline;
• a negative generation timeout is invalid; and
• the cloned http.Client supplies the whole-request transport cap, retaining a positive supplied-client timeout or applying the configured/default timeout when the supplied value is not positive.

The earliest applicable caller, generation, or transport deadline controls the request. Caller cancellation retains context.Canceled; caller, generation, and whole-request timeout failures retain context.DeadlineExceeded, together with the request-failure identity. Constructing the internal client does not mutate a supplied http.Client.