promptkit/docs/integrations/openai-compatible-chat.md

# OpenAI-Compatible Chat Integration

## Purpose

This document defines the outbound HTTP behavior implemented by Promptkit's
internal OpenAI-compatible model client. The
[internal model-client document](../internal/llm.md) owns implementation flow,
errors, and test ownership. The root Promptkit engine uses this client by
default unless a consumer injects another implementation. The
[framework format reference](../formats.md) owns the profile and prompt values
that produce these outbound settings.

## Endpoint And Method

Generation sends an HTTP `POST` with `Content-Type: application/json`.
Before the client is called, the engine resolves framework, backend, profile,
and request values into one execution target. Endpoint configuration is trimmed
and must be an absolute HTTP or HTTPS URL with a host and without user
information, a query, or a fragment. A non-empty endpoint from the target
overrides the client's configured base URL. The final selected endpoint is
validated again before transport.

The completion URL is composed through parsed URL path operations. Nested base
paths are retained, repeated trailing slashes are normalized, and the result
has exactly one appended `/chat/completions` suffix. Generation fails before
sending when neither source supplies a valid endpoint.

The target's backend ID is routing metadata for prepared values, results, and
injected clients. The built-in client does not derive the URL from that ID and
does not serialize it in the provider request.

## Authentication

A usable API key supplied directly on the execution target takes precedence.
Otherwise, when an API-key environment-variable name is supplied, the client
reads and trims that variable. A bearer header is sent only when the resolved
direct or environment credential is non-empty. When neither source is usable,
the client omits `Authorization` and handles the provider response normally.
An explicitly required target with no usable source is rejected before
transport.

The target contains the already resolved environment-variable name: an
explicit request override takes precedence over profile metadata, which takes
precedence over the backend default. Only the name reaches prepared metadata;
the environment value is read just before the provider call and is never added
to the JSON body.

## Explicit Smoke Diagnostics

Promptkit's explicit smoke diagnostics are separate consumer operations; they
are not part of ordinary `Run`, the offline examples, or release validation.
With the built-in client, inference diagnostics use the same configured client
and `/chat/completions` request as generation. Their content cases are private
to Promptkit and are not application prompt, schema, or provider protocol
extensions. The public
[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go) owns request selection,
report facts, and injected-client behavior.

HTTP-only diagnostics use the configured built-in HTTP client. Reachability is
one unauthenticated `GET` to the selected diagnostic URL; an observed response
counts as reachable even when its status is non-2xx. Authentication behavior
is selected from backend metadata, not inferred from the host. The
`openrouter_key` adapter makes one `GET` to `/key` relative to the effective
backend base, unless configuration supplies a full diagnostic endpoint (see
the [backend format reference](../formats.md#built-in-backends)). It may send
the available bearer credential, but can dispatch without one. A 401 is
recognized as rejection. A 200 is recognized as acceptance only when its JSON
object has a `data` object containing string `label`, Boolean `is_free_tier`,
and numeric `usage`; unrelated fields and account limits are ignored. This is
only a minimal identifying subset of OpenRouter's documented
[current-key response](https://github.com/OpenRouterTeam/docs/blob/main/api_reference/limits.mdx#checking-your-limits),
not validation of the full account document. Other statuses and unrecognized
200 responses do not establish acceptance.

Diagnostic HTTP evidence describes the last request observed at Promptkit's
RoundTripper boundary and any response associated with it. It includes the
selected destination without URL user information, response status, and
credential presence when observed. Redirects can change the last observed
destination or strip credentials. This evidence does not prove that bytes
reached a peer or reveal dispatch hidden inside a custom transport. Diagnostic
reports do not retain headers, credentials, provider
bodies, or generated text. A consumer-injected `LLMClient` can support inference
diagnostics, but does not establish HTTP status, destination, reachability, or
upstream credential-supply facts; Promptkit does not fall back to a built-in
transport for HTTP-only diagnostics.

## Request Body

The request body always contains `model` and `messages`. The execution
target's model takes precedence over the client's configured model, and one
must be available.

Each ordinary message contains its `role` and string `content`. A
cache-controlled message instead uses a text content block containing `type`,
`text`, and `cache_control`; an empty cache-control TTL is omitted.

Promptkit sends only `developer`, `system`, `user`, and `assistant` roles and
does so without provider-specific translation. Tool and deprecated function
payloads are outside this text-message contract. A backend or model that
rejects an otherwise supported role or context returns its ordinary provider
error, which follows the normal generation-error path.

The effective direct or prompt-rendered session ID is trimmed, limited to 256
Unicode code points, and sent when nonempty as top-level `session_id`. It is
never also sent as a session header.

The client conditionally includes:

- `temperature`, `max_tokens`, and `top_p` only when selected by a profile or
  runtime override, including an explicit runtime zero; they are absent when
  unspecified;
- non-empty `service_tier` and effective `reasoning_effort`; an explicitly
  disabled reasoning setting is empty and therefore omitted; and
- `response_format` for JSON Schema structured output, including its name,
  strict flag, and schema document. Plain JSON validation does not add an
  object-only response constraint.

The engine resolves backend, profile, and request extra-parameter maps by
whole-map replacement rather than key merging. The resulting effective map is
merged into the top-level body. Empty keys, values that cannot be encoded, and
collisions with these reserved fields are rejected before any provider call:

- `model`
- `session_id`
- `messages`
- `temperature`
- `max_tokens`
- `top_p`
- `service_tier`
- `reasoning_effort`
- `response_format`

`backend_id`, `api_key_env`, and resolved credential values are not provider
request fields.

## Response Handling

Any 2xx response body is limited to 16 MiB (16,777,216 bytes). A larger
declared `Content-Length` is rejected before the body is read, and streamed,
chunked, or underreported bodies are read through the same bound with at most
one additional byte used to detect overflow. A body exactly at the limit is
allowed. The body is closed on every outcome and an oversized stream is not
drained.

The bounded body must contain exactly one OpenAI-compatible JSON response
object followed only by JSON whitespace and EOF. The client returns the first
choice's explicitly present string message content, including an empty or
whitespace-only string, and maps prompt, completion, total, cached, and
cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
data, a second JSON value, absent choices, missing content, `null` content,
non-string content, and size overflow are malformed responses and return no
partial result.

Each of the five token fields retains whether the provider reported a
non-null integer, including zero. Omitted and `null` fields are unknown for
observations. A strictly decoded successful envelope can yield observed usage
even when a later missing choice or content makes the attempt malformed.
Invalid framing and non-2xx error bodies supply no observed usage; provider
work may still have occurred. Reported counts are not inferred, reconciled,
or proof of complete cost accounting.

For a non-2xx status, Promptkit recognizes one JSON document with a top-level
object-valued `error` member. Its optional `message` and `type` fields must be
strings, and `code` may be a string or JSON number. Valid supported fields are
handled independently, numeric codes retain their JSON number text, and
unknown fields are ignored. Missing, invalid, malformed, or multiply framed
envelopes contribute no provider detail.

Non-success bodies have a 65,536-byte limit. A larger declared
`Content-Length` is not read; otherwise the client reads at most one additional
byte to detect streamed or underreported overflow. Empty, unreadable,
oversized, malformed, and unrecognized bodies retain only the received status.
The body is always closed and no oversized stream is drained beyond that probe.

Extracted strings are made valid UTF-8, trimmed, and converted to one line by
collapsing Unicode whitespace, control, and format-character runs. Blank
values are omitted. Codes and types longer than 256 Unicode code points are
omitted; messages longer than 4,096 code points are truncated at a code-point
boundary with an ellipsis inside the limit. Promptkit never exposes raw bodies,
headers, endpoints, credentials, request data, schemas, generated content, or
unsupported provider metadata through this handling.

An outbound `http.Client.Do` failure retains both Promptkit's request-failure
identity and the exact transport error for `errors.Is` and `errors.As` checks.
The rendered error does not include the selected endpoint, request headers,
request content, credentials, or provider body.

## Transient Status Retries

The built-in path retries only HTTP 429, 500, 502, 503, and 504. It makes two
additional attempts by default; engine configuration or an individual run can
choose a different nonnegative count, including zero for single-attempt
behavior. The same budget applies separately to initial generation and each
structural repair generation. An injected `LLMClient` is dispatched once by
Promptkit regardless of this setting. A custom `http.Client` on the built-in
path remains subject to these retries.

Each retry uses equal jitter between half and all of an exponential local
delay, starting at 500 milliseconds and capped at 8 seconds. The first
`Retry-After` header value sets a minimum not-before time measured from
response receipt if it is valid unsigned decimal seconds or an HTTP date.
Invalid values use local delay; past dates impose no extra delay. The provider
minimum is not capped at 8 seconds. Retry waiting is cancellable and holds no
active backend permit. A caller without a deadline can wait a long time for
provider timing.

The request body, selected URL, and resolved authorization are fixed across
attempts of one generation. Each attempt creates a fresh HTTP request and
receives its own configured generation and HTTP-client timeout after capacity
admission. The caller context covers every attempt and wait without a reset.
Transport failures, malformed responses, noneligible statuses, and invalid
generated content are not transient retries. A positive budget ending in
eligible statuses reports retry exhaustion with the final provider cause.

## Timeout And Cancellation

Timeouts are layered:

- the caller context remains the outer cancellation boundary;
- a positive generation timeout adds a request context deadline;
- zero adds no generation-specific deadline;
- a negative generation timeout is invalid; and
- the cloned `http.Client` supplies the whole-request transport cap, retaining
  a positive supplied-client timeout or applying the configured/default
  timeout when the supplied value is not positive.

The earliest applicable caller, generation, or transport deadline controls the
request. Caller cancellation retains `context.Canceled`; caller, generation,
and whole-request timeout failures retain `context.DeadlineExceeded`, together
with the request-failure identity. Constructing the internal client does not
mutate a supplied `http.Client`.