150 lines
7.0 KiB
Markdown
150 lines
7.0 KiB
Markdown
# OpenAI-Compatible Chat Integration
|
|
|
|
## Purpose
|
|
|
|
This document defines the outbound HTTP behavior implemented by Promptkit's
|
|
internal OpenAI-compatible model client. The
|
|
[internal model-client document](../internal/llm.md) owns implementation flow,
|
|
errors, and test ownership. The root Promptkit engine uses this client by
|
|
default unless a consumer injects another implementation. The
|
|
[framework format reference](../formats.md) owns the profile and prompt values
|
|
that produce these outbound settings.
|
|
|
|
## Endpoint And Method
|
|
|
|
Generation sends an HTTP `POST` with `Content-Type: application/json`.
|
|
Before the client is called, the engine resolves framework, backend, profile,
|
|
and request values into one execution target. Endpoint configuration is trimmed
|
|
and must be an absolute HTTP or HTTPS URL with a host and without user
|
|
information, a query, or a fragment. A non-empty endpoint from the target
|
|
overrides the client's configured base URL. The final selected endpoint is
|
|
validated again before transport.
|
|
|
|
The completion URL is composed through parsed URL path operations. Nested base
|
|
paths are retained, repeated trailing slashes are normalized, and the result
|
|
has exactly one appended `/chat/completions` suffix. Generation fails before
|
|
sending when neither source supplies a valid endpoint.
|
|
|
|
The target's backend ID is routing metadata for prepared values, results, and
|
|
injected clients. The built-in client does not derive the URL from that ID and
|
|
does not serialize it in the provider request.
|
|
|
|
## Authentication
|
|
|
|
A usable API key supplied directly on the execution target takes precedence.
|
|
Otherwise, when an API-key environment-variable name is supplied, the client
|
|
reads and trims that variable. A bearer header is sent only when the resolved
|
|
direct or environment credential is non-empty. When neither source is usable,
|
|
the client omits `Authorization` and handles the provider response normally.
|
|
An explicitly required target with no usable source is rejected before
|
|
transport.
|
|
|
|
The target contains the already resolved environment-variable name: an
|
|
explicit request override takes precedence over profile metadata, which takes
|
|
precedence over the backend default. Only the name reaches prepared metadata;
|
|
the environment value is read just before the provider call and is never added
|
|
to the JSON body.
|
|
|
|
## Request Body
|
|
|
|
The request body always contains `model` and `messages`. The execution
|
|
target's model takes precedence over the client's configured model, and one
|
|
must be available.
|
|
|
|
Each ordinary message contains its `role` and string `content`. A
|
|
cache-controlled message instead uses a text content block containing `type`,
|
|
`text`, and `cache_control`; an empty cache-control TTL is omitted.
|
|
|
|
The effective direct or prompt-rendered session ID is trimmed, limited to 256
|
|
Unicode code points, and sent when nonempty as top-level `session_id`. It is
|
|
never also sent as a session header.
|
|
|
|
The client conditionally includes:
|
|
|
|
- `temperature`, `max_tokens`, and `top_p` only when selected by a profile or
|
|
runtime override, including an explicit runtime zero; they are absent when
|
|
unspecified;
|
|
- non-empty `service_tier` and effective `reasoning_effort`; an explicitly
|
|
disabled reasoning setting is empty and therefore omitted; and
|
|
- `response_format` for JSON Schema structured output, including its name,
|
|
strict flag, and schema document.
|
|
|
|
The engine resolves backend, profile, and request extra-parameter maps by
|
|
whole-map replacement rather than key merging. The resulting effective map is
|
|
then merged directly into the top-level body after JSON serialization is
|
|
verified. Empty keys and collisions with these reserved fields are rejected
|
|
before any provider call:
|
|
|
|
- `model`
|
|
- `session_id`
|
|
- `messages`
|
|
- `temperature`
|
|
- `max_tokens`
|
|
- `top_p`
|
|
- `service_tier`
|
|
- `reasoning_effort`
|
|
- `response_format`
|
|
|
|
`backend_id`, `api_key_env`, and resolved credential values are not provider
|
|
request fields.
|
|
|
|
## Response Handling
|
|
|
|
Any 2xx response body is limited to 16 MiB (16,777,216 bytes). A larger
|
|
declared `Content-Length` is rejected before the body is read, and streamed,
|
|
chunked, or underreported bodies are read through the same bound with at most
|
|
one additional byte used to detect overflow. A body exactly at the limit is
|
|
allowed. The body is closed on every outcome and an oversized stream is not
|
|
drained.
|
|
|
|
The bounded body must contain exactly one OpenAI-compatible JSON response
|
|
object followed only by JSON whitespace and EOF. The client returns the first
|
|
choice's non-empty message content and maps prompt, completion, total, cached,
|
|
and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
|
|
data, a second JSON value, absent choices, empty first-choice content, and size
|
|
overflow are malformed responses and return no partial result.
|
|
|
|
For a non-2xx status, Promptkit recognizes one JSON document with a top-level
|
|
object-valued `error` member. Its optional `message` and `type` fields must be
|
|
strings, and `code` may be a string or JSON number. Valid supported fields are
|
|
handled independently, numeric codes retain their JSON number text, and
|
|
unknown fields are ignored. Missing, invalid, malformed, or multiply framed
|
|
envelopes contribute no provider detail.
|
|
|
|
Non-success bodies have a 65,536-byte limit. A larger declared
|
|
`Content-Length` is not read; otherwise the client reads at most one additional
|
|
byte to detect streamed or underreported overflow. Empty, unreadable,
|
|
oversized, malformed, and unrecognized bodies retain only the received status.
|
|
The body is always closed and no oversized stream is drained beyond that probe.
|
|
|
|
Extracted strings are made valid UTF-8, trimmed, and converted to one line by
|
|
collapsing Unicode whitespace, control, and format-character runs. Blank
|
|
values are omitted. Codes and types longer than 256 Unicode code points are
|
|
omitted; messages longer than 4,096 code points are truncated at a code-point
|
|
boundary with an ellipsis inside the limit. Promptkit never exposes raw bodies,
|
|
headers, endpoints, credentials, request data, schemas, generated content, or
|
|
unsupported provider metadata through this handling.
|
|
|
|
An outbound `http.Client.Do` failure retains both Promptkit's request-failure
|
|
identity and the exact transport error for `errors.Is` and `errors.As` checks.
|
|
The rendered error does not include the selected endpoint, request headers,
|
|
request content, credentials, or provider body.
|
|
|
|
## Timeout And Cancellation
|
|
|
|
Timeouts are layered:
|
|
|
|
- the caller context remains the outer cancellation boundary;
|
|
- a positive generation timeout adds a request context deadline;
|
|
- zero adds no generation-specific deadline;
|
|
- a negative generation timeout is invalid; and
|
|
- the cloned `http.Client` supplies the whole-request transport cap, retaining
|
|
a positive supplied-client timeout or applying the configured/default
|
|
timeout when the supplied value is not positive.
|
|
|
|
The earliest applicable caller, generation, or transport deadline controls the
|
|
request. Caller cancellation retains `context.Canceled`; caller, generation,
|
|
and whole-request timeout failures retain `context.DeadlineExceeded`, together
|
|
with the request-failure identity. Constructing the internal client does not
|
|
mutate a supplied `http.Client`.
|