180 lines
10 KiB
Markdown
180 lines
10 KiB
Markdown
# Internal Model Client
|
|
|
|
## Purpose
|
|
|
|
This document describes Promptkit's internal model-client implementation. The
|
|
[architecture policy](../policy/architecture.md) owns the library boundary,
|
|
and the
|
|
[OpenAI-compatible chat integration](../integrations/openai-compatible-chat.md)
|
|
owns the observable outbound HTTP contract. The
|
|
[framework format reference](../formats.md) owns the profile and prompt
|
|
settings consumed by the client.
|
|
|
|
The concrete client remains under `internal/llm`. The internal engine assembles it
|
|
as the default implementation behind Promptkit's public client boundary.
|
|
|
|
## Components And Flow
|
|
|
|
`Client` is the provider-neutral generation boundary consumed by later
|
|
orchestration. `OpenAICompatibleClient` is the built-in implementation. It
|
|
uses internal domain values for rendered prompts, execution targets,
|
|
structured output, responses, and token usage.
|
|
|
|
The runner supplies a fully resolved target after applying backend, profile,
|
|
and request precedence, plus canonical provider-bound text messages. The
|
|
client uses its endpoint, credential metadata, generation fields, and extra
|
|
parameters. `BackendID` remains routing metadata
|
|
for the generation boundary and is not mapped into the provider payload.
|
|
|
|
Construction trims and validates a nonempty configured base URL and clones any
|
|
supplied `http.Client` so Promptkit can apply its timeout default without
|
|
mutating the caller's client. An empty configured base remains valid because a
|
|
resolved request target may supply the endpoint. Generation preparation then:
|
|
|
|
1. validates shared execution-setting invariants and the final selected base
|
|
endpoint;
|
|
2. maps the internal request into the OpenAI-compatible chat payload;
|
|
3. checks extra-parameter keys against empty and reserved names, then merges
|
|
their values into the payload;
|
|
4. composes `/chat/completions` through parsed URL path operations;
|
|
5. encodes the complete body once on success. If encoding fails, extra values
|
|
are checked individually to retain invalid-request classification; otherwise
|
|
the complete-request failure remains a request failure;
|
|
6. resolves authentication and freezes the URL, body, and optional credential.
|
|
|
|
Each execution of that prepared request creates a fresh HTTP request, body
|
|
reader, and attempt timeout context. It closes the response body and returns
|
|
the HTTP status, decoded response or error, reported token values and presence,
|
|
and any valid provider `Retry-After` not-before time. A strictly decoded
|
|
successful envelope can retain usage even if its first choice or content is
|
|
missing. Invalid framing and non-success error bodies supply no usage. The
|
|
ordinary `Generate` method prepares and executes once and returns no partial
|
|
response on failure.
|
|
|
|
`internal/llm` owns the set of reserved OpenAI-compatible request fields used
|
|
when validating extra parameters. Backend registration consumes the same rule
|
|
without making the model client depend on registry configuration.
|
|
|
|
This client does not schedule retries; the [generation coordinator](generation.md)
|
|
does so around its prepared attempts. The client has no tool-call support,
|
|
provider catalog, inbound HTTP behavior, or durable session store.
|
|
|
|
## Diagnostic HTTP Evidence
|
|
|
|
Built-in diagnostics wrap the selected HTTP client's `RoundTripper` for one
|
|
invocation using a shallow client copy, preserving its timeout, redirect policy,
|
|
cookie jar, and underlying transport without mutating the configured client.
|
|
Evidence distinguishes a diagnostic dispatch invocation from an observed
|
|
request and response. A newly observed request replaces the previous
|
|
destination-specific response evidence, so redirects retain only the last
|
|
observed hop. A response is recorded before envelope decoding or later context
|
|
checks can discard a generation result.
|
|
|
|
The collector keeps only the last observed destination (without URL user
|
|
information), status, credential-presence, response-availability, decoded
|
|
generation, and presence-aware usage facts needed by the diagnostic. It does
|
|
not retain headers, credentials, raw request or response bodies, or a redirect
|
|
trace. These are observations at Promptkit's RoundTripper boundary, not proof
|
|
that bytes reached a peer or that a custom transport did no hidden dispatch.
|
|
Reachability and named authentication probes use the same configured built-in
|
|
client but discard their response bodies after local recognition. The
|
|
[integration contract](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics)
|
|
owns their observable HTTP semantics.
|
|
|
|
An injected `LLMClient` does not pass through this HTTP boundary. Its returned
|
|
response can establish response and inference facts, but not an HTTP
|
|
destination, status, reachability, or upstream credential-supply fact. The
|
|
facade does not silently install a built-in transport for those diagnostics.
|
|
|
|
## Prepared Generation
|
|
|
|
For [`RunPrepared`](../../engine.go), the runner supplies the model client with
|
|
the target, rendered messages, and structured-output constraint retained by
|
|
executable preparation. Execution does not reopen or rerender consumer
|
|
sources.
|
|
|
|
Before backend admission, the runner rechecks a frozen credential
|
|
environment-variable name only when the target explicitly requires a
|
|
credential. The handle does not retain the environment value; the model client
|
|
resolves the value visible when generation begins. For optional sources with no
|
|
usable value, the built-in client omits `Authorization` and continues to the
|
|
provider. A direct request key remains in private execution state only until
|
|
the claimed execution finishes or an unclaimed handle is discarded. Exact
|
|
public ownership and redaction semantics belong to the
|
|
[`PreparedExecution` GoDoc](../../prepared_execution.go).
|
|
|
|
## Failure Categories
|
|
|
|
The package preserves distinct error identities for invalid client
|
|
configuration, invalid generation requests, request execution failures,
|
|
non-success provider statuses, and malformed successful responses. Provider
|
|
response bodies are never exposed in raw form through non-success errors.
|
|
|
|
Invalid nonempty configured endpoints are configuration failures. A missing or
|
|
invalid final selected endpoint is an invalid generation request and is
|
|
rejected before transport.
|
|
|
|
Authentication resolves a trimmed direct key before a trimmed configured
|
|
environment value. Optional missing, empty, or whitespace-only sources do not
|
|
block transport and produce no `Authorization` header. An explicitly required
|
|
target with no usable source is rejected before transport with the existing
|
|
invalid-request diagnostics.
|
|
|
|
Successful response bodies have a fixed 16 MiB limit enforced by declared
|
|
length and by reading at most one byte beyond the boundary. The decoder accepts
|
|
exactly one JSON object plus trailing whitespace and EOF. Size overflow,
|
|
truncation, malformed JSON, trailing data, and a second value are malformed
|
|
responses with no partial result or provider content in the error. Every body
|
|
is closed, and an unbounded oversized stream is not drained.
|
|
|
|
After framing succeeds, the first choice must contain an explicitly present
|
|
string `message.content`. The string is returned exactly, including empty or
|
|
whitespace-only content. Missing choices, missing or `null` content, and
|
|
non-string content are malformed responses. Output validation and correction
|
|
eligibility remain outside this package.
|
|
|
|
Successful-envelope usage is decoded with independent presence bits for each
|
|
token field: omission and `null` are unknown, while a reported integer zero is
|
|
known. Strictly decoded usage can still be observed if later choice/content
|
|
checks fail; invalid framing or a non-success error body contributes none.
|
|
Existing successful-result accounting remains value-based.
|
|
|
|
For a non-success response, `ProviderHTTPError` retains the HTTP status and
|
|
only normalized detail from the bounded recognized envelope. It retains
|
|
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;
|
|
its bounded reader and parser never close or drain a body themselves. The root
|
|
facade converts this concrete internal error into the public
|
|
[`GenerationError`](../../generation_error.go), while arbitrary injected-client
|
|
errors continue through the ordinary generation-error mapping unchanged.
|
|
|
|
An `http.Client.Do` failure is represented by a redacting multi-cause error:
|
|
the package request-failure sentinel and the exact returned transport error are
|
|
both available through `errors.Is` and `errors.As`, while the rendered text
|
|
does not expose the endpoint, headers, request content, credential, transport
|
|
detail, or provider body. Caller cancellation retains `context.Canceled`;
|
|
caller deadlines, generation deadlines, and whole-request client timeouts
|
|
retain `context.DeadlineExceeded`. The runner adds its generation category
|
|
without discarding those identities or depending on HTTP status mapping.
|
|
|
|
## Test Ownership
|
|
|
|
The
|
|
[OpenAI-compatible client tests](../../internal/llm/openai_compatible_client_test.go)
|
|
own configuration, client cloning, deterministic deadline precedence,
|
|
authentication, request and response mapping, malformed data, error identity,
|
|
cancellation, endpoint selection and composition, pre-transport rejection, and
|
|
bounded single-document successful-response framing, closure, and
|
|
response-body suppression. The [HTTP evidence tests](../../internal/llm/http_evidence_test.go)
|
|
own invocation-local last-hop request/response facts, while the
|
|
[prepared-generation evidence tests](../../internal/llm/prepared_generation_evidence_test.go)
|
|
own retention across decoding and cancellation. The focused
|
|
[provider HTTP error tests](../../internal/llm/provider_http_error_test.go)
|
|
own envelope parsing, normalization, and bounded-reader cases; their
|
|
[transport tests](../../internal/llm/provider_http_error_transport_test.go)
|
|
own non-success response closure and integration. Root transport contract tests
|
|
own public `GenerationError` conversion, while also verifying that resolved
|
|
backend settings reach this client without serializing backend identity and
|
|
that ordinary-run cancellation retains its public generation and context
|
|
identities. All use local test servers or controlled test transports; the
|
|
default suite makes no live or paid provider requests.
|