133 lines
6.9 KiB
Markdown
133 lines
6.9 KiB
Markdown
# Internal Model Client
|
|
|
|
## Purpose
|
|
|
|
This document describes Promptkit's internal model-client implementation. The
|
|
[architecture policy](../policy/architecture.md) owns the library boundary,
|
|
and the
|
|
[OpenAI-compatible chat integration](../integrations/openai-compatible-chat.md)
|
|
owns the observable outbound HTTP contract. The
|
|
[framework format reference](../formats.md) owns the profile and prompt
|
|
settings consumed by the client.
|
|
|
|
The concrete client remains under `internal/llm`. The root engine assembles it
|
|
as the default implementation behind Promptkit's public client boundary.
|
|
|
|
## Components And Flow
|
|
|
|
`Client` is the provider-neutral generation boundary consumed by later
|
|
orchestration. `OpenAICompatibleClient` is the built-in implementation. It
|
|
uses internal domain values for rendered prompts, execution targets,
|
|
structured output, responses, and token usage.
|
|
|
|
The runner supplies a fully resolved target after applying backend, profile,
|
|
and request precedence. The client uses its endpoint, credential metadata,
|
|
generation fields, and extra parameters. `BackendID` remains routing metadata
|
|
for the generation boundary and is not mapped into the provider payload.
|
|
|
|
Construction trims and validates a nonempty configured base URL and clones any
|
|
supplied `http.Client` so Promptkit can apply its timeout default without
|
|
mutating the caller's client. An empty configured base remains valid because a
|
|
resolved request target may supply the endpoint. Generation then:
|
|
|
|
1. validates shared execution-setting invariants and the final selected base
|
|
endpoint;
|
|
2. maps the internal request into the OpenAI-compatible chat payload;
|
|
3. validates and merges extra parameters;
|
|
4. composes `/chat/completions` through parsed URL path operations;
|
|
5. resolves authentication;
|
|
6. performs the outbound request under the applicable deadlines; and
|
|
7. decodes one strictly framed, size-bounded successful response object and
|
|
maps its first choice and token usage, or decodes bounded structured
|
|
non-success detail.
|
|
|
|
`internal/llm` owns the set of reserved OpenAI-compatible request fields used
|
|
when validating extra parameters. Backend registration consumes the same rule
|
|
without making the model client depend on registry configuration.
|
|
|
|
The implementation has no retry loop, tool-call support, provider catalog,
|
|
inbound HTTP behavior, or durable session store.
|
|
|
|
## Prepared Generation
|
|
|
|
For [`RunPrepared`](../../engine.go), the runner supplies the model client with
|
|
the target, rendered messages, and structured-output constraint retained by
|
|
executable preparation. Execution does not reopen or rerender consumer
|
|
sources.
|
|
|
|
Before backend admission, the runner rechecks a frozen credential
|
|
environment-variable name only when the target explicitly requires a
|
|
credential. The handle does not retain the environment value; the model client
|
|
resolves the value visible when generation begins. For optional sources with no
|
|
usable value, the built-in client omits `Authorization` and continues to the
|
|
provider. A direct request key remains in private execution state only until
|
|
the claimed execution finishes or an unclaimed handle is discarded. Exact
|
|
public ownership and redaction semantics belong to the
|
|
[`PreparedExecution` GoDoc](../../prepared_execution.go).
|
|
|
|
## Failure Categories
|
|
|
|
The package preserves distinct error identities for invalid client
|
|
configuration, invalid generation requests, request execution failures,
|
|
non-success provider statuses, and malformed successful responses. Provider
|
|
response bodies are never exposed in raw form through non-success errors.
|
|
|
|
Invalid nonempty configured endpoints are configuration failures. A missing or
|
|
invalid final selected endpoint is an invalid generation request and is
|
|
rejected before transport.
|
|
|
|
Authentication resolves a trimmed direct key before a trimmed configured
|
|
environment value. Optional missing, empty, or whitespace-only sources do not
|
|
block transport and produce no `Authorization` header. An explicitly required
|
|
target with no usable source is rejected before transport with the existing
|
|
invalid-request diagnostics.
|
|
|
|
Successful response bodies have a fixed 16 MiB limit enforced by declared
|
|
length and by reading at most one byte beyond the boundary. The decoder accepts
|
|
exactly one JSON object plus trailing whitespace and EOF. Size overflow,
|
|
truncation, malformed JSON, trailing data, and a second value are malformed
|
|
responses with no partial result or provider content in the error. Every body
|
|
is closed, and an unbounded oversized stream is not drained.
|
|
|
|
After framing succeeds, the first choice must contain an explicitly present
|
|
string `message.content`. The string is returned exactly, including empty or
|
|
whitespace-only content. Missing choices, missing or `null` content, and
|
|
non-string content are malformed responses. Output validation and correction
|
|
eligibility remain outside this package.
|
|
|
|
For a non-success response, `ProviderHTTPError` retains the HTTP status and
|
|
only normalized detail from the bounded recognized envelope. It retains
|
|
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;
|
|
its bounded reader and parser never close or drain a body themselves. The root
|
|
facade converts this concrete internal error into the public
|
|
[`GenerationError`](../../generation_error.go), while arbitrary injected-client
|
|
errors continue through the ordinary generation-error mapping unchanged.
|
|
|
|
An `http.Client.Do` failure is represented by a redacting multi-cause error:
|
|
the package request-failure sentinel and the exact returned transport error are
|
|
both available through `errors.Is` and `errors.As`, while the rendered text
|
|
does not expose the endpoint, headers, request content, credential, transport
|
|
detail, or provider body. Caller cancellation retains `context.Canceled`;
|
|
caller deadlines, generation deadlines, and whole-request client timeouts
|
|
retain `context.DeadlineExceeded`. The runner adds its generation category
|
|
without discarding those identities or depending on HTTP status mapping.
|
|
|
|
## Test Ownership
|
|
|
|
The
|
|
[OpenAI-compatible client tests](../../internal/llm/openai_compatible_client_test.go)
|
|
own configuration, client cloning, deterministic deadline precedence,
|
|
authentication, request and response mapping, malformed data, error identity,
|
|
cancellation, endpoint selection and composition, pre-transport rejection, and
|
|
bounded single-document successful-response framing, closure, and
|
|
response-body suppression. The focused
|
|
[provider HTTP error tests](../../internal/llm/provider_http_error_test.go)
|
|
own envelope parsing, normalization, and bounded-reader cases; their
|
|
[transport tests](../../internal/llm/provider_http_error_transport_test.go)
|
|
own non-success response closure and integration. Root transport contract tests
|
|
own public `GenerationError` conversion, while also verifying that resolved
|
|
backend settings reach this client without serializing backend identity and
|
|
that ordinary-run cancellation retains its public generation and context
|
|
identities. All use local test servers or controlled test transports; the
|
|
default suite makes no live or paid provider requests.
|