promptkit/docs/internal/llm.md

Internal Model Client

Purpose

This document describes Promptkit's internal model-client implementation. The architecture policy owns the library boundary, and the OpenAI-compatible chat integration owns the observable outbound HTTP contract. The framework format reference owns the profile and prompt settings consumed by the client.

The concrete client remains under internal/llm. The root engine assembles it as the default implementation behind Promptkit's public client boundary.

Components And Flow

Client is the provider-neutral generation boundary consumed by later orchestration. OpenAICompatibleClient is the built-in implementation. It uses internal domain values for rendered prompts, execution targets, structured output, responses, and token usage.

The runner supplies a fully resolved target after applying backend, profile, and request precedence. The client uses its endpoint, credential metadata, generation fields, and extra parameters. BackendID remains routing metadata for the generation boundary and is not mapped into the provider payload.

Construction trims and validates a nonempty configured base URL and clones any supplied http.Client so Promptkit can apply its timeout default without mutating the caller's client. An empty configured base remains valid because a resolved request target may supply the endpoint. Generation then:

1. validates shared execution-setting invariants and the final selected base endpoint;
2. maps the internal request into the OpenAI-compatible chat payload;
3. validates and merges extra parameters;
4. composes /chat/completions through parsed URL path operations;
5. resolves authentication;
6. performs the outbound request under the applicable deadlines; and
7. decodes one strictly framed, size-bounded successful response object and maps its first choice and token usage, or decodes bounded structured non-success detail.

internal/llm owns the set of reserved OpenAI-compatible request fields used when validating extra parameters. Backend registration consumes the same rule without making the model client depend on registry configuration.

The implementation has no retry loop, tool-call support, provider catalog, inbound HTTP behavior, or durable session store.

Prepared Generation

For RunPrepared, the runner supplies the model client with the target, rendered messages, and structured-output constraint retained by executable preparation. Execution does not reopen or rerender consumer sources.

Before backend admission, the runner rechecks that the frozen credential environment-variable name is available. The handle does not retain the environment value; the model client resolves the value visible when generation begins. A direct request key remains in private execution state only until the claimed execution finishes or an unclaimed handle is discarded. Exact public ownership and redaction semantics belong to the PreparedExecution GoDoc.

Failure Categories

The package preserves distinct error identities for invalid client configuration, invalid generation requests, request execution failures, non-success provider statuses, and malformed successful responses. Provider response bodies are not included in non-success errors.

Invalid nonempty configured endpoints are configuration failures. A missing or invalid final selected endpoint is an invalid generation request and is rejected before transport.

Successful response bodies have a fixed 16 MiB limit enforced by declared length and by reading at most one byte beyond the boundary. The decoder accepts exactly one JSON object plus trailing whitespace and EOF. Size overflow, truncation, malformed JSON, trailing data, and a second value are malformed responses with no partial result or provider content in the error. Every body is closed, and an unbounded oversized stream is not drained.

For a non-success response, ProviderHTTPError retains the HTTP status and only normalized detail from the bounded recognized envelope. It retains ErrUnexpectedStatus through unwrapping. The client owns response closure; its bounded reader and parser never close or drain a body themselves. The root facade converts this concrete internal error into the public GenerationError, while arbitrary injected-client errors continue through the ordinary generation-error mapping unchanged.

An http.Client.Do failure is represented by a redacting multi-cause error: the package request-failure sentinel and the exact returned transport error are both available through errors.Is and errors.As, while the rendered text does not expose the endpoint, headers, request content, credential, transport detail, or provider body. Caller cancellation retains context.Canceled; caller deadlines, generation deadlines, and whole-request client timeouts retain context.DeadlineExceeded. The runner adds its generation category without discarding those identities or depending on HTTP status mapping.

Test Ownership

The OpenAI-compatible client tests own configuration, client cloning, deterministic deadline precedence, authentication, request and response mapping, malformed data, error identity, cancellation, endpoint selection and composition, pre-transport rejection, and bounded single-document successful-response framing, closure, and response-body suppression. The focused provider HTTP error tests own envelope parsing, normalization, and bounded-reader cases; their transport tests own non-success response closure and integration. Root transport contract tests own public GenerationError conversion, while also verifying that resolved backend settings reach this client without serializing backend identity and that ordinary-run cancellation retains its public generation and context identities. All use local test servers or controlled test transports; the default suite makes no live or paid provider requests.