scriptorium/docs/internal/llm.md

LLM Internals

Purpose

internal/llm defines the provider-neutral Client interface and the OpenAI-compatible client implementation. The OpenAI-compatible integration contract owns the outbound HTTP wire format and protocol behavior.

Construction

NewOpenAICompatibleClient validates a non-empty configured base URL, records an optional default model, and establishes the default timeout. A non-positive configured timeout uses the internal default.

When callers supply an http.Client, construction clones it rather than mutating the caller's instance. A supplied client with no timeout receives the resolved default in the clone; a supplied non-zero timeout is retained. The client stores the trimmed base URL, default model, timeout, and cloned client.

Generate Flow

Generate receives a domain.GenerateRequest from the runner:

1. validate the effective timeout and choose the request endpoint;
2. map the domain request to the internal wire-request representation;
3. validate and flatten extra parameters, encode JSON, and create the HTTP request;
4. prefer a direct API key, otherwise resolve the configured key environment variable;
5. derive a request HTTP client when an explicit timeout changes the configured client;
6. execute the request, reject non-success status responses without returning provider response bodies; and
7. decode the response subset into domain.GenerateResponse.

openAIChatRequestFromGenerateRequest is the conversion boundary for effective model defaults, explicit numeric-presence state, rendered messages, structured output, and session-ID validation. openAIChatRequestPayload protects reserved fields and JSON encoding before an HTTP call. The external payload shape is defined only in the integration contract.

Error Categories

The package uses these internal sentinels:

• ErrInvalidConfig for invalid client construction;
• ErrInvalidRequest for invalid effective generation input;
• ErrRequestFailed for request construction or transport failures;
• ErrUnexpectedStatus for non-success HTTP responses; and
• ErrMalformedResponse for invalid or incomplete successful-response data.

The runner maps an invalid LLM request to its invalid-request category and other LLM failures to its generation category. Adapters then apply their public error contracts.

Package-Local Guarantees

• The default-model fallback happens before wire encoding.
• Per-request timeout handling clones a configured HTTP client when needed; it does not mutate shared client state.
• Direct API keys take precedence over environment lookup within this client.
• Provider response bodies are discarded for non-success status responses.
• The client does not implement retries, tool calls, or a stateful session store.

Verification And Change Recipe

Inspect:

• internal/llm/openai_compatible_client_test.go
• internal/usecase/runner_test.go
• internal/adapter/http/handler_test.go

When changing the client:

1. keep domain-to-wire mapping inside internal/llm and preserve the Client interface;
2. test construction, timeout selection, mapping, and error categorization;
3. update the OpenAI-compatible integration contract for any observable wire or protocol change; and
4. update runner internals if the client boundary or structured output handoff changes.

The testing policy owns global test sufficiency.