scriptorium/docs/internal/llm.md

LLM Internals

Purpose

internal/llm defines the provider-neutral Client interface and the OpenAI-compatible client implementation. The OpenAI-compatible integration contract owns the outbound HTTP wire format and protocol behavior.

Construction

NewOpenAICompatibleClient validates a non-empty configured base URL, records an optional default model, and resolves one transport cap. A supplied client with a positive timeout supplies that cap; otherwise a positive configured timeout is used, then the internal default.

When callers supply an http.Client, construction clones it rather than mutating the caller's instance. A supplied client with a zero or negative timeout receives the resolved transport cap in the clone. The client stores the trimmed base URL, default model, and cloned client.

Generate Flow

Generate receives a domain.GenerateRequest from the runner:

1. validate the effective timeout and choose the request endpoint;
2. map the domain request to the internal wire-request representation;
3. validate and flatten extra parameters and encode JSON;
4. derive a child context when the effective generation timeout is positive, then create the HTTP request with that context;
5. prefer a direct API key, otherwise resolve the configured key environment variable;
6. execute with the construction-time HTTP client, reject non-success status responses without returning provider response bodies; and
7. decode the response subset into domain.GenerateResponse.

openAIChatRequestFromGenerateRequest is the conversion boundary for effective model defaults, explicit numeric-presence state, rendered messages, structured output, and session-ID validation. openAIChatRequestPayload protects reserved fields and JSON encoding before an HTTP call. The external payload shape is defined only in the integration contract.

Error Categories

The package uses these internal sentinels:

• ErrInvalidConfig for invalid client construction;
• ErrInvalidRequest for invalid effective generation input;
• ErrRequestFailed for request construction or transport failures;
• ErrUnexpectedStatus for non-success HTTP responses; and
• ErrMalformedResponse for invalid or incomplete successful-response data.

The runner maps an invalid LLM request to its invalid-request category and other LLM failures to its generation category. Adapters then apply their public error contracts.

Package-Local Guarantees

• The default-model fallback happens before wire encoding.
• Per-generation timeout handling derives a request context; it never replaces or mutates the configured HTTP client's transport cap.
• Direct API keys take precedence over environment lookup within this client.
• Provider response bodies are discarded for non-success status responses.
• The client does not implement retries, tool calls, or a stateful session store.

Verification And Change Recipe

Inspect:

• internal/llm/openai_compatible_client_test.go
• internal/usecase/runner_test.go
• internal/adapter/http/handler_test.go

When changing the client:

1. keep domain-to-wire mapping inside internal/llm and preserve the Client interface;
2. test construction, timeout selection, mapping, and error categorization;
3. update the OpenAI-compatible integration contract for any observable wire or protocol change; and
4. update runner internals if the client boundary or structured output handoff changes.

The testing policy owns global test sufficiency.