87 lines
3.6 KiB
Markdown
87 lines
3.6 KiB
Markdown
# LLM Internals
|
|
|
|
## Purpose
|
|
|
|
`internal/llm` defines the provider-neutral `Client` interface and the
|
|
OpenAI-compatible client implementation. The [OpenAI-compatible integration
|
|
contract](../integrations/openai-compatible-chat.md) owns the outbound HTTP wire
|
|
format and protocol behavior.
|
|
|
|
## Construction
|
|
|
|
`NewOpenAICompatibleClient` validates a non-empty configured base URL, records
|
|
an optional default model, and resolves one transport cap. A supplied client
|
|
with a positive timeout supplies that cap; otherwise a positive configured
|
|
timeout is used, then the internal default.
|
|
|
|
When callers supply an `http.Client`, construction clones it rather than
|
|
mutating the caller's instance. A supplied client with a zero or negative
|
|
timeout receives the resolved transport cap in the clone. The client stores the
|
|
trimmed base URL, default model, and cloned client.
|
|
|
|
## Generate Flow
|
|
|
|
`Generate` receives a `domain.GenerateRequest` from the runner:
|
|
|
|
1. validate the effective timeout and choose the request endpoint;
|
|
2. map the domain request to the internal wire-request representation;
|
|
3. validate and flatten extra parameters and encode JSON;
|
|
4. derive a child context when the effective generation timeout is positive,
|
|
then create the HTTP request with that context;
|
|
5. prefer a direct API key, otherwise resolve the configured key environment
|
|
variable;
|
|
6. execute with the construction-time HTTP client, reject non-success status
|
|
responses without returning
|
|
provider response bodies; and
|
|
7. decode the response subset into `domain.GenerateResponse`.
|
|
|
|
`openAIChatRequestFromGenerateRequest` is the conversion boundary for effective
|
|
model defaults, explicit numeric-presence state, rendered messages, structured
|
|
output, and session-ID validation. `openAIChatRequestPayload` protects reserved
|
|
fields and JSON encoding before an HTTP call. The external payload shape is
|
|
defined only in the [integration contract](../integrations/openai-compatible-chat.md).
|
|
|
|
## Error Categories
|
|
|
|
The package uses these internal sentinels:
|
|
|
|
- `ErrInvalidConfig` for invalid client construction;
|
|
- `ErrInvalidRequest` for invalid effective generation input;
|
|
- `ErrRequestFailed` for request construction or transport failures;
|
|
- `ErrUnexpectedStatus` for non-success HTTP responses; and
|
|
- `ErrMalformedResponse` for invalid or incomplete successful-response data.
|
|
|
|
The runner maps an invalid LLM request to its invalid-request category and
|
|
other LLM failures to its generation category. Adapters then apply their public
|
|
error contracts.
|
|
|
|
## Package-Local Guarantees
|
|
|
|
- The default-model fallback happens before wire encoding.
|
|
- Per-generation timeout handling derives a request context; it never replaces
|
|
or mutates the configured HTTP client's transport cap.
|
|
- Direct API keys take precedence over environment lookup within this client.
|
|
- Provider response bodies are discarded for non-success status responses.
|
|
- The client does not implement retries, tool calls, or a stateful session
|
|
store.
|
|
|
|
## Verification And Change Recipe
|
|
|
|
Inspect:
|
|
|
|
- `internal/llm/openai_compatible_client_test.go`
|
|
- `internal/usecase/runner_test.go`
|
|
- `internal/adapter/http/handler_test.go`
|
|
|
|
When changing the client:
|
|
|
|
1. keep domain-to-wire mapping inside `internal/llm` and preserve the `Client`
|
|
interface;
|
|
2. test construction, timeout selection, mapping, and error categorization;
|
|
3. update the [OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
|
for any observable wire or protocol change; and
|
|
4. update [runner internals](runner.md) if the client boundary or structured
|
|
output handoff changes.
|
|
|
|
The [testing policy](../policy/testing.md) owns global test sufficiency.
|