117 lines
3.3 KiB
Markdown
117 lines
3.3 KiB
Markdown
# LLM Runtime
|
|
|
|
The implemented LLM runtime lives in `internal/framework/llm`. It provides
|
|
transport-neutral structured completion contracts, an OpenAI-compatible HTTP
|
|
adapter, concurrency scheduling, schema registry helpers, retry behavior, and
|
|
secret redaction.
|
|
|
|
## Contract
|
|
|
|
Modules depend on `contracts.StructuredLLMClient`:
|
|
|
|
```go
|
|
CompleteStructured(ctx, request, out) (response, error)
|
|
```
|
|
|
|
The request contains messages, optional model override, response schema name,
|
|
and response schema JSON. The caller supplies a pointer target for decoded
|
|
structured output.
|
|
|
|
Extractors own prompts and schemas. Provider adapters should not contain
|
|
domain-specific prompt logic.
|
|
|
|
## Production Client Construction
|
|
|
|
`internal/cli` builds the production LLM client from the effective config:
|
|
|
|
1. find the effective LLM profile;
|
|
2. build `OpenAICompatibleClientConfig`;
|
|
3. create an OpenAI-compatible client;
|
|
4. create a scheduler from profile or global concurrency;
|
|
5. wrap the client with `NewScheduledClient`;
|
|
6. return non-secret LLM profile manifest metadata.
|
|
|
|
The current run command requires exactly one distinct effective LLM profile for
|
|
the resolved pipeline.
|
|
|
|
## OpenAI-Compatible Adapter
|
|
|
|
`OpenAICompatibleClient` posts JSON to:
|
|
|
|
```text
|
|
<base_url>/chat/completions
|
|
```
|
|
|
|
It sends:
|
|
|
|
- `model`
|
|
- `messages`
|
|
- `response_format.type = "json_schema"`
|
|
- `response_format.json_schema.name`
|
|
- `response_format.json_schema.strict = true`
|
|
- `response_format.json_schema.schema`
|
|
|
|
If an API key is configured, the adapter sends an `Authorization: Bearer ...`
|
|
header.
|
|
|
|
The adapter accepts assistant content either as a JSON string containing JSON or
|
|
as raw JSON content. It then unmarshals that content into the caller-provided
|
|
target.
|
|
|
|
External wire-contract details belong in the
|
|
[OpenAI-compatible integration doc](../integrations/openai-compatible.md).
|
|
|
|
## Retries And Timeouts
|
|
|
|
The adapter retries:
|
|
|
|
- provider request failures;
|
|
- response read failures;
|
|
- HTTP `429`;
|
|
- HTTP `5xx`;
|
|
- malformed provider envelopes;
|
|
- malformed assistant JSON;
|
|
- structured-output decode failures.
|
|
|
|
Non-retryable `4xx` responses are returned without retry. Request timeout comes
|
|
from the effective LLM profile. Context cancellation is respected.
|
|
|
|
## Scheduler
|
|
|
|
`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
|
|
FIFO queue of waiters. Cancellation removes queued waiters or releases granted
|
|
permits.
|
|
|
|
`NewScheduledClient` wraps any structured LLM client and runs each completion
|
|
inside the scheduler.
|
|
|
|
Effective concurrency is:
|
|
|
|
1. `llm_profiles.<id>.max_concurrency`, when greater than zero;
|
|
2. `concurrency.total_llm`, when greater than zero;
|
|
3. `1`.
|
|
|
|
## Schema Registry
|
|
|
|
The framework schema registry embeds generic test schemas. It also exposes
|
|
helpers for caller-owned schemas:
|
|
|
|
- `LoadResponseSchema`
|
|
- `LookupResponseSchema`
|
|
- `MustLookupResponseSchema`
|
|
- `ResponseSchema.DiagnosticsMap`
|
|
|
|
`DiagnosticsMap` omits raw schema content and includes metadata such as key,
|
|
ID, version, name, and SHA-256.
|
|
|
|
The D&D spell extractor owns and loads its own embedded response schema.
|
|
|
|
## Secret Redaction
|
|
|
|
Provider errors are passed through `ErrorWithSecretsRedacted` with the API key
|
|
and bearer-token value. Config diagnostics use redacted effective config
|
|
payloads.
|
|
|
|
Do not add raw provider request bodies, response bodies, API keys, or prompt
|
|
payloads to diagnostics by default.
|