140 lines
4.6 KiB
Markdown
140 lines
4.6 KiB
Markdown
# LLM Runtime
|
|
|
|
The implemented LLM runtime lives in `internal/framework/llm`. It provides
|
|
transport-neutral structured completion contracts, a Scriptorium-backed
|
|
production client, an OpenAI-compatible HTTP adapter retained for legacy tests
|
|
and helpers, concurrency scheduling, prompt/schema asset registration, schema
|
|
registry helpers, and secret redaction.
|
|
|
|
## Contract
|
|
|
|
Modules depend on `contracts.StructuredLLMClient`:
|
|
|
|
```go
|
|
CompleteStructured(ctx, request, out) (response, error)
|
|
```
|
|
|
|
The request contains prompt ID/version, profile ID, session ID, prompt input
|
|
materials, variables, and legacy rendered-message/schema fields used by modules
|
|
that have not yet moved to prompt-asset execution. The caller supplies a pointer
|
|
target for decoded structured output.
|
|
|
|
Modules that call the LLM own their prompts, schemas, prompt IDs, validators,
|
|
and domain-specific interpretation. Provider adapters should not contain
|
|
domain-specific prompt logic.
|
|
|
|
## Production Client Construction
|
|
|
|
`internal/cli` builds the production LLM client from the effective config:
|
|
|
|
1. collect production Scriptorium prompt and schema assets from module packages;
|
|
2. create a Scriptorium-backed structured client using effective Scriptorium
|
|
profile source settings;
|
|
3. create a scheduler from global LLM concurrency;
|
|
4. wrap the client with `NewScheduledClient`;
|
|
5. let the runtime report non-secret profile manifest metadata after calls.
|
|
|
|
The runtime records the actual selected Scriptorium profile, provider, and model
|
|
used during execution. Manifest population does not rely on a precomputed
|
|
profile ID before pipeline execution.
|
|
|
|
## Scriptorium Adapter
|
|
|
|
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting
|
|
Notarius prompt requests into Scriptorium `RunRequest` values. It:
|
|
|
|
- validates the caller output target and prompt ID;
|
|
- converts `LLMInputMaterial` values into inline Scriptorium artifacts;
|
|
- passes `session_id` through Scriptorium variables when present;
|
|
- sends explicit profile IDs only when the request supplies one;
|
|
- lets Scriptorium render prompts, call the configured provider, and validate
|
|
structured output;
|
|
- unmarshals successful JSON into the caller-provided target;
|
|
- maps token usage and selected profile/model metadata into the Notarius
|
|
response and manifest profile recorder.
|
|
|
|
Generated-output validation failures are returned as Notarius errors. Provider
|
|
and runtime errors are wrapped with prompt context and bearer tokens are
|
|
redacted from error strings.
|
|
|
|
## OpenAI-Compatible Adapter
|
|
|
|
`OpenAICompatibleClient` posts JSON to:
|
|
|
|
```text
|
|
<base_url>/chat/completions
|
|
```
|
|
|
|
It sends:
|
|
|
|
- `model`
|
|
- `messages`
|
|
- `response_format.type = "json_schema"`
|
|
- `response_format.json_schema.name`
|
|
- `response_format.json_schema.strict = true`
|
|
- `response_format.json_schema.schema`
|
|
|
|
If an API key is configured, the adapter sends an `Authorization: Bearer ...`
|
|
header.
|
|
|
|
The adapter accepts assistant content either as a JSON string containing JSON or
|
|
as raw JSON content. It then unmarshals that content into the caller-provided
|
|
target.
|
|
|
|
External wire-contract details belong in the
|
|
[OpenAI-compatible integration doc](../integrations/openai-compatible.md).
|
|
|
|
## Retries And Timeouts
|
|
|
|
The adapter retries:
|
|
|
|
- provider request failures;
|
|
- response read failures;
|
|
- HTTP `429`;
|
|
- HTTP `5xx`;
|
|
- malformed provider envelopes;
|
|
- malformed assistant JSON;
|
|
- structured-output decode failures.
|
|
|
|
Non-retryable `4xx` responses are returned without retry. Context cancellation
|
|
is respected.
|
|
|
|
## Scheduler
|
|
|
|
`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
|
|
FIFO queue of waiters. Cancellation removes queued waiters or releases granted
|
|
permits.
|
|
|
|
`NewScheduledClient` wraps any structured LLM client and runs each completion
|
|
inside the scheduler.
|
|
|
|
Effective concurrency is:
|
|
|
|
1. `concurrency.total_llm`, when greater than zero;
|
|
2. `1`.
|
|
|
|
## Schema Registry
|
|
|
|
The framework schema registry embeds generic test schemas. It also exposes
|
|
helpers for caller-owned schemas:
|
|
|
|
- `LoadResponseSchema`
|
|
- `LookupResponseSchema`
|
|
- `MustLookupResponseSchema`
|
|
- `ResponseSchema.DiagnosticsMap`
|
|
|
|
`DiagnosticsMap` omits raw schema content and includes metadata such as key,
|
|
ID, version, name, and SHA-256.
|
|
|
|
Production modules own and register their Scriptorium prompt and schema assets.
|
|
Framework packages may collect those files but must not contain D&D-specific
|
|
prompt content.
|
|
|
|
## Secret Redaction
|
|
|
|
Provider errors are redacted before surfacing through the Scriptorium-backed
|
|
client. Config diagnostics use redacted effective config payloads.
|
|
|
|
Do not add raw provider request bodies, response bodies, API keys, or prompt
|
|
payloads to diagnostics by default.
|