Add Scriptorium-backed LLM runtime
This commit is contained in:
@@ -1,9 +1,10 @@
|
||||
# LLM Runtime
|
||||
|
||||
The implemented LLM runtime lives in `internal/framework/llm`. It provides
|
||||
transport-neutral structured completion contracts, an OpenAI-compatible HTTP
|
||||
adapter, concurrency scheduling, schema registry helpers, retry behavior, and
|
||||
secret redaction.
|
||||
transport-neutral structured completion contracts, a Scriptorium-backed
|
||||
production client, an OpenAI-compatible HTTP adapter retained for legacy tests
|
||||
and helpers, concurrency scheduling, prompt/schema asset registration, schema
|
||||
registry helpers, and secret redaction.
|
||||
|
||||
## Contract
|
||||
|
||||
@@ -13,26 +14,48 @@ Modules depend on `contracts.StructuredLLMClient`:
|
||||
CompleteStructured(ctx, request, out) (response, error)
|
||||
```
|
||||
|
||||
The request contains messages, optional model override, response schema name,
|
||||
and response schema JSON. The caller supplies a pointer target for decoded
|
||||
structured output.
|
||||
The request contains prompt ID/version, profile ID, session ID, prompt input
|
||||
materials, variables, and legacy rendered-message/schema fields used by modules
|
||||
that have not yet moved to prompt-asset execution. The caller supplies a pointer
|
||||
target for decoded structured output.
|
||||
|
||||
Modules that call the LLM own their prompts and schemas. Provider adapters
|
||||
should not contain domain-specific prompt logic.
|
||||
Modules that call the LLM own their prompts, schemas, prompt IDs, validators,
|
||||
and domain-specific interpretation. Provider adapters should not contain
|
||||
domain-specific prompt logic.
|
||||
|
||||
## Production Client Construction
|
||||
|
||||
`internal/cli` builds the production LLM client from the effective config:
|
||||
|
||||
1. find the effective LLM profile;
|
||||
2. build `OpenAICompatibleClientConfig`;
|
||||
3. create an OpenAI-compatible client;
|
||||
4. create a scheduler from profile or global concurrency;
|
||||
5. wrap the client with `NewScheduledClient`;
|
||||
6. return non-secret LLM profile manifest metadata.
|
||||
1. collect production Scriptorium prompt and schema assets from module packages;
|
||||
2. create a Scriptorium-backed structured client using effective Scriptorium
|
||||
profile source settings;
|
||||
3. create a scheduler from global LLM concurrency;
|
||||
4. wrap the client with `NewScheduledClient`;
|
||||
5. let the runtime report non-secret profile manifest metadata after calls.
|
||||
|
||||
The current run command requires exactly one distinct effective LLM profile for
|
||||
the resolved pipeline.
|
||||
The runtime records the actual selected Scriptorium profile, provider, and model
|
||||
used during execution. Manifest population does not rely on a precomputed
|
||||
profile ID before pipeline execution.
|
||||
|
||||
## Scriptorium Adapter
|
||||
|
||||
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting
|
||||
Notarius prompt requests into Scriptorium `RunRequest` values. It:
|
||||
|
||||
- validates the caller output target and prompt ID;
|
||||
- converts `LLMInputMaterial` values into inline Scriptorium artifacts;
|
||||
- passes `session_id` through Scriptorium variables when present;
|
||||
- sends explicit profile IDs only when the request supplies one;
|
||||
- lets Scriptorium render prompts, call the configured provider, and validate
|
||||
structured output;
|
||||
- unmarshals successful JSON into the caller-provided target;
|
||||
- maps token usage and selected profile/model metadata into the Notarius
|
||||
response and manifest profile recorder.
|
||||
|
||||
Generated-output validation failures are returned as Notarius errors. Provider
|
||||
and runtime errors are wrapped with prompt context and bearer tokens are
|
||||
redacted from error strings.
|
||||
|
||||
## OpenAI-Compatible Adapter
|
||||
|
||||
@@ -73,8 +96,8 @@ The adapter retries:
|
||||
- malformed assistant JSON;
|
||||
- structured-output decode failures.
|
||||
|
||||
Non-retryable `4xx` responses are returned without retry. Request timeout comes
|
||||
from the effective LLM profile. Context cancellation is respected.
|
||||
Non-retryable `4xx` responses are returned without retry. Context cancellation
|
||||
is respected.
|
||||
|
||||
## Scheduler
|
||||
|
||||
@@ -87,9 +110,8 @@ inside the scheduler.
|
||||
|
||||
Effective concurrency is:
|
||||
|
||||
1. `llm_profiles.<id>.max_concurrency`, when greater than zero;
|
||||
2. `concurrency.total_llm`, when greater than zero;
|
||||
3. `1`.
|
||||
1. `concurrency.total_llm`, when greater than zero;
|
||||
2. `1`.
|
||||
|
||||
## Schema Registry
|
||||
|
||||
@@ -104,13 +126,14 @@ helpers for caller-owned schemas:
|
||||
`DiagnosticsMap` omits raw schema content and includes metadata such as key,
|
||||
ID, version, name, and SHA-256.
|
||||
|
||||
The D&D spell extractor owns and loads its own embedded response schema.
|
||||
Production modules own and register their Scriptorium prompt and schema assets.
|
||||
Framework packages may collect those files but must not contain D&D-specific
|
||||
prompt content.
|
||||
|
||||
## Secret Redaction
|
||||
|
||||
Provider errors are passed through `ErrorWithSecretsRedacted` with the API key
|
||||
and bearer-token value. Config diagnostics use redacted effective config
|
||||
payloads.
|
||||
Provider errors are redacted before surfacing through the Scriptorium-backed
|
||||
client. Config diagnostics use redacted effective config payloads.
|
||||
|
||||
Do not add raw provider request bodies, response bodies, API keys, or prompt
|
||||
payloads to diagnostics by default.
|
||||
|
||||
@@ -41,8 +41,9 @@ production modules.
|
||||
- `internal/framework/pipeline`: module registries, module specs, profile
|
||||
resolution, capability checks, run orchestration, warnings, validation, and
|
||||
manifest population.
|
||||
- `internal/framework/llm`: OpenAI-compatible structured-output client,
|
||||
scheduler, schema registry, retries, and secret redaction.
|
||||
- `internal/framework/llm`: Scriptorium-backed structured-output client,
|
||||
prompt/schema asset registry, scheduler, schema registry, retries, and secret
|
||||
redaction.
|
||||
- `internal/framework/prompt`: embedded prompt registry and template rendering.
|
||||
- `internal/framework/validate`: validator decision helpers and cardinality
|
||||
enforcement.
|
||||
|
||||
Reference in New Issue
Block a user