notarius/docs/internal/llm.md

LLM Runtime

The implemented LLM runtime lives in internal/framework/llm. It provides transport-neutral structured completion contracts, a Scriptorium-backed production client, an OpenAI-compatible HTTP adapter retained for legacy tests and helpers, concurrency scheduling, prompt/schema asset registration, schema registry helpers, and secret redaction.

Contract

Modules depend on contracts.StructuredLLMClient:

CompleteStructured(ctx, request, out) (response, error)

The request contains prompt ID/version, profile ID, session ID, prompt input materials, variables, and legacy rendered-message/schema fields used by modules that have not yet moved to prompt-asset execution. The caller supplies a pointer target for decoded structured output.

Modules that call the LLM own their prompts, schemas, prompt IDs, validators, and domain-specific interpretation. Provider adapters should not contain domain-specific prompt logic.

Production Client Construction

internal/cli builds the production LLM client from the effective config:

1. collect production Scriptorium prompt and schema assets from module packages;
2. create a Scriptorium-backed structured client using effective Scriptorium profile source settings;
3. create a scheduler from global LLM concurrency;
4. wrap the client with NewScheduledClient;
5. let the runtime report non-secret profile manifest metadata after calls.

The runtime records the actual selected Scriptorium profile, provider, and model used during execution. Manifest population does not rely on a precomputed profile ID before pipeline execution.

Scriptorium Adapter

ScriptoriumClient implements contracts.StructuredLLMClient by converting Notarius prompt requests into Scriptorium RunRequest values. It:

• validates the caller output target and prompt ID;
• converts LLMInputMaterial values into inline Scriptorium artifacts;
• passes session_id through Scriptorium variables when present;
• sends explicit profile IDs only when the request supplies one;
• lets Scriptorium render prompts, call the configured provider, and validate structured output;
• unmarshals successful JSON into the caller-provided target;
• maps token usage and selected profile/model metadata into the Notarius response and manifest profile recorder.

Generated-output validation failures are returned as Notarius errors. Provider and runtime errors are wrapped with prompt context and bearer tokens are redacted from error strings.

OpenAI-Compatible Adapter

OpenAICompatibleClient posts JSON to:

<base_url>/chat/completions

It sends:

• model
• messages
• response_format.type = "json_schema"
• response_format.json_schema.name
• response_format.json_schema.strict = true
• response_format.json_schema.schema

If an API key is configured, the adapter sends an Authorization: Bearer ... header.

The adapter accepts assistant content either as a JSON string containing JSON or as raw JSON content. It then unmarshals that content into the caller-provided target.

External wire-contract details belong in the OpenAI-compatible integration doc.

Retries And Timeouts

The adapter retries:

• provider request failures;
• response read failures;
• HTTP 429;
• HTTP 5xx;
• malformed provider envelopes;
• malformed assistant JSON;
• structured-output decode failures.

Non-retryable 4xx responses are returned without retry. Context cancellation is respected.

Scheduler

Scheduler bounds concurrent provider calls. It tracks in-flight calls and a FIFO queue of waiters. Cancellation removes queued waiters or releases granted permits.

NewScheduledClient wraps any structured LLM client and runs each completion inside the scheduler.

Effective concurrency is:

1. concurrency.total_llm, when greater than zero;
2. 1.

Schema Registry

The framework schema registry embeds generic test schemas. It also exposes helpers for caller-owned schemas:

• LoadResponseSchema
• LookupResponseSchema
• MustLookupResponseSchema
• ResponseSchema.DiagnosticsMap

DiagnosticsMap omits raw schema content and includes metadata such as key, ID, version, name, and SHA-256.

Production modules own and register their Scriptorium prompt and schema assets. Framework packages may collect those files but must not contain D&D-specific prompt content.

Secret Redaction

Provider errors are redacted before surfacing through the Scriptorium-backed client. Config diagnostics use redacted effective config payloads.

Do not add raw provider request bodies, response bodies, API keys, or prompt payloads to diagnostics by default.