notarius/docs/internal/llm.md

LLM Runtime

The implemented LLM runtime lives in internal/framework/llm. It provides transport-neutral structured completion contracts, a Scriptorium-backed production client, concurrency scheduling, prompt/schema asset registration, schema registry helpers, and secret redaction.

Contract

Modules depend on contracts.StructuredLLMClient:

CompleteStructured(ctx, request, out) (response, error)

The request contains prompt ID/version, profile ID, session ID, prompt input materials, and variables. The caller supplies a pointer target for decoded structured output.

Modules that call the LLM own their prompts, schemas, prompt IDs, validators, and domain-specific interpretation. Provider adapters should not contain domain-specific prompt logic.

Prompt input materials carry source or reference bytes with optional origin metadata. The Scriptorium-backed runtime receives them as named artifacts rather than rendered prompt strings owned by Notarius modules.

Production Client Construction

internal/cli builds the production LLM client from the effective config:

1. collect production Scriptorium prompt and schema assets from module packages;
2. create a Scriptorium-backed structured client using effective Scriptorium profile source settings from scriptorium.profile_dir or scriptorium.profile_file;
3. create a scheduler from global LLM concurrency;
4. wrap the client with NewScheduledClient;
5. let the runtime report non-secret profile manifest metadata after calls.

The runtime records the actual selected Scriptorium profile, provider, and model used during execution. Manifest population does not rely on a precomputed profile ID before pipeline execution.

Scriptorium Adapter

ScriptoriumClient implements contracts.StructuredLLMClient by converting Notarius prompt requests into Scriptorium RunRequest values. It:

• validates the caller output target and prompt ID;
• converts LLMInputMaterial values into inline Scriptorium artifacts, using a single space for empty material so optional blank references remain explicit;
• passes session_id through Scriptorium variables and request metadata when present;
• sends explicit profile IDs only when the request supplies one;
• lets Scriptorium render prompts, call the configured provider, and validate structured output;
• unmarshals successful JSON into the caller-provided target;
• maps token usage and selected profile/model metadata into the Notarius response and manifest profile recorder.

Generated-output validation failures are returned as Notarius errors. Provider and runtime errors are wrapped with prompt context and bearer tokens are redacted from error strings. Prompt text, raw source input, reference content, schema JSON, API keys, and bearer tokens are not added to default diagnostics or run manifests.

Scheduler

Scheduler bounds concurrent provider calls. It tracks in-flight calls and a FIFO queue of waiters. Cancellation removes queued waiters or releases granted permits.

NewScheduledClient wraps any structured LLM client and runs each completion inside the scheduler.

Effective concurrency is:

1. concurrency.total_llm, when greater than zero;
2. 1.

Schema Registry

The framework schema registry embeds generic test schemas. It also exposes helpers for caller-owned schemas:

• LoadResponseSchema
• LookupResponseSchema
• MustLookupResponseSchema
• ResponseSchema.DiagnosticsMap

DiagnosticsMap omits raw schema content and includes metadata such as key, ID, version, name, and SHA-256.

Production modules own and register their Scriptorium prompt and schema assets. Framework packages may collect those files but must not contain D&D-specific prompt content.

Secret Redaction

Provider errors are redacted before surfacing through the Scriptorium-backed client. Config diagnostics use redacted effective config payloads.

Do not add raw provider request bodies, response bodies, API keys, or prompt payloads to diagnostics by default.