notarius/docs/internal/llm.md

LLM Runtime

The implemented LLM runtime lives in internal/framework/llm. It provides transport-neutral structured completion contracts, an OpenAI-compatible HTTP adapter, concurrency scheduling, schema registry helpers, retry behavior, and secret redaction.

Contract

Modules depend on contracts.StructuredLLMClient:

CompleteStructured(ctx, request, out) (response, error)

The request contains messages, optional model override, response schema name, and response schema JSON. The caller supplies a pointer target for decoded structured output.

Extractors own prompts and schemas. Provider adapters should not contain domain-specific prompt logic.

Production Client Construction

internal/cli builds the production LLM client from the effective config:

1. find the effective LLM profile;
2. build OpenAICompatibleClientConfig;
3. create an OpenAI-compatible client;
4. create a scheduler from profile or global concurrency;
5. wrap the client with NewScheduledClient;
6. return non-secret LLM profile manifest metadata.

The current run command requires exactly one distinct effective LLM profile for the resolved pipeline.

OpenAI-Compatible Adapter

OpenAICompatibleClient posts JSON to:

<base_url>/chat/completions

It sends:

• model
• messages
• response_format.type = "json_schema"
• response_format.json_schema.name
• response_format.json_schema.strict = true
• response_format.json_schema.schema

If an API key is configured, the adapter sends an Authorization: Bearer ... header.

The adapter accepts assistant content either as a JSON string containing JSON or as raw JSON content. It then unmarshals that content into the caller-provided target.

External wire-contract details belong in the OpenAI-compatible integration doc.

Retries And Timeouts

The adapter retries:

• provider request failures;
• response read failures;
• HTTP 429;
• HTTP 5xx;
• malformed provider envelopes;
• malformed assistant JSON;
• structured-output decode failures.

Non-retryable 4xx responses are returned without retry. Request timeout comes from the effective LLM profile. Context cancellation is respected.

Scheduler

Scheduler bounds concurrent provider calls. It tracks in-flight calls and a FIFO queue of waiters. Cancellation removes queued waiters or releases granted permits.

NewScheduledClient wraps any structured LLM client and runs each completion inside the scheduler.

Effective concurrency is:

1. llm_profiles.<id>.max_concurrency, when greater than zero;
2. concurrency.total_llm, when greater than zero;
3. 1.

Schema Registry

The framework schema registry embeds generic test schemas. It also exposes helpers for caller-owned schemas:

• LoadResponseSchema
• LookupResponseSchema
• MustLookupResponseSchema
• ResponseSchema.DiagnosticsMap

DiagnosticsMap omits raw schema content and includes metadata such as key, ID, version, name, and SHA-256.

The D&D spell extractor owns and loads its own embedded response schema.

Secret Redaction

Provider errors are passed through ErrorWithSecretsRedacted with the API key and bearer-token value. Config diagnostics use redacted effective config payloads.

Do not add raw provider request bodies, response bodies, API keys, or prompt payloads to diagnostics by default.