notarius/docs/internal/llm.md

LLM Runtime Internals

internal/framework/llm implements Notarius's transport boundary for structured completion. It contains the Scriptorium adapter, concurrency scheduler, prompt/schema registries, selected-profile recording, and provider-error redaction.

Provider-neutral ownership rules are defined in Architecture. Profile sources, credentials, and concurrency settings are defined in Configuration.

Structured Contract

Modules and LLM-backed validators depend on contracts.StructuredLLMClient.CompleteStructured. A request identifies a prompt and optional profile/session, supplies named input materials and variables, and provides a caller-owned decoding target. A successful response contains the validated raw structured bytes plus non-secret provider, model, profile, and token metadata.

The caller owns prompt selection, response-schema selection, and interpretation of the decoded result. LLMInputMaterial keeps source and reference bytes with their origin metadata so the adapter can pass named artifacts to Scriptorium without exposing Scriptorium types through stage contracts.

Production Construction

internal/cli constructs the production runtime by:

1. collecting embedded prompt and response-schema assets from production module packages;
2. creating a ScriptoriumClient from the effective profile source;
3. attaching an LLMProfileRecorder;
4. creating a scheduler from the effective concurrency limit;
5. returning a ScheduledClient wrapper.

The CLI separately gathers explicit profile IDs from resolved LLM-capable stage and validator bindings. It prepares a small internal check prompt for each ID so missing or invalid profiles fail before pipeline execution. The runtime profile override syntax and scope are defined in the CLI reference; binding rules are defined in Configuration.

Scriptorium Adapter

ScriptoriumClient converts a Notarius request into a Scriptorium RunRequest. It validates the decoding target and prompt identity, maps named input materials to inline artifacts, forwards explicit profile and session context, delegates rendering/provider execution/structured validation, and unmarshals successful JSON into the caller target.

Empty optional input material is represented by a single space so Scriptorium retains the named input. The client returns Scriptorium's validated structured bytes rather than re-encoding the caller target, allowing modules to preserve the runtime result exactly.

Selected profile, provider, model, and token metadata are mapped into the Notarius response. The recorder deduplicates profiles by identity and supplies manifest-safe profile summaries after actual calls; manifest population does not guess the selected prompt default in advance.

Generated-output validation failures and provider failures are wrapped with prompt context. Error strings pass through bearer-token redaction before they cross the runtime boundary.

Scheduling

Scheduler uses a bounded permit count and a FIFO waiter queue. Immediate acquisition increments the in-flight count; queued acquisition waits for a permit or context cancellation. Cancellation removes a queued waiter, while a cancelled waiter that has already received a permit releases it.

ScheduledClient acquires a permit around each structured completion and defers release on every result path. The effective limit and default are configuration facts in Configuration.

Prompt And Schema Assets

AssetRegistry combines caller-owned prompt filesystems under stable prefixes and rejects invalid or conflicting registrations. Production module packages register their own prompt and schema assets; generic framework code contains no D&D prompt content.

Schema helpers load embedded JSON Schema with identity and digest metadata, return defensive copies, and expose a diagnostics map that omits schema bytes. The small framework registry contains only generic test schemas; production schemas remain package-owned.

Debug And Redaction Boundaries

The pipeline may wrap the client with a debug recorder that captures prepared prompt/response material for an explicitly enabled debug run. Default diagnostics and manifests receive identities, hashes, usage, and selected profile summaries rather than prompt, source, reference, schema, or response content.

The Scriptorium error wrapper removes bearer credential values from surfaced provider errors; RedactSecrets and ErrorWithSecretsRedacted support known secret values elsewhere in the runtime. Config diagnostics use a separate clone-and-redact path in internal/core/config. These mechanisms implement the security invariant in Architecture; operator handling of debug data is defined in Operations.

Failure Behavior

• Invalid targets, missing prompt IDs, malformed structured output, and Scriptorium failures return contextual errors to the calling module.
• Scheduler construction rejects non-positive limits; acquisition respects context cancellation.
• Asset registration rejects invalid roots, missing content, and path conflicts.
• Schema loading distinguishes missing assets, invalid JSON, and invalid metadata.
• Profile validation errors occur during CLI preparation when an explicit selected ID cannot be prepared.

Tests To Inspect

• internal/framework/llm/scriptorium_client_test.go and scriptorium_api_test.go: adapter mapping and local HTTP integration.
• internal/framework/llm/scheduler_test.go and scheduled_client_test.go: permits, FIFO behavior, cancellation, and wrapper release.
• internal/framework/llm/asset_registry_test.go and schema_registry_test.go: asset composition, validation, and defensive copies.
• internal/framework/llm/secrets_test.go: provider-error redaction.
• internal/cli/run_test.go: profile validation, production client wiring, manifest recording, and debug integration.
• Module-local scriptorium_assets_test.go files: prompt inputs and package asset registration.