notarius/docs/internal/llm.md

LLM Runtime Internals

internal/framework/llm implements Notarius's transport boundary for structured completion. It contains the Scriptorium adapter, concurrency scheduler, prompt/schema registries, selected-profile recording, and provider-error redaction.

Provider-neutral ownership rules are defined in Architecture. Profile sources, credentials, and concurrency settings are defined in Configuration.

Structured Contract

Modules and LLM-backed validators depend on contracts.StructuredLLMClient.CompleteStructured. A request identifies a prompt and optional profile/session, supplies named input materials and variables, and provides a caller-owned decoding target. A successful response contains the validated raw structured bytes plus non-secret provider, model, profile, and token metadata.

The caller owns prompt selection, response-schema selection, and interpretation of the decoded result. LLMInputMaterial keeps source and reference bytes with their origin metadata so the adapter can pass named artifacts to Scriptorium without exposing Scriptorium types through stage contracts.

Production Construction

internal/cli constructs the production runtime by:

1. allocating the asset registry populated by the generic, Seriatim, and D&D package-family registrars;
2. creating a ScriptoriumClient from the effective profile source;
3. attaching an LLMProfileRecorder;
4. creating a scheduler from the effective concurrency limit;
5. returning a ScheduledClient wrapper;
6. decorating that shared client before preparation when debug recording is enabled; and
7. injecting that one shared client into complete pipeline preparation before the source file is read or the runner is invoked.

The D&D scene chunker and spell, NPC, and combat extractors retain this injected client and use it for every structured completion. Operation requests do not carry an LLM client.

The CLI separately gathers explicit profile IDs from resolved LLM-capable stage and validator bindings. It prepares a small internal check prompt for each ID so missing or invalid profiles fail before pipeline execution. The runtime profile override syntax and scope are defined in the CLI reference; binding rules are defined in Configuration.

Scriptorium Adapter

ScriptoriumClient converts a Notarius request into a Scriptorium RunRequest. It validates the decoding target and prompt identity, maps named input materials to inline artifacts, forwards explicit profile and session context, delegates rendering/provider execution/structured validation, and unmarshals successful JSON into the caller target.

Empty optional input material is represented by a single space so Scriptorium retains the named input. The client returns Scriptorium's validated structured bytes rather than re-encoding the caller target, allowing modules to preserve the runtime result exactly.

Selected profile, provider, model, and token metadata are mapped into the Notarius response. The recorder deduplicates profiles by identity and supplies manifest-safe profile summaries after actual calls; manifest population does not guess the selected prompt default in advance.

Generated-output validation failures and provider failures are wrapped with prompt context. Error strings pass through bearer-token redaction before they cross the runtime boundary.

Scheduling

Scheduler uses a bounded permit count and a FIFO waiter queue. Immediate acquisition increments the in-flight count; queued acquisition waits for a permit or context cancellation. Cancellation removes a queued waiter, while a cancelled waiter that has already received a permit releases it.

ScheduledClient acquires a permit around each structured completion and defers release on every result path. The effective limit and default are configuration facts in Configuration.

This provider-call ceiling is independent of the pipeline's extract worker limit. Concurrent lanes, retries, and validators all use the same scheduled client, so increasing framework workers cannot exceed total_llm. Pipeline dispatch and cancellation mechanics are documented in Pipeline Internals.

Prompt And Schema Assets

AssetRegistry combines caller-owned prompt filesystems under stable prefixes and rejects invalid or conflicting registrations. Production module packages register their own prompt and schema assets; generic framework code contains no D&D prompt content. internal/framework/promptfs provides the domain-neutral filesystem composition helper used to combine module-owned files with shared domain prompt fragments.

The D&D scene chunker and spell, NPC, and combat-turn extractors each declare an ordered prompt asset manifest. The manifest lists the package-owned YAML and Markdown files, then the exact shared fragments rendered by that prompt; the same ordered list drives both filesystem mounting and the prompt fingerprint. Unused shared assets are neither mounted nor fingerprinted. Universal extraction-evidence and output policy lives only in the shared extraction assets; package-owned prompt files retain artifact-specific rules. The scene prompt keeps its separate output rule because it does not render the extraction-evidence asset.

D&D Extraction Prompt Ordering And Cache Boundaries

D&D extraction prompts order messages from the most reusable content to the most variable content. New extraction lanes use these tiers in order:

1. universal shared content, including the system, extraction-evidence, and in-world identity messages;
2. stable campaign or run context shared across lanes, including campaign references;
3. stable subset- and lane-specific context and instructions, including an NPC registry, catalog, task, or extraction instructions when applicable;
4. the chunk transcript as the final user message.

This ordering lets requests reuse the longest identical prefix before the per-chunk transcript changes. Cache reuse requires the preceding message sequence and content to be exactly identical; semantic similarity is not sufficient. Cache boundaries belong at the ends of reusable stable tiers, subject to the provider's cache-boundary limit. The shared identity and campaign-reference messages form the first two extraction boundaries. Spell and combat prompts add a boundary at the shared NPC registry. Each extraction prompt places its final boundary on its lane-specific instructions, immediately before the transcript. The transcript does not carry cache control because no reusable content follows it.

Accordingly, the common prefix of all three extraction prompts is system, extraction evidence, identity, and campaign references. The NPC prompt then renders task, instructions, and transcript. Spell renders the NPC registry, catalog, task, instructions, and transcript. Combat renders immediate resolution, NPC registry, task, instructions, and transcript. The scene chunker is not an extraction lane: it retains its separate system, transcript, campaign-reference, task, and instruction order and marks its transcript and campaign-reference messages ephemeral.

Shared wording belongs in the canonical assets under internal/modules/dnd/shared; extraction packages reference those assets in their manifests instead of copying similar text into package-local files. Package-local assets contain only lane-specific content. An extraction lane may depart from the tier order only when prompt-quality evidence or a provider constraint makes the exception necessary; document the exception and rationale here when it becomes implemented behavior.

Schema helpers load embedded JSON Schema with identity and digest metadata, return defensive copies, and expose a diagnostics map that omits schema bytes. The small framework registry contains only generic test schemas; production schemas remain package-owned.

The spell, NPC, and combat extractors' package-owned prompts declare their structured JSON inputs and private response schemas. Each private response schema remains separate from its durable artifact codec schema; this work does not use shared schema fragments or schema generation. Those private schemas own the transport envelope—required fields, JSON types, nullability, and unknown-field rejection—while deterministic validators own semantic constraints such as enum membership, non-empty values and collections, and positive numbers. The spell extractor's prompt declares a required application/json spell_catalog input and an optional application/json npcs input. The extractor generates the catalog input from its prepared effective catalog as {"spell_names":[...]} using sorted canonical names only. The shared D&D prompt assets include a generic NPC grounding fragment directly after the campaign reference message for spell and combat prompts. When an NPC registry is bound, the domain registry boundary strictly decodes and identity-validates one durable artifact, re-encodes canonical JSON for provenance, and separately generates a names-only prompt projection. The unbound projection is exactly {"npcs":[]}. Prompt input and component-local checkpoint digests cover the projected bytes; manifests retain the optional full registry digest/count rather than names, overlay bytes, registry paths, or source metadata. Combat prompt, response-schema, mapping, normalization, identity, and registry-projection fingerprints remain separate semantic inputs to checkpoint identity.

Debug And Redaction Boundaries

The pipeline may wrap the client with a debug recorder that captures prepared prompt/response material for an explicitly requested debug run. Debug summaries and manifests receive identities, hashes, usage, and selected profile summaries rather than prompt, source, reference, schema, or response content.

The Scriptorium error wrapper removes bearer credential values from surfaced provider errors; RedactSecrets and ErrorWithSecretsRedacted support known secret values elsewhere in the runtime. Config summaries use a separate clone-and-redact path in internal/core/config. These mechanisms implement the security invariant in Architecture; operator handling of debug data is defined in Operations.

Failure Behavior

• Invalid targets, missing prompt IDs, malformed structured output, and Scriptorium failures return contextual errors to the calling module.
• Scheduler construction rejects non-positive limits; acquisition respects context cancellation.
• Asset registration rejects invalid roots, missing content, and path conflicts.
• Schema loading distinguishes missing assets, invalid JSON, and invalid metadata.
• Profile validation errors occur during CLI preparation when an explicit selected ID cannot be prepared.

Tests To Inspect

• internal/framework/llm/scriptorium_client_test.go: adapter mapping and local HTTP integration.
• internal/framework/llm/scheduler_test.go and scheduled_client_test.go: permits, FIFO behavior, cancellation, and wrapper release.
• internal/framework/llm/asset_registry_test.go and schema_registry_test.go: asset composition, validation, and defensive copies.
• internal/framework/llm/secrets_test.go: provider-error redaction.
• internal/cli/run_contract_test.go: profile validation, production client wiring, manifest recording, and debug integration.
• Module-local scriptorium_assets_test.go files: prompt inputs and package asset registration.