LLM Runtime Internals

internal/framework/llm is Notarius’s provider-independent structured completion boundary. It adapts framework requests to PromptKit, bounds provider calls, assembles registered prompt and schema assets, records selected profiles, and redacts provider errors. The architectural boundary is defined in Architecture; profile sources, credentials, and concurrency settings belong in Configuration and Configuration.

Structured Completion Boundary

Modules and LLM-backed validators depend only on contracts.StructuredLLMClient. A completion request supplies a prompt ID and version, optional profile and session IDs, named input material, variables, and a caller-owned decode target. The successful response returns the validated raw structured bytes together with non-secret provider, model, profile, and token metadata.

The caller owns the domain behavior: it chooses the prompt, prepares inputs, selects the private response schema, and interprets the decoded result. The adapter does not own source evidence, artifact conversion, normalization, or durable schemas. Those responsibilities remain with the module and its integration contract.

PromptKitClient validates the request target and prompt identity, maps each named material to a PromptKit inline artifact while preserving its origin URI, passes the supplied request session through to PromptKit's direct per-run session field, retains the same value as the session_id prompt variable for maintained prompt compatibility, and forwards profile selection. It does not derive or replace session values; the CLI owns that policy. It then creates one frozen prepared execution, captures its caller-owned credential-redacted details for debug material, and executes that exact snapshot through PromptKit's prepared-execution boundary. The direct field is authoritative for provider session behavior. A session ID is a stable, non-secret correlation identifier and may be exposed to providers and provider observability. The adapter returns PromptKit’s validated raw bytes rather than re-encoding the decoded target. An empty optional material is represented as one space so its named input is retained by PromptKit.

Client construction may also receive a run-wide reasoning-effort override from the CLI factory boundary. The adapter copies the caller-owned pointer and creates a fresh PromptKit execution override for each request: a nil pointer inherits the selected profile, a non-empty value replaces it, and an empty value clears inherited reasoning. The CLI's mutually exclusive --reasoning-effort and --clear-reasoning-effort controls select those states. With neither flag, profile behavior remains unchanged. Because production constructs one shared client, the selected state applies uniformly to module calls, retries, and LLM-backed validators for the whole run.

An empty request profile lets the prompt select its configured default. Before a run begins, the CLI asks the adapter to inspect every explicit profile on the resolved selected LLM-backed bindings and validators, including inherited pipeline profiles. Inspection resolves the profile and its selected backend and target without loading a prompt, reading credentials, admitting capacity, or contacting a provider, so a missing or invalid explicit profile fails before stage execution while a valid api_key_env may remain unset. Calls record the profile actually selected by PromptKit. The recorder trims and deduplicates non-secret profile identity, provider, model, selected backend ID, and effective reasoning values for manifest use. Entries that differ in backend or reasoning remain distinct and deterministically ordered. Endpoint-only profiles retain an empty backend ID, which the published JSON omits. Successful completion responses and recorded profile manifests identify the adapter provider as promptkit.

The CLI's profile-inspection engine and the production adapter use the same profile-source construction to apply the configured profile directory or file, the optional registered fallback profile assets, and the optional conventional local backend. Preflight therefore resolves the same profile sources and backend membership as runtime without performing generation. Fallback assets are mounted only when at least one source is registered. The production D&D registrar contributes its dnd-extraction fallback, and the maintained D&D prompts select that logical ID by default. PromptKit owns source precedence and profile parsing: an operator-provided matching profile takes precedence over a fallback profile without Notarius merging either document. When the registration is absent, a profile selecting backend: local fails inspection instead of falling back to a built-in or endpoint-only target.

Before execution, the adapter also contributes a non-secret checkpoint fingerprint for the effective PromptKit profile source. It combines the identity of PromptKit's compiled-in profile catalog with a deterministic digest of every YAML profile in the configured profile directory, or of the configured profile file, and a deterministic digest of the flattened fallback profile assets. The fingerprint contains neither profile content nor source paths. It covers inherited pipeline profiles, explicit binding profiles, and prompt-selected defaults, so changing a model or other profile setting cannot reuse checkpoints created under the prior profile source. This cache identity is independent of durable profile provenance: run manifests continue to list only profiles actually observed during LLM calls. When the local backend is registered, a second fingerprint hashes its trimmed endpoint behind a stable marker. Changing that semantic execution target invalidates checkpoint reuse. The raw endpoint is not stored in checkpoint identity, and the local concurrency limit is excluded because it changes scheduling rather than execution semantics.

Shared Provider-Call Limit

Production construction creates one PromptKit client and wraps it in one scheduled client. The scheduler has a fixed, positive permit limit, serves queued calls in FIFO order, and removes a queued call when its context is cancelled. A granted permit is released exactly once on every completion path.

The scheduled wrapper surrounds every CompleteStructured call, so concurrent lanes, pipeline retries, and LLM-backed validators share the same provider-call ceiling. This ceiling is independent of pipeline worker concurrency; changing worker counts cannot exceed the configured LLM limit. The configuration field and its effective default are owned by Configuration.

PromptKit applies a second, independent admission limit when the selected profile names a limited backend. It sits beneath the Notarius scheduled client, so it may narrow but cannot expand the application-wide limit. Built-in OpenRouter profiles select PromptKit's reserved backend and its upstream capacity policy. A positive configured local-backend limit bounds active local generations inside PromptKit; zero leaves that backend unlimited there. Endpoint-only profiles do not select a PromptKit backend and remain limited only by the Notarius scheduler.

Prompt And Schema Assets

An AssetRegistry collects prompt, schema, and optional fallback-profile filesystems from production module families. It flattens registered roots into the corresponding PromptKit filesystems and rejects invalid roots, unreadable assets, duplicate paths, and missing prompt or schema files during preparation. Fallback assets receive a safe content digest for checkpoint identity; raw paths and bytes are never included. The framework’s promptfs helper combines module-owned prompt files with reusable domain fragments without making the framework depend on D&D content.

Each LLM-backed module owns its prompt declaration, package-specific assets, and private response schema. Shared D&D wording is owned by the D&D shared asset package; the detailed D&D conventions are in D&D Module Internals. The mounted prompt assets used by a module also determine its prompt fingerprint. Schema loaders validate JSON, attach identity and digest metadata, make defensive copies, and expose diagnostics without raw schema bytes.

Private response schemas validate a model transport envelope. They are not the durable artifact schema and should not be documented as an external wire contract. Durable formats and compatibility rules remain in the integration contracts.

Prompt Maintenance And Backend Caching

Prompt message order and shared asset bytes are runtime behavior. Backend cache reuse depends on identical preceding roles, rendered bytes, and cache-control metadata—not merely equivalent meaning. Keep reusable shared assets byte-identical and preserve each prompt’s declared ordering and cache controls when editing it.

For sibling prompts that can reuse the same source material, order universal shared context first, request source material next, and module-specific suffixes last. Put a cache boundary at a reusable prefix that is useful to the backend. Redundant intermediate cache boundaries do not extend that reusable prefix and add no value.

Prompt-family owners may choose a different sequence when their inputs and reuse pattern differ. The D&D family’s extraction, scene-chunking, and NPC normalization policies are maintained in D&D Module Internals. Do not add tests that enforce prompt prose; prompt tests should verify the meaningful input placement and cache controls of the prompt being changed.

Validation, Repair, And Retries

PromptKit performs prompt rendering, provider execution, and the prompt’s structured-output validation. The adapter reports an empty result, validation failure, empty structured body, or decode failure as ErrInvalidStructuredOutput, while retaining the returned raw bytes and debug material when they exist. Provider failures remain operational errors rather than output-validation failures.

When PromptKit rejects backend admission before generation, the adapter maps promptkit.ErrCapacityExceeded to contracts.ErrLLMCapacityExceeded, retaining prompt context and a redacted upstream diagnostic without exposing the PromptKit sentinel or capacity-error type as a framework contract. When supplied, the normalized selected backend ID appears only in that safe application-owned diagnostic context. A canceled caller context takes precedence. The adapter does not retry capacity failures; the pipeline's existing binding attempt policy sees the operational error and decides whether to rerun the complete operation.

Prompt-declared repair is executed within PromptKit’s structured-output flow. The current production D&D prompt manifests set repair attempts to zero. That setting does not replace pipeline retry behavior: a binding’s configured retry count reruns its stage attempt after an error or rejection, and an exhausted rejection is a recorded output rather than a provider error. The pipeline owns attempt lifecycle, validation chains, and retry diagnostics; see Pipeline Internals and the binding reference.

Timeout Ownership

The caller context remains the outer cancellation authority. PromptKit applies a positive effective generation timeout as an inner request deadline; an explicit zero disables only that generation deadline. The HTTP client timeout is a separate transport-wide cap. Notarius forwards the caller context and does not install another timeout wrapper around PromptKit.

The selected PromptKit profile owns generation settings. Notarius binding retries remain outside the adapter and repeat the complete module operation and validation chain. PromptKit does not add a provider retry loop. Operator-facing behavior is summarized in Operations, and the pinned upstream contract is identified in PromptKit Integration.

Observability And Redaction

When debug recording is enabled, the pipeline decorates the shared client. The wrapper records prepared prompt and response material, timing, selected profile and backend, effective model parameters, and call identifiers in the run’s debug bundle, including material available from a failed structured completion. Effective parameters use PromptKit's stable lower-case JSON field names and may include backend_id. For a successful completion, a debug-write failure is surfaced; when the completion already failed, its call error remains the result. Debug-bundle location, retention, and handling are operational concerns documented in Operations.

Run manifests receive selected profile summaries, including optional effective backend and reasoning provenance, and component identities—not prompt, schema, source, reference, or response content. The published field semantics belong to the JSON output contract. Provider error text is wrapped with prompt context and bearer credentials are redacted before it crosses the runtime boundary. Known-secret redaction is available to other runtime collaborators; it does not make prompt or response contents safe for general logging.

Failure Boundaries

Construction fails for missing asset registries, mutually exclusive profile sources, invalid asset registration, or a non-positive scheduler limit.
Preparation failures, unavailable explicit profiles, provider failures, and context cancellation propagate to the calling stage with context.
Backend admission exhaustion is a provider-neutral operational error and is not classified as invalid structured output or validator rejection.
Malformed or schema-invalid provider output is classified separately as invalid structured output so the module or pipeline can apply its own retry and rejection policy.
Domain semantic checks, evidence decisions, and deterministic normalization run outside the provider adapter.

Focused Verification

Read the LLM adapter, scheduler, asset registry, schema loader, and redaction tests when changing this boundary. Prompt changes also require the owning module’s asset tests, and retry or debug changes require focused pipeline or CLI coverage. The focused runtime and D&D checks are:

go test ./internal/framework/llm/... ./internal/modules/dnd/...

14 KiB Raw Blame History Unescape Escape