252 lines
14 KiB
Markdown
252 lines
14 KiB
Markdown
# LLM Runtime Internals
|
||
|
||
`internal/framework/llm` is Notarius’s provider-independent structured
|
||
completion boundary. It adapts framework requests to PromptKit, bounds
|
||
provider calls, assembles registered prompt and schema assets, records selected
|
||
profiles, and redacts provider errors. The architectural boundary is defined in
|
||
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||
credentials, and concurrency settings belong in
|
||
[Configuration](../config.md#promptkit-profiles) and
|
||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||
|
||
## Structured Completion Boundary
|
||
|
||
Modules and LLM-backed validators depend only on
|
||
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
|
||
version, optional profile and session IDs, named input material, variables, and
|
||
a caller-owned decode target. The successful response returns the validated raw
|
||
structured bytes together with non-secret provider, model, profile, and token
|
||
metadata.
|
||
|
||
The caller owns the domain behavior: it chooses the prompt, prepares inputs,
|
||
selects the private response schema, and interprets the decoded result. The
|
||
adapter does not own source evidence, artifact conversion, normalization, or
|
||
durable schemas. Those responsibilities remain with the module and its
|
||
[integration contract](../integrations/).
|
||
|
||
`PromptKitClient` validates the request target and prompt identity, maps each
|
||
named material to a PromptKit inline artifact while preserving its origin URI,
|
||
maps the trimmed request session to PromptKit's direct per-run session field,
|
||
retains the same value as the `session_id` prompt variable for maintained
|
||
prompt compatibility, and forwards profile selection. It then creates one
|
||
frozen prepared execution, captures its caller-owned credential-redacted
|
||
details for debug material, and executes that exact snapshot through
|
||
PromptKit's prepared-execution boundary. The direct field
|
||
is authoritative for provider session behavior. A session ID is a stable,
|
||
non-secret correlation identifier and may be exposed to providers and provider
|
||
observability. The adapter returns PromptKit’s validated raw bytes rather than
|
||
re-encoding the decoded target. An empty optional material is represented as
|
||
one space so its named input is retained by PromptKit.
|
||
|
||
Client construction may also receive a run-wide reasoning-effort override from
|
||
the CLI factory boundary. The adapter copies the caller-owned pointer and
|
||
creates a fresh PromptKit execution override for each request: a nil pointer
|
||
inherits the selected profile, a non-empty value replaces it, and an empty
|
||
value clears inherited reasoning. The CLI's mutually exclusive
|
||
`--reasoning-effort` and `--clear-reasoning-effort` controls select those
|
||
states. With neither flag, profile behavior remains unchanged. Because
|
||
production constructs one shared client, the selected state applies uniformly
|
||
to module calls, retries, and LLM-backed validators for the whole run.
|
||
|
||
An empty request profile lets the prompt select its configured default. Before a
|
||
run begins, the CLI asks the adapter to inspect every explicit profile on the
|
||
resolved selected LLM-backed bindings and validators, including inherited
|
||
pipeline profiles. Inspection resolves the profile and its selected backend and
|
||
target without loading a prompt, reading credentials, admitting capacity, or
|
||
contacting a provider, so a missing or invalid explicit profile fails before
|
||
stage execution while a valid `api_key_env` may remain unset. Calls record the
|
||
profile actually selected by PromptKit. The recorder trims and deduplicates
|
||
non-secret profile identity, provider, model, selected backend ID, and
|
||
effective reasoning values for manifest use. Entries that differ in backend or
|
||
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
|
||
retain an empty backend ID, which the published JSON omits. Successful
|
||
completion responses and recorded profile manifests identify the adapter
|
||
provider as `promptkit`.
|
||
|
||
The CLI's profile-inspection engine and the production adapter use the same
|
||
profile-source construction to apply the configured profile directory or file,
|
||
the optional registered fallback profile assets, and the optional conventional
|
||
`local` backend. Preflight therefore resolves the same profile sources and
|
||
backend membership as runtime without performing generation. Fallback assets
|
||
are mounted only when at least one source is registered. The production D&D
|
||
registrar contributes its `dnd-extraction` fallback, and the maintained D&D
|
||
prompts select that logical ID by default. PromptKit owns source precedence and
|
||
profile parsing: an operator-provided matching profile takes precedence over a
|
||
fallback profile without Notarius merging either document.
|
||
When the registration is absent, a profile selecting `backend: local` fails
|
||
inspection instead of falling back to a built-in or endpoint-only target.
|
||
|
||
Before execution, the adapter also contributes a non-secret checkpoint
|
||
fingerprint for the effective PromptKit profile source. It combines the
|
||
identity of PromptKit's compiled-in profile catalog with a deterministic digest
|
||
of every YAML profile in the configured profile directory, or of the configured
|
||
profile file, and a deterministic digest of the flattened fallback profile
|
||
assets. The fingerprint contains neither profile content nor source paths. It
|
||
covers inherited pipeline profiles, explicit binding profiles, and
|
||
prompt-selected defaults, so changing a model or other profile setting cannot
|
||
reuse checkpoints created under the
|
||
prior profile source. This cache identity is independent of durable
|
||
profile provenance: run manifests continue to list only profiles actually
|
||
observed during LLM calls. When the local backend is registered, a second
|
||
fingerprint hashes its trimmed endpoint behind a stable marker. Changing that
|
||
semantic execution target invalidates checkpoint reuse. The raw endpoint is not
|
||
stored in checkpoint identity, and the local concurrency limit is excluded
|
||
because it changes scheduling rather than execution semantics.
|
||
|
||
## Shared Provider-Call Limit
|
||
|
||
Production construction creates one PromptKit client and wraps it in one
|
||
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||
queued calls in FIFO order, and removes a queued call when its context is
|
||
cancelled. A granted permit is released exactly once on every completion path.
|
||
|
||
The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
|
||
lanes, pipeline retries, and LLM-backed validators share the same provider-call
|
||
ceiling. This ceiling is independent of pipeline worker concurrency; changing
|
||
worker counts cannot exceed the configured LLM limit. The configuration field
|
||
and its effective default are owned by
|
||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||
|
||
PromptKit applies a second, independent admission limit when the selected
|
||
profile names a limited backend. It sits beneath the Notarius scheduled client,
|
||
so it may narrow but cannot expand the application-wide limit. Built-in
|
||
OpenRouter profiles select PromptKit's reserved backend and its upstream
|
||
capacity policy. A positive configured local-backend limit bounds active local
|
||
generations inside PromptKit; zero leaves that backend unlimited there.
|
||
Endpoint-only profiles do not select a PromptKit backend and remain limited
|
||
only by the Notarius scheduler.
|
||
|
||
## Prompt And Schema Assets
|
||
|
||
An `AssetRegistry` collects prompt, schema, and optional fallback-profile
|
||
filesystems from production module families. It flattens registered roots into
|
||
the corresponding PromptKit filesystems and rejects invalid roots, unreadable
|
||
assets, duplicate paths, and missing prompt or schema files during preparation.
|
||
Fallback assets receive a safe content digest for checkpoint identity; raw
|
||
paths and bytes are never included. The framework’s `promptfs` helper combines
|
||
module-owned prompt files with reusable domain fragments without making the
|
||
framework depend on D&D content.
|
||
|
||
Each LLM-backed module owns its prompt declaration, package-specific assets,
|
||
and private response schema. Shared D&D wording is owned by the D&D shared
|
||
asset package; the detailed D&D conventions are in
|
||
[D&D Module Internals](dnd.md). The mounted prompt assets used by a module also
|
||
determine its prompt fingerprint. Schema loaders validate JSON, attach identity
|
||
and digest metadata, make defensive copies, and expose diagnostics without raw
|
||
schema bytes.
|
||
|
||
Private response schemas validate a model transport envelope. They are not the
|
||
durable artifact schema and should not be documented as an external wire
|
||
contract. Durable formats and compatibility rules remain in the
|
||
[integration contracts](../integrations/).
|
||
|
||
## Prompt Maintenance And Backend Caching
|
||
|
||
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
||
reuse depends on identical preceding roles, rendered bytes, and cache-control
|
||
metadata—not merely equivalent meaning. Keep reusable shared assets
|
||
byte-identical and preserve each prompt’s declared ordering and cache controls
|
||
when editing it.
|
||
|
||
For sibling prompts that can reuse the same source material, order universal
|
||
shared context first, request source material next, and module-specific
|
||
suffixes last. Put a cache boundary at a reusable prefix that is useful to the
|
||
backend. Redundant intermediate cache boundaries do not extend that reusable
|
||
prefix and add no value.
|
||
|
||
Prompt-family owners may choose a different sequence when their inputs and
|
||
reuse pattern differ. The D&D family’s extraction, scene-chunking, and NPC
|
||
normalization policies are maintained in [D&D Module Internals](dnd.md#prompt-construction).
|
||
Do not add tests that enforce prompt prose; prompt tests should verify the
|
||
meaningful input placement and cache controls of the prompt being changed.
|
||
|
||
## Validation, Repair, And Retries
|
||
|
||
PromptKit performs prompt rendering, provider execution, and the prompt’s
|
||
structured-output validation. The adapter reports an empty result, validation
|
||
failure, empty structured body, or decode failure as
|
||
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||
material when they exist. Provider failures remain operational errors rather
|
||
than output-validation failures.
|
||
|
||
When PromptKit rejects backend admission before generation, the adapter maps
|
||
`promptkit.ErrCapacityExceeded` to
|
||
`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted
|
||
upstream diagnostic without exposing the PromptKit sentinel or capacity-error
|
||
type as a framework contract. When supplied, the normalized selected backend
|
||
ID appears only in that safe application-owned diagnostic context. A canceled
|
||
caller context takes precedence. The adapter does not retry capacity failures;
|
||
the pipeline's existing binding attempt policy sees the operational error and
|
||
decides whether to rerun the complete operation.
|
||
|
||
Prompt-declared repair is executed within PromptKit’s structured-output flow.
|
||
The current production D&D prompt manifests set repair attempts to zero. That
|
||
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||
count reruns its stage attempt after an error or rejection, and an exhausted
|
||
rejection is a recorded output rather than a provider error. The pipeline owns
|
||
attempt lifecycle, validation chains, and retry diagnostics; see
|
||
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||
[binding reference](../config.md#module-bindings-and-validators).
|
||
|
||
## Timeout Ownership
|
||
|
||
The caller context remains the outer cancellation authority. PromptKit applies
|
||
a positive effective generation timeout as an inner request deadline; an
|
||
explicit zero disables only that generation deadline. The HTTP client timeout
|
||
is a separate transport-wide cap. Notarius forwards the caller context and
|
||
does not install another timeout wrapper around PromptKit.
|
||
|
||
The selected PromptKit profile owns generation settings. Notarius binding
|
||
retries remain outside the adapter and repeat the complete module operation
|
||
and validation chain. PromptKit does not add a provider retry loop.
|
||
Operator-facing behavior is summarized in
|
||
[Operations](../operations.md#operational-limits), and the pinned upstream
|
||
contract is identified in
|
||
[PromptKit Integration](../integrations/pkg-promptkit.md).
|
||
|
||
## Observability And Redaction
|
||
|
||
When debug recording is enabled, the pipeline decorates the shared client. The
|
||
wrapper records prepared prompt and response material, timing, selected profile
|
||
and backend, effective model parameters, and call identifiers in the run’s
|
||
debug bundle, including material available from a failed structured completion.
|
||
Effective parameters use PromptKit's stable lower-case JSON field names and may
|
||
include `backend_id`. For a successful completion, a debug-write failure is
|
||
surfaced; when the completion already failed, its call error remains the
|
||
result. Debug-bundle location, retention, and handling are operational concerns
|
||
documented in [Operations](../operations.md#debug-bundles).
|
||
|
||
Run manifests receive selected profile summaries, including optional effective
|
||
backend and reasoning provenance, and component identities—not prompt, schema,
|
||
source, reference, or response content. The published field semantics belong
|
||
to the [JSON output contract](../integrations/json-output.md#manifestjson).
|
||
Provider error text is wrapped with prompt context and bearer credentials are
|
||
redacted before it crosses the runtime boundary. Known-secret redaction is
|
||
available to other runtime collaborators; it does not make prompt or response
|
||
contents safe for general logging.
|
||
|
||
## Failure Boundaries
|
||
|
||
- Construction fails for missing asset registries, mutually exclusive profile
|
||
sources, invalid asset registration, or a non-positive scheduler limit.
|
||
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||
context cancellation propagate to the calling stage with context.
|
||
- Backend admission exhaustion is a provider-neutral operational error and is
|
||
not classified as invalid structured output or validator rejection.
|
||
- Malformed or schema-invalid provider output is classified separately as
|
||
invalid structured output so the module or pipeline can apply its own retry
|
||
and rejection policy.
|
||
- Domain semantic checks, evidence decisions, and deterministic normalization
|
||
run outside the provider adapter.
|
||
|
||
## Focused Verification
|
||
|
||
Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
|
||
tests when changing this boundary. Prompt changes also require the owning
|
||
module’s asset tests, and retry or debug changes require focused pipeline or
|
||
CLI coverage. The focused runtime and D&D checks are:
|
||
|
||
~~~sh
|
||
go test ./internal/framework/llm/... ./internal/modules/dnd/...
|
||
~~~
|