150 lines
7.9 KiB
Markdown
150 lines
7.9 KiB
Markdown
# LLM Runtime Internals
|
||
|
||
`internal/framework/llm` is Notarius’s provider-independent structured
|
||
completion boundary. It adapts framework requests to Scriptorium, bounds
|
||
provider calls, assembles registered prompt and schema assets, records selected
|
||
profiles, and redacts provider errors. The architectural boundary is defined in
|
||
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||
credentials, and concurrency settings belong in
|
||
[Configuration](../config.md#scriptorium-profiles) and
|
||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||
|
||
## Structured Completion Boundary
|
||
|
||
Modules and LLM-backed validators depend only on
|
||
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
|
||
version, optional profile and session IDs, named input material, variables, and
|
||
a caller-owned decode target. The successful response returns the validated raw
|
||
structured bytes together with non-secret provider, model, profile, and token
|
||
metadata.
|
||
|
||
The caller owns the domain behavior: it chooses the prompt, prepares inputs,
|
||
selects the private response schema, and interprets the decoded result. The
|
||
adapter does not own source evidence, artifact conversion, normalization, or
|
||
durable schemas. Those responsibilities remain with the module and its
|
||
[integration contract](../integrations/).
|
||
|
||
`ScriptoriumClient` validates the request target and prompt identity, maps each
|
||
named material to a Scriptorium inline artifact while preserving its origin URI,
|
||
forwards session and profile selection, then prepares and runs the prompt. It
|
||
returns Scriptorium’s validated raw bytes rather than re-encoding the decoded
|
||
target. An empty optional material is represented as one space so its named
|
||
input is retained by Scriptorium.
|
||
|
||
An empty request profile lets the prompt select its configured default. The CLI
|
||
prepares every explicitly selected binding profile before a run begins, so a
|
||
missing explicit profile fails before stage execution. Calls record the profile
|
||
actually selected by Scriptorium; the recorder deduplicates non-secret profile
|
||
identity, provider, and model values for manifest use.
|
||
|
||
## Shared Provider-Call Limit
|
||
|
||
Production construction creates one Scriptorium client and wraps it in one
|
||
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||
queued calls in FIFO order, and removes a queued call when its context is
|
||
cancelled. A granted permit is released exactly once on every completion path.
|
||
|
||
The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
|
||
lanes, pipeline retries, and LLM-backed validators share the same provider-call
|
||
ceiling. This ceiling is independent of pipeline worker concurrency; changing
|
||
worker counts cannot exceed the configured LLM limit. The configuration field
|
||
and its effective default are owned by
|
||
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||
|
||
## Prompt And Schema Assets
|
||
|
||
An `AssetRegistry` collects prompt and schema filesystems from production module
|
||
families. It flattens registered roots into the Scriptorium filesystems and
|
||
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
||
or schema files during preparation. The framework’s `promptfs` helper combines
|
||
module-owned prompt files with reusable domain fragments without making the
|
||
framework depend on D&D content.
|
||
|
||
Each LLM-backed module owns its prompt declaration, package-specific assets,
|
||
and private response schema. Shared D&D wording is owned by the D&D shared
|
||
asset package; the detailed D&D conventions are in
|
||
[D&D Module Internals](dnd.md). The mounted prompt assets used by a module also
|
||
determine its prompt fingerprint. Schema loaders validate JSON, attach identity
|
||
and digest metadata, make defensive copies, and expose diagnostics without raw
|
||
schema bytes.
|
||
|
||
Private response schemas validate a model transport envelope. They are not the
|
||
durable artifact schema and should not be documented as an external wire
|
||
contract. Durable formats and compatibility rules remain in the
|
||
[integration contracts](../integrations/).
|
||
|
||
## Prompt Maintenance And Backend Caching
|
||
|
||
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
||
reuse depends on the same preceding messages and content, not merely equivalent
|
||
meaning. Keep reusable shared assets byte-identical and keep stable material
|
||
before the inputs that vary per request wherever a prompt’s declared sequence
|
||
supports caching. Preserve the existing manifest order and cache-control hints
|
||
when editing a prompt.
|
||
|
||
D&D extraction manifests place the changing chunk transcript at the end of the
|
||
prompt after their reusable context. Scene chunking and NPC normalization use
|
||
their own declared message sequences because their inputs and work differ. The
|
||
family-specific asset and ordering rules belong in [D&D Module Internals](dnd.md).
|
||
Do not add tests that enforce a fixed message-prefix length; prompt-asset tests
|
||
should instead verify the meaningful asset sequence, inputs, and cache controls
|
||
of the prompt being changed.
|
||
|
||
## Validation, Repair, And Retries
|
||
|
||
Scriptorium performs prompt rendering, provider execution, and the prompt’s
|
||
structured-output validation. The adapter reports an empty result, validation
|
||
failure, empty structured body, or decode failure as
|
||
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||
material when they exist. Provider failures remain operational errors rather
|
||
than output-validation failures.
|
||
|
||
Prompt-declared repair is executed within Scriptorium’s structured-output flow.
|
||
The current production D&D prompt manifests set repair attempts to zero. That
|
||
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||
count reruns its stage attempt after an error or rejection, and an exhausted
|
||
rejection is a recorded output rather than a provider error. The pipeline owns
|
||
attempt lifecycle, validation chains, and retry diagnostics; see
|
||
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||
[binding reference](../config.md#module-bindings-and-validators).
|
||
|
||
## Observability And Redaction
|
||
|
||
When debug recording is enabled, the pipeline decorates the shared client. The
|
||
wrapper records prepared prompt and response material, timing, selected profile
|
||
and model, and call identifiers in the run’s debug bundle, including material
|
||
available from a failed structured completion. For a successful completion, a
|
||
debug-write failure is surfaced; when the completion already failed, its call
|
||
error remains the result. Debug-bundle location, retention, and handling are
|
||
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
||
|
||
Run manifests receive selected profile summaries and component identities, not
|
||
prompt, schema, source, reference, or response content. Provider error text is
|
||
wrapped with prompt context and bearer credentials are redacted before it
|
||
crosses the runtime boundary. Known-secret redaction is available to other
|
||
runtime collaborators; it does not make prompt or response contents safe for
|
||
general logging.
|
||
|
||
## Failure Boundaries
|
||
|
||
- Construction fails for missing asset registries, mutually exclusive profile
|
||
sources, invalid asset registration, or a non-positive scheduler limit.
|
||
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||
context cancellation propagate to the calling stage with context.
|
||
- Malformed or schema-invalid provider output is classified separately as
|
||
invalid structured output so the module or pipeline can apply its own retry
|
||
and rejection policy.
|
||
- Domain semantic checks, evidence decisions, and deterministic normalization
|
||
run outside the provider adapter.
|
||
|
||
## Focused Verification
|
||
|
||
Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
|
||
tests when changing this boundary. Prompt changes also require the owning
|
||
module’s asset tests, and retry or debug changes require focused pipeline or
|
||
CLI coverage. The focused runtime and D&D checks are:
|
||
|
||
~~~sh
|
||
go test ./internal/framework/llm/... ./internal/modules/dnd/...
|
||
~~~
|