155 lines
7.3 KiB
Markdown
155 lines
7.3 KiB
Markdown
# LLM Runtime Internals
|
|
|
|
`internal/framework/llm` implements Notarius's transport boundary for structured
|
|
completion. It contains the Scriptorium adapter, concurrency scheduler,
|
|
prompt/schema registries, selected-profile recording, and provider-error
|
|
redaction.
|
|
|
|
Provider-neutral ownership rules are defined in
|
|
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
|
|
credentials, and concurrency settings are defined in
|
|
[Configuration](../config.md).
|
|
|
|
## Structured Contract
|
|
|
|
Modules and LLM-backed validators depend on
|
|
`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
|
|
prompt and optional profile/session, supplies named input materials and
|
|
variables, and provides a caller-owned decoding target. A successful response
|
|
contains the validated raw structured bytes plus non-secret provider, model,
|
|
profile, and token metadata.
|
|
|
|
The caller owns prompt selection, response-schema selection, and interpretation
|
|
of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
|
|
their origin metadata so the adapter can pass named artifacts to Scriptorium
|
|
without exposing Scriptorium types through stage contracts.
|
|
|
|
## Production Construction
|
|
|
|
`internal/cli` constructs the production runtime by:
|
|
|
|
1. allocating the asset registry populated by the generic, Seriatim, and D&D
|
|
package-family registrars;
|
|
2. creating a `ScriptoriumClient` from the effective profile source;
|
|
3. attaching an `LLMProfileRecorder`;
|
|
4. creating a scheduler from the effective concurrency limit;
|
|
5. returning a `ScheduledClient` wrapper;
|
|
6. decorating that shared client before preparation when debug recording is
|
|
enabled; and
|
|
7. injecting that one shared client into complete pipeline preparation before
|
|
the source file is read or the runner is invoked.
|
|
|
|
The D&D scene chunker and spell extractor retain this injected client and use
|
|
it for every structured completion. Operation requests do not carry an LLM
|
|
client.
|
|
|
|
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
|
and validator bindings. It prepares a small internal check prompt for each ID so
|
|
missing or invalid profiles fail before pipeline execution. The runtime profile
|
|
override syntax and scope are defined in the
|
|
[CLI reference](../cli.md#run); binding rules are defined in
|
|
[Configuration](../config.md#module-bindings).
|
|
|
|
## Scriptorium Adapter
|
|
|
|
`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
|
|
It validates the decoding target and prompt identity, maps named input materials
|
|
to inline artifacts, forwards explicit profile and session context, delegates
|
|
rendering/provider execution/structured validation, and unmarshals successful
|
|
JSON into the caller target.
|
|
|
|
Empty optional input material is represented by a single space so Scriptorium
|
|
retains the named input. The client returns Scriptorium's validated structured
|
|
bytes rather than re-encoding the caller target, allowing modules to preserve
|
|
the runtime result exactly.
|
|
|
|
Selected profile, provider, model, and token metadata are mapped into the
|
|
Notarius response. The recorder deduplicates profiles by identity and supplies
|
|
manifest-safe profile summaries after actual calls; manifest population does
|
|
not guess the selected prompt default in advance.
|
|
|
|
Generated-output validation failures and provider failures are wrapped with
|
|
prompt context. Error strings pass through bearer-token redaction before they
|
|
cross the runtime boundary.
|
|
|
|
## Scheduling
|
|
|
|
`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
|
|
acquisition increments the in-flight count; queued acquisition waits for a
|
|
permit or context cancellation. Cancellation removes a queued waiter, while a
|
|
cancelled waiter that has already received a permit releases it.
|
|
|
|
`ScheduledClient` acquires a permit around each structured completion and
|
|
defers release on every result path. The effective limit and default are
|
|
configuration facts in [Configuration](../config.md#defaults).
|
|
|
|
This provider-call ceiling is independent of the pipeline's extract worker
|
|
limit. Concurrent lanes, retries, and validators all use the same scheduled
|
|
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
|
|
dispatch and cancellation mechanics are documented in
|
|
[Pipeline Internals](pipeline.md#execution-flow).
|
|
|
|
## Prompt And Schema Assets
|
|
|
|
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
|
and rejects invalid or conflicting registrations. Production module packages
|
|
register their own prompt and schema assets; generic framework code contains no
|
|
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
|
|
filesystem composition helper used to combine module-owned files with shared
|
|
domain prompt fragments.
|
|
|
|
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
|
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
|
The small framework registry contains only generic test schemas; production
|
|
schemas remain package-owned.
|
|
|
|
The spell extractor's package-owned prompt declares a required
|
|
`application/json` `spell_catalog` input. The extractor generates that input
|
|
from its prepared effective catalog as `{"spell_names":[...]}` using sorted
|
|
canonical names only. Its input digest covers those generated bytes; manifests
|
|
record catalog identity and digest rather than names, aliases, overlay bytes,
|
|
or source metadata.
|
|
|
|
## Debug And Redaction Boundaries
|
|
|
|
The pipeline may wrap the client with a debug recorder that captures prepared
|
|
prompt/response material for an explicitly requested debug run. Debug summaries
|
|
and manifests receive identities, hashes, usage, and selected profile summaries
|
|
rather than prompt, source, reference, schema, or response content.
|
|
|
|
The Scriptorium error wrapper removes bearer credential values from surfaced
|
|
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
|
|
secret values elsewhere in the runtime. Config summaries use a separate
|
|
clone-and-redact path in `internal/core/config`. These mechanisms implement the
|
|
security invariant in
|
|
[Architecture](../policy/architecture.md#state-output-and-safety); operator
|
|
handling of debug data is defined in [Operations](../operations.md#debug).
|
|
|
|
## Failure Behavior
|
|
|
|
- Invalid targets, missing prompt IDs, malformed structured output, and
|
|
Scriptorium failures return contextual errors to the calling module.
|
|
- Scheduler construction rejects non-positive limits; acquisition respects
|
|
context cancellation.
|
|
- Asset registration rejects invalid roots, missing content, and path conflicts.
|
|
- Schema loading distinguishes missing assets, invalid JSON, and invalid
|
|
metadata.
|
|
- Profile validation errors occur during CLI preparation when an explicit
|
|
selected ID cannot be prepared.
|
|
|
|
## Tests To Inspect
|
|
|
|
- `internal/framework/llm/scriptorium_client_test.go`: adapter mapping and local
|
|
HTTP integration.
|
|
- `internal/framework/llm/scheduler_test.go` and
|
|
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
|
|
release.
|
|
- `internal/framework/llm/asset_registry_test.go` and
|
|
`schema_registry_test.go`: asset composition, validation, and defensive
|
|
copies.
|
|
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
|
|
- `internal/cli/run_contract_test.go`: profile validation, production client
|
|
wiring, manifest recording, and debug integration.
|
|
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
|
|
asset registration.
|