Rewrite LLM runtime documentation
This commit is contained in:
@@ -1,252 +1,149 @@
|
|||||||
# LLM Runtime Internals
|
# LLM Runtime Internals
|
||||||
|
|
||||||
`internal/framework/llm` implements Notarius's transport boundary for structured
|
`internal/framework/llm` is Notarius’s provider-independent structured
|
||||||
completion. It contains the Scriptorium adapter, concurrency scheduler,
|
completion boundary. It adapts framework requests to Scriptorium, bounds
|
||||||
prompt/schema registries, selected-profile recording, and provider-error
|
provider calls, assembles registered prompt and schema assets, records selected
|
||||||
redaction.
|
profiles, and redacts provider errors. The architectural boundary is defined in
|
||||||
|
[Architecture](../policy/architecture.md#llm-boundary); profile sources,
|
||||||
|
credentials, and concurrency settings belong in
|
||||||
|
[Configuration](../config.md#scriptorium-profiles) and
|
||||||
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
|
|
||||||
Provider-neutral ownership rules are defined in
|
## Structured Completion Boundary
|
||||||
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
|
|
||||||
credentials, and concurrency settings are defined in
|
|
||||||
[Configuration](../config.md).
|
|
||||||
|
|
||||||
## Structured Contract
|
Modules and LLM-backed validators depend only on
|
||||||
|
`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and
|
||||||
|
version, optional profile and session IDs, named input material, variables, and
|
||||||
|
a caller-owned decode target. The successful response returns the validated raw
|
||||||
|
structured bytes together with non-secret provider, model, profile, and token
|
||||||
|
metadata.
|
||||||
|
|
||||||
Modules and LLM-backed validators depend on
|
The caller owns the domain behavior: it chooses the prompt, prepares inputs,
|
||||||
`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
|
selects the private response schema, and interprets the decoded result. The
|
||||||
prompt and optional profile/session, supplies named input materials and
|
adapter does not own source evidence, artifact conversion, normalization, or
|
||||||
variables, and provides a caller-owned decoding target. A successful response
|
durable schemas. Those responsibilities remain with the module and its
|
||||||
contains the validated raw structured bytes plus non-secret provider, model,
|
[integration contract](../integrations/).
|
||||||
profile, and token metadata.
|
|
||||||
|
|
||||||
The caller owns prompt selection, response-schema selection, and interpretation
|
`ScriptoriumClient` validates the request target and prompt identity, maps each
|
||||||
of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
|
named material to a Scriptorium inline artifact while preserving its origin URI,
|
||||||
their origin metadata so the adapter can pass named artifacts to Scriptorium
|
forwards session and profile selection, then prepares and runs the prompt. It
|
||||||
without exposing Scriptorium types through stage contracts.
|
returns Scriptorium’s validated raw bytes rather than re-encoding the decoded
|
||||||
|
target. An empty optional material is represented as one space so its named
|
||||||
|
input is retained by Scriptorium.
|
||||||
|
|
||||||
## Production Construction
|
An empty request profile lets the prompt select its configured default. The CLI
|
||||||
|
prepares every explicitly selected binding profile before a run begins, so a
|
||||||
|
missing explicit profile fails before stage execution. Calls record the profile
|
||||||
|
actually selected by Scriptorium; the recorder deduplicates non-secret profile
|
||||||
|
identity, provider, and model values for manifest use.
|
||||||
|
|
||||||
`internal/cli` constructs the production runtime by:
|
## Shared Provider-Call Limit
|
||||||
|
|
||||||
1. allocating the asset registry populated by the generic, Seriatim, and D&D
|
Production construction creates one Scriptorium client and wraps it in one
|
||||||
package-family registrars;
|
scheduled client. The scheduler has a fixed, positive permit limit, serves
|
||||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
queued calls in FIFO order, and removes a queued call when its context is
|
||||||
3. attaching an `LLMProfileRecorder`;
|
cancelled. A granted permit is released exactly once on every completion path.
|
||||||
4. creating a scheduler from the effective concurrency limit;
|
|
||||||
5. returning a `ScheduledClient` wrapper;
|
|
||||||
6. decorating that shared client before preparation when debug recording is
|
|
||||||
enabled; and
|
|
||||||
7. injecting that one shared client into complete pipeline preparation before
|
|
||||||
the source file is read or the runner is invoked.
|
|
||||||
|
|
||||||
The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and
|
The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent
|
||||||
scene-description extractors retain this
|
lanes, pipeline retries, and LLM-backed validators share the same provider-call
|
||||||
injected client and use it for every structured completion. Operation requests
|
ceiling. This ceiling is independent of pipeline worker concurrency; changing
|
||||||
do not carry an LLM client.
|
worker counts cannot exceed the configured LLM limit. The configuration field
|
||||||
|
and its effective default are owned by
|
||||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
[Configuration](../config.md#concurrency-output-cache-and-debug).
|
||||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
|
||||||
missing or invalid profiles fail before pipeline execution. The runtime profile
|
|
||||||
override syntax and scope are defined in the
|
|
||||||
[CLI reference](../cli.md#run); binding rules are defined in
|
|
||||||
[Configuration](../config.md#module-bindings).
|
|
||||||
|
|
||||||
## Scriptorium Adapter
|
|
||||||
|
|
||||||
`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
|
|
||||||
It validates the decoding target and prompt identity, maps named input materials
|
|
||||||
to inline artifacts, forwards explicit profile and session context, delegates
|
|
||||||
rendering/provider execution/structured validation, and unmarshals successful
|
|
||||||
JSON into the caller target.
|
|
||||||
|
|
||||||
Empty optional input material is represented by a single space so Scriptorium
|
|
||||||
retains the named input. The client returns Scriptorium's validated structured
|
|
||||||
bytes rather than re-encoding the caller target, allowing modules to preserve
|
|
||||||
the runtime result exactly.
|
|
||||||
|
|
||||||
Selected profile, provider, model, and token metadata are mapped into the
|
|
||||||
Notarius response. The recorder deduplicates profiles by identity and supplies
|
|
||||||
manifest-safe profile summaries after actual calls; manifest population does
|
|
||||||
not guess the selected prompt default in advance.
|
|
||||||
|
|
||||||
Generated-output validation failures and provider failures are wrapped with
|
|
||||||
prompt context. Error strings pass through bearer-token redaction before they
|
|
||||||
cross the runtime boundary.
|
|
||||||
|
|
||||||
## Scheduling
|
|
||||||
|
|
||||||
`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
|
|
||||||
acquisition increments the in-flight count; queued acquisition waits for a
|
|
||||||
permit or context cancellation. Cancellation removes a queued waiter, while a
|
|
||||||
cancelled waiter that has already received a permit releases it.
|
|
||||||
|
|
||||||
`ScheduledClient` acquires a permit around each structured completion and
|
|
||||||
defers release on every result path. The effective limit and default are
|
|
||||||
configuration facts in [Configuration](../config.md#defaults).
|
|
||||||
|
|
||||||
This provider-call ceiling is independent of the pipeline's extract worker
|
|
||||||
limit. Concurrent lanes, retries, and validators all use the same scheduled
|
|
||||||
client, so increasing framework workers cannot exceed `total_llm`. Pipeline
|
|
||||||
dispatch and cancellation mechanics are documented in
|
|
||||||
[Pipeline Internals](pipeline.md#execution-flow).
|
|
||||||
|
|
||||||
## Prompt And Schema Assets
|
## Prompt And Schema Assets
|
||||||
|
|
||||||
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
An `AssetRegistry` collects prompt and schema filesystems from production module
|
||||||
and rejects invalid or conflicting registrations. Production module packages
|
families. It flattens registered roots into the Scriptorium filesystems and
|
||||||
register their own prompt and schema assets; generic framework code contains no
|
rejects invalid roots, unreadable assets, duplicate paths, and missing prompt
|
||||||
D&D prompt content. `internal/framework/promptfs` provides the domain-neutral
|
or schema files during preparation. The framework’s `promptfs` helper combines
|
||||||
filesystem composition helper used to combine module-owned files with shared
|
module-owned prompt files with reusable domain fragments without making the
|
||||||
domain prompt fragments.
|
framework depend on D&D content.
|
||||||
|
|
||||||
The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and
|
Each LLM-backed module owns its prompt declaration, package-specific assets,
|
||||||
scene-description extractors each declare an
|
and private response schema. Shared D&D wording is owned by the D&D shared
|
||||||
ordered prompt asset manifest. The manifest lists the package-owned YAML and
|
asset package; the detailed D&D conventions are in
|
||||||
Markdown files, then the exact shared fragments rendered by that prompt; the
|
[D&D Module Internals](dnd.md). The mounted prompt assets used by a module also
|
||||||
same ordered list drives both filesystem mounting and the prompt fingerprint.
|
determine its prompt fingerprint. Schema loaders validate JSON, attach identity
|
||||||
Unused shared assets are neither mounted nor fingerprinted. Universal
|
and digest metadata, make defensive copies, and expose diagnostics without raw
|
||||||
extraction-evidence and output policy lives only in the shared extraction
|
schema bytes.
|
||||||
assets; package-owned prompt files retain artifact-specific rules. The scene
|
|
||||||
prompt keeps its separate output rule because it does not render the
|
|
||||||
extraction-evidence asset.
|
|
||||||
|
|
||||||
### D&D Extraction Prompt Ordering And Cache Boundaries
|
Private response schemas validate a model transport envelope. They are not the
|
||||||
|
durable artifact schema and should not be documented as an external wire
|
||||||
|
contract. Durable formats and compatibility rules remain in the
|
||||||
|
[integration contracts](../integrations/).
|
||||||
|
|
||||||
D&D extraction prompts order messages from the most reusable content to the
|
## Prompt Maintenance And Backend Caching
|
||||||
most variable content. New extraction lanes use these tiers in order:
|
|
||||||
|
|
||||||
1. universal shared content, including the system, extraction-evidence, and
|
Prompt message order and shared asset bytes are runtime behavior. Backend cache
|
||||||
in-world identity messages;
|
reuse depends on the same preceding messages and content, not merely equivalent
|
||||||
2. stable campaign or run context shared across lanes, including campaign
|
meaning. Keep reusable shared assets byte-identical and keep stable material
|
||||||
references;
|
before the inputs that vary per request wherever a prompt’s declared sequence
|
||||||
3. stable subset- and lane-specific context and instructions, including an NPC
|
supports caching. Preserve the existing manifest order and cache-control hints
|
||||||
registry, catalog, task, or extraction instructions when applicable;
|
when editing a prompt.
|
||||||
4. the chunk transcript as the final user message.
|
|
||||||
|
|
||||||
This ordering lets requests reuse the longest identical prefix before the
|
D&D extraction manifests place the changing chunk transcript at the end of the
|
||||||
per-chunk transcript changes. Cache reuse requires the preceding message
|
prompt after their reusable context. Scene chunking and NPC normalization use
|
||||||
sequence and content to be exactly identical; semantic similarity is not
|
their own declared message sequences because their inputs and work differ. The
|
||||||
sufficient. Cache boundaries belong at the ends of reusable stable tiers,
|
family-specific asset and ordering rules belong in [D&D Module Internals](dnd.md).
|
||||||
subject to the provider's cache-boundary limit. The shared identity and
|
Do not add tests that enforce a fixed message-prefix length; prompt-asset tests
|
||||||
campaign-reference messages form the first two extraction boundaries. Spell,
|
should instead verify the meaningful asset sequence, inputs, and cache controls
|
||||||
combat, and interaction prompts add a boundary at the shared NPC registry. Each extraction
|
of the prompt being changed.
|
||||||
prompt places its final boundary on its lane-specific instructions, immediately
|
|
||||||
before the transcript. The transcript does not carry cache control because no
|
|
||||||
reusable content follows it.
|
|
||||||
|
|
||||||
Accordingly, the common prefix of the spell, NPC, combat, item-event, and interaction
|
## Validation, Repair, And Retries
|
||||||
extraction prompts is system,
|
|
||||||
extraction evidence, identity, and campaign references. The NPC prompt then
|
|
||||||
renders task, instructions, and transcript. Spell renders the NPC registry,
|
|
||||||
catalog, task, instructions, and transcript. Combat renders the NPC registry,
|
|
||||||
task, instructions, and transcript. Item-event renders task, instructions, and
|
|
||||||
transcript without a generated-artifact input. NPC interaction renders the names-only NPC
|
|
||||||
registry, task, instructions, and transcript. The
|
|
||||||
scene chunker is not an extraction lane: it retains its separate system,
|
|
||||||
transcript, campaign-reference, task, and instruction order and marks its
|
|
||||||
transcript and campaign-reference messages ephemeral.
|
|
||||||
|
|
||||||
The scene-description extractor deliberately omits the citation-oriented
|
Scriptorium performs prompt rendering, provider execution, and the prompt’s
|
||||||
`common-dnd-extraction-evidence.md` asset because Notarius attaches the whole
|
structured-output validation. The adapter reports an empty result, validation
|
||||||
accepted chunk range itself. Its manifest is system, shared identity, shared
|
failure, empty structured body, or decode failure as
|
||||||
campaign references, lane task, lane instructions, then the transcript. The
|
`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug
|
||||||
identity, campaign-reference, and instruction messages are ephemeral cache
|
material when they exist. Provider failures remain operational errors rather
|
||||||
boundaries; the transcript is last and has no cache control. Compatible shared
|
than output-validation failures.
|
||||||
messages remain canonical shared assets rather than copied package text.
|
|
||||||
|
|
||||||
### D&D NPC Normalization Prompt Ordering And Cache Boundaries
|
Prompt-declared repair is executed within Scriptorium’s structured-output flow.
|
||||||
|
The current production D&D prompt manifests set repair attempts to zero. That
|
||||||
|
setting does not replace pipeline retry behavior: a binding’s configured retry
|
||||||
|
count reruns its stage attempt after an error or rejection, and an exhausted
|
||||||
|
rejection is a recorded output rather than a provider error. The pipeline owns
|
||||||
|
attempt lifecycle, validation chains, and retry diagnostics; see
|
||||||
|
[Pipeline Internals](pipeline.md#validation-retries-and-output) and the
|
||||||
|
[binding reference](../config.md#module-bindings-and-validators).
|
||||||
|
|
||||||
NPC normalization has a distinct prompt and response-schema identity from NPC
|
## Observability And Redaction
|
||||||
extraction. Its stable message tiers are the common D&D system asset, followed
|
|
||||||
by package-owned task and normalization instructions. Cache boundaries follow
|
|
||||||
the shared system tier and the package instructions. The variable tail contains
|
|
||||||
the private candidate-name-and-range input and a windowed transcript input
|
|
||||||
whose cited units provide local context; neither has a cache boundary because
|
|
||||||
it changes with the document.
|
|
||||||
|
|
||||||
This prompt intentionally omits shared identity guidance,
|
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||||
extraction-evidence, and campaign-reference assets: it reconciles existing
|
wrapper records prepared prompt and response material, timing, selected profile
|
||||||
records rather than extracting events or adding evidence. Its package-owned
|
and model, and call identifiers in the run’s debug bundle, including material
|
||||||
manifest and schema identity are fingerprinted separately, so a normalization
|
available from a failed structured completion. For a successful completion, a
|
||||||
prompt or schema change cannot reuse a prior normalization checkpoint.
|
debug-write failure is surfaced; when the completion already failed, its call
|
||||||
|
error remains the result. Debug-bundle location, retention, and handling are
|
||||||
|
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
||||||
|
|
||||||
Shared wording belongs in the canonical assets under
|
Run manifests receive selected profile summaries and component identities, not
|
||||||
`internal/modules/dnd/shared`; extraction packages reference those assets in
|
prompt, schema, source, reference, or response content. Provider error text is
|
||||||
their manifests instead of copying similar text into package-local files.
|
wrapped with prompt context and bearer credentials are redacted before it
|
||||||
Package-local assets contain only lane-specific content. An extraction lane may
|
crosses the runtime boundary. Known-secret redaction is available to other
|
||||||
depart from the tier order only when prompt-quality evidence or a provider
|
runtime collaborators; it does not make prompt or response contents safe for
|
||||||
constraint makes the exception necessary; document the exception and rationale
|
general logging.
|
||||||
here when it becomes implemented behavior.
|
|
||||||
|
|
||||||
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
## Failure Boundaries
|
||||||
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
|
||||||
The small framework registry contains only generic test schemas; production
|
|
||||||
schemas remain package-owned.
|
|
||||||
|
|
||||||
The spell, NPC, combat, item-event, NPC-interaction, and scene-description extractors'
|
- Construction fails for missing asset registries, mutually exclusive profile
|
||||||
package-owned prompts declare their
|
sources, invalid asset registration, or a non-positive scheduler limit.
|
||||||
structured JSON inputs and private response schemas. Each private response
|
- Preparation failures, unavailable explicit profiles, provider failures, and
|
||||||
schema remains separate from its durable artifact codec schema; this work does
|
context cancellation propagate to the calling stage with context.
|
||||||
not use shared schema fragments or schema generation. Those private schemas own
|
- Malformed or schema-invalid provider output is classified separately as
|
||||||
the transport envelope—required fields, JSON types, nullability, and
|
invalid structured output so the module or pipeline can apply its own retry
|
||||||
unknown-field rejection—while deterministic validators own semantic constraints
|
and rejection policy.
|
||||||
such as enum membership, non-empty values and collections, and positive
|
- Domain semantic checks, evidence decisions, and deterministic normalization
|
||||||
numbers. The spell extractor's prompt declares a required
|
run outside the provider adapter.
|
||||||
`application/json` `spell_catalog` input and an optional `application/json`
|
|
||||||
`npcs` input. The extractor generates
|
|
||||||
the catalog input from its prepared
|
|
||||||
effective catalog as `{"spell_names":[...]}` using sorted canonical names only.
|
|
||||||
The shared D&D prompt assets include a generic NPC grounding fragment directly
|
|
||||||
after the campaign reference message for spell, combat, and interaction prompts. When an NPC
|
|
||||||
registry is bound, the
|
|
||||||
domain registry boundary strictly decodes and identity-validates one durable
|
|
||||||
artifact, re-encodes canonical JSON for provenance, and separately generates a
|
|
||||||
names-only prompt projection. The unbound projection is exactly `{"npcs":[]}`.
|
|
||||||
Prompt input and component-local checkpoint digests cover the projected bytes;
|
|
||||||
manifests retain the optional full registry digest/count rather than names,
|
|
||||||
overlay bytes, registry paths, or source metadata. Combat and interaction prompt,
|
|
||||||
response-schema, mapping, normalization, identity, and registry-projection
|
|
||||||
fingerprints remain separate semantic inputs to checkpoint identity.
|
|
||||||
|
|
||||||
## Debug And Redaction Boundaries
|
## Focused Verification
|
||||||
|
|
||||||
The pipeline may wrap the client with a debug recorder that captures prepared
|
Read the LLM adapter, scheduler, asset registry, schema loader, and redaction
|
||||||
prompt/response material for an explicitly requested debug run. Debug summaries
|
tests when changing this boundary. Prompt changes also require the owning
|
||||||
and manifests receive identities, hashes, usage, and selected profile summaries
|
module’s asset tests, and retry or debug changes require focused pipeline or
|
||||||
rather than prompt, source, reference, schema, or response content.
|
CLI coverage. The focused runtime and D&D checks are:
|
||||||
|
|
||||||
The Scriptorium error wrapper removes bearer credential values from surfaced
|
~~~sh
|
||||||
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
|
go test ./internal/framework/llm/... ./internal/modules/dnd/...
|
||||||
secret values elsewhere in the runtime. Config summaries use a separate
|
~~~
|
||||||
clone-and-redact path in `internal/core/config`. These mechanisms implement the
|
|
||||||
security invariant in
|
|
||||||
[Architecture](../policy/architecture.md#state-output-and-safety); operator
|
|
||||||
handling of debug data is defined in [Operations](../operations.md#debug).
|
|
||||||
|
|
||||||
## Failure Behavior
|
|
||||||
|
|
||||||
- Invalid targets, missing prompt IDs, malformed structured output, and
|
|
||||||
Scriptorium failures return contextual errors to the calling module.
|
|
||||||
- Scheduler construction rejects non-positive limits; acquisition respects
|
|
||||||
context cancellation.
|
|
||||||
- Asset registration rejects invalid roots, missing content, and path conflicts.
|
|
||||||
- Schema loading distinguishes missing assets, invalid JSON, and invalid
|
|
||||||
metadata.
|
|
||||||
- Profile validation errors occur during CLI preparation when an explicit
|
|
||||||
selected ID cannot be prepared.
|
|
||||||
|
|
||||||
## Tests To Inspect
|
|
||||||
|
|
||||||
- `internal/framework/llm/scriptorium_client_test.go`: adapter mapping and local
|
|
||||||
HTTP integration.
|
|
||||||
- `internal/framework/llm/scheduler_test.go` and
|
|
||||||
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
|
|
||||||
release.
|
|
||||||
- `internal/framework/llm/asset_registry_test.go` and
|
|
||||||
`schema_registry_test.go`: asset composition, validation, and defensive
|
|
||||||
copies.
|
|
||||||
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
|
|
||||||
- `internal/cli/run_contract_test.go`: profile validation, production client
|
|
||||||
wiring, manifest recording, and debug integration.
|
|
||||||
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
|
|
||||||
asset registration.
|
|
||||||
|
|||||||
Reference in New Issue
Block a user