Refocus developer and internal documentation
This commit is contained in:
@@ -1,114 +1,132 @@
|
||||
# LLM Runtime
|
||||
# LLM Runtime Internals
|
||||
|
||||
The implemented LLM runtime lives in `internal/framework/llm`. It provides
|
||||
transport-neutral structured completion contracts, a Scriptorium-backed
|
||||
production client, concurrency scheduling, prompt/schema asset registration,
|
||||
schema registry helpers, and secret redaction.
|
||||
`internal/framework/llm` implements Notarius's transport boundary for structured
|
||||
completion. It contains the Scriptorium adapter, concurrency scheduler,
|
||||
prompt/schema registries, selected-profile recording, and provider-error
|
||||
redaction.
|
||||
|
||||
## Contract
|
||||
Provider-neutral ownership rules are defined in
|
||||
[Architecture](../policy/architecture.md#llm-boundary). Profile sources,
|
||||
credentials, and concurrency settings are defined in
|
||||
[Configuration](../config.md).
|
||||
|
||||
Modules depend on `contracts.StructuredLLMClient`:
|
||||
## Structured Contract
|
||||
|
||||
```go
|
||||
CompleteStructured(ctx, request, out) (response, error)
|
||||
```
|
||||
Modules and LLM-backed validators depend on
|
||||
`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a
|
||||
prompt and optional profile/session, supplies named input materials and
|
||||
variables, and provides a caller-owned decoding target. A successful response
|
||||
contains the validated raw structured bytes plus non-secret provider, model,
|
||||
profile, and token metadata.
|
||||
|
||||
The request contains prompt ID/version, profile ID, session ID, prompt input
|
||||
materials, and variables. The caller supplies a pointer target for decoded
|
||||
structured output. The response also carries the raw structured output bytes
|
||||
returned by the runtime so modules can preserve raw payloads in pipeline stage
|
||||
outputs.
|
||||
The caller owns prompt selection, response-schema selection, and interpretation
|
||||
of the decoded result. `LLMInputMaterial` keeps source and reference bytes with
|
||||
their origin metadata so the adapter can pass named artifacts to Scriptorium
|
||||
without exposing Scriptorium types through stage contracts.
|
||||
|
||||
Modules that call the LLM own their prompts, schemas, prompt IDs, and
|
||||
domain-specific interpretation. Validator packages own approve/reject policy,
|
||||
and central catalog mappings decide which validators run by default. Provider
|
||||
adapters should not contain domain-specific prompt logic.
|
||||
## Production Construction
|
||||
|
||||
Prompt input materials carry source or reference bytes with optional origin
|
||||
metadata. The Scriptorium-backed runtime receives them as named artifacts rather
|
||||
than rendered prompt strings owned by Notarius modules.
|
||||
`internal/cli` constructs the production runtime by:
|
||||
|
||||
## Production Client Construction
|
||||
1. collecting embedded prompt and response-schema assets from production module
|
||||
packages;
|
||||
2. creating a `ScriptoriumClient` from the effective profile source;
|
||||
3. attaching an `LLMProfileRecorder`;
|
||||
4. creating a scheduler from the effective concurrency limit;
|
||||
5. returning a `ScheduledClient` wrapper.
|
||||
|
||||
`internal/cli` builds the production LLM client from the effective config:
|
||||
|
||||
1. collect production Scriptorium prompt and schema assets from module packages;
|
||||
2. create a Scriptorium-backed structured client using effective Scriptorium
|
||||
profile source settings from `scriptorium.profile_dir` or
|
||||
`scriptorium.profile_file`;
|
||||
3. create a scheduler from global LLM concurrency;
|
||||
4. wrap the client with `NewScheduledClient`;
|
||||
5. let the runtime report non-secret profile manifest metadata after calls.
|
||||
|
||||
The runtime records the actual selected Scriptorium profile, provider, and model
|
||||
used during execution. Manifest population does not rely on a precomputed
|
||||
profile ID before pipeline execution.
|
||||
|
||||
Explicit profile validation applies to LLM-capable pipeline stages: chunk,
|
||||
extract, merge, normalize, and LLM-backed validators with explicit
|
||||
`llm_profile` values. Input, output, and deterministic validators do not call
|
||||
the LLM. The `--llm-profile` run flag overrides effective chunk, extract, merge,
|
||||
and normalize bindings; it does not override validator-specific profiles.
|
||||
The CLI separately gathers explicit profile IDs from resolved LLM-capable stage
|
||||
and validator bindings. It prepares a small internal check prompt for each ID so
|
||||
missing or invalid profiles fail before pipeline execution. The runtime profile
|
||||
override syntax and scope are defined in the
|
||||
[CLI reference](../cli.md#run); binding rules are defined in
|
||||
[Configuration](../config.md#module-bindings).
|
||||
|
||||
## Scriptorium Adapter
|
||||
|
||||
`ScriptoriumClient` implements `contracts.StructuredLLMClient` by converting
|
||||
Notarius prompt requests into Scriptorium `RunRequest` values. It:
|
||||
`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`.
|
||||
It validates the decoding target and prompt identity, maps named input materials
|
||||
to inline artifacts, forwards explicit profile and session context, delegates
|
||||
rendering/provider execution/structured validation, and unmarshals successful
|
||||
JSON into the caller target.
|
||||
|
||||
- validates the caller output target and prompt ID;
|
||||
- converts `LLMInputMaterial` values into inline Scriptorium artifacts, using a
|
||||
single space for empty material so optional blank references remain explicit;
|
||||
- passes `session_id` through Scriptorium variables and request metadata when
|
||||
present;
|
||||
- sends explicit profile IDs only when the request supplies one;
|
||||
- lets Scriptorium render prompts, call the configured provider, and validate
|
||||
structured output;
|
||||
- unmarshals successful JSON into the caller-provided target;
|
||||
- returns the validated raw structured output bytes to the caller;
|
||||
- maps token usage and selected profile/model metadata into the Notarius
|
||||
response and manifest profile recorder.
|
||||
Empty optional input material is represented by a single space so Scriptorium
|
||||
retains the named input. The client returns Scriptorium's validated structured
|
||||
bytes rather than re-encoding the caller target, allowing modules to preserve
|
||||
the runtime result exactly.
|
||||
|
||||
Generated-output validation failures are returned as Notarius errors. Provider
|
||||
and runtime errors are wrapped with prompt context and bearer tokens are
|
||||
redacted from error strings. Prompt text, raw source input, reference content,
|
||||
schema JSON, API keys, and bearer tokens are not added to default diagnostics or
|
||||
run manifests.
|
||||
Selected profile, provider, model, and token metadata are mapped into the
|
||||
Notarius response. The recorder deduplicates profiles by identity and supplies
|
||||
manifest-safe profile summaries after actual calls; manifest population does
|
||||
not guess the selected prompt default in advance.
|
||||
|
||||
## Scheduler
|
||||
Generated-output validation failures and provider failures are wrapped with
|
||||
prompt context. Error strings pass through bearer-token redaction before they
|
||||
cross the runtime boundary.
|
||||
|
||||
`Scheduler` bounds concurrent provider calls. It tracks in-flight calls and a
|
||||
FIFO queue of waiters. Cancellation removes queued waiters or releases granted
|
||||
permits.
|
||||
## Scheduling
|
||||
|
||||
`NewScheduledClient` wraps any structured LLM client and runs each completion
|
||||
inside the scheduler.
|
||||
`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate
|
||||
acquisition increments the in-flight count; queued acquisition waits for a
|
||||
permit or context cancellation. Cancellation removes a queued waiter, while a
|
||||
cancelled waiter that has already received a permit releases it.
|
||||
|
||||
Effective concurrency is:
|
||||
`ScheduledClient` acquires a permit around each structured completion and
|
||||
defers release on every result path. The effective limit and default are
|
||||
configuration facts in [Configuration](../config.md#defaults).
|
||||
|
||||
1. `concurrency.total_llm`, when greater than zero;
|
||||
2. `1`.
|
||||
## Prompt And Schema Assets
|
||||
|
||||
## Schema Registry
|
||||
`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes
|
||||
and rejects invalid or conflicting registrations. Production module packages
|
||||
register their own prompt and schema assets; generic framework code contains no
|
||||
D&D prompt content.
|
||||
|
||||
The framework schema registry embeds generic test schemas. It also exposes
|
||||
helpers for caller-owned schemas:
|
||||
Schema helpers load embedded JSON Schema with identity and digest metadata,
|
||||
return defensive copies, and expose a diagnostics map that omits schema bytes.
|
||||
The small framework registry contains only generic test schemas; production
|
||||
schemas remain package-owned.
|
||||
|
||||
- `LoadResponseSchema`
|
||||
- `LookupResponseSchema`
|
||||
- `MustLookupResponseSchema`
|
||||
- `ResponseSchema.DiagnosticsMap`
|
||||
## Debug And Redaction Boundaries
|
||||
|
||||
`DiagnosticsMap` omits raw schema content and includes metadata such as key,
|
||||
ID, version, name, and SHA-256.
|
||||
The pipeline may wrap the client with a debug recorder that captures prepared
|
||||
prompt/response material for an explicitly enabled debug run. Default
|
||||
diagnostics and manifests receive identities, hashes, usage, and selected
|
||||
profile summaries rather than prompt, source, reference, schema, or response
|
||||
content.
|
||||
|
||||
Production modules own and register their Scriptorium prompt and schema assets.
|
||||
Framework packages may collect those files but must not contain D&D-specific
|
||||
prompt content.
|
||||
The Scriptorium error wrapper removes bearer credential values from surfaced
|
||||
provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known
|
||||
secret values elsewhere in the runtime. Config diagnostics use a separate
|
||||
clone-and-redact path in `internal/core/config`. These mechanisms implement the
|
||||
security invariant in
|
||||
[Architecture](../policy/architecture.md#state-output-and-safety); operator
|
||||
handling of debug data is defined in [Operations](../operations.md#debug).
|
||||
|
||||
## Secret Redaction
|
||||
## Failure Behavior
|
||||
|
||||
Provider errors are redacted before surfacing through the Scriptorium-backed
|
||||
client. Config diagnostics use redacted effective config payloads.
|
||||
- Invalid targets, missing prompt IDs, malformed structured output, and
|
||||
Scriptorium failures return contextual errors to the calling module.
|
||||
- Scheduler construction rejects non-positive limits; acquisition respects
|
||||
context cancellation.
|
||||
- Asset registration rejects invalid roots, missing content, and path conflicts.
|
||||
- Schema loading distinguishes missing assets, invalid JSON, and invalid
|
||||
metadata.
|
||||
- Profile validation errors occur during CLI preparation when an explicit
|
||||
selected ID cannot be prepared.
|
||||
|
||||
Do not add raw provider request bodies, response bodies, API keys, or prompt
|
||||
payloads to diagnostics by default.
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/framework/llm/scriptorium_client_test.go` and
|
||||
`scriptorium_api_test.go`: adapter mapping and local HTTP integration.
|
||||
- `internal/framework/llm/scheduler_test.go` and
|
||||
`scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper
|
||||
release.
|
||||
- `internal/framework/llm/asset_registry_test.go` and
|
||||
`schema_registry_test.go`: asset composition, validation, and defensive
|
||||
copies.
|
||||
- `internal/framework/llm/secrets_test.go`: provider-error redaction.
|
||||
- `internal/cli/run_test.go`: profile validation, production client wiring,
|
||||
manifest recording, and debug integration.
|
||||
- Module-local `scriptorium_assets_test.go` files: prompt inputs and package
|
||||
asset registration.
|
||||
|
||||
Reference in New Issue
Block a user