From fdf8c4afd4df08e7e102bd211694bec92e45bfb5 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 26 Jul 2026 13:33:45 +0000 Subject: [PATCH] Rewrite LLM runtime documentation --- docs/internal/llm.md | 343 +++++++++++++++---------------------------- 1 file changed, 120 insertions(+), 223 deletions(-) diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 5d264e2..85d66b6 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -1,252 +1,149 @@ # LLM Runtime Internals -`internal/framework/llm` implements Notarius's transport boundary for structured -completion. It contains the Scriptorium adapter, concurrency scheduler, -prompt/schema registries, selected-profile recording, and provider-error -redaction. +`internal/framework/llm` is Notarius’s provider-independent structured +completion boundary. It adapts framework requests to Scriptorium, bounds +provider calls, assembles registered prompt and schema assets, records selected +profiles, and redacts provider errors. The architectural boundary is defined in +[Architecture](../policy/architecture.md#llm-boundary); profile sources, +credentials, and concurrency settings belong in +[Configuration](../config.md#scriptorium-profiles) and +[Configuration](../config.md#concurrency-output-cache-and-debug). -Provider-neutral ownership rules are defined in -[Architecture](../policy/architecture.md#llm-boundary). Profile sources, -credentials, and concurrency settings are defined in -[Configuration](../config.md). +## Structured Completion Boundary -## Structured Contract +Modules and LLM-backed validators depend only on +`contracts.StructuredLLMClient`. A completion request supplies a prompt ID and +version, optional profile and session IDs, named input material, variables, and +a caller-owned decode target. The successful response returns the validated raw +structured bytes together with non-secret provider, model, profile, and token +metadata. -Modules and LLM-backed validators depend on -`contracts.StructuredLLMClient.CompleteStructured`. A request identifies a -prompt and optional profile/session, supplies named input materials and -variables, and provides a caller-owned decoding target. A successful response -contains the validated raw structured bytes plus non-secret provider, model, -profile, and token metadata. +The caller owns the domain behavior: it chooses the prompt, prepares inputs, +selects the private response schema, and interprets the decoded result. The +adapter does not own source evidence, artifact conversion, normalization, or +durable schemas. Those responsibilities remain with the module and its +[integration contract](../integrations/). -The caller owns prompt selection, response-schema selection, and interpretation -of the decoded result. `LLMInputMaterial` keeps source and reference bytes with -their origin metadata so the adapter can pass named artifacts to Scriptorium -without exposing Scriptorium types through stage contracts. +`ScriptoriumClient` validates the request target and prompt identity, maps each +named material to a Scriptorium inline artifact while preserving its origin URI, +forwards session and profile selection, then prepares and runs the prompt. It +returns Scriptorium’s validated raw bytes rather than re-encoding the decoded +target. An empty optional material is represented as one space so its named +input is retained by Scriptorium. -## Production Construction +An empty request profile lets the prompt select its configured default. The CLI +prepares every explicitly selected binding profile before a run begins, so a +missing explicit profile fails before stage execution. Calls record the profile +actually selected by Scriptorium; the recorder deduplicates non-secret profile +identity, provider, and model values for manifest use. -`internal/cli` constructs the production runtime by: +## Shared Provider-Call Limit -1. allocating the asset registry populated by the generic, Seriatim, and D&D - package-family registrars; -2. creating a `ScriptoriumClient` from the effective profile source; -3. attaching an `LLMProfileRecorder`; -4. creating a scheduler from the effective concurrency limit; -5. returning a `ScheduledClient` wrapper; -6. decorating that shared client before preparation when debug recording is - enabled; and -7. injecting that one shared client into complete pipeline preparation before - the source file is read or the runner is invoked. +Production construction creates one Scriptorium client and wraps it in one +scheduled client. The scheduler has a fixed, positive permit limit, serves +queued calls in FIFO order, and removes a queued call when its context is +cancelled. A granted permit is released exactly once on every completion path. -The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and -scene-description extractors retain this -injected client and use it for every structured completion. Operation requests -do not carry an LLM client. - -The CLI separately gathers explicit profile IDs from resolved LLM-capable stage -and validator bindings. It prepares a small internal check prompt for each ID so -missing or invalid profiles fail before pipeline execution. The runtime profile -override syntax and scope are defined in the -[CLI reference](../cli.md#run); binding rules are defined in -[Configuration](../config.md#module-bindings). - -## Scriptorium Adapter - -`ScriptoriumClient` converts a Notarius request into a Scriptorium `RunRequest`. -It validates the decoding target and prompt identity, maps named input materials -to inline artifacts, forwards explicit profile and session context, delegates -rendering/provider execution/structured validation, and unmarshals successful -JSON into the caller target. - -Empty optional input material is represented by a single space so Scriptorium -retains the named input. The client returns Scriptorium's validated structured -bytes rather than re-encoding the caller target, allowing modules to preserve -the runtime result exactly. - -Selected profile, provider, model, and token metadata are mapped into the -Notarius response. The recorder deduplicates profiles by identity and supplies -manifest-safe profile summaries after actual calls; manifest population does -not guess the selected prompt default in advance. - -Generated-output validation failures and provider failures are wrapped with -prompt context. Error strings pass through bearer-token redaction before they -cross the runtime boundary. - -## Scheduling - -`Scheduler` uses a bounded permit count and a FIFO waiter queue. Immediate -acquisition increments the in-flight count; queued acquisition waits for a -permit or context cancellation. Cancellation removes a queued waiter, while a -cancelled waiter that has already received a permit releases it. - -`ScheduledClient` acquires a permit around each structured completion and -defers release on every result path. The effective limit and default are -configuration facts in [Configuration](../config.md#defaults). - -This provider-call ceiling is independent of the pipeline's extract worker -limit. Concurrent lanes, retries, and validators all use the same scheduled -client, so increasing framework workers cannot exceed `total_llm`. Pipeline -dispatch and cancellation mechanics are documented in -[Pipeline Internals](pipeline.md#execution-flow). +The scheduled wrapper surrounds every `CompleteStructured` call, so concurrent +lanes, pipeline retries, and LLM-backed validators share the same provider-call +ceiling. This ceiling is independent of pipeline worker concurrency; changing +worker counts cannot exceed the configured LLM limit. The configuration field +and its effective default are owned by +[Configuration](../config.md#concurrency-output-cache-and-debug). ## Prompt And Schema Assets -`AssetRegistry` combines caller-owned prompt filesystems under stable prefixes -and rejects invalid or conflicting registrations. Production module packages -register their own prompt and schema assets; generic framework code contains no -D&D prompt content. `internal/framework/promptfs` provides the domain-neutral -filesystem composition helper used to combine module-owned files with shared -domain prompt fragments. +An `AssetRegistry` collects prompt and schema filesystems from production module +families. It flattens registered roots into the Scriptorium filesystems and +rejects invalid roots, unreadable assets, duplicate paths, and missing prompt +or schema files during preparation. The framework’s `promptfs` helper combines +module-owned prompt files with reusable domain fragments without making the +framework depend on D&D content. -The D&D scene chunker and spell, NPC, combat-turn, item-event, NPC-interaction, and -scene-description extractors each declare an -ordered prompt asset manifest. The manifest lists the package-owned YAML and -Markdown files, then the exact shared fragments rendered by that prompt; the -same ordered list drives both filesystem mounting and the prompt fingerprint. -Unused shared assets are neither mounted nor fingerprinted. Universal -extraction-evidence and output policy lives only in the shared extraction -assets; package-owned prompt files retain artifact-specific rules. The scene -prompt keeps its separate output rule because it does not render the -extraction-evidence asset. +Each LLM-backed module owns its prompt declaration, package-specific assets, +and private response schema. Shared D&D wording is owned by the D&D shared +asset package; the detailed D&D conventions are in +[D&D Module Internals](dnd.md). The mounted prompt assets used by a module also +determine its prompt fingerprint. Schema loaders validate JSON, attach identity +and digest metadata, make defensive copies, and expose diagnostics without raw +schema bytes. -### D&D Extraction Prompt Ordering And Cache Boundaries +Private response schemas validate a model transport envelope. They are not the +durable artifact schema and should not be documented as an external wire +contract. Durable formats and compatibility rules remain in the +[integration contracts](../integrations/). -D&D extraction prompts order messages from the most reusable content to the -most variable content. New extraction lanes use these tiers in order: +## Prompt Maintenance And Backend Caching -1. universal shared content, including the system, extraction-evidence, and - in-world identity messages; -2. stable campaign or run context shared across lanes, including campaign - references; -3. stable subset- and lane-specific context and instructions, including an NPC - registry, catalog, task, or extraction instructions when applicable; -4. the chunk transcript as the final user message. +Prompt message order and shared asset bytes are runtime behavior. Backend cache +reuse depends on the same preceding messages and content, not merely equivalent +meaning. Keep reusable shared assets byte-identical and keep stable material +before the inputs that vary per request wherever a prompt’s declared sequence +supports caching. Preserve the existing manifest order and cache-control hints +when editing a prompt. -This ordering lets requests reuse the longest identical prefix before the -per-chunk transcript changes. Cache reuse requires the preceding message -sequence and content to be exactly identical; semantic similarity is not -sufficient. Cache boundaries belong at the ends of reusable stable tiers, -subject to the provider's cache-boundary limit. The shared identity and -campaign-reference messages form the first two extraction boundaries. Spell, -combat, and interaction prompts add a boundary at the shared NPC registry. Each extraction -prompt places its final boundary on its lane-specific instructions, immediately -before the transcript. The transcript does not carry cache control because no -reusable content follows it. +D&D extraction manifests place the changing chunk transcript at the end of the +prompt after their reusable context. Scene chunking and NPC normalization use +their own declared message sequences because their inputs and work differ. The +family-specific asset and ordering rules belong in [D&D Module Internals](dnd.md). +Do not add tests that enforce a fixed message-prefix length; prompt-asset tests +should instead verify the meaningful asset sequence, inputs, and cache controls +of the prompt being changed. -Accordingly, the common prefix of the spell, NPC, combat, item-event, and interaction -extraction prompts is system, -extraction evidence, identity, and campaign references. The NPC prompt then -renders task, instructions, and transcript. Spell renders the NPC registry, -catalog, task, instructions, and transcript. Combat renders the NPC registry, -task, instructions, and transcript. Item-event renders task, instructions, and -transcript without a generated-artifact input. NPC interaction renders the names-only NPC -registry, task, instructions, and transcript. The -scene chunker is not an extraction lane: it retains its separate system, -transcript, campaign-reference, task, and instruction order and marks its -transcript and campaign-reference messages ephemeral. +## Validation, Repair, And Retries -The scene-description extractor deliberately omits the citation-oriented -`common-dnd-extraction-evidence.md` asset because Notarius attaches the whole -accepted chunk range itself. Its manifest is system, shared identity, shared -campaign references, lane task, lane instructions, then the transcript. The -identity, campaign-reference, and instruction messages are ephemeral cache -boundaries; the transcript is last and has no cache control. Compatible shared -messages remain canonical shared assets rather than copied package text. +Scriptorium performs prompt rendering, provider execution, and the prompt’s +structured-output validation. The adapter reports an empty result, validation +failure, empty structured body, or decode failure as +`ErrInvalidStructuredOutput`, while retaining the returned raw bytes and debug +material when they exist. Provider failures remain operational errors rather +than output-validation failures. -### D&D NPC Normalization Prompt Ordering And Cache Boundaries +Prompt-declared repair is executed within Scriptorium’s structured-output flow. +The current production D&D prompt manifests set repair attempts to zero. That +setting does not replace pipeline retry behavior: a binding’s configured retry +count reruns its stage attempt after an error or rejection, and an exhausted +rejection is a recorded output rather than a provider error. The pipeline owns +attempt lifecycle, validation chains, and retry diagnostics; see +[Pipeline Internals](pipeline.md#validation-retries-and-output) and the +[binding reference](../config.md#module-bindings-and-validators). -NPC normalization has a distinct prompt and response-schema identity from NPC -extraction. Its stable message tiers are the common D&D system asset, followed -by package-owned task and normalization instructions. Cache boundaries follow -the shared system tier and the package instructions. The variable tail contains -the private candidate-name-and-range input and a windowed transcript input -whose cited units provide local context; neither has a cache boundary because -it changes with the document. +## Observability And Redaction -This prompt intentionally omits shared identity guidance, -extraction-evidence, and campaign-reference assets: it reconciles existing -records rather than extracting events or adding evidence. Its package-owned -manifest and schema identity are fingerprinted separately, so a normalization -prompt or schema change cannot reuse a prior normalization checkpoint. +When debug recording is enabled, the pipeline decorates the shared client. The +wrapper records prepared prompt and response material, timing, selected profile +and model, and call identifiers in the run’s debug bundle, including material +available from a failed structured completion. For a successful completion, a +debug-write failure is surfaced; when the completion already failed, its call +error remains the result. Debug-bundle location, retention, and handling are +operational concerns documented in [Operations](../operations.md#debug-bundles). -Shared wording belongs in the canonical assets under -`internal/modules/dnd/shared`; extraction packages reference those assets in -their manifests instead of copying similar text into package-local files. -Package-local assets contain only lane-specific content. An extraction lane may -depart from the tier order only when prompt-quality evidence or a provider -constraint makes the exception necessary; document the exception and rationale -here when it becomes implemented behavior. +Run manifests receive selected profile summaries and component identities, not +prompt, schema, source, reference, or response content. Provider error text is +wrapped with prompt context and bearer credentials are redacted before it +crosses the runtime boundary. Known-secret redaction is available to other +runtime collaborators; it does not make prompt or response contents safe for +general logging. -Schema helpers load embedded JSON Schema with identity and digest metadata, -return defensive copies, and expose a diagnostics map that omits schema bytes. -The small framework registry contains only generic test schemas; production -schemas remain package-owned. +## Failure Boundaries -The spell, NPC, combat, item-event, NPC-interaction, and scene-description extractors' -package-owned prompts declare their -structured JSON inputs and private response schemas. Each private response -schema remains separate from its durable artifact codec schema; this work does -not use shared schema fragments or schema generation. Those private schemas own -the transport envelope—required fields, JSON types, nullability, and -unknown-field rejection—while deterministic validators own semantic constraints -such as enum membership, non-empty values and collections, and positive -numbers. The spell extractor's prompt declares a required -`application/json` `spell_catalog` input and an optional `application/json` -`npcs` input. The extractor generates -the catalog input from its prepared -effective catalog as `{"spell_names":[...]}` using sorted canonical names only. -The shared D&D prompt assets include a generic NPC grounding fragment directly -after the campaign reference message for spell, combat, and interaction prompts. When an NPC -registry is bound, the -domain registry boundary strictly decodes and identity-validates one durable -artifact, re-encodes canonical JSON for provenance, and separately generates a -names-only prompt projection. The unbound projection is exactly `{"npcs":[]}`. -Prompt input and component-local checkpoint digests cover the projected bytes; -manifests retain the optional full registry digest/count rather than names, -overlay bytes, registry paths, or source metadata. Combat and interaction prompt, -response-schema, mapping, normalization, identity, and registry-projection -fingerprints remain separate semantic inputs to checkpoint identity. +- Construction fails for missing asset registries, mutually exclusive profile + sources, invalid asset registration, or a non-positive scheduler limit. +- Preparation failures, unavailable explicit profiles, provider failures, and + context cancellation propagate to the calling stage with context. +- Malformed or schema-invalid provider output is classified separately as + invalid structured output so the module or pipeline can apply its own retry + and rejection policy. +- Domain semantic checks, evidence decisions, and deterministic normalization + run outside the provider adapter. -## Debug And Redaction Boundaries +## Focused Verification -The pipeline may wrap the client with a debug recorder that captures prepared -prompt/response material for an explicitly requested debug run. Debug summaries -and manifests receive identities, hashes, usage, and selected profile summaries -rather than prompt, source, reference, schema, or response content. +Read the LLM adapter, scheduler, asset registry, schema loader, and redaction +tests when changing this boundary. Prompt changes also require the owning +module’s asset tests, and retry or debug changes require focused pipeline or +CLI coverage. The focused runtime and D&D checks are: -The Scriptorium error wrapper removes bearer credential values from surfaced -provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known -secret values elsewhere in the runtime. Config summaries use a separate -clone-and-redact path in `internal/core/config`. These mechanisms implement the -security invariant in -[Architecture](../policy/architecture.md#state-output-and-safety); operator -handling of debug data is defined in [Operations](../operations.md#debug). - -## Failure Behavior - -- Invalid targets, missing prompt IDs, malformed structured output, and - Scriptorium failures return contextual errors to the calling module. -- Scheduler construction rejects non-positive limits; acquisition respects - context cancellation. -- Asset registration rejects invalid roots, missing content, and path conflicts. -- Schema loading distinguishes missing assets, invalid JSON, and invalid - metadata. -- Profile validation errors occur during CLI preparation when an explicit - selected ID cannot be prepared. - -## Tests To Inspect - -- `internal/framework/llm/scriptorium_client_test.go`: adapter mapping and local - HTTP integration. -- `internal/framework/llm/scheduler_test.go` and - `scheduled_client_test.go`: permits, FIFO behavior, cancellation, and wrapper - release. -- `internal/framework/llm/asset_registry_test.go` and - `schema_registry_test.go`: asset composition, validation, and defensive - copies. -- `internal/framework/llm/secrets_test.go`: provider-error redaction. -- `internal/cli/run_contract_test.go`: profile validation, production client - wiring, manifest recording, and debug integration. -- Module-local `scriptorium_assets_test.go` files: prompt inputs and package - asset registration. +~~~sh +go test ./internal/framework/llm/... ./internal/modules/dnd/... +~~~