eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Inspect PromptKit profiles during preflight

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-03 16:26:15 +00:00
parent 67b315099d
commit 4829f94157
6 changed files with 271 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
docs/internal/llm.md
View File

@@ -47,22 +47,27 @@ states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. The CLI
prepares every explicitly selected binding profile before a run begins, so a
missing explicit profile fails before stage execution. Calls record the profile
actually selected by PromptKit. The recorder trims and deduplicates non-secret
profile identity, provider, model, selected backend ID, and effective reasoning
values for manifest use. Entries that differ in backend or reasoning remain
distinct and deterministically ordered. Endpoint-only profiles retain an empty
backend ID, which the published JSON omits. Successful completion responses and
recorded profile manifests identify the adapter provider as `promptkit`.
An empty request profile lets the prompt select its configured default. Before a
run begins, the CLI asks the adapter to inspect every explicitly selected
binding profile. Inspection resolves the profile and its selected backend and
target without loading a prompt, reading credentials, admitting capacity, or
contacting a provider, so a missing or invalid explicit profile fails before
stage execution while a valid `api_key_env` may remain unset. Calls record the
profile actually selected by PromptKit. The recorder trims and deduplicates
non-secret profile identity, provider, model, selected backend ID, and
effective reasoning values for manifest use. Entries that differ in backend or
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
retain an empty backend ID, which the published JSON omits. Successful
completion responses and recorded profile manifests identify the adapter
provider as `promptkit`.
The CLI's preparation-only engine and the production adapter use the same
conversion helper to register the optional conventional `local` backend.
Preflight therefore resolves the same backend membership as runtime without
performing generation. When the registration is absent, a profile selecting
`backend: local` fails preparation instead of falling back to a built-in or
endpoint-only target.
The CLI's profile-inspection engine and the production adapter use the same
profile-source construction to apply the configured profile directory or file
and register the optional conventional `local` backend. Preflight therefore
resolves the same ordinary profile source and backend membership as runtime
without performing generation. When the registration is absent, a profile
selecting `backend: local` fails inspection instead of falling back to a
built-in or endpoint-only target.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the