eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Inspect PromptKit profiles during preflight

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-03 16:26:15 +00:00
parent 67b315099d
commit 4829f94157
6 changed files with 271 additions and 109 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/internal/cli.md
View File

@@ -39,10 +39,12 @@ in [Configuration Internals](configuration.md).
Configuration validation without a selected pipeline checks structural
configuration only. Validation with a selected pipeline also builds the
effective catalog, resolves the pipeline, and verifies explicitly selected
PromptKit profiles. Each explicit binding or validator profile is prepared
against the configured PromptKit source without performing generation, so an
unknown profile fails before pipeline preparation. Pipeline listing validates
configuration before returning normalized, sorted identifiers.
PromptKit profiles. Each explicit binding or validator profile is inspected
against the configured PromptKit source and backend registrations without
loading a prompt or performing generation, so an unknown or invalid profile
fails before pipeline preparation. Credential availability remains an
execution-time concern. Pipeline listing validates configuration before
returning normalized, sorted identifiers.
## Production Composition

35
docs/internal/llm.md
View File

@@ -47,22 +47,27 @@ states. With neither flag, profile behavior remains unchanged. Because
production constructs one shared client, the selected state applies uniformly
to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. The CLI
prepares every explicitly selected binding profile before a run begins, so a
missing explicit profile fails before stage execution. Calls record the profile
actually selected by PromptKit. The recorder trims and deduplicates non-secret
profile identity, provider, model, selected backend ID, and effective reasoning
values for manifest use. Entries that differ in backend or reasoning remain
distinct and deterministically ordered. Endpoint-only profiles retain an empty
backend ID, which the published JSON omits. Successful completion responses and
recorded profile manifests identify the adapter provider as `promptkit`.
An empty request profile lets the prompt select its configured default. Before a
run begins, the CLI asks the adapter to inspect every explicitly selected
binding profile. Inspection resolves the profile and its selected backend and
target without loading a prompt, reading credentials, admitting capacity, or
contacting a provider, so a missing or invalid explicit profile fails before
stage execution while a valid `api_key_env` may remain unset. Calls record the
profile actually selected by PromptKit. The recorder trims and deduplicates
non-secret profile identity, provider, model, selected backend ID, and
effective reasoning values for manifest use. Entries that differ in backend or
reasoning remain distinct and deterministically ordered. Endpoint-only profiles
retain an empty backend ID, which the published JSON omits. Successful
completion responses and recorded profile manifests identify the adapter
provider as `promptkit`.
The CLI's preparation-only engine and the production adapter use the same
conversion helper to register the optional conventional `local` backend.
Preflight therefore resolves the same backend membership as runtime without
performing generation. When the registration is absent, a profile selecting
`backend: local` fails preparation instead of falling back to a built-in or
endpoint-only target.
The CLI's profile-inspection engine and the production adapter use the same
profile-source construction to apply the configured profile directory or file
and register the optional conventional `local` backend. Preflight therefore
resolves the same ordinary profile source and backend membership as runtime
without performing generation. When the registration is absent, a profile
selecting `backend: local` fails inspection instead of falling back to a
built-in or endpoint-only target.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the