Publish effective LLM backend provenance

This commit is contained in:
2026-07-30 02:18:14 +00:00
parent f8333f2c15
commit 71a004bfc8
11 changed files with 194 additions and 32 deletions

View File

@@ -98,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
provenance for the plan used by this run; cache records, debug artifacts, and
other operational state are not published as bundle files.
Each `llm_profiles` entry identifies effective, non-secret LLM execution
provenance:
| Field | Required | Meaning |
| --- | --- | --- |
| `id` | Yes | Selected PromptKit profile identifier. |
| `provider` | No | Notarius adapter provider identifier. |
| `model` | No | Effective provider model identifier. |
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
These values describe observed execution; they are not a backend-registration
interface. Entries that differ by backend or effective reasoning remain
distinct even when their profile, provider, and model are otherwise equal.
## Rejections And Warnings
`rejected.json` is always an object with a `rejected` array. Each entry has

View File

@@ -17,7 +17,7 @@ Notarius relies on the root `promptkit` package to:
- prepare and run a `RunRequest` with named inline artifacts, variables,
a direct session ID, prompt identity, and profile selection;
- return rendered debug material, validated structured output, selected
profile and model metadata, and token usage;
profile, backend, effective model metadata, and token usage;
- distinguish structured-output validation failure from execution failure; and
- identify a missing explicit profile through `ErrProfileNotFound`.
@@ -33,6 +33,13 @@ the same value as the `session_id` prompt variable for maintained prompt
compatibility. Session IDs are stable, non-secret correlation identifiers and
may be exposed to providers and provider observability.
Notarius records PromptKit's selected backend ID and effective reasoning
setting as optional run-manifest provenance. Endpoint-only profiles have no
backend ID. Debug prompt material also retains the selected backend ID and
PromptKit's stable lower-case `effective_model_params` JSON, which may include
`backend_id`. Notarius production configuration does not expose user-defined
PromptKit backend registration.
## Notarius Ownership
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts

View File

@@ -49,10 +49,12 @@ to module calls, retries, and LLM-backed validators for the whole run.
An empty request profile lets the prompt select its configured default. The CLI
prepares every explicitly selected binding profile before a run begins, so a
missing explicit profile fails before stage execution. Calls record the profile
actually selected by PromptKit; the recorder deduplicates non-secret profile
identity, provider, and model values for manifest use. Successful completion
responses and recorded profile manifests identify the adapter provider as
`promptkit`.
actually selected by PromptKit. The recorder trims and deduplicates non-secret
profile identity, provider, model, selected backend ID, and effective reasoning
values for manifest use. Entries that differ in backend or reasoning remain
distinct and deterministically ordered. Endpoint-only profiles retain an empty
backend ID, which the published JSON omits. Successful completion responses and
recorded profile manifests identify the adapter provider as `promptkit`.
Before execution, the adapter also contributes a non-secret checkpoint
fingerprint for the effective PromptKit profile source. It combines the
@@ -159,18 +161,22 @@ contract is identified in
When debug recording is enabled, the pipeline decorates the shared client. The
wrapper records prepared prompt and response material, timing, selected profile
and model, and call identifiers in the runs debug bundle, including material
available from a failed structured completion. For a successful completion, a
debug-write failure is surfaced; when the completion already failed, its call
error remains the result. Debug-bundle location, retention, and handling are
operational concerns documented in [Operations](../operations.md#debug-bundles).
and backend, effective model parameters, and call identifiers in the runs
debug bundle, including material available from a failed structured completion.
Effective parameters use PromptKit's stable lower-case JSON field names and may
include `backend_id`. For a successful completion, a debug-write failure is
surfaced; when the completion already failed, its call error remains the
result. Debug-bundle location, retention, and handling are operational concerns
documented in [Operations](../operations.md#debug-bundles).
Run manifests receive selected profile summaries and component identities, not
prompt, schema, source, reference, or response content. Provider error text is
wrapped with prompt context and bearer credentials are redacted before it
crosses the runtime boundary. Known-secret redaction is available to other
runtime collaborators; it does not make prompt or response contents safe for
general logging.
Run manifests receive selected profile summaries, including optional effective
backend and reasoning provenance, and component identities—not prompt, schema,
source, reference, or response content. The published field semantics belong
to the [JSON output contract](../integrations/json-output.md#manifestjson).
Provider error text is wrapped with prompt context and bearer credentials are
redacted before it crosses the runtime boundary. Known-secret redaction is
available to other runtime collaborators; it does not make prompt or response
contents safe for general logging.
## Failure Boundaries