Publish effective LLM backend provenance
This commit is contained in:
@@ -98,6 +98,21 @@ summarize results without embedding lane payload bytes. A chunk-plan summary is
|
||||
provenance for the plan used by this run; cache records, debug artifacts, and
|
||||
other operational state are not published as bundle files.
|
||||
|
||||
Each `llm_profiles` entry identifies effective, non-secret LLM execution
|
||||
provenance:
|
||||
|
||||
| Field | Required | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `id` | Yes | Selected PromptKit profile identifier. |
|
||||
| `provider` | No | Notarius adapter provider identifier. |
|
||||
| `model` | No | Effective provider model identifier. |
|
||||
| `backend_id` | No | Effective PromptKit backend registration identifier. Endpoint-only profiles omit it. |
|
||||
| `reasoning_effort` | No | Effective opaque provider reasoning setting. An empty or explicitly cleared setting is omitted. |
|
||||
|
||||
These values describe observed execution; they are not a backend-registration
|
||||
interface. Entries that differ by backend or effective reasoning remain
|
||||
distinct even when their profile, provider, and model are otherwise equal.
|
||||
|
||||
## Rejections And Warnings
|
||||
|
||||
`rejected.json` is always an object with a `rejected` array. Each entry has
|
||||
|
||||
@@ -17,7 +17,7 @@ Notarius relies on the root `promptkit` package to:
|
||||
- prepare and run a `RunRequest` with named inline artifacts, variables,
|
||||
a direct session ID, prompt identity, and profile selection;
|
||||
- return rendered debug material, validated structured output, selected
|
||||
profile and model metadata, and token usage;
|
||||
profile, backend, effective model metadata, and token usage;
|
||||
- distinguish structured-output validation failure from execution failure; and
|
||||
- identify a missing explicit profile through `ErrProfileNotFound`.
|
||||
|
||||
@@ -33,6 +33,13 @@ the same value as the `session_id` prompt variable for maintained prompt
|
||||
compatibility. Session IDs are stable, non-secret correlation identifiers and
|
||||
may be exposed to providers and provider observability.
|
||||
|
||||
Notarius records PromptKit's selected backend ID and effective reasoning
|
||||
setting as optional run-manifest provenance. Endpoint-only profiles have no
|
||||
backend ID. Debug prompt material also retains the selected backend ID and
|
||||
PromptKit's stable lower-case `effective_model_params` JSON, which may include
|
||||
`backend_id`. Notarius production configuration does not expose user-defined
|
||||
PromptKit backend registration.
|
||||
|
||||
## Notarius Ownership
|
||||
|
||||
[LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts
|
||||
|
||||
@@ -49,10 +49,12 @@ to module calls, retries, and LLM-backed validators for the whole run.
|
||||
An empty request profile lets the prompt select its configured default. The CLI
|
||||
prepares every explicitly selected binding profile before a run begins, so a
|
||||
missing explicit profile fails before stage execution. Calls record the profile
|
||||
actually selected by PromptKit; the recorder deduplicates non-secret profile
|
||||
identity, provider, and model values for manifest use. Successful completion
|
||||
responses and recorded profile manifests identify the adapter provider as
|
||||
`promptkit`.
|
||||
actually selected by PromptKit. The recorder trims and deduplicates non-secret
|
||||
profile identity, provider, model, selected backend ID, and effective reasoning
|
||||
values for manifest use. Entries that differ in backend or reasoning remain
|
||||
distinct and deterministically ordered. Endpoint-only profiles retain an empty
|
||||
backend ID, which the published JSON omits. Successful completion responses and
|
||||
recorded profile manifests identify the adapter provider as `promptkit`.
|
||||
|
||||
Before execution, the adapter also contributes a non-secret checkpoint
|
||||
fingerprint for the effective PromptKit profile source. It combines the
|
||||
@@ -159,18 +161,22 @@ contract is identified in
|
||||
|
||||
When debug recording is enabled, the pipeline decorates the shared client. The
|
||||
wrapper records prepared prompt and response material, timing, selected profile
|
||||
and model, and call identifiers in the run’s debug bundle, including material
|
||||
available from a failed structured completion. For a successful completion, a
|
||||
debug-write failure is surfaced; when the completion already failed, its call
|
||||
error remains the result. Debug-bundle location, retention, and handling are
|
||||
operational concerns documented in [Operations](../operations.md#debug-bundles).
|
||||
and backend, effective model parameters, and call identifiers in the run’s
|
||||
debug bundle, including material available from a failed structured completion.
|
||||
Effective parameters use PromptKit's stable lower-case JSON field names and may
|
||||
include `backend_id`. For a successful completion, a debug-write failure is
|
||||
surfaced; when the completion already failed, its call error remains the
|
||||
result. Debug-bundle location, retention, and handling are operational concerns
|
||||
documented in [Operations](../operations.md#debug-bundles).
|
||||
|
||||
Run manifests receive selected profile summaries and component identities, not
|
||||
prompt, schema, source, reference, or response content. Provider error text is
|
||||
wrapped with prompt context and bearer credentials are redacted before it
|
||||
crosses the runtime boundary. Known-secret redaction is available to other
|
||||
runtime collaborators; it does not make prompt or response contents safe for
|
||||
general logging.
|
||||
Run manifests receive selected profile summaries, including optional effective
|
||||
backend and reasoning provenance, and component identities—not prompt, schema,
|
||||
source, reference, or response content. The published field semantics belong
|
||||
to the [JSON output contract](../integrations/json-output.md#manifestjson).
|
||||
Provider error text is wrapped with prompt context and bearer credentials are
|
||||
redacted before it crosses the runtime boundary. Known-secret redaction is
|
||||
available to other runtime collaborators; it does not make prompt or response
|
||||
contents safe for general logging.
|
||||
|
||||
## Failure Boundaries
|
||||
|
||||
|
||||
Reference in New Issue
Block a user