Expose opt-in HTTP execution reports

This commit is contained in:
2026-09-26 15:51:47 +00:00
parent edc1f8abd9
commit c60813e490
8 changed files with 304 additions and 44 deletions

View File

@@ -20,7 +20,7 @@ of their `Content-Type`; successful JSON responses use
The configured request-body limit includes inline artifact bodies. The artifact
limit applies to HTTP `file` inputs. The response limit applies to the encoded
response, including the artifact body and optional raw output. A limit of zero
response, including the artifact body, optional raw output, and optional execution report. A limit of zero
disables that limit.
A request body over its limit returns `413 request_too_large`; an oversized
@@ -50,6 +50,7 @@ copyable shape. At the HTTP adapter boundary, the smallest valid shape is:
| `vars` | no | Object mapping template-variable names to strings. |
| `model` | no | Runtime model-override object. |
| `include_raw_output` | no | Include `raw_model_output` when true. |
| `include_execution_report` | no | Include a terminal `execution` report when true and Promptkit supplies a terminal observation. Omitted, `null`, or false leaves reporting off. |
| `transient_retries` | no | Nonnegative additional transient attempt count for this run. Omitted or `null` inherits the configured engine value and then Promptkit's default; `0` disables additional attempts. |
| `run_timeout_seconds` | no | Optional signed 64-bit whole-second limit for engine execution. Omitted, `null`, or `0` adds no deadline; positive values shorten the caller context deadline. |
@@ -118,6 +119,7 @@ validation contract. The response contains:
- `metadata`: run, prompt, rendered-prompt, profile, optional backend identity, model, input-hash, usage,
timing, validation, and repair-attempt metadata; and
- optional `raw_model_output` when requested.
- optional `execution` when requested and a terminal observation exists.
`metadata.model_params` has `endpoint`, `model`, `temperature`,
`max_tokens`, `top_p`, and `timeout_seconds`, plus optional
@@ -139,6 +141,38 @@ A validation failure has `validation.status: "failed"`, `is_valid: false`,
and any available diagnostic errors, while still returning the artifact and
metadata.
### Execution Report
The opt-in `execution` object projects one Promptkit terminal observation. It
is independent of `metadata.usage`, which retains its existing result-based
meaning. A terminal event can describe a failed invocation even when no provider
call was dispatched; if there is no terminal event, `execution` is omitted.
| Field | Meaning |
| --- | --- |
| `run_id`, `prompt_id`, `prompt_version`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
| `outcome` | `success`, `invalid_output`, `error`, or `other` for an unrecognized upstream value. |
| `failure` | The known terminal failure category, `none`, or `other` for an unrecognized value. |
| `status_code` | Provider status, omitted when unavailable. |
| `retries_exhausted` | Boolean independent of the failure category. |
| `attempts`, `transient_retries`, `repair_calls` | Unsigned terminal aggregate counts. |
| `duration_ms`, `capacity_wait_ms`, `generation_duration_ms`, `retry_delay_ms` | Terminal observed durations in integer milliseconds; their sum need not equal wall time. |
| `usage` | Object containing `prompt_tokens`, `completion_tokens`, `total_tokens`, `cached_tokens`, and `cache_write_tokens`. |
Each `usage` count contains `value` (integer), `known` (boolean), and
`complete` (boolean). Reported zero differs from an unknown count; an
incomplete total does not establish complete billing. All counts, durations,
`retries_exhausted`, and `usage` are present even when zero or unknown. The
recognized failure values are `none`, `canceled`, `deadline`, `transport`,
`provider_status`, `malformed_response`, `capacity`, `invalid_config`,
`invalid_request`, `prompt_not_found`, `profile_not_found`, `prompt_load`,
`profile_load`, `artifact_load`, `prompt_render`, `validation`, `generation`,
and `other`.
The report excludes content, credentials, URLs, headers, and raw errors, but
its identifiers can still be sensitive. Apply appropriate access and retention
controls to responses that include it.
## Error Responses
Errors have this shape:
@@ -148,6 +182,12 @@ Errors have this shape:
```
Messages are concise and do not expose wrapped internal causes.
For an opted-in failed invocation, the error envelope can also include the
optional top-level `execution` object described above. Decode and validation
failures before engine invocation do not have a report. Report-bearing error
responses are subject to the complete encoded response-size limit; if they
exceed it, the response is the compact `413 response_too_large` error without
`execution`. The existing compact error fallback itself is not re-limited.
| Status | Code | Meaning |
| --- | --- | --- |