eric/scriptorium
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 9 Wiki Activity

Expose opt-in HTTP execution reports

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-26 15:51:47 +00:00
parent edc1f8abd9
commit c60813e490
8 changed files with 304 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
docs/api.md
View File

@@ -20,7 +20,7 @@ of their `Content-Type`; successful JSON responses use
The configured request-body limit includes inline artifact bodies. The artifact
limit applies to HTTP `file` inputs. The response limit applies to the encoded
response, including the artifact body and optional raw output. A limit of zero
response, including the artifact body, optional raw output, and optional execution report. A limit of zero
disables that limit.
A request body over its limit returns `413 request_too_large`; an oversized
@@ -50,6 +50,7 @@ copyable shape. At the HTTP adapter boundary, the smallest valid shape is:
| `vars` | no | Object mapping template-variable names to strings. |
| `model` | no | Runtime model-override object. |
| `include_raw_output` | no | Include `raw_model_output` when true. |
| `include_execution_report` | no | Include a terminal `execution` report when true and Promptkit supplies a terminal observation. Omitted, `null`, or false leaves reporting off. |
| `transient_retries` | no | Nonnegative additional transient attempt count for this run. Omitted or `null` inherits the configured engine value and then Promptkit's default; `0` disables additional attempts. |
| `run_timeout_seconds` | no | Optional signed 64-bit whole-second limit for engine execution. Omitted, `null`, or `0` adds no deadline; positive values shorten the caller context deadline. |
@@ -118,6 +119,7 @@ validation contract. The response contains:
- `metadata`: run, prompt, rendered-prompt, profile, optional backend identity, model, input-hash, usage,
timing, validation, and repair-attempt metadata; and
- optional `raw_model_output` when requested.
- optional `execution` when requested and a terminal observation exists.
`metadata.model_params` has `endpoint`, `model`, `temperature`,
`max_tokens`, `top_p`, and `timeout_seconds`, plus optional
@@ -139,6 +141,38 @@ A validation failure has `validation.status: "failed"`, `is_valid: false`,
and any available diagnostic errors, while still returning the artifact and
metadata.
### Execution Report
The opt-in `execution` object projects one Promptkit terminal observation. It
is independent of `metadata.usage`, which retains its existing result-based
meaning. A terminal event can describe a failed invocation even when no provider
call was dispatched; if there is no terminal event, `execution` is omitted.
| Field | Meaning |
| --- | --- |
| `run_id`, `prompt_id`, `prompt_version`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
| `outcome` | `success`, `invalid_output`, `error`, or `other` for an unrecognized upstream value. |
| `failure` | The known terminal failure category, `none`, or `other` for an unrecognized value. |
| `status_code` | Provider status, omitted when unavailable. |
| `retries_exhausted` | Boolean independent of the failure category. |
| `attempts`, `transient_retries`, `repair_calls` | Unsigned terminal aggregate counts. |
| `duration_ms`, `capacity_wait_ms`, `generation_duration_ms`, `retry_delay_ms` | Terminal observed durations in integer milliseconds; their sum need not equal wall time. |
| `usage` | Object containing `prompt_tokens`, `completion_tokens`, `total_tokens`, `cached_tokens`, and `cache_write_tokens`. |
Each `usage` count contains `value` (integer), `known` (boolean), and
`complete` (boolean). Reported zero differs from an unknown count; an
incomplete total does not establish complete billing. All counts, durations,
`retries_exhausted`, and `usage` are present even when zero or unknown. The
recognized failure values are `none`, `canceled`, `deadline`, `transport`,
`provider_status`, `malformed_response`, `capacity`, `invalid_config`,
`invalid_request`, `prompt_not_found`, `profile_not_found`, `prompt_load`,
`profile_load`, `artifact_load`, `prompt_render`, `validation`, `generation`,
and `other`.
The report excludes content, credentials, URLs, headers, and raw errors, but
its identifiers can still be sensitive. Apply appropriate access and retention
controls to responses that include it.
## Error Responses
Errors have this shape:
@@ -148,6 +182,12 @@ Errors have this shape:
```
Messages are concise and do not expose wrapped internal causes.
For an opted-in failed invocation, the error envelope can also include the
optional top-level `execution` object described above. Decode and validation
failures before engine invocation do not have a report. Report-bearing error
responses are subject to the complete encoded response-size limit; if they
exceed it, the response is the compact `413 response_too_large` error without
`execution`. The existing compact error fallback itself is not re-limited.
| Status | Code | Meaning |
| --- | --- | --- |

25
docs/cli.md
View File

@@ -129,28 +129,9 @@ successful run, and report writing is attempted even if artifact writing fails.
Any output or report write failure exits 1; already-written output is not
undone. Without a write failure, the actual run keeps its usual exit status.
The JSON object contains the following fields:
| Field | Meaning |
| --- | --- |
| `run_id`, `prompt_id`, `prompt_version`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
| `outcome` | `success`, `invalid_output`, `error`, or `other` for an unrecognized upstream value. |
| `failure` | The known terminal failure category, `none`, or `other` for an unrecognized value. |
| `status_code` | Provider status, omitted when unavailable. |
| `retries_exhausted` | Boolean independent of the failure category. |
| `attempts`, `transient_retries`, `repair_calls` | Unsigned terminal aggregate counts. |
| `duration_ms`, `capacity_wait_ms`, `generation_duration_ms`, `retry_delay_ms` | Terminal observed durations in integer milliseconds; their sum need not equal wall time. |
| `usage` | Object containing `prompt_tokens`, `completion_tokens`, `total_tokens`, `cached_tokens`, and `cache_write_tokens`. |
Each `usage` count contains `value` (integer), `known` (boolean), and
`complete` (boolean). Reported zero differs from an unknown count; an
incomplete total does not establish complete billing. All counts, durations,
`retries_exhausted`, and `usage` are present even when zero or unknown. The
recognized failure values are `none`, `canceled`, `deadline`, `transport`,
`provider_status`, `malformed_response`, `capacity`, `invalid_config`,
`invalid_request`, `prompt_not_found`, `profile_not_found`, `prompt_load`,
`profile_load`, `artifact_load`, `prompt_render`, `validation`, `generation`,
and `other`.
The [HTTP API reference](api.md#execution-report) defines the shared report
fields and usage-presence semantics. The file contains that JSON object followed
by a newline.
The report excludes content, credentials, URLs, headers, and raw errors, but
its identifiers can still be sensitive. The caller owns file retention and

7
docs/internal/adapters.md
View File

@@ -81,6 +81,13 @@ CLI `run` attaches one only when a report path is selected, writes generated
output first on success, and then writes the terminal snapshot when present.
Its report path remains untouched for pre-invocation failures or absent
terminal observations; report and artifact writes are independent.
HTTP `POST /v1/runs` attaches a separate collector to each invocation only when
`include_execution_report` is true. Its optional top-level `execution` field
appears in success or error envelopes only when Promptkit emits a terminal
event. Pre-invocation adapter errors and absent terminal events leave it out.
The complete serialized success or report-bearing error response is subject to
the configured response-size limit; overflow returns the compact 413 error
without a report. Field meanings are owned by the [API contract](../api.md#execution-report).
## Package-Local Guarantees

5
docs/operations.md
View File

@@ -31,6 +31,11 @@ contain sensitive identifiers even though they omit prompt and response bodies.
Keep report paths separate from generated output and apply the same access and
retention controls. Output and report writes are not transactional; an older
report may remain when no terminal observation was produced.
HTTP callers may also request a [terminal execution report](api.md#execution-report).
It shares the same identifier sensitivity and should receive response access
and retention controls. An opted-in report increases the encoded response size;
an oversized success or report-bearing error becomes the compact 413 response
without a report.
## Deploy The Filesystem And Process

2
docs/roadmap/implementation.md
View File

@@ -349,7 +349,7 @@ do not erase the original run failure. Render and inspect remain unchanged.
## Stage 8: Expose HTTP Execution Reports
**Status: Pending.**
**Status: Complete.**
Selected interface: optional top-level request boolean
`include_execution_report`. Omission, null, or false leaves reporting off,