Add opt-in CLI execution report files
This commit is contained in:
46
docs/cli.md
46
docs/cli.md
@@ -56,6 +56,7 @@ Optional flags:
|
||||
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
|
||||
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
|
||||
| `--out <path>` | Write generated content to this file instead of stdout. |
|
||||
| `--execution-report <path>` | Write an opt-in terminal execution report as newline-terminated JSON to a separate file. |
|
||||
| `--llm-base-url <url>` | Runtime endpoint override. |
|
||||
| `--model <name>` | Runtime model override. |
|
||||
| `--api-key-env <name>` | Runtime API-key environment-variable name override. |
|
||||
@@ -110,6 +111,51 @@ profile value, a nonblank value replaces that value, and
|
||||
`--reasoning-effort=` explicitly clears inherited reasoning. Promptkit treats
|
||||
nonblank values as provider-specific opaque strings.
|
||||
|
||||
### Execution report file
|
||||
|
||||
`--execution-report` requires a nonempty path. It is available only for `run`;
|
||||
`-` is an ordinary filename. Parent directories are not created. An existing
|
||||
file is replaced through ordinary file writing, without rollback or an atomic
|
||||
replacement guarantee. Generated content still goes to stdout or `--out`.
|
||||
Equal output/report paths, including existing paths that refer to the same
|
||||
file, are rejected before execution.
|
||||
|
||||
The file is written after engine execution when Promptkit supplies a terminal
|
||||
event, including for a failed run or invalid generated output. A failure before
|
||||
invocation leaves the report path untouched. If no terminal event is supplied,
|
||||
Scriptorium leaves the path untouched and emits a diagnostic; an existing file
|
||||
may therefore contain an older report. Artifact output is attempted first on a
|
||||
successful run, and report writing is attempted even if artifact writing fails.
|
||||
Any output or report write failure exits 1; already-written output is not
|
||||
undone. Without a write failure, the actual run keeps its usual exit status.
|
||||
|
||||
The JSON object contains the following fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `run_id`, `prompt_id`, `prompt_version`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
|
||||
| `outcome` | `success`, `invalid_output`, `error`, or `other` for an unrecognized upstream value. |
|
||||
| `failure` | The known terminal failure category, `none`, or `other` for an unrecognized value. |
|
||||
| `status_code` | Provider status, omitted when unavailable. |
|
||||
| `retries_exhausted` | Boolean independent of the failure category. |
|
||||
| `attempts`, `transient_retries`, `repair_calls` | Unsigned terminal aggregate counts. |
|
||||
| `duration_ms`, `capacity_wait_ms`, `generation_duration_ms`, `retry_delay_ms` | Terminal observed durations in integer milliseconds; their sum need not equal wall time. |
|
||||
| `usage` | Object containing `prompt_tokens`, `completion_tokens`, `total_tokens`, `cached_tokens`, and `cache_write_tokens`. |
|
||||
|
||||
Each `usage` count contains `value` (integer), `known` (boolean), and
|
||||
`complete` (boolean). Reported zero differs from an unknown count; an
|
||||
incomplete total does not establish complete billing. All counts, durations,
|
||||
`retries_exhausted`, and `usage` are present even when zero or unknown. The
|
||||
recognized failure values are `none`, `canceled`, `deadline`, `transport`,
|
||||
`provider_status`, `malformed_response`, `capacity`, `invalid_config`,
|
||||
`invalid_request`, `prompt_not_found`, `profile_not_found`, `prompt_load`,
|
||||
`profile_load`, `artifact_load`, `prompt_render`, `validation`, `generation`,
|
||||
and `other`.
|
||||
|
||||
The report excludes content, credentials, URLs, headers, and raw errors, but
|
||||
its identifiers can still be sensitive. The caller owns file retention and
|
||||
access.
|
||||
|
||||
## `scriptorium render`
|
||||
|
||||
```text
|
||||
|
||||
Reference in New Issue
Block a user