Add opt-in CLI execution report files
This commit is contained in:
46
docs/cli.md
46
docs/cli.md
@@ -56,6 +56,7 @@ Optional flags:
|
||||
| `--input name=path` | Optional input file mapping; repeat or use comma-separated mappings. |
|
||||
| `--var name=value` | Template-variable mapping; repeat or use comma-separated mappings. |
|
||||
| `--out <path>` | Write generated content to this file instead of stdout. |
|
||||
| `--execution-report <path>` | Write an opt-in terminal execution report as newline-terminated JSON to a separate file. |
|
||||
| `--llm-base-url <url>` | Runtime endpoint override. |
|
||||
| `--model <name>` | Runtime model override. |
|
||||
| `--api-key-env <name>` | Runtime API-key environment-variable name override. |
|
||||
@@ -110,6 +111,51 @@ profile value, a nonblank value replaces that value, and
|
||||
`--reasoning-effort=` explicitly clears inherited reasoning. Promptkit treats
|
||||
nonblank values as provider-specific opaque strings.
|
||||
|
||||
### Execution report file
|
||||
|
||||
`--execution-report` requires a nonempty path. It is available only for `run`;
|
||||
`-` is an ordinary filename. Parent directories are not created. An existing
|
||||
file is replaced through ordinary file writing, without rollback or an atomic
|
||||
replacement guarantee. Generated content still goes to stdout or `--out`.
|
||||
Equal output/report paths, including existing paths that refer to the same
|
||||
file, are rejected before execution.
|
||||
|
||||
The file is written after engine execution when Promptkit supplies a terminal
|
||||
event, including for a failed run or invalid generated output. A failure before
|
||||
invocation leaves the report path untouched. If no terminal event is supplied,
|
||||
Scriptorium leaves the path untouched and emits a diagnostic; an existing file
|
||||
may therefore contain an older report. Artifact output is attempted first on a
|
||||
successful run, and report writing is attempted even if artifact writing fails.
|
||||
Any output or report write failure exits 1; already-written output is not
|
||||
undone. Without a write failure, the actual run keeps its usual exit status.
|
||||
|
||||
The JSON object contains the following fields:
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `run_id`, `prompt_id`, `prompt_version`, `profile_id`, `backend_id`, `model` | Terminal identifiers; absent values are omitted. |
|
||||
| `outcome` | `success`, `invalid_output`, `error`, or `other` for an unrecognized upstream value. |
|
||||
| `failure` | The known terminal failure category, `none`, or `other` for an unrecognized value. |
|
||||
| `status_code` | Provider status, omitted when unavailable. |
|
||||
| `retries_exhausted` | Boolean independent of the failure category. |
|
||||
| `attempts`, `transient_retries`, `repair_calls` | Unsigned terminal aggregate counts. |
|
||||
| `duration_ms`, `capacity_wait_ms`, `generation_duration_ms`, `retry_delay_ms` | Terminal observed durations in integer milliseconds; their sum need not equal wall time. |
|
||||
| `usage` | Object containing `prompt_tokens`, `completion_tokens`, `total_tokens`, `cached_tokens`, and `cache_write_tokens`. |
|
||||
|
||||
Each `usage` count contains `value` (integer), `known` (boolean), and
|
||||
`complete` (boolean). Reported zero differs from an unknown count; an
|
||||
incomplete total does not establish complete billing. All counts, durations,
|
||||
`retries_exhausted`, and `usage` are present even when zero or unknown. The
|
||||
recognized failure values are `none`, `canceled`, `deadline`, `transport`,
|
||||
`provider_status`, `malformed_response`, `capacity`, `invalid_config`,
|
||||
`invalid_request`, `prompt_not_found`, `profile_not_found`, `prompt_load`,
|
||||
`profile_load`, `artifact_load`, `prompt_render`, `validation`, `generation`,
|
||||
and `other`.
|
||||
|
||||
The report excludes content, credentials, URLs, headers, and raw errors, but
|
||||
its identifiers can still be sensitive. The caller owns file retention and
|
||||
access.
|
||||
|
||||
## `scriptorium render`
|
||||
|
||||
```text
|
||||
|
||||
@@ -29,6 +29,13 @@ When using `--out`, the caller owns the output path, its permissions, and
|
||||
cleanup. Treat rendered prompts, generated artifacts, stdout, and stderr as
|
||||
potentially sensitive.
|
||||
|
||||
For `run`, an optional [execution report file](../cli.md) contains terminal
|
||||
metadata separate from generated stdout or `--out`. The caller owns that
|
||||
path's permissions and retention. Output and report files are not a
|
||||
transaction: one may be written when the other fails, and a preexisting report
|
||||
may remain when the invocation has no terminal event. Check the exit status
|
||||
and diagnostics rather than assuming a report path contains this run's data.
|
||||
|
||||
## Cancellation And Recovery
|
||||
|
||||
A CLI invocation performs one synchronous request and creates no durable run
|
||||
|
||||
@@ -77,6 +77,10 @@ The report collector is per invocation. It ignores nonterminal events and
|
||||
copies terminal aggregates once, including usage presence and completeness.
|
||||
It retains no history or shared state; adapters decide whether to attach it and
|
||||
how to present its snapshot.
|
||||
CLI `run` attaches one only when a report path is selected, writes generated
|
||||
output first on success, and then writes the terminal snapshot when present.
|
||||
Its report path remains untouched for pre-invocation failures or absent
|
||||
terminal observations; report and artifact writes are independent.
|
||||
|
||||
## Package-Local Guarantees
|
||||
|
||||
|
||||
@@ -26,6 +26,11 @@ configuration, or environment and submitting a new request.
|
||||
Generated artifacts, rendered prompts, model output, and run metadata are
|
||||
caller-owned data. Retention, encryption, backup, and deletion are deployment
|
||||
responsibilities.
|
||||
Optional [CLI execution reports](cli.md) also belong to the caller. They can
|
||||
contain sensitive identifiers even though they omit prompt and response bodies.
|
||||
Keep report paths separate from generated output and apply the same access and
|
||||
retention controls. Output and report writes are not transactional; an older
|
||||
report may remain when no terminal observation was produced.
|
||||
|
||||
## Deploy The Filesystem And Process
|
||||
|
||||
|
||||
@@ -302,7 +302,7 @@ no new orchestration layer or durable state.
|
||||
|
||||
## Stage 7: Expose CLI Execution Report Files
|
||||
|
||||
**Status: Pending.**
|
||||
**Status: Complete.**
|
||||
|
||||
Selected interface: run-only `--execution-report <path>`, writing one
|
||||
newline-terminated JSON report to a separate file. A nonempty path opts in;
|
||||
|
||||
Reference in New Issue
Block a user