Document prompt cache control behavior

This commit is contained in:
2026-07-02 23:10:24 +00:00
parent 5dcb3cd4fc
commit 4d4bb7a121
7 changed files with 113 additions and 4 deletions

View File

@@ -32,6 +32,7 @@ Integration references:
- an effective `prompt_dir` and `profile_dir` (from flags or config) - an effective `prompt_dir` and `profile_dir` (from flags or config)
- `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config). - `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config).
- Positional arguments are rejected. - Positional arguments are rejected.
- Prompt cache control is configured in prompt YAML (`messages[].cache_control`), not with CLI flags.
## Flag Reference ## Flag Reference
@@ -93,6 +94,7 @@ Notes:
- Writes generated artifact content to stdout by default. - Writes generated artifact content to stdout by default.
- Writes generated artifact content to `--out` when provided. - Writes generated artifact content to `--out` when provided.
- Prints run summary metadata to stderr on success. - Prints run summary metadata to stderr on success.
- Appends `cached_tokens=<n> cache_write_tokens=<n>` to the summary only when the provider reports non-zero cache usage.
- Prints errors to stderr on failure. - Prints errors to stderr on failure.
`render`: `render`:

View File

@@ -119,6 +119,7 @@ Field reference:
- `role` (required) - `role` (required)
- `content` or `content_file` (exactly one is required) - `content` or `content_file` (exactly one is required)
- `cache_control` (optional object): provider prompt-cache metadata for this message
Message rules: Message rules:
@@ -128,6 +129,27 @@ Message rules:
- Prompt decoding is strict; unknown YAML fields are rejected. - Prompt decoding is strict; unknown YAML fields are rejected.
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one. - Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
`messages[].cache_control` fields:
- `type` (required when `cache_control` is present): currently only `ephemeral`.
- `ttl` (optional): currently only `1h`; omitted from outbound requests when unset.
Example cache-controlled message:
```yaml
messages:
- role: system
content_file: ./stable_context.md
cache_control:
type: ephemeral
ttl: 1h
- role: user
content: |
{{input "transcript"}}
```
Use cache control on stable reusable prompt content. Dynamic per-run inputs before the cache-controlled message change the provider cache key.
`output` fields: `output` fields:
- `format` (required): `text`, `markdown`, or `json`. - `format` (required): `text`, `markdown`, or `json`.
@@ -184,6 +206,8 @@ Profile rules:
Current outbound request behavior: Current outbound request behavior:
- The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts. - The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts.
- Messages without `cache_control` serialize with string `content`.
- Messages with `cache_control` serialize as a single text content-block array containing `cache_control`.
- `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests. - `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests.
## Schema Behavior ## Schema Behavior

View File

@@ -128,7 +128,9 @@ Response shape:
"usage": { "usage": {
"prompt_tokens": 11, "prompt_tokens": 11,
"completion_tokens": 22, "completion_tokens": 22,
"total_tokens": 33 "total_tokens": 33,
"cached_tokens": 0,
"cache_write_tokens": 0
}, },
"start_time": "2026-05-04T12:00:00Z", "start_time": "2026-05-04T12:00:00Z",
"end_time": "2026-05-04T12:00:01Z", "end_time": "2026-05-04T12:00:01Z",
@@ -142,6 +144,8 @@ Response shape:
`raw_model_output` is omitted by default. `raw_model_output` is omitted by default.
`metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are always present as numbers. They are `0` when the provider omits compatible cache usage fields or reports no cache activity.
To include it, send: To include it, send:
- `"include_raw_output": true` - `"include_raw_output": true`

View File

@@ -26,7 +26,7 @@ Example:
Serialized JSON fields: Serialized JSON fields:
- `model` (required after fallback resolution) - `model` (required after fallback resolution)
- `messages` (role/content pairs from rendered prompt) - `messages` (rendered prompt messages)
- `temperature` (only when non-zero) - `temperature` (only when non-zero)
- `max_tokens` (only when non-zero) - `max_tokens` (only when non-zero)
- `top_p` (only when non-zero) - `top_p` (only when non-zero)
@@ -35,6 +35,35 @@ Serialized JSON fields:
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support. `service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
Messages without prompt cache control serialize with string `content`:
```json
{
"role": "system",
"content": "rendered text"
}
```
Messages with prompt cache control serialize as a single text content-block array:
```json
{
"role": "system",
"content": [
{
"type": "text",
"text": "rendered text",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
}
]
}
```
When cache-control `ttl` is unset in the prompt definition, `ttl` is omitted from the outbound payload.
Structured output is currently `json_schema` only, serialized as: Structured output is currently `json_schema` only, serialized as:
```json ```json
@@ -82,6 +111,13 @@ Expected successful response shape (subset used):
- `usage.prompt_tokens` - `usage.prompt_tokens`
- `usage.completion_tokens` - `usage.completion_tokens`
- `usage.total_tokens` - `usage.total_tokens`
- `usage.prompt_tokens_details.cached_tokens` (optional)
- `usage.cache_write_tokens` (optional)
Absent cache usage fields are treated as zero. Parsed cache usage is exposed through run results and adapter response surfaces as:
- `cached_tokens`
- `cache_write_tokens`
Malformed response conditions include: Malformed response conditions include:
@@ -104,6 +140,8 @@ The following fields may exist in profile/effective settings but are not current
- `reasoning_effort` - `reasoning_effort`
- `extra_params` - `extra_params`
The client also does not serialize top-level `cache_control` or `session_id`.
No built-in retries, tool-calls, or multi-request payload modes are implemented in this client. No built-in retries, tool-calls, or multi-request payload modes are implemented in this client.
## Relationship To Runner ## Relationship To Runner

View File

@@ -22,11 +22,13 @@ CLI adapter:
- Input: process args, filesystem config/assets, environment. - Input: process args, filesystem config/assets, environment.
- Output: exit code, stdout artifact/prepared output, stderr summaries/errors. - Output: exit code, stdout artifact/prepared output, stderr summaries/errors.
- `run` summaries include cache usage counters only when either parsed cache counter is non-zero.
HTTP adapter: HTTP adapter:
- Input: JSON request body (`runRequestDTO`). - Input: JSON request body (`runRequestDTO`).
- Output: JSON success/error body with mapped status codes. - Output: JSON success/error body with mapped status codes.
- Success metadata includes token usage plus cache usage counters.
Filesystem repositories: Filesystem repositories:
@@ -93,6 +95,9 @@ Artifact refs:
LLM adapter: LLM adapter:
- endpoint appends `/chat/completions`. - endpoint appends `/chat/completions`.
- rendered messages without cache control serialize with string `content`.
- rendered messages with cache control serialize as one text content block with `cache_control`.
- compatible cache usage response fields are parsed into domain token usage.
- non-2xx responses map to request failure errors. - non-2xx responses map to request failure errors.
- malformed responses (including missing/empty first choice content) are errors. - malformed responses (including missing/empty first choice content) are errors.
@@ -142,3 +147,4 @@ Behavior highlights:
- External request/response strictness is part of contract stability. - External request/response strictness is part of contract stability.
- Prepared-render output never includes resolved API key values. - Prepared-render output never includes resolved API key values.
- Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`). - Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`).
- Outbound cache control is message-level only; no top-level cache-control/session fields are serialized.

View File

@@ -108,9 +108,11 @@ Validation content failures are not run errors:
- only the environment-variable name is retained; secret value is never returned - only the environment-variable name is retained; secret value is never returned
7. resolve output contract and structured-output schema payload when `json_schema` mode is active. 7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
8. read input artifacts. 8. read input artifacts.
9. render prompt messages. 9. render prompt messages, including any normalized message cache-control metadata.
10. compute prompt/input/render hashes and return `PreparedRun`. 10. compute prompt/input/render hashes and return `PreparedRun`.
`rendered_prompt_hash` includes cache-control metadata when present because it affects the outbound provider request. Prompts without cache control keep the role/content hash behavior.
`Prepare` does not call the LLM. `Prepare` does not call the LLM.
## Run Flow ## Run Flow
@@ -123,7 +125,7 @@ Validation content failures are not run errors:
4. build output artifact content type from output format. 4. build output artifact content type from output format.
5. validate output. 5. validate output.
6. optionally attempt bounded repair when repairer is injected and contract allows it. 6. optionally attempt bounded repair when repairer is injected and contract allows it.
7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, usage, and timestamps. 7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, token/cache usage, and timestamps.
## Repair Hook Boundary ## Repair Hook Boundary

View File

@@ -252,6 +252,39 @@ Relevant links:
- [Configuration reference](config.md) - [Configuration reference](config.md)
- [Operations guide](operations.md) - [Operations guide](operations.md)
## Prompt Cache Misses Or No Cache Usage
Symptom:
- CLI run summary omits `cached_tokens` / `cache_write_tokens`.
- HTTP `metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are both `0`.
- Provider cost or latency does not improve after repeated similar runs.
Likely cause:
- The selected prompt has no `messages[].cache_control`.
- Dynamic per-run input appears before the cache-controlled message and changes the provider cache key.
- The provider does not support the serialized cache-control shape for the selected model.
- The provider imposes minimum token thresholds or cache-breakpoint limits.
Diagnostic step:
- Run `render --format json` and verify the intended rendered message includes `cache_control`.
- Confirm stable reusable context appears before the cache-controlled message, with dynamic input after it.
- Check provider docs/logs for model support, minimum token thresholds, and breakpoint limits.
Safe fix:
- Move stable reusable context before the cache-controlled message.
- Move highly dynamic input after the cache breakpoint.
- Keep `cache_control.type: ephemeral` and, when using `ttl`, set `ttl: 1h`.
- Use CLI cache counters or HTTP cache usage fields to verify cache reads/writes after rerunning.
Relevant links:
- [Configuration reference](config.md)
- [OpenAI-compatible chat integration](integrations/openai-compatible-chat.md)
## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status) ## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status)
Symptom: Symptom: