diff --git a/docs/cli.md b/docs/cli.md index 264cf2a..8c72e6d 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -32,6 +32,7 @@ Integration references: - an effective `prompt_dir` and `profile_dir` (from flags or config) - `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config). - Positional arguments are rejected. +- Prompt cache control is configured in prompt YAML (`messages[].cache_control`), not with CLI flags. ## Flag Reference @@ -93,6 +94,7 @@ Notes: - Writes generated artifact content to stdout by default. - Writes generated artifact content to `--out` when provided. - Prints run summary metadata to stderr on success. +- Appends `cached_tokens= cache_write_tokens=` to the summary only when the provider reports non-zero cache usage. - Prints errors to stderr on failure. `render`: diff --git a/docs/config.md b/docs/config.md index 64d3ba5..a40705f 100644 --- a/docs/config.md +++ b/docs/config.md @@ -119,6 +119,7 @@ Field reference: - `role` (required) - `content` or `content_file` (exactly one is required) +- `cache_control` (optional object): provider prompt-cache metadata for this message Message rules: @@ -128,6 +129,27 @@ Message rules: - Prompt decoding is strict; unknown YAML fields are rejected. - Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one. +`messages[].cache_control` fields: + +- `type` (required when `cache_control` is present): currently only `ephemeral`. +- `ttl` (optional): currently only `1h`; omitted from outbound requests when unset. + +Example cache-controlled message: + +```yaml +messages: + - role: system + content_file: ./stable_context.md + cache_control: + type: ephemeral + ttl: 1h + - role: user + content: | + {{input "transcript"}} +``` + +Use cache control on stable reusable prompt content. Dynamic per-run inputs before the cache-controlled message change the provider cache key. + `output` fields: - `format` (required): `text`, `markdown`, or `json`. @@ -184,6 +206,8 @@ Profile rules: Current outbound request behavior: - The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts. +- Messages without `cache_control` serialize with string `content`. +- Messages with `cache_control` serialize as a single text content-block array containing `cache_control`. - `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests. ## Schema Behavior diff --git a/docs/integrations/http-api.md b/docs/integrations/http-api.md index 83b97f3..e8d59fc 100644 --- a/docs/integrations/http-api.md +++ b/docs/integrations/http-api.md @@ -128,7 +128,9 @@ Response shape: "usage": { "prompt_tokens": 11, "completion_tokens": 22, - "total_tokens": 33 + "total_tokens": 33, + "cached_tokens": 0, + "cache_write_tokens": 0 }, "start_time": "2026-05-04T12:00:00Z", "end_time": "2026-05-04T12:00:01Z", @@ -142,6 +144,8 @@ Response shape: `raw_model_output` is omitted by default. +`metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are always present as numbers. They are `0` when the provider omits compatible cache usage fields or reports no cache activity. + To include it, send: - `"include_raw_output": true` diff --git a/docs/integrations/openai-compatible-chat.md b/docs/integrations/openai-compatible-chat.md index 3434ad3..5e3cb0a 100644 --- a/docs/integrations/openai-compatible-chat.md +++ b/docs/integrations/openai-compatible-chat.md @@ -26,7 +26,7 @@ Example: Serialized JSON fields: - `model` (required after fallback resolution) -- `messages` (role/content pairs from rendered prompt) +- `messages` (rendered prompt messages) - `temperature` (only when non-zero) - `max_tokens` (only when non-zero) - `top_p` (only when non-zero) @@ -35,6 +35,35 @@ Serialized JSON fields: `service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support. +Messages without prompt cache control serialize with string `content`: + +```json +{ + "role": "system", + "content": "rendered text" +} +``` + +Messages with prompt cache control serialize as a single text content-block array: + +```json +{ + "role": "system", + "content": [ + { + "type": "text", + "text": "rendered text", + "cache_control": { + "type": "ephemeral", + "ttl": "1h" + } + } + ] +} +``` + +When cache-control `ttl` is unset in the prompt definition, `ttl` is omitted from the outbound payload. + Structured output is currently `json_schema` only, serialized as: ```json @@ -82,6 +111,13 @@ Expected successful response shape (subset used): - `usage.prompt_tokens` - `usage.completion_tokens` - `usage.total_tokens` +- `usage.prompt_tokens_details.cached_tokens` (optional) +- `usage.cache_write_tokens` (optional) + +Absent cache usage fields are treated as zero. Parsed cache usage is exposed through run results and adapter response surfaces as: + +- `cached_tokens` +- `cache_write_tokens` Malformed response conditions include: @@ -104,6 +140,8 @@ The following fields may exist in profile/effective settings but are not current - `reasoning_effort` - `extra_params` +The client also does not serialize top-level `cache_control` or `session_id`. + No built-in retries, tool-calls, or multi-request payload modes are implemented in this client. ## Relationship To Runner diff --git a/docs/internal/adapters.md b/docs/internal/adapters.md index 60bf383..a993be7 100644 --- a/docs/internal/adapters.md +++ b/docs/internal/adapters.md @@ -22,11 +22,13 @@ CLI adapter: - Input: process args, filesystem config/assets, environment. - Output: exit code, stdout artifact/prepared output, stderr summaries/errors. +- `run` summaries include cache usage counters only when either parsed cache counter is non-zero. HTTP adapter: - Input: JSON request body (`runRequestDTO`). - Output: JSON success/error body with mapped status codes. +- Success metadata includes token usage plus cache usage counters. Filesystem repositories: @@ -93,6 +95,9 @@ Artifact refs: LLM adapter: - endpoint appends `/chat/completions`. +- rendered messages without cache control serialize with string `content`. +- rendered messages with cache control serialize as one text content block with `cache_control`. +- compatible cache usage response fields are parsed into domain token usage. - non-2xx responses map to request failure errors. - malformed responses (including missing/empty first choice content) are errors. @@ -142,3 +147,4 @@ Behavior highlights: - External request/response strictness is part of contract stability. - Prepared-render output never includes resolved API key values. - Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`). +- Outbound cache control is message-level only; no top-level cache-control/session fields are serialized. diff --git a/docs/internal/runner.md b/docs/internal/runner.md index 929ae65..5aeb0d7 100644 --- a/docs/internal/runner.md +++ b/docs/internal/runner.md @@ -108,9 +108,11 @@ Validation content failures are not run errors: - only the environment-variable name is retained; secret value is never returned 7. resolve output contract and structured-output schema payload when `json_schema` mode is active. 8. read input artifacts. -9. render prompt messages. +9. render prompt messages, including any normalized message cache-control metadata. 10. compute prompt/input/render hashes and return `PreparedRun`. +`rendered_prompt_hash` includes cache-control metadata when present because it affects the outbound provider request. Prompts without cache control keep the role/content hash behavior. + `Prepare` does not call the LLM. ## Run Flow @@ -123,7 +125,7 @@ Validation content failures are not run errors: 4. build output artifact content type from output format. 5. validate output. 6. optionally attempt bounded repair when repairer is injected and contract allows it. -7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, usage, and timestamps. +7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, token/cache usage, and timestamps. ## Repair Hook Boundary diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 4e3e898..c88c2bd 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -252,6 +252,39 @@ Relevant links: - [Configuration reference](config.md) - [Operations guide](operations.md) +## Prompt Cache Misses Or No Cache Usage + +Symptom: + +- CLI run summary omits `cached_tokens` / `cache_write_tokens`. +- HTTP `metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are both `0`. +- Provider cost or latency does not improve after repeated similar runs. + +Likely cause: + +- The selected prompt has no `messages[].cache_control`. +- Dynamic per-run input appears before the cache-controlled message and changes the provider cache key. +- The provider does not support the serialized cache-control shape for the selected model. +- The provider imposes minimum token thresholds or cache-breakpoint limits. + +Diagnostic step: + +- Run `render --format json` and verify the intended rendered message includes `cache_control`. +- Confirm stable reusable context appears before the cache-controlled message, with dynamic input after it. +- Check provider docs/logs for model support, minimum token thresholds, and breakpoint limits. + +Safe fix: + +- Move stable reusable context before the cache-controlled message. +- Move highly dynamic input after the cache breakpoint. +- Keep `cache_control.type: ephemeral` and, when using `ttl`, set `ttl: 1h`. +- Use CLI cache counters or HTTP cache usage fields to verify cache reads/writes after rerunning. + +Relevant links: + +- [Configuration reference](config.md) +- [OpenAI-compatible chat integration](integrations/openai-compatible-chat.md) + ## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status) Symptom: