137 lines
3.8 KiB
Markdown
137 lines
3.8 KiB
Markdown
# Prompt Cache Control Roadmap
|
|
|
|
## Purpose
|
|
|
|
Scriptorium should support provider prompt-cache controls for OpenAI-compatible gateways that expose Anthropic-style cache breakpoints, especially OpenRouter.
|
|
|
|
The feature should preserve existing prompt definitions. Prompt authors should opt in with optional message-level cache metadata, and Scriptorium should report cache usage when compatible providers return it.
|
|
|
|
Implementation steps belong in `docs/roadmap/implementation.md`.
|
|
|
|
## Target State
|
|
|
|
Prompt authors can mark a rendered message as an explicit cache breakpoint:
|
|
|
|
```yaml
|
|
messages:
|
|
- role: system
|
|
content_file: ../common/transcript.system.md
|
|
cache_control:
|
|
type: ephemeral
|
|
ttl: 1h
|
|
```
|
|
|
|
Existing messages without `cache_control` continue to render and serialize as string content:
|
|
|
|
```json
|
|
{
|
|
"role": "system",
|
|
"content": "rendered text"
|
|
}
|
|
```
|
|
|
|
Messages with `cache_control` serialize as a single text content block:
|
|
|
|
```json
|
|
{
|
|
"role": "system",
|
|
"content": [
|
|
{
|
|
"type": "text",
|
|
"text": "rendered text",
|
|
"cache_control": {
|
|
"type": "ephemeral",
|
|
"ttl": "1h"
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
Scriptorium should parse and expose provider cache usage when compatible response fields are present, including cached prompt tokens and cache-write tokens.
|
|
|
|
## Prompt Authoring Policy
|
|
|
|
Cache breakpoints should be used for stable reusable prompt prefixes.
|
|
|
|
Recommended ordering:
|
|
|
|
1. Put stable, reusable context first.
|
|
2. Put `cache_control` on the last stable message that should be part of the reusable prefix.
|
|
3. Put per-run dynamic inputs after that breakpoint.
|
|
|
|
Example:
|
|
|
|
```yaml
|
|
messages:
|
|
- role: system
|
|
content_file: ../common/transcript.system.md
|
|
- role: user
|
|
content_file: ./character_meta_analysis.task.md
|
|
- role: user
|
|
content_file: ./character_meta_analysis.instructions.md
|
|
cache_control:
|
|
type: ephemeral
|
|
ttl: 1h
|
|
- role: user
|
|
content: |
|
|
<<<PREVIOUS_SESSION_RECAP
|
|
{{input "recap"}}
|
|
PREVIOUS_SESSION_RECAP>>>
|
|
- role: user
|
|
content: |
|
|
<<<CURRENT_SESSION_TRANSCRIPT
|
|
{{input "transcript"}}
|
|
CURRENT_SESSION_TRANSCRIPT>>>
|
|
```
|
|
|
|
## Supported Cache-Control Shape
|
|
|
|
Initial support is message-level only:
|
|
|
|
```yaml
|
|
cache_control:
|
|
type: ephemeral
|
|
ttl: 1h
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `cache_control` is optional on each message.
|
|
- `cache_control.type` is required when `cache_control` is present.
|
|
- The only supported `type` value is `ephemeral`.
|
|
- `ttl` is optional.
|
|
- When set, the only supported `ttl` value is `1h`.
|
|
- Empty `ttl` is omitted from the outbound payload.
|
|
- Prompt decoding remains strict; unknown cache-control fields are rejected.
|
|
- Existing `content` and `content_file` rules remain unchanged.
|
|
|
|
## Compatibility
|
|
|
|
This feature should be backward compatible for existing prompt definitions.
|
|
|
|
Compatibility requirements:
|
|
|
|
- Existing prompt YAML without `cache_control` loads unchanged.
|
|
- Existing `render` output remains valid.
|
|
- Existing outbound request payloads remain string-content messages unless `cache_control` is configured.
|
|
- Existing integrations do not need to send new request fields.
|
|
- CLI and HTTP callers do not need new request options for the initial feature.
|
|
|
|
The only intentional prompt-definition contract change is the new optional `messages[].cache_control` object.
|
|
|
|
## Deferred Work
|
|
|
|
These are intentionally out of scope for the initial feature:
|
|
|
|
- `content_blocks` prompt syntax.
|
|
- Multiple text blocks inside a single message.
|
|
- Image, tool, or non-text content blocks.
|
|
- Provider-specific automatic prompt caching toggles.
|
|
- Top-level OpenRouter `cache_control`.
|
|
- Top-level OpenRouter `session_id`.
|
|
- General-purpose serialization of `extra_params`.
|
|
- Provider-specific validation profiles for cache-control limits.
|
|
|
|
These can be added later without changing the message-level cache-control contract.
|