6 Commits

32 changed files with 1376 additions and 40 deletions

View File

@@ -32,6 +32,7 @@ Integration references:
- an effective `prompt_dir` and `profile_dir` (from flags or config) - an effective `prompt_dir` and `profile_dir` (from flags or config)
- `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config). - `serve` requires an effective `prompt_dir` and `profile_dir` (from flags or config).
- Positional arguments are rejected. - Positional arguments are rejected.
- Prompt cache control is configured in prompt YAML (`messages[].cache_control`), not with CLI flags.
## Flag Reference ## Flag Reference
@@ -93,6 +94,7 @@ Notes:
- Writes generated artifact content to stdout by default. - Writes generated artifact content to stdout by default.
- Writes generated artifact content to `--out` when provided. - Writes generated artifact content to `--out` when provided.
- Prints run summary metadata to stderr on success. - Prints run summary metadata to stderr on success.
- Appends `cached_tokens=<n> cache_write_tokens=<n>` to the summary only when the provider reports non-zero cache usage.
- Prints errors to stderr on failure. - Prints errors to stderr on failure.
`render`: `render`:

View File

@@ -119,6 +119,7 @@ Field reference:
- `role` (required) - `role` (required)
- `content` or `content_file` (exactly one is required) - `content` or `content_file` (exactly one is required)
- `cache_control` (optional object): provider prompt-cache metadata for this message
Message rules: Message rules:
@@ -128,6 +129,27 @@ Message rules:
- Prompt decoding is strict; unknown YAML fields are rejected. - Prompt decoding is strict; unknown YAML fields are rejected.
- Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one. - Duplicate prompt IDs are invalid. If multiple files declare the requested prompt ID, Scriptorium fails instead of choosing one.
`messages[].cache_control` fields:
- `type` (required when `cache_control` is present): currently only `ephemeral`.
- `ttl` (optional): currently only `1h`; omitted from outbound requests when unset.
Example cache-controlled message:
```yaml
messages:
- role: system
content_file: ./stable_context.md
cache_control:
type: ephemeral
ttl: 1h
- role: user
content: |
{{input "transcript"}}
```
Use cache control on stable reusable prompt content. Dynamic per-run inputs before the cache-controlled message change the provider cache key.
`output` fields: `output` fields:
- `format` (required): `text`, `markdown`, or `json`. - `format` (required): `text`, `markdown`, or `json`.
@@ -184,6 +206,8 @@ Profile rules:
Current outbound request behavior: Current outbound request behavior:
- The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts. - The OpenAI-compatible client currently serializes: `model`, `messages`, `temperature`, `max_tokens`, `top_p`, `service_tier`, and optional `response_format` for `json_schema` prompts.
- Messages without `cache_control` serialize with string `content`.
- Messages with `cache_control` serialize as a single text content-block array containing `cache_control`.
- `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests. - `reasoning_effort` and `extra_params` are parsed and carried in effective settings, but are not currently serialized into outbound chat-completions requests.
## Schema Behavior ## Schema Behavior

View File

@@ -128,7 +128,9 @@ Response shape:
"usage": { "usage": {
"prompt_tokens": 11, "prompt_tokens": 11,
"completion_tokens": 22, "completion_tokens": 22,
"total_tokens": 33 "total_tokens": 33,
"cached_tokens": 0,
"cache_write_tokens": 0
}, },
"start_time": "2026-05-04T12:00:00Z", "start_time": "2026-05-04T12:00:00Z",
"end_time": "2026-05-04T12:00:01Z", "end_time": "2026-05-04T12:00:01Z",
@@ -142,6 +144,8 @@ Response shape:
`raw_model_output` is omitted by default. `raw_model_output` is omitted by default.
`metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are always present as numbers. They are `0` when the provider omits compatible cache usage fields or reports no cache activity.
To include it, send: To include it, send:
- `"include_raw_output": true` - `"include_raw_output": true`

View File

@@ -26,7 +26,7 @@ Example:
Serialized JSON fields: Serialized JSON fields:
- `model` (required after fallback resolution) - `model` (required after fallback resolution)
- `messages` (role/content pairs from rendered prompt) - `messages` (rendered prompt messages)
- `temperature` (only when non-zero) - `temperature` (only when non-zero)
- `max_tokens` (only when non-zero) - `max_tokens` (only when non-zero)
- `top_p` (only when non-zero) - `top_p` (only when non-zero)
@@ -35,6 +35,35 @@ Serialized JSON fields:
`service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support. `service_tier` is provider-specific. OpenRouter currently documents request values such as `flex` and `priority`; Scriptorium forwards any non-empty configured value and lets the backend validate support.
Messages without prompt cache control serialize with string `content`:
```json
{
"role": "system",
"content": "rendered text"
}
```
Messages with prompt cache control serialize as a single text content-block array:
```json
{
"role": "system",
"content": [
{
"type": "text",
"text": "rendered text",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
}
]
}
```
When cache-control `ttl` is unset in the prompt definition, `ttl` is omitted from the outbound payload.
Structured output is currently `json_schema` only, serialized as: Structured output is currently `json_schema` only, serialized as:
```json ```json
@@ -82,6 +111,13 @@ Expected successful response shape (subset used):
- `usage.prompt_tokens` - `usage.prompt_tokens`
- `usage.completion_tokens` - `usage.completion_tokens`
- `usage.total_tokens` - `usage.total_tokens`
- `usage.prompt_tokens_details.cached_tokens` (optional)
- `usage.cache_write_tokens` (optional)
Absent cache usage fields are treated as zero. Parsed cache usage is exposed through run results and adapter response surfaces as:
- `cached_tokens`
- `cache_write_tokens`
Malformed response conditions include: Malformed response conditions include:
@@ -104,6 +140,8 @@ The following fields may exist in profile/effective settings but are not current
- `reasoning_effort` - `reasoning_effort`
- `extra_params` - `extra_params`
The client also does not serialize top-level `cache_control` or `session_id`.
No built-in retries, tool-calls, or multi-request payload modes are implemented in this client. No built-in retries, tool-calls, or multi-request payload modes are implemented in this client.
## Relationship To Runner ## Relationship To Runner

View File

@@ -22,11 +22,13 @@ CLI adapter:
- Input: process args, filesystem config/assets, environment. - Input: process args, filesystem config/assets, environment.
- Output: exit code, stdout artifact/prepared output, stderr summaries/errors. - Output: exit code, stdout artifact/prepared output, stderr summaries/errors.
- `run` summaries include cache usage counters only when either parsed cache counter is non-zero.
HTTP adapter: HTTP adapter:
- Input: JSON request body (`runRequestDTO`). - Input: JSON request body (`runRequestDTO`).
- Output: JSON success/error body with mapped status codes. - Output: JSON success/error body with mapped status codes.
- Success metadata includes token usage plus cache usage counters.
Filesystem repositories: Filesystem repositories:
@@ -93,6 +95,9 @@ Artifact refs:
LLM adapter: LLM adapter:
- endpoint appends `/chat/completions`. - endpoint appends `/chat/completions`.
- rendered messages without cache control serialize with string `content`.
- rendered messages with cache control serialize as one text content block with `cache_control`.
- compatible cache usage response fields are parsed into domain token usage.
- non-2xx responses map to request failure errors. - non-2xx responses map to request failure errors.
- malformed responses (including missing/empty first choice content) are errors. - malformed responses (including missing/empty first choice content) are errors.
@@ -142,3 +147,4 @@ Behavior highlights:
- External request/response strictness is part of contract stability. - External request/response strictness is part of contract stability.
- Prepared-render output never includes resolved API key values. - Prepared-render output never includes resolved API key values.
- Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`). - Outbound OpenAI-compatible request includes only currently serialized fields (`model`, `messages`, optional `temperature`, `max_tokens`, `top_p`, optional `service_tier`, optional `response_format`).
- Outbound cache control is message-level only; no top-level cache-control/session fields are serialized.

View File

@@ -108,9 +108,11 @@ Validation content failures are not run errors:
- only the environment-variable name is retained; secret value is never returned - only the environment-variable name is retained; secret value is never returned
7. resolve output contract and structured-output schema payload when `json_schema` mode is active. 7. resolve output contract and structured-output schema payload when `json_schema` mode is active.
8. read input artifacts. 8. read input artifacts.
9. render prompt messages. 9. render prompt messages, including any normalized message cache-control metadata.
10. compute prompt/input/render hashes and return `PreparedRun`. 10. compute prompt/input/render hashes and return `PreparedRun`.
`rendered_prompt_hash` includes cache-control metadata when present because it affects the outbound provider request. Prompts without cache control keep the role/content hash behavior.
`Prepare` does not call the LLM. `Prepare` does not call the LLM.
## Run Flow ## Run Flow
@@ -123,7 +125,7 @@ Validation content failures are not run errors:
4. build output artifact content type from output format. 4. build output artifact content type from output format.
5. validate output. 5. validate output.
6. optionally attempt bounded repair when repairer is injected and contract allows it. 6. optionally attempt bounded repair when repairer is injected and contract allows it.
7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, usage, and timestamps. 7. return `RunResult` with artifact, raw output, validation, hashes, profile/model metadata, token/cache usage, and timestamps.
## Repair Hook Boundary ## Repair Hook Boundary

136
docs/roadmap/cache.md Normal file
View File

@@ -0,0 +1,136 @@
# Prompt Cache Control Roadmap
## Purpose
Scriptorium should support provider prompt-cache controls for OpenAI-compatible gateways that expose Anthropic-style cache breakpoints, especially OpenRouter.
The feature should preserve existing prompt definitions. Prompt authors should opt in with optional message-level cache metadata, and Scriptorium should report cache usage when compatible providers return it.
Implementation steps belong in `docs/roadmap/implementation.md`.
## Target State
Prompt authors can mark a rendered message as an explicit cache breakpoint:
```yaml
messages:
- role: system
content_file: ../common/transcript.system.md
cache_control:
type: ephemeral
ttl: 1h
```
Existing messages without `cache_control` continue to render and serialize as string content:
```json
{
"role": "system",
"content": "rendered text"
}
```
Messages with `cache_control` serialize as a single text content block:
```json
{
"role": "system",
"content": [
{
"type": "text",
"text": "rendered text",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
}
]
}
```
Scriptorium should parse and expose provider cache usage when compatible response fields are present, including cached prompt tokens and cache-write tokens.
## Prompt Authoring Policy
Cache breakpoints should be used for stable reusable prompt prefixes.
Recommended ordering:
1. Put stable, reusable context first.
2. Put `cache_control` on the last stable message that should be part of the reusable prefix.
3. Put per-run dynamic inputs after that breakpoint.
Example:
```yaml
messages:
- role: system
content_file: ../common/transcript.system.md
- role: user
content_file: ./character_meta_analysis.task.md
- role: user
content_file: ./character_meta_analysis.instructions.md
cache_control:
type: ephemeral
ttl: 1h
- role: user
content: |
<<<PREVIOUS_SESSION_RECAP
{{input "recap"}}
PREVIOUS_SESSION_RECAP>>>
- role: user
content: |
<<<CURRENT_SESSION_TRANSCRIPT
{{input "transcript"}}
CURRENT_SESSION_TRANSCRIPT>>>
```
## Supported Cache-Control Shape
Initial support is message-level only:
```yaml
cache_control:
type: ephemeral
ttl: 1h
```
Rules:
- `cache_control` is optional on each message.
- `cache_control.type` is required when `cache_control` is present.
- The only supported `type` value is `ephemeral`.
- `ttl` is optional.
- When set, the only supported `ttl` value is `1h`.
- Empty `ttl` is omitted from the outbound payload.
- Prompt decoding remains strict; unknown cache-control fields are rejected.
- Existing `content` and `content_file` rules remain unchanged.
## Compatibility
This feature should be backward compatible for existing prompt definitions.
Compatibility requirements:
- Existing prompt YAML without `cache_control` loads unchanged.
- Existing `render` output remains valid.
- Existing outbound request payloads remain string-content messages unless `cache_control` is configured.
- Existing integrations do not need to send new request fields.
- CLI and HTTP callers do not need new request options for the initial feature.
The only intentional prompt-definition contract change is the new optional `messages[].cache_control` object.
## Deferred Work
These are intentionally out of scope for the initial feature:
- `content_blocks` prompt syntax.
- Multiple text blocks inside a single message.
- Image, tool, or non-text content blocks.
- Provider-specific automatic prompt caching toggles.
- Top-level OpenRouter `cache_control`.
- Top-level OpenRouter `session_id`.
- General-purpose serialization of `extra_params`.
- Provider-specific validation profiles for cache-control limits.
These can be added later without changing the message-level cache-control contract.

View File

@@ -0,0 +1,390 @@
# Prompt Cache Control Implementation Plan
## Scope
This plan implements the feature defined in `docs/roadmap/cache.md`.
The implementation target is message-level Anthropic/OpenRouter-style cache control for rendered prompt messages, plus cache usage observability. Do not implement deferred roadmap items such as `content_blocks`, top-level `cache_control`, `session_id`, or general `extra_params` serialization.
Follow the policy documents under `docs/policy/` while implementing:
- Keep prompt/profile/HTTP/config decoding strict.
- Keep adapters thin.
- Keep provider request serialization in `internal/llm`.
- Keep prompt loading in `internal/promptdef`.
- Update canonical non-roadmap docs only when the behavior is implemented.
- Do not accept or emit raw secret values.
## Stage 1: Domain Types And Prompt Loader
Objective: make prompt YAML accept and validate optional `messages[].cache_control`, and carry normalized cache-control metadata in domain prompt definitions.
Files to update:
- `internal/domain/domain.go`
- `internal/promptdef/filesystem_repository.go`
- `internal/promptdef/repository_test.go`
- `internal/promptdef/testdata/`
Domain decisions:
- Add a typed cache-control value in `internal/domain`.
- Use a pointer on message structs so absence is distinguishable from an empty object.
- Use `omitempty` on JSON tags for the new field so existing no-cache prompt and render hashes do not gain `null` fields.
Use this domain shape:
```go
type CacheControlType string
const (
CacheControlEphemeral CacheControlType = "ephemeral"
)
type CacheControl struct {
Type CacheControlType `yaml:"type" json:"type"`
TTL string `yaml:"ttl,omitempty" json:"ttl,omitempty"`
}
```
Add cache control to:
```go
type PromptMessageTemplate struct {
Role string `yaml:"role"`
Content string `yaml:"content"`
ContentFile string `yaml:"content_file"`
CacheControl *CacheControl `yaml:"cache_control,omitempty" json:"cache_control,omitempty"`
}
```
Prompt loader changes:
- Add this field to `promptMessageFile`:
```go
CacheControl *cacheControlFile `yaml:"cache_control"`
```
- Add a private `cacheControlFile` struct with `type` and `ttl`.
- In `normalizePromptDefinition`, validate `cache_control` per message.
- Trim whitespace from `type` and `ttl`.
- Reject present `cache_control` when `type` is empty.
- Reject any `type` other than `ephemeral`.
- Accept omitted or empty `ttl`.
- Reject any non-empty `ttl` other than `1h`.
- Copy normalized values into `domain.PromptMessageTemplate.CacheControl`.
- Preserve the existing rule that each message must set exactly one of `content` or `content_file`.
Testing requirements:
- Add a valid fixture with `cache_control: {type: ephemeral, ttl: 1h}`.
- Add a valid fixture with `cache_control: {type: ephemeral}`.
- Add invalid fixtures for:
- empty `cache_control.type`
- unsupported `cache_control.type`
- unsupported `cache_control.ttl`
- unknown field under `cache_control`
- Assert valid fixtures load with the expected `CacheControl` values.
- Assert invalid fixtures fail with `promptdef.ErrInvalidPromptDefinition` or `promptdef.ErrInvalidYAML` as appropriate.
- Re-run:
```bash
go test ./internal/promptdef
```
Do not update renderer, formatter, LLM serialization, adapters, or user-facing docs in this stage.
## Stage 2: Rendering, Prepared Output, And Render Hashing
Objective: preserve cache-control metadata through rendering, expose it in prepared-run output, and make `rendered_prompt_hash` reflect outbound-affecting cache metadata.
Files to update:
- `internal/domain/domain.go`
- `internal/prompt/go_renderer.go`
- `internal/prompt/renderer_test.go`
- `internal/usecase/runner.go`
- `internal/usecase/runner_test.go`
- `internal/domain/prepared_run_test.go`
- `internal/format/prepared_run.go`
- `internal/format/prepared_run_test.go`
Domain changes:
- Add cache control to rendered messages:
```go
type RenderedMessage struct {
Role string `json:"role"`
Content string `json:"content"`
CacheControl *CacheControl `json:"cache_control,omitempty"`
}
```
Renderer changes:
- When a `PromptMessageTemplate` has `CacheControl`, copy it to the returned `RenderedMessage`.
- Do not mutate the template's pointer in place.
- Add and use a small helper such as `cloneCacheControl` to avoid aliasing.
- Do not change template execution behavior for `{{input "name"}}` or `{{.var}}`.
Rendered hash decision:
- `rendered_prompt_hash` must change when cache-control metadata changes, because the outbound provider request changes.
- Preserve existing hashes for no-cache prompts by keeping the current role/content hash flow.
- Append normalized cache-control metadata only when present, for example:
- `cache_control.type=<type>`
- `cache_control.ttl=<ttl>` only when non-empty
- Include cache-control fields in a stable order.
Prepared-run formatting:
- JSON prepared-run output should include `cache_control` on messages only when present.
- Text prepared-run output should show cache-control metadata under the specific message, before `content`, for example:
```text
cache_control: ephemeral ttl=1h
content: |
```
- If `ttl` is empty, omit the `ttl=...` suffix.
- Do not print cache-control lines for messages without cache control.
Testing requirements:
- Renderer test: cache control is copied to rendered messages.
- Renderer test: changing returned rendered cache control does not mutate the source template.
- Runner test: `rendered_prompt_hash` changes when cache control differs and remains stable for identical rendered messages.
- Format JSON test: `cache_control` appears only when present.
- Format text test: cache control appears under the correct message.
- Existing render/format tests still pass.
- Re-run:
```bash
go test ./internal/prompt ./internal/usecase ./internal/domain ./internal/format
```
Do not update outbound LLM serialization or adapter response surfaces in this stage.
## Stage 3: Outbound Chat Serialization And Usage Parsing
Objective: serialize cache-controlled messages as one-item text content-block arrays and parse cache usage fields from compatible provider responses.
Files to update:
- `internal/llm/openai_compatible_client.go`
- `internal/llm/openai_compatible_client_test.go`
- `internal/domain/domain.go`
Request serialization decisions:
- Keep string content for messages without cache control.
- Use content blocks only when `RenderedMessage.CacheControl` is non-nil.
- Do not introduce `content_blocks` prompt syntax.
- Do not serialize top-level `cache_control`, top-level `session_id`, or `extra_params`.
- Split request and response wire message structs. The current `openAIChatMessage` is used in both request and response paths; changing request content to `any` should not make response decoding less strict than necessary.
Use request wire types equivalent to:
```go
type openAIChatRequestMessage struct {
Role string `json:"role"`
Content any `json:"content"`
}
type openAIChatTextContentBlock struct {
Type string `json:"type"`
Text string `json:"text"`
CacheControl *openAICacheControl `json:"cache_control,omitempty"`
}
type openAICacheControl struct {
Type string `json:"type"`
TTL string `json:"ttl,omitempty"`
}
```
When cache control is present, content should serialize as:
```json
[
{
"type": "text",
"text": "rendered text",
"cache_control": {
"type": "ephemeral",
"ttl": "1h"
}
}
]
```
When `ttl` is empty, omit `ttl`.
Usage parsing decisions:
- Extend `domain.TokenUsage`:
```go
type TokenUsage struct {
PromptTokens int
CompletionTokens int
TotalTokens int
CachedTokens int
CacheWriteTokens int
}
```
- Parse `usage.prompt_tokens_details.cached_tokens` into `CachedTokens`.
- Parse `usage.cache_write_tokens` into `CacheWriteTokens`.
- Treat absent cache fields as zero.
- Do not fail response decoding if cache fields are absent.
- If a provider later returns a different cache-write field, add it in a later change after observing a real response.
Testing requirements:
- Existing request serialization test still sees string message content for no-cache messages.
- New request serialization test sees content-block array for cache-controlled messages.
- New request serialization test verifies empty `ttl` is omitted.
- Response parsing test verifies cached and cache-write tokens are mapped.
- Response parsing test verifies absent cache usage fields remain zero.
- Existing structured-output and service-tier behavior remains unchanged.
- Re-run:
```bash
go test ./internal/llm
```
Do not update CLI or HTTP observability in this stage unless required by compilation after extending `TokenUsage`.
## Stage 4: CLI And HTTP Cache Usage Observability
Objective: surface parsed cache usage through existing CLI and HTTP result paths without adding request knobs.
Files to update:
- `internal/adapter/cli/run.go`
- `internal/adapter/cli/run_test.go`
- `internal/adapter/http/dto.go`
- `internal/adapter/http/handler.go`
- `internal/adapter/http/handler_test.go`
CLI decisions:
- Do not add CLI flags.
- Keep success summaries concise.
- Preserve the existing summary output when cache usage is zero.
- When either cache field is non-zero, append:
```text
cached_tokens=<n> cache_write_tokens=<n>
```
HTTP decisions:
- Do not add request fields.
- Add numeric fields to `metadata.usage`:
- `cached_tokens`
- `cache_write_tokens`
- Always include these fields as numbers, matching the current non-omitempty numeric usage fields.
- Continue omitting raw model output unless `include_raw_output` is true.
Testing requirements:
- CLI summary omits cache fields when both are zero.
- CLI summary includes cache fields when either is non-zero.
- HTTP success response includes `metadata.usage.cached_tokens`.
- HTTP success response includes `metadata.usage.cache_write_tokens`.
- Existing HTTP request mapping is unchanged.
- Re-run:
```bash
go test ./internal/adapter/cli ./internal/adapter/http
```
## Stage 5: Canonical Documentation
Objective: update implemented-behavior docs after the feature works and tests pass.
Files to update:
- `docs/config.md`
- `docs/integrations/openai-compatible-chat.md`
- `docs/cli.md`
- `docs/troubleshooting.md`
- `docs/internal/runner.md`
- `docs/internal/adapters.md`, if adapter response behavior needs an explicit note
- `docs/roadmap/cache.md`, only if the accepted target changes during implementation
Documentation requirements:
- Keep all unimplemented ideas under `docs/roadmap/`.
- Document `messages[].cache_control` in the prompt definition reference in `docs/config.md`.
- Document the outbound message serialization rule in `docs/integrations/openai-compatible-chat.md`.
- Document parsed cache usage fields in `docs/integrations/openai-compatible-chat.md`.
- In `docs/cli.md`, note that prompt cache control is configured in prompt YAML, not with CLI flags.
- In `docs/troubleshooting.md`, add cache miss guidance:
- stable context should appear before the cache breakpoint
- dynamic input before the breakpoint changes the cache key
- providers may impose minimum token thresholds and breakpoint limits
- use cached/cache-write token counts to verify behavior
- Do not claim support for deferred roadmap items.
Testing and verification:
```bash
go test ./...
```
If documentation examples include new YAML snippets, ensure they match the strict prompt loader schema.
## Stage 6: Final Verification
Objective: verify the whole change set end to end.
Required commands:
```bash
go test ./...
go build ./cmd/scriptorium
```
Recommended smoke checks:
```bash
go run ./cmd/scriptorium render \
--config ./examples/config.yml \
--prompt generic.markdown_summary \
--input transcript=./examples/fixtures/transcript.md \
--input glossary=./examples/fixtures/glossary.yml \
--format json
```
If adding a new example prompt with `cache_control`, also smoke-render that prompt with `--format json` and inspect that the rendered message includes `cache_control`.
Final review checklist:
- Existing prompt definitions without `cache_control` still load.
- Prompt definitions with `cache_control` render successfully.
- Unknown cache-control YAML fields are rejected.
- No-cache outbound messages still serialize with string `content`.
- Cache-controlled outbound messages serialize with text content-block arrays.
- `rendered_prompt_hash` changes when cache-control metadata changes.
- Cache usage fields are zero when providers omit them.
- CLI and HTTP request surfaces have no new knobs.
- CLI and HTTP response surfaces expose cache usage.
- Non-roadmap docs describe only implemented behavior.
- `docs/roadmap/cache.md` still defines target state and deferred work, not task steps.
## Stage Boundaries
Stages 1 through 4 can be implemented in separate prompts if needed. Stage 5 should happen only after behavior is implemented. Stage 6 should be performed after all implementation and documentation changes are complete.
If a stage fails because prior stage changes were not implemented, stop and complete the earlier stage first rather than papering over missing domain fields or DTO mappings.
## Open Questions
None required for the initial implementation.
The plan chooses message-level `cache_control`, string content for legacy messages, text-block array content only when cache control is present, and no new CLI/HTTP request knobs. These choices match `docs/roadmap/cache.md` and preserve existing prompt-definition compatibility.

View File

@@ -252,6 +252,39 @@ Relevant links:
- [Configuration reference](config.md) - [Configuration reference](config.md)
- [Operations guide](operations.md) - [Operations guide](operations.md)
## Prompt Cache Misses Or No Cache Usage
Symptom:
- CLI run summary omits `cached_tokens` / `cache_write_tokens`.
- HTTP `metadata.usage.cached_tokens` and `metadata.usage.cache_write_tokens` are both `0`.
- Provider cost or latency does not improve after repeated similar runs.
Likely cause:
- The selected prompt has no `messages[].cache_control`.
- Dynamic per-run input appears before the cache-controlled message and changes the provider cache key.
- The provider does not support the serialized cache-control shape for the selected model.
- The provider imposes minimum token thresholds or cache-breakpoint limits.
Diagnostic step:
- Run `render --format json` and verify the intended rendered message includes `cache_control`.
- Confirm stable reusable context appears before the cache-controlled message, with dynamic input after it.
- Check provider docs/logs for model support, minimum token thresholds, and breakpoint limits.
Safe fix:
- Move stable reusable context before the cache-controlled message.
- Move highly dynamic input after the cache breakpoint.
- Keep `cache_control.type: ephemeral` and, when using `ttl`, set `ttl: 1h`.
- Use CLI cache counters or HTTP cache usage fields to verify cache reads/writes after rerunning.
Relevant links:
- [Configuration reference](config.md)
- [OpenAI-compatible chat integration](integrations/openai-compatible-chat.md)
## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status) ## Validation Status Failed (`run` Exit 2 Or HTTP 200 With Failed Status)
Symptom: Symptom:

View File

@@ -606,7 +606,7 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
if res == nil { if res == nil {
return return
} }
fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d\n", fmt.Fprintf(stderr, "prompt=%s@%s selected_profile=%s model=%s validation=%s mode=%s validation_errors=%d prompt_hash=%s inputs=%d usage=%d/%d/%d",
res.PromptID, res.PromptID,
res.PromptVersion, res.PromptVersion,
res.SelectedProfileID, res.SelectedProfileID,
@@ -620,6 +620,10 @@ func printSummary(stderr io.Writer, res *domain.RunResult) {
res.Usage.CompletionTokens, res.Usage.CompletionTokens,
res.Usage.TotalTokens, res.Usage.TotalTokens,
) )
if res.Usage.CachedTokens != 0 || res.Usage.CacheWriteTokens != 0 {
fmt.Fprintf(stderr, " cached_tokens=%d cache_write_tokens=%d", res.Usage.CachedTokens, res.Usage.CacheWriteTokens)
}
fmt.Fprintln(stderr)
} }
func printUsage(w io.Writer) { func printUsage(w io.Writer) {

View File

@@ -1064,6 +1064,38 @@ func TestWriteOutputAndSummaryUseSeparateWriters(t *testing.T) {
if !strings.Contains(stderr.String(), "prompt=p@1") { if !strings.Contains(stderr.String(), "prompt=p@1") {
t.Fatalf("expected summary on stderr, got %q", stderr.String()) t.Fatalf("expected summary on stderr, got %q", stderr.String())
} }
if strings.Contains(stderr.String(), "cached_tokens=") || strings.Contains(stderr.String(), "cache_write_tokens=") {
t.Fatalf("expected zero cache usage to be omitted from summary, got %q", stderr.String())
}
}
func TestPrintSummaryIncludesCacheUsageWhenPresent(t *testing.T) {
var stderr bytes.Buffer
printSummary(&stderr, &domain.RunResult{
PromptID: "p",
PromptVersion: "1",
SelectedProfileID: "exec",
ModelName: "m",
Validation: domain.ValidationResult{Status: domain.ValidationPassed, Mode: domain.ValidationBasic},
RenderedPromptHash: "h",
InputHashes: map[string]string{"in": "x"},
Usage: domain.TokenUsage{
PromptTokens: 10,
CompletionTokens: 5,
TotalTokens: 15,
CachedTokens: 0,
CacheWriteTokens: 3,
},
})
summary := stderr.String()
if !strings.Contains(summary, "usage=10/5/15") {
t.Fatalf("expected base usage summary, got %q", summary)
}
if !strings.Contains(summary, "cached_tokens=0 cache_write_tokens=3") {
t.Fatalf("expected cache usage in summary, got %q", summary)
}
} }
type cliTestLibrary struct { type cliTestLibrary struct {

View File

@@ -86,6 +86,8 @@ type tokenUsageDTO struct {
PromptTokens int `json:"prompt_tokens"` PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"` CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"` TotalTokens int `json:"total_tokens"`
CachedTokens int `json:"cached_tokens"`
CacheWriteTokens int `json:"cache_write_tokens"`
} }
type validationDTO struct { type validationDTO struct {

View File

@@ -105,6 +105,8 @@ func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
PromptTokens: res.Usage.PromptTokens, PromptTokens: res.Usage.PromptTokens,
CompletionTokens: res.Usage.CompletionTokens, CompletionTokens: res.Usage.CompletionTokens,
TotalTokens: res.Usage.TotalTokens, TotalTokens: res.Usage.TotalTokens,
CachedTokens: res.Usage.CachedTokens,
CacheWriteTokens: res.Usage.CacheWriteTokens,
}, },
StartTime: res.StartTime, StartTime: res.StartTime,
EndTime: res.EndTime, EndTime: res.EndTime,

View File

@@ -66,7 +66,13 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
APIKeyEnv: envName, APIKeyEnv: envName,
}, },
InputHashes: map[string]string{"transcript": "h1"}, InputHashes: map[string]string{"transcript": "h1"},
Usage: domain.TokenUsage{PromptTokens: 1, CompletionTokens: 2, TotalTokens: 3}, Usage: domain.TokenUsage{
PromptTokens: 1,
CompletionTokens: 2,
TotalTokens: 3,
CachedTokens: 4,
CacheWriteTokens: 5,
},
StartTime: start, StartTime: start,
EndTime: end, EndTime: end,
Duration: 2 * time.Second, Duration: 2 * time.Second,
@@ -111,6 +117,13 @@ func TestHandlerPostRunsSuccessWithExplicitProfileID(t *testing.T) {
if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" { if metadata["model_name"] != "m1" || metadata["endpoint"] != "http://llm/v1" {
t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"]) t.Fatalf("unexpected model metadata: name=%#v endpoint=%#v", metadata["model_name"], metadata["endpoint"])
} }
usage := metadata["usage"].(map[string]any)
if usage["prompt_tokens"] != float64(1) || usage["completion_tokens"] != float64(2) || usage["total_tokens"] != float64(3) {
t.Fatalf("unexpected base usage metadata: %#v", usage)
}
if usage["cached_tokens"] != float64(4) || usage["cache_write_tokens"] != float64(5) {
t.Fatalf("unexpected cache usage metadata: %#v", usage)
}
modelParams := metadata["model_params"].(map[string]any) modelParams := metadata["model_params"].(map[string]any)
if modelParams["api_key_env"] != envName { if modelParams["api_key_env"] != envName {
t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"]) t.Fatalf("expected model_params.api_key_env=%q, got %#v", envName, modelParams["api_key_env"])
@@ -171,6 +184,10 @@ func TestHandlerPostRunsSuccessUsingPromptDefaultProfile(t *testing.T) {
if metadata["selected_profile_id"] != "prompt-default" { if metadata["selected_profile_id"] != "prompt-default" {
t.Fatalf("expected selected_profile_id from result, got %#v", metadata["selected_profile_id"]) t.Fatalf("expected selected_profile_id from result, got %#v", metadata["selected_profile_id"])
} }
usage := metadata["usage"].(map[string]any)
if usage["cached_tokens"] != float64(0) || usage["cache_write_tokens"] != float64(0) {
t.Fatalf("expected zero cache usage fields to be included, got %#v", usage)
}
} }
func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) { func TestHandlerModelOverrideMapsAllSupportedExecutionFields(t *testing.T) {

View File

@@ -40,6 +40,19 @@ const (
ValidationSkipped ValidationStatus = "skipped" ValidationSkipped ValidationStatus = "skipped"
) )
// CacheControlType defines provider cache behavior for prompt content.
type CacheControlType string
const (
CacheControlEphemeral CacheControlType = "ephemeral"
)
// CacheControl describes provider cache metadata attached to prompt content.
type CacheControl struct {
Type CacheControlType `yaml:"type" json:"type"`
TTL string `yaml:"ttl,omitempty" json:"ttl,omitempty"`
}
// RunRequest represents a request to generate a single artifact. // RunRequest represents a request to generate a single artifact.
type RunRequest struct { type RunRequest struct {
PromptID string PromptID string
@@ -134,6 +147,7 @@ type PromptMessageTemplate struct {
Role string `yaml:"role"` Role string `yaml:"role"`
Content string `yaml:"content"` Content string `yaml:"content"`
ContentFile string `yaml:"content_file"` ContentFile string `yaml:"content_file"`
CacheControl *CacheControl `yaml:"cache_control,omitempty" json:"cache_control,omitempty"`
} }
// ExecutionProfile describes how and where to execute a model. // ExecutionProfile describes how and where to execute a model.
@@ -182,6 +196,7 @@ type RenderedPrompt struct {
type RenderedMessage struct { type RenderedMessage struct {
Role string `json:"role"` Role string `json:"role"`
Content string `json:"content"` Content string `json:"content"`
CacheControl *CacheControl `json:"cache_control,omitempty"`
} }
// GenerateRequest is the internal request passed to the LLM client. // GenerateRequest is the internal request passed to the LLM client.
@@ -222,6 +237,8 @@ type TokenUsage struct {
PromptTokens int PromptTokens int
CompletionTokens int CompletionTokens int
TotalTokens int TotalTokens int
CachedTokens int
CacheWriteTokens int
} }
// ValidationResult represents the outcome of an output validation. // ValidationResult represents the outcome of an output validation.

View File

@@ -53,3 +53,52 @@ func TestPreparedRunJSONDoesNotIncludeSecretValues(t *testing.T) {
} }
} }
} }
func TestPreparedRunJSONIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := PreparedRun{
PromptID: "prompt.id",
SelectedProfileID: "local-fast",
EffectiveModelParams: ExecutionTarget{
Endpoint: "http://llm/v1",
Model: "gpt-test",
},
RenderedPromptHash: "rendered-hash",
Messages: []RenderedMessage{
{
Role: "system",
Content: "You are helpful.",
CacheControl: &CacheControl{
Type: CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize this."},
},
}
b, err := json.Marshal(prepared)
if err != nil {
t.Fatalf("marshal failed: %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(b, &decoded); err != nil {
t.Fatalf("unmarshal failed: %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected cache_control on first message, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}

View File

@@ -151,6 +151,13 @@ func (textPreparedRunFormatter) Format(prepared *domain.PreparedRun) ([]byte, er
messages := byRole[role] messages := byRole[role]
for i, msg := range messages { for i, msg := range messages {
fmt.Fprintf(&b, " - message: %d\n", i+1) fmt.Fprintf(&b, " - message: %d\n", i+1)
if msg.CacheControl != nil {
fmt.Fprintf(&b, " cache_control: %s", msg.CacheControl.Type)
if msg.CacheControl.TTL != "" {
fmt.Fprintf(&b, " ttl=%s", msg.CacheControl.TTL)
}
fmt.Fprintln(&b)
}
fmt.Fprintln(&b, " content: |") fmt.Fprintln(&b, " content: |")
content := msg.Content content := msg.Content
if content == "" { if content == "" {

View File

@@ -62,6 +62,58 @@ func TestTextFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
} }
} }
func TestTextFormatterIncludesMessageCacheControlBeforeContent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []domain.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " system:\n - message: 1\n cache_control: ephemeral ttl=1h\n content: |") {
t.Fatalf("expected system message cache control before content, got:\n%s", s)
}
if strings.Count(s, "cache_control:") != 1 {
t.Fatalf("expected exactly one cache_control line, got:\n%s", s)
}
}
func TestTextFormatterOmitsEmptyCacheControlTTL(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []domain.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
},
},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatText)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
s := string(out)
if !strings.Contains(s, " cache_control: ephemeral\n") {
t.Fatalf("expected cache_control line without ttl, got:\n%s", s)
}
if strings.Contains(s, "ttl=") {
t.Fatalf("expected empty ttl to be omitted, got:\n%s", s)
}
}
func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) { func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
prepared := samplePreparedRun() prepared := samplePreparedRun()
@@ -98,6 +150,47 @@ func TestJSONFormatterEmitsValidJSONAndIncludesPreparedRunFields(t *testing.T) {
} }
} }
func TestJSONFormatterIncludesMessageCacheControlOnlyWhenPresent(t *testing.T) {
prepared := samplePreparedRun()
prepared.Messages = []domain.RenderedMessage{
{
Role: "system",
Content: "System guidance.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Summarize the transcript."},
}
out, err := FormatPreparedRun(prepared, PreparedRunFormatJSON)
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
var decoded struct {
Messages []map[string]any `json:"messages"`
}
if err := json.Unmarshal(out, &decoded); err != nil {
t.Fatalf("expected valid json output, got %v", err)
}
if len(decoded.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(decoded.Messages))
}
cacheControl, ok := decoded.Messages[0]["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected first message cache_control, got %#v", decoded.Messages[0])
}
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
if _, ok := decoded.Messages[1]["cache_control"]; ok {
t.Fatalf("expected second message to omit cache_control, got %#v", decoded.Messages[1])
}
}
func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) { func TestJSONFormatterDoesNotIncludeResolvedAPIKeyValue(t *testing.T) {
const secret = "super-secret-api-key" const secret = "super-secret-api-key"
t.Setenv("SCRIPTORIUM_API_KEY", secret) t.Setenv("SCRIPTORIUM_API_KEY", secret)

View File

@@ -151,6 +151,8 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
PromptTokens: wireResp.Usage.PromptTokens, PromptTokens: wireResp.Usage.PromptTokens,
CompletionTokens: wireResp.Usage.CompletionTokens, CompletionTokens: wireResp.Usage.CompletionTokens,
TotalTokens: wireResp.Usage.TotalTokens, TotalTokens: wireResp.Usage.TotalTokens,
CachedTokens: wireResp.Usage.PromptTokensDetails.CachedTokens,
CacheWriteTokens: wireResp.Usage.CacheWriteTokens,
}, },
}, nil }, nil
} }
@@ -168,12 +170,9 @@ func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultMod
Model: model, Model: model,
} }
wireReq.Messages = make([]openAIChatMessage, 0, len(req.Prompt.Messages)) wireReq.Messages = make([]openAIChatRequestMessage, 0, len(req.Prompt.Messages))
for _, msg := range req.Prompt.Messages { for _, msg := range req.Prompt.Messages {
wireReq.Messages = append(wireReq.Messages, openAIChatMessage{ wireReq.Messages = append(wireReq.Messages, openAIChatRequestMessageFromRenderedMessage(msg))
Role: msg.Role,
Content: msg.Content,
})
} }
if req.Target.Temperature != 0 { if req.Target.Temperature != 0 {
@@ -201,7 +200,7 @@ func openAIChatRequestFromGenerateRequest(req domain.GenerateRequest, defaultMod
type openAIChatRequest struct { type openAIChatRequest struct {
Model string `json:"model"` Model string `json:"model"`
Messages []openAIChatMessage `json:"messages"` Messages []openAIChatRequestMessage `json:"messages"`
Temperature *float64 `json:"temperature,omitempty"` Temperature *float64 `json:"temperature,omitempty"`
MaxTokens *int `json:"max_tokens,omitempty"` MaxTokens *int `json:"max_tokens,omitempty"`
TopP *float64 `json:"top_p,omitempty"` TopP *float64 `json:"top_p,omitempty"`
@@ -209,19 +208,39 @@ type openAIChatRequest struct {
ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"` ResponseFormat *openAIResponseFormat `json:"response_format,omitempty"`
} }
type openAIChatMessage struct { type openAIChatRequestMessage struct {
Role string `json:"role"`
Content any `json:"content"`
}
type openAIChatTextContentBlock struct {
Type string `json:"type"`
Text string `json:"text"`
CacheControl *openAICacheControl `json:"cache_control,omitempty"`
}
type openAICacheControl struct {
Type string `json:"type"`
TTL string `json:"ttl,omitempty"`
}
type openAIChatResponseMessage struct {
Role string `json:"role"` Role string `json:"role"`
Content string `json:"content"` Content string `json:"content"`
} }
type openAIChatResponse struct { type openAIChatResponse struct {
Choices []struct { Choices []struct {
Message openAIChatMessage `json:"message"` Message openAIChatResponseMessage `json:"message"`
} `json:"choices"` } `json:"choices"`
Usage struct { Usage struct {
PromptTokens int `json:"prompt_tokens"` PromptTokens int `json:"prompt_tokens"`
CompletionTokens int `json:"completion_tokens"` CompletionTokens int `json:"completion_tokens"`
TotalTokens int `json:"total_tokens"` TotalTokens int `json:"total_tokens"`
PromptTokensDetails struct {
CachedTokens int `json:"cached_tokens"`
} `json:"prompt_tokens_details"`
CacheWriteTokens int `json:"cache_write_tokens"`
} `json:"usage"` } `json:"usage"`
} }
@@ -236,6 +255,28 @@ type openAIJSONSchemaEnvelope struct {
Schema any `json:"schema"` Schema any `json:"schema"`
} }
func openAIChatRequestMessageFromRenderedMessage(msg domain.RenderedMessage) openAIChatRequestMessage {
wireMsg := openAIChatRequestMessage{
Role: msg.Role,
Content: msg.Content,
}
if msg.CacheControl == nil {
return wireMsg
}
wireMsg.Content = []openAIChatTextContentBlock{
{
Type: "text",
Text: msg.Content,
CacheControl: &openAICacheControl{
Type: string(msg.CacheControl.Type),
TTL: msg.CacheControl.TTL,
},
},
}
return wireMsg
}
func toOpenAIResponseFormat(spec *domain.StructuredOutputSpec) (*openAIResponseFormat, error) { func toOpenAIResponseFormat(spec *domain.StructuredOutputSpec) (*openAIResponseFormat, error) {
if spec == nil { if spec == nil {
return nil, nil return nil, nil

View File

@@ -89,6 +89,9 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
if resp.Usage.PromptTokens != 11 || resp.Usage.CompletionTokens != 22 || resp.Usage.TotalTokens != 33 { if resp.Usage.PromptTokens != 11 || resp.Usage.CompletionTokens != 22 || resp.Usage.TotalTokens != 33 {
t.Fatalf("unexpected usage: %+v", resp.Usage) t.Fatalf("unexpected usage: %+v", resp.Usage)
} }
if resp.Usage.CachedTokens != 0 || resp.Usage.CacheWriteTokens != 0 {
t.Fatalf("expected absent cache usage fields to remain zero, got %+v", resp.Usage)
}
if obs.Authorization != "Bearer secret-key" { if obs.Authorization != "Bearer secret-key" {
t.Fatalf("unexpected Authorization header: %q", obs.Authorization) t.Fatalf("unexpected Authorization header: %q", obs.Authorization)
@@ -144,6 +147,155 @@ func TestOpenAICompatibleClientGenerateSuccess(t *testing.T) {
} }
} }
func TestOpenAICompatibleClientSerializesCacheControlledMessageAsContentBlock(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "Stable instructions.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Dynamic request."},
}},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
for _, forbidden := range []string{"cache_control", "session_id", "extra_params"} {
if _, exists := observedBody[forbidden]; exists {
t.Fatalf("expected top-level %s to be omitted, got %#v", forbidden, observedBody[forbidden])
}
}
msgs, ok := observedBody["messages"].([]any)
if !ok || len(msgs) != 2 {
t.Fatalf("unexpected messages payload: %#v", observedBody["messages"])
}
msg0 := msgs[0].(map[string]any)
if msg0["role"] != "system" {
t.Fatalf("unexpected first message role: %#v", msg0["role"])
}
contentBlocks, ok := msg0["content"].([]any)
if !ok || len(contentBlocks) != 1 {
t.Fatalf("expected first message content block array, got %#v", msg0["content"])
}
block := contentBlocks[0].(map[string]any)
if block["type"] != "text" || block["text"] != "Stable instructions." {
t.Fatalf("unexpected text content block: %#v", block)
}
cacheControl, ok := block["cache_control"].(map[string]any)
if !ok {
t.Fatalf("expected cache_control on content block, got %#v", block)
}
if cacheControl["type"] != string(domain.CacheControlEphemeral) || cacheControl["ttl"] != "1h" {
t.Fatalf("unexpected cache_control payload: %#v", cacheControl)
}
msg1 := msgs[1].(map[string]any)
if msg1["role"] != "user" || msg1["content"] != "Dynamic request." {
t.Fatalf("expected uncached message to keep string content, got %#v", msg1)
}
}
func TestOpenAICompatibleClientOmitsEmptyCacheControlTTL(t *testing.T) {
var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
if err := json.NewDecoder(r.Body).Decode(&observedBody); err != nil {
t.Fatalf("failed to decode request body: %v", err)
}
_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1"})
if err != nil {
t.Fatal(err)
}
_, err = client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "Stable instructions.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
},
},
}},
Target: domain.ExecutionTarget{Model: "model"},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
msgs := observedBody["messages"].([]any)
msg0 := msgs[0].(map[string]any)
contentBlocks := msg0["content"].([]any)
block := contentBlocks[0].(map[string]any)
cacheControl := block["cache_control"].(map[string]any)
if cacheControl["type"] != string(domain.CacheControlEphemeral) {
t.Fatalf("unexpected cache_control type: %#v", cacheControl)
}
if _, exists := cacheControl["ttl"]; exists {
t.Fatalf("expected empty ttl to be omitted, got %#v", cacheControl)
}
}
func TestOpenAICompatibleClientParsesCacheUsage(t *testing.T) {
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{
"choices": [{"message": {"role": "assistant", "content": "ok"}}],
"usage": {
"prompt_tokens": 100,
"completion_tokens": 20,
"total_tokens": 120,
"prompt_tokens_details": {"cached_tokens": 80},
"cache_write_tokens": 60
}
}`))
}))
defer ts.Close()
client, err := NewOpenAICompatibleClient(OpenAICompatibleConfig{BaseURL: ts.URL + "/v1", Model: "model"})
if err != nil {
t.Fatal(err)
}
resp, err := client.Generate(context.Background(), domain.GenerateRequest{
Prompt: domain.RenderedPrompt{Messages: []domain.RenderedMessage{{Role: "user", Content: "hi"}}},
})
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if resp.Usage.PromptTokens != 100 || resp.Usage.CompletionTokens != 20 || resp.Usage.TotalTokens != 120 {
t.Fatalf("unexpected base usage fields: %+v", resp.Usage)
}
if resp.Usage.CachedTokens != 80 || resp.Usage.CacheWriteTokens != 60 {
t.Fatalf("unexpected cache usage fields: %+v", resp.Usage)
}
}
func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *testing.T) { func TestOpenAICompatibleClientOmitsResponseFormatWhenNoStructuredOutput(t *testing.T) {
var observedBody map[string]any var observedBody map[string]any
ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

View File

@@ -77,6 +77,7 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
renderedMessages = append(renderedMessages, domain.RenderedMessage{ renderedMessages = append(renderedMessages, domain.RenderedMessage{
Role: tmplMsg.Role, Role: tmplMsg.Role,
Content: buf.String(), Content: buf.String(),
CacheControl: cloneCacheControl(tmplMsg.CacheControl),
}) })
} }
@@ -84,3 +85,11 @@ func (r *goRenderer) Render(ctx context.Context, definition *domain.PromptDefini
Messages: renderedMessages, Messages: renderedMessages,
}, nil }, nil
} }
func cloneCacheControl(in *domain.CacheControl) *domain.CacheControl {
if in == nil {
return nil
}
out := *in
return &out
}

View File

@@ -78,6 +78,66 @@ func TestGoRenderer_Render(t *testing.T) {
} }
}) })
t.Run("copying cache control to rendered messages", func(t *testing.T) {
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{
Role: "system",
Content: "You are concise.",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "Analyze this: {{input \"transcript\"}}"},
},
}
res, err := renderer.Render(ctx, def, inputs, vars)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(res.Messages) != 2 {
t.Fatalf("expected 2 messages, got %d", len(res.Messages))
}
if res.Messages[0].CacheControl == nil {
t.Fatal("expected rendered cache control")
}
if res.Messages[0].CacheControl.Type != domain.CacheControlEphemeral {
t.Fatalf("unexpected cache control type: %q", res.Messages[0].CacheControl.Type)
}
if res.Messages[0].CacheControl.TTL != "1h" {
t.Fatalf("unexpected cache control ttl: %q", res.Messages[0].CacheControl.TTL)
}
if res.Messages[1].CacheControl != nil {
t.Fatalf("expected no cache control on second message, got %#v", res.Messages[1].CacheControl)
}
})
t.Run("rendered cache control does not alias source template", func(t *testing.T) {
source := &domain.CacheControl{Type: domain.CacheControlEphemeral, TTL: "1h"}
def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},
Templates: []domain.PromptMessageTemplate{
{Role: "system", Content: "You are concise.", CacheControl: source},
},
}
res, err := renderer.Render(ctx, def, inputs, vars)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if res.Messages[0].CacheControl == source {
t.Fatal("expected rendered cache control to be cloned")
}
res.Messages[0].CacheControl.TTL = ""
if source.TTL != "1h" {
t.Fatalf("source cache control was mutated, ttl=%q", source.TTL)
}
})
t.Run("accessing vars", func(t *testing.T) { t.Run("accessing vars", func(t *testing.T) {
def := &domain.PromptDefinition{ def := &domain.PromptDefinition{
Inputs: []domain.PromptInput{{Name: "transcript", Required: true}}, Inputs: []domain.PromptInput{{Name: "transcript", Required: true}},

View File

@@ -45,6 +45,12 @@ type promptMessageFile struct {
Role string `yaml:"role"` Role string `yaml:"role"`
Content string `yaml:"content"` Content string `yaml:"content"`
ContentFile string `yaml:"content_file"` ContentFile string `yaml:"content_file"`
CacheControl *cacheControlFile `yaml:"cache_control"`
}
type cacheControlFile struct {
Type string `yaml:"type"`
TTL string `yaml:"ttl"`
} }
type promptOutputContractFile struct { type promptOutputContractFile struct {
@@ -212,6 +218,11 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
return nil, fmt.Errorf("message %d (%s) must set exactly one of content or content_file", i, role) return nil, fmt.Errorf("message %d (%s) must set exactly one of content or content_file", i, role)
} }
cacheControl, err := normalizeCacheControl(msg.CacheControl)
if err != nil {
return nil, fmt.Errorf("message %d (%s) cache_control: %w", i, role, err)
}
templateContent := msg.Content templateContent := msg.Content
resolvedContentFile := "" resolvedContentFile := ""
if hasContentFile { if hasContentFile {
@@ -233,6 +244,7 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
Role: role, Role: role,
Content: templateContent, Content: templateContent,
ContentFile: resolvedContentFile, ContentFile: resolvedContentFile,
CacheControl: cacheControl,
}) })
} }
@@ -274,6 +286,30 @@ func normalizePromptDefinition(raw *promptDefinitionFile, sourcePath string) (*d
}, nil }, nil
} }
func normalizeCacheControl(raw *cacheControlFile) (*domain.CacheControl, error) {
if raw == nil {
return nil, nil
}
cacheType := strings.TrimSpace(raw.Type)
if cacheType == "" {
return nil, errors.New("type is required")
}
if domain.CacheControlType(cacheType) != domain.CacheControlEphemeral {
return nil, fmt.Errorf("unsupported type %q", cacheType)
}
ttl := strings.TrimSpace(raw.TTL)
if ttl != "" && ttl != "1h" {
return nil, fmt.Errorf("unsupported ttl %q", ttl)
}
return &domain.CacheControl{
Type: domain.CacheControlType(cacheType),
TTL: ttl,
}, nil
}
func isValidOutputFormat(f domain.OutputFormat) bool { func isValidOutputFormat(f domain.OutputFormat) bool {
switch f { switch f {
case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON: case domain.FormatText, domain.FormatMarkdown, domain.FormatJSON:

View File

@@ -68,6 +68,34 @@ func TestFilesystemRepository_GetPromptDefinition(t *testing.T) {
} }
}) })
t.Run("valid cache control with ttl", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "valid-cache-control-ttl", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if len(p.Templates) != 2 {
t.Fatalf("expected 2 messages, got %d", len(p.Templates))
}
assertCacheControl(t, p.Templates[0].CacheControl, domain.CacheControlEphemeral, "1h")
if p.Templates[1].CacheControl != nil {
t.Fatalf("expected second message cache control to be nil, got %#v", p.Templates[1].CacheControl)
}
})
t.Run("valid cache control without ttl", func(t *testing.T) {
p, err := repo.GetPromptDefinition(ctx, "valid-cache-control-without-ttl", "")
if err != nil {
t.Fatalf("expected no error, got %v", err)
}
if len(p.Templates) != 2 {
t.Fatalf("expected 2 messages, got %d", len(p.Templates))
}
assertCacheControl(t, p.Templates[0].CacheControl, domain.CacheControlEphemeral, "")
if p.Templates[1].CacheControl != nil {
t.Fatalf("expected second message cache control to be nil, got %#v", p.Templates[1].CacheControl)
}
})
t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) { t.Run("valid nested file-backed prompt resolves content file relative to nested YAML", func(t *testing.T) {
nestedDir := filepath.Join(tmpDir, "dnd", "recap") nestedDir := filepath.Join(tmpDir, "dnd", "recap")
if err := os.MkdirAll(nestedDir, 0o755); err != nil { if err := os.MkdirAll(nestedDir, 0o755); err != nil {
@@ -258,6 +286,10 @@ output:
{name: "invalid validation mode", id: "invalid_validation_mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}}, {name: "invalid validation mode", id: "invalid_validation_mode", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"invalid validation mode"}},
{name: "json_schema without schema_path", id: "json_schema_without_schema_path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}}, {name: "json_schema without schema_path", id: "json_schema_without_schema_path", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"schema_path"}},
{name: "unknown input field", id: "unknown_input_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}}, {name: "unknown input field", id: "unknown_input_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unknown_input_setting not found"}},
{name: "empty cache control type", id: "empty_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "type is required"}},
{name: "unsupported cache control type", id: "unsupported_cache_control_type", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported type"}},
{name: "unsupported cache control ttl", id: "unsupported_cache_control_ttl", targetErr: ErrInvalidPromptDefinition, errSubstrs: []string{"cache_control", "unsupported ttl"}},
{name: "unknown cache control field", id: "unknown_cache_control_field", targetErr: ErrInvalidYAML, errSubstrs: []string{"field unexpected not found"}},
} }
for _, tc := range cases { for _, tc := range cases {
@@ -282,6 +314,19 @@ output:
}) })
} }
func assertCacheControl(t *testing.T, got *domain.CacheControl, wantType domain.CacheControlType, wantTTL string) {
t.Helper()
if got == nil {
t.Fatal("expected cache control, got nil")
}
if got.Type != wantType {
t.Fatalf("unexpected cache control type: got %q want %q", got.Type, wantType)
}
if got.TTL != wantTTL {
t.Fatalf("unexpected cache control ttl: got %q want %q", got.TTL, wantTTL)
}
}
func writePromptTestFile(t *testing.T, path string, content string) { func writePromptTestFile(t *testing.T, path string, content string) {
t.Helper() t.Helper()
if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil { if err := os.WriteFile(path, []byte(strings.TrimLeft(content, "\n")), 0o644); err != nil {

View File

@@ -0,0 +1,10 @@
id: empty-cache-control-type
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control: {}
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,12 @@
id: unknown-cache-control-field
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
unexpected: true
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,12 @@
id: unsupported-cache-control-ttl
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
ttl: 5m
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,11 @@
id: unsupported-cache-control-type
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: persistent
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,14 @@
id: valid-cache-control-ttl
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
ttl: 1h
- role: user
content: "Summarize the input."
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -0,0 +1,13 @@
id: valid-cache-control-without-ttl
version: "1.0.0"
messages:
- role: system
content: "Use cached instructions."
cache_control:
type: ephemeral
- role: user
content: "Summarize the input."
output:
format: markdown
validation_mode: basic
repair_attempts: 0

View File

@@ -428,6 +428,14 @@ func hashRenderedPrompt(p domain.RenderedPrompt) string {
b.WriteString(msg.Role) b.WriteString(msg.Role)
b.WriteByte('\n') b.WriteByte('\n')
b.WriteString(msg.Content) b.WriteString(msg.Content)
if msg.CacheControl != nil {
b.WriteString("\ncache_control.type=")
b.WriteString(string(msg.CacheControl.Type))
if msg.CacheControl.TTL != "" {
b.WriteString("\ncache_control.ttl=")
b.WriteString(msg.CacheControl.TTL)
}
}
b.WriteString("\n---\n") b.WriteString("\n---\n")
} }
h := sha256.Sum256([]byte(b.String())) h := sha256.Sum256([]byte(b.String()))

View File

@@ -577,6 +577,61 @@ func TestDeriveStructuredSchemaName(t *testing.T) {
} }
} }
func TestHashRenderedPromptIncludesCacheControlWhenPresent(t *testing.T) {
uncached := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{Role: "system", Content: "sys"},
{Role: "user", Content: "usr"},
}}
wantLegacyHash := hashString("system\nsys\n---\nuser\nusr\n---\n")
if got := hashRenderedPrompt(uncached); got != wantLegacyHash {
t.Fatalf("expected no-cache hash to preserve legacy input, got %q want %q", got, wantLegacyHash)
}
withCache := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "sys",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "usr"},
}}
alsoWithCache := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "sys",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
TTL: "1h",
},
},
{Role: "user", Content: "usr"},
}}
withoutTTL := domain.RenderedPrompt{Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "sys",
CacheControl: &domain.CacheControl{
Type: domain.CacheControlEphemeral,
},
},
{Role: "user", Content: "usr"},
}}
cachedHash := hashRenderedPrompt(withCache)
if cachedHash == hashRenderedPrompt(uncached) {
t.Fatal("expected cache control to change rendered prompt hash")
}
if cachedHash != hashRenderedPrompt(alsoWithCache) {
t.Fatal("expected identical cache control metadata to produce stable hash")
}
if cachedHash == hashRenderedPrompt(withoutTTL) {
t.Fatal("expected ttl changes to affect rendered prompt hash")
}
}
func TestRunnerRunSuccessful(t *testing.T) { func TestRunnerRunSuccessful(t *testing.T) {
promptRepo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)} promptRepo := &fakePromptRepo{def: promptDef(domain.FormatMarkdown, domain.ValidationBasic, 0)}
execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}} execRepo := &fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}}