diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md index 463ca94..52cfceb 100644 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -8,7 +8,9 @@ Allow consumers to select execution observation once for an application-owned operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should participate automatically. Include optional provider-reported cost alongside -token usage, with built-in cost support initially limited to OpenRouter. +token usage, with built-in cost support initially limited to OpenRouter. Extend +OpenRouter token reporting with reasoning counts, documented cache reads and +writes, and a derived token cache-hit ratio. The motivating consumer is CommandKit's optional `ckprompt` integration, which could use this facility to report token usage for an entire command or pipeline. @@ -29,15 +31,18 @@ PromptKit v0.24.0 already provides: execution, and [CoordinateRequest.Observer](../../coordinate.go) for coordinator-selected continuation runs. -The missing capabilities are contextual observer selection and provider-reported -cost accounting. Applications currently have to enroll each request explicitly, -and PromptKit does not expose provider charges. The existing +The missing capabilities are contextual observer selection, provider-reported +cost accounting, reasoning-token counts, and a presence-aware cache-hit helper. +OpenRouter cache writes also need their documented nested response mapping; +the current client reads a top-level `usage.cache_write_tokens` field instead. +Applications currently have to enroll each request explicitly, and PromptKit +does not expose provider charges. The existing [consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) describes that workflow. ## Target Public API -Add one public helper: +Add a public context helper: ```go func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context @@ -49,8 +54,8 @@ associated with the context; it does not replace the inherited observer. A nil observer leaves the inherited selection unchanged. No engine option, process-wide default, or replacement group type is needed. -Extend the existing result, event, and group surfaces with cost metadata as -described below. +Extend the existing token types and result, event, and group surfaces with the +token and cost metadata described below, plus a derived cache-hit helper. The intended consumer workflow is: @@ -258,6 +263,91 @@ requiring every provider to implement every capability is needed. This is configurable selection among PromptKit-supported provider implementations, not a public API for downstream registration of additional implementations. +## OpenRouter Token And Cache Reporting + +### Response Mapping + +Use OpenRouter's documented Chat Completions usage fields. Its +[response schema](https://openrouter.ai/docs/api_reference/overview), +[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting), +and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching) +define the following mapping, relative to `usage`: + +| Quantity | Response field | PromptKit field | +| --- | --- | --- | +| Input tokens | `prompt_tokens` | Existing `PromptTokens` | +| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` | +| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` | +| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` | +| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` | +| Total tokens | `total_tokens` | Existing `TotalTokens` | + +Preserve `CachedTokens` as the public field name and clarify its meaning as +cache-read tokens. Correct the OpenRouter cache-write mapping to the nested +field above; retain the generic provider's existing top-level handling. Do not +add the nested and top-level values together. OpenRouter uses its documented +nested value; an omitted nested value remains unknown. + +Add `ReasoningTokens` to the public/domain token values, presence indicators, +and observed token usage, including conversions, result serialization, attempt +and terminal events, and group summaries. Initially, OpenRouter supplies this +additional built-in response interpretation; the generic provider preserves its +existing behavior. Injected clients may supply the canonical count and its +presence through the existing model-client boundary. + +### Count Relationships And Availability + +Reasoning tokens are included in completion tokens, as explained in OpenRouter's +[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens). +Cache reads and writes describe input-token breakdowns. Present these as subsets +or details rather than additional consumption to add to input/output totals. +Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it. + +Consumers can derive non-reasoning output as completion tokens minus reasoning +tokens when both counts are available and consistent. No additional stored +non-reasoning count is needed in this scope. Capturing reasoning-token counts +does not capture reasoning text or change reasoning request settings. + +Reuse existing per-field presence and aggregation rules: omitted/null means +unknown, explicit zero means known zero, and aggregate `Known` and `Complete` +flags describe the reported contributions across dispatched attempts. Include +initial generations, retries, and repairs through the existing observation +accounting, retaining terminal-only group aggregation and deduplication. Preserve +the existing distinction between integer-only result usage and presence-aware +observations; zero in `TokenUsage` alone does not establish provider reporting. + +### Token Cache-Hit Ratio + +Add this derived helper on the existing presence-aware usage type: + +```go +func (u ObservedTokenUsage) CacheHitRatio() (float64, bool) +``` + +The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero +to one. It measures the proportion of input tokens served from cache, not the +proportion of requests with a cache hit or monetary savings. The Boolean reports +whether a ratio is available; an unavailable result returns zero and false. + +Return an available ratio only when both counts are known and complete, input +tokens are positive, and cache reads are between zero and input tokens inclusive. +Zero cache reads with known positive input yields an available zero ratio. Missing +or incomplete counts, zero input, or inconsistent counts leave it unavailable; +preserve the raw reported counts rather than clamping them or failing generation. +Cache-write availability does not affect this calculation. + +For a run or group, divide the aggregated cache-read count by the aggregated +input count; do not average per-attempt or per-run percentages. Do not use +`reads / (reads + writes)`, because cache writes need not account for all +uncached input. The same helper works on attempts, terminal events, and group +summaries without storing another independently maintained field. + +The ratio describes the included observations. A group ratio does not supersede +`TerminalCoverageComplete` or imply that all application work has finished. +For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts +are known and complete, even if cache writes are unknown. Output 840 with +reasoning 600 still represents 840 output tokens, not 1,440. + ## Optional Provider-Reported Cost ### Initial Provider Support @@ -360,10 +450,11 @@ delivery, and interpretation. This feature adds no CommandKit dependency or application-specific concepts to PromptKit. Smoke diagnostics retain their separate report/evidence model and are not -enrolled in ordinary execution observation by this change. Reasoning-token -capture, additional built-in cost providers, multiple currencies, currency -conversion, pricing estimates, billing reconciliation, persistent history, and -logging infrastructure are outside this scope. +enrolled in ordinary execution observation by this change. Reasoning-text +capture, changes to reasoning or caching request behavior, additional built-in +cost providers, multiple currencies, currency conversion, pricing estimates, +billing reconciliation, persistent history, and logging infrastructure are +outside this scope. ## Work Needed And Completion Criteria @@ -389,6 +480,15 @@ logging infrastructure are outside this scope. - Verify that provider diagnostic defaults do not become explicit values in the loaded configuration or replace source provenance in inspection. Cover the changed clearing behavior and endpoint-only authentication overrides. +- Carry OpenRouter reasoning-token counts through token values, presence, + results, events, and group summaries. Verify nested OpenRouter cache-write + decoding, including omitted/null/zero values and the nested value taking + precedence when a top-level value is also present. Preserve generic-provider + token mappings and existing field meanings. +- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and + incomplete counts, zero input, inconsistent counts, independence from + cache-write availability, and token-weighted aggregate ratios. Verify subset + accounting without adding reasoning or cached tokens to the reported totals. - Carry optional OpenRouter cost through response decoding, internal/public conversions, successful run results, attempt and terminal events, and existing group summaries. Keep result cost available without registering an observer. @@ -413,7 +513,9 @@ logging infrastructure are outside this scope. turn telemetry into an exact-decimal guarantee. - Document the exact helper contract in GoDoc and add a focused invocation-level aggregation example to the consumer observation guide, including cost - availability, units, and completeness. Document OpenRouter cost decoding in + availability, units, and completeness. Document the reasoning/cache fields and + cache-hit helper in their owning GoDoc and illustrate input/output breakdowns + without double-counting. Document OpenRouter token and cost decoding in the [integration contract](../integrations/openai-compatible-chat.md#response-handling), and update internal observation and model-client documentation for changed flows. Document `provider`, its smoke-test defaults and override semantics in the @@ -428,13 +530,15 @@ Organize the eventual implementation plan around two workstreams: 1. Contextual observer selection, including ordinary, prepared, coordinated, and nested execution boundaries and the existing group collector. 2. Backend provider selection and consolidation within `internal/llm`, including - smoke-test defaults and explicit overrides, followed by provider-reported cost - through decoding, accounting, public results, events, and group summaries. + smoke-test defaults and explicit overrides, followed by OpenRouter token + extensions and provider-reported cost through decoding, accounting, public + results, events, and group summaries, plus the derived cache-hit helper. Include the corresponding configuration and API documentation. The second area is more extensive than the context helper. Keep backend provider -configuration, diagnostic consolidation, and cost accounting in separately -reviewable implementation units rather than one large implementation prompt. +configuration, diagnostic consolidation, token/cache reporting, and cost +accounting in separately reviewable implementation units rather than one large +implementation prompt. Complete the remaining provider discussion before preparing that plan; avoid coupling delivery of contextual observation to provider cost support.