From 03209947f697b11bd633247c5800e9653bee13ef Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 09:48:38 +0000 Subject: [PATCH] Add optional OpenRouter cost reporting to the observation roadmap --- docs/roadmap/invocation-scoped-token-usage.md | 126 +++++++++++++++--- 1 file changed, 109 insertions(+), 17 deletions(-) diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md index 2ad11a9..2d5b8d4 100644 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -7,7 +7,8 @@ Status: accepted scope; implementation pending. Allow consumers to select execution observation once for an application-owned operation, rather than attach an observer to every PromptKit request. Nested, concurrent, and multi-engine calls using that operation's context should -participate automatically. +participate automatically. Include optional provider-reported cost alongside +token usage, with built-in cost support initially limited to OpenRouter. The motivating consumer is CommandKit's optional `ckprompt` integration, which could use this facility to report token usage for an entire command or pipeline. @@ -28,8 +29,9 @@ PromptKit v0.24.0 already provides: execution, and [CoordinateRequest.Observer](../../coordinate.go) for coordinator-selected continuation runs. -The missing capability is contextual observer selection. Applications currently -have to enroll each request explicitly. The existing +The missing capabilities are contextual observer selection and provider-reported +cost accounting. Applications currently have to enroll each request explicitly, +and PromptKit does not expose provider charges. The existing [consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) describes that workflow. @@ -46,7 +48,10 @@ or derived contexts that retain its values. It composes with any observer alread associated with the context; it does not replace the inherited observer. A nil observer leaves the inherited selection unchanged. -No engine option, process-wide default, or new aggregation type is needed. +No engine option, process-wide default, or replacement group type is needed. +Extend the existing result, event, and group surfaces with cost metadata as +described below. + The intended consumer workflow is: ```go @@ -118,14 +123,90 @@ arbitrary Go function values. Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing terminal-event deduplication. Selecting the same group through multiple paths must not inflate its totals. Terminal usage is an aggregate, not an additional -contribution to be added to attempt usage. +contribution to be added to attempt usage. These rules apply to cost as well as +token counts. + +## Optional Provider-Reported Cost + +### Initial Provider Support + +Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter +defines as the amount charged to the account, denominated in USD. See its +[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting) +and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/). +Do not add `cost_details` breakdowns or combine upstream inference cost with the +top-level charge; those describe different facts and could double-count cost. + +For the built-in client, select this interpretation for the resolved +`openrouter` backend ID. Do not infer USD semantics from an arbitrary backend's +field named `cost`, or add endpoint discovery or verification. An endpoint +override on that backend continues to select its OpenRouter response contract. +Other built-in backend selections report unknown cost initially. Additional +provider adapters or cost-capability configuration are not needed for this scope. + +Use the cost included in the normal completion response. No additional usage +lookup, pricing request, or opt-in provider request parameter is needed. + +### Monetary Representation + +Use `float64` amounts in USD. Make the unit explicit in public field names such +as `CostUSD`, rather than introducing a generic amount with an implicit currency. +At the individual model-client response boundary, use an optional amount +(`*float64`): nil means unknown, and a pointer to zero means reported zero. +Injected clients may supply the same optional USD value without needing an +OpenRouter backend; no provider-specific interface is needed. + +Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields +remain token counts. For run results, execution events, and group summaries, +use a small presence-aware cost value with `Value float64`, `Known bool`, and +`Complete bool`, following the existing observed-count conventions. Successful +run results should expose cost even when no observer was registered; observers +also provide per-attempt and terminal cost, including available facts on failure. + +Binary floating point is sufficient for operational telemetry and keeps decoding, +addition, and the public API lightweight. Do not round to cents or another fixed +scale during capture or aggregation; consumers choose display precision. +Decimal strings or a decimal arithmetic dependency would be appropriate for +exact financial ledgers, while fixed-scale integers would require a selected +precision and rounding policy. Neither is warranted for this reporting feature. +The API does not promise exact decimal arithmetic or invoice reconciliation. + +### Presence, Aggregation, And Failures + +- An absent or null cost is unknown. Explicit numeric zero is known zero. + A backend that reports no cost remains unknown even when a consumer knows it + has no external charge. +- On a dispatched attempt, a usable reported cost is known and complete for + that attempt. Cost completeness is independent of token-field completeness. +- Sum known costs across dispatched attempts, including initial generations, + retries, and repairs. `Known` means at least one attempt reported cost; + `Complete` means every dispatched attempt did. Unknown values contribute no + amount, but make the total incomplete. With no dispatched attempts, follow + the existing known, complete zero convention. +- Group summaries reuse existing terminal-only aggregation and deduplication: + sum known terminal values, OR their `Known` flags, and AND their `Complete` + flags. Retain existing empty-group and terminal-coverage semantics. A subtotal + from OpenRouter mixed with unreported local-backend cost is incomplete. +- Keep valid cost observed from a decoded successful envelope when a later + choice/content check fails, as with token usage. Invalid framing and non-2xx + error bodies continue to supply no usage. Cancellation or failure may leave + incomplete reporting even when billable upstream work occurred. +- Treat an unusable optional cost value, including a nonnumeric value or one + not representable as a finite `float64`, as unknown rather than making an + otherwise valid generation fail. This does not relax existing response framing + or content validation. An injected client likewise cannot supply a nonfinite + usable cost. + +Completeness describes coverage of reported values, not verification of the +provider's billing. Preserve reported usable amounts without pricing inference +or reconciliation. ## Accounting And Responsibility Boundaries -Reuse existing events, usage presence/completeness, and group summaries. Available -usage remains observable after an error or cancellation under the existing -execution-observation behavior; this does not promise that upstream providers -report usage for every attempt. +Reuse existing events, token presence/completeness, and group summaries, extending +them with the cost facts above. Available usage remains observable after an error +or cancellation under the existing execution-observation behavior; this does not +promise that upstream providers report usage for every attempt. PromptKit owns observer selection, event generation, and aggregation. CommandKit and other consumers own invocation lifecycle, CLI flags, presentation, output @@ -133,18 +214,22 @@ delivery, and interpretation. This feature adds no CommandKit dependency or application-specific concepts to PromptKit. Smoke diagnostics retain their separate report/evidence model and are not -enrolled in ordinary execution observation by this change. Additional usage -fields such as reasoning tokens and monetary cost, pricing estimates, billing -reconciliation, persistent history, and logging infrastructure are outside this -scope. +enrolled in ordinary execution observation by this change. Reasoning-token +capture, additional built-in cost providers, multiple currencies, currency +conversion, pricing estimates, billing reconciliation, persistent history, and +logging infrastructure are outside this scope. ## Work Needed And Completion Criteria - Add the context helper and connect observer selection to ordinary and prepared execution, keeping contextual selection separate from private run state. - Preserve existing explicit selection and coordinator behavior while applying - the composition and lifetime rules above. Keep existing execution behavior - unchanged when no contextual observer is selected. + the composition and lifetime rules above. Preserve existing observer selection + when no contextual observer is selected. +- Carry optional OpenRouter cost through response decoding, internal/public + conversions, successful run results, attempt and terminal events, and existing + group summaries. Keep result cost available without registering an observer. + Preserve existing token fields and accounting. - Cover ordinary and prepared execution, coordinated continuations, nested calls using derived contexts, concurrent calls, multiple engines sharing a scope, and independent scopes on a reused engine. @@ -154,9 +239,16 @@ scope. - Verify aggregate accounting through retries and repairs, available usage after failures or cancellation, and known zero versus unknown or incomplete counts. Use offline fixtures and injected clients; live inference is unnecessary. +- Cover fractional USD values, explicit zero, omitted/null and unusable cost, + non-OpenRouter built-in selection, optional injected-client cost, mixed known + and unknown totals, repair/retry aggregation, and duplicate group registration. + Use tolerant comparisons where floating-point addition requires them; do not + turn telemetry into an exact-decimal guarantee. - Document the exact helper contract in GoDoc and add a focused invocation-level - aggregation example to the consumer observation guide. Update internal - observation documentation where its selection flow changes. + aggregation example to the consumer observation guide, including cost + availability, units, and completeness. Document OpenRouter cost decoding in + the [integration contract](../integrations/openai-compatible-chat.md#response-handling), + and update internal observation and model-client documentation for changed flows. This roadmap defines the target behavior. A separate implementation plan can divide the work into appropriately sized stages.