Add optional OpenRouter cost reporting to the observation roadmap
This commit is contained in:
@@ -7,7 +7,8 @@ Status: accepted scope; implementation pending.
|
||||
Allow consumers to select execution observation once for an application-owned
|
||||
operation, rather than attach an observer to every PromptKit request. Nested,
|
||||
concurrent, and multi-engine calls using that operation's context should
|
||||
participate automatically.
|
||||
participate automatically. Include optional provider-reported cost alongside
|
||||
token usage, with built-in cost support initially limited to OpenRouter.
|
||||
|
||||
The motivating consumer is CommandKit's optional `ckprompt` integration, which
|
||||
could use this facility to report token usage for an entire command or pipeline.
|
||||
@@ -28,8 +29,9 @@ PromptKit v0.24.0 already provides:
|
||||
execution, and [CoordinateRequest.Observer](../../coordinate.go) for
|
||||
coordinator-selected continuation runs.
|
||||
|
||||
The missing capability is contextual observer selection. Applications currently
|
||||
have to enroll each request explicitly. The existing
|
||||
The missing capabilities are contextual observer selection and provider-reported
|
||||
cost accounting. Applications currently have to enroll each request explicitly,
|
||||
and PromptKit does not expose provider charges. The existing
|
||||
[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run)
|
||||
describes that workflow.
|
||||
|
||||
@@ -46,7 +48,10 @@ or derived contexts that retain its values. It composes with any observer alread
|
||||
associated with the context; it does not replace the inherited observer. A nil
|
||||
observer leaves the inherited selection unchanged.
|
||||
|
||||
No engine option, process-wide default, or new aggregation type is needed.
|
||||
No engine option, process-wide default, or replacement group type is needed.
|
||||
Extend the existing result, event, and group surfaces with cost metadata as
|
||||
described below.
|
||||
|
||||
The intended consumer workflow is:
|
||||
|
||||
```go
|
||||
@@ -118,14 +123,90 @@ arbitrary Go function values.
|
||||
Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing
|
||||
terminal-event deduplication. Selecting the same group through multiple paths
|
||||
must not inflate its totals. Terminal usage is an aggregate, not an additional
|
||||
contribution to be added to attempt usage.
|
||||
contribution to be added to attempt usage. These rules apply to cost as well as
|
||||
token counts.
|
||||
|
||||
## Optional Provider-Reported Cost
|
||||
|
||||
### Initial Provider Support
|
||||
|
||||
Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter
|
||||
defines as the amount charged to the account, denominated in USD. See its
|
||||
[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting)
|
||||
and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/).
|
||||
Do not add `cost_details` breakdowns or combine upstream inference cost with the
|
||||
top-level charge; those describe different facts and could double-count cost.
|
||||
|
||||
For the built-in client, select this interpretation for the resolved
|
||||
`openrouter` backend ID. Do not infer USD semantics from an arbitrary backend's
|
||||
field named `cost`, or add endpoint discovery or verification. An endpoint
|
||||
override on that backend continues to select its OpenRouter response contract.
|
||||
Other built-in backend selections report unknown cost initially. Additional
|
||||
provider adapters or cost-capability configuration are not needed for this scope.
|
||||
|
||||
Use the cost included in the normal completion response. No additional usage
|
||||
lookup, pricing request, or opt-in provider request parameter is needed.
|
||||
|
||||
### Monetary Representation
|
||||
|
||||
Use `float64` amounts in USD. Make the unit explicit in public field names such
|
||||
as `CostUSD`, rather than introducing a generic amount with an implicit currency.
|
||||
At the individual model-client response boundary, use an optional amount
|
||||
(`*float64`): nil means unknown, and a pointer to zero means reported zero.
|
||||
Injected clients may supply the same optional USD value without needing an
|
||||
OpenRouter backend; no provider-specific interface is needed.
|
||||
|
||||
Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields
|
||||
remain token counts. For run results, execution events, and group summaries,
|
||||
use a small presence-aware cost value with `Value float64`, `Known bool`, and
|
||||
`Complete bool`, following the existing observed-count conventions. Successful
|
||||
run results should expose cost even when no observer was registered; observers
|
||||
also provide per-attempt and terminal cost, including available facts on failure.
|
||||
|
||||
Binary floating point is sufficient for operational telemetry and keeps decoding,
|
||||
addition, and the public API lightweight. Do not round to cents or another fixed
|
||||
scale during capture or aggregation; consumers choose display precision.
|
||||
Decimal strings or a decimal arithmetic dependency would be appropriate for
|
||||
exact financial ledgers, while fixed-scale integers would require a selected
|
||||
precision and rounding policy. Neither is warranted for this reporting feature.
|
||||
The API does not promise exact decimal arithmetic or invoice reconciliation.
|
||||
|
||||
### Presence, Aggregation, And Failures
|
||||
|
||||
- An absent or null cost is unknown. Explicit numeric zero is known zero.
|
||||
A backend that reports no cost remains unknown even when a consumer knows it
|
||||
has no external charge.
|
||||
- On a dispatched attempt, a usable reported cost is known and complete for
|
||||
that attempt. Cost completeness is independent of token-field completeness.
|
||||
- Sum known costs across dispatched attempts, including initial generations,
|
||||
retries, and repairs. `Known` means at least one attempt reported cost;
|
||||
`Complete` means every dispatched attempt did. Unknown values contribute no
|
||||
amount, but make the total incomplete. With no dispatched attempts, follow
|
||||
the existing known, complete zero convention.
|
||||
- Group summaries reuse existing terminal-only aggregation and deduplication:
|
||||
sum known terminal values, OR their `Known` flags, and AND their `Complete`
|
||||
flags. Retain existing empty-group and terminal-coverage semantics. A subtotal
|
||||
from OpenRouter mixed with unreported local-backend cost is incomplete.
|
||||
- Keep valid cost observed from a decoded successful envelope when a later
|
||||
choice/content check fails, as with token usage. Invalid framing and non-2xx
|
||||
error bodies continue to supply no usage. Cancellation or failure may leave
|
||||
incomplete reporting even when billable upstream work occurred.
|
||||
- Treat an unusable optional cost value, including a nonnumeric value or one
|
||||
not representable as a finite `float64`, as unknown rather than making an
|
||||
otherwise valid generation fail. This does not relax existing response framing
|
||||
or content validation. An injected client likewise cannot supply a nonfinite
|
||||
usable cost.
|
||||
|
||||
Completeness describes coverage of reported values, not verification of the
|
||||
provider's billing. Preserve reported usable amounts without pricing inference
|
||||
or reconciliation.
|
||||
|
||||
## Accounting And Responsibility Boundaries
|
||||
|
||||
Reuse existing events, usage presence/completeness, and group summaries. Available
|
||||
usage remains observable after an error or cancellation under the existing
|
||||
execution-observation behavior; this does not promise that upstream providers
|
||||
report usage for every attempt.
|
||||
Reuse existing events, token presence/completeness, and group summaries, extending
|
||||
them with the cost facts above. Available usage remains observable after an error
|
||||
or cancellation under the existing execution-observation behavior; this does not
|
||||
promise that upstream providers report usage for every attempt.
|
||||
|
||||
PromptKit owns observer selection, event generation, and aggregation. CommandKit
|
||||
and other consumers own invocation lifecycle, CLI flags, presentation, output
|
||||
@@ -133,18 +214,22 @@ delivery, and interpretation. This feature adds no CommandKit dependency or
|
||||
application-specific concepts to PromptKit.
|
||||
|
||||
Smoke diagnostics retain their separate report/evidence model and are not
|
||||
enrolled in ordinary execution observation by this change. Additional usage
|
||||
fields such as reasoning tokens and monetary cost, pricing estimates, billing
|
||||
reconciliation, persistent history, and logging infrastructure are outside this
|
||||
scope.
|
||||
enrolled in ordinary execution observation by this change. Reasoning-token
|
||||
capture, additional built-in cost providers, multiple currencies, currency
|
||||
conversion, pricing estimates, billing reconciliation, persistent history, and
|
||||
logging infrastructure are outside this scope.
|
||||
|
||||
## Work Needed And Completion Criteria
|
||||
|
||||
- Add the context helper and connect observer selection to ordinary and prepared
|
||||
execution, keeping contextual selection separate from private run state.
|
||||
- Preserve existing explicit selection and coordinator behavior while applying
|
||||
the composition and lifetime rules above. Keep existing execution behavior
|
||||
unchanged when no contextual observer is selected.
|
||||
the composition and lifetime rules above. Preserve existing observer selection
|
||||
when no contextual observer is selected.
|
||||
- Carry optional OpenRouter cost through response decoding, internal/public
|
||||
conversions, successful run results, attempt and terminal events, and existing
|
||||
group summaries. Keep result cost available without registering an observer.
|
||||
Preserve existing token fields and accounting.
|
||||
- Cover ordinary and prepared execution, coordinated continuations, nested calls
|
||||
using derived contexts, concurrent calls, multiple engines sharing a scope,
|
||||
and independent scopes on a reused engine.
|
||||
@@ -154,9 +239,16 @@ scope.
|
||||
- Verify aggregate accounting through retries and repairs, available usage after
|
||||
failures or cancellation, and known zero versus unknown or incomplete counts.
|
||||
Use offline fixtures and injected clients; live inference is unnecessary.
|
||||
- Cover fractional USD values, explicit zero, omitted/null and unusable cost,
|
||||
non-OpenRouter built-in selection, optional injected-client cost, mixed known
|
||||
and unknown totals, repair/retry aggregation, and duplicate group registration.
|
||||
Use tolerant comparisons where floating-point addition requires them; do not
|
||||
turn telemetry into an exact-decimal guarantee.
|
||||
- Document the exact helper contract in GoDoc and add a focused invocation-level
|
||||
aggregation example to the consumer observation guide. Update internal
|
||||
observation documentation where its selection flow changes.
|
||||
aggregation example to the consumer observation guide, including cost
|
||||
availability, units, and completeness. Document OpenRouter cost decoding in
|
||||
the [integration contract](../integrations/openai-compatible-chat.md#response-handling),
|
||||
and update internal observation and model-client documentation for changed flows.
|
||||
|
||||
This roadmap defines the target behavior. A separate implementation plan can
|
||||
divide the work into appropriately sized stages.
|
||||
|
||||
Reference in New Issue
Block a user