Files
promptkit/docs/roadmap/invocation-scoped-token-usage.md

27 KiB

Invocation-Scoped Execution Observation

Status: accepted scope; implementation complete. Current behavior is documented in the linked canonical contracts; roadmap retirement is separate housekeeping.

Purpose And Scope

Let consumers select observation once for an application-owned operation and obtain usage across its nested, concurrent, coordinated, and multi-engine PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a token cache-hit ratio, and optional USD cost with its source and completeness.

CommandKit's optional ckprompt integration motivates this work, but the API remains application-neutral. PromptKit owns observation, provider response interpretation, and aggregation. Consumers own context propagation, waiting for their work, invocation lifecycle, presentation, and interpretation.

This document defines the target state and the changes required to reach it. A separate implementation plan will own sequencing and prompt-sized stages.

Current Baseline

PromptKit v0.24.0 already provides execution observers and ordered composition, a concurrency-safe execution group, and request-level observation. Groups aggregate the first terminal event for each observed run. Existing observation distinguishes known, unknown, and incomplete prompt, completion, total, cached, and cache-write token counts across retries and repairs.

Today, consumers explicitly enroll requests. Prepared handles retain the request observer; coordinator observers select continuation runs. There is no contextual selection, reasoning-token count, cache-hit helper, or monetary cost reporting. OpenRouter cache writes also need their documented nested mapping: the current client reads usage.cache_write_tokens instead.

Contextual Observer Selection

Add the public helper:

func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context

The returned context composes the supplied observer with any inherited contextual observers. A nil observer leaves that selection unchanged. Executions using the context, or derived contexts retaining its values, participate automatically; unrelated contexts do not. No engine option or process-wide default is needed.

A consumer can create an invocation-owned group, associate group.Observe with the context, run its pipeline, and snapshot the group after its owned work finishes, including when the pipeline returns an error. Group coverage describes observed members, not future work or overall application progress.

Delivery And Execution Boundaries

  • Run selects contextual observers from its invocation context.
  • RunPrepared selects them from its execution context, not the context used to prepare the handle. The explicitly supplied request observer remains retained by the handle and composes with those execution-time observers.
  • Preparation emits no execution events. Discarded handles and rejected execution claims retain their existing observation behavior.
  • Coordinated continuations acquire contextual observation through their ordinary Run boundary. CoordinateRequest.Observer still selects continuation runs only; additional calls made by application callbacks participate when they use the scoped context.
  • Every execution, including a nested one using a callback or injected-client context, has its own identity, recorder, and totals. Clearing an enclosing execution's private state at a public boundary preserves contextual observer selection, not the enclosing recorder.

Deliver events to explicit request observers first, then any coordinator observer already composed with them, then contextual observers from inherited to newly registered. Incorporate contextual selection once per execution, not again in forwarding or coordination layers.

Preserve synchronous callback delivery and panic behavior. Concurrent runs may invoke a shared observer concurrently; consumers synchronize their callback state. The existing group remains concurrency-safe.

Registering the same callback through multiple paths may deliver duplicate events; arbitrary Go functions are not deduplicated. Retain one lifecycle per execution and reuse the group's terminal-event deduplication. Terminal totals are aggregates, not additional contributions to add to attempt totals.

Backend Provider And Accounting Metadata

Add two optional backend fields:

Field Meaning
provider Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor.
zero_cost Enables a configured-zero cost fallback when no usable reported cost is available.

For example:

promptkit:
  backends:
    - id: research-router
      provider: openrouter
      endpoint: https://openrouter.ai/api/v1
      api_key_env: OPENROUTER_API_KEY
    - id: local-inference
      provider: vllm
      endpoint: http://localhost:8000/v1
      zero_cost: true

Trim and lowercase the final effective provider value. Omitted, null, empty, and whitespace-only values select openai-compatible. Reject unsupported nonempty normalized values as ordinary configuration errors. The supported selections are openai-compatible, openrouter, and vllm.

zero_cost is Boolean. Omitted, null, or false disables its fallback; true enables it. It is independent of provider selection and does not suppress a usable reported cost.

Resolve both fields through ordinary final ConfigKit composition and inheritance. Carry their effective selections through capture and prepared execution without reloading configuration. Declare provider: openrouter in the bundled OpenRouter backend. Do not automatically declare any backend zero-cost.

These fields are local metadata, not outbound request parameters or entries added to extra_params. Provider selection does not choose an endpoint or model, and endpoint overrides retain the selected provider and accounting policy.

Inspection Boundary

InspectBackend shows the normalized/resolved declared provider and zero-cost setting, preserving explicit false for zero_cost. Omitted/null declarations may remain absent, and blank provider values need no extra history representation. Do not manufacture an explicit openai-compatible declaration or false zero_cost value from runtime defaults.

InspectConfiguration owns ConfigKit source history and explicit/null/omitted distinctions. Resource inspection does not acquire source-history fields or null-tracking machinery. Provider diagnostic defaults likewise remain runtime fallbacks, not values written into loaded configuration or recorded as explicit source declarations.

Provider Conventions

All selections reuse the existing OpenAI-compatible text Chat Completions foundation: base-relative /chat/completions JSON POST requests, existing credential rules, text messages, selected generation controls and structured output, extra parameters, first-choice content extraction, common token fields, and existing response validation, errors, cancellation, timeouts, and transient retries.

Provider Additional response interpretation Default authentication adapter
openai-compatible Existing token mappings; no provider-reported monetary cost decoding. None.
openrouter OpenRouter reasoning/cache details and usage.cost in USD. openrouter_key.
vllm vLLM reasoning/cache details; no provider-reported monetary cost decoding. vllm_models.

The default provider names PromptKit's current supported conventions, not the entire OpenAI API. "Lowest common denominator" does not strip existing optional features or promise every backend supports them. Continue sending selected supported settings and return backend errors normally.

Provider selection supplies no default endpoint, model, credential environment variable, or sampling settings, and does not create backend definitions. Keep vLLM reasoning, chat-template, and other model-specific controls caller-selected; do not automatically enable thinking or translate those controls. No new generation protocol, capability negotiation, or version discovery is needed. vLLM documents its OpenAI-compatible interface.

Authentication Diagnostics

Provider defaults apply only to explicitly requested authentication diagnostics. Loading configuration, constructing an engine, and ordinary generation do not initiate probes. Existing reachability and inference-default behavior remains.

Explicit smoke_tests.authentication metadata overrides provider defaults:

  • A supplied kind selects the adapter; none declares authentication testing inapplicable.
  • Omitted/null kind uses the provider default, allowing endpoint-only overrides.
  • A supplied endpoint overrides the selected adapter's destination; omitted/null uses that adapter's existing fallback.
  • An absent/null authentication block removes its overrides and leaves provider defaults applicable. kind: none, not clearing the block, suppresses them.

Without a provider default or explicit adapter, preserve the existing unsupported diagnostic outcome rather than assuming authentication is inapplicable. Explicit adapters remain selectable without a matching provider declaration.

Retain openrouter_key behavior. The new vllm_models adapter makes one GET to /models relative to the configured API base, supplying an available credential under existing rules and allowing dispatch without one. A base ending in /v1 therefore produces /v1/models; explicit diagnostic endpoint overrides apply.

For vllm_models, HTTP 200 with a valid JSON object containing object: "list" and array-valued data means acceptance of the probe request. An empty array is accepted; recognition does not inspect model capabilities or require a particular model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown. Reuse existing bounded diagnostic decoding and framing. Report observed credential presence and response facts without claiming authentication is enforced or models-list acceptance establishes inference authorization.

vLLM documents /v1/models as protected when API-key authentication is configured; /health is not an authentication check. See its authentication documentation. No health-monitoring facility is added here.

Token And Cache Reporting

Provider Response Mapping

Use common usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens for PromptTokens, CompletionTokens, and TotalTokens. Normalize these additional provider details, relative to usage:

Public field OpenRouter vLLM
New ReasoningTokens completion_tokens_details.reasoning_tokens completion_tokens_details.reasoning_tokens
Existing CachedTokens prompt_tokens_details.cached_tokens prompt_tokens_details.cached_tokens
Existing CacheWriteTokens prompt_tokens_details.cache_write_tokens prompt_tokens_details.created_cache_tokens

OpenRouter documents these in its response schema, usage-accounting guide, and caching guide. vLLM's response construction uses created_cache_tokens; its output definitions describe local prefix-cache writes, not a billable write charge.

Preserve CachedTokens as the public name and clarify its meaning as cache reads. Use each provider's documented cache-write field, not a sum or guessed alias; a missing nested value is unknown even if an alternative field is present. Retain the generic provider's existing mappings, including its top-level usage.cache_write_tokens handling.

Add ReasoningTokens to public/domain token values, presence indicators, observed usage, conversions, and result serialization, and carry it through attempt events, terminal events, and group summaries. Injected clients may supply canonical counts and presence through the existing model-client boundary, retaining its legacy UsagePresence fallback; explicit presence distinguishes measured zero.

Availability And Optional Metadata

Omitted/null details are unknown, not zero. Explicit integer zero is known zero. For the newly interpreted provider-specific reasoning and cache-write details, wrong-type, nonintegral, or unrepresentable integer values are also unknown. An unusable new detail does not invalidate otherwise usable content or erase independently available usage. Preserve existing common-field decoding and response framing/content validation; this is a targeted optional-metadata rule, not a general relaxation of the response contract. Keep this interpretation local to the model-client boundary without a general numeric-conversion framework.

vLLM detail availability depends on version and configuration: prompt details currently require --enable-prompt-tokens-details, and reasoning details depend on a configured reasoning parser. Document those reporting settings without configuring or discovering them automatically. Older or differently configured servers may omit details and still produce usable responses. Do not estimate reasoning counts from text or cache writes from input minus cache reads.

Count Relationships And Aggregation

Reasoning is included in completion tokens, not additional output to add to them. OpenRouter explains this in its reasoning documentation; vLLM similarly includes reasoning detail within generated-token usage. Cache reads and writes are input-token details, not extra input consumption. Preserve the reported total without recomputation or reconciliation.

Consumers can derive non-reasoning output from consistent, known completion and reasoning counts; no additional stored count is needed. Capturing counts does not capture reasoning text or change generation/caching settings.

Reuse existing per-field token accounting: aggregates sum known contributions, Known identifies available reporting, and Complete describes its coverage across dispatched attempts. Preserve initial-generation, retry, and repair observation accounting and terminal-only group deduplication. Existing integer result usage remains distinct from presence-aware observations; a zero result field alone does not establish that the provider reported it.

Token Cache-Hit Ratio

Add a derived helper, not another stored total:

func (u ObservedTokenUsage) CacheHitRatio() (float64, bool)

Compute CachedTokens / PromptTokens as a fraction from zero to one. This is the proportion of input tokens served from cache, not request hit frequency or cost savings. For aggregates, divide summed reads by summed input; do not average individual percentages or use reads / (reads + writes).

Return an available ratio only when input and cache-read counts are known and complete, input is positive, and reads lie between zero and input inclusive. Known zero reads with positive input yields an available zero. Missing, incomplete, zero-denominator, or inconsistent data yields zero and false while preserving raw counts; it does not fail generation or clamp reported values. Cache-write availability is irrelevant.

The same helper serves attempts, terminals, and group summaries. A group ratio describes its included observations and does not supersede terminal coverage or imply all application work has finished.

Cost Reporting

Reported Cost And Configured-Zero Precedence

Initially, only the OpenRouter built-in provider interprets usage.cost as the account charge in USD. See its usage-accounting reference and USD explanation. Do not add cost_details amounts to that charge or interpret arbitrary providers' fields named cost. The generic and vLLM providers do not decode monetary cost.

Use the cost included in the ordinary response; no extra request or usage lookup is needed. At the model-client boundary, represent an optional USD amount as *float64: nil means unknown; a pointer to zero means reported zero. Injected clients may supply the same value regardless of provider selection.

Resolve cost at completion of each dispatched attempt, outside provider-specific decoding and whether or not decoding occurred or succeeded:

  • A usable provider/model-client-reported cost takes precedence, including zero.
  • Otherwise, zero_cost: true on the effective backend supplies configured USD zero.
  • Otherwise, cost remains unknown.

This applies to built-in and injected clients, including failed or canceled dispatched attempts and undecodable responses. Without a selected backend declaration, no configured-zero fallback applies. Non-dispatched work contributes no cost. Do not infer zero cost from local hosting, provider name, credentials, or token counts, or install the declaration automatically on bundled backends.

An unusable reported amount, including a nonnumeric value or one not representable as a finite float64, is unknown before fallback; it does not make an otherwise usable generation fail. Keep available cost from a decoded successful envelope when a later choice/content check fails. Invalid framing and non-2xx bodies supply no provider usage, although the configured-zero fallback still applies to a dispatched attempt. Existing framing, content, and error handling remain intact.

Monetary Representation And Sources

Expose cost separately from token counts and their presence indicators. Use explicit USD field naming such as CostUSD and a small public cost value for results, events, and group summaries with:

  • Value float64: the available USD amount or subtotal;
  • Known bool: availability;
  • Complete bool: coverage under the selected cost policy; and
  • Source: the contribution-source discriminator below.
Source Meaning
provider Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification.
configured_zero Supplied by the explicit backend fallback.
mixed An aggregate includes both reported and configured-zero contributions.
Empty No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts.

Use binary floating point for operational telemetry, without rounding to cents or another fixed scale during capture or aggregation. Consumers choose display precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency rounding, pricing inference, and invoice reconciliation are not part of this work.

Attempt, Run, And Group Accounting

A dispatched attempt with usable reported cost or configured zero is known and complete for cost, independently of token completeness. Sum known amounts across all dispatched attempts, including retries and repairs. Aggregate Known means at least one known contribution; Complete means every dispatched attempt has a cost from either source. Unknown contributions add no amount and make coverage incomplete. Completeness does not assert that every provider reported a charge.

Aggregate sources by union of known contributions, including zero amounts. Unknown contributions add no source: a provider-only subtotal can be incomplete without becoming mixed. A reported zero plus a configured zero is mixed.

Successful results expose cost even without an observer. Result and terminal cost agree on amount, availability, completeness, and source, and enabling observation does not change result cost. For example, an unreported failed attempt followed by a reported USD 0.002 yields an incomplete provider-source subtotal when no zero-cost fallback applies.

Groups aggregate terminal costs only, counting each run's first terminal once. Combine known amounts, OR Known, AND Complete, and union sources. Preserve existing empty/no-terminal group semantics: a completed execution with no dispatched attempts yields known, complete zero with empty source; an empty group does likewise, while a group with members but no terminals has unknown, incomplete usage.

Cost completeness concerns included terminal observations, independently of TerminalCoverageComplete. Two completed runs with full cost coverage can yield a complete subtotal while a third observed run lacks a terminal. Conversely, fully reported OpenRouter runs and explicitly zero-cost local runs can yield a complete, mixed-source total. Neither case verifies a provider invoice or future application progress.

Internal Responsibilities And Scope Limits

Provider-specific metadata interpretation and authentication conventions belong in internal/llm, sharing common transport, request construction, credentials, framing, and response fields. Use one provider-selection mechanism there. Generation/observation coordination consumes normalized facts or generic diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator method with a generic operation while retaining explicit adapter selection.

Provider-specific files and straightforward dispatch suffice; no public provider registration API, new internal/providers package, or broad mandatory-capability interface is needed. This is configuration-driven selection among supported implementations, not downstream registration of implementations.

Reuse a small shared cost-accounting component for result and observation paths so dispatched attempts, fallback, and source union follow the same rules even when observation is disabled. Provider code interprets wire metadata; common accounting owns fallback and aggregation. No ConfigKit provenance traversal belongs in ordinary execution.

Smoke diagnostics retain their separate report/evidence model and are not enrolled in ordinary execution observation. CLI integration, output delivery, persistent history, logging infrastructure, reasoning-text capture, changes to reasoning/caching request behavior, server administration/metrics, additional built-in cost decoders, multiple currencies, configurable token rates, fixed nonzero request prices, and cost estimation remain outside scope.

Work Needed And Acceptance

The following areas describe required changes, not an execution order.

  • Extend the public context API and ordinary/prepared execution wiring while preserving independent nested lifecycles, explicit observers, coordinator selection, and existing preparation/discard/rejected-claim behavior.
  • Extend backend settings, typed helpers, final-value normalization/validation, capture, prepared metadata, and inspection for provider and zero_cost. Update the bundled OpenRouter declaration. Preserve configuration provenance and resource-inspection simplicity, and keep metadata out of request bodies.
  • Consolidate provider behavior in internal/llm, adding vLLM usage normalization and the models-list diagnostic alongside OpenRouter interpretation and the shared default path. Preserve common decoding while tolerating unusable new optional details and cost locally to the model-client boundary.
  • Extend public/domain token types, presence, serialization, conversions, events, and groups with reasoning counts and correct provider-specific cache writes. Add the presence-aware cache-hit helper without changing existing token meanings or introducing a separate ratio accumulator.
  • Carry optional reported cost and configured-zero policy through attempt completion and shared accounting into results, events, and groups, including source metadata and observer-independent result accounting.

Select focused offline checks under the testing policy. Use real internal collaborators and controlled HTTP/injected-client boundaries, with proportionate public contract coverage. Important behaviors include:

  • Context inheritance, registration order, nil/duplicate enrollment, concurrent calls, multiple engines, independent scopes, nested calls, and execution-time selection for a handle prepared elsewhere.
  • Provider selection independent of backend identity; normalization and unsupported value handling; omitted/null/blank defaults; inheritance, endpoint overrides, explicit false, and inspection without manufactured values or source history.
  • Provider-default and explicit diagnostic selection, endpoint-only overrides, clearing metadata versus kind: none, vLLM acceptance including an empty list, rejection, unrecognized/framing-limited responses, and optional credentials.
  • Provider token mappings, absent/null/zero details, malformed new optional counts without generation failure, unchanged common-field handling, and responses representative of vLLM servers with detail reporting disabled.
  • Cache-hit availability, inconsistent counts without generation failure, token-weighted aggregates, independence from cache writes, and no double-counting of reasoning or cache details.
  • Reported cost precedence, configured-zero fallback on dispatched failure and cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources, zero-valued contributions, and independently incomplete token reporting.
  • Result/terminal cost agreement with observation enabled or disabled, duplicate terminal deduplication, empty/no-terminal groups, and subtotal completeness distinct from terminal coverage.

Synthetic fixtures and offline examples suffice; live providers are not required. Keep ownership localized and avoid repeating the same assertions across layers.

Documentation And Migration

When implemented, exact exported contracts belong in GoDoc; the consumer guide should show invocation-scoped aggregation, reasoning/cache breakdowns without double-counting, and cost availability, source, and completeness. Document provider wire mappings, optional-metadata handling, and diagnostic response recognition in the integration contract. The framework format reference owns the new backend fields, default selection, inheritance, and diagnostic overrides. Update internal documentation for changed ownership and flows.

Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache creation meaning without promising availability on every deployment. Consumers own presentation. Models-list acceptance does not establish inference authorization, and complete cost metadata does not independently verify billing.

Release notes should call out that authentication objects no longer require kind when a provider default supplies it, endpoint-only overrides are supported, and clearing an authentication block restores provider defaults. kind: none suppresses that default. Preserve existing explicit adapters without requiring a provider declaration.