From a47df4ec20ab7516e6182e5618728a10b37d83c0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 17:52:58 +0000 Subject: [PATCH] Consolidate the observation roadmap around the agreed target state --- docs/roadmap/invocation-scoped-token-usage.md | 1115 +++++++---------- 1 file changed, 423 insertions(+), 692 deletions(-) diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md index b3f81fc..08a7086 100644 --- a/docs/roadmap/invocation-scoped-token-usage.md +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -2,145 +2,96 @@ Status: accepted scope; implementation pending. -## Purpose +## Purpose And Scope -Allow consumers to select execution observation once for an application-owned -operation, rather than attach an observer to every PromptKit request. Nested, -concurrent, and multi-engine calls using that operation's context should -participate automatically. Include optional provider-reported cost alongside -token usage, with built-in cost decoding initially limited to OpenRouter and an -explicit backend zero-cost fallback. Distinguish reported and configured cost -sources in individual and aggregate metadata. Extend OpenRouter and vLLM token -reporting with reasoning counts, documented cache reads and writes, and a shared -derived token cache-hit ratio. +Let consumers select observation once for an application-owned operation and +obtain usage across its nested, concurrent, coordinated, and multi-engine +PromptKit calls. Extend that reporting with reasoning tokens, cache activity, a +token cache-hit ratio, and optional USD cost with its source and completeness. -The motivating consumer is CommandKit's optional `ckprompt` integration, which -could use this facility to report token usage for an entire command or pipeline. -PromptKit supplies application-neutral observation and aggregation; consumers -choose how to present or use the results. +CommandKit's optional `ckprompt` integration motivates this work, but the API +remains application-neutral. PromptKit owns observation, provider response +interpretation, and aggregation. Consumers own context propagation, waiting for +their work, invocation lifecycle, presentation, and interpretation. + +This document defines the target state and the changes required to reach it. +A separate implementation plan will own sequencing and prompt-sized stages. ## Current Baseline -PromptKit v0.24.0 already provides: +PromptKit v0.24.0 already provides +[execution observers and ordered composition](../../observations.go), +a concurrency-safe [execution group](../../execution_group.go), and +[request-level observation](../consumers/pkg-promptkit.md#observe-an-ordinary-run). +Groups aggregate the first terminal event for each observed run. Existing +observation distinguishes known, unknown, and incomplete prompt, completion, +total, cached, and cache-write token counts across retries and repairs. -- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous - metadata callbacks and ordered composition. -- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached - snapshots, and aggregation of the first terminal event for each observed run. -- Presence-aware prompt, completion, total, cached, and cache-write token counts, - including aggregate accounting across retries and repairs. -- Explicit selection through `RunRequest.Observer`, retained for prepared - execution, and [CoordinateRequest.Observer](../../coordinate.go) for - coordinator-selected continuation runs. +Today, consumers explicitly enroll requests. Prepared handles retain the request +observer; [coordinator observers](../../coordinate.go) select continuation runs. +There is no contextual selection, reasoning-token count, cache-hit helper, or +monetary cost reporting. OpenRouter cache writes also need their documented +nested mapping: the current client reads `usage.cache_write_tokens` instead. -The missing capabilities are contextual observer selection, cost accounting and -source metadata, an explicit zero-cost declaration, reasoning-token counts, and -a presence-aware cache-hit helper. -OpenRouter cache writes also need their documented nested response mapping; -the current client reads a top-level `usage.cache_write_tokens` field instead. -Applications currently have to enroll each request explicitly, and PromptKit -does not expose provider charges. The existing -[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) -describes that workflow. +## Contextual Observer Selection -## Target Public API - -Add a public context helper: +Add the public helper: ```go func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context ``` -The returned context associates the observer with executions using that context -or derived contexts that retain its values. It composes with any observer already -associated with the context; it does not replace the inherited observer. A nil -observer leaves the inherited selection unchanged. +The returned context composes the supplied observer with any inherited contextual +observers. A nil observer leaves that selection unchanged. Executions using the +context, or derived contexts retaining its values, participate automatically; +unrelated contexts do not. No engine option or process-wide default is needed. -No engine option, process-wide default, or replacement group type is needed. -Extend the existing token types and result, event, and group surfaces with the -token and cost metadata described below, plus a derived cache-hit helper. +A consumer can create an invocation-owned group, associate `group.Observe` with +the context, run its pipeline, and snapshot the group after its owned work +finishes, including when the pipeline returns an error. Group coverage describes +observed members, not future work or overall application progress. -The intended consumer workflow is: +### Delivery And Execution Boundaries -```go -var group promptkit.ExecutionGroup -ctx = promptkit.WithExecutionObserver(ctx, group.Observe) +- `Run` selects contextual observers from its invocation context. +- `RunPrepared` selects them from its execution context, not the context used + to prepare the handle. The explicitly supplied request observer remains + retained by the handle and composes with those execution-time observers. +- Preparation emits no execution events. Discarded handles and rejected + execution claims retain their existing observation behavior. +- Coordinated continuations acquire contextual observation through their + ordinary `Run` boundary. `CoordinateRequest.Observer` still selects + continuation runs only; additional calls made by application callbacks + participate when they use the scoped context. +- Every execution, including a nested one using a callback or injected-client + context, has its own identity, recorder, and totals. Clearing an enclosing + execution's private state at a public boundary preserves contextual observer + selection, not the enclosing recorder. -err := runApplicationPipeline(ctx) -// After all application-owned work has finished, including on failure: -summary := group.Snapshot() -``` +Deliver events to explicit request observers first, then any coordinator observer +already composed with them, then contextual observers from inherited to newly +registered. Incorporate contextual selection once per execution, not again in +forwarding or coordination layers. -The consumer owns context propagation, waiting for its work, and using the -snapshot and error. Calls using unrelated contexts are not automatically -enrolled. Group coverage continues to describe observed members, not future -work or overall application progress. +Preserve synchronous callback delivery and panic behavior. Concurrent runs may +invoke a shared observer concurrently; consumers synchronize their callback state. +The existing group remains concurrency-safe. -## Observer Selection And Delivery +Registering the same callback through multiple paths may deliver duplicate +events; arbitrary Go functions are not deduplicated. Retain one lifecycle per +execution and reuse the group's terminal-event deduplication. Terminal totals +are aggregates, not additional contributions to add to attempt totals. -### Composition +## Backend Provider And Accounting Metadata -Preserve existing explicit observers. Deliver each event to the explicit request -observer first, then any coordinator observer already composed with it, then -contextual observers in registration order, from inherited to newly added. +Add two optional backend fields: -Preserve the existing synchronous callback and panic behavior. Concurrent -executions may invoke a shared observer concurrently; consumers remain -responsible for synchronizing their callback state. `ExecutionGroup` already -supports concurrent use. +| Field | Meaning | +| --- | --- | +| `provider` | Selects the supported API conventions, independently of backend identity, namespace, endpoint hostname, and model vendor. | +| `zero_cost` | Enables a configured-zero cost fallback when no usable reported cost is available. | -### Ordinary And Nested Execution - -`Run` selects contextual observation from its invocation context. Each execution -has its own observation identity, recorder, and accounting, including a nested -execution using a context passed to an application callback or injected client. - -Inherited observer selection is distinct from private execution state. Existing -clearing of an enclosing execution's private state at a public execution boundary -must preserve the contextual observer selection. Inheriting observation does not -reuse an enclosing run's recorder, totals, or lifecycle. - -### Prepared Execution - -`RunPrepared` selects contextual observation from the context supplied when the -handle is executed. A handle prepared in one scope and executed in another -participates in the execution scope, not the preparation scope. - -The explicit `RunRequest.Observer` remains retained by a successfully prepared -handle as today and composes with the execution-time contextual observers. -Preparation itself emits no execution events. Discarded handles and rejected -execution claims retain their existing observation behavior. - -### Coordinated Execution - -Continuation runs receive contextual observation through their ordinary `Run` -boundary. `CoordinateRequest.Observer` continues to select continuation runs -only. Additional calls made by application callbacks participate when they use -the scoped context, without being automatically selected by the coordinator's -explicit observer. - -Contextual observers are incorporated once per execution, rather than being -re-added by coordination or another forwarding layer. - -### Duplicate Registration And Accounting - -Registering the same callback through multiple explicit or contextual paths may -deliver the same event more than once. Do not attempt to identify or deduplicate -arbitrary Go function values. - -Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing -terminal-event deduplication. Selecting the same group through multiple paths -must not inflate its totals. Terminal usage is an aggregate, not an additional -contribution to be added to attempt usage. These rules apply to cost as well as -token counts. - -## Backend Provider Selection - -Add an optional backend `provider` field identifying the API implementation -whose conventions PromptKit should use. It identifies the service contacted, -not the organization that trained the selected model. Backend IDs remain resource -identities; names, namespaces, and endpoint hostnames do not select provider -behavior. +For example: ```yaml promptkit: @@ -149,617 +100,397 @@ promptkit: provider: openrouter endpoint: https://openrouter.ai/api/v1 api_key_env: OPENROUTER_API_KEY -``` - -Normalize the final effective provider value by trimming whitespace and -lowercasing. Omitted, null, empty, and whitespace-only values are equivalent. -Reject an unsupported nonempty normalized value as an ordinary configuration -error rather than silently using generic behavior. The supported selections for -this feature are `openai-compatible`, `openrouter`, and `vllm`. - -Omitted/null/blank provider values select the same effective behavior as explicit -`provider: openai-compatible`. This is a runtime fallback, not an explicitly -sourced configuration value. Backend inspection retains the distinction between -an explicit provider declaration and an omitted/null/blank declaration. - -The `openrouter` and `vllm` providers reuse the shared OpenAI-compatible foundation -and add their response-metadata and diagnostic conventions below. - -Resolve the field through ordinary ConfigKit composition and inheritance. Carry -the selected value through configuration capture and prepared execution, expose -it in backend inspection, and declare it in the bundled OpenRouter backend. -It is PromptKit metadata, not an outbound `extra_params` entry. Selecting a -provider does not choose a model or alter the configured endpoint. An endpoint -override retains the provider selection. - -### Default OpenAI-Compatible Provider - -`openai-compatible` names PromptKit's current supported text Chat Completions -conventions. Preserve the existing shared behavior documented in the -[integration contract](../integrations/openai-compatible-chat.md), including: - -- JSON POST requests to `/chat/completions` relative to the configured base; -- existing optional bearer-credential handling, including explicitly required - credential settings when selected; -- text messages, explicitly selected generation controls and structured-output - requests, and configured extra parameters; -- first-choice text extraction, existing supported token counts and their - presence-aware observation; and -- existing response validation, error handling, timeouts, cancellation, and - transient retries. - -This selection supplies no default endpoint, model, API-key environment variable, -or sampling settings. It does not decode provider-reported monetary cost or -provide a default authentication-probe adapter. The backend zero-cost declaration -below remains available independently of provider selection. Without an explicit -authentication adapter, report the existing unsupported diagnostic outcome rather -than assuming authentication is inapplicable. Existing reachability and inference -diagnostics and explicit authentication adapters remain available. - -"Lowest common denominator" identifies the shared protocol foundation; it does -not remove existing optional features or promise every backend supports them. -Continue sending caller-selected supported settings and return backend errors -normally, without capability negotiation or fallback requests. This is the -existing text Chat Completions scope, not a claim to support the full OpenAI API. - -### vLLM Provider - -`provider: vllm` selects the shared generation path plus vLLM-specific usage -normalization and a default authentication diagnostic. It supplies no default -endpoint, model, API-key environment variable, or sampling settings, and does not -create a backend definition automatically. - -Preserve caller-selected reasoning settings, structured-output requests, and -extra parameters. Do not automatically change chat-template arguments, enable -thinking, or translate model-specific controls based on the provider label. -No separate generation protocol is needed; vLLM documents an -[OpenAI-compatible Chat Completions API](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/). - -The token mappings are specified below. vLLM supplies no provider-reported cost; -cost remains unknown unless the backend explicitly declares the zero-cost -fallback below. Local hosting alone does not select that fallback. This provider -adds no price estimator, server administration, metrics collection, or automatic -capability/version probe. - -### Smoke-Test Defaults And Explicit Overrides - -When an authentication smoke test is explicitly requested, the selected provider -supplies its default probe convention. `provider: openrouter` supplies the -existing `openrouter_key` convention; `provider: vllm` supplies the `vllm_models` -convention described below. Preserve explicit -`smoke_tests.authentication.kind` and `smoke_tests.authentication.endpoint` -settings as overrides: - -- A supplied kind selects the diagnostic adapter instead of the provider default. - `kind: none` continues to declare authentication testing inapplicable. -- An omitted/null kind uses the provider default, if available. This permits an - endpoint override without repeating the provider's default kind. -- A supplied endpoint overrides the selected adapter's default destination. - An omitted/null endpoint uses that adapter's existing endpoint fallback. - -An absent/null authentication block likewise leaves selection to the provider -default. Clearing the block removes its explicit overrides; `kind: none` is the -explicit way to declare authentication inapplicable despite a provider default. -Without a provider default or explicit adapter, retain existing unsupported -diagnostic behavior. Existing explicit adapters continue to work without a -provider declaration. Reachability and inference-default settings keep their -existing behavior. - -These are conventions for explicitly requested diagnostics. Loading a backend, -constructing an engine, and ordinary generation do not initiate probes. - -Resolve provider-supplied diagnostic defaults at the provider boundary without -writing them into the loaded configuration or recording them as explicit source -values. Configuration inspection preserves the distinction between configured -overrides and provider-supplied fallbacks, including their existing source -provenance. - -### vLLM Authentication Diagnostic - -The `vllm_models` adapter makes one GET request to `/models` relative to the -configured API base, supplying an available credential through the existing -credential rules. A base ending in `/v1` therefore produces `/v1/models`. -Existing explicit diagnostic endpoint overrides remain available. The adapter -can dispatch without a credential; selecting vLLM does not declare authentication -required or inapplicable. - -Recognize HTTP 200 with a valid JSON object containing `object: "list"` and an -array-valued `data` as acceptance of the probe request, and a 401 as rejection. -An empty `data` array counts as acceptance; recognition does not require a -particular model or inspect model capabilities. Reuse the existing bounded -diagnostic decoding and framing rules. Unrecognized responses leave acceptance -unknown under the existing diagnostic reporting model. Report observed credential presence and -response facts without claiming that authentication is enforced or that models -listing establishes inference authorization. Preserve the shared override rules, -including `kind: none`; explicit `vllm_models` selection also works without a -`vllm` provider declaration, just as explicit OpenRouter adapters do. - -vLLM documents `/v1/models` as protected when its API-key authentication is -configured, while `/health` does not establish authentication acceptance. See its -[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations). -Retain existing reachability and inference diagnostic behavior rather than -adding health monitoring. These requests occur only during explicitly requested -diagnostics, never during ordinary generation or engine construction. - -### Diagnostic Migration - -This changes the current authentication metadata contract: an authentication -object currently requires `kind`, and clearing the object removes its configured -adapter. Under this roadmap, endpoint-only overrides become valid, and clearing -the object restores any selected provider's default. `kind: none` suppresses -that default. Explain these changes in the framework format documentation and -release notes, alongside the new provider field. - -### Provider Behavior Ownership - -Centralize provider-specific behavior within `internal/llm`, which already owns -OpenAI-compatible transport and OpenRouter authentication-response recognition. -Keep common request construction, credentials, HTTP execution, framing, and -common response fields shared. Provider-specific code owns additional response -metadata interpretation and authentication-probe conventions. - -Use one provider-selection mechanism inside that package. Generation, -observation, and smoke-test coordination consume provider-neutral facts or invoke -generic diagnostic operations rather than repeating provider dispatch and -response interpretation. Explicit diagnostic adapter selection remains supported -at this boundary. Update the current OpenRouter-specific diagnostic collaborator -method to a generic operation as part of this consolidation. - -Provider-specific files and straightforward dispatch are sufficient initially. -No new `internal/providers` package, registration framework, or broad interface -requiring every provider to implement every capability is needed. -This is configurable selection among PromptKit-supported provider implementations, -not a public API for downstream registration of additional implementations. - -## Token And Cache Reporting - -### OpenRouter Response Mapping - -Use OpenRouter's documented Chat Completions usage fields. Its -[response schema](https://openrouter.ai/docs/api_reference/overview), -[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting), -and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching) -define the following mapping, relative to `usage`: - -| Quantity | Response field | PromptKit field | -| --- | --- | --- | -| Input tokens | `prompt_tokens` | Existing `PromptTokens` | -| Output tokens, including reasoning | `completion_tokens` | Existing `CompletionTokens` | -| Reasoning tokens | `completion_tokens_details.reasoning_tokens` | New `ReasoningTokens` | -| Cache reads | `prompt_tokens_details.cached_tokens` | Existing `CachedTokens` | -| Cache writes | `prompt_tokens_details.cache_write_tokens` | Existing `CacheWriteTokens` | -| Total tokens | `total_tokens` | Existing `TotalTokens` | - -Preserve `CachedTokens` as the public field name and clarify its meaning as -cache-read tokens. Correct the OpenRouter cache-write mapping to the nested -field above; retain the generic provider's existing top-level handling. Do not -add the nested and top-level values together. OpenRouter uses its documented -nested value; an omitted nested value remains unknown. - -Add `ReasoningTokens` to the public/domain token values, presence indicators, -and observed token usage, including conversions, result serialization, attempt -and terminal events, and group summaries. OpenRouter and vLLM supply this -additional built-in response interpretation; the generic provider preserves its -existing behavior. Injected clients may supply the canonical count and its -presence through the existing model-client boundary. - -### vLLM Response Mapping And Availability - -Reuse the ordinary `prompt_tokens`, `completion_tokens`, and `total_tokens` -counts. For `provider: vllm`, map these additional fields relative to `usage`: - -| Response field | PromptKit field | -| --- | --- | -| `completion_tokens_details.reasoning_tokens` | `ReasoningTokens` | -| `prompt_tokens_details.cached_tokens` | `CachedTokens` | -| `prompt_tokens_details.created_cache_tokens` | `CacheWriteTokens` | - -vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/) -uses `created_cache_tokens`, unlike OpenRouter's `cache_write_tokens`. Its -[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/) -describe the creation count as prompt tokens counted as local prefix-cache -writes. Normalize that token activity to `CacheWriteTokens` without implying a -billable write charge. Use the selected provider's documented field, not a sum -of alternative cache-write fields; an absent vLLM creation count stays unknown. - -Detail availability depends on server version and configuration. Current vLLM -gates prompt details on `--enable-prompt-tokens-details` and reasoning details on -a configured reasoning parser. Document these reporting settings, but do not -configure or discover them automatically. Older or differently configured -servers may omit the fields; accept those responses with the affected counts -unknown. Do not infer reasoning counts from generated text or cache writes from -input tokens minus cache reads. - -Once normalized, these counts use the shared presence, aggregation, and -cache-hit rules below. No vLLM-specific result type, ratio, or group accounting -is needed. - -### Count Relationships And Availability - -Reasoning tokens are included in completion tokens, as explained in OpenRouter's -[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens). -vLLM likewise reports its reasoning detail within the generated-token usage. -Cache reads and writes describe input-token breakdowns. Present these as subsets -or details rather than additional consumption to add to input/output totals. -Preserve the provider's reported `TotalTokens`; do not recompute or reconcile it. - -Consumers can derive non-reasoning output as completion tokens minus reasoning -tokens when both counts are available and consistent. No additional stored -non-reasoning count is needed in this scope. Capturing reasoning-token counts -does not capture reasoning text or change reasoning request settings. - -Reuse existing per-field presence and aggregation rules: omitted/null means -unknown, explicit zero means known zero, and aggregate `Known` and `Complete` -flags describe the reported contributions across dispatched attempts. Include -initial generations, retries, and repairs through the existing observation -accounting, retaining terminal-only group aggregation and deduplication. Preserve -the existing distinction between integer-only result usage and presence-aware -observations; zero in `TokenUsage` alone does not establish provider reporting. - -### Token Cache-Hit Ratio - -Add this derived helper on the existing presence-aware usage type: - -```go -func (u ObservedTokenUsage) CacheHitRatio() (float64, bool) -``` - -The ratio is `CachedTokens / PromptTokens`, represented as a fraction from zero -to one. It measures the proportion of input tokens served from cache, not the -proportion of requests with a cache hit or monetary savings. The Boolean reports -whether a ratio is available; an unavailable result returns zero and false. - -Return an available ratio only when both counts are known and complete, input -tokens are positive, and cache reads are between zero and input tokens inclusive. -Zero cache reads with known positive input yields an available zero ratio. Missing -or incomplete counts, zero input, or inconsistent counts leave it unavailable; -preserve the raw reported counts rather than clamping them or failing generation. -Cache-write availability does not affect this calculation. - -For a run or group, divide the aggregated cache-read count by the aggregated -input count; do not average per-attempt or per-run percentages. Do not use -`reads / (reads + writes)`, because cache writes need not account for all -uncached input. The same helper works on attempts, terminal events, and group -summaries without storing another independently maintained field. - -The ratio describes the included observations. A group ratio does not supersede -`TerminalCoverageComplete` or imply that all application work has finished. -For example, input 1,250 and cache reads 250 yield 0.2 (20%) when both counts -are known and complete, even if cache writes are unknown. Output 840 with -reasoning 600 still represents 840 output tokens, not 1,440. - -## Optional Cost Reporting - -### Initial Provider Support - -Capture OpenRouter Chat Completions' top-level `usage.cost`, which OpenRouter -defines as the amount charged to the account, denominated in USD. See its -[usage accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting) -and [USD cost explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/). -Do not add `cost_details` breakdowns or combine upstream inference cost with the -top-level charge; those describe different facts and could double-count cost. - -For the built-in client, select this interpretation through `provider: openrouter`, -regardless of backend identity or namespace. Do not infer USD semantics from an -arbitrary backend's field named `cost`, or add endpoint discovery or verification. -Built-in provider-reported cost interpretation is an extension unique to -`openrouter` initially; `openai-compatible` and `vllm` do not interpret -`usage.cost`. Without an applicable zero-cost declaration, their built-in cost -remains unknown. No separate provider cost-decoding selector is needed. -Aggregation remains provider-neutral after the model-client boundary supplies -interpreted USD facts. - -Use the cost included in the normal completion response. No additional usage -lookup, pricing request, or opt-in provider request parameter is needed. - -### Explicit Backend Zero-Cost Declaration - -Add optional Boolean backend metadata `zero_cost`, independent of `provider`: - -```yaml -promptkit: - backends: - id: local-inference provider: vllm endpoint: http://localhost:8000/v1 zero_cost: true ``` -For each dispatched attempt, use a usable provider/model-client-reported cost -first, including an explicitly reported zero. If no usable reported cost is -available and the effective backend has `zero_cost: true`, supply USD zero with -a configured-zero source. Omitted, null, or false leaves the fallback disabled; -it does not suppress a reported cost. Apply this policy to built-in and injected -client execution using the selected backend, regardless of its provider type. +Trim and lowercase the final effective provider value. Omitted, null, empty, and +whitespace-only values select `openai-compatible`. Reject unsupported nonempty +normalized values as ordinary configuration errors. The supported selections are +`openai-compatible`, `openrouter`, and `vllm`. -The fallback also applies to dispatched attempts that fail, are canceled, or -produce no decodable response. It is an operator declaration, not a provider -observation. Non-dispatched work contributes no cost. Do not infer this setting -from a local address, provider name, token counts, or missing credentials, and -do not install it automatically on any backend. Without a selected backend -declaration, no configured-zero fallback applies. +`zero_cost` is Boolean. Omitted, null, or false disables its fallback; true +enables it. It is independent of provider selection and does not suppress a +usable reported cost. -Resolve the field through ordinary final ConfigKit composition and inheritance; -include it in typed backend helpers, configuration capture, and backend -inspection. Retain the effective policy through prepared execution without -reloading configuration. It is local accounting metadata, not an outbound -request parameter. `InspectBackend` shows the resolved declared setting, -preserving explicit `false`; omitted and null values can remain absent. Do not -manufacture an explicit false value for the runtime fallback or add null-tracking -machinery to resource inspection. `InspectConfiguration` retains ConfigKit's -source history and explicit/null/omitted distinctions. Source provenance belongs -to configuration inspection, not resource inspection. +Resolve both fields through ordinary final ConfigKit composition and inheritance. +Carry their effective selections through capture and prepared execution without +reloading configuration. Declare `provider: openrouter` in the bundled +OpenRouter backend. Do not automatically declare any backend zero-cost. -This supports an explicit zero declaration, not configurable input/output or -cache-read/write token rates, fixed nonzero request charges, pricing tables, or -cost estimation. It does not alter credentials, generation, or token reporting. +These fields are local metadata, not outbound request parameters or entries +added to `extra_params`. Provider selection does not choose an endpoint or model, +and endpoint overrides retain the selected provider and accounting policy. -### Monetary Representation +### Inspection Boundary -Use `float64` amounts in USD. Make the unit explicit in public field names such -as `CostUSD`, rather than introducing a generic amount with an implicit currency. -At the individual model-client response boundary, use an optional amount -(`*float64`): nil means unknown, and a pointer to zero means reported zero. -Injected clients may supply the same optional USD value without needing an -OpenRouter backend; no provider-specific interface is needed. +`InspectBackend` shows the normalized/resolved declared provider and zero-cost +setting, preserving explicit false for `zero_cost`. Omitted/null declarations +may remain absent, and blank provider values need no extra history representation. +Do not manufacture an explicit `openai-compatible` declaration or false +`zero_cost` value from runtime defaults. -Expose cost separately from `TokenUsage` and `TokenUsagePresence`, whose fields -remain token counts. For run results, execution events, and group summaries, -use a small presence-aware cost value with `Value float64`, `Known bool`, and -`Complete bool`, following the existing observed-count conventions, plus the -source metadata below. Successful run results should expose cost even when no -observer was registered; observers also provide per-attempt and terminal cost, -including available facts on failure. +`InspectConfiguration` owns ConfigKit source history and explicit/null/omitted +distinctions. Resource inspection does not acquire source-history fields or +null-tracking machinery. Provider diagnostic defaults likewise remain runtime +fallbacks, not values written into loaded configuration or recorded as explicit +source declarations. -Binary floating point is sufficient for operational telemetry and keeps decoding, -addition, and the public API lightweight. Do not round to cents or another fixed -scale during capture or aggregation; consumers choose display precision. -Decimal strings or a decimal arithmetic dependency would be appropriate for -exact financial ledgers, while fixed-scale integers would require a selected -precision and rounding policy. Neither is warranted for this reporting feature. -The API does not promise exact decimal arithmetic or invoice reconciliation. +## Provider Conventions -### Cost-Source Metadata +All selections reuse the existing +[OpenAI-compatible text Chat Completions foundation](../integrations/openai-compatible-chat.md): +base-relative `/chat/completions` JSON POST requests, existing credential rules, +text messages, selected generation controls and structured output, extra +parameters, first-choice content extraction, common token fields, and existing +response validation, errors, cancellation, timeouts, and transient retries. -Give the presence-aware cost value a `Source` discriminator with these meanings: +| Provider | Additional response interpretation | Default authentication adapter | +| --- | --- | --- | +| `openai-compatible` | Existing token mappings; no provider-reported monetary cost decoding. | None. | +| `openrouter` | OpenRouter reasoning/cache details and `usage.cost` in USD. | `openrouter_key`. | +| `vllm` | vLLM reasoning/cache details; no provider-reported monetary cost decoding. | `vllm_models`. | -- `provider`: a cost supplied by the provider or injected model-client response. - This records the reporting path, not independent verification of a charge. -- `configured_zero`: zero supplied by the explicit backend fallback. -- `mixed`: an aggregate includes both provider and configured-zero contributions. -- Empty: no known contribution has a source. This covers unknown cost and the - known, complete zero aggregate when no attempts were dispatched. +The default provider names PromptKit's current supported conventions, not the +entire OpenAI API. "Lowest common denominator" does not strip existing optional +features or promise every backend supports them. Continue sending selected +supported settings and return backend errors normally. -Derive aggregate source from the union of known contribution sources, including -zero-valued contributions. Unknown contributions supply no source but affect -completeness. A provider-only subtotal can therefore be incomplete without being -`mixed`. Carry this metadata consistently through attempt events, terminal -events, successful results, and group summaries. Reuse existing aggregation and -terminal deduplication rather than adding a parallel collector or cost history. +Provider selection supplies no default endpoint, model, credential environment +variable, or sampling settings, and does not create backend definitions. Keep +vLLM reasoning, chat-template, and other model-specific controls caller-selected; +do not automatically enable thinking or translate those controls. No new +generation protocol, capability negotiation, or version discovery is needed. +vLLM documents its +[OpenAI-compatible interface](https://docs.vllm.ai/en/stable/serving/online_serving/openai_compatible_server/). -Consumers can distinguish reported charges, configured zeros, and mixed totals -without treating configured values as provider reports. Source metadata is -separate from configuration inspection and does not require provenance traversal -during execution. +### Authentication Diagnostics -### Presence, Aggregation, And Failures +Provider defaults apply only to explicitly requested authentication diagnostics. +Loading configuration, constructing an engine, and ordinary generation do not +initiate probes. Existing reachability and inference-default behavior remains. -- An absent or null provider cost is unknown unless the backend's explicit - zero-cost fallback supplies it. A reported numeric zero remains known zero - with a provider source, even when the fallback is enabled. -- On a dispatched attempt, a usable reported cost or configured-zero fallback - is known and complete for that attempt. Cost completeness is independent of - token-field completeness and does not assert provider reporting. -- Sum known costs across dispatched attempts, including initial generations, - retries, and repairs. `Known` means at least one attempt has an available cost - from either source; `Complete` means every dispatched attempt does under this - policy. Unknown values contribute no amount, but make the total incomplete. - With no dispatched attempts, follow the existing known, complete zero - convention, with an empty source. -- Group summaries reuse existing terminal-only aggregation and deduplication: - sum known terminal values, OR their `Known` flags, and AND their `Complete` - flags. Retain existing empty-group and terminal-coverage semantics. A subtotal - from OpenRouter mixed with unreported local-backend cost is incomplete unless - those local attempts have an explicit configured-zero fallback. Combine - cost-source metadata as described above without changing token accounting. -- Keep valid cost observed from a decoded successful envelope when a later - choice/content check fails, as with token usage. Invalid framing and non-2xx - error bodies continue to supply no provider usage, but the configured-zero - fallback still applies to dispatched attempts. Without that fallback, - cancellation or failure may leave incomplete cost reporting even when billable - upstream work occurred. -- Treat an unusable optional cost value, including a nonnumeric value or one - not representable as a finite `float64`, as unknown rather than making an - otherwise valid generation fail; the zero-cost fallback can then apply. This - does not relax existing response framing or content validation. An injected - client likewise cannot supply a nonfinite usable cost. +Explicit `smoke_tests.authentication` metadata overrides provider defaults: -Completeness describes coverage by available reported or explicitly configured -cost, not verification of the provider's billing or proof that every provider -reported a charge. Preserve usable reported amounts without pricing inference or -reconciliation. +- A supplied `kind` selects the adapter; `none` declares authentication testing + inapplicable. +- Omitted/null `kind` uses the provider default, allowing endpoint-only overrides. +- A supplied endpoint overrides the selected adapter's destination; omitted/null + uses that adapter's existing fallback. +- An absent/null authentication block removes its overrides and leaves provider + defaults applicable. `kind: none`, not clearing the block, suppresses them. -For a successful run, result cost and terminal-event cost describe the same -dispatched attempts and must agree on value, presence, completeness, and source. -Result cost must also be the same with observation enabled or disabled. For -example, an unreported failed attempt followed by a successful attempt reporting USD -0.002 yields `Value: 0.002`, `Known: true`, `Complete: false`, and a provider -source when no zero-cost fallback applies. Apply the same accounting through -repairs and ordinary or prepared execution. An invocation with fully reported -OpenRouter charges and configured-zero local calls can have a complete subtotal -with a mixed source, without claiming all providers reported costs. +Without a provider default or explicit adapter, preserve the existing unsupported +diagnostic outcome rather than assuming authentication is inapplicable. Explicit +adapters remain selectable without a matching provider declaration. -Group cost completeness concerns the terminal observations included in its -subtotal. If two completed runs reported all their costs but a third observed -run has no terminal event, the subtotal may have `Complete: true` while -`TerminalCoverageComplete` is false. Preserve this distinction using the existing -coverage field; no additional completion flag or group lifecycle is needed. +Retain `openrouter_key` behavior. The new `vllm_models` adapter makes one GET to +`/models` relative to the configured API base, supplying an available credential +under existing rules and allowing dispatch without one. A base ending in `/v1` +therefore produces `/v1/models`; explicit diagnostic endpoint overrides apply. -## Accounting And Responsibility Boundaries +For `vllm_models`, HTTP 200 with a valid JSON object containing `object: "list"` +and array-valued `data` means acceptance of the probe request. An empty array is +accepted; recognition does not inspect model capabilities or require a particular +model. HTTP 401 means rejection; unrecognized responses leave acceptance unknown. +Reuse existing bounded diagnostic decoding and framing. Report observed +credential presence and response facts without claiming authentication is +enforced or models-list acceptance establishes inference authorization. -Reuse existing events, token presence/completeness, and group summaries, extending -them with the cost facts above. Available usage remains observable after an error -or cancellation under the existing execution-observation behavior; this does not -promise that upstream providers report usage for every attempt. +vLLM documents `/v1/models` as protected when API-key authentication is configured; +`/health` is not an authentication check. See its +[authentication documentation](https://docs.vllm.ai/en/stable/usage/security/#api-key-authentication-limitations). +No health-monitoring facility is added here. -PromptKit owns observer selection, event generation, and aggregation. CommandKit -and other consumers own invocation lifecycle, CLI flags, presentation, output -delivery, and interpretation. This feature adds no CommandKit dependency or -application-specific concepts to PromptKit. +## Token And Cache Reporting + +### Provider Response Mapping + +Use common `usage.prompt_tokens`, `usage.completion_tokens`, and +`usage.total_tokens` for `PromptTokens`, `CompletionTokens`, and `TotalTokens`. +Normalize these additional provider details, relative to `usage`: + +| Public field | OpenRouter | vLLM | +| --- | --- | --- | +| New `ReasoningTokens` | `completion_tokens_details.reasoning_tokens` | `completion_tokens_details.reasoning_tokens` | +| Existing `CachedTokens` | `prompt_tokens_details.cached_tokens` | `prompt_tokens_details.cached_tokens` | +| Existing `CacheWriteTokens` | `prompt_tokens_details.cache_write_tokens` | `prompt_tokens_details.created_cache_tokens` | + +OpenRouter documents these in its +[response schema](https://openrouter.ai/docs/api_reference/overview), +[usage-accounting guide](https://openrouter.ai/docs/cookbook/administration/usage-accounting), +and [caching guide](https://openrouter.ai/docs/guides/best-practices/prompt-caching). +vLLM's [response construction](https://docs.vllm.ai/en/stable/api/vllm/entrypoints/openai/chat_completion/serving/) +uses `created_cache_tokens`; its +[output definitions](https://docs.vllm.ai/en/stable/api/vllm/outputs/) +describe local prefix-cache writes, not a billable write charge. + +Preserve `CachedTokens` as the public name and clarify its meaning as cache reads. +Use each provider's documented cache-write field, not a sum or guessed alias; +a missing nested value is unknown even if an alternative field is present. +Retain the generic provider's existing mappings, including its top-level +`usage.cache_write_tokens` handling. + +Add `ReasoningTokens` to public/domain token values, presence indicators, observed +usage, conversions, and result serialization, and carry it through attempt events, +terminal events, and group summaries. Injected clients may supply canonical +counts and presence through the existing model-client boundary, retaining its +legacy `UsagePresence` fallback; explicit presence distinguishes measured zero. + +### Availability And Optional Metadata + +Omitted/null details are unknown, not zero. Explicit integer zero is known zero. +For the newly interpreted provider-specific reasoning and cache-write details, +wrong-type, nonintegral, or unrepresentable integer values are also unknown. +An unusable new detail does not invalidate otherwise usable content or erase +independently available usage. Preserve existing common-field decoding and +response framing/content validation; this is a targeted optional-metadata rule, +not a general relaxation of the response contract. Keep this interpretation local +to the model-client boundary without a general numeric-conversion framework. + +vLLM detail availability depends on version and configuration: prompt details +currently require `--enable-prompt-tokens-details`, and reasoning details depend +on a configured reasoning parser. Document those reporting settings without +configuring or discovering them automatically. Older or differently configured +servers may omit details and still produce usable responses. Do not estimate +reasoning counts from text or cache writes from input minus cache reads. + +### Count Relationships And Aggregation + +Reasoning is included in completion tokens, not additional output to add to them. +OpenRouter explains this in its +[reasoning documentation](https://openrouter.ai/docs/guides/best-practices/reasoning-tokens); +vLLM similarly includes reasoning detail within generated-token usage. Cache reads +and writes are input-token details, not extra input consumption. Preserve the +reported total without recomputation or reconciliation. + +Consumers can derive non-reasoning output from consistent, known completion and +reasoning counts; no additional stored count is needed. Capturing counts does +not capture reasoning text or change generation/caching settings. + +Reuse existing per-field token accounting: aggregates sum known contributions, +`Known` identifies available reporting, and `Complete` describes its coverage +across dispatched attempts. Preserve initial-generation, retry, and repair +observation accounting and terminal-only group deduplication. Existing integer +result usage remains distinct from presence-aware observations; a zero result +field alone does not establish that the provider reported it. + +### Token Cache-Hit Ratio + +Add a derived helper, not another stored total: + +```go +func (u ObservedTokenUsage) CacheHitRatio() (float64, bool) +``` + +Compute `CachedTokens / PromptTokens` as a fraction from zero to one. This is the +proportion of input tokens served from cache, not request hit frequency or cost +savings. For aggregates, divide summed reads by summed input; do not average +individual percentages or use `reads / (reads + writes)`. + +Return an available ratio only when input and cache-read counts are known and +complete, input is positive, and reads lie between zero and input inclusive. +Known zero reads with positive input yields an available zero. Missing, +incomplete, zero-denominator, or inconsistent data yields zero and false while +preserving raw counts; it does not fail generation or clamp reported values. +Cache-write availability is irrelevant. + +The same helper serves attempts, terminals, and group summaries. A group ratio +describes its included observations and does not supersede terminal coverage or +imply all application work has finished. + +## Cost Reporting + +### Reported Cost And Configured-Zero Precedence + +Initially, only the OpenRouter built-in provider interprets `usage.cost` as the +account charge in USD. See its +[usage-accounting reference](https://openrouter.ai/docs/cookbook/administration/usage-accounting) +and [USD explanation](https://openrouter.ai/blog/insights/cost-vs-quality-tradeoff-framework-for-agent-models/). +Do not add `cost_details` amounts to that charge or interpret arbitrary providers' +fields named `cost`. The generic and vLLM providers do not decode monetary cost. + +Use the cost included in the ordinary response; no extra request or usage lookup +is needed. At the model-client boundary, represent an optional USD amount as +`*float64`: nil means unknown; a pointer to zero means reported zero. Injected +clients may supply the same value regardless of provider selection. + +Resolve cost at completion of each dispatched attempt, outside provider-specific +decoding and whether or not decoding occurred or succeeded: + +- A usable provider/model-client-reported cost takes precedence, including zero. +- Otherwise, `zero_cost: true` on the effective backend supplies configured USD zero. +- Otherwise, cost remains unknown. + +This applies to built-in and injected clients, including failed or canceled +dispatched attempts and undecodable responses. Without a selected backend +declaration, no configured-zero fallback applies. Non-dispatched work contributes +no cost. Do not infer zero cost from local hosting, provider name, credentials, +or token counts, or install the declaration automatically on bundled backends. + +An unusable reported amount, including a nonnumeric value or one not representable +as a finite `float64`, is unknown before fallback; it does not make an otherwise +usable generation fail. Keep available cost from a decoded successful envelope +when a later choice/content check fails. Invalid framing and non-2xx bodies supply +no provider usage, although the configured-zero fallback still applies to a +dispatched attempt. Existing framing, content, and error handling remain intact. + +### Monetary Representation And Sources + +Expose cost separately from token counts and their presence indicators. Use +explicit USD field naming such as `CostUSD` and a small public cost value for +results, events, and group summaries with: + +- `Value float64`: the available USD amount or subtotal; +- `Known bool`: availability; +- `Complete bool`: coverage under the selected cost policy; and +- `Source`: the contribution-source discriminator below. + +| Source | Meaning | +| --- | --- | +| `provider` | Supplied by the provider or injected model-client response; identifies the reporting path, not independent billing verification. | +| `configured_zero` | Supplied by the explicit backend fallback. | +| `mixed` | An aggregate includes both reported and configured-zero contributions. | +| Empty | No known contribution has a source, including an unknown cost or an aggregate with no dispatched attempts. | + +Use binary floating point for operational telemetry, without rounding to cents +or another fixed scale during capture or aggregation. Consumers choose display +precision. Exact-decimal ledgers, decimal dependencies, fixed-scale currency +rounding, pricing inference, and invoice reconciliation are not part of this work. + +### Attempt, Run, And Group Accounting + +A dispatched attempt with usable reported cost or configured zero is known and +complete for cost, independently of token completeness. Sum known amounts across +all dispatched attempts, including retries and repairs. Aggregate `Known` means +at least one known contribution; `Complete` means every dispatched attempt has a +cost from either source. Unknown contributions add no amount and make coverage +incomplete. Completeness does not assert that every provider reported a charge. + +Aggregate sources by union of known contributions, including zero amounts. +Unknown contributions add no source: a provider-only subtotal can be incomplete +without becoming `mixed`. A reported zero plus a configured zero is `mixed`. + +Successful results expose cost even without an observer. Result and terminal +cost agree on amount, availability, completeness, and source, and enabling +observation does not change result cost. For example, an unreported failed +attempt followed by a reported USD 0.002 yields an incomplete provider-source +subtotal when no zero-cost fallback applies. + +Groups aggregate terminal costs only, counting each run's first terminal once. +Combine known amounts, OR `Known`, AND `Complete`, and union sources. Preserve +existing empty/no-terminal group semantics: a completed execution with no +dispatched attempts yields known, complete zero with empty source; an empty group +does likewise, while a group with members but no terminals has unknown, +incomplete usage. + +Cost completeness concerns included terminal observations, independently of +`TerminalCoverageComplete`. Two completed runs with full cost coverage can yield +a complete subtotal while a third observed run lacks a terminal. Conversely, +fully reported OpenRouter runs and explicitly zero-cost local runs can yield a +complete, mixed-source total. Neither case verifies a provider invoice or future +application progress. + +## Internal Responsibilities And Scope Limits + +Provider-specific metadata interpretation and authentication conventions belong +in `internal/llm`, sharing common transport, request construction, credentials, +framing, and response fields. Use one provider-selection mechanism there. +Generation/observation coordination consumes normalized facts or generic +diagnostic operations, replacing the OpenRouter-specific diagnostic collaborator +method with a generic operation while retaining explicit adapter selection. + +Provider-specific files and straightforward dispatch suffice; no public provider +registration API, new `internal/providers` package, or broad mandatory-capability +interface is needed. This is configuration-driven selection among supported +implementations, not downstream registration of implementations. + +Reuse a small shared cost-accounting component for result and observation paths +so dispatched attempts, fallback, and source union follow the same rules even +when observation is disabled. Provider code interprets wire metadata; common +accounting owns fallback and aggregation. No ConfigKit provenance traversal +belongs in ordinary execution. Smoke diagnostics retain their separate report/evidence model and are not -enrolled in ordinary execution observation by this change. Reasoning-text -capture, changes to reasoning or caching request behavior, additional built-in -cost providers, configurable token-rate estimation, fixed nonzero request -pricing, multiple currencies, currency conversion, pricing estimates, -billing reconciliation, persistent history, and logging infrastructure are -outside this scope. +enrolled in ordinary execution observation. CLI integration, output delivery, +persistent history, logging infrastructure, reasoning-text capture, changes to +reasoning/caching request behavior, server administration/metrics, additional +built-in cost decoders, multiple currencies, configurable token rates, fixed +nonzero request prices, and cost estimation remain outside scope. -## Work Needed And Completion Criteria +## Work Needed And Acceptance -- Add the context helper and connect observer selection to ordinary and prepared - execution, keeping contextual selection separate from private run state. -- Preserve existing explicit selection and coordinator behavior while applying - the composition and lifetime rules above. Preserve existing observer selection - when no contextual observer is selected. -- Add `provider` to the backend configuration model, typed source helpers, - capture, inspection, and execution plumbing. Update the bundled OpenRouter - backend to select it explicitly. Retain the selection through prepared runs. -- Add `zero_cost` to backend settings, typed helpers, capture, inspection, and - prepared-execution plumbing. Apply the fallback outside provider-specific - decoding so it works with built-in and injected clients. Verify composition - and inheritance, omitted/null/false behavior, explicit overrides, provider - independence, configuration-inspection provenance, and absence from provider - bodies. Verify that resource inspection preserves explicit false without - manufacturing values or source-history fields for omitted/null settings. -- Consolidate provider-specific response and authentication-probe behavior in - `internal/llm`; keep coordination and accounting provider-neutral. Cover - provider-default probes, explicit kind and endpoint overrides, `kind: none`, - and existing explicit probes without a provider selection. -- Verify provider selection independently of backend names and namespaces, - including renamed and inherited backends, equivalent omitted/null/blank values, - trimming and case normalization, rejection of unsupported nonempty values, - and retention through endpoint overrides. Confirm provider metadata is not - serialized into the completion request body. Verify that explicit - `openai-compatible` and omitted/null/blank selection preserve the same existing - shared behavior without manufacturing explicit configuration provenance. -- Verify that provider diagnostic defaults do not become explicit values in the - loaded configuration or replace source provenance in inspection. Cover the - changed clearing behavior and endpoint-only authentication overrides. -- Carry OpenRouter reasoning-token counts through token values, presence, - results, events, and group summaries. Verify nested OpenRouter cache-write - decoding, including omitted/null/zero values and the nested value taking - precedence when a top-level value is also present. Preserve generic-provider - token mappings and existing field meanings. -- Add the `vllm` provider selection, reasoning and cache-detail mappings, and - `vllm_models` authentication adapter. Cover supplied/missing credentials, - recognized acceptance (including an empty models list), rejection, - unrecognized responses, provider defaults, and explicit kind/endpoint overrides, - including `none` and adapter selection - without a provider declaration. Verify that ordinary generation performs no - diagnostic requests and leaves monetary cost unknown unless the backend - explicitly declares the zero-cost fallback. -- Cover vLLM fixtures with reported, zero, omitted, and null reasoning/cache - details, including responses representative of servers without detail - reporting enabled. Verify `created_cache_tokens` mapping independently of - OpenRouter's cache-write field and carry normalized counts through results, - events, and groups. Do not require a live server or version discovery. -- Add `ObservedTokenUsage.CacheHitRatio`. Cover known zero reads, unknown and - incomplete counts, zero input, inconsistent counts, independence from - cache-write availability, and token-weighted aggregate ratios. Verify subset - accounting without adding reasoning or cached tokens to the reported totals. -- Carry optional OpenRouter cost through response decoding, internal/public - conversions, successful run results, attempt and terminal events, and existing - group summaries. Keep result cost available without registering an observer. - Preserve existing token fields and accounting. -- Carry cost-source metadata through results, events, conversions, and group - snapshots. Cover provider-only, configured-zero-only, mixed, and unknown costs, - including zero-valued reported contributions and empty/non-dispatched cases. - Verify source union and terminal deduplication independently of cost value. -- Verify that usable reported cost wins over `zero_cost: true`, and missing, - null, or unusable reported cost permits fallback. Cover dispatched failures, - cancellation, undecodable responses, retries, repairs, ordinary/prepared - execution, and observer-enabled/disabled results. Verify that mixed OpenRouter - and declared zero-cost local calls can produce complete cost totals while - token completeness remains independent. -- Verify that successful-result cost agrees with terminal-event cost and is - unchanged when observation is disabled, including the unreported-retry example - above. Verify group snapshots with fully reported completed runs alongside a - member without a terminal observation. -- Cover ordinary and prepared execution, coordinated continuations, nested calls - using derived contexts, concurrent calls, multiple engines sharing a scope, - and independent scopes on a reused engine. -- Cover callback ordering, nested contextual registration, nil registration, - duplicate group enrollment, prepared-handle scope selection, and existing - preparation/discard/rejected-claim behavior. -- Verify aggregate accounting through retries and repairs, available usage after - failures or cancellation, and known zero versus unknown or incomplete counts. - Use offline fixtures and injected clients; live inference is unnecessary. -- Cover fractional USD values, explicit zero, omitted/null and unusable cost, - generic built-in selection, optional injected-client cost, mixed known - and unknown totals, repair/retry aggregation, and duplicate group registration. - Use tolerant comparisons where floating-point addition requires them; do not - turn telemetry into an exact-decimal guarantee. -- Document the exact helper contract in GoDoc and add a focused invocation-level - aggregation example to the consumer observation guide, including cost - availability, units, completeness, and source. Explain that configured zeros - can make cost complete without provider reporting. Document the reasoning/cache - fields and cache-hit helper in their owning GoDoc and illustrate input/output - breakdowns without double-counting. Document OpenRouter/vLLM token decoding and - OpenRouter cost decoding in the - [integration contract](../integrations/openai-compatible-chat.md#response-handling), - and update internal observation and model-client documentation for changed flows. - Document `provider`, its smoke-test defaults and override semantics in the - framework format reference and relevant public GoDoc; update inspection and - diagnostic documentation with the implemented behavior. Document `zero_cost` - and its fallback precedence, inspection behavior, and prepared lifetime in the - relevant format, GoDoc, and consumer owners. Include the diagnostic - migration in release notes. Include vLLM usage availability, server reporting - settings, the meaning of cache creation, and the limits of its authentication - diagnostic in the relevant consumer and integration documentation. +The following areas describe required changes, not an execution order. -## Implementation Planning Guidance +- Extend the public context API and ordinary/prepared execution wiring while + preserving independent nested lifecycles, explicit observers, coordinator + selection, and existing preparation/discard/rejected-claim behavior. +- Extend backend settings, typed helpers, final-value normalization/validation, + capture, prepared metadata, and inspection for `provider` and `zero_cost`. + Update the bundled OpenRouter declaration. Preserve configuration provenance + and resource-inspection simplicity, and keep metadata out of request bodies. +- Consolidate provider behavior in `internal/llm`, adding vLLM usage normalization + and the models-list diagnostic alongside OpenRouter interpretation and the + shared default path. Preserve common decoding while tolerating unusable new + optional details and cost locally to the model-client boundary. +- Extend public/domain token types, presence, serialization, conversions, events, + and groups with reasoning counts and correct provider-specific cache writes. + Add the presence-aware cache-hit helper without changing existing token + meanings or introducing a separate ratio accumulator. +- Carry optional reported cost and configured-zero policy through attempt + completion and shared accounting into results, events, and groups, including + source metadata and observer-independent result accounting. -Organize the eventual implementation plan around two workstreams: +Select focused offline checks under the [testing policy](../policy/testing.md). +Use real internal collaborators and controlled HTTP/injected-client boundaries, +with proportionate public contract coverage. Important behaviors include: -1. Contextual observer selection, including ordinary, prepared, coordinated, - and nested execution boundaries and the existing group collector. -2. Backend provider selection and consolidation within `internal/llm`, including - smoke-test defaults and explicit overrides, followed by OpenRouter/vLLM token - extensions, provider-reported cost, and configured-zero fallback through - decoding, accounting, public results, events, and group summaries, plus the - derived cache-hit helper. Include the corresponding configuration and API - documentation. +- Context inheritance, registration order, nil/duplicate enrollment, concurrent + calls, multiple engines, independent scopes, nested calls, and execution-time + selection for a handle prepared elsewhere. +- Provider selection independent of backend identity; normalization and unsupported + value handling; omitted/null/blank defaults; inheritance, endpoint overrides, + explicit false, and inspection without manufactured values or source history. +- Provider-default and explicit diagnostic selection, endpoint-only overrides, + clearing metadata versus `kind: none`, vLLM acceptance including an empty list, + rejection, unrecognized/framing-limited responses, and optional credentials. +- Provider token mappings, absent/null/zero details, malformed new optional counts + without generation failure, unchanged common-field handling, and responses + representative of vLLM servers with detail reporting disabled. +- Cache-hit availability, inconsistent counts without generation failure, + token-weighted aggregates, independence from cache writes, and no double-counting + of reasoning or cache details. +- Reported cost precedence, configured-zero fallback on dispatched failure and + cancellation, built-in/injected clients, retries/repairs, mixed/unknown sources, + zero-valued contributions, and independently incomplete token reporting. +- Result/terminal cost agreement with observation enabled or disabled, duplicate + terminal deduplication, empty/no-terminal groups, and subtotal completeness + distinct from terminal coverage. -The second area is more extensive than the context helper. Keep backend provider -configuration, diagnostic consolidation, token/cache reporting, and cost/source -accounting in separately reviewable implementation units rather than one large -implementation prompt. -Avoid coupling delivery of contextual observation to provider cost support. +Synthetic fixtures and offline examples suffice; live providers are not required. +Keep ownership localized and avoid repeating the same assertions across layers. -Today, successful results accumulate generation responses in the runner, while -optional observation recorders account for dispatched attempts. Adding cost to -response objects alone does not preserve completeness across unreported retries. -Reuse a small accounting component for result and observation paths so they -apply the same rules, including when no observer is selected. Preserve existing -token accounting rather than expanding this work into a redesign of its public -contract. +## Documentation And Migration -Keep tolerant optional-cost decoding local to the model-client boundary. Adding -a plain `*float64` JSON field to the typed envelope would make a nonnumeric or -overflowing value fail decoding of the entire response. Decode the optional -field separately so an unusable provider cost becomes unknown before applying -any configured-zero fallback, without changing existing framing or content -validation. This needs no general numeric-conversion layer. +When implemented, exact exported contracts belong in GoDoc; the +[consumer guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) should show +invocation-scoped aggregation, reasoning/cache breakdowns without double-counting, +and cost availability, source, and completeness. Document provider wire mappings, +optional-metadata handling, and diagnostic response recognition in the +[integration contract](../integrations/openai-compatible-chat.md#response-handling). +The [framework format reference](../formats.md#backend-definitions) owns the new +backend fields, default selection, inheritance, and diagnostic overrides. +Update internal documentation for changed ownership and flows. -Apply configured-zero fallback after decoding, outside provider-specific code, -so built-in and injected clients share the same policy. Aggregate cost sources -from known contributions even when their amounts are zero; a reported zero and -a configured zero together have a mixed source. Keep subtotal completeness -separate from terminal coverage, using the existing coverage field and the -examples above. These rules belong in shared accounting rather than separate -provider, result, and group implementations. +Explain vLLM's server reporting settings, unknown-versus-zero behavior, and cache +creation meaning without promising availability on every deployment. Consumers +own presentation. Models-list acceptance does not establish inference +authorization, and complete cost metadata does not independently verify billing. -Use the completion criteria above to select focused offline checks for each -unit. This roadmap owns target behavior; a separate implementation plan owns -the detailed work sequence. +Release notes should call out that authentication objects no longer require +`kind` when a provider default supplies it, endpoint-only overrides are supported, +and clearing an authentication block restores provider defaults. `kind: none` +suppresses that default. Preserve existing explicit adapters without requiring a +provider declaration.