From 1070be8d89d97d7675167fa08164bbec00c16bec Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 22 Nov 2026 09:39:27 +0000 Subject: [PATCH] Define invocation-scoped observation roadmap --- docs/roadmap/invocation-scoped-token-usage.md | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docs/roadmap/invocation-scoped-token-usage.md diff --git a/docs/roadmap/invocation-scoped-token-usage.md b/docs/roadmap/invocation-scoped-token-usage.md new file mode 100644 index 0000000..2ad11a9 --- /dev/null +++ b/docs/roadmap/invocation-scoped-token-usage.md @@ -0,0 +1,162 @@ +# Invocation-Scoped Execution Observation + +Status: accepted scope; implementation pending. + +## Purpose + +Allow consumers to select execution observation once for an application-owned +operation, rather than attach an observer to every PromptKit request. Nested, +concurrent, and multi-engine calls using that operation's context should +participate automatically. + +The motivating consumer is CommandKit's optional `ckprompt` integration, which +could use this facility to report token usage for an entire command or pipeline. +PromptKit supplies application-neutral observation and aggregation; consumers +choose how to present or use the results. + +## Current Baseline + +PromptKit v0.24.0 already provides: + +- [ExecutionObserver and ComposeObservers](../../observations.go) for synchronous + metadata callbacks and ordered composition. +- [ExecutionGroup](../../execution_group.go) for concurrent collection, detached + snapshots, and aggregation of the first terminal event for each observed run. +- Presence-aware prompt, completion, total, cached, and cache-write token counts, + including aggregate accounting across retries and repairs. +- Explicit selection through `RunRequest.Observer`, retained for prepared + execution, and [CoordinateRequest.Observer](../../coordinate.go) for + coordinator-selected continuation runs. + +The missing capability is contextual observer selection. Applications currently +have to enroll each request explicitly. The existing +[consumer observation guide](../consumers/pkg-promptkit.md#observe-an-ordinary-run) +describes that workflow. + +## Target Public API + +Add one public helper: + +```go +func WithExecutionObserver(ctx context.Context, observer ExecutionObserver) context.Context +``` + +The returned context associates the observer with executions using that context +or derived contexts that retain its values. It composes with any observer already +associated with the context; it does not replace the inherited observer. A nil +observer leaves the inherited selection unchanged. + +No engine option, process-wide default, or new aggregation type is needed. +The intended consumer workflow is: + +```go +var group promptkit.ExecutionGroup +ctx = promptkit.WithExecutionObserver(ctx, group.Observe) + +err := runApplicationPipeline(ctx) +// After all application-owned work has finished, including on failure: +summary := group.Snapshot() +``` + +The consumer owns context propagation, waiting for its work, and using the +snapshot and error. Calls using unrelated contexts are not automatically +enrolled. Group coverage continues to describe observed members, not future +work or overall application progress. + +## Observer Selection And Delivery + +### Composition + +Preserve existing explicit observers. Deliver each event to the explicit request +observer first, then any coordinator observer already composed with it, then +contextual observers in registration order, from inherited to newly added. + +Preserve the existing synchronous callback and panic behavior. Concurrent +executions may invoke a shared observer concurrently; consumers remain +responsible for synchronizing their callback state. `ExecutionGroup` already +supports concurrent use. + +### Ordinary And Nested Execution + +`Run` selects contextual observation from its invocation context. Each execution +has its own observation identity, recorder, and accounting, including a nested +execution using a context passed to an application callback or injected client. + +Inherited observer selection is distinct from private execution state. Existing +clearing of an enclosing execution's private state at a public execution boundary +must preserve the contextual observer selection. Inheriting observation does not +reuse an enclosing run's recorder, totals, or lifecycle. + +### Prepared Execution + +`RunPrepared` selects contextual observation from the context supplied when the +handle is executed. A handle prepared in one scope and executed in another +participates in the execution scope, not the preparation scope. + +The explicit `RunRequest.Observer` remains retained by a successfully prepared +handle as today and composes with the execution-time contextual observers. +Preparation itself emits no execution events. Discarded handles and rejected +execution claims retain their existing observation behavior. + +### Coordinated Execution + +Continuation runs receive contextual observation through their ordinary `Run` +boundary. `CoordinateRequest.Observer` continues to select continuation runs +only. Additional calls made by application callbacks participate when they use +the scoped context, without being automatically selected by the coordinator's +explicit observer. + +Contextual observers are incorporated once per execution, rather than being +re-added by coordination or another forwarding layer. + +### Duplicate Registration And Accounting + +Registering the same callback through multiple explicit or contextual paths may +deliver the same event more than once. Do not attempt to identify or deduplicate +arbitrary Go function values. + +Retain one execution lifecycle per run and reuse `ExecutionGroup`'s existing +terminal-event deduplication. Selecting the same group through multiple paths +must not inflate its totals. Terminal usage is an aggregate, not an additional +contribution to be added to attempt usage. + +## Accounting And Responsibility Boundaries + +Reuse existing events, usage presence/completeness, and group summaries. Available +usage remains observable after an error or cancellation under the existing +execution-observation behavior; this does not promise that upstream providers +report usage for every attempt. + +PromptKit owns observer selection, event generation, and aggregation. CommandKit +and other consumers own invocation lifecycle, CLI flags, presentation, output +delivery, and interpretation. This feature adds no CommandKit dependency or +application-specific concepts to PromptKit. + +Smoke diagnostics retain their separate report/evidence model and are not +enrolled in ordinary execution observation by this change. Additional usage +fields such as reasoning tokens and monetary cost, pricing estimates, billing +reconciliation, persistent history, and logging infrastructure are outside this +scope. + +## Work Needed And Completion Criteria + +- Add the context helper and connect observer selection to ordinary and prepared + execution, keeping contextual selection separate from private run state. +- Preserve existing explicit selection and coordinator behavior while applying + the composition and lifetime rules above. Keep existing execution behavior + unchanged when no contextual observer is selected. +- Cover ordinary and prepared execution, coordinated continuations, nested calls + using derived contexts, concurrent calls, multiple engines sharing a scope, + and independent scopes on a reused engine. +- Cover callback ordering, nested contextual registration, nil registration, + duplicate group enrollment, prepared-handle scope selection, and existing + preparation/discard/rejected-claim behavior. +- Verify aggregate accounting through retries and repairs, available usage after + failures or cancellation, and known zero versus unknown or incomplete counts. + Use offline fixtures and injected clients; live inference is unnecessary. +- Document the exact helper contract in GoDoc and add a focused invocation-level + aggregation example to the consumer observation guide. Update internal + observation documentation where its selection flow changes. + +This roadmap defines the target behavior. A separate implementation plan can +divide the work into appropriately sized stages.