From 45bbf7bb2c742dff21850e3f9814ebb9df69ea26 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 13 Nov 2026 16:33:21 +0000 Subject: [PATCH] Record per-run accounting and continuation relationships --- docs/roadmap/conversation-continuation.md | 94 +++++++++++++++-------- 1 file changed, 63 insertions(+), 31 deletions(-) diff --git a/docs/roadmap/conversation-continuation.md b/docs/roadmap/conversation-continuation.md index de9fa7d..d5b1064 100644 --- a/docs/roadmap/conversation-continuation.md +++ b/docs/roadmap/conversation-continuation.md @@ -54,9 +54,10 @@ The proposed capabilities are: Continuation has direct value by removing consumer capture, reconstruction, prefix comparison, and history assembly. The coordinator centralizes turn -progression, execution relationships, cancellation, budgets, and diagnostics. -Its value should come from removing that repeated lifecycle work, beyond merely -replacing a consumer's loop counter. +progression, cancellation, and budgets while reusing ordinary execution +observations. Captured exchanges supply the minimal parent relationship selected +in Decision 5. Its value comes from removing repeated lifecycle work, beyond +merely replacing a consumer's loop counter. Audita would continue owning candidate selection, domain validation, feedback meaning, branch scheduling, accepted sibling proposals, and transcript changes. Its domain-specific repair code would not move wholesale into PromptKit. @@ -178,9 +179,9 @@ interpret the purpose of that work. **Implementation planning:** select exact request/stop and outcome types under Decision 4's accepted response delivery, turn-budget consumption, zero-turn behavior, cancellation, and minimal error/result semantics. -Execution association and usage accounting belong to Decision 5. The sketches -below illustrate the selected responsibility split; they do not settle the -remaining public contracts. +Decision 5 records the selected execution association and accounting approach. +The sketches below illustrate the selected responsibility split; they do not +settle the remaining public contracts. #### Illustrative Consumer Sketches @@ -270,7 +271,8 @@ already receive each completed response through the callback. Completed work must not be mistaken for application acceptance of its content. The concrete interface should let both examples use the same coordination mechanism without manual message-history assembly or per-turn lifecycle -bookkeeping. Callback model-call association remains part of Decision 5. +bookkeeping. Calls made inside callbacks retain independent observations; +applications explicitly group them as selected in Decision 5. ### 4. How Are Application Next Actions And Execution Outcomes Represented? @@ -386,8 +388,9 @@ Failure-stage fields, additional run-ID references, a separate failure classification hierarchy, and aggregate reporting are not part of this minimal termination contract. They require a demonstrated consumer need before addition. This does not remove fields from existing response values or observation APIs. -Decision 5 separately evaluates accounting, and Decision 6 inspection, without -presuming that either needs to expand the coordinator's termination result. +Decision 5 reuses ordinary run accounting without expanding this result. +Decision 6 separately evaluates inspection without presuming that it needs to +expand the coordinator's termination result. Audita retains its own meanings for acceptance, rejection, withdrawal, and assessment unavailability. A normal stop does not assert any of those outcomes. @@ -408,33 +411,59 @@ result. ### 5. How Are Related Executions And Usage Accounted For? -**Status: open.** Define accounting for the initial exchange, each continuation, -its structural repairs and transport retries, and model calls made by -application callbacks, both for standalone continuations and coordinated turns. +**Status: accepted; exact placement of relationship metadata remains for +implementation planning.** Reuse ordinary per-run accounting, expose a minimal +parent relationship for continuations, and leave grouping of other calls and +overall totals to applications. This applies to both standalone continuations +and coordinated turns. The current [observation boundary](../internal/observation.md) keeps nested invocations independent. A callback's arbitrary model calls cannot simply be assumed to belong to the enclosing interaction's totals. -**Recommendation:** preserve independent run accounting and add explicit -relationships between exchanges, branches, sequences, and associated runs. -Choose a small association mechanism, such as a scoped observation facility or -explicit execution references, so consumers need not rebuild lifecycle tracking. -Shared initial work should be referenced by branches and counted once when -reporting the combined operation. Report available usage with its existing -presence/completeness meaning; do not promise complete cost accounting. +**Accepted per-run accounting:** each continuation reports its own usage and +ordinary execution observations, including the existing distinctions between +transport attempts, structural repairs, and available failure-path diagnostics. +Keep existing usage presence/completeness semantics. The coordinator adds no +second accounting system or cumulative totals to its termination result. -**Alternative:** have PromptKit automatically capture every nested run during an -application callback. This minimizes explicit association but can include -unrelated work and changes the existing invocation-isolation boundary. -Another simpler option is to report continuation usage only and let applications -attach callback accounting; that leaves more work downstream and needs checking -against Audita's stated reporting requirement. +An initial exchange A that produces branches B and C contributes its usage only +to A. B and C do not inherit that usage. Applications reporting the combined work +can count A, B, and C once each. Usage reported for sending captured history again +in B or C belongs to those new executions and still counts. Reusing content does +not eliminate newly reported usage. Existing observation totals must not be +added to the attempt events they already summarize. -**Details to settle:** association of concurrent callback calls, observer -composition, failure-path diagnostics, and which totals include which child -executions. A turn budget would bound continuation attempts, not all provider -calls performed by structural repair, transport retries, or application work. +**Accepted relationship metadata:** retain the originating run identity with +each captured exchange and expose the immediate parent relationship in +continuation metadata. Use existing run identifiers without introducing mandatory +conversation, branch, or sequence identifiers. This supplies information that +PromptKit knows directly without requiring consumers to reconstruct which +exchange each continuation extends. Its exact public placement remains for +implementation planning and need not enlarge the coordinator's termination +result. + +**Accepted application boundary:** model calls made inside a callback remain +independently observed and accounted for. PromptKit does not infer that they +belong to a continuation or aggregate merely because they execute inside its +callback. Applications explicitly group those existing observations when useful, +including concurrent calls. Audita retains assembly of its overall usage report; +PromptKit supplies execution facts without owning an application-wide reporting +model or promising complete cost accounting. + +**Alternatives considered, not selected:** + +- Automatically group nested calls. This could simplify obtaining an overall + total, but requires rules about which calls belong together and how observation + context propagates, and could include unrelated work. +- Add no relationship metadata. This would keep the API smaller, but observation + consumers would have to reconstruct continuation relationships themselves. + +**Implementation planning:** place the minimal parent reference and preserve +ordinary per-request observation behavior for every continuation. Broader scoped +observation facilities or aggregate reporting require a separate demonstrated +need. The additional-turn budget continues to bound continuation executions, +not all provider calls performed inside those executions or application callbacks. ### 6. Where Should Continuation Inspection Live? @@ -483,8 +512,11 @@ coverage should establish: - Independent branches retaining their own history and outcomes. - Assessment calls against the same backend at capacity one, without capacity retained by the coordinator blocking them. -- Associated validator usage and shared initial usage reported without - double-counting, including incomplete usage and operational failures. +- Per-run usage and failure diagnostics available through ordinary observations, + including incomplete usage. Consumer examples explicitly group callback calls + and count shared initial work once, without coordinator-provided totals. +- Immediate parent relationships distinguish successive turns and independent + branches using existing run identities. These are planning targets, not new implemented guarantees. Keep additional mechanisms and constraints tied to the selected decisions and demonstrated