Record per-run accounting and continuation relationships
This commit is contained in:
@@ -54,9 +54,10 @@ The proposed capabilities are:
|
||||
|
||||
Continuation has direct value by removing consumer capture, reconstruction,
|
||||
prefix comparison, and history assembly. The coordinator centralizes turn
|
||||
progression, execution relationships, cancellation, budgets, and diagnostics.
|
||||
Its value should come from removing that repeated lifecycle work, beyond merely
|
||||
replacing a consumer's loop counter.
|
||||
progression, cancellation, and budgets while reusing ordinary execution
|
||||
observations. Captured exchanges supply the minimal parent relationship selected
|
||||
in Decision 5. Its value comes from removing repeated lifecycle work, beyond
|
||||
merely replacing a consumer's loop counter.
|
||||
Audita would continue owning candidate selection, domain validation, feedback
|
||||
meaning, branch scheduling, accepted sibling proposals, and transcript changes.
|
||||
Its domain-specific repair code would not move wholesale into PromptKit.
|
||||
@@ -178,9 +179,9 @@ interpret the purpose of that work.
|
||||
**Implementation planning:** select exact request/stop and outcome types under
|
||||
Decision 4's accepted response delivery, turn-budget consumption, zero-turn
|
||||
behavior, cancellation, and minimal error/result semantics.
|
||||
Execution association and usage accounting belong to Decision 5. The sketches
|
||||
below illustrate the selected responsibility split; they do not settle the
|
||||
remaining public contracts.
|
||||
Decision 5 records the selected execution association and accounting approach.
|
||||
The sketches below illustrate the selected responsibility split; they do not
|
||||
settle the remaining public contracts.
|
||||
|
||||
#### Illustrative Consumer Sketches
|
||||
|
||||
@@ -270,7 +271,8 @@ already receive each completed response through the callback. Completed work
|
||||
must not be mistaken for application acceptance of its content.
|
||||
The concrete interface should let both examples use the same coordination
|
||||
mechanism without manual message-history assembly or per-turn lifecycle
|
||||
bookkeeping. Callback model-call association remains part of Decision 5.
|
||||
bookkeeping. Calls made inside callbacks retain independent observations;
|
||||
applications explicitly group them as selected in Decision 5.
|
||||
|
||||
### 4. How Are Application Next Actions And Execution Outcomes Represented?
|
||||
|
||||
@@ -386,8 +388,9 @@ Failure-stage fields, additional run-ID references, a separate failure
|
||||
classification hierarchy, and aggregate reporting are not part of this minimal
|
||||
termination contract. They require a demonstrated consumer need before addition.
|
||||
This does not remove fields from existing response values or observation APIs.
|
||||
Decision 5 separately evaluates accounting, and Decision 6 inspection, without
|
||||
presuming that either needs to expand the coordinator's termination result.
|
||||
Decision 5 reuses ordinary run accounting without expanding this result.
|
||||
Decision 6 separately evaluates inspection without presuming that it needs to
|
||||
expand the coordinator's termination result.
|
||||
|
||||
Audita retains its own meanings for acceptance, rejection, withdrawal, and
|
||||
assessment unavailability. A normal stop does not assert any of those outcomes.
|
||||
@@ -408,33 +411,59 @@ result.
|
||||
|
||||
### 5. How Are Related Executions And Usage Accounted For?
|
||||
|
||||
**Status: open.** Define accounting for the initial exchange, each continuation,
|
||||
its structural repairs and transport retries, and model calls made by
|
||||
application callbacks, both for standalone continuations and coordinated turns.
|
||||
**Status: accepted; exact placement of relationship metadata remains for
|
||||
implementation planning.** Reuse ordinary per-run accounting, expose a minimal
|
||||
parent relationship for continuations, and leave grouping of other calls and
|
||||
overall totals to applications. This applies to both standalone continuations
|
||||
and coordinated turns.
|
||||
|
||||
The current [observation boundary](../internal/observation.md) keeps nested
|
||||
invocations independent. A callback's arbitrary model calls cannot simply be
|
||||
assumed to belong to the enclosing interaction's totals.
|
||||
|
||||
**Recommendation:** preserve independent run accounting and add explicit
|
||||
relationships between exchanges, branches, sequences, and associated runs.
|
||||
Choose a small association mechanism, such as a scoped observation facility or
|
||||
explicit execution references, so consumers need not rebuild lifecycle tracking.
|
||||
Shared initial work should be referenced by branches and counted once when
|
||||
reporting the combined operation. Report available usage with its existing
|
||||
presence/completeness meaning; do not promise complete cost accounting.
|
||||
**Accepted per-run accounting:** each continuation reports its own usage and
|
||||
ordinary execution observations, including the existing distinctions between
|
||||
transport attempts, structural repairs, and available failure-path diagnostics.
|
||||
Keep existing usage presence/completeness semantics. The coordinator adds no
|
||||
second accounting system or cumulative totals to its termination result.
|
||||
|
||||
**Alternative:** have PromptKit automatically capture every nested run during an
|
||||
application callback. This minimizes explicit association but can include
|
||||
unrelated work and changes the existing invocation-isolation boundary.
|
||||
Another simpler option is to report continuation usage only and let applications
|
||||
attach callback accounting; that leaves more work downstream and needs checking
|
||||
against Audita's stated reporting requirement.
|
||||
An initial exchange A that produces branches B and C contributes its usage only
|
||||
to A. B and C do not inherit that usage. Applications reporting the combined work
|
||||
can count A, B, and C once each. Usage reported for sending captured history again
|
||||
in B or C belongs to those new executions and still counts. Reusing content does
|
||||
not eliminate newly reported usage. Existing observation totals must not be
|
||||
added to the attempt events they already summarize.
|
||||
|
||||
**Details to settle:** association of concurrent callback calls, observer
|
||||
composition, failure-path diagnostics, and which totals include which child
|
||||
executions. A turn budget would bound continuation attempts, not all provider
|
||||
calls performed by structural repair, transport retries, or application work.
|
||||
**Accepted relationship metadata:** retain the originating run identity with
|
||||
each captured exchange and expose the immediate parent relationship in
|
||||
continuation metadata. Use existing run identifiers without introducing mandatory
|
||||
conversation, branch, or sequence identifiers. This supplies information that
|
||||
PromptKit knows directly without requiring consumers to reconstruct which
|
||||
exchange each continuation extends. Its exact public placement remains for
|
||||
implementation planning and need not enlarge the coordinator's termination
|
||||
result.
|
||||
|
||||
**Accepted application boundary:** model calls made inside a callback remain
|
||||
independently observed and accounted for. PromptKit does not infer that they
|
||||
belong to a continuation or aggregate merely because they execute inside its
|
||||
callback. Applications explicitly group those existing observations when useful,
|
||||
including concurrent calls. Audita retains assembly of its overall usage report;
|
||||
PromptKit supplies execution facts without owning an application-wide reporting
|
||||
model or promising complete cost accounting.
|
||||
|
||||
**Alternatives considered, not selected:**
|
||||
|
||||
- Automatically group nested calls. This could simplify obtaining an overall
|
||||
total, but requires rules about which calls belong together and how observation
|
||||
context propagates, and could include unrelated work.
|
||||
- Add no relationship metadata. This would keep the API smaller, but observation
|
||||
consumers would have to reconstruct continuation relationships themselves.
|
||||
|
||||
**Implementation planning:** place the minimal parent reference and preserve
|
||||
ordinary per-request observation behavior for every continuation. Broader scoped
|
||||
observation facilities or aggregate reporting require a separate demonstrated
|
||||
need. The additional-turn budget continues to bound continuation executions,
|
||||
not all provider calls performed inside those executions or application callbacks.
|
||||
|
||||
### 6. Where Should Continuation Inspection Live?
|
||||
|
||||
@@ -483,8 +512,11 @@ coverage should establish:
|
||||
- Independent branches retaining their own history and outcomes.
|
||||
- Assessment calls against the same backend at capacity one, without capacity
|
||||
retained by the coordinator blocking them.
|
||||
- Associated validator usage and shared initial usage reported without
|
||||
double-counting, including incomplete usage and operational failures.
|
||||
- Per-run usage and failure diagnostics available through ordinary observations,
|
||||
including incomplete usage. Consumer examples explicitly group callback calls
|
||||
and count shared initial work once, without coordinator-provided totals.
|
||||
- Immediate parent relationships distinguish successive turns and independent
|
||||
branches using existing run identities.
|
||||
|
||||
These are planning targets, not new implemented guarantees. Keep additional
|
||||
mechanisms and constraints tied to the selected decisions and demonstrated
|
||||
|
||||
Reference in New Issue
Block a user