From 72955aaaa31ec17dba2946fb403fc0a4364e1eee Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 13 Nov 2026 17:32:22 +0000 Subject: [PATCH] Rewrite conversation roadmap around the target capabilities --- docs/roadmap/conversation-continuation.md | 1015 +++++++-------------- 1 file changed, 354 insertions(+), 661 deletions(-) diff --git a/docs/roadmap/conversation-continuation.md b/docs/roadmap/conversation-continuation.md index d2cb3b4..559b6cc 100644 --- a/docs/roadmap/conversation-continuation.md +++ b/docs/roadmap/conversation-continuation.md @@ -1,258 +1,314 @@ -# Conversation Continuation And Application-Directed Interaction +# Conversation Continuation And Execution Groups -Status: continuation, coordinator, and optional execution-group behavioral -decisions accepted; ready for API and staged implementation planning. Execution -groups remain independently deliverable. No implementation is authorized by this -document. Exact public types and signatures remain for implementation planning. +Status: target behavior accepted; ready for API and staged implementation planning. +No implementation is authorized by this document. Exact public types and +signatures remain to be designed. Execution groups are independently deliverable. -## North Star: Content-Neutral, Generally Useful Capabilities +## Purpose And Responsibility Boundary -**Accepted direction:** evaluate each capability for its usefulness across -different downstream applications. PromptKit must be able to implement it and -explain its value without assuming or inferring the meaning of the content -passed through prompts and responses. +PromptKit will let applications continue logical conversations, optionally +coordinate successive turns, and summarize explicitly selected executions. +These capabilities remove repeated history reconstruction, turn lifecycle work, +and observation aggregation without interpreting prompt or response content. -PromptKit can own conversation and execution mechanics: captured messages, -continuations, independent branches, execution relationships, budgets, -cancellation, observations, and usage. Applications own the interpretation of -responses and their reasons for choosing the next action. Repair, analysis, -clarification, and branching exploration are examples of consumer workflows -that could use the same mechanisms. +[Audita's request](audita-request.md) supplies consumer evidence and acceptance +scenarios. Repair, analysis, clarification, and branching exploration should all +use the same content-neutral capabilities. New concepts and names must be useful +without translating an application's domain into repair terminology. This does +not rename or expand PromptKit's existing structural-output repair functionality. -Decision 3 selects an optional sequential coordinator built on independently -usable continuation primitives. Its value is generally useful execution and -conversation management, rather than implementation of Audita's repair loop. -It executes application-directed interactions without needing to understand why -the application requests another turn or stops. Applications retain orchestration -across branches; that initial scope does not prohibit future capabilities. +| Capability | PromptKit owns | Application owns | +| --- | --- | --- | +| Captured exchanges and continuation | Logical history, reusable branches, ordinary follow-up execution, exchange details and parent metadata | Which exchange to continue and which request to execute | +| Optional sequential coordinator | One branch's turn progression, additional-turn budget, callback delivery and cancellation checks | Response interpretation, next action, branch scheduling and broader failure policy | +| Optional execution groups | Explicitly selected observations, deduplication and execution-fact summaries | Membership selection, domain identifiers, final-report timing and presentation | -Apply two questions to proposed public concepts and implementation machinery: +Continuation is independently usable. The coordinator builds on it; applications +can instead use their own loops. Groups work with ordinary runs as well as +continuations and require neither capture nor a coordinator. Selecting a group +does not change scheduling, capacity, cancellation, retries, or validation. -1. Can their value be explained using conversation and execution concepts alone? -2. Can different applications use them without translating their domain into - repair-specific concepts? +The scope is in-process use. Persistence, restartable workflows, cross-process +resumption, a conversation registry, batch reconciliation, and a branching-workflow +scheduler are outside this feature. Applications schedule branches, join results, +and decide whether one branch's failure should affect other work. This initial +scope is not a permanent prohibition on future capabilities. -Audita supplies concrete evidence of need and acceptance examples; its workflow -does not define PromptKit's abstraction. Terms such as rejected candidate, -accepted revision, and withdrawal belong to the consuming application. This -direction concerns the new capabilities and does not rename or expand -PromptKit's existing structural-output repair functionality. +## Captured Exchanges And Continuation -## Purpose And Scope +### Capture And Availability -This roadmap records PromptKit's design questions arising from -[Audita's feature request](audita-request.md). That request owns the consumer -workflow and requested acceptance scenarios; this document owns the proposed -PromptKit direction, discussion order, and eventual implementation sequencing. +Capture is opt-in per ordinary or prepared execution. Ordinary execution returns +the exchange alongside its usual result. A prepared handle retains the capture +choice; executing that exact prepared snapshot creates the exchange. Preparation +alone creates no completed exchange. Continuation implies capture so its result +can itself be continued. -The proposed capabilities are: +Capture the request actually executed, including the complete rendered messages +with their content, roles, and cache metadata, followed by the full final +assistant content returned by the run. Consumers must not reconstruct the request +or pair a response with separately prepared data. “Full” means content supported +by PromptKit, not a new raw provider-envelope API. -- Capture a rendered request and its final assistant content, then continue it - using ordinary configured prompts without reconstructing the original request. -- Allow independent continuations from one captured exchange. -- Provide an optional sequential coordinator for bounded looping using - application-supplied next actions and follow-up requests. -- Provide optional execution groups for explicitly selected runs and summaries - of their available execution facts, independently of conversation continuation. - -Continuation has direct value by removing consumer capture, reconstruction, -prefix comparison, and history assembly. The coordinator centralizes turn -progression, cancellation, and budgets while reusing ordinary execution -observations. Captured exchanges supply the minimal parent relationship selected -in Decision 5. Its value comes from removing repeated lifecycle work, beyond -merely replacing a consumer's loop counter. -Audita would continue owning candidate selection, domain validation, feedback -meaning, branch scheduling, accepted sibling proposals, and transcript changes. -Its domain-specific repair code would not move wholesale into PromptKit. - -The request needs in-process use. Persistent sessions, restartable workflows, -conversation storage, and batch-item reconciliation are outside its scope. -The existing [prepared execution handle](../../prepared_execution.go) remains a -single-use execution capability; continuation needs its own semantics. - -## Decisions In Discussion Order - -### 1. What Does A Captured Exchange Contain? - -**Status: accepted; exact capture API remains for implementation planning.** Capture -represents the logical conversation. Structural-output repair attempts and -corrective instructions are internal execution details, not conversation turns. - -**Accepted approach:** capture the complete rendered request, preserving message -content, roles, and cache metadata, followed by the final assistant content -returned by the run. When structural repair succeeds, retain only the final -corrected assistant content, excluding intermediate candidates and corrective -instructions from the logical conversation. This supplies reusable context for -subsequent turns while keeping transport attempts and structural repair separate -from application turns. It also matches Audita's requested history. - -**Accepted capture availability:** when capture is requested, follow the ordinary -distinction between a completed result and an operational error: +Capture represents logical conversation history. Internal structural-repair +instructions and intermediate candidates are excluded; only the final returned +assistant content follows the original request. Existing attempt observations +remain separate from conversation content. | Execution outcome | Capture behavior | | --- | --- | -| Completed response that satisfies its output contract | Capture the logical exchange. | -| Completed response with failed structural validation, including repair exhaustion | Capture the final returned response; preserve the failed validation status in the ordinary result. | -| Operational error with no completed result | Produce no new exchange; the preceding exchange remains usable. | -| Completed response followed by cancellation before coordinator callback delivery | Preserve the undelivered response under Decision 4's accepted optional-result behavior. | +| Completed response satisfying its output contract | Return the captured exchange. | +| Completed response with failed structural validation, including repair exhaustion | Return the exchange with the ordinary validation result intact. | +| Operational error with no completed result | Create no new exchange; the preceding exchange remains usable. | +| Completed response obtained before cancellation prevents coordinator delivery | Preserve that single undelivered response under the coordinator return contract. | -Capture does not imply validation success or application acceptance. The -coordinator delivers completed results with their ordinary validation status; -the application decides whether to stop or continue, potentially using another -output contract. Failed structural validation is not reclassified as an -operational error. Only the final returned content enters the logical history, -never intermediate structural-repair responses or corrective instructions. +Capture does not imply structural validity or application acceptance. A completed +response with failed validation follows the normal coordinator callback path, +subject to cancellation; it is not reclassified as an operational error. -**Accepted opt-in behavior:** support capture per execution through both ordinary -and prepared execution. Ordinary execution returns the captured exchange -alongside its usual result. For prepared execution, retain the capture choice -with the prepared handle and create the exchange when that exact execution -completes. Preparation alone produces no completed exchange because it has no -assistant response. A continuation operation implies capture so it can produce -the next reusable exchange. +### Ownership And Reuse -Construct capture from the request actually executed. Consumers must not need -to reconstruct history or pair independently prepared data with a response. -An ordinary request option is the preferred API direction; separate capture -methods are also viable. Select the precise representation during API planning. +An exchange is an immutable, reusable snapshot held by the caller. Continuing it +adds the rendered follow-up messages and final response to a new exchange, leaving +the source and independent branches unchanged. No previous templates or input +files need to be read or rendered again. -**Accepted ownership and lifetime:** a captured exchange is an immutable, -reusable snapshot held by the caller. Continuing it appends the rendered -follow-up messages and final response to produce a new exchange, leaving the -source exchange and other branches unchanged. Independent branches may share -their starting exchange. Inspection values cannot mutate retained history. -Previously rendered history does not require the original input files or -templates to be rendered again. +Another engine instance in the same process can continue the exchange, applying +its own configuration to the new request. The originating engine is not required. +This permits application engine replacement without promising serialized or +cross-process resumption. -A completed exchange retains no backend admission lease, direct credential, -or execution callback. Ordinary Go memory management handles its lifetime when -references are released; it adds no `Close` or `Discard` lifecycle. The existing -single-use prepared execution lifecycle remains separate. +Completed exchanges retain no backend admission lease, direct credential, or +execution callback. Ordinary Go memory management controls their lifetime; there +is no new `Close` or `Discard` lifecycle. The existing +[prepared execution handle](../../prepared_execution.go) remains a separate, +single-use execution capability. -**Accepted engine independence:** another engine instance in the same process -may continue a captured exchange. The receiving engine resolves the new prompt, -profile, output contract, and credentials under Decision 2's rules. It does not -need the originating engine to rerender prior messages. This permits retaining -conversations while an application rebuilds its engine with new configuration. -It does not imply persistence, restartable sessions, serialization for later -resumption, or cross-process portability. +### Exchange Details -**Alternative considered, not selected:** retain the corrective messages and -responses that led to the final output. This provides more execution detail, -but increases retained content and carries internal repair instructions into -later model context. -Fine-grained execution tracing is outside this feature's purpose. Existing -attempt and usage observations remain separate from logical conversation -content; this decision adds no requirement to capture internal repair text. +Provide explicit access to the held exchange's logical message history, including +its final assistant content, originating run identity, and immediate parent +reference. Reuse existing run identifiers; no mandatory conversation, branch, or +sequence identifiers are introduced. -**Additional alternatives considered, not selected:** +Follow the general pattern of `PreparedExecution.Details()`. Details must be +generically renderable through ordinary encoders. Reading or modifying returned +values must not affect retained history or another branch. Inspection reads the +snapshot without rendering, loading inputs, or making provider calls. -- Capture only responses that satisfy their output contract. This would offer a - validated-output condition but unnecessarily restrict continuing from otherwise - available conversation content. Applications retain that choice. -- Bind exchanges to their originating engine. Prepared executions need ownership - of execution-specific state, but captured conversation data has no equivalent - requirement under the selected fresh-execution settings model. -- Use a mutable conversation object. This would make appending convenient but - require explicit copying or forking to preserve independent branches. Immutable - snapshots fit the selected branch behavior directly. +Existing results and preparation details continue to supply ordinary execution +information. Exchange inspection does not create a second comprehensive report, +a result archive, or an engine-level conversation lookup API. `InspectPrompt`, +`InspectProfile`, and `InspectBackend` keep their resource-inspection boundary. +Captured message content remains separate from metadata-only observations. -**Implementation planning:** select the opt-in API and exchange representation -within these behaviors. The request's “full final assistant response” refers to -content supported by PromptKit, without adding a raw provider-envelope API. +## Optional Sequential Coordinator -### 2. How Are Follow-Up Execution Settings Selected? +The coordinator starts from a captured exchange and executes one branch using +an application callback, a caller-selected additional-turn budget, and context. +The callback selects a new ordinary request, stops, or returns an error. It can +keep application state in a closure; PromptKit defines no domain-state or +feedback schema. -**Status: accepted.** A continuation supplies existing conversational context -to a normally configured new execution. Conversation history does not introduce -another source of execution-setting inheritance. +### Progression And Budget -**Accepted approach:** preserve conversation content while resolving the new -profile and output contract through the follow-up prompt and ordinary request -overrides. The follow-up can therefore use a different model and response -schema without modifying the original exchange. Explicit request overrides use -ordinary precedence, including complete replacement of the output contract -when supplied. Reuse existing selection and precedence rules. +With an active context, invoke the initial callback once, even with a zero-turn +budget. It receives the starting exchange and remaining budget; the starting +exchange is not a newly delivered response. Existing application findings can +be used without repeating earlier assessment. -In particular, prefer the follow-up prompt's default profile when no explicit -profile is supplied, with the ordinary missing-profile behavior if neither is -available. Do not retain the original direct credential merely to retain -conversation history; resolve credential supply for the new execution using -the ordinary credential rules. +After each continuation, deliver its completed response once through the same +callback before starting another execution, including the final permitted +response. Internal retries and structural repairs do not create extra callbacks. -Select the session ID using the new request's explicit value or the follow-up -prompt's session template under the existing rules. Do not implicitly inherit -the preceding execution's session ID. An application can explicitly supply the -same session ID across turns. Conversation relationships are recorded separately -and do not depend on a shared session ID. +| Callback action | Coordinator behavior | +| --- | --- | +| Stop | Return application-directed completion, including after the last permitted response. | +| Continue with budget remaining | Start the selected continuation. | +| Continue without budget remaining | Return exhaustion without executing the requested turn. | +| Error | Stop and propagate the error, subject to context checks. | -**Alternative considered, not selected:** inherit the originating execution's -profile and settings unless overridden. This is convenient for repeated turns -against the same model, but adds another precedence source and can make a -follow-up prompt's defaults surprising. Inheriting an old output contract is -especially unsuitable when successive turns expect different response shapes. -An explicit inheritance option is not selected for this feature; reconsider it -only if ordinary selection leaves a demonstrated usability gap. +The budget permits at most N additional continuation executions to start. +Starting a continuation consumes one turn even if preparation or generation +fails. The initial exchange and callback invocations consume no turns; internal +transport retries and structural repairs stay within one turn. The first +operational error stops coordination, without refunds or recovery machinery. -**Implementation planning detail:** specify message assembly order for captured -history, follow-up definition messages, and caller-appended messages, preserving -authored roles. This does not reopen the accepted execution-setting precedence. +Using the full budget alone does not mean exhaustion: exhaustion means the +application requested more work than the budget permits. The budget does not +bound provider attempts, cost, or work inside callbacks. Applications wanting +no callback work can skip the coordinator entirely. -### 3. Should PromptKit Coordinate Turns, And Where Does Application Work Run? +### Cancellation And Failures -**Status: accepted; exact public types remain for implementation planning.** Provide -independently usable continuation primitives and an optional sequential -coordinator for one branch. The application selects each next request or decides -to stop. PromptKit manages turn progression and execution bookkeeping under the -caller's chosen additional-turn budget and context. +Check context immediately before every callback, including the initial callback. +If cancellation is already observed, skip delivery and return the context error +with its identity preserved. Retain a completed response already obtained but +not yet delivered; do not recover raw output from an underlying failed call that +returned no completed result. -Applications can create several independent continuations or coordinators from -the same exchange. Scheduling branches, joining results, and deciding whether a -branch failure should stop other work remain application responsibilities in -this initial scope. Consumers can also use continuation primitives in their own -loops without adopting the coordinator. +Callbacks receive the same context and cooperate with cancellation. Check it +again after the callback returns and before starting another execution. +Cancellation may race with invocation; PromptKit does not forcibly terminate +callback code or promise callback delivery despite cancellation or panic. -The current [run flow](../internal/runner.md#run-flow) retains logical admission -through structural validation and repair. Releasing an active generation permit -alone does not release that admission lease. +An execution or callback error stops coordination. Preserve ordinary Go error +identities, including `errors.Is`/`errors.As`. Do not automatically replay the +failed continuation or callback. Ordinary transport retries and structural repair +remain execution-internal. Earlier deliveries and application work remain intact; +the application owns any broader consequences. -**Accepted execution lifecycle:** compose completed executions. Finish a turn -and release its logical admission lease before invoking application work. Each -later turn receives ordinary admission. The coordinator holds no backend -capacity while awaiting application work. Callbacks can inspect responses, -consult application data, wait for application input, or make model calls against -the same engine and backend, including at capacity one. PromptKit need not -interpret the purpose of that work. +### Minimal Return Contract -**Alternatives considered, not selected:** +Return only: -- Provide only continuation and observation primitives. This would keep the - public API smaller, but leave repeated lifecycle and accounting coordination - downstream. Independent primitive use remains supported alongside the selected - coordinator. -- Provide a coordinator for an entire branching tree. This could manage more - work, but introduces scheduling, concurrency, joining, and failure-policy - questions without demonstrated shared answers across consumers. Applications - can build those workflows from the selected branch-level capabilities. +- An ordinary Go error for execution failure, callback failure, or cancellation. +- A small normal outcome distinguishing application-directed stop from budget + exhaustion; neither outcome is an error or a claim about content acceptance. +- An optional single completed response when cancellation prevented callback + delivery, reusing existing response types where practical. -**Implementation planning:** select exact request/stop and outcome types under -Decision 4's accepted response delivery, turn-budget consumption, zero-turn -behavior, cancellation, and minimal error/result semantics. -Decision 5 records the selected execution association and accounting approach. -The sketches below illustrate the selected responsibility split; they do not -settle the remaining public contracts. +Delivery means callback invocation, not successful processing or acceptance. If +the callback returns an error, its response is already delivered and is neither +returned as undelivered nor replayed. Previously delivered responses are not +archived in the termination result. No additional failure hierarchy, failure-stage +fields, run-ID references, or aggregate reporting is added to this contract. +Existing response and observation fields remain available; group summaries are +obtained separately. -#### Illustrative Consumer Sketches +## Shared Execution And Accounting Rules -These are pseudocode, not implemented Go APIs or selected exported names. -`coordinate`, `Continue`, `Stop`, and callback state fields are illustrative. -`request` represents an ordinary PromptKit request, with the selection rules -accepted in Decision 2. Each sketch starts from an already captured exchange. -The callback can keep application state in a closure; PromptKit need not define -a universal domain-state or feedback schema. +### Follow-Up Configuration -**Analysis loop:** the application examines the latest response and selects a -question or decides that its investigation is complete. +A continuation is an ordinary new execution supplied with captured history. +Resolve its prompt, profile, credentials, output contract, and request overrides +using the receiving engine's existing rules. In particular: + +- Use an explicit request profile or the follow-up prompt's default profile, + with ordinary missing-profile behavior when neither is available. +- A supplied output contract completely replaces the default; do not merge it + with or inherit the preceding execution's contract. +- Resolve credentials for the new execution; retained history does not preserve + the original direct key. +- Select session ID from the new request or follow-up prompt's session template. + Do not implicitly inherit it. Applications may explicitly reuse a session ID; + ancestry does not depend on one. + +History introduces no additional execution-setting precedence. The follow-up +can change model or response schema without altering the original exchange. +Exact assembly order for captured history, follow-up definition messages, and +caller-appended messages remains an API-planning detail; preserve authored roles. + +### Capacity And Application Work + +Compose completed executions. Finish each turn and release its logical admission +lease before calling application code. Releasing only the active generation +permit is insufficient under the existing [run flow](../internal/runner.md#run-flow). +Each later execution acquires ordinary admission. + +The coordinator holds no backend capacity while awaiting callback work. +Callbacks may inspect responses, consult data, wait for application input, or +call the same engine and backend, including when backend capacity is one. + +### Ancestry And Per-Run Accounting + +Each continuation retains its own originating run ID and exposes its immediate +parent exchange's run ID. Parent metadata describes history provenance; group +membership describes application-selected accounting. Neither implies the other. +A validator can join a continuation's accounting group without extending its +conversation. + +Reuse ordinary per-run usage and observations, including attempt, retry, repair, +failure-path, and usage-presence/completeness semantics. Preserve the existing +[observation boundary](../internal/observation.md): arbitrary nested calls are +independent executions. Do not infer membership from a callback's purpose or +context. + +If initial run A supplies branches B and C, its usage belongs only to A. B and C +do not inherit that cost. Usage reported for resending captured history in B and +C still belongs to those new runs. Count selected runs once each; never add +terminal totals to the attempt events they already summarize. + +## Optional Execution Groups + +### Explicit Membership And Observer Composition + +Use an observer-compatible collector attached explicitly to selected executions. +Prepared requests retain the selection; membership starts when execution emits +observations. Preparation alone, discarded handles, and failed claims do not +enroll runs through observations. + +Coordinator selection covers its continuation runs. Callback-initiated calls +require their own explicit selection. Neither nested context nor a captured +starting exchange automatically enrolls another run. + +Support an application's existing diagnostic observer alongside one or more +collectors through a small observer-composition helper. Preserve synchronous +delivery and existing panic behavior. Do not introduce asynchronous queues, +panic recovery, or guaranteed delivery to later observers after an earlier one +panics; interrupted delivery can leave accounting incomplete. + +A run may explicitly belong to multiple independent groups. Each group counts +its run identity once. Applications obtain operation-level and overall views by +attaching the corresponding collectors to each selected run. Overlapping totals +are not generally additive; no group hierarchy, inherited membership, or merging +facility is needed. + +### Concurrent Snapshots And Lifetime + +Provide detached, internally consistent snapshots from a caller-held group while +selected executions and snapshot reads may occur concurrently. Reading or +modifying a snapshot must not change the collector or another snapshot. There +is no mandatory close, seal, or wait lifecycle. Applications own scheduling and +decide when to read or discard the group. + +Distinguish observed runs with terminal observations from those without them. +Missing terminal observations mean incomplete accounting; the group does not +infer whether work is active, delivery was interrupted, or another cause applies. +It promises no terminal delivery beyond ordinary observation semantics. + +“All observed members have terminal observations” does not mean the application +operation is complete. More runs can join later, and PromptKit cannot know whether +the application selected all relevant calls. The application chooses when a +snapshot represents its final report. + +### Summary Contents + +The initial summary exposes: + +- Member run identities and terminal-observation coverage. +- Existing provider-attempt, transient-retry, and structural-repair counters. +- Available token totals with per-field known/completeness information. + +Aggregate one terminal observation per run. These observations already summarize +retries, repairs, and available failure-path usage, including executions returning +no ordinary result. Do not reconstruct totals from attempts. + +Keep terminal coverage distinct from usage completeness. A terminal observation +may itself have incomplete usage. Two finished runs with fully reported usage +provide a useful known subtotal, but a third observed run lacking a terminal +event leaves group accounting incomplete. Preserve existing usage semantics and +expose missing coverage separately; unavailable evidence must not become known +zero. + +Live attempt-level totals and duration aggregation are omitted from the initial +implementation. Applications can use existing events for progress displays. If +durations are later added, summed execution durations must be distinguished from +elapsed group time. Groups supply execution facts, not monetary cost, application +outcomes, or an application-wide reporting framework. Applications retain domain +mappings, interpretation, and report assembly. + +## Consumer Sketches + +These are pseudocode, not selected exported names or implemented Go APIs. +Each begins with an already captured exchange and uses ordinary follow-up request +selection. Both omit application-specific error handling, including checks for +failed structural validation before schema-specific decoding. + +### Analysis Loop + +The application records responses and selects its next question or decides that +its investigation is complete. PromptKit retains history and executes the turns. ```text analysis = application.start_analysis(initial_exchange) @@ -277,14 +333,10 @@ termination, error = coordinate( application.present_analysis(analysis, termination, error) ``` -PromptKit preserves the history and performs each selected execution. It does -not decide whether the response answered the question, whether the evidence is -sufficient, or what to investigate next. The application callback may consult -other sources or call the engine before returning its next action. +### Audita Branch -**Audita branch:** the application starts with one already-assessed correction -and its rejection findings. It interprets and assesses each new response; the -coordinator uses the same generic controls as the analysis loop. +Audita starts with an already-assessed candidate and existing findings. Only +Audita interprets acceptance, withdrawal, or assessment unavailability. ```text branch = audita.branch(candidate, existing_findings) @@ -314,450 +366,91 @@ termination, error = coordinate( audita.record_branch_outcome(branch, termination, error) ``` -The first callback uses existing findings, so initial batch validation is not -repeated. `audita.assess` may make model calls after the turn's admission lease -has been released. Only Audita decides that a correction is accepted, withdrawn, -or unavailable, and retains those meanings in its own state. Further corrections -can start separate coordinators from the same initial exchange; accepted sibling -corrections and other branches remain with Audita. - -Both sketches omit application-specific error handling, including checks for -failed structural validation before schema-specific decoding, and depict normal -turn progression. The final permitted response follows the same callback path as -earlier responses, as accepted in Decision 4. That decision also defines zero-turn -budgets, cancellation around callback delivery, and a minimal termination result. -The sketches do not imply an archive of responses in that result: applications -already receive each completed response through the callback. Completed work -must not be mistaken for application acceptance of its content. -The concrete interface should let both examples use the same coordination -mechanism without manual message-history assembly or per-turn lifecycle -bookkeeping. Calls made inside callbacks retain independent observations; -applications explicitly group them as selected in Decision 5. - -### 4. How Are Application Next Actions And Execution Outcomes Represented? - -**Status: accepted; exact public types remain for implementation planning.** -The coordinator uses content-neutral next actions and distinguishes -application-directed completion from exhaustion. - -**Accepted normal flow:** deliver each completed response once through the -ordinary application callback, including the last permitted response. If the -callback returns `Stop`, report application-directed completion. If it requests -another turn and none remain, report exhaustion without executing that request. -Using the entire budget does not itself imply exhaustion; exhaustion means the -application requested additional work that the budget would not permit. - -This gives consumers one response-processing path, including any application -validation, without a separate path for the final response. Internal transport -retries and structural repairs do not produce additional conversation-response -callbacks. Cancellation and operational errors can interrupt normal delivery as -specified below. `Stop` and other API spellings remain illustrative until public -types are selected. - -**Alternative considered, not selected:** stop immediately when the turn budget -is used and expose the final response only in the report. This avoids another -callback but requires separate consumer processing of the final response. - -**Accepted zero-turn behavior:** with an active context, invoke the initial -callback once even when the additional-turn budget is zero. The callback receives -the starting exchange and knows that no additional turns remain; this is not -delivery of a newly generated response. It can use existing application state -without repeating earlier assessment. - -- `Stop` produces application-directed completion. -- Requesting another turn produces exhaustion without executing the request. -- Returning an error produces an operational failure, subject to the - cancellation checks below. - -Zero bounds coordinator-driven continuation executions, not work inside the -application callback. The callback can still consult data or make model calls. -Applications wanting to disable all such work can skip invoking the coordinator. -The alternative of treating zero as an immediate no-op with no callback is not -selected: it would introduce special behavior and would not distinguish an -already-complete interaction from one requesting more work. - -**Accepted turn-budget consumption:** at most N additional continuation -executions may be started. The first operational error ends coordination. -Starting a continuation consumes one turn even if preparation or generation -subsequently fails. Callback invocations do not consume turns, and internal -transport retries and structural repairs remain within the same turn. The -starting exchange is outside the additional-turn budget. Requesting a turn -when none remain produces exhaustion without starting an execution. - -For example, with a three-turn budget, a preparation failure in the second -continuation returns that error and ends coordination. It does not start a -third continuation or report exhaustion. The budget does not measure provider -calls, work performed inside application callbacks, or cost. - -Counting only completed continuations was considered and not selected. Because -operational errors already stop coordination, that alternative would primarily -change bookkeeping. Use the straightforward executions-started interpretation -without introducing public counters, budget refunds, or recovery machinery for -this purpose. - -**Accepted cancellation behavior:** check the caller's context immediately -before callback invocation, including the initial callback. If cancellation is -already observed, do not invoke the callback; return the context error with its -identity preserved. If the coordinator already obtained a completed response -that has not been delivered to the callback, preserve that single response in -the optional result described below. This does not imply application processing -or acceptance, or require recovering a response that a failed underlying -execution did not return. - -Cancellation can race with callback invocation. The callback receives the same -context and is responsible for cooperating with cancellation during its work. -PromptKit checks the context again after the callback returns and before starting -another execution. It does not forcibly terminate callback code. Thus ordinary -response delivery is once per completed turn during normal progression, while -cancellation can interrupt that progression. Preserving available completed work -on interruption does not create a separate final-response processing path during -normal budget exhaustion. - -The alternative of invoking the callback after cancellation with a canceled -context is not selected. It could help callbacks that only record results, but -callbacks may perform substantial application work or model calls; PromptKit -cannot assume they are merely bookkeeping. - -**Accepted operational-failure behavior:** each continuation is an ordinary -execution supplied with captured conversation history. Applications receive -completed responses through the callback before another continuation begins. -An execution or callback error stops coordination and propagates to the caller, -preserving useful Go error identities. PromptKit does not automatically replay -the failed continuation or callback. Existing transport retries and structural -repair remain within ordinary execution. Earlier callback deliveries and -application work remain intact; applications decide whether the failure affects -other branches or the overall workflow. - -**Accepted minimal return contract:** provide only the following information: - -- An ordinary Go error for execution failures, callback failures, and - cancellation, preserving `errors.Is`/`errors.As` behavior. -- A small outcome value distinguishing application-directed stopping from - budget exhaustion on normal termination. These outcomes are not errors. -- An optional completed response when cancellation prevented its delivery to - the callback. Reuse existing response types where practical. - -Delivery means the callback was invoked with the response; it does not imply -the application finished processing or accepted it. If that callback returns -an error, the response is not classified as undelivered or automatically -delivered again. The coordinator returns no archive of previously delivered -responses. A small optional result accommodates the undelivered-response case; -exact Go declarations remain for implementation planning. - -Failure-stage fields, additional run-ID references, a separate failure -classification hierarchy, and aggregate reporting are not part of this minimal -termination contract. They require a demonstrated consumer need before addition. -This does not remove fields from existing response values or observation APIs. -Decision 5 reuses ordinary run accounting without expanding this result. -Decision 6 exposes captured-exchange details without expanding the coordinator's -termination result. - -Audita retains its own meanings for acceptance, rejection, withdrawal, and -assessment unavailability. A normal stop does not assert any of those outcomes. - -**Alternative considered, not selected for normal termination:** use callback -errors to signal ordinary termination as well as operational failure. This can -reduce the number of result types, but makes normal control flow harder to -distinguish from failure. Normal stopping uses an explicit action, while failures -use ordinary Go errors. A broader structured termination report was also -considered and not selected: its additional metadata is unnecessary for the -demonstrated minimal contract. - -**Implementation planning:** select exact types for next actions, outcomes, and -the optional result under these accepted behaviors. As selected in Decision 1, -a completed run with failed structural validation produces a reusable exchange -when capture is requested and retains its ordinary validation result. That -result follows the normal response callback path, subject to cancellation; -the operational-error behavior does not reclassify it. - -### 5. How Are Related Executions And Usage Accounted For? - -**Status: accepted; exact placement of relationship metadata remains for -implementation planning.** Reuse ordinary per-run accounting, expose a minimal -parent relationship for continuations, and let applications select accounting -membership explicitly. The separable execution-group addition below supplies -optional summaries without expanding the coordinator. This applies to both -standalone continuations and coordinated turns. - -The current [observation boundary](../internal/observation.md) keeps nested -invocations independent. A callback's arbitrary model calls cannot simply be -assumed to belong to the enclosing interaction's totals. - -**Accepted per-run accounting:** each continuation reports its own usage and -ordinary execution observations, including the existing distinctions between -transport attempts, structural repairs, and available failure-path diagnostics. -Keep existing usage presence/completeness semantics. The coordinator adds no -second accounting system or cumulative totals to its termination result. - -An initial exchange A that produces branches B and C contributes its usage only -to A. B and C do not inherit that usage. Applications reporting the combined work -can count A, B, and C once each. Usage reported for sending captured history again -in B or C belongs to those new executions and still counts. Reusing content does -not eliminate newly reported usage. Existing observation totals must not be -added to the attempt events they already summarize. - -**Accepted relationship metadata:** retain the originating run identity with -each captured exchange and expose the immediate parent relationship in -continuation metadata. Use existing run identifiers without introducing mandatory -conversation, branch, or sequence identifiers. This supplies information that -PromptKit knows directly without requiring consumers to reconstruct which -exchange each continuation extends. Its exact public placement remains for -implementation planning and need not enlarge the coordinator's termination -result. - -**Accepted application boundary:** model calls made inside a callback remain -independently observed and accounted for. PromptKit does not infer that they -belong to a continuation or aggregate merely because they execute inside its -callback. Applications explicitly select which runs belong together, including -concurrent calls, and may use the optional execution groups below to collect and -summarize those facts. This revises the earlier choice to leave all aggregation -to applications; membership and domain interpretation remain application-owned. -Audita retains assembly of its overall usage report and mappings to its domain -identifiers. PromptKit supplies execution facts without owning an -application-wide reporting model or promising complete cost accounting. - -**Alternatives considered, not selected:** - -- Automatically group nested calls. This could simplify obtaining an overall - total, but requires rules about which calls belong together and how observation - context propagates, and could include unrelated work. -- Add no relationship metadata. This would keep the API smaller, but observation - consumers would have to reconstruct continuation relationships themselves. - -**Implementation planning:** place the minimal parent reference and preserve -ordinary per-request observation behavior for every continuation. Execution -groups address the separately demonstrated need for explicit accounting across -related calls; broader reporting remains outside this scope. The additional-turn -budget continues to bound continuation executions, not all provider calls -performed inside those executions or application callbacks. - -### 6. Where Should Continuation Inspection Live? - -**Status: accepted; exact method and type shape remain for implementation -planning.** Provide explicit access to the captured exchange's existing content -and minimal identity metadata, following the general pattern of -`PreparedExecution.Details()`. - -**Accepted inspection scope:** expose the logical message history, including -the final assistant response, the exchange's originating run identity, and the -immediate parent relationship selected in Decision 5. Existing execution results -and preparation details continue supplying their ordinary information. Reuse -existing types where useful rather than creating a second comprehensive -execution report. - -Details should be generically renderable through ordinary encoders so consumers -can display them without field-by-field adapters. Reading or modifying a -returned details value must not alter retained history or another branch's -execution. Access reads captured values without rerendering templates, reloading -inputs, or making provider calls. Captured message content remains separate -from ordinary metadata-only execution observations. - -The exchange contains logical conversation history because continuation needs -it. This does not create a separate archive of execution results or expand the -coordinator's minimal termination result. `InspectPrompt`, `InspectProfile`, and -`InspectBackend` retain their configured-resource responsibilities. - -**Representation alternatives for implementation planning:** a small details -accessor on a retained exchange is the preferred pattern. A public data value -may also satisfy the accepted behavior with fewer API elements, provided its -ownership rules prevent caller mutations from affecting retained branch state. -Choose the exact representation and serialization details during API planning. - -**Alternative considered, not selected:** an engine-level conversation lookup -and inspection API. Such an API could serve a searchable conversation registry, -but this feature does not select that registry. Explicit access through an -already-held exchange satisfies the present need. Richer reporting or registry -capabilities require separate consumer justification. - -## Separable Addition: Optional Execution Groups - -**Status: all four group decisions accepted; exact public API remains for -implementation planning.** Audita's follow-up request identifies a distinct -need: a validator call can belong to the same logical operation as a continuation -without extending its conversation. -Conversation ancestry describes which exchange supplies history; accounting -membership describes which executions the application chooses to report together. -Neither relationship implies the other. - -An application creates a group for a logical operation and explicitly associates -selected ordinary runs or coordinated continuations with it. Calls made by a -validator or other callback join only when the application chooses. Groups use -existing run identities and observation/usage semantics to summarize available -execution facts. They are usable without capture or a coordinator, and continuation -and coordination remain usable without them. Selecting a group does not change -execution scheduling, capacity, cancellation, retry, or validation behavior. - -For example, an application may group a continuation and a separate validator -run while keeping their conversations independent. It may also include the -initial run shared by several branches. Replaying that run's captured history -does not enroll it or count its usage again; newly reported usage for sending -history in later executions still belongs to those later runs. - -The group does not infer content meaning, automatically capture arbitrary nested -calls, schedule work, determine application outcomes, or introduce a reporting -framework. Applications retain domain identifiers, presentation, and any broader -report assembly. Group summaries are obtained separately from the coordinator's -minimal termination result. They preserve unavailable versus explicit zero usage -and do not add terminal totals to the attempts they already summarize. - -### Group Decision 1: How Does An Application Select Membership? - -**Status: accepted.** Start with an observer-compatible collector using existing -execution observations. Applications explicitly attach the collector to selected -ordinary requests. Prepared requests retain the selection, with membership -beginning when execution emits observations; preparation alone, discarded handles, -and failed claims do not enroll a run through observations. - -Coordinator selection explicitly covers its continuation runs. Calls made inside -application callbacks join only through their own explicit selection. There is -no implicit context-based membership or automatic enrollment of the starting -exchange's run. - -Preserving an application's existing observer alongside one or more group -collectors is supported usage. Provide a small observer-composition helper so -consumers need not rebuild event forwarding throughout their adapters. Preserve -ordinary synchronous delivery and panic behavior; do not add asynchronous queues, -panic recovery, or guaranteed delivery to later observers after an earlier -observer panics. Interrupted delivery can leave accounting incomplete. - -**API planning:** sketch ordinary, prepared, coordinated, and callback-initiated -usage with the collector and composition helper. Explicit group fields remain a -fallback if those sketches demonstrate awkward usage; they would still use the -same observation machinery internally. The observer-compatible approach is the -selected starting point, not an unresolved product decision. - -### Group Decision 2: What Is The Group's Lifetime? - -**Status: accepted.** Provide detached, internally consistent summary snapshots -from a caller-held group, supporting concurrent selected executions and snapshot -reads. Reading or modifying a snapshot must not alter the collector or other -snapshots. No mandatory close, seal, or wait lifecycle is introduced. Applications -own execution scheduling and decide when to read or discard the group. - -Distinguish runs with terminal observations from observed runs without them. -Missing terminal observations remain visible as incomplete accounting; the group -does not infer whether the cause is active work, interrupted observer delivery, -or something else. Preserve the existing observation contract rather than -promising additional terminal delivery. - -“All observed members have terminal observations” does not mean “the application -operation is complete.” Additional runs may join later, and PromptKit cannot know -whether the application selected every relevant call. The application decides -when a snapshot represents its final report. - -**Alternative considered, not selected:** a close/seal or wait lifecycle could -provide fixed membership or a completion wait, but would duplicate application -coordination without a demonstrated need. - -### Group Decision 3: May Groups Overlap? - -**Status: accepted.** Allow explicit membership in multiple groups, counting each -run identity once within each group. Groups are independent accounting views; -overlapping totals are not generally additive. - -An application can obtain both operation-level and overall summaries by attaching -the respective collectors to each selected execution. This requires no group -nesting, inherited membership, scheduling, or facility for merging overlapping -summaries. - -**Alternative considered, not selected:** one group per run would simplify -membership plumbing but restrict useful views and force consumers to rebuild -broader aggregation. There is no demonstrated need for that restriction. - -### Group Decision 4: Which Facts Does A Summary Expose And Aggregate? - -**Status: accepted.** The initial summary exposes: - -- Member run identities and terminal-observation coverage. -- Existing execution counters: provider attempts, transient retries, and - structural repairs. -- Available token-usage totals with their known/completeness information. - -Aggregate one terminal observation per run, without reconstructing totals from -attempt events or adding terminal summaries to the attempts they summarize. -Terminal observations already include retries, structural repairs, reported -usage, and available failure-path facts, including executions that return no -ordinary result. Summaries describe execution facts, not application acceptance -or monetary cost. - -Distinguish terminal coverage from usage completeness. A terminal observation -can itself report incomplete usage. Conversely, fully reported usage for every -finished member does not establish complete accounting when another observed -member lacks a terminal observation. Preserve existing per-field usage semantics -and expose missing terminal coverage separately. For example, two finished runs -may supply a useful known subtotal while a third observed run without a terminal -event leaves group accounting incomplete. Absence of evidence must not become -known zero. Exact field names and representation remain for API planning. - -**Alternatives considered, not selected for the initial implementation:** live -attempt-level totals would expose additional partial facts but require reconciling -attempt and terminal observations. Applications can consume ordinary events for -progress displays. Duration aggregation is also omitted absent a concrete need; -if added later, summed execution durations must be distinguished from elapsed -group time. Neither requires expanding the initial summary into a comprehensive -report schema or a new outcome hierarchy. - -## Proposed Delivery And Validation - -All six continuation/coordinator decisions and all four execution-group decisions -are accepted. No product-level decisions remain open. Remaining work is API and -staged implementation planning: exact types and signatures, capture opt-in syntax, -message assembly details, placement of relationship metadata, the concrete -inspection representation, and collector/composition and summary API shapes. -Execution groups retain a separate delivery track; their implementation is not a -prerequisite for continuation. The recommended continuation delivery sequence is: - -1. Deliver capture and independent continuation through the public facade, - allowing Audita to remove its reconstruction and prefix-verification work. -2. Build the optional sequential coordinator on that capability, reusing - ordinary execution, validation, capacity, and observation machinery. Refine - both consumer sketches against the selected public contracts to verify that - the same mechanism removes useful repeated work in each application. - -Plan execution groups independently under the accepted decisions, reusing -ordinary observations and preserving existing application observers. -They can ship separately without changing the coordinator termination contract. +The initial callback uses existing findings without repeating batch assessment. +`audita.assess` may call the same backend after the turn's admission lease is +released. Those calls join any group only through their own explicit selection. +Other candidates can start independent branches; Audita owns sibling outcomes. + +Both sketches process the final permitted response through the normal callback. +Applications retain their received results; the coordinator's termination value +is not an archive. These examples should remain usable through the same generic +API without manual history assembly or per-turn lifecycle bookkeeping. + +## API Planning And Delivery + +No product-level decisions remain open. Design exact types and signatures within +the accepted behavior above: + +- Capture opt-in syntax and exchange representation. Prefer a request option; + separate capture methods remain viable. Prefer a details accessor, while a + public value is viable if it preserves ownership and generic rendering. +- Continuation message assembly order and placement of immediate-parent metadata. +- Callback state, next-action types, normal outcomes, and the optional + undelivered-response result. +- Collector attachment, observer composition, and summary field representation. + Start with the observer-compatible API. Explicit group fields remain a fallback + only if concrete ordinary/prepared/coordinator/callback usage sketches expose + awkwardness; either approach uses existing observation machinery. + +Deliver in separable stages: + +1. **Capture and independent continuation:** expose execution-derived exchanges, + reusable branches, fresh follow-up settings, parent metadata, and details. + This lets consumers remove reconstruction and prefix-verification machinery. +2. **Optional sequential coordinator:** build on continuation and ordinary + execution, validation, capacity, and observations. Refine both consumer + sketches against the concrete API. +3. **Optional execution groups, on an independent track:** provide collection, + composition, snapshots, and aggregation. This can ship separately and need + not wait for or delay continuation or coordination. + +These stages guide a subsequent implementation plan; they do not authorize code +changes or impose an ordering dependency on execution groups. + +## Acceptance Scenarios Use [Audita's acceptance scenarios](audita-request.md#suggested-delivery-and-acceptance-scenarios) -as consumer checks, alongside analysis and branching-conversation examples that -exercise the same APIs without repair-specific concepts. Additional proposed -coverage should establish: +alongside analysis and branching-conversation examples. Validate each capability +at its public boundary without adding guarantees beyond the selected behavior. -- The selected conversation history when the initial run needed structural - repair, including use of its full final assistant content. -- Capture of the final returned response after structural validation fails, - preserving its ordinary validation status through the coordinator callback. - Operational failures with no completed result create no new exchange. -- Capture opt-in through ordinary and prepared execution, using the executed - snapshot and producing no completed exchange during preparation alone. -- Continuation through another engine instance in the same process, retaining - prior content while applying the receiving engine's follow-up configuration. -- Continuation after the original input files change or disappear, without - altering the captured history. -- Independent branches retaining their own history and outcomes. -- Assessment calls against the same backend at capacity one, without capacity - retained by the coordinator blocking them. -- Per-run usage and failure diagnostics available through ordinary observations, - including incomplete usage. Consumer examples explicitly select accounting - membership and count shared initial work once, without coordinator-provided - totals; optional groups may supply those summaries when available. -- Immediate parent relationships distinguish successive turns and independent - branches using existing run identities. -- Explicit exchange details expose captured logical history and minimal identity - metadata through ordinary encoding. Mutating returned details does not affect - retained branch state, and inspection requires no rendering or external reads. +### Capture And Continuation -Execution-group validation should separately cover explicit ordinary, prepared, -coordinator, and callback membership; unrelated nested calls remaining outside -the group; independent ancestry and membership; concurrent collection alongside -an existing application observer; detached, internally consistent snapshots; -failure-path terminal facts; unavailable and explicit-zero usage; missing terminal -coverage distinct from usage completeness; and no attempt/terminal double counting. -Verify once-per-run counting within overlapping groups without implying that -their totals can be added. Check that new members can join after a snapshot with -full terminal coverage and that observer composition retains ordinary synchronous -delivery and panic behavior. +- Ordinary and prepared opt-in capture uses the actual executed snapshot; + preparation alone produces no exchange. +- Internal structural repair contributes only the full final assistant content + to history. Final validation failure still permits capture; an operational + failure without a completed result does not. +- Independent branches preserve prior history and outcomes, including after + original inputs change or disappear and when continued by another engine. +- Follow-up configuration, credentials, session selection, and complete output + overrides follow ordinary rules without implicit inheritance. +- Details encode generically, expose originating and parent identities, and are + detached from retained state; inspection performs no rendering or external reads. -These are planning targets, not new implemented guarantees. Keep additional -mechanisms and constraints tied to the selected decisions and demonstrated -consumer needs. +### Coordination + +- Initial callback, zero-turn behavior, last-response delivery, stop versus + exhaustion, and execution-start budget consumption follow the specified rules. +- Cancellation before delivery retains only an already-obtained undelivered + response; callback errors do not replay delivered responses or earlier work. +- Failed structural validation follows normal response delivery, while operational + errors preserve Go error identities and stop progression. +- Same-backend callback calls at capacity one proceed after logical admission + release. Internal retries/repairs do not create extra conversation callbacks. +- Both consumer sketches work without PromptKit interpreting content or owning + application outcomes, branch scheduling, or a response archive. + +### Observations And Groups + +- Per-run usage and failure diagnostics preserve ordinary presence/completeness + semantics and parent references distinguish turns and branches. +- Explicit ordinary, prepared, coordinator, and callback membership works alongside + an existing observer; unrelated nested calls remain independent. +- Concurrent collection and snapshots preserve detached, consistent results. + More members may join after a snapshot with full terminal coverage. +- Overlapping groups count each run once within each group. Shared initial work, + newly reported replay usage, and attempt/terminal events are not double counted. +- Known subtotals, explicit zeros, incomplete usage, and missing terminal coverage + remain distinguishable, including on failure paths. +- Observer composition preserves synchronous delivery and panic behavior without + stronger terminal-delivery promises.