From bd975f17f26293cd589ff08b202e873a5518f889 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 23 Sep 2026 16:21:59 +0000 Subject: [PATCH] Finalize reliable execution policy decisions --- docs/roadmap/reliable-execution.md | 98 ++++++++++-------------------- 1 file changed, 33 insertions(+), 65 deletions(-) diff --git a/docs/roadmap/reliable-execution.md b/docs/roadmap/reliable-execution.md index 506d1d8..10ae565 100644 --- a/docs/roadmap/reliable-execution.md +++ b/docs/roadmap/reliable-execution.md @@ -2,7 +2,9 @@ **Status:** Feature roadmap; not implemented. This document defines the target end state. A separate implementation plan will supply sequential stages and -prompt-level steps after the remaining decisions are resolved. +prompt-level steps. The four policy decisions reviewed with the maintainer are +settled below; exact API spelling, event field layout, and backoff timing values +remain implementation-plan details. ## Purpose @@ -42,13 +44,26 @@ retries. The initial policy uses two additional attempts by default and permits an unambiguous explicit zero to disable them. Initial eligible statuses are 429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status. +Configure an engine default with a request-scoped retry-count override, using +presence to distinguish omission from zero. Keep retry policy out of profile +YAML initially. Per-request control supports mixed workflows without requiring +separate engines and splitting their capacity pools. Use a modest fixed backoff +policy; concrete timing values belong in the implementation plan, not a new +public timing-configuration surface. + - Retry delays use exponential backoff with jitter and honor valid provider - `Retry-After` timing. Invalid or absent timing falls back to local policy. + `Retry-After` timing as a minimum delay, subject to cancellation, rather than + truncating it and retrying too early. Invalid or absent timing falls back to + local policy. - Caller cancellation and deadlines interrupt both waiting and active attempts. Cancellation is never itself a reason to retry. - Permanent request failures, malformed successful response envelopes, and structurally invalid generated content do not enter this transient loop. Structural invalidity remains the separate output-repair mechanism. +- Transport failures are classified and exposed without automatic replay in + this scope. A generic network error does not prove that no billable work + occurred. Conservative eligibility reduces ambiguity; even HTTP-status + retries do not promise zero duplicate billing. - Retry one resolved generation request: retain its messages, model, endpoint, settings, structured-output constraint, and resolved credential throughout that retry sequence. Do not rerender or reload application sources between @@ -70,6 +85,14 @@ provider attempts for that execution is (R + 1) × (P + 1). Cancellation and non-retryable failures can reduce that count. Application-owned semantic retries are separate executions and may multiply it further. +Existing generation/HTTP timeouts remain per-attempt limits. The caller's +context bounds the complete operation, including capacity waiting, retry delays, +and repair; its deadline is never reset between attempts. Callers needing a +total time budget must provide a deadline, since multiple attempts can exceed +one attempt's timeout. An expired caller deadline ends the run rather than +starting another attempt. This preserves existing transport-setting semantics +without adding a competing total-timeout setting. + ### Shared Capacity, Not A Second Scheduler Every actual generation attempt, including retries and repair, participates in @@ -110,6 +133,14 @@ Provide an optional request-scoped observer through the public facade. It receiv metadata about an execution without requiring a logging or metrics framework. Promptkit does not retain a global execution history or persist reports. +Deliver callbacks synchronously and in order within an execution. Concurrent +executions may invoke a shared callback concurrently. Do not invoke consumer +code under capacity-manager locks or add a background delivery queue. Callbacks +cannot return execution-policy errors; consumers keep them fast and handle their +own reporting failures. Panics follow ordinary injected-code behavior rather +than being silently swallowed. This keeps delivery straightforward without +buffering, dropping, or shutdown machinery; slow callbacks can still delay a run. + The observable lifecycle covers successful execution, completed invalid output, and operational failure, including failures before any model call. Expose only identities available at that point; do not invent selected profiles or backends @@ -187,66 +218,3 @@ failover, streaming/tool support, telemetry dependency, persistent run store, or additional content-retention facility. Existing fallback-profile behavior is not repurposed as runtime provider failover. Licensing is already complete in source and is not a further implementation dependency. - -## Open Decisions - -Resolve these before making the companion implementation plan decision-complete. -API spelling and event field layout also belong in that plan. - -### Retry Configuration And Timing - -**Recommendation:** An engine default with a request-scoped retry-count override, -using presence to distinguish omission from zero. Keep policy out of profile -YAML initially. Use a modest fixed exponential backoff policy with jitter; -select concrete timing values in the plan. Treat valid `Retry-After` as a minimum -delay subject to cancellation rather than truncating it and retrying too early. -This supplies normal defaults and per-operation control without another profile -inheritance dimension. - -**Alternative:** Engine-only retry configuration is smaller but makes mixed -single-attempt and retrying workflows require separate engines, splitting their -capacity pools. A richer public timing policy is viable for consumer tuning but -adds a broader contract before there is demonstrated need. - -### Transport-Failure Eligibility - -**Recommendation:** Initially retry only the selected HTTP statuses, and expose -transport failures for consumer handling without automatic replay. This avoids -pretending a generic network error proves that no billable work occurred. - -**Alternative:** Include a narrow documented set of transient transport failures. -This recovers more outages but requires precise classification, testing, and -clear duplicate-work warnings. Even HTTP-status retries do not promise zero -duplicate billing; conservative eligibility only reduces ambiguity. - -### Timeout Scope - -**Recommendation:** Keep existing generation/HTTP timeouts as per-attempt limits; -the caller's context bounds the complete operation, including capacity waiting, -retry delays, and repair. Do not reset the caller's deadline between attempts. -This preserves the meaning of the existing transport settings and avoids adding -another competing total-timeout setting. Document that callers needing a total -time budget must provide a deadline; multiple attempts can exceed one attempt's -timeout. An expired caller deadline ends the run, never starts another attempt. - -**Alternative:** Apply the configured generation timeout across an entire retry -sequence. This provides a tighter aggregate bound without caller changes, but -reinterprets an existing setting and gives later attempts less time following -capacity contention or provider-directed delays. It also needs a clear boundary -between separate repair generations and their retry sequences. - -### Observer Delivery - -**Recommendation:** Synchronous callbacks, ordered within an execution, with -concurrent invocations possible across executions sharing a callback. Do not -invoke consumer code under capacity-manager locks or add a background delivery -queue. Callbacks cannot return execution-policy errors; callers keep them fast -and handle their own reporting failures. Panics follow ordinary injected-code -behavior rather than being silently swallowed. This minimizes machinery and -makes delivery timing understandable; slow callbacks can still delay a run. - -**Alternative:** Return a separate execution report on both success and failure. -This avoids callback latency but requires a broader result/error API and retains -attempt history internally. Async delivery offers isolation but introduces -buffering, dropping, shutdown, and completion semantics disproportionate to the -current need.