promptkit/docs/roadmap/reliable-execution.md

Reliable And Observable Execution

Status: Accepted feature scope; not implemented. This document owns the target end state and approved policy choices. The companion implementation plan owns implementation steps and concrete API and algorithm decisions.

Purpose

Make Promptkit responsible for reusable provider-failure recovery and execution accounting, including failed operations, without becoming an application workflow or telemetry framework. Consumers should not need separate retry loops merely to handle temporary provider failures, nor parse error strings to understand what happened.

This scope incorporates requests 1–5 from the Audita wishlist. Retries, error classification, observations, shared capacity, and consumer evidence belong together because they describe the same execution lifecycle. Licensing was handled separately. Audita motivates the work but does not define an application-specific API.

Existing Foundations

Promptkit already provides structural output validation and bounded repair, prepared execution, engine-local backend capacity, run metadata, aggregate token usage, and typed provider-status and capacity errors. Operational errors do not return partial run results. The built-in client has no transient retry loop, and existing integer usage fields do not distinguish missing reports from zero.

The current contracts remain owned by public GoDoc, the consumer guide, and the provider integration contract. The capacity document and runner document describe the existing implementation. This roadmap does not redefine those documents as already implementing new work.

Target End State

Bounded Transient-Failure Retries

The built-in OpenAI-compatible client path supports configurable transient retries. The initial policy uses two additional attempts by default and permits an unambiguous explicit zero to disable them. Initial eligible statuses are 429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status.

Configure an engine default with a request-scoped retry-count override, using presence to distinguish omission from zero. Keep retry policy out of profile YAML initially. Per-request control supports mixed workflows without requiring separate engines and splitting their capacity pools. Use a modest fixed backoff policy without a new public timing-configuration surface.

• Retry delays use exponential backoff with jitter and honor valid provider Retry-After timing as a minimum delay, subject to cancellation, rather than truncating it and retrying too early. Invalid or absent timing falls back to local policy.
• Caller cancellation and deadlines interrupt both waiting and active attempts. Cancellation is never itself a reason to retry.
• Permanent request failures, malformed successful response envelopes, and structurally invalid generated content do not enter this transient loop. Structural invalidity remains the separate output-repair mechanism.
• Transport failures are classified and exposed without automatic replay in this scope. A generic network error does not prove that no billable work occurred. Conservative eligibility reduces ambiguity; even HTTP-status retries do not promise zero duplicate billing.
• Retry one resolved generation request: retain its messages, model, endpoint, settings, structured-output constraint, and resolved credential throughout that retry sequence. Do not rerender or reload application sources between attempts. Resolve environment credentials at execution, not when preparing a handle that may remain unexecuted for a long time.
• Apply the same behavior to ordinary and prepared execution and to generation performed for structural repair. A prepared handle remains single-use; its execution may contain multiple internal provider attempts.
• Do not silently retry arbitrary injected LLMClient implementations. They continue to own their retry policy. A custom HTTP transport under the built-in client uses the built-in policy; consumers can disable it when their transport already retries. Hidden attempts inside collaborators cannot be counted by Promptkit and must not be presented as if they were observable.

Retries are not exactly-once generation. A failed or uncertain request may have performed billable work. With R additional transient attempts per generation and P additional structural repair generations, the maximum Promptkit-controlled provider attempts for that execution is (R + 1) × (P + 1). Cancellation and non-retryable failures can reduce that count. Application-owned semantic retries are separate executions and may multiply it further.

Existing generation/HTTP timeouts remain per-attempt limits. The caller's context bounds the complete operation, including capacity waiting, retry delays, and repair; its deadline is never reset between attempts. Callers needing a total time budget must provide a deadline, since multiple attempts can exceed one attempt's timeout. An expired caller deadline ends the run rather than starting another attempt. This preserves existing transport-setting semantics without adding a competing total-timeout setting.

Shared Capacity, Not A Second Scheduler

Every actual generation attempt, including retries and repair, participates in the existing engine-local pool for its backend. Release active-generation permits before backoff; waiting to retry must not monopolize active capacity. Keep the logical execution's bounded admission reservation until it finishes. Reacquiring a permit uses ordinary capacity scheduling, not a new priority lane.

Proposal and assessment calls in the representative example share one engine and a finite backend limit. Existing unlimited custom backends and endpoint-only profiles remain supported; do not impose a new global cap or silently change catalog defaults. If a consumer requires a different limit, current custom backend IDs and application-owned profiles remain the starting point.

Concurrency bounds simultaneous activity, not cumulative cost or request rate. Independent engines, processes, and other clients do not share these limits.

Actionable Public Failures

Extend existing public error contracts rather than replace them with an unrelated taxonomy. Consumers must distinguish cancellation, deadlines, transport failures, non-success provider status, malformed provider responses, retry exhaustion, capacity rejection, and completed output-validation failure without parsing text.

Exhaustion retains the final underlying cause and actual attempt count. Existing context identities, provider status/detail access, and errors.Is/errors.As behavior remain useful through any added wrapping. An error can be both an exhaustion and a particular provider failure; categories need not be exclusive.

Completed invalid output remains a validation result, not an operational error. Operational failures continue to return no partial RunResult; observations provide accounting independently. Preserve redacted default diagnostics and deliberate access to untrusted provider detail under application disclosure policy.

Optional Execution Observations

Provide an optional request-scoped observer through the public facade. It receives metadata about an execution without requiring a logging or metrics framework. Promptkit does not retain a global execution history or persist reports.

Deliver callbacks synchronously and in order within an execution. Concurrent executions may invoke a shared callback concurrently. Do not invoke consumer code under capacity-manager locks or add a background delivery queue. Callbacks cannot return execution-policy errors; consumers keep them fast and handle their own reporting failures. Panics follow ordinary injected-code behavior rather than being silently swallowed. This keeps delivery straightforward without buffering, dropping, or shutdown machinery; slow callbacks can still delay a run.

The observable lifecycle covers successful execution, completed invalid output, and operational failure, including failures before any model call. Expose only identities available at that point; do not invent selected profiles or backends when resolution failed. Prepared execution observations distinguish execution from earlier preparation and consumer-held delay.

Consumers can correlate the logical execution with generation attempts and learn:

• the available prompt, profile, backend, and effective model identities;
• whether generation is initial or structural repair, and which transient attempt occurred within it;
• capacity-wait, generation, and retry-delay measurements with clear boundaries;
• classified outcomes and actual retry and repair counts; and
• provider-reported usage already received, even when a subsequent step fails.

Attempt measurements and terminal totals have explicit accounting semantics so consumers do not add aggregates to their constituent attempts. Missing usage is distinguished from measured zero; partial provider reporting must not become falsely complete accounting. Do not estimate unknown usage or claim a complete billing ledger. Preserve existing successful-result usage semantics and use a presence-aware observation representation rather than silently redefining zero.

Observation payloads contain metadata, not credentials, request/response bodies, raw provider messages, or arbitrary collaborator error strings. Metadata itself may be sensitive and remains subject to application handling. Content capture is not part of this scope. Public contracts must state event ownership, delivery ordering, concurrency, and callback behavior without exposing internal types.

Representative Consumer Evidence

Add a compact maintained offline example of a structured proposal followed by an independent LLM assessment. Use embedded prompts/schemas, application-owned profiles, synthetic content, a shared finite backend, and collected observations. Use a controlled HTTP boundary when demonstrating built-in transient retry behavior; injecting an already-retrying model client would not prove that path.

Show that schema validity does not imply domain acceptance. Keep transcript processing, semantic acceptance, reports, and retained artifacts application-owned. Do not turn the example into a miniature Audita pipeline or a failure-case menu.

Focused tests, rather than one oversized example, establish recovery, exhaustion, disablement, retry timing, cancellation during capacity/backoff/transport, repair interaction, peak concurrency, failure accounting, usage presence, and redaction. Use deterministic clocks/waits or controlled transports where needed; no paid calls, real credentials, timing-sensitive sleeps, or duplicated test matrices.

Architecture And API Boundaries

Keep execution machinery under internal/; the root package remains a facade for intentional consumer contracts. Maintain one retry-policy owner and one capacity manager. Keep provider wire decoding in its adapter, lifecycle coordination in the execution layer, and application reporting downstream.

Do not add a second run API solely for diagnostics. Preserve existing operation signatures and optional-observation behavior for callers that do not use it. Avoid public clock/sleeper abstractions introduced only for internal testing. No profile/backend catalog schema changes are assumed by this roadmap.

Documentation And Compatibility

Default retries change request counts, latency, and potential cost. Target a minor release with explicit migration guidance, including how to retain single-attempt behavior and avoid duplicate retries. Canonical GoDoc owns exact API and error semantics; integration docs own HTTP retry behavior; consumer docs own adoption examples. Update internal documentation and maintained-example validation only where responsibilities actually change. Release notes summarize and link rather than become a second contract owner.

Non-Goals

No application pipeline, domain validator, cross-process scheduler, new task queue, monetary budget, rate limiter, circuit breaker, automatic backend/model failover, streaming/tool support, telemetry dependency, persistent run store, or additional content-retention facility. Existing fallback-profile behavior is not repurposed as runtime provider failover. Licensing is already complete in source and is not a further implementation dependency.