promptkit/docs/roadmap/reliable-execution.md

# Reliable And Observable Execution

**Status:** Implemented in source; not yet released. This document records the
accepted target and approved policy choices. The companion
[implementation plan](implementation.md) records implementation steps and
concrete API and algorithm decisions. Current contracts are owned by public
GoDoc and the linked consumer and integration documentation.

## Purpose

Make Promptkit responsible for reusable provider-failure recovery and execution
accounting, including failed operations, without becoming an application workflow
or telemetry framework. Consumers should not need separate retry loops merely
to handle temporary provider failures, nor parse error strings to understand
what happened.

This scope incorporates requests 1–5 from the
[Audita wishlist](audita-feature-requests.md). Retries, error classification,
observations, shared capacity, and consumer evidence belong together because
they describe the same execution lifecycle. Licensing was handled separately.
Audita motivates the work but does not define an application-specific API.

## Foundations At Planning Time

At the time this scope was accepted, Promptkit provided structural output
validation and bounded repair, prepared execution, engine-local backend
capacity, run metadata, aggregate token usage, and typed provider-status and
capacity errors. Operational errors did not return partial run results. The
built-in path had no transient retry loop, and existing integer usage fields
did not distinguish missing reports from zero.

The implemented contracts are owned by public GoDoc, the
[consumer guide](../consumers/pkg-promptkit.md), and the
[provider integration contract](../integrations/openai-compatible-chat.md).
The [capacity document](../internal/capacity.md) and
[runner document](../internal/runner.md) describe the existing implementation.
This historical baseline does not override their current-state descriptions.

## Target End State

### Bounded Transient-Failure Retries

The built-in OpenAI-compatible client path supports configurable transient
retries. The initial policy uses two additional attempts by default and permits
an unambiguous explicit zero to disable them. Initial eligible statuses are
429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status.

Configure an engine default with a request-scoped retry-count override, using
presence to distinguish omission from zero. Keep retry policy out of profile
YAML initially. Per-request control supports mixed workflows without requiring
separate engines and splitting their capacity pools. Use a modest fixed backoff
policy without a new public timing-configuration surface.

- Retry delays use exponential backoff with jitter and honor valid provider
  `Retry-After` timing as a minimum delay, subject to cancellation, rather than
  truncating it and retrying too early. Invalid or absent timing falls back to
  local policy.
- Caller cancellation and deadlines interrupt both waiting and active attempts.
  Cancellation is never itself a reason to retry.
- Permanent request failures, malformed successful response envelopes, and
  structurally invalid generated content do not enter this transient loop.
  Structural invalidity remains the separate output-repair mechanism.
- Transport failures are classified and exposed without automatic replay in
  this scope. A generic network error does not prove that no billable work
  occurred. Conservative eligibility reduces ambiguity; even HTTP-status
  retries do not promise zero duplicate billing.
- Retry one resolved generation request: retain its messages, model, endpoint,
  settings, structured-output constraint, and resolved credential throughout
  that retry sequence. Do not rerender or reload application sources between
  attempts. Resolve environment credentials at execution, not when preparing
  a handle that may remain unexecuted for a long time.
- Apply the same behavior to ordinary and prepared execution and to generation
  performed for structural repair. A prepared handle remains single-use; its
  execution may contain multiple internal provider attempts.
- Do not silently retry arbitrary injected `LLMClient` implementations. They
  continue to own their retry policy. A custom HTTP transport under the built-in
  client uses the built-in policy; consumers can disable it when their transport
  already retries. Hidden attempts inside collaborators cannot be counted by
  Promptkit and must not be presented as if they were observable.

Retries are not exactly-once generation. A failed or uncertain request may have
performed billable work. With R additional transient attempts per generation
and P additional structural repair generations, the maximum Promptkit-controlled
provider attempts for that execution is (R + 1) × (P + 1). Cancellation and
non-retryable failures can reduce that count. Application-owned semantic retries
are separate executions and may multiply it further.

Existing generation/HTTP timeouts remain per-attempt limits. The caller's
context bounds the complete operation, including capacity waiting, retry delays,
and repair; its deadline is never reset between attempts. Callers needing a
total time budget must provide a deadline, since multiple attempts can exceed
one attempt's timeout. An expired caller deadline ends the run rather than
starting another attempt. This preserves existing transport-setting semantics
without adding a competing total-timeout setting.

### Shared Capacity, Not A Second Scheduler

Every actual generation attempt, including retries and repair, participates in
the existing engine-local pool for its backend. Release active-generation
permits before backoff; waiting to retry must not monopolize active capacity.
Keep the logical execution's bounded admission reservation until it finishes.
Reacquiring a permit uses ordinary capacity scheduling, not a new priority lane.

Proposal and assessment calls in the representative example share one engine
and a finite backend limit. Existing unlimited custom backends and endpoint-only
profiles remain supported; do not impose a new global cap or silently change
catalog defaults. If a consumer requires a different limit, current custom
backend IDs and application-owned profiles remain the starting point.

Concurrency bounds simultaneous activity, not cumulative cost or request rate.
Independent engines, processes, and other clients do not share these limits.

### Actionable Public Failures

Extend existing public error contracts rather than replace them with an unrelated
taxonomy. Consumers must distinguish cancellation, deadlines, transport failures,
non-success provider status, malformed provider responses, retry exhaustion,
capacity rejection, and completed output-validation failure without parsing text.

Exhaustion retains the final underlying cause and actual attempt count. Existing
context identities, provider status/detail access, and `errors.Is`/`errors.As`
behavior remain useful through any added wrapping. An error can be both an
exhaustion and a particular provider failure; categories need not be exclusive.

Completed invalid output remains a validation result, not an operational error.
Operational failures continue to return no partial `RunResult`; observations
provide accounting independently. Preserve redacted default diagnostics and
deliberate access to untrusted provider detail under application disclosure policy.

### Optional Execution Observations

Provide an optional request-scoped observer through the public facade. It receives
metadata about an execution without requiring a logging or metrics framework.
Promptkit does not retain a global execution history or persist reports.

Deliver callbacks synchronously and in order within an execution. Concurrent
executions may invoke a shared callback concurrently. Do not invoke consumer
code under capacity-manager locks or add a background delivery queue. Callbacks
cannot return execution-policy errors; consumers keep them fast and handle their
own reporting failures. Panics follow ordinary injected-code behavior rather
than being silently swallowed. This keeps delivery straightforward without
buffering, dropping, or shutdown machinery; slow callbacks can still delay a run.

The observable lifecycle covers successful execution, completed invalid output,
and operational failure, including failures before any model call. Expose only
identities available at that point; do not invent selected profiles or backends
when resolution failed. Prepared execution observations distinguish execution
from earlier preparation and consumer-held delay.

Consumers can correlate the logical execution with generation attempts and learn:

- the available prompt, profile, backend, and effective model identities;
- whether generation is initial or structural repair, and which transient
  attempt occurred within it;
- capacity-wait, generation, and retry-delay measurements with clear boundaries;
- classified outcomes and actual retry and repair counts; and
- provider-reported usage already received, even when a subsequent step fails.

Attempt measurements and terminal totals have explicit accounting semantics so
consumers do not add aggregates to their constituent attempts. Missing usage is
distinguished from measured zero; partial provider reporting must not become
falsely complete accounting. Do not estimate unknown usage or claim a complete
billing ledger. Preserve existing successful-result usage semantics and use a
presence-aware observation representation rather than silently redefining zero.

Observation payloads contain metadata, not credentials, request/response bodies,
raw provider messages, or arbitrary collaborator error strings. Metadata itself
may be sensitive and remains subject to application handling. Content capture
is not part of this scope. Public contracts must state event ownership, delivery
ordering, concurrency, and callback behavior without exposing internal types.

### Representative Consumer Evidence

Add a compact maintained offline example of a structured proposal followed by
an independent LLM assessment. Use embedded prompts/schemas, application-owned
profiles, synthetic content, a shared finite backend, and collected observations.
Use a controlled HTTP boundary when demonstrating built-in transient retry
behavior; injecting an already-retrying model client would not prove that path.

Show that schema validity does not imply domain acceptance. Keep transcript
processing, semantic acceptance, reports, and retained artifacts application-owned.
Do not turn the example into a miniature Audita pipeline or a failure-case menu.

Focused tests, rather than one oversized example, establish recovery, exhaustion,
disablement, retry timing, cancellation during capacity/backoff/transport, repair
interaction, peak concurrency, failure accounting, usage presence, and redaction.
Use deterministic clocks/waits or controlled transports where needed; no paid
calls, real credentials, timing-sensitive sleeps, or duplicated test matrices.

## Architecture And API Boundaries

Keep execution machinery under `internal/`; the root package remains a facade
for intentional consumer contracts. Maintain one retry-policy owner and one
capacity manager. Keep provider wire decoding in its adapter, lifecycle
coordination in the execution layer, and application reporting downstream.

Do not add a second run API solely for diagnostics. Preserve existing operation
signatures and optional-observation behavior for callers that do not use it.
Avoid public clock/sleeper abstractions introduced only for internal testing.
No profile/backend catalog schema changes are assumed by this roadmap.

## Documentation And Compatibility

Default retries change request counts, latency, and potential cost. Target a
minor release with explicit migration guidance, including how to retain
single-attempt behavior and avoid duplicate retries. Canonical GoDoc owns exact
API and error semantics; integration docs own HTTP retry behavior; consumer docs
own adoption examples. Update internal documentation and maintained-example
validation only where responsibilities actually change. Release notes summarize
and link rather than become a second contract owner.

## Non-Goals

No application pipeline, domain validator, cross-process scheduler, new task
queue, monetary budget, rate limiter, circuit breaker, automatic backend/model
failover, streaming/tool support, telemetry dependency, persistent run store,
or additional content-retention facility. Existing fallback-profile behavior
is not repurposed as runtime provider failover. Licensing is already complete
in source and is not a further implementation dependency.