diff --git a/docs/roadmap/audita-feature-requests.md b/docs/roadmap/audita-feature-requests.md index ce05dc7..59880d6 100644 --- a/docs/roadmap/audita-feature-requests.md +++ b/docs/roadmap/audita-feature-requests.md @@ -7,6 +7,12 @@ These are not approved features, implementation plans, delivery commitments, or claims about current behavior. Priorities reflect Audita's prospective adoption; Promptkit should evaluate them against its broader consumer needs. +Requests 1–5 are now incorporated into the active +[reliable-execution roadmap](reliable-execution.md), which owns the selected +Promptkit-native scope and remaining decisions. The requests below retain the +consumer context, not a separate implementation specification. Request 6 is +implemented in source as noted below. + ## Consumer Context Audita polishes transcripts through ordered correction modules. It makes LLM calls diff --git a/docs/roadmap/reliable-execution.md b/docs/roadmap/reliable-execution.md new file mode 100644 index 0000000..506d1d8 --- /dev/null +++ b/docs/roadmap/reliable-execution.md @@ -0,0 +1,252 @@ +# Reliable And Observable Execution + +**Status:** Feature roadmap; not implemented. This document defines the target +end state. A separate implementation plan will supply sequential stages and +prompt-level steps after the remaining decisions are resolved. + +## Purpose + +Make Promptkit responsible for reusable provider-failure recovery and execution +accounting, including failed operations, without becoming an application workflow +or telemetry framework. Consumers should not need separate retry loops merely +to handle temporary provider failures, nor parse error strings to understand +what happened. + +This scope incorporates requests 1–5 from the +[Audita wishlist](audita-feature-requests.md). Retries, error classification, +observations, shared capacity, and consumer evidence belong together because +they describe the same execution lifecycle. Licensing was handled separately. +Audita motivates the work but does not define an application-specific API. + +## Existing Foundations + +Promptkit already provides structural output validation and bounded repair, +prepared execution, engine-local backend capacity, run metadata, aggregate token +usage, and typed provider-status and capacity errors. Operational errors do not +return partial run results. The built-in client has no transient retry loop, +and existing integer usage fields do not distinguish missing reports from zero. + +The current contracts remain owned by public GoDoc, the +[consumer guide](../consumers/pkg-promptkit.md), and the +[provider integration contract](../integrations/openai-compatible-chat.md). +The [capacity document](../internal/capacity.md) and +[runner document](../internal/runner.md) describe the existing implementation. +This roadmap does not redefine those documents as already implementing new work. + +## Target End State + +### Bounded Transient-Failure Retries + +The built-in OpenAI-compatible client path supports configurable transient +retries. The initial policy uses two additional attempts by default and permits +an unambiguous explicit zero to disable them. Initial eligible statuses are +429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status. + +- Retry delays use exponential backoff with jitter and honor valid provider + `Retry-After` timing. Invalid or absent timing falls back to local policy. +- Caller cancellation and deadlines interrupt both waiting and active attempts. + Cancellation is never itself a reason to retry. +- Permanent request failures, malformed successful response envelopes, and + structurally invalid generated content do not enter this transient loop. + Structural invalidity remains the separate output-repair mechanism. +- Retry one resolved generation request: retain its messages, model, endpoint, + settings, structured-output constraint, and resolved credential throughout + that retry sequence. Do not rerender or reload application sources between + attempts. Resolve environment credentials at execution, not when preparing + a handle that may remain unexecuted for a long time. +- Apply the same behavior to ordinary and prepared execution and to generation + performed for structural repair. A prepared handle remains single-use; its + execution may contain multiple internal provider attempts. +- Do not silently retry arbitrary injected `LLMClient` implementations. They + continue to own their retry policy. A custom HTTP transport under the built-in + client uses the built-in policy; consumers can disable it when their transport + already retries. Hidden attempts inside collaborators cannot be counted by + Promptkit and must not be presented as if they were observable. + +Retries are not exactly-once generation. A failed or uncertain request may have +performed billable work. With R additional transient attempts per generation +and P additional structural repair generations, the maximum Promptkit-controlled +provider attempts for that execution is (R + 1) × (P + 1). Cancellation and +non-retryable failures can reduce that count. Application-owned semantic retries +are separate executions and may multiply it further. + +### Shared Capacity, Not A Second Scheduler + +Every actual generation attempt, including retries and repair, participates in +the existing engine-local pool for its backend. Release active-generation +permits before backoff; waiting to retry must not monopolize active capacity. +Keep the logical execution's bounded admission reservation until it finishes. +Reacquiring a permit uses ordinary capacity scheduling, not a new priority lane. + +Proposal and assessment calls in the representative example share one engine +and a finite backend limit. Existing unlimited custom backends and endpoint-only +profiles remain supported; do not impose a new global cap or silently change +catalog defaults. If a consumer requires a different limit, current custom +backend IDs and application-owned profiles remain the starting point. + +Concurrency bounds simultaneous activity, not cumulative cost or request rate. +Independent engines, processes, and other clients do not share these limits. + +### Actionable Public Failures + +Extend existing public error contracts rather than replace them with an unrelated +taxonomy. Consumers must distinguish cancellation, deadlines, transport failures, +non-success provider status, malformed provider responses, retry exhaustion, +capacity rejection, and completed output-validation failure without parsing text. + +Exhaustion retains the final underlying cause and actual attempt count. Existing +context identities, provider status/detail access, and `errors.Is`/`errors.As` +behavior remain useful through any added wrapping. An error can be both an +exhaustion and a particular provider failure; categories need not be exclusive. + +Completed invalid output remains a validation result, not an operational error. +Operational failures continue to return no partial `RunResult`; observations +provide accounting independently. Preserve redacted default diagnostics and +deliberate access to untrusted provider detail under application disclosure policy. + +### Optional Execution Observations + +Provide an optional request-scoped observer through the public facade. It receives +metadata about an execution without requiring a logging or metrics framework. +Promptkit does not retain a global execution history or persist reports. + +The observable lifecycle covers successful execution, completed invalid output, +and operational failure, including failures before any model call. Expose only +identities available at that point; do not invent selected profiles or backends +when resolution failed. Prepared execution observations distinguish execution +from earlier preparation and consumer-held delay. + +Consumers can correlate the logical execution with generation attempts and learn: + +- the available prompt, profile, backend, and effective model identities; +- whether generation is initial or structural repair, and which transient + attempt occurred within it; +- capacity-wait, generation, and retry-delay measurements with clear boundaries; +- classified outcomes and actual retry and repair counts; and +- provider-reported usage already received, even when a subsequent step fails. + +Attempt measurements and terminal totals have explicit accounting semantics so +consumers do not add aggregates to their constituent attempts. Missing usage is +distinguished from measured zero; partial provider reporting must not become +falsely complete accounting. Do not estimate unknown usage or claim a complete +billing ledger. Preserve existing successful-result usage semantics and use a +presence-aware observation representation rather than silently redefining zero. + +Observation payloads contain metadata, not credentials, request/response bodies, +raw provider messages, or arbitrary collaborator error strings. Metadata itself +may be sensitive and remains subject to application handling. Content capture +is not part of this scope. Public contracts must state event ownership, delivery +ordering, concurrency, and callback behavior without exposing internal types. + +### Representative Consumer Evidence + +Add a compact maintained offline example of a structured proposal followed by +an independent LLM assessment. Use embedded prompts/schemas, application-owned +profiles, synthetic content, a shared finite backend, and collected observations. +Use a controlled HTTP boundary when demonstrating built-in transient retry +behavior; injecting an already-retrying model client would not prove that path. + +Show that schema validity does not imply domain acceptance. Keep transcript +processing, semantic acceptance, reports, and retained artifacts application-owned. +Do not turn the example into a miniature Audita pipeline or a failure-case menu. + +Focused tests, rather than one oversized example, establish recovery, exhaustion, +disablement, retry timing, cancellation during capacity/backoff/transport, repair +interaction, peak concurrency, failure accounting, usage presence, and redaction. +Use deterministic clocks/waits or controlled transports where needed; no paid +calls, real credentials, timing-sensitive sleeps, or duplicated test matrices. + +## Architecture And API Boundaries + +Keep execution machinery under `internal/`; the root package remains a facade +for intentional consumer contracts. Maintain one retry-policy owner and one +capacity manager. Keep provider wire decoding in its adapter, lifecycle +coordination in the execution layer, and application reporting downstream. +Exact internal package arrangement belongs in the implementation plan. + +Do not add a second run API solely for diagnostics. Preserve existing operation +signatures and optional-observation behavior for callers that do not use it. +Avoid public clock/sleeper abstractions introduced only for internal testing. +No profile/backend catalog schema changes are assumed by this roadmap. + +## Documentation And Compatibility + +Default retries change request counts, latency, and potential cost. Target a +minor release with explicit migration guidance, including how to retain +single-attempt behavior and avoid duplicate retries. Canonical GoDoc owns exact +API and error semantics; integration docs own HTTP retry behavior; consumer docs +own adoption examples. Update internal documentation and maintained-example +validation only where responsibilities actually change. Release notes summarize +and link rather than become a second contract owner. + +## Non-Goals + +No application pipeline, domain validator, cross-process scheduler, new task +queue, monetary budget, rate limiter, circuit breaker, automatic backend/model +failover, streaming/tool support, telemetry dependency, persistent run store, +or additional content-retention facility. Existing fallback-profile behavior +is not repurposed as runtime provider failover. Licensing is already complete +in source and is not a further implementation dependency. + +## Open Decisions + +Resolve these before making the companion implementation plan decision-complete. +API spelling and event field layout also belong in that plan. + +### Retry Configuration And Timing + +**Recommendation:** An engine default with a request-scoped retry-count override, +using presence to distinguish omission from zero. Keep policy out of profile +YAML initially. Use a modest fixed exponential backoff policy with jitter; +select concrete timing values in the plan. Treat valid `Retry-After` as a minimum +delay subject to cancellation rather than truncating it and retrying too early. +This supplies normal defaults and per-operation control without another profile +inheritance dimension. + +**Alternative:** Engine-only retry configuration is smaller but makes mixed +single-attempt and retrying workflows require separate engines, splitting their +capacity pools. A richer public timing policy is viable for consumer tuning but +adds a broader contract before there is demonstrated need. + +### Transport-Failure Eligibility + +**Recommendation:** Initially retry only the selected HTTP statuses, and expose +transport failures for consumer handling without automatic replay. This avoids +pretending a generic network error proves that no billable work occurred. + +**Alternative:** Include a narrow documented set of transient transport failures. +This recovers more outages but requires precise classification, testing, and +clear duplicate-work warnings. Even HTTP-status retries do not promise zero +duplicate billing; conservative eligibility only reduces ambiguity. + +### Timeout Scope + +**Recommendation:** Keep existing generation/HTTP timeouts as per-attempt limits; +the caller's context bounds the complete operation, including capacity waiting, +retry delays, and repair. Do not reset the caller's deadline between attempts. +This preserves the meaning of the existing transport settings and avoids adding +another competing total-timeout setting. Document that callers needing a total +time budget must provide a deadline; multiple attempts can exceed one attempt's +timeout. An expired caller deadline ends the run, never starts another attempt. + +**Alternative:** Apply the configured generation timeout across an entire retry +sequence. This provides a tighter aggregate bound without caller changes, but +reinterprets an existing setting and gives later attempts less time following +capacity contention or provider-directed delays. It also needs a clear boundary +between separate repair generations and their retry sequences. + +### Observer Delivery + +**Recommendation:** Synchronous callbacks, ordered within an execution, with +concurrent invocations possible across executions sharing a callback. Do not +invoke consumer code under capacity-manager locks or add a background delivery +queue. Callbacks cannot return execution-policy errors; callers keep them fast +and handle their own reporting failures. Panics follow ordinary injected-code +behavior rather than being silently swallowed. This minimizes machinery and +makes delivery timing understandable; slow callbacks can still delay a run. + +**Alternative:** Return a separate execution report on both success and failure. +This avoids callback latency but requires a broader result/error API and retains +attempt history internally. Async delivery offers isolation but introduces +buffering, dropping, shutdown, and completion semantics disproportionate to the +current need.