Plan reliable and observable execution

docs/roadmap/audita-feature-requests.md

@@ -7,6 +7,12 @@ These are not approved features, implementation plans, delivery commitments, or
 claims about current behavior. Priorities reflect Audita's prospective adoption;
 Promptkit should evaluate them against its broader consumer needs.
 
+Requests 1–5 are now incorporated into the active
+[reliable-execution roadmap](reliable-execution.md), which owns the selected
+Promptkit-native scope and remaining decisions. The requests below retain the
+consumer context, not a separate implementation specification. Request 6 is
+implemented in source as noted below.
+
 ## Consumer Context
 
 Audita polishes transcripts through ordered correction modules. It makes LLM calls

docs/roadmap/reliable-execution.md

@@ -0,0 +1,252 @@
+# Reliable And Observable Execution
+
+**Status:** Feature roadmap; not implemented. This document defines the target
+end state. A separate implementation plan will supply sequential stages and
+prompt-level steps after the remaining decisions are resolved.
+
+## Purpose
+
+Make Promptkit responsible for reusable provider-failure recovery and execution
+accounting, including failed operations, without becoming an application workflow
+or telemetry framework. Consumers should not need separate retry loops merely
+to handle temporary provider failures, nor parse error strings to understand
+what happened.
+
+This scope incorporates requests 1–5 from the
+[Audita wishlist](audita-feature-requests.md). Retries, error classification,
+observations, shared capacity, and consumer evidence belong together because
+they describe the same execution lifecycle. Licensing was handled separately.
+Audita motivates the work but does not define an application-specific API.
+
+## Existing Foundations
+
+Promptkit already provides structural output validation and bounded repair,
+prepared execution, engine-local backend capacity, run metadata, aggregate token
+usage, and typed provider-status and capacity errors. Operational errors do not
+return partial run results. The built-in client has no transient retry loop,
+and existing integer usage fields do not distinguish missing reports from zero.
+
+The current contracts remain owned by public GoDoc, the
+[consumer guide](../consumers/pkg-promptkit.md), and the
+[provider integration contract](../integrations/openai-compatible-chat.md).
+The [capacity document](../internal/capacity.md) and
+[runner document](../internal/runner.md) describe the existing implementation.
+This roadmap does not redefine those documents as already implementing new work.
+
+## Target End State
+
+### Bounded Transient-Failure Retries
+
+The built-in OpenAI-compatible client path supports configurable transient
+retries. The initial policy uses two additional attempts by default and permits
+an unambiguous explicit zero to disable them. Initial eligible statuses are
+429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status.
+
+- Retry delays use exponential backoff with jitter and honor valid provider
+  `Retry-After` timing. Invalid or absent timing falls back to local policy.
+- Caller cancellation and deadlines interrupt both waiting and active attempts.
+  Cancellation is never itself a reason to retry.
+- Permanent request failures, malformed successful response envelopes, and
+  structurally invalid generated content do not enter this transient loop.
+  Structural invalidity remains the separate output-repair mechanism.
+- Retry one resolved generation request: retain its messages, model, endpoint,
+  settings, structured-output constraint, and resolved credential throughout
+  that retry sequence. Do not rerender or reload application sources between
+  attempts. Resolve environment credentials at execution, not when preparing
+  a handle that may remain unexecuted for a long time.
+- Apply the same behavior to ordinary and prepared execution and to generation
+  performed for structural repair. A prepared handle remains single-use; its
+  execution may contain multiple internal provider attempts.
+- Do not silently retry arbitrary injected `LLMClient` implementations. They
+  continue to own their retry policy. A custom HTTP transport under the built-in
+  client uses the built-in policy; consumers can disable it when their transport
+  already retries. Hidden attempts inside collaborators cannot be counted by
+  Promptkit and must not be presented as if they were observable.
+
+Retries are not exactly-once generation. A failed or uncertain request may have
+performed billable work. With R additional transient attempts per generation
+and P additional structural repair generations, the maximum Promptkit-controlled
+provider attempts for that execution is (R + 1) × (P + 1). Cancellation and
+non-retryable failures can reduce that count. Application-owned semantic retries
+are separate executions and may multiply it further.
+
+### Shared Capacity, Not A Second Scheduler
+
+Every actual generation attempt, including retries and repair, participates in
+the existing engine-local pool for its backend. Release active-generation
+permits before backoff; waiting to retry must not monopolize active capacity.
+Keep the logical execution's bounded admission reservation until it finishes.
+Reacquiring a permit uses ordinary capacity scheduling, not a new priority lane.
+
+Proposal and assessment calls in the representative example share one engine
+and a finite backend limit. Existing unlimited custom backends and endpoint-only
+profiles remain supported; do not impose a new global cap or silently change
+catalog defaults. If a consumer requires a different limit, current custom
+backend IDs and application-owned profiles remain the starting point.
+
+Concurrency bounds simultaneous activity, not cumulative cost or request rate.
+Independent engines, processes, and other clients do not share these limits.
+
+### Actionable Public Failures
+
+Extend existing public error contracts rather than replace them with an unrelated
+taxonomy. Consumers must distinguish cancellation, deadlines, transport failures,
+non-success provider status, malformed provider responses, retry exhaustion,
+capacity rejection, and completed output-validation failure without parsing text.
+
+Exhaustion retains the final underlying cause and actual attempt count. Existing
+context identities, provider status/detail access, and `errors.Is`/`errors.As`
+behavior remain useful through any added wrapping. An error can be both an
+exhaustion and a particular provider failure; categories need not be exclusive.
+
+Completed invalid output remains a validation result, not an operational error.
+Operational failures continue to return no partial `RunResult`; observations
+provide accounting independently. Preserve redacted default diagnostics and
+deliberate access to untrusted provider detail under application disclosure policy.
+
+### Optional Execution Observations
+
+Provide an optional request-scoped observer through the public facade. It receives
+metadata about an execution without requiring a logging or metrics framework.
+Promptkit does not retain a global execution history or persist reports.
+
+The observable lifecycle covers successful execution, completed invalid output,
+and operational failure, including failures before any model call. Expose only
+identities available at that point; do not invent selected profiles or backends
+when resolution failed. Prepared execution observations distinguish execution
+from earlier preparation and consumer-held delay.
+
+Consumers can correlate the logical execution with generation attempts and learn:
+
+- the available prompt, profile, backend, and effective model identities;
+- whether generation is initial or structural repair, and which transient
+  attempt occurred within it;
+- capacity-wait, generation, and retry-delay measurements with clear boundaries;
+- classified outcomes and actual retry and repair counts; and
+- provider-reported usage already received, even when a subsequent step fails.
+
+Attempt measurements and terminal totals have explicit accounting semantics so
+consumers do not add aggregates to their constituent attempts. Missing usage is
+distinguished from measured zero; partial provider reporting must not become
+falsely complete accounting. Do not estimate unknown usage or claim a complete
+billing ledger. Preserve existing successful-result usage semantics and use a
+presence-aware observation representation rather than silently redefining zero.
+
+Observation payloads contain metadata, not credentials, request/response bodies,
+raw provider messages, or arbitrary collaborator error strings. Metadata itself
+may be sensitive and remains subject to application handling. Content capture
+is not part of this scope. Public contracts must state event ownership, delivery
+ordering, concurrency, and callback behavior without exposing internal types.
+
+### Representative Consumer Evidence
+
+Add a compact maintained offline example of a structured proposal followed by
+an independent LLM assessment. Use embedded prompts/schemas, application-owned
+profiles, synthetic content, a shared finite backend, and collected observations.
+Use a controlled HTTP boundary when demonstrating built-in transient retry
+behavior; injecting an already-retrying model client would not prove that path.
+
+Show that schema validity does not imply domain acceptance. Keep transcript
+processing, semantic acceptance, reports, and retained artifacts application-owned.
+Do not turn the example into a miniature Audita pipeline or a failure-case menu.
+
+Focused tests, rather than one oversized example, establish recovery, exhaustion,
+disablement, retry timing, cancellation during capacity/backoff/transport, repair
+interaction, peak concurrency, failure accounting, usage presence, and redaction.
+Use deterministic clocks/waits or controlled transports where needed; no paid
+calls, real credentials, timing-sensitive sleeps, or duplicated test matrices.
+
+## Architecture And API Boundaries
+
+Keep execution machinery under `internal/`; the root package remains a facade
+for intentional consumer contracts. Maintain one retry-policy owner and one
+capacity manager. Keep provider wire decoding in its adapter, lifecycle
+coordination in the execution layer, and application reporting downstream.
+Exact internal package arrangement belongs in the implementation plan.
+
+Do not add a second run API solely for diagnostics. Preserve existing operation
+signatures and optional-observation behavior for callers that do not use it.
+Avoid public clock/sleeper abstractions introduced only for internal testing.
+No profile/backend catalog schema changes are assumed by this roadmap.
+
+## Documentation And Compatibility
+
+Default retries change request counts, latency, and potential cost. Target a
+minor release with explicit migration guidance, including how to retain
+single-attempt behavior and avoid duplicate retries. Canonical GoDoc owns exact
+API and error semantics; integration docs own HTTP retry behavior; consumer docs
+own adoption examples. Update internal documentation and maintained-example
+validation only where responsibilities actually change. Release notes summarize
+and link rather than become a second contract owner.
+
+## Non-Goals
+
+No application pipeline, domain validator, cross-process scheduler, new task
+queue, monetary budget, rate limiter, circuit breaker, automatic backend/model
+failover, streaming/tool support, telemetry dependency, persistent run store,
+or additional content-retention facility. Existing fallback-profile behavior
+is not repurposed as runtime provider failover. Licensing is already complete
+in source and is not a further implementation dependency.
+
+## Open Decisions
+
+Resolve these before making the companion implementation plan decision-complete.
+API spelling and event field layout also belong in that plan.
+
+### Retry Configuration And Timing
+
+**Recommendation:** An engine default with a request-scoped retry-count override,
+using presence to distinguish omission from zero. Keep policy out of profile
+YAML initially. Use a modest fixed exponential backoff policy with jitter;
+select concrete timing values in the plan. Treat valid `Retry-After` as a minimum
+delay subject to cancellation rather than truncating it and retrying too early.
+This supplies normal defaults and per-operation control without another profile
+inheritance dimension.
+
+**Alternative:** Engine-only retry configuration is smaller but makes mixed
+single-attempt and retrying workflows require separate engines, splitting their
+capacity pools. A richer public timing policy is viable for consumer tuning but
+adds a broader contract before there is demonstrated need.
+
+### Transport-Failure Eligibility
+
+**Recommendation:** Initially retry only the selected HTTP statuses, and expose
+transport failures for consumer handling without automatic replay. This avoids
+pretending a generic network error proves that no billable work occurred.
+
+**Alternative:** Include a narrow documented set of transient transport failures.
+This recovers more outages but requires precise classification, testing, and
+clear duplicate-work warnings. Even HTTP-status retries do not promise zero
+duplicate billing; conservative eligibility only reduces ambiguity.
+
+### Timeout Scope
+
+**Recommendation:** Keep existing generation/HTTP timeouts as per-attempt limits;
+the caller's context bounds the complete operation, including capacity waiting,
+retry delays, and repair. Do not reset the caller's deadline between attempts.
+This preserves the meaning of the existing transport settings and avoids adding
+another competing total-timeout setting. Document that callers needing a total
+time budget must provide a deadline; multiple attempts can exceed one attempt's
+timeout. An expired caller deadline ends the run, never starts another attempt.
+
+**Alternative:** Apply the configured generation timeout across an entire retry
+sequence. This provides a tighter aggregate bound without caller changes, but
+reinterprets an existing setting and gives later attempts less time following
+capacity contention or provider-directed delays. It also needs a clear boundary
+between separate repair generations and their retry sequences.
+
+### Observer Delivery
+
+**Recommendation:** Synchronous callbacks, ordered within an execution, with
+concurrent invocations possible across executions sharing a callback. Do not
+invoke consumer code under capacity-manager locks or add a background delivery
+queue. Callbacks cannot return execution-policy errors; callers keep them fast
+and handle their own reporting failures. Panics follow ordinary injected-code
+behavior rather than being silently swallowed. This minimizes machinery and
+makes delivery timing understandable; slow callbacks can still delay a run.
+
+**Alternative:** Return a separate execution report on both success and failure.
+This avoids callback latency but requires a broader result/error API and retains
+attempt history internally. Async delivery offers isolation but introduces
+buffering, dropping, shutdown, and completion semantics disproportionate to the
+current need.