promptkit/docs/roadmap/implementation.md

Reliable Execution Implementation Plan

Scope And Working Rules

Implement the target in reliable-execution.md. This plan is for sequential execution by a coding agent, one numbered stage per prompt. Stages intentionally separate transport refactoring, retry behavior, lifecycle instrumentation, public integration, and consumer evidence. Do not merge stages merely because their production changes look small.

Read all repository policies, especially the agent, architecture, documentation, testing, and release policies. Preserve unrelated changes. Commit each completed, validated stage with a plain-English message and report remaining work. Do not push, tag, publish, rewrite history, or change dependencies. Licensing is complete. No new live-provider checks or release gates are needed.

Use focused tests at each stage; run race tests for concurrency changes. Keep intermediate commits building, and expose public controls only when their behavior works. Update owning documentation alongside implemented behavior, not ahead of it. Exact internal helper names may change without approval; the consumer semantics and explicit choices below must not drift silently.

Decisions Shared By All Stages

Public Controls And Prepared Execution

• Add Config.TransientRetries *int and RunRequest.TransientRetries *int. Nil selects two additional attempts at construction or inherits the engine value on a request. Zero disables retries. Reject negative values with ErrInvalidConfig or ErrInvalidRequest, respectively. Do not invent a small maximum; count actual attempts with uint64 and convert the nonnegative budget before adding the initial attempt, avoiding int overflow at MaxInt. Use overflow-safe duration arithmetic. Copy pointed values.
• Validate request retry values in Prepare, PrepareExecution, and Run. Prepare performs no retries; PrepareExecution freezes the effective retry count privately. Do not add retry settings to profile YAML, execution targets, provider payloads, prepared metadata, or prompt hashes.
• Add RunRequest.Observer ExecutionObserver, with type ExecutionObserver func(ExecutionEvent). It has no error return and is excluded from JSON. Ordinary Run observes that invocation. Successful PrepareExecution retains the function in its private payload for the owning RunPrepared; Prepare and PrepareExecution do not emit execution events. No new RunPrepared parameters or parallel execution API are introduced.
• RunPrepared observation starts only after successful atomic claim. Invalid, foreign, reused, discarded, or nil handles cannot supply an execution observer and emit no events. Preparation failures return their existing errors without execution events. Ordinary Run does observe pre-generation errors, including facade conversion errors and a nil engine if an observer was supplied.
• Clear retained observers and private execution settings on claim completion or discard along with the existing private payload. Never invoke observers while holding the prepared-handle mutex. A callback is retained by identity; its closure is not cloned. Document caller synchronization responsibilities.
• Injected LLMClient values are never automatically retried, even when configured retry values are positive. Values are still validated. A custom HTTP client used by the built-in implementation does receive built-in retries; explicit zero disables them when a transport already owns retries.

Retry Semantics

Only HTTP 429, 500, 502, 503, and 504 permit a retry. Transport failures, timeout, malformed envelopes, invalid requests, and nil collaborator responses do not. Content repair remains separate. Each repair generation gets the same configured transient budget as initial generation, never a budget shared across the run.

For retry ordinal k starting at 1, use equal jitter in [base/2, base], where base is min(500 milliseconds × 2^(k-1), 8 seconds). Cap before doubling to avoid overflow. These are documented defaults, not public clock/backoff options. Internal tests can supply a clock, jitter function, and cancellable wait function through private constructors; do not change package globals or sleep in tests.

Parse Retry-After as unsigned decimal seconds or an HTTP date via http.ParseTime, following RFC 9110 section 10.2.3. Use the first header value; trim whitespace; negative, fractional, or malformed values use local backoff. Past dates supply zero additional minimum. Measure a valid delay from response receipt, not from the end of body decoding; wait until the later of that not-before time and the local backoff deadline. Never apply the 8-second local cap to the provider minimum. Saturate unrepresentably large valid delays safely rather than wrapping or treating them as permission to retry early. Cancellation interrupts the wait; it must not busy-loop.

Check the caller context before waiting, before permit acquisition, and before dispatch. Caller cancellation/deadline wins when observed at those boundaries. Do not proactively reject long provider delays; wait cancellably. A context without a deadline can wait a long time, which must be documented.

Existing HTTP-client and target generation timeouts apply separately to each actual attempt after capacity admission. Retry waiting and capacity waiting use the original caller context, not a previously canceled attempt context. No new overall timeout is imposed. Never reset the caller's deadline.

Freeze built-in request payload, URL, and resolved authentication once per generation sequence, before its first attempt; create a fresh HTTP request/body reader and timeout context for every attempt. An absent optional key must remain absent if the environment changes during retries. Repair is a new generation sequence and may resolve the environment again, consistent with existing calls. Do not freeze environment values at PrepareExecution time.

When R is zero, return the ordinary first failure, not retry exhaustion. When R is positive and all R+1 attempts return eligible statuses, return exhaustion. If a later failure is noneligible or the context stops the sequence, return that failure rather than exhaustion. Count attempts actually dispatched, not waits scheduled. For one run, the bound is (R+1)×(P+1), where P is the repair budget; document the formula but do not compute it using overflow-prone int products.

Error Contract

Add public ErrTransport, ErrMalformedResponse, and ErrRetriesExhausted. Transport/malformed failures during generation also retain ErrLLMGenerate. Preserve standard context identities and original transport causes for errors.Is/errors.As. Do not expose new internal concrete error types as the consumer contract or stringify sensitive transport errors while mapping them.

Add immutable RetryExhaustedError with Attempts() uint64, Unwrap() error, Is(error) bool matching ErrRetriesExhausted, and redacted Error/GoString. Map its cause recursively so errors.As still finds the existing public GenerationError and status/detail accessors. Nil receivers and zero values must be safe, consistent with existing typed errors. No stable JSON contract. Do not let the current provider-error special case in mapPublicError discard the exhaustion wrapper or outer validation/generation identities.

Injected errors retain their identities; do not guess transport or HTTP classifications by text or structural resemblance. An injected client can use the public sentinels deliberately. A nil successful collaborator response is malformed generation. Completed structural rejection remains a normal result with failed validation; operational validation errors remain errors with no partial RunResult.

Observation Values And Accounting

Define public values in observations.go, with separate internal equivalents and facade conversion, not aliases. They have no stable serialized contract; consumers translate them into application-owned records.

ExecutionEvent contains:

• Kind ExecutionEventKind: constants ExecutionStarted, ExecutionAttempt, ExecutionRetryDelay, ExecutionFinished (string values started, attempt, retry_delay, finished).
• RunID string, Time time.Time, and PromptID, PromptVersion, ProfileID, BackendID, Model string fields. Use the same RunID as a successful result. Start events for ordinary runs have no resolved identities; prepared runs may supply their already-resolved snapshot. Enrich later events as resolution succeeds. Do not label requested but unresolved values as selected values.
• Generation uint64 (1 initial, 2 first repair), Attempt uint64 (1-based within a generation), and Dispatched bool. Non-attempt events use zero indexes unless describing the retry delay after a specific failed attempt.
• CapacityWait, GenerationDuration, RetryDelay, Duration time.Duration.
• Outcome ExecutionOutcome: success, invalid_output, error, or empty when not applicable; corresponding constants use the ExecutionOutcome prefix.
• Failure ExecutionFailureKind, StatusCode int, RetriesExhausted bool. Failure constants use the ExecutionFailure prefix; ExecutionFailureNone is the zero value empty string. Other values are canceled, deadline, transport, provider_status, malformed_response, capacity, invalid_config, invalid_request, prompt_not_found, profile_not_found, prompt_load, profile_load, artifact_load, prompt_render, validation, generation, other. Missing credentials/profile selection use invalid_request. Classify context identities first, then specific causes before broad operation categories. Exhaustion is a separate flag, not a replacement for provider_status.
• Usage ObservedTokenUsage and terminal Attempts, TransientRetries, RepairCalls uint64. Nonterminal aggregate counters are zero.

ObservedTokenUsage has the existing five token field names, each an ObservedTokenCount{Value int; Known bool; Complete bool}. An attempt field is known/complete exactly when the provider supplied that field as a non-null integer. An aggregate Value sums known contributions, Known means at least one known contribution, and Complete means every dispatched attempt reported that field. For zero dispatched attempts use known complete zero totals. Preserve reported values without deriving totals or reconciling fields. Unknown fields have Value zero. Counts on non-dispatched attempts are unknown and excluded from usage aggregation. This distinguishes partial accounting from complete zero.

Add optional GenerateResponse.UsagePresence *TokenUsagePresence with JSON tag usage_presence,omitempty; TokenUsagePresence contains five bool fields with the corresponding snake_case JSON names. For nil from legacy injected clients, nonzero fields are known and zero fields are unknown: the old interface cannot distinguish omitted zero from measured zero. An explicit presence value permits known zeros and governs all five fields. Copy it on conversion. Existing result accounting still sums all Usage values regardless of this observation metadata; TokenUsage and existing RunResult JSON remain unchanged. New GenerateResponse JSON round-trips the additive field and omits nil. The built-in adapter always supplies explicit presence; absent/null wire counts are unknown.

Emit one attempt-completion event per acquisition attempt, including interrupted capacity waiting with Dispatched=false. GenerationDuration covers dispatch through response decoding/closure, excludes capacity/backoff/callback time. CapacityWait covers acquiring a permit, and is zero for the unlimited path. Release the active permit before invoking observers. Dispatched means invoking the prepared HTTP attempt or injected Generate, not a proven network packet or billable request; hidden HTTP redirects/transport retries are not observable.

Emit retry-delay completion after each actual wait, including interrupted waits, with elapsed RetryDelay and the failed attempt indexes. No delay event when no wait starts. Emit terminal totals once on each normal return, including errors, after admission release. Attempts counts dispatched calls; TransientRetries counts dispatched attempts numbered above 1; RepairCalls counts generations above 1 with at least one dispatched attempt. Do not count scheduled work. Terminal phase-duration fields sum their respective measured phases; Duration is wall-clock operation time and can additionally include preparation, validation, callbacks, and other work. Do not add terminal usage to attempt usage. Attempt success means generation succeeded, not that later output validation accepted it.

Synchronous callbacks are ordered per operation, may overlap across operations, and cannot return errors. Event values contain no mutable containers or error objects. Never include endpoint URLs, headers, keys, session values, bodies, validation diagnostics, or provider text. Metadata identifiers can still be sensitive. Callback panics propagate; defer cleanup before first callback after resource acquisition. Do not promise terminal events after panics or process termination and do not invoke further callbacks while unwinding a panic. Callbacks must return for the operation to progress; cancellation cannot forcibly interrupt consumer code. Warn against blocking on work requiring the same run's admission slot. No asynchronous delivery or durable history.

Stage 1 — Expose Missing Generation Failure Categories

Completed: 2026-09-23.

Read errors.go, generation_error.go, llm_adapter.go, the LLM error types, and public error-boundary tests. Add public transport/malformed sentinels and map current built-in errors while preserving cause identity and redacted text. Classify actual request execution failures as transport, not unrelated request encoding failures merely sharing the old broad ErrRequestFailed sentinel. Normalize nil successful injected responses into the malformed category. Do not add retries or observers yet. Retain existing provider-status and validation behavior. Add the error GoDoc and a short consumer-guide classification update.

Verify public tests for transport causes, caller cancellation/deadline, malformed framing/choices, nil response, provider detail redaction, and injected arbitrary errors. Reuse current fixtures; avoid duplicating the entire transport matrix. Run go test ./.... Commit when mappings and existing tests pass.

Stage 2 — Preserve Usage Presence Through The Adapter

Completed: 2026-09-23.

Read wire response decoding, domain/public response types, facade conversion, and JSON contract tests. Add the presence types and optional public response field above. Convert wire usage counts to pointer-aware decoding, retaining existing integer values and response validation behavior. Explicit null and omission are unknown; numeric zero is known. Invalid numeric types retain current malformed-response handling. Apply the nil-presence fallback above only to observation metadata, never to existing successful-result accounting.

Test absent usage, partial nested usage, nulls, zero, and all five reported fields at the decoder boundary. Test public optional-field JSON round trips and defensive copying once at the facade. Existing successful RunResult totals and JSON must be unchanged. Document injected-client presence semantics in GoDoc. No observation API yet. Validate affected packages and commit.

Stage 3 — Separate Frozen HTTP Preparation From One Attempt

Completed: 2026-09-23.

Refactor internal/llm/OpenAICompatibleClient without changing its one-shot Generate behavior. Add an internal PrepareGeneration operation that validates and encodes one request, resolves credentials once, and returns private frozen state with an Execute(ctx) method. It creates a fresh body/request/deadline on every Execute and closes every response before returning.

Return an internal attempt outcome containing response, error, HTTP status, usage plus presence, and parsed provider not-before timing. Preserve known usage from a strictly decoded successful envelope even if missing choices/content subsequently make it malformed; do not salvage data from invalid framing or non-success error bodies. Preserve current body-size limits and redaction. The existing Generate method delegates once and returns no partial response on error. After failed body reading/decoding, retain an observed attempt-context cancellation/deadline instead of hiding it behind malformed-response formatting. Timing/header data remains internal, not provider-error strings.

Implement Retry-After parsing here, but no sleeping or retry loop. Test frozen payload/authentication across repeated Execute calls, including an initially absent optional key; independent per-attempt timeout contexts; body closure; usage on malformed choices; and header parsing with an explicit receipt time. Use separate Execute contexts to verify a canceled attempt does not poison the next call. All existing wire tests must pass. Commit this refactoring separately.

Stage 4 — Implement The Internal Retry And Capacity Coordinator

Completed: 2026-09-23.

Create internal/generation, depending on domain, llm, and capacity. It implements the existing llm.Client consumed by the runner and repairer. For built-in generation it prepares once through Stage 3, then loops over capacity-controlled Execute calls. For injected generation it dispatches once. Constructor arguments explicitly select the path; do not infer retry eligibility from arbitrary errors.

Expose an internal Manager active-permit acquisition method returning an idempotent release function. Reuse pool logic; never build another queue. Keep the current capacity client as a delegating one-shot wrapper until engine wiring moves, sharing that acquisition primitive. The coordinator defers permit release inside each attempt; no permit survives a wait or callback. Admission remains runner-owned. Invalid frozen-request preparation dispatches nothing.

Implement the exact delay/context/count semantics above with private test seams. Introduce an internal exhaustion wrapper retaining the cause and attempt count. The coordinator accepts a resolved retry count on the internal GenerateRequest; it has no public API or engine behavior change yet. Repair propagation arrives in Stage 5. Leave observation hooks for later stages rather than adding no-op public callbacks now.

Tests own the retry status table, R=0, exhaustion, a noneligible later failure, context interruption, provider minimum versus jitter, overflow-safe backoff, frozen absent/present credentials, and independent attempt deadlines. A blocked backoff must allow another call to acquire the only active permit. Cancellation and panic must not leak permits. Run focused race tests and commit.

Stage 5 — Wire Retry Configuration Through Both Execution Paths

Add public/internal configuration and request fields. Thread the copied effective retry count through ordinary resolution, private prepared payloads, executePreparedRun, newGenerationRequest, and RepairRequest. Do not add it to public GenerateRequest or provider settings: injected clients own their policy. Keep one shared resolution helper for default/override/negative validation.

In internal/engine.New, assemble the generation coordinator around the selected built-in or injected client and shared manager. Do not wrap it again in capacity.NewClient: that would hold a permit over waits or deadlock reacquisition. Initial generation and default repairer must receive the same coordinator. Remove obsolete capacity-wrapper machinery/tests only when its distinct behavior is covered by the coordinator and manager tests; do not retain two live schedulers.

Add the public exhaustion type and recursive error mapping. Update old tests expecting one eligible HTTP failure to set explicit zero where their purpose is not retry behavior, rather than making unrelated tests wait through backoff. Document the newly active controls, default two retries, and per-attempt timeout semantics now. Add public tests for request/engine precedence, copied pointers, prepared freezing, repair budget multiplication, exhaustion cause access, and injected-client single dispatch. Run full tests and focused race tests; commit.

Stage 6 — Build The Internal Observation Recorder

Create internal/observation for invocation-local counters, presence-aware aggregation, and synchronous dispatch. Internal event/value counterparts and the callback function type belong in domain so domain requests/payloads can carry them without importing the recorder package. The recorder depends only on domain/standard library, not provider or root types. No global map, history slice, mutex held across callbacks, goroutine, or background queue is needed. Keep outcome classification at owning boundaries; recorder consumes normalized failure metadata, never raw error strings.

Move/reuse run-ID generation here so observation and successful results have one identity owner. Provide start/update/attempt/delay/finish operations and a private context key to pass the invocation-local recorder through the existing internal client interfaces. Context propagation is solely internal plumbing, not public control policy. A nil observer avoids event delivery, but counts needed for errors and existing results must still work. If run-ID creation fails, report an error terminal event with an empty RunID rather than invent a valid identifier.

Test phase sums, dispatched-only counters, partial/unknown/zero usage, no double counting, callback order, retained-value independence, and panic propagation. Do not promise event delivery across panics. There is no public observer or runner wiring in this stage. Update the internal inventory when the package lands.

Stage 7 — Instrument Attempts And Retry Delays

Connect the coordinator to the internal recorder through context. Record measured capacity wait even on interrupted acquisition, actual dispatch and outcome, usage from Stage 3, and completed/canceled backoff. Release the active permit before callback delivery. Collect injected response usage using presence from Stage 2; arbitrary injected failures have unknown usage. Error response bodies are not mined for usage.

Do not add an event for request freezing failure: the terminal operation will report it without pretending a dispatch happened. A completed attempt callback that cancels context must prevent another attempt. Measure phase end before its callback so observer work is not reported as provider time. Feed normalized HTTP and context classifications without exposing sensitive strings.

Tests cover retry then success, exhaustion, interrupted wait, canceled capacity, success then later repair failure (using internal recorder plumbing), legacy injected usage and explicit unknown fields, and callbacks outside permits/locks. Use synchronization rather than elapsed-time thresholds. Run race tests; commit.

Stage 8 — Add Ordinary-Run Lifecycle And Public Observation API

Add public observation types/conversion and RunRequest.Observer. The facade adapts the function and delegates bookkeeping to the internal recorder. Start ordinary Run observation before request conversion and nil-engine rejection; finish after public error mapping so the terminal classification matches consumer errors. Use the recorder's RunID/start in the runner instead of generating another ID. Without an observer preserve existing Result timing conventions; with an observer its Duration includes the facade boundary and can be slightly longer.

Enrich resolved identities at the runner's resolution boundary, before admission; do not copy unresolved input strings into selected fields. Generation indexes are passed through initial and repair paths. Ensure final totals include completed generation usage before validation or repair errors. Emit finished only on normal return, after all admission cleanup; do not use a defer that calls user callbacks during panic unwinding. Keep Run signatures and no-partial-result behavior.

Tests cover source/conversion failure, nil engine, admission rejection, validation operational failure, valid/invalid/skipped validation, success followed by repair failure, matching RunID, redaction, and observer panic cleanup. Verify callback mutation/retention cannot alter execution state. Update public GoDoc and consumer guidance for ordinary observations only; run root and internal race tests; commit.

Stage 9 — Observe Prepared Execution Without Weakening Its Lifecycle

Retain the adapted observer with the effective retries in the private prepared payload. Claim atomically before starting observation, release the claim mutex, install payload cleanup, then emit started. Preparation emits nothing; successful claim followed by credential failure, capacity rejection, or cancellation still emits finished on a normal return. Invalid/reused/foreign handles emit nothing and must neither invoke nor discard another execution's callback.

Use the same recorder, identity enrichment, failure normalization, and terminal accounting as ordinary execution. Keep callback work outside handle and manager locks. Discard clears observer references without calling them. Concurrent calls using one handle result in one executing observation sequence, not duplicate runs.

Add focused prepared public tests for frozen retry choice/observer, discarded and foreign handles, claim races, no preparation events, later failures with usage, and panic cleanup. Preserve existing preparation/result timing distinctions. Document the prepared observer lifetime and exclusions. Run race tests; commit.

Stage 10 — Verify Cross-Cutting Public Contracts

Review assembled behavior through the public facade using controlled HTTP transports and small synthetic fixtures. Add only missing integration evidence:

• Two profiles sharing one finite backend coordinate initial, retry, and repair traffic. Backoff frees the active slot but not logical admission. Different backend pools remain independent; unlimited/endpoint-only paths still work.
• A sequence of transient failure, successful invalid output, repair transient failure, and successful valid output produces consistent counts, known partial usage, result usage, events, and error identities. Exercise exhaustion and cancellation as focused variants, not a Cartesian product.
• Frozen request/authentication and prepared source immutability survive retries. Noneligible errors never replay. Injected client attempts remain opaque.
• Safe default error/event formatting and JSON never disclose credential markers, request bodies, provider message markers, or retained callback internals.

Do not require fixed callback timing, random samples, exact error wording, or internal helper-call sequences. Fix interaction defects in their owning layer. Run go test -race -vet=off ./... and go vet ./...; commit the verified scope.

Stage 11 — Add The Offline Proposal And Assessment Example

Add examples/go-library/observe with embedded prompt/schema assets, two application-owned profiles sharing a finite custom backend, and one engine. Use a controlled RoundTripper under the built-in client: return one eligible status before a valid structured proposal, then a structurally valid assessment that the application can accept/reject independently. Collect metadata events and print a compact structured summary, including actual attempt/retry counts and assessment outcome. Do not log prompt/response bodies or provider detail. Accept the real short default backoff in this executable demonstration; unit tests still use controlled waiting. Do not add public timing controls for it.

Keep cancellation/exhaustion/schema-invalid branch coverage in focused tests from earlier stages, not a large example switchboard. Add this example once to the existing release-check script with semantic output assertions; do not run a redundant second integration suite there. Update the development guide, testing policy's example inventory, and internal overview accordingly. Run the example and its relevant tests; commit.

Stage 12 — Complete Documentation And Final Validation

Reconcile GoDoc, consumer error/observer/retry guidance, provider integration docs, and internal runner/LLM/capacity/generation/observation documentation with the implemented design. Remove claims that no retries exist or that usage can only be learned on success. Clearly distinguish observations from complete cost accounting and explain nil legacy injected usage presence.

Document single-attempt migration via explicit zero, multiplication with repair and application semantic retries, provider-delay/caller-deadline behavior, callback lifetime/concurrency/panics, and no-partial-result compatibility. Do not copy the same field reference into multiple documents. Link the new example. Update Audita's wishlist status to implemented where verified. Do not invent a release version or tag; versioned release notes and completed-roadmap retirement belong to the later authorized release-preparation task.

Run ./scripts/release-check.sh once for the final implementation, inspect the diff for accidental contracts, dependencies, or artifacts, and verify no public signature exposes internal types. Record results and any real remaining gaps. Commit the final coherent documentation/integration work; report readiness for review without publishing or waiting for CI.

Open Questions

None requiring a maintainer decision remain. The approved roadmap policies and the explicit API, accounting, timing, and lifecycle choices above are the basis for this implementation. If implementation reveals a conflict with an existing explicit contract, report the concrete conflict rather than silently adding a restriction or weakening that contract.