diff --git a/README.md b/README.md index f80601b..c63c48f 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,10 @@ boundary and constraints that framework work must preserve. ## Release Guidance +Consumers upgrading to `v0.11.0` should read the +[v0.11.0 changelog and migration guide](docs/releases/v0.11.0.md), particularly +the default retry behavior and license change. + Consumers upgrading to the `v0.10.0` profile-policy transition should read the [v0.10.0 transition guide](docs/releases/v0.10.0.md). diff --git a/docs/releases/v0.11.0.md b/docs/releases/v0.11.0.md new file mode 100644 index 0000000..eaca839 --- /dev/null +++ b/docs/releases/v0.11.0.md @@ -0,0 +1,76 @@ +# Promptkit v0.11.0 + +This release adds reliable, observable model execution and changes Promptkit's +license to LGPL-3.0-or-later. + +## Summary + +- Built-in model calls can recover from selected transient HTTP failures. +- Optional execution observations report attempts, timing, outcomes, and known + token usage, including work completed before a run fails. +- Public errors distinguish transport failures, malformed responses, and retry + exhaustion without parsing diagnostic text. + +## Compatibility + +The built-in client now defaults to two additional transient attempts. This can +increase latency, request counts, and cost; retries do not guarantee exactly-once +generation. Injected `LLMClient` implementations retain their own retry policy +and are not automatically replayed. Existing generation timeouts apply per +attempt, while the caller context governs the complete operation. See the +[provider integration contract](../integrations/openai-compatible-chat.md). + +Public retry and observation controls are additive. Existing successful-result +usage semantics remain unchanged, and operational failures still return no +partial `RunResult`. Observations distinguish unknown usage from reported zero; +they are not a complete billing ledger. See the +[consumer guide](../consumers/pkg-promptkit.md) and +[observation GoDoc](../../observations.go). + +Promptkit is now licensed under LGPL-3.0-or-later. Review the +[license summary](../../README.md#license), [LGPL text](../../LICENSE), and +[accompanying GPL text](../../COPYING) before adoption or distribution. +Previously published releases and dependency licenses are unchanged. + +## Upgrade + +Update the dependency: + +```sh +go get gitea.maximumdirect.net/eric/promptkit@v0.11.0 +``` + +To retain single-attempt behavior for built-in generation, set +`Config.TransientRetries` to a pointer to zero. `RunRequest.TransientRetries` +can override the engine choice per request. Review any existing application or +HTTP-transport retry loops to avoid unintentionally multiplying attempts. +Transient retries and structural repairs have separate budgets; provide a +caller deadline when a total execution time limit is needed. + +Observation is opt-in through `RunRequest.Observer`, including for prepared +execution. Callbacks are synchronous; shared observers must support concurrent +runs. Injected clients can supply `GenerateResponse.UsagePresence` to explicitly +report known zero counts. No observer or usage-presence changes are required +for existing consumers. See the [consumer guide](../consumers/pkg-promptkit.md) +for adoption and the [offline example](../../examples/go-library/observe/main.go) +for a structured proposal-and-assessment workflow. + +## Changes + +- Centralized built-in retry coordination with cancellable backoff and provider + retry timing. Retries and structural repair share existing backend capacity; + backoff releases active permits without releasing logical run admission. +- Added `ErrTransport`, `ErrMalformedResponse`, `ErrRetriesExhausted`, and + `RetryExhaustedError`, retaining applicable underlying error identities. +- Added request-scoped lifecycle observations and presence-aware token accounting + for ordinary and prepared execution, without recording prompt/response bodies. +- Isolated delegated operations' observation state and made injected-client + attempt classifications consistent with terminal classifications. +- Added an offline observation example and regression coverage for retry, + repair, capacity, error, and observation interactions. +- Adopted LGPL-3.0-or-later and streamlined the documented local release process. + +Backend catalog selections are unchanged from v0.10.0: OpenRouter +`v2.20260922.1` and Rakestrawhome `v2.20260922.0`. See the +[catalog compatibility policy](../policy/release.md#catalog-profile-compatibility) +and [profile inventories](../formats.md#external-profile-inventories). diff --git a/docs/roadmap/audita-feature-requests.md b/docs/roadmap/audita-feature-requests.md deleted file mode 100644 index 87ed49d..0000000 --- a/docs/roadmap/audita-feature-requests.md +++ /dev/null @@ -1,200 +0,0 @@ -# Audita Consumer Feature Requests - -**Last updated:** 2026-09-23 - -**Status:** Requests 1–6 are implemented in source but not yet released. -These requests originated from a review of Promptkit v0.10.0. They retain -Audita's consumer context and priorities, not a current API specification or -a delivery commitment for Audita adoption. - -Requests 1–5 were resolved through the -[reliable-execution roadmap](reliable-execution.md) and its -[implementation plan](implementation.md). The [consumer guide](../consumers/pkg-promptkit.md) -and public GoDoc own current behavior. No release or Audita adoption is -claimed here. - -## Consumer Context - -Audita polishes transcripts through ordered correction modules. It makes LLM calls -both to propose corrections and to assess proposed changes, alongside deterministic -domain validation. It needs bounded execution, understandable failures, and useful -run diagnostics without disclosing credentials or transcript content unnecessarily. - -Audita would like Promptkit to own shared prompt and LLM execution capabilities, -while Audita retains transcript processing, module orchestration, domain acceptance, -correction application, reports, and diagnostic artifact retention. Suite-wide -consistency is the adoption goal; these requests do not ask Promptkit to reproduce -Audita's existing APIs or internal architecture. - -The [consumer guide](../consumers/pkg-promptkit.md), -[format reference](../formats.md), and -[provider integration contract](../integrations/openai-compatible-chat.md) -remain the owners of implemented behavior. This document specifies desired -consumer outcomes only; API shape and implementation are left to Promptkit. - -## 1. Transient-Failure Retries - -**Implemented in source; not yet released.** The built-in HTTP path owns the -selected transient-status policy; injected clients keep their own policy. - -**Priority: High.** Audita should not need its own general-purpose provider retry -policy to recover from temporary rate limiting or service failures. - -Requested behavior: - -- Retry HTTP 429 and selected transient server failures. Suggested initial defaults - cover 500, 502, 503, and 504; retrying every 5xx status is not a requirement. -- Supply reasonable documented defaults. Two additional attempts is a proposed - starting point, with a configurable maximum and an unambiguous explicit zero - that disables retries. -- Delay retries to avoid immediate repeated requests, honor applicable provider - retry timing, and bound the operation by the caller's cancellation/deadline. - Cancellation must interrupt waiting as well as active requests. -- Define which transport failures are eligible. Document that an uncertain response - can mean the provider already performed billable work; retries do not imply - exactly-once generation. -- Keep retry attempts tied to the same prepared generation request. An automatic - retry must not silently select different prompt content, model settings, or - credential identity. -- Distinguish transient-failure retries from content-repair calls. Document how - their budgets combine so consumers can reason about maximum request counts, - latency, and potential cost. -- State whether the policy applies to built-in and injected clients. Consumers - must be able to avoid duplicate retries when another collaborator already owns - retry behavior. - -Expected consumer evidence includes a transient failure followed by success, -exhaustion, explicit retry disablement, cancellation during a delay, and interaction -with content repair. Permanent request failures must remain distinguishable. - -## 2. Execution Observations, Including Failed Runs - -**Implemented in source; not yet released.** Optional metadata callbacks cover -ordinary and claimed prepared execution, including normal error returns. - -**Priority: High.** Audita needs to explain the time and calls consumed by a logical -operation, including when that operation never produces a successful result. - -Requested observations include: - -- Correlation between the logical operation and its individual attempts, selected - prompt, profile, and backend. -- Attempt outcomes and enough timing information to distinguish capacity waiting, - generation, and retry delays. -- Separate counts for transient-failure retries and content-repair calls. -- Available provider-reported token usage, including completed calls preceding a - later failure, with unknown usage distinguishable from measured zero. -- Clear accounting boundaries so consumers do not double-count aggregate and - individual-attempt measurements. - -Observations should be usable by application-owned reports, logging, and metrics -without requiring a particular telemetry system or artifact layout. Default -observations should contain metadata rather than credentials or prompt/response -bodies. Any content disclosure should be an explicit consumer choice, with its -sensitivity clearly documented. - -Current result metadata and usage remain useful; the request is to make execution -and failure accounting sufficiently complete for downstream diagnostics. - -## 3. Actionable Failure Classification - -**Implemented in source; not yet released.** Public sentinels and typed errors -preserve provider status, retry exhaustion, transport, malformed-response, -capacity, validation, and context distinctions. - -**Priority: High; closely related to retries and observations.** Audita needs to -choose between failing a run, rejecting affected work, and recording a warning -without interpreting human-readable error messages. - -Build on existing public error and validation-result contracts so consumers can -reliably distinguish cancellation, deadline expiration, transport failure, -non-success provider status, malformed provider response, exhausted retries, -capacity rejection, and completed output-validation failure. - -On retry exhaustion, retain the underlying failure classification and attempt -count. Preserve safe default diagnostics while allowing deliberate access to -provider detail under the consumer's disclosure policy. Document which outcomes -carry generated output, validation findings, and partial execution observations. - -Promptkit need not decide Audita's recovery or correction-acceptance policy. - -## 4. Simple Per-Backend Capacity Limits - -**Implemented in source; not yet released.** One engine-local backend pool -coordinates initial, retry, and repair attempts across profiles selecting it. -Audita must still configure finite limits for the paid backends it adopts. - -**Priority: Adoption requirement, largely supported by existing facilities.** -Audita needs only two protections: - -- For a local backend with a hard simultaneous-query limit, keep combined Audita - traffic within its assigned capacity. For example, a DGX Spark serving - gemma-4-31b-it may allow four simultaneous queries. -- For a paid API backend, use a reasonable finite concurrency cap so a large - transcript or pipeline cannot accidentally launch dozens or thousands of - simultaneous billable calls. - -Proposal and validation calls targeting the same backend should share its limit. -The protection must cover initial calls, transient retries, and content-repair -calls. Audita does not need separate proposal/validation budgets, named workload -limits, or an additional aggregate ceiling across backends. - -Promptkit's existing engine-local, per-backend capacity controls are the intended -foundation. Audita should select finite limits for the backends it uses; adoption -must not accidentally leave paid endpoints on an unlimited path. Exact defaults -remain to be selected. No new fine-grained scheduling feature is requested. - -Verify the documented capacity scope in the consumer example, especially as retries -are added. An engine-local limit does not coordinate other processes or independent -clients using the same service; a shared local backend must have capacity allocated -accordingly or enforced at the service boundary. Cross-process coordination is not -an upstream feature request here. - -Concurrency limits bound simultaneous activity, not total token usage or spending -across a long run. A monetary budget or rate-limiting system is not requested. - -## 5. Representative Consumer Example - -**Implemented in source; not yet released.** The -[offline example](../../examples/go-library/observe/main.go) demonstrates the -built-in retry, structured proposal and assessment, metadata collection, and -separate application acceptance decision. Focused tests, not example switches, -cover cancellation and invalid-output branches. - -**Priority: High as adoption evidence.** Provide a maintained offline example of a -structured proposal call followed by an independent LLM validation call, using -embedded prompts/schemas and separate application-owned execution profiles. - -Demonstrate transient failure handling, schema-invalid output, cancellation, and -application collection of execution observations. Use synthetic inputs and -controlled model responses without credentials or paid provider calls. - -Keep schema validity distinct from domain acceptance: a structurally valid response -may still be rejected by Audita's deterministic or semantic checks. The example -should show how an application uses Promptkit's public contracts, not duplicate -Audita's processing pipeline. - -## 6. Licensing Recommendation - -**Implemented in source; not yet released.** The maintainer authorized -`LGPL-3.0-or-later`. See the [license summary](../../README.md#license), -[LGPL notice and text](../../LICENSE), and [accompanying GPL text](../../COPYING). -Existing release tags remain unchanged. Audita must select a subsequent published -release containing this change and satisfy the applicable distribution obligations. - -## Suggested Evaluation Order and Scope - -The implemented retry, classification, observation, and per-backend capacity -contracts can now be evaluated together with the representative example. -Audita still owns its migration decision and must select a later published -release before adopting this source state. - -Existing facilities for embedded sources, application-owned profiles, prepared -execution, JSON-schema validation, and request-scoped credentials are useful -adoption foundations rather than new requests. Audita can bridge its application -configuration and credentials into Promptkit's public API; a dependency between -Promptkit and ConfigKit is not requested. - -Transcript normalization, chunking, module ordering, semantic validators, correction -application, correction-ledger assembly, and filesystem retention remain Audita's -responsibility. No requested feature requires Promptkit to own those concerns. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 623f82a..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,503 +0,0 @@ -# Reliable Execution Implementation Plan - -## Scope And Working Rules - -Implement the target in [reliable-execution.md](reliable-execution.md). This plan -is for sequential execution by a coding agent, one numbered stage per prompt. -Stages intentionally separate transport refactoring, retry behavior, lifecycle -instrumentation, public integration, and consumer evidence. Do not merge stages -merely because their production changes look small. - -Read all [repository policies](../development.md#initial-orientation), especially -the [agent](../policy/agents.md), [architecture](../policy/architecture.md), -[documentation](../policy/documentation.md), [testing](../policy/testing.md), and -[release](../policy/release.md) policies. Preserve unrelated changes. Commit each -completed, validated stage with a plain-English message and report remaining -work. Do not push, tag, publish, rewrite history, or change dependencies. Licensing -is complete. No new live-provider checks or release gates are needed. - -Use focused tests at each stage; run race tests for concurrency changes. Keep -intermediate commits building, and expose public controls only when their -behavior works. Update owning documentation alongside implemented behavior, -not ahead of it. Exact internal helper names may change without approval; the -consumer semantics and explicit choices below must not drift silently. - -## Decisions Shared By All Stages - -### Public Controls And Prepared Execution - -- Add `Config.TransientRetries *int` and `RunRequest.TransientRetries *int`. - Nil selects two additional attempts at construction or inherits the engine - value on a request. Zero disables retries. Reject negative values with - `ErrInvalidConfig` or `ErrInvalidRequest`, respectively. Do not invent a small - maximum; count actual attempts with uint64 and convert the nonnegative budget - before adding the initial attempt, avoiding int overflow at MaxInt. Use - overflow-safe duration arithmetic. Copy pointed values. -- Validate request retry values in `Prepare`, `PrepareExecution`, and `Run`. - `Prepare` performs no retries; `PrepareExecution` freezes the effective retry - count privately. Do not add retry settings to profile YAML, execution targets, - provider payloads, prepared metadata, or prompt hashes. -- Add `RunRequest.Observer ExecutionObserver`, with - `type ExecutionObserver func(ExecutionEvent)`. It has no error return and is - excluded from JSON. Ordinary `Run` observes that invocation. Successful - `PrepareExecution` retains the function in its private payload for the owning - `RunPrepared`; `Prepare` and `PrepareExecution` do not emit execution events. - No new `RunPrepared` parameters or parallel execution API are introduced. -- `RunPrepared` observation starts only after successful atomic claim. Invalid, - foreign, reused, discarded, or nil handles cannot supply an execution observer - and emit no events. Preparation failures return their existing errors without - execution events. Ordinary `Run` does observe pre-generation errors, including - facade conversion errors and a nil engine if an observer was supplied. -- Clear retained observers and private execution settings on claim completion - or discard along with the existing private payload. Never invoke observers - while holding the prepared-handle mutex. A callback is retained by identity; - its closure is not cloned. Document caller synchronization responsibilities. -- Injected `LLMClient` values are never automatically retried, even when configured - retry values are positive. Values are still validated. A custom HTTP client - used by the built-in implementation does receive built-in retries; explicit - zero disables them when a transport already owns retries. - -### Retry Semantics - -Only HTTP 429, 500, 502, 503, and 504 permit a retry. Transport failures, timeout, -malformed envelopes, invalid requests, and nil collaborator responses do not. -Content repair remains separate. Each repair generation gets the same configured -transient budget as initial generation, never a budget shared across the run. - -For retry ordinal k starting at 1, use equal jitter in [base/2, base], where -base is min(500 milliseconds × 2^(k-1), 8 seconds). Cap before doubling to avoid -overflow. These are documented defaults, not public clock/backoff options. -Internal tests can supply a clock, jitter function, and cancellable wait function -through private constructors; do not change package globals or sleep in tests. - -Parse `Retry-After` as unsigned decimal seconds or an HTTP date via -`http.ParseTime`, following [RFC 9110 section 10.2.3](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3). -Use the first header value; trim whitespace; negative, fractional, or malformed -values use local backoff. Past dates supply zero additional minimum. Measure a -valid delay from response receipt, not from the end of body decoding; wait until -the later of that not-before time and the local backoff deadline. Never apply -the 8-second local cap to the provider minimum. Saturate unrepresentably large -valid delays safely rather than wrapping or treating them as permission to -retry early. Cancellation interrupts the wait; it must not busy-loop. - -Check the caller context before waiting, before permit acquisition, and before -dispatch. Caller cancellation/deadline wins when observed at those boundaries. -Do not proactively reject long provider delays; wait cancellably. A context -without a deadline can wait a long time, which must be documented. - -Existing HTTP-client and target generation timeouts apply separately to each -actual attempt after capacity admission. Retry waiting and capacity waiting use -the original caller context, not a previously canceled attempt context. No -new overall timeout is imposed. Never reset the caller's deadline. - -Freeze built-in request payload, URL, and resolved authentication once per -generation sequence, before its first attempt; create a fresh HTTP request/body -reader and timeout context for every attempt. An absent optional key must remain -absent if the environment changes during retries. Repair is a new generation -sequence and may resolve the environment again, consistent with existing calls. -Do not freeze environment values at `PrepareExecution` time. - -When R is zero, return the ordinary first failure, not retry exhaustion. When -R is positive and all R+1 attempts return eligible statuses, return exhaustion. -If a later failure is noneligible or the context stops the sequence, return that -failure rather than exhaustion. Count attempts actually dispatched, not waits -scheduled. For one run, the bound is (R+1)×(P+1), where P is the repair budget; -document the formula but do not compute it using overflow-prone `int` products. - -### Error Contract - -Add public `ErrTransport`, `ErrMalformedResponse`, and `ErrRetriesExhausted`. -Transport/malformed failures during generation also retain `ErrLLMGenerate`. -Preserve standard context identities and original transport causes for -`errors.Is`/`errors.As`. Do not expose new internal concrete error types as the -consumer contract or stringify sensitive transport errors while mapping them. - -Add immutable `RetryExhaustedError` with `Attempts() uint64`, `Unwrap() error`, -`Is(error) bool` matching `ErrRetriesExhausted`, and redacted `Error`/`GoString`. -Map its cause recursively so `errors.As` still finds the existing public -`GenerationError` and status/detail accessors. Nil receivers and zero values -must be safe, consistent with existing typed errors. No stable JSON contract. -Do not let the current provider-error special case in `mapPublicError` discard -the exhaustion wrapper or outer validation/generation identities. - -Injected errors retain their identities; do not guess transport or HTTP -classifications by text or structural resemblance. An injected client can use -the public sentinels deliberately. A nil successful collaborator response is -malformed generation. Completed structural rejection remains a normal result -with failed validation; operational validation errors remain errors with no -partial `RunResult`. - -### Observation Values And Accounting - -Define public values in `observations.go`, with separate internal equivalents -and facade conversion, not aliases. They have no stable serialized contract; -consumers translate them into application-owned records. - -`ExecutionEvent` contains: - -- `Kind ExecutionEventKind`: constants `ExecutionStarted`, `ExecutionAttempt`, - `ExecutionRetryDelay`, `ExecutionFinished` (string values `started`, `attempt`, - `retry_delay`, `finished`). -- `RunID string`, `Time time.Time`, and `PromptID`, `PromptVersion`, `ProfileID`, - `BackendID`, `Model` string fields. Use the same RunID as a successful result. - Start events for ordinary runs have no resolved identities; prepared runs - may supply their already-resolved snapshot. Enrich later events as resolution - succeeds. Do not label requested but unresolved values as selected values. -- `Generation uint64` (1 initial, 2 first repair), `Attempt uint64` (1-based within a - generation), and `Dispatched bool`. Non-attempt events use zero indexes unless - describing the retry delay after a specific failed attempt. -- `CapacityWait`, `GenerationDuration`, `RetryDelay`, `Duration time.Duration`. -- `Outcome ExecutionOutcome`: `success`, `invalid_output`, `error`, or empty - when not applicable; corresponding constants use the `ExecutionOutcome` prefix. -- `Failure ExecutionFailureKind`, `StatusCode int`, `RetriesExhausted bool`. - Failure constants use the `ExecutionFailure` prefix; `ExecutionFailureNone` - is the zero value empty string. Other values are `canceled`, `deadline`, - `transport`, `provider_status`, - `malformed_response`, `capacity`, `invalid_config`, `invalid_request`, - `prompt_not_found`, `profile_not_found`, `prompt_load`, `profile_load`, - `artifact_load`, `prompt_render`, `validation`, `generation`, `other`. - Missing credentials/profile selection use invalid_request. Classify context - identities first, then specific causes before broad operation categories. - Exhaustion is a separate flag, not a replacement for provider_status. -- `Usage ObservedTokenUsage` and terminal `Attempts`, `TransientRetries`, - `RepairCalls uint64`. Nonterminal aggregate counters are zero. - -`ObservedTokenUsage` has the existing five token field names, each an -`ObservedTokenCount{Value int; Known bool; Complete bool}`. An attempt field is -known/complete exactly when the provider supplied that field as a non-null -integer. An aggregate Value sums known contributions, Known means at least one -known contribution, and Complete means every dispatched attempt reported that -field. For zero dispatched attempts use known complete zero totals. Preserve -reported values without deriving totals or reconciling fields. Unknown fields -have Value zero. Counts on non-dispatched attempts are unknown and excluded from -usage aggregation. This distinguishes partial accounting from complete zero. - -Add optional `GenerateResponse.UsagePresence *TokenUsagePresence` with JSON tag -`usage_presence,omitempty`; `TokenUsagePresence` contains five bool fields with -the corresponding snake_case JSON names. For nil from legacy injected clients, -nonzero fields are known and zero fields are unknown: the old interface cannot -distinguish omitted zero from measured zero. An explicit presence value permits -known zeros and governs all five fields. Copy it on conversion. Existing result -accounting still sums all Usage values regardless of this observation metadata; -`TokenUsage` and existing `RunResult` JSON remain unchanged. New -GenerateResponse JSON round-trips the additive field and omits nil. The built-in -adapter always supplies explicit presence; absent/null wire counts are unknown. - -Emit one attempt-completion event per acquisition attempt, including interrupted -capacity waiting with Dispatched=false. GenerationDuration covers dispatch -through response decoding/closure, excludes capacity/backoff/callback time. -CapacityWait covers acquiring a permit, and is zero for the unlimited path. -Release the active permit before invoking observers. Dispatched means invoking -the prepared HTTP attempt or injected Generate, not a proven network packet or -billable request; hidden HTTP redirects/transport retries are not observable. - -Emit retry-delay completion after each actual wait, including interrupted waits, -with elapsed RetryDelay and the failed attempt indexes. No delay event when no -wait starts. Emit terminal totals once on each normal return, including errors, -after admission release. Attempts counts dispatched calls; TransientRetries -counts dispatched attempts numbered above 1; RepairCalls counts generations -above 1 with at least one dispatched attempt. Do not count scheduled work. -Terminal phase-duration fields sum their respective measured phases; Duration -is wall-clock operation time -and can additionally include preparation, validation, callbacks, and other work. -Do not add terminal usage to attempt usage. Attempt success means generation -succeeded, not that later output validation accepted it. - -Synchronous callbacks are ordered per operation, may overlap across operations, -and cannot return errors. Event values contain no mutable containers or error -objects. Never include endpoint URLs, headers, keys, session values, bodies, -validation diagnostics, or provider text. Metadata identifiers can still be -sensitive. Callback panics propagate; defer cleanup before first callback after -resource acquisition. Do not promise terminal events after panics or process -termination and do not invoke further callbacks while unwinding a panic. -Callbacks must return for the operation to progress; cancellation cannot forcibly -interrupt consumer code. Warn against blocking on work requiring the same run's -admission slot. No asynchronous delivery or durable history. - -## Stage 1 — Expose Missing Generation Failure Categories - -**Completed:** 2026-09-23. - -Read `errors.go`, `generation_error.go`, `llm_adapter.go`, the LLM error types, -and public error-boundary tests. Add public transport/malformed sentinels and -map current built-in errors while preserving cause identity and redacted text. -Classify actual request execution failures as transport, not unrelated request -encoding failures merely sharing the old broad ErrRequestFailed sentinel. -Normalize nil successful injected responses into the malformed category. Do not -add retries or observers yet. Retain existing provider-status and validation -behavior. Add the error GoDoc and a short consumer-guide classification update. - -Verify public tests for transport causes, caller cancellation/deadline, malformed -framing/choices, nil response, provider detail redaction, and injected arbitrary -errors. Reuse current fixtures; avoid duplicating the entire transport matrix. -Run `go test ./...`. Commit when mappings and existing tests pass. - -## Stage 2 — Preserve Usage Presence Through The Adapter - -**Completed:** 2026-09-23. - -Read wire response decoding, domain/public response types, facade conversion, -and JSON contract tests. Add the presence types and optional public response -field above. Convert wire usage counts to pointer-aware decoding, retaining -existing integer values and response validation behavior. Explicit null and -omission are unknown; numeric zero is known. Invalid numeric types retain current -malformed-response handling. Apply the nil-presence fallback above only to -observation metadata, never to existing successful-result accounting. - -Test absent usage, partial nested usage, nulls, zero, and all five reported -fields at the decoder boundary. Test public optional-field JSON round trips and -defensive copying once at the facade. Existing successful RunResult totals and -JSON must be unchanged. Document injected-client presence semantics in GoDoc. -No observation API yet. Validate affected packages and commit. - -## Stage 3 — Separate Frozen HTTP Preparation From One Attempt - -**Completed:** 2026-09-23. - -Refactor `internal/llm/OpenAICompatibleClient` without changing its one-shot -`Generate` behavior. Add an internal `PrepareGeneration` operation that validates -and encodes one request, resolves credentials once, and returns private frozen -state with an `Execute(ctx)` method. It creates a fresh body/request/deadline -on every Execute and closes every response before returning. - -Return an internal attempt outcome containing response, error, HTTP status, -usage plus presence, and parsed provider not-before timing. Preserve known usage -from a strictly decoded successful envelope even if missing choices/content -subsequently make it malformed; do not salvage data from invalid framing or -non-success error bodies. Preserve current body-size limits and redaction. -The existing Generate method delegates once and returns no partial response on -error. After failed body reading/decoding, retain an observed attempt-context -cancellation/deadline instead of hiding it behind malformed-response formatting. -Timing/header data remains internal, not provider-error strings. - -Implement Retry-After parsing here, but no sleeping or retry loop. Test frozen -payload/authentication across repeated Execute calls, including an initially -absent optional key; independent per-attempt timeout contexts; body closure; -usage on malformed choices; and header parsing with an explicit receipt time. -Use separate Execute contexts to verify a canceled attempt does not poison the -next call. All existing wire tests must pass. Commit this refactoring separately. - -## Stage 4 — Implement The Internal Retry And Capacity Coordinator - -**Completed:** 2026-09-23. - -Create `internal/generation`, depending on domain, llm, and capacity. It implements -the existing `llm.Client` consumed by the runner and repairer. For built-in -generation it prepares once through Stage 3, then loops over capacity-controlled -Execute calls. For injected generation it dispatches once. Constructor arguments -explicitly select the path; do not infer retry eligibility from arbitrary errors. - -Expose an internal Manager active-permit acquisition method returning an -idempotent release function. Reuse pool logic; never build another queue. Keep -the current capacity client as a delegating one-shot wrapper until engine wiring -moves, sharing that acquisition primitive. The coordinator defers permit release -inside each attempt; no permit survives a wait or callback. Admission remains -runner-owned. Invalid frozen-request preparation dispatches nothing. - -Implement the exact delay/context/count semantics above with private test seams. -Introduce an internal exhaustion wrapper retaining the cause and attempt count. -The coordinator accepts a resolved retry count on the internal GenerateRequest; -it has no public API or engine behavior change yet. Repair propagation arrives -in Stage 5. Leave observation hooks for later stages rather than adding no-op -public callbacks now. - -Tests own the retry status table, R=0, exhaustion, a noneligible later failure, -context interruption, provider minimum versus jitter, overflow-safe backoff, -frozen absent/present credentials, and independent attempt deadlines. A blocked -backoff must allow another call to acquire the only active permit. Cancellation -and panic must not leak permits. Run focused race tests and commit. - -## Stage 5 — Wire Retry Configuration Through Both Execution Paths - -**Completed:** 2026-09-23. - -Add public/internal configuration and request fields. Thread the copied effective -retry count through ordinary resolution, private prepared payloads, -`executePreparedRun`, `newGenerationRequest`, and `RepairRequest`. Do not add it -to public GenerateRequest or provider settings: injected clients own their policy. -Keep one shared resolution helper for default/override/negative validation. - -In `internal/engine.New`, assemble the generation coordinator around the selected -built-in or injected client and shared manager. Do not wrap it again in -capacity.NewClient: that would hold a permit over waits or deadlock reacquisition. -Initial generation and default repairer must receive the same coordinator. -Remove obsolete capacity-wrapper machinery/tests only when its distinct behavior -is covered by the coordinator and manager tests; do not retain two live schedulers. - -Add the public exhaustion type and recursive error mapping. Update old tests -expecting one eligible HTTP failure to set explicit zero where their purpose is -not retry behavior, rather than making unrelated tests wait through backoff. -Document the newly active controls, default two retries, and per-attempt timeout -semantics now. Add public tests for request/engine precedence, copied pointers, -prepared freezing, repair budget multiplication, exhaustion cause access, and -injected-client single dispatch. Run full tests and focused race tests; commit. - -## Stage 6 — Build The Internal Observation Recorder - -**Completed:** 2026-09-23. - -Create `internal/observation` for invocation-local counters, presence-aware -aggregation, and synchronous dispatch. Internal event/value counterparts and -the callback function type belong in domain so domain requests/payloads can -carry them without importing the recorder package. The recorder depends only -on domain/standard library, not provider or root types. No global map, history -slice, mutex held across callbacks, goroutine, or background queue is needed. -Keep outcome classification at owning boundaries; recorder consumes normalized -failure metadata, never raw error strings. - -Move/reuse run-ID generation here so observation and successful results have one -identity owner. Provide start/update/attempt/delay/finish operations and a private -context key to pass the invocation-local recorder through the existing internal -client interfaces. Context propagation is solely internal plumbing, not public -control policy. A nil observer avoids event delivery, but counts needed for errors -and existing results must still work. If run-ID creation fails, report an error -terminal event with an empty RunID rather than invent a valid identifier. - -Test phase sums, dispatched-only counters, partial/unknown/zero usage, no double -counting, callback order, retained-value independence, and panic propagation. -Do not promise event delivery across panics. There is no public observer or runner -wiring in this stage. Update the internal inventory when the package lands. - -## Stage 7 — Instrument Attempts And Retry Delays - -**Completed:** 2026-09-23. - -Connect the coordinator to the internal recorder through context. Record measured -capacity wait even on interrupted acquisition, actual dispatch and outcome, usage -from Stage 3, and completed/canceled backoff. Release the active permit before -callback delivery. Collect injected response usage using presence from Stage 2; -arbitrary injected failures have unknown usage. Error response bodies are not -mined for usage. - -Do not add an event for request freezing failure: the terminal operation will -report it without pretending a dispatch happened. A completed attempt callback -that cancels context must prevent another attempt. Measure phase end before its -callback so observer work is not reported as provider time. Feed normalized HTTP -and context classifications without exposing sensitive strings. - -Tests cover retry then success, exhaustion, interrupted wait, canceled capacity, -success then later repair failure (using internal recorder plumbing), legacy -injected usage and explicit unknown fields, and callbacks outside permits/locks. -Use synchronization rather than elapsed-time thresholds. Run race tests; commit. - -## Stage 8 — Add Ordinary-Run Lifecycle And Public Observation API - -**Completed:** 2026-09-23. - -Add public observation types/conversion and RunRequest.Observer. The facade adapts -the function and delegates bookkeeping to the internal recorder. Start ordinary -Run observation before request conversion and nil-engine rejection; finish after -public error mapping so the terminal classification matches consumer errors. -Use the recorder's RunID/start in the runner instead of generating another ID. -Without an observer preserve existing Result timing conventions; with an observer -its Duration includes the facade boundary and can be slightly longer. - -Enrich resolved identities at the runner's resolution boundary, before admission; -do not copy unresolved input strings into selected fields. Generation indexes are -passed through initial and repair paths. Ensure final totals include completed -generation usage before validation or repair errors. Emit finished only on normal -return, after all admission cleanup; do not use a defer that calls user callbacks -during panic unwinding. Keep `Run` signatures and no-partial-result behavior. - -Tests cover source/conversion failure, nil engine, admission rejection, validation -operational failure, valid/invalid/skipped validation, success followed by repair -failure, matching RunID, redaction, and observer panic cleanup. Verify callback -mutation/retention cannot alter execution state. Update public GoDoc and consumer -guidance for ordinary observations only; run root and internal race tests; commit. - -## Stage 9 — Observe Prepared Execution Without Weakening Its Lifecycle - -**Completed:** 2026-09-23. - -Retain the adapted observer with the effective retries in the private prepared -payload. Claim atomically before starting observation, release the claim mutex, -install payload cleanup, then emit started. Preparation emits nothing; successful -claim followed by credential failure, capacity rejection, or cancellation still -emits finished on a normal return. Invalid/reused/foreign handles emit nothing -and must neither invoke nor discard another execution's callback. - -Use the same recorder, identity enrichment, failure normalization, and terminal -accounting as ordinary execution. Keep callback work outside handle and manager -locks. Discard clears observer references without calling them. Concurrent calls -using one handle result in one executing observation sequence, not duplicate runs. - -Add focused prepared public tests for frozen retry choice/observer, discarded and -foreign handles, claim races, no preparation events, later failures with usage, -and panic cleanup. Preserve existing preparation/result timing distinctions. -Document the prepared observer lifetime and exclusions. Run race tests; commit. - -## Stage 10 — Verify Cross-Cutting Public Contracts - -**Completed:** 2026-09-23. - -Review assembled behavior through the public facade using controlled HTTP -transports and small synthetic fixtures. Add only missing integration evidence: - -- Two profiles sharing one finite backend coordinate initial, retry, and repair - traffic. Backoff frees the active slot but not logical admission. Different - backend pools remain independent; unlimited/endpoint-only paths still work. -- A sequence of transient failure, successful invalid output, repair transient - failure, and successful valid output produces consistent counts, known partial - usage, result usage, events, and error identities. Exercise exhaustion and - cancellation as focused variants, not a Cartesian product. -- Frozen request/authentication and prepared source immutability survive retries. - Noneligible errors never replay. Injected client attempts remain opaque. -- Safe default error/event formatting and JSON never disclose credential markers, - request bodies, provider message markers, or retained callback internals. - -Do not require fixed callback timing, random samples, exact error wording, or -internal helper-call sequences. Fix interaction defects in their owning layer. -Run `go test -race -vet=off ./...` and `go vet ./...`; commit the verified scope. - -## Stage 11 — Add The Offline Proposal And Assessment Example - -**Completed:** 2026-09-23. - -Add `examples/go-library/observe` with embedded prompt/schema assets, two -application-owned profiles sharing a finite custom backend, and one engine. -Use a controlled RoundTripper under the built-in client: return one eligible -status before a valid structured proposal, then a structurally valid assessment -that the application can accept/reject independently. Collect metadata events -and print a compact structured summary, including actual attempt/retry counts -and assessment outcome. Do not log prompt/response bodies or provider detail. -Accept the real short default backoff in this executable demonstration; unit -tests still use controlled waiting. Do not add public timing controls for it. - -Keep cancellation/exhaustion/schema-invalid branch coverage in focused tests -from earlier stages, not a large example switchboard. Add this example once to -the existing release-check script with semantic output assertions; do not run -a redundant second integration suite there. Update the development guide, -testing policy's example inventory, and internal overview accordingly. Run the -example and its relevant tests; commit. - -## Stage 12 — Complete Documentation And Final Validation - -**Completed:** 2026-09-23. - -Reconcile GoDoc, consumer error/observer/retry guidance, provider integration -docs, and internal runner/LLM/capacity/generation/observation documentation with -the implemented design. Remove claims that no retries exist or that usage can -only be learned on success. Clearly distinguish observations from complete cost -accounting and explain nil legacy injected usage presence. - -Document single-attempt migration via explicit zero, multiplication with repair -and application semantic retries, provider-delay/caller-deadline behavior, -callback lifetime/concurrency/panics, and no-partial-result compatibility. Do not -copy the same field reference into multiple documents. Link the new example. -Update Audita's wishlist status to implemented where verified. Do not invent a -release version or tag; versioned release notes and completed-roadmap retirement -belong to the later authorized release-preparation task. - -Run `./scripts/release-check.sh` once for the final implementation, inspect the -diff for accidental contracts, dependencies, or artifacts, and verify no public -signature exposes internal types. Record results and any real remaining gaps. -Commit the final coherent documentation/integration work; report readiness for -review without publishing or waiting for CI. - -## Open Questions - -None requiring a maintainer decision remain. The approved roadmap policies and -the explicit API, accounting, timing, and lifecycle choices above are the basis -for this implementation. If implementation reveals a conflict with an existing -explicit contract, report the concrete conflict rather than silently adding a -restriction or weakening that contract. diff --git a/docs/roadmap/reliable-execution.md b/docs/roadmap/reliable-execution.md deleted file mode 100644 index 21bbcd2..0000000 --- a/docs/roadmap/reliable-execution.md +++ /dev/null @@ -1,219 +0,0 @@ -# Reliable And Observable Execution - -**Status:** Implemented in source; not yet released. This document records the -accepted target and approved policy choices. The companion -[implementation plan](implementation.md) records implementation steps and -concrete API and algorithm decisions. Current contracts are owned by public -GoDoc and the linked consumer and integration documentation. - -## Purpose - -Make Promptkit responsible for reusable provider-failure recovery and execution -accounting, including failed operations, without becoming an application workflow -or telemetry framework. Consumers should not need separate retry loops merely -to handle temporary provider failures, nor parse error strings to understand -what happened. - -This scope incorporates requests 1–5 from the -[Audita wishlist](audita-feature-requests.md). Retries, error classification, -observations, shared capacity, and consumer evidence belong together because -they describe the same execution lifecycle. Licensing was handled separately. -Audita motivates the work but does not define an application-specific API. - -## Foundations At Planning Time - -At the time this scope was accepted, Promptkit provided structural output -validation and bounded repair, prepared execution, engine-local backend -capacity, run metadata, aggregate token usage, and typed provider-status and -capacity errors. Operational errors did not return partial run results. The -built-in path had no transient retry loop, and existing integer usage fields -did not distinguish missing reports from zero. - -The implemented contracts are owned by public GoDoc, the -[consumer guide](../consumers/pkg-promptkit.md), and the -[provider integration contract](../integrations/openai-compatible-chat.md). -The [capacity document](../internal/capacity.md) and -[runner document](../internal/runner.md) describe the existing implementation. -This historical baseline does not override their current-state descriptions. - -## Target End State - -### Bounded Transient-Failure Retries - -The built-in OpenAI-compatible client path supports configurable transient -retries. The initial policy uses two additional attempts by default and permits -an unambiguous explicit zero to disable them. Initial eligible statuses are -429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status. - -Configure an engine default with a request-scoped retry-count override, using -presence to distinguish omission from zero. Keep retry policy out of profile -YAML initially. Per-request control supports mixed workflows without requiring -separate engines and splitting their capacity pools. Use a modest fixed backoff -policy without a new public timing-configuration surface. - -- Retry delays use exponential backoff with jitter and honor valid provider - `Retry-After` timing as a minimum delay, subject to cancellation, rather than - truncating it and retrying too early. Invalid or absent timing falls back to - local policy. -- Caller cancellation and deadlines interrupt both waiting and active attempts. - Cancellation is never itself a reason to retry. -- Permanent request failures, malformed successful response envelopes, and - structurally invalid generated content do not enter this transient loop. - Structural invalidity remains the separate output-repair mechanism. -- Transport failures are classified and exposed without automatic replay in - this scope. A generic network error does not prove that no billable work - occurred. Conservative eligibility reduces ambiguity; even HTTP-status - retries do not promise zero duplicate billing. -- Retry one resolved generation request: retain its messages, model, endpoint, - settings, structured-output constraint, and resolved credential throughout - that retry sequence. Do not rerender or reload application sources between - attempts. Resolve environment credentials at execution, not when preparing - a handle that may remain unexecuted for a long time. -- Apply the same behavior to ordinary and prepared execution and to generation - performed for structural repair. A prepared handle remains single-use; its - execution may contain multiple internal provider attempts. -- Do not silently retry arbitrary injected `LLMClient` implementations. They - continue to own their retry policy. A custom HTTP transport under the built-in - client uses the built-in policy; consumers can disable it when their transport - already retries. Hidden attempts inside collaborators cannot be counted by - Promptkit and must not be presented as if they were observable. - -Retries are not exactly-once generation. A failed or uncertain request may have -performed billable work. With R additional transient attempts per generation -and P additional structural repair generations, the maximum Promptkit-controlled -provider attempts for that execution is (R + 1) × (P + 1). Cancellation and -non-retryable failures can reduce that count. Application-owned semantic retries -are separate executions and may multiply it further. - -Existing generation/HTTP timeouts remain per-attempt limits. The caller's -context bounds the complete operation, including capacity waiting, retry delays, -and repair; its deadline is never reset between attempts. Callers needing a -total time budget must provide a deadline, since multiple attempts can exceed -one attempt's timeout. An expired caller deadline ends the run rather than -starting another attempt. This preserves existing transport-setting semantics -without adding a competing total-timeout setting. - -### Shared Capacity, Not A Second Scheduler - -Every actual generation attempt, including retries and repair, participates in -the existing engine-local pool for its backend. Release active-generation -permits before backoff; waiting to retry must not monopolize active capacity. -Keep the logical execution's bounded admission reservation until it finishes. -Reacquiring a permit uses ordinary capacity scheduling, not a new priority lane. - -Proposal and assessment calls in the representative example share one engine -and a finite backend limit. Existing unlimited custom backends and endpoint-only -profiles remain supported; do not impose a new global cap or silently change -catalog defaults. If a consumer requires a different limit, current custom -backend IDs and application-owned profiles remain the starting point. - -Concurrency bounds simultaneous activity, not cumulative cost or request rate. -Independent engines, processes, and other clients do not share these limits. - -### Actionable Public Failures - -Extend existing public error contracts rather than replace them with an unrelated -taxonomy. Consumers must distinguish cancellation, deadlines, transport failures, -non-success provider status, malformed provider responses, retry exhaustion, -capacity rejection, and completed output-validation failure without parsing text. - -Exhaustion retains the final underlying cause and actual attempt count. Existing -context identities, provider status/detail access, and `errors.Is`/`errors.As` -behavior remain useful through any added wrapping. An error can be both an -exhaustion and a particular provider failure; categories need not be exclusive. - -Completed invalid output remains a validation result, not an operational error. -Operational failures continue to return no partial `RunResult`; observations -provide accounting independently. Preserve redacted default diagnostics and -deliberate access to untrusted provider detail under application disclosure policy. - -### Optional Execution Observations - -Provide an optional request-scoped observer through the public facade. It receives -metadata about an execution without requiring a logging or metrics framework. -Promptkit does not retain a global execution history or persist reports. - -Deliver callbacks synchronously and in order within an execution. Concurrent -executions may invoke a shared callback concurrently. Do not invoke consumer -code under capacity-manager locks or add a background delivery queue. Callbacks -cannot return execution-policy errors; consumers keep them fast and handle their -own reporting failures. Panics follow ordinary injected-code behavior rather -than being silently swallowed. This keeps delivery straightforward without -buffering, dropping, or shutdown machinery; slow callbacks can still delay a run. - -The observable lifecycle covers successful execution, completed invalid output, -and operational failure, including failures before any model call. Expose only -identities available at that point; do not invent selected profiles or backends -when resolution failed. Prepared execution observations distinguish execution -from earlier preparation and consumer-held delay. - -Consumers can correlate the logical execution with generation attempts and learn: - -- the available prompt, profile, backend, and effective model identities; -- whether generation is initial or structural repair, and which transient - attempt occurred within it; -- capacity-wait, generation, and retry-delay measurements with clear boundaries; -- classified outcomes and actual retry and repair counts; and -- provider-reported usage already received, even when a subsequent step fails. - -Attempt measurements and terminal totals have explicit accounting semantics so -consumers do not add aggregates to their constituent attempts. Missing usage is -distinguished from measured zero; partial provider reporting must not become -falsely complete accounting. Do not estimate unknown usage or claim a complete -billing ledger. Preserve existing successful-result usage semantics and use a -presence-aware observation representation rather than silently redefining zero. - -Observation payloads contain metadata, not credentials, request/response bodies, -raw provider messages, or arbitrary collaborator error strings. Metadata itself -may be sensitive and remains subject to application handling. Content capture -is not part of this scope. Public contracts must state event ownership, delivery -ordering, concurrency, and callback behavior without exposing internal types. - -### Representative Consumer Evidence - -Add a compact maintained offline example of a structured proposal followed by -an independent LLM assessment. Use embedded prompts/schemas, application-owned -profiles, synthetic content, a shared finite backend, and collected observations. -Use a controlled HTTP boundary when demonstrating built-in transient retry -behavior; injecting an already-retrying model client would not prove that path. - -Show that schema validity does not imply domain acceptance. Keep transcript -processing, semantic acceptance, reports, and retained artifacts application-owned. -Do not turn the example into a miniature Audita pipeline or a failure-case menu. - -Focused tests, rather than one oversized example, establish recovery, exhaustion, -disablement, retry timing, cancellation during capacity/backoff/transport, repair -interaction, peak concurrency, failure accounting, usage presence, and redaction. -Use deterministic clocks/waits or controlled transports where needed; no paid -calls, real credentials, timing-sensitive sleeps, or duplicated test matrices. - -## Architecture And API Boundaries - -Keep execution machinery under `internal/`; the root package remains a facade -for intentional consumer contracts. Maintain one retry-policy owner and one -capacity manager. Keep provider wire decoding in its adapter, lifecycle -coordination in the execution layer, and application reporting downstream. - -Do not add a second run API solely for diagnostics. Preserve existing operation -signatures and optional-observation behavior for callers that do not use it. -Avoid public clock/sleeper abstractions introduced only for internal testing. -No profile/backend catalog schema changes are assumed by this roadmap. - -## Documentation And Compatibility - -Default retries change request counts, latency, and potential cost. Target a -minor release with explicit migration guidance, including how to retain -single-attempt behavior and avoid duplicate retries. Canonical GoDoc owns exact -API and error semantics; integration docs own HTTP retry behavior; consumer docs -own adoption examples. Update internal documentation and maintained-example -validation only where responsibilities actually change. Release notes summarize -and link rather than become a second contract owner. - -## Non-Goals - -No application pipeline, domain validator, cross-process scheduler, new task -queue, monetary budget, rate limiter, circuit breaker, automatic backend/model -failover, streaming/tool support, telemetry dependency, persistent run store, -or additional content-retention facility. Existing fallback-profile behavior -is not repurposed as runtime provider failover. Licensing is already complete -in source and is not a further implementation dependency. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index e7f41b6..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,110 +0,0 @@ -# Reliable Execution Review Findings - -Review baseline: `cef8524`, following the completed -[implementation plan](implementation.md) and the -[reliable execution roadmap](reliable-execution.md). - -The intended architecture is substantially implemented: the public facade -adapts contracts, the runner owns execution and repair, one generation -coordinator owns retries and active permits, the HTTP adapter freezes requests -and decodes responses, and an invocation-local recorder produces observations. -The two correctness gaps below have been remediated. This document preserves -the review evidence and resolution, not a new public contract. - -## 1. Isolate Observation State Across Public Invocations (P2) - -**Status:** Resolved. - -**Location:** [Engine.Run](../../engine.go); -[Runner.Run](../../internal/usecase/runner.go); -[recorder context propagation](../../internal/observation/recorder.go). - -The observer is request-scoped, but its recorder is attached to the context -passed to injected collaborators. If an injected client delegates to another -engine using that context and a request without an observer, `Engine.Run` -passes the inherited context through unchanged. The nested runner discovers -the outer recorder and treats it as its own. - -A focused public-API reproduction with two engines and one nested injected -client produced identical inner and outer result RunIDs, four outer events -instead of three, and two outer attempts instead of one. The inner call had no -observer. Nested selection and usage also flow into the shared recorder; -parallel delegated calls could concurrently mutate state intended to be local -to one sequential invocation. This is not a request to add nested tracing: -injected-client work is intentionally opaque to its caller's accounting. - -**Remediation direction:** Establish an observation-context boundary for each -public operation, retaining caller cancellation, deadlines, and unrelated -context values while preventing inherited Promptkit recorder/claim state from -being reused. Install only that invocation's own observation state when -appropriate. Check `RunPrepared` without a retained observer as well as -preparation operations, which must not mutate an enclosing run's selected -identities. Do not prohibit delegation or add a global recorder/locking layer. - -**Regression evidence required:** Exercise delegation through the public facade -with the received context. Separate calls must have distinct result identities; -an unobserved child must not emit to or inflate its parent's observations; a -child with its own observer must have its own lifecycle. Include focused -prepared/preparation coverage where the same context plumbing is used, and -race-enabled coverage for concurrent delegation without sharing recorder state. - -**Resolution:** Preparation and execution facade entry points mask inherited -recorder/claim state before installing any invocation-owned state. The context -boundary preserves caller values, deadlines, cancellation, and cancellation -causes. Public regression tests cover observed and unobserved delegated ordinary -and prepared runs, preparation isolation, independent accounting/identities, -and concurrent delegation. A focused recorder test checks context preservation -and claim-sink isolation. - -## 2. Preserve Injected Failure Categories In Attempt Events (P2) - -**Status:** Resolved. - -**Location:** [public client adapter](../../llm_adapter.go); -[attempt classification](../../internal/generation/coordinator.go); -[terminal classification](../../observations.go). - -The public adapter returns public error sentinels, including -`ErrMalformedResponse` for a successful nil response. The coordinator recognizes -only internal LLM sentinels. As a result, attempt observations report the broad -`generation` category while the terminal observation reports the specific -`malformed_response` or `transport` category for the very same failure. - -Focused public-API reproductions confirmed all three cases: injected `(nil, -nil)`, an explicit public `ErrMalformedResponse`, and an explicit public -`ErrTransport`. Each emitted `generation` on the attempt and the expected -specific category on the terminal event. Existing repair-failure tests check -the terminal category but do not assert the failing attempt's category. - -**Remediation direction:** Normalize deliberately supplied public failure -identities at the facade adapter boundary into metadata or internal identities -the coordinator understands, preserving the original error for `errors.Is` and -`errors.As`. Classify nil success consistently through the same boundary. -Internal packages must not import the public package. Do not infer categories -from arbitrary error text or enable retries for injected clients. - -**Regression evidence required:** A compact public-boundary table covering nil -success, wrapped public malformed/transport sentinels, and an arbitrary injected -error should assert both attempt and terminal classifications, preserved error -identity, and single dispatch. Retain context-first classification for wrapped -cancellation/deadline causes. - -**Resolution:** The facade adapter adds internal classification identities for -explicit public transport/malformed errors, retaining the original diagnostic -and error chain. Nil success uses the same normalization path. Public tests -check attempt/terminal agreement, `errors.Is`/`errors.As`, single dispatch even -with a positive retry budget, arbitrary errors, and context-first classification. - -## Validation And Review Outcome - -- `./scripts/release-check.sh` passed on the review baseline and remediated tree, including - race-enabled tests, vet, the three offline examples, formatting, documentation - links, and repository hygiene checks. -- Temporary public-API probes reproduced both findings during review. They were - removed and replaced by focused permanent regression coverage during remediation. -- The existing tests provide useful coverage of retries, repair interaction, - capacity, prepared claims, usage presence, and redaction, but did not protect - these two observation-boundary cases before remediation. The new - [boundary tests](../../observation_boundary_contract_test.go) protect them. -- No other must-fix issue was identified in this review. This does not establish - live-provider behavior; all verification was offline.