Reconcile reliable execution documentation

doc.go

@@ -55,7 +55,8 @@
 // [Backend], [RunRequest], [ArtifactRef], [ExecutionTargetOverride], [Profile],
 // [OpenAICompatibleProfileConfig], [ProfileInspection],
 // [PromptInputDefinition], [PromptInspection], [PreparedExecution],
-// [CapacityError], and [GenerationError], do not have stable JSON
+// [CapacityError], [GenerationError], [RetryExhaustedError], and
+// [ExecutionEvent], do not have stable JSON
 // representations. Direct API keys are nevertheless excluded from JSON for
 // every public value.
 

docs/consumers/pkg-promptkit.md

@@ -201,6 +201,10 @@ repair-call, duration, and presence-aware usage totals. Its usage is an
 aggregate, not an additional billable attempt. A known zero differs from an
 unknown count, and incomplete totals must not be treated as complete cost
 accounting. A failed operation still returns no partial `RunResult`.
+An injected client that leaves `UsagePresence` nil can report known nonzero
+counts, but its zero counts remain unknown to observations; supply explicit
+presence when measured zeros matter. Existing `RunResult.Usage` sums returned
+values independently of observation presence.
 
 Callbacks run synchronously, in order for one operation, and can overlap for
 different operations. Synchronize shared callback state and avoid blocking on
@@ -208,6 +212,9 @@ work that needs the same run's admission slot. Callback panics propagate;
 there is no durable event history or promise of a terminal event after panic.
 Events omit content and error objects, but selected identifiers may still be
 sensitive. See the [observation GoDoc](../../observations.go) for exact fields.
+The [offline proposal and assessment example](../../examples/go-library/observe/main.go)
+shows application-owned event collection and a separate semantic acceptance
+decision using the built-in HTTP path without network access.
 
 ### Repair A Structured Result
 
@@ -240,6 +247,9 @@ previous single-attempt behavior, or set `RunRequest.TransientRetries` to
 override one execution. A custom HTTP transport still uses this policy;
 injected `LLMClient` implementations retain their own retry policy. Application
 semantic retries are separate runs and can multiply provider traffic further.
+The caller context covers capacity and retry waiting as well as attempts;
+without a deadline a valid provider delay can be long. Promptkit does not
+infer exactly-once or complete cost accounting from a retry outcome.
 See the [provider integration contract](../integrations/openai-compatible-chat.md#transient-status-retries)
 for eligible statuses, waits, timeouts, and provider timing.
 
@@ -612,6 +622,12 @@ decoded or an injected client returns a nil successful response. Both retain
 `errors.Is` and `errors.As`. Caller cancellation and deadlines retain their
 standard context identities. Arbitrary injected-client errors retain their own
 identities without Promptkit guessing a failure category from their text.
+An eligible built-in status that consumes a positive retry budget additionally
+matches `ErrRetriesExhausted`; use `errors.As` to read
+`RetryExhaustedError.Attempts()` and the final `GenerationError` status. With
+explicit zero retries, the first eligible status is an ordinary generation
+error, not exhaustion. Completed structural rejection remains a normal result
+with failed validation, while an operational error returns no partial result.
 
 ## Application Boundary
 

docs/integrations/openai-compatible-chat.md

@@ -113,6 +113,14 @@ data, a second JSON value, absent choices, missing content, `null` content,
 non-string content, and size overflow are malformed responses and return no
 partial result.
 
+Each of the five token fields retains whether the provider reported a
+non-null integer, including zero. Omitted and `null` fields are unknown for
+observations. A strictly decoded successful envelope can yield observed usage
+even when a later missing choice or content makes the attempt malformed.
+Invalid framing and non-2xx error bodies supply no observed usage; provider
+work may still have occurred. Reported counts are not inferred, reconciled,
+or proof of complete cost accounting.
+
 For a non-2xx status, Promptkit recognizes one JSON document with a top-level
 object-valued `error` member. Its optional `message` and `type` fields must be
 strings, and `code` may be a string or JSON number. Valid supported fields are
@@ -150,12 +158,13 @@ Promptkit regardless of this setting. A custom `http.Client` on the built-in
 path remains subject to these retries.
 
 Each retry uses equal jitter between half and all of an exponential local
-delay, starting at 500 milliseconds and capped at 8 seconds. The first valid
-`Retry-After` header value, as unsigned decimal seconds or an HTTP date, sets
-a minimum not-before time measured from response receipt. Invalid values use
-local delay; past dates impose no extra delay. The provider minimum is not
-capped at 8 seconds. Retry waiting is cancellable and holds no active backend
-permit. A caller without a deadline can wait a long time for provider timing.
+delay, starting at 500 milliseconds and capped at 8 seconds. The first
+`Retry-After` header value sets a minimum not-before time measured from
+response receipt if it is valid unsigned decimal seconds or an HTTP date.
+Invalid values use local delay; past dates impose no extra delay. The provider
+minimum is not capped at 8 seconds. Retry waiting is cancellable and holds no
+active backend permit. A caller without a deadline can wait a long time for
+provider timing.
 
 The request body, selected URL, and resolved authorization are fixed across
 attempts of one generation. Each attempt creates a fresh HTTP request and

docs/internal/capacity.md

@@ -77,12 +77,17 @@ list. Releasing a permit transfers it directly to the oldest remaining waiter
 before making it generally available. Pools do not order work relative to
 other backend IDs.
 
-The coordinator passes injected generation requests, responses, and errors
-through unchanged. The built-in HTTP client remains responsible for provider
-transport behavior. The runner, rather than the
+The coordinator does not retry injected clients; it normalizes a nil successful
+response as malformed generation. The built-in HTTP client remains responsible
+for provider transport behavior. The runner, rather than the
 capacity layer, sums all five usage fields from the initial response and every
 completed repair response into the successful run result.
 
+An invocation-local observer receives attempt events after active permit
+release, including canceled acquisition without dispatch. A retry wait holds
+the logical admission lease but no active permit. Observer callbacks must not
+block on work that requires the same run's admission slot.
+
 ## Cancellation And Release
 
 Admission checks the caller context before reserving a slot. A call canceled
@@ -116,6 +121,8 @@ release. The
 [external package capacity tests](../../capacity_contract_test.go) own the
 assembled public-engine behavior for configured limits, capacity errors,
 endpoint identity, engine independence, and injected clients. The
+[reliable execution contract tests](../../reliable_execution_contract_test.go)
+cover shared and independent backend pools during retry and repair. The
 [prepared-execution contract tests](../../prepared_execution_contract_test.go)
 own the public prepared-capacity boundary. The
 [root error-boundary tests](../../errors_internal_test.go) own preservation of

docs/internal/generation.md

@@ -29,8 +29,16 @@ including interrupted waits. Active permits are released before callbacks;
 generation duration ends before callback delivery. Built-in successful-envelope
 usage retains reported-field presence, while injected clients use explicit
 presence when supplied and otherwise treat zero fields as unknown.
+Malformed successful envelopes can retain strictly decoded usage; error
+response bodies are never mined for it. Arbitrary injected failures carry
+unknown usage. Transport, timeout, malformed, and noneligible status failures
+stop the sequence without retry exhaustion. Context is checked before each
+wait, acquisition, and dispatch.
 
 Focused coordinator tests use controlled transport and wait functions. They
 cover eligibility, exhaustion, cancellation, backoff, request freezing, permit
 release, and the injected-client path. The capacity manager tests own permit
-acquisition and idempotent release.
+acquisition and idempotent release. Public
+[integration tests](../../reliable_execution_contract_test.go) cover retry,
+repair, observation, frozen prepared execution, and shared-backend behavior
+through the assembled facade.

docs/internal/llm.md

@@ -51,8 +51,9 @@ response on failure.
 when validating extra parameters. Backend registration consumes the same rule
 without making the model client depend on registry configuration.
 
-The implementation has no retry loop, tool-call support, provider catalog,
-inbound HTTP behavior, or durable session store.
+This client does not schedule retries; the [generation coordinator](generation.md)
+does so around its prepared attempts. The client has no tool-call support,
+provider catalog, inbound HTTP behavior, or durable session store.
 
 ## Prepared Generation
 
@@ -101,6 +102,12 @@ whitespace-only content. Missing choices, missing or `null` content, and
 non-string content are malformed responses. Output validation and correction
 eligibility remain outside this package.
 
+Successful-envelope usage is decoded with independent presence bits for each
+token field: omission and `null` are unknown, while a reported integer zero is
+known. Strictly decoded usage can still be observed if later choice/content
+checks fail; invalid framing or a non-success error body contributes none.
+Existing successful-result accounting remains value-based.
+
 For a non-success response, `ProviderHTTPError` retains the HTTP status and
 only normalized detail from the bounded recognized envelope. It retains
 `ErrUnexpectedStatus` through unwrapping. The client owns response closure;

docs/internal/observation.md

@@ -4,9 +4,9 @@
 aggregation, measured phase totals, and synchronous metadata event delivery.
 Its event values and callback type live in `internal/domain` so internal
 requests can carry them without depending on the recorder package. The
-recorder currently supplies run IDs to the runner and receives attempt and
-retry-delay measurements from the generation coordinator. Run lifecycle
-instrumentation for ordinary `Run` runs at the facade and runner boundaries.
+recorder supplies run IDs to the runner and receives attempt and
+retry-delay measurements from the generation coordinator. Ordinary `Run`
+lifecycle instrumentation spans the facade and runner boundaries.
 Prepared execution starts only after its private payload is atomically claimed;
 the facade receives that invocation's recorder privately and emits the terminal
 event after public error mapping. Failed claims and preparation emit nothing.
@@ -17,6 +17,12 @@ callback. The caller supplies normalized failure metadata; the recorder never
 parses an error string or receives provider bodies, credentials, headers, or
 other content. Events are values with no mutable containers. Callback panics
 propagate to the caller.
+Per-operation callbacks are ordered and synchronous; different operations may
+invoke the same callback concurrently. The recorder marks each measured phase's
+end before delivery, so callback work does not inflate provider duration.
+Terminal phase durations sum measured phases, while wall duration may also
+include preparation, validation, and callbacks. Panics do not promise a
+terminal event.
 
 Attempt usage is known when its explicit presence bit is true. For legacy
 injected clients without presence metadata, nonzero counts are known and

docs/internal/overview.md

@@ -20,7 +20,7 @@ contributor workflow and validation.
 | `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
 | `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, and output contracts. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [endpoint invariant](../../internal/domain/endpoint.go) |
 | `internal/engine` | Assembles external catalogs, Promptkit-owned profiles, repositories, registry, capacity, collaborators, repairer, and the one use-case runner for each engine instance. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#profiles-and-built-ins), [internal runner](runner.md) |
-| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries and cancellable backoff. | [Internal generation coordination](generation.md) |
+| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, and attempt observations. | [Internal generation coordination](generation.md) |
 | `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
 | `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
 | `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
@@ -30,8 +30,8 @@ contributor workflow and validation.
 | `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
 | `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
 | `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
-| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, and synchronous metadata event delivery for ordinary runs and generation attempts. | [Internal execution observation](observation.md) |
-| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
+| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, and synchronous metadata event delivery for ordinary and claimed prepared runs and generation attempts. | [Internal execution observation](observation.md) |
+| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary and single-claim prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
 
 The internal engine assembles these internal components without exposing their
 representations. Consumers depend only on the root facade.

docs/internal/runner.md

@@ -189,7 +189,8 @@ credential during execution, reaches generation and every repair attempt.
 Result usage is the field-wise sum of all five usage values from the initial
 response and every completed repair response. Final raw output, artifact, and
 validation state still come from the last candidate. A repair error returns no
-partial run result or partial usage.
+partial run result or partial result usage, although the invocation-local
+observer retains completed-attempt usage and reports it in the terminal event.
 
 ## Failure Categories
 

docs/roadmap/audita-feature-requests.md

@@ -2,16 +2,16 @@
 
 **Last updated:** 2026-09-23
 
-**Status:** Proposed consumer requests, based on a review of Promptkit v0.10.0.
-These are not approved features, implementation plans, delivery commitments, or
-claims about current behavior. Priorities reflect Audita's prospective adoption;
-Promptkit should evaluate them against its broader consumer needs.
+**Status:** Requests 1–6 are implemented in source but not yet released.
+These requests originated from a review of Promptkit v0.10.0. They retain
+Audita's consumer context and priorities, not a current API specification or
+a delivery commitment for Audita adoption.
 
-Requests 1–5 are now incorporated into the active
-[reliable-execution roadmap](reliable-execution.md), which owns the selected
-Promptkit-native scope and remaining decisions. The requests below retain the
-consumer context, not a separate implementation specification. Request 6 is
-implemented in source as noted below.
+Requests 1–5 were resolved through the
+[reliable-execution roadmap](reliable-execution.md) and its
+[implementation plan](implementation.md). The [consumer guide](../consumers/pkg-promptkit.md)
+and public GoDoc own current behavior. No release or Audita adoption is
+claimed here.
 
 ## Consumer Context
 
@@ -34,6 +34,9 @@ consumer outcomes only; API shape and implementation are left to Promptkit.
 
 ## 1. Transient-Failure Retries
 
+**Implemented in source; not yet released.** The built-in HTTP path owns the
+selected transient-status policy; injected clients keep their own policy.
+
 **Priority: High.** Audita should not need its own general-purpose provider retry
 policy to recover from temporary rate limiting or service failures.
 
@@ -66,6 +69,9 @@ with content repair. Permanent request failures must remain distinguishable.
 
 ## 2. Execution Observations, Including Failed Runs
 
+**Implemented in source; not yet released.** Optional metadata callbacks cover
+ordinary and claimed prepared execution, including normal error returns.
+
 **Priority: High.** Audita needs to explain the time and calls consumed by a logical
 operation, including when that operation never produces a successful result.
 
@@ -92,6 +98,10 @@ and failure accounting sufficiently complete for downstream diagnostics.
 
 ## 3. Actionable Failure Classification
 
+**Implemented in source; not yet released.** Public sentinels and typed errors
+preserve provider status, retry exhaustion, transport, malformed-response,
+capacity, validation, and context distinctions.
+
 **Priority: High; closely related to retries and observations.** Audita needs to
 choose between failing a run, rejecting affected work, and recording a warning
 without interpreting human-readable error messages.
@@ -110,6 +120,10 @@ Promptkit need not decide Audita's recovery or correction-acceptance policy.
 
 ## 4. Simple Per-Backend Capacity Limits
 
+**Implemented in source; not yet released.** One engine-local backend pool
+coordinates initial, retry, and repair attempts across profiles selecting it.
+Audita must still configure finite limits for the paid backends it adopts.
+
 **Priority: Adoption requirement, largely supported by existing facilities.**
 Audita needs only two protections:
 
@@ -141,6 +155,12 @@ across a long run. A monetary budget or rate-limiting system is not requested.
 
 ## 5. Representative Consumer Example
 
+**Implemented in source; not yet released.** The
+[offline example](../../examples/go-library/observe/main.go) demonstrates the
+built-in retry, structured proposal and assessment, metadata collection, and
+separate application acceptance decision. Focused tests, not example switches,
+cover cancellation and invalid-output branches.
+
 **Priority: High as adoption evidence.** Provide a maintained offline example of a
 structured proposal call followed by an independent LLM validation call, using
 embedded prompts/schemas and separate application-owned execution profiles.
@@ -164,11 +184,10 @@ release containing this change and satisfy the applicable distribution obligatio
 
 ## Suggested Evaluation Order and Scope
 
-Evaluate retries, failure classification, and execution observations together, then
-use the representative consumer example to assess readiness and verify shared
-per-backend limits. Existing capacity controls should suffice unless that exercise
-reveals a concrete gap. Resolve licensing before selecting a release for Audita's
-migration.
+The implemented retry, classification, observation, and per-backend capacity
+contracts can now be evaluated together with the representative example.
+Audita still owns its migration decision and must select a later published
+release before adopting this source state.
 
 Existing facilities for embedded sources, application-owned profiles, prepared
 execution, JSON-schema validation, and request-scoped credentials are useful

docs/roadmap/implementation.md

@@ -472,6 +472,8 @@ example and its relevant tests; commit.
 
 ## Stage 12 — Complete Documentation And Final Validation
 
+**Completed:** 2026-09-23.
+
 Reconcile GoDoc, consumer error/observer/retry guidance, provider integration
 docs, and internal runner/LLM/capacity/generation/observation documentation with
 the implemented design. Remove claims that no retries exist or that usage can

docs/roadmap/reliable-execution.md

@@ -1,9 +1,10 @@
 # Reliable And Observable Execution
 
-**Status:** Accepted feature scope; not implemented. This document owns the
-target end state and approved policy choices. The companion
-[implementation plan](implementation.md) owns implementation steps and concrete
-API and algorithm decisions.
+**Status:** Implemented in source; not yet released. This document records the
+accepted target and approved policy choices. The companion
+[implementation plan](implementation.md) records implementation steps and
+concrete API and algorithm decisions. Current contracts are owned by public
+GoDoc and the linked consumer and integration documentation.
 
 ## Purpose
 
@@ -19,20 +20,21 @@ observations, shared capacity, and consumer evidence belong together because
 they describe the same execution lifecycle. Licensing was handled separately.
 Audita motivates the work but does not define an application-specific API.
 
-## Existing Foundations
+## Foundations At Planning Time
 
-Promptkit already provides structural output validation and bounded repair,
-prepared execution, engine-local backend capacity, run metadata, aggregate token
-usage, and typed provider-status and capacity errors. Operational errors do not
-return partial run results. The built-in client has no transient retry loop,
-and existing integer usage fields do not distinguish missing reports from zero.
+At the time this scope was accepted, Promptkit provided structural output
+validation and bounded repair, prepared execution, engine-local backend
+capacity, run metadata, aggregate token usage, and typed provider-status and
+capacity errors. Operational errors did not return partial run results. The
+built-in path had no transient retry loop, and existing integer usage fields
+did not distinguish missing reports from zero.
 
-The current contracts remain owned by public GoDoc, the
+The implemented contracts are owned by public GoDoc, the
 [consumer guide](../consumers/pkg-promptkit.md), and the
 [provider integration contract](../integrations/openai-compatible-chat.md).
 The [capacity document](../internal/capacity.md) and
 [runner document](../internal/runner.md) describe the existing implementation.
-This roadmap does not redefine those documents as already implementing new work.
+This historical baseline does not override their current-state descriptions.
 
 ## Target End State
 

engine.go

@@ -510,6 +510,11 @@ func (e *Engine) PrepareExecution(ctx context.Context, req RunRequest) (*Prepare
 // model attempts. Each dispatched attempt receives its own generation and
 // HTTP-client timeout; the caller context governs the full operation.
 // Injected LLMClient implementations are not retried automatically.
+// With Observer, Run emits started before facade request conversion or a nil
+// engine check, and finished on every normal return after public error mapping
+// and resource release. The event RunID matches a successful result. Without
+// Observer, result timing retains its existing convention; observed wall
+// duration can be slightly longer because it spans the facade boundary.
 //
 // Run can return every error category documented by [Engine.Prepare], plus
 // ErrCapacityExceeded and ErrLLMGenerate. Built-in HTTP execution failures

observations.go

@@ -10,7 +10,10 @@ import (
 
 // ExecutionObserver receives immutable metadata events synchronously. Callers
 // must synchronize shared state if separate runs use the same observer. A
-// callback must return for its run to progress; panics propagate to the caller.
+// callback must return for its run to progress; cancellation cannot interrupt
+// consumer code already in the callback. Do not block on work requiring the
+// same run's admission slot. Panics propagate without a promise of a terminal
+// event, and Promptkit keeps no asynchronous queue or durable history.
 type ExecutionObserver func(ExecutionEvent)
 
 // ExecutionEventKind identifies one point in an execution's lifecycle.
@@ -57,7 +60,11 @@ const (
 )
 
 // ObservedTokenCount distinguishes an omitted count from a reported zero.
-// Complete means every dispatched attempt reported the field.
+// On an attempt, Known and Complete are true together only for a reported
+// non-null integer. An aggregate Value sums known contributions; Known means
+// at least one attempt reported the field, and Complete means every dispatched
+// attempt did. With no dispatched attempts, aggregate zero is known and
+// complete. Unknown fields have Value zero.
 type ObservedTokenCount struct {
 	Value    int
 	Known    bool
@@ -78,6 +85,20 @@ type ObservedTokenUsage struct {
 // credentials, URLs, headers, or error objects. It has no stable JSON contract.
 // Identifiers themselves may be sensitive. Nonterminal aggregate counters are
 // zero; a terminal usage total must not be added to attempt usage.
+//
+// Generation is one for the initial generation and increases for repairs;
+// Attempt is one-based within each generation. Dispatched means Promptkit
+// invoked an HTTP attempt or injected Generate, not that a network packet was
+// sent or billable work occurred. A canceled capacity acquisition emits a
+// non-dispatched attempt with no usage. A retry-delay event describes the
+// completed or interrupted wait after the indexed failed attempt.
+//
+// GenerationDuration excludes capacity waiting, retry delay, and callback
+// work. Terminal phase durations sum measured phases; terminal Duration is
+// wall time and may also include preparation, validation, and callbacks.
+// Attempt success means generation succeeded, even if later validation rejects
+// its output. Failure is classified metadata; RetriesExhausted is a separate
+// flag and does not replace the underlying failure kind.
 type ExecutionEvent struct {
 	Kind               ExecutionEventKind
 	RunID              string