Prepare the v0.11.0 release
This commit is contained in:
@@ -33,6 +33,10 @@ boundary and constraints that framework work must preserve.
|
|||||||
|
|
||||||
## Release Guidance
|
## Release Guidance
|
||||||
|
|
||||||
|
Consumers upgrading to `v0.11.0` should read the
|
||||||
|
[v0.11.0 changelog and migration guide](docs/releases/v0.11.0.md), particularly
|
||||||
|
the default retry behavior and license change.
|
||||||
|
|
||||||
Consumers upgrading to the `v0.10.0` profile-policy transition
|
Consumers upgrading to the `v0.10.0` profile-policy transition
|
||||||
should read the [v0.10.0 transition guide](docs/releases/v0.10.0.md).
|
should read the [v0.10.0 transition guide](docs/releases/v0.10.0.md).
|
||||||
|
|
||||||
|
|||||||
76
docs/releases/v0.11.0.md
Normal file
76
docs/releases/v0.11.0.md
Normal file
@@ -0,0 +1,76 @@
|
|||||||
|
# Promptkit v0.11.0
|
||||||
|
|
||||||
|
This release adds reliable, observable model execution and changes Promptkit's
|
||||||
|
license to LGPL-3.0-or-later.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
- Built-in model calls can recover from selected transient HTTP failures.
|
||||||
|
- Optional execution observations report attempts, timing, outcomes, and known
|
||||||
|
token usage, including work completed before a run fails.
|
||||||
|
- Public errors distinguish transport failures, malformed responses, and retry
|
||||||
|
exhaustion without parsing diagnostic text.
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
The built-in client now defaults to two additional transient attempts. This can
|
||||||
|
increase latency, request counts, and cost; retries do not guarantee exactly-once
|
||||||
|
generation. Injected `LLMClient` implementations retain their own retry policy
|
||||||
|
and are not automatically replayed. Existing generation timeouts apply per
|
||||||
|
attempt, while the caller context governs the complete operation. See the
|
||||||
|
[provider integration contract](../integrations/openai-compatible-chat.md).
|
||||||
|
|
||||||
|
Public retry and observation controls are additive. Existing successful-result
|
||||||
|
usage semantics remain unchanged, and operational failures still return no
|
||||||
|
partial `RunResult`. Observations distinguish unknown usage from reported zero;
|
||||||
|
they are not a complete billing ledger. See the
|
||||||
|
[consumer guide](../consumers/pkg-promptkit.md) and
|
||||||
|
[observation GoDoc](../../observations.go).
|
||||||
|
|
||||||
|
Promptkit is now licensed under LGPL-3.0-or-later. Review the
|
||||||
|
[license summary](../../README.md#license), [LGPL text](../../LICENSE), and
|
||||||
|
[accompanying GPL text](../../COPYING) before adoption or distribution.
|
||||||
|
Previously published releases and dependency licenses are unchanged.
|
||||||
|
|
||||||
|
## Upgrade
|
||||||
|
|
||||||
|
Update the dependency:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go get gitea.maximumdirect.net/eric/promptkit@v0.11.0
|
||||||
|
```
|
||||||
|
|
||||||
|
To retain single-attempt behavior for built-in generation, set
|
||||||
|
`Config.TransientRetries` to a pointer to zero. `RunRequest.TransientRetries`
|
||||||
|
can override the engine choice per request. Review any existing application or
|
||||||
|
HTTP-transport retry loops to avoid unintentionally multiplying attempts.
|
||||||
|
Transient retries and structural repairs have separate budgets; provide a
|
||||||
|
caller deadline when a total execution time limit is needed.
|
||||||
|
|
||||||
|
Observation is opt-in through `RunRequest.Observer`, including for prepared
|
||||||
|
execution. Callbacks are synchronous; shared observers must support concurrent
|
||||||
|
runs. Injected clients can supply `GenerateResponse.UsagePresence` to explicitly
|
||||||
|
report known zero counts. No observer or usage-presence changes are required
|
||||||
|
for existing consumers. See the [consumer guide](../consumers/pkg-promptkit.md)
|
||||||
|
for adoption and the [offline example](../../examples/go-library/observe/main.go)
|
||||||
|
for a structured proposal-and-assessment workflow.
|
||||||
|
|
||||||
|
## Changes
|
||||||
|
|
||||||
|
- Centralized built-in retry coordination with cancellable backoff and provider
|
||||||
|
retry timing. Retries and structural repair share existing backend capacity;
|
||||||
|
backoff releases active permits without releasing logical run admission.
|
||||||
|
- Added `ErrTransport`, `ErrMalformedResponse`, `ErrRetriesExhausted`, and
|
||||||
|
`RetryExhaustedError`, retaining applicable underlying error identities.
|
||||||
|
- Added request-scoped lifecycle observations and presence-aware token accounting
|
||||||
|
for ordinary and prepared execution, without recording prompt/response bodies.
|
||||||
|
- Isolated delegated operations' observation state and made injected-client
|
||||||
|
attempt classifications consistent with terminal classifications.
|
||||||
|
- Added an offline observation example and regression coverage for retry,
|
||||||
|
repair, capacity, error, and observation interactions.
|
||||||
|
- Adopted LGPL-3.0-or-later and streamlined the documented local release process.
|
||||||
|
|
||||||
|
Backend catalog selections are unchanged from v0.10.0: OpenRouter
|
||||||
|
`v2.20260922.1` and Rakestrawhome `v2.20260922.0`. See the
|
||||||
|
[catalog compatibility policy](../policy/release.md#catalog-profile-compatibility)
|
||||||
|
and [profile inventories](../formats.md#external-profile-inventories).
|
||||||
@@ -1,200 +0,0 @@
|
|||||||
# Audita Consumer Feature Requests
|
|
||||||
|
|
||||||
**Last updated:** 2026-09-23
|
|
||||||
|
|
||||||
**Status:** Requests 1–6 are implemented in source but not yet released.
|
|
||||||
These requests originated from a review of Promptkit v0.10.0. They retain
|
|
||||||
Audita's consumer context and priorities, not a current API specification or
|
|
||||||
a delivery commitment for Audita adoption.
|
|
||||||
|
|
||||||
Requests 1–5 were resolved through the
|
|
||||||
[reliable-execution roadmap](reliable-execution.md) and its
|
|
||||||
[implementation plan](implementation.md). The [consumer guide](../consumers/pkg-promptkit.md)
|
|
||||||
and public GoDoc own current behavior. No release or Audita adoption is
|
|
||||||
claimed here.
|
|
||||||
|
|
||||||
## Consumer Context
|
|
||||||
|
|
||||||
Audita polishes transcripts through ordered correction modules. It makes LLM calls
|
|
||||||
both to propose corrections and to assess proposed changes, alongside deterministic
|
|
||||||
domain validation. It needs bounded execution, understandable failures, and useful
|
|
||||||
run diagnostics without disclosing credentials or transcript content unnecessarily.
|
|
||||||
|
|
||||||
Audita would like Promptkit to own shared prompt and LLM execution capabilities,
|
|
||||||
while Audita retains transcript processing, module orchestration, domain acceptance,
|
|
||||||
correction application, reports, and diagnostic artifact retention. Suite-wide
|
|
||||||
consistency is the adoption goal; these requests do not ask Promptkit to reproduce
|
|
||||||
Audita's existing APIs or internal architecture.
|
|
||||||
|
|
||||||
The [consumer guide](../consumers/pkg-promptkit.md),
|
|
||||||
[format reference](../formats.md), and
|
|
||||||
[provider integration contract](../integrations/openai-compatible-chat.md)
|
|
||||||
remain the owners of implemented behavior. This document specifies desired
|
|
||||||
consumer outcomes only; API shape and implementation are left to Promptkit.
|
|
||||||
|
|
||||||
## 1. Transient-Failure Retries
|
|
||||||
|
|
||||||
**Implemented in source; not yet released.** The built-in HTTP path owns the
|
|
||||||
selected transient-status policy; injected clients keep their own policy.
|
|
||||||
|
|
||||||
**Priority: High.** Audita should not need its own general-purpose provider retry
|
|
||||||
policy to recover from temporary rate limiting or service failures.
|
|
||||||
|
|
||||||
Requested behavior:
|
|
||||||
|
|
||||||
- Retry HTTP 429 and selected transient server failures. Suggested initial defaults
|
|
||||||
cover 500, 502, 503, and 504; retrying every 5xx status is not a requirement.
|
|
||||||
- Supply reasonable documented defaults. Two additional attempts is a proposed
|
|
||||||
starting point, with a configurable maximum and an unambiguous explicit zero
|
|
||||||
that disables retries.
|
|
||||||
- Delay retries to avoid immediate repeated requests, honor applicable provider
|
|
||||||
retry timing, and bound the operation by the caller's cancellation/deadline.
|
|
||||||
Cancellation must interrupt waiting as well as active requests.
|
|
||||||
- Define which transport failures are eligible. Document that an uncertain response
|
|
||||||
can mean the provider already performed billable work; retries do not imply
|
|
||||||
exactly-once generation.
|
|
||||||
- Keep retry attempts tied to the same prepared generation request. An automatic
|
|
||||||
retry must not silently select different prompt content, model settings, or
|
|
||||||
credential identity.
|
|
||||||
- Distinguish transient-failure retries from content-repair calls. Document how
|
|
||||||
their budgets combine so consumers can reason about maximum request counts,
|
|
||||||
latency, and potential cost.
|
|
||||||
- State whether the policy applies to built-in and injected clients. Consumers
|
|
||||||
must be able to avoid duplicate retries when another collaborator already owns
|
|
||||||
retry behavior.
|
|
||||||
|
|
||||||
Expected consumer evidence includes a transient failure followed by success,
|
|
||||||
exhaustion, explicit retry disablement, cancellation during a delay, and interaction
|
|
||||||
with content repair. Permanent request failures must remain distinguishable.
|
|
||||||
|
|
||||||
## 2. Execution Observations, Including Failed Runs
|
|
||||||
|
|
||||||
**Implemented in source; not yet released.** Optional metadata callbacks cover
|
|
||||||
ordinary and claimed prepared execution, including normal error returns.
|
|
||||||
|
|
||||||
**Priority: High.** Audita needs to explain the time and calls consumed by a logical
|
|
||||||
operation, including when that operation never produces a successful result.
|
|
||||||
|
|
||||||
Requested observations include:
|
|
||||||
|
|
||||||
- Correlation between the logical operation and its individual attempts, selected
|
|
||||||
prompt, profile, and backend.
|
|
||||||
- Attempt outcomes and enough timing information to distinguish capacity waiting,
|
|
||||||
generation, and retry delays.
|
|
||||||
- Separate counts for transient-failure retries and content-repair calls.
|
|
||||||
- Available provider-reported token usage, including completed calls preceding a
|
|
||||||
later failure, with unknown usage distinguishable from measured zero.
|
|
||||||
- Clear accounting boundaries so consumers do not double-count aggregate and
|
|
||||||
individual-attempt measurements.
|
|
||||||
|
|
||||||
Observations should be usable by application-owned reports, logging, and metrics
|
|
||||||
without requiring a particular telemetry system or artifact layout. Default
|
|
||||||
observations should contain metadata rather than credentials or prompt/response
|
|
||||||
bodies. Any content disclosure should be an explicit consumer choice, with its
|
|
||||||
sensitivity clearly documented.
|
|
||||||
|
|
||||||
Current result metadata and usage remain useful; the request is to make execution
|
|
||||||
and failure accounting sufficiently complete for downstream diagnostics.
|
|
||||||
|
|
||||||
## 3. Actionable Failure Classification
|
|
||||||
|
|
||||||
**Implemented in source; not yet released.** Public sentinels and typed errors
|
|
||||||
preserve provider status, retry exhaustion, transport, malformed-response,
|
|
||||||
capacity, validation, and context distinctions.
|
|
||||||
|
|
||||||
**Priority: High; closely related to retries and observations.** Audita needs to
|
|
||||||
choose between failing a run, rejecting affected work, and recording a warning
|
|
||||||
without interpreting human-readable error messages.
|
|
||||||
|
|
||||||
Build on existing public error and validation-result contracts so consumers can
|
|
||||||
reliably distinguish cancellation, deadline expiration, transport failure,
|
|
||||||
non-success provider status, malformed provider response, exhausted retries,
|
|
||||||
capacity rejection, and completed output-validation failure.
|
|
||||||
|
|
||||||
On retry exhaustion, retain the underlying failure classification and attempt
|
|
||||||
count. Preserve safe default diagnostics while allowing deliberate access to
|
|
||||||
provider detail under the consumer's disclosure policy. Document which outcomes
|
|
||||||
carry generated output, validation findings, and partial execution observations.
|
|
||||||
|
|
||||||
Promptkit need not decide Audita's recovery or correction-acceptance policy.
|
|
||||||
|
|
||||||
## 4. Simple Per-Backend Capacity Limits
|
|
||||||
|
|
||||||
**Implemented in source; not yet released.** One engine-local backend pool
|
|
||||||
coordinates initial, retry, and repair attempts across profiles selecting it.
|
|
||||||
Audita must still configure finite limits for the paid backends it adopts.
|
|
||||||
|
|
||||||
**Priority: Adoption requirement, largely supported by existing facilities.**
|
|
||||||
Audita needs only two protections:
|
|
||||||
|
|
||||||
- For a local backend with a hard simultaneous-query limit, keep combined Audita
|
|
||||||
traffic within its assigned capacity. For example, a DGX Spark serving
|
|
||||||
gemma-4-31b-it may allow four simultaneous queries.
|
|
||||||
- For a paid API backend, use a reasonable finite concurrency cap so a large
|
|
||||||
transcript or pipeline cannot accidentally launch dozens or thousands of
|
|
||||||
simultaneous billable calls.
|
|
||||||
|
|
||||||
Proposal and validation calls targeting the same backend should share its limit.
|
|
||||||
The protection must cover initial calls, transient retries, and content-repair
|
|
||||||
calls. Audita does not need separate proposal/validation budgets, named workload
|
|
||||||
limits, or an additional aggregate ceiling across backends.
|
|
||||||
|
|
||||||
Promptkit's existing engine-local, per-backend capacity controls are the intended
|
|
||||||
foundation. Audita should select finite limits for the backends it uses; adoption
|
|
||||||
must not accidentally leave paid endpoints on an unlimited path. Exact defaults
|
|
||||||
remain to be selected. No new fine-grained scheduling feature is requested.
|
|
||||||
|
|
||||||
Verify the documented capacity scope in the consumer example, especially as retries
|
|
||||||
are added. An engine-local limit does not coordinate other processes or independent
|
|
||||||
clients using the same service; a shared local backend must have capacity allocated
|
|
||||||
accordingly or enforced at the service boundary. Cross-process coordination is not
|
|
||||||
an upstream feature request here.
|
|
||||||
|
|
||||||
Concurrency limits bound simultaneous activity, not total token usage or spending
|
|
||||||
across a long run. A monetary budget or rate-limiting system is not requested.
|
|
||||||
|
|
||||||
## 5. Representative Consumer Example
|
|
||||||
|
|
||||||
**Implemented in source; not yet released.** The
|
|
||||||
[offline example](../../examples/go-library/observe/main.go) demonstrates the
|
|
||||||
built-in retry, structured proposal and assessment, metadata collection, and
|
|
||||||
separate application acceptance decision. Focused tests, not example switches,
|
|
||||||
cover cancellation and invalid-output branches.
|
|
||||||
|
|
||||||
**Priority: High as adoption evidence.** Provide a maintained offline example of a
|
|
||||||
structured proposal call followed by an independent LLM validation call, using
|
|
||||||
embedded prompts/schemas and separate application-owned execution profiles.
|
|
||||||
|
|
||||||
Demonstrate transient failure handling, schema-invalid output, cancellation, and
|
|
||||||
application collection of execution observations. Use synthetic inputs and
|
|
||||||
controlled model responses without credentials or paid provider calls.
|
|
||||||
|
|
||||||
Keep schema validity distinct from domain acceptance: a structurally valid response
|
|
||||||
may still be rejected by Audita's deterministic or semantic checks. The example
|
|
||||||
should show how an application uses Promptkit's public contracts, not duplicate
|
|
||||||
Audita's processing pipeline.
|
|
||||||
|
|
||||||
## 6. Licensing Recommendation
|
|
||||||
|
|
||||||
**Implemented in source; not yet released.** The maintainer authorized
|
|
||||||
`LGPL-3.0-or-later`. See the [license summary](../../README.md#license),
|
|
||||||
[LGPL notice and text](../../LICENSE), and [accompanying GPL text](../../COPYING).
|
|
||||||
Existing release tags remain unchanged. Audita must select a subsequent published
|
|
||||||
release containing this change and satisfy the applicable distribution obligations.
|
|
||||||
|
|
||||||
## Suggested Evaluation Order and Scope
|
|
||||||
|
|
||||||
The implemented retry, classification, observation, and per-backend capacity
|
|
||||||
contracts can now be evaluated together with the representative example.
|
|
||||||
Audita still owns its migration decision and must select a later published
|
|
||||||
release before adopting this source state.
|
|
||||||
|
|
||||||
Existing facilities for embedded sources, application-owned profiles, prepared
|
|
||||||
execution, JSON-schema validation, and request-scoped credentials are useful
|
|
||||||
adoption foundations rather than new requests. Audita can bridge its application
|
|
||||||
configuration and credentials into Promptkit's public API; a dependency between
|
|
||||||
Promptkit and ConfigKit is not requested.
|
|
||||||
|
|
||||||
Transcript normalization, chunking, module ordering, semantic validators, correction
|
|
||||||
application, correction-ledger assembly, and filesystem retention remain Audita's
|
|
||||||
responsibility. No requested feature requires Promptkit to own those concerns.
|
|
||||||
@@ -1,503 +0,0 @@
|
|||||||
# Reliable Execution Implementation Plan
|
|
||||||
|
|
||||||
## Scope And Working Rules
|
|
||||||
|
|
||||||
Implement the target in [reliable-execution.md](reliable-execution.md). This plan
|
|
||||||
is for sequential execution by a coding agent, one numbered stage per prompt.
|
|
||||||
Stages intentionally separate transport refactoring, retry behavior, lifecycle
|
|
||||||
instrumentation, public integration, and consumer evidence. Do not merge stages
|
|
||||||
merely because their production changes look small.
|
|
||||||
|
|
||||||
Read all [repository policies](../development.md#initial-orientation), especially
|
|
||||||
the [agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
|
||||||
[documentation](../policy/documentation.md), [testing](../policy/testing.md), and
|
|
||||||
[release](../policy/release.md) policies. Preserve unrelated changes. Commit each
|
|
||||||
completed, validated stage with a plain-English message and report remaining
|
|
||||||
work. Do not push, tag, publish, rewrite history, or change dependencies. Licensing
|
|
||||||
is complete. No new live-provider checks or release gates are needed.
|
|
||||||
|
|
||||||
Use focused tests at each stage; run race tests for concurrency changes. Keep
|
|
||||||
intermediate commits building, and expose public controls only when their
|
|
||||||
behavior works. Update owning documentation alongside implemented behavior,
|
|
||||||
not ahead of it. Exact internal helper names may change without approval; the
|
|
||||||
consumer semantics and explicit choices below must not drift silently.
|
|
||||||
|
|
||||||
## Decisions Shared By All Stages
|
|
||||||
|
|
||||||
### Public Controls And Prepared Execution
|
|
||||||
|
|
||||||
- Add `Config.TransientRetries *int` and `RunRequest.TransientRetries *int`.
|
|
||||||
Nil selects two additional attempts at construction or inherits the engine
|
|
||||||
value on a request. Zero disables retries. Reject negative values with
|
|
||||||
`ErrInvalidConfig` or `ErrInvalidRequest`, respectively. Do not invent a small
|
|
||||||
maximum; count actual attempts with uint64 and convert the nonnegative budget
|
|
||||||
before adding the initial attempt, avoiding int overflow at MaxInt. Use
|
|
||||||
overflow-safe duration arithmetic. Copy pointed values.
|
|
||||||
- Validate request retry values in `Prepare`, `PrepareExecution`, and `Run`.
|
|
||||||
`Prepare` performs no retries; `PrepareExecution` freezes the effective retry
|
|
||||||
count privately. Do not add retry settings to profile YAML, execution targets,
|
|
||||||
provider payloads, prepared metadata, or prompt hashes.
|
|
||||||
- Add `RunRequest.Observer ExecutionObserver`, with
|
|
||||||
`type ExecutionObserver func(ExecutionEvent)`. It has no error return and is
|
|
||||||
excluded from JSON. Ordinary `Run` observes that invocation. Successful
|
|
||||||
`PrepareExecution` retains the function in its private payload for the owning
|
|
||||||
`RunPrepared`; `Prepare` and `PrepareExecution` do not emit execution events.
|
|
||||||
No new `RunPrepared` parameters or parallel execution API are introduced.
|
|
||||||
- `RunPrepared` observation starts only after successful atomic claim. Invalid,
|
|
||||||
foreign, reused, discarded, or nil handles cannot supply an execution observer
|
|
||||||
and emit no events. Preparation failures return their existing errors without
|
|
||||||
execution events. Ordinary `Run` does observe pre-generation errors, including
|
|
||||||
facade conversion errors and a nil engine if an observer was supplied.
|
|
||||||
- Clear retained observers and private execution settings on claim completion
|
|
||||||
or discard along with the existing private payload. Never invoke observers
|
|
||||||
while holding the prepared-handle mutex. A callback is retained by identity;
|
|
||||||
its closure is not cloned. Document caller synchronization responsibilities.
|
|
||||||
- Injected `LLMClient` values are never automatically retried, even when configured
|
|
||||||
retry values are positive. Values are still validated. A custom HTTP client
|
|
||||||
used by the built-in implementation does receive built-in retries; explicit
|
|
||||||
zero disables them when a transport already owns retries.
|
|
||||||
|
|
||||||
### Retry Semantics
|
|
||||||
|
|
||||||
Only HTTP 429, 500, 502, 503, and 504 permit a retry. Transport failures, timeout,
|
|
||||||
malformed envelopes, invalid requests, and nil collaborator responses do not.
|
|
||||||
Content repair remains separate. Each repair generation gets the same configured
|
|
||||||
transient budget as initial generation, never a budget shared across the run.
|
|
||||||
|
|
||||||
For retry ordinal k starting at 1, use equal jitter in [base/2, base], where
|
|
||||||
base is min(500 milliseconds × 2^(k-1), 8 seconds). Cap before doubling to avoid
|
|
||||||
overflow. These are documented defaults, not public clock/backoff options.
|
|
||||||
Internal tests can supply a clock, jitter function, and cancellable wait function
|
|
||||||
through private constructors; do not change package globals or sleep in tests.
|
|
||||||
|
|
||||||
Parse `Retry-After` as unsigned decimal seconds or an HTTP date via
|
|
||||||
`http.ParseTime`, following [RFC 9110 section 10.2.3](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3).
|
|
||||||
Use the first header value; trim whitespace; negative, fractional, or malformed
|
|
||||||
values use local backoff. Past dates supply zero additional minimum. Measure a
|
|
||||||
valid delay from response receipt, not from the end of body decoding; wait until
|
|
||||||
the later of that not-before time and the local backoff deadline. Never apply
|
|
||||||
the 8-second local cap to the provider minimum. Saturate unrepresentably large
|
|
||||||
valid delays safely rather than wrapping or treating them as permission to
|
|
||||||
retry early. Cancellation interrupts the wait; it must not busy-loop.
|
|
||||||
|
|
||||||
Check the caller context before waiting, before permit acquisition, and before
|
|
||||||
dispatch. Caller cancellation/deadline wins when observed at those boundaries.
|
|
||||||
Do not proactively reject long provider delays; wait cancellably. A context
|
|
||||||
without a deadline can wait a long time, which must be documented.
|
|
||||||
|
|
||||||
Existing HTTP-client and target generation timeouts apply separately to each
|
|
||||||
actual attempt after capacity admission. Retry waiting and capacity waiting use
|
|
||||||
the original caller context, not a previously canceled attempt context. No
|
|
||||||
new overall timeout is imposed. Never reset the caller's deadline.
|
|
||||||
|
|
||||||
Freeze built-in request payload, URL, and resolved authentication once per
|
|
||||||
generation sequence, before its first attempt; create a fresh HTTP request/body
|
|
||||||
reader and timeout context for every attempt. An absent optional key must remain
|
|
||||||
absent if the environment changes during retries. Repair is a new generation
|
|
||||||
sequence and may resolve the environment again, consistent with existing calls.
|
|
||||||
Do not freeze environment values at `PrepareExecution` time.
|
|
||||||
|
|
||||||
When R is zero, return the ordinary first failure, not retry exhaustion. When
|
|
||||||
R is positive and all R+1 attempts return eligible statuses, return exhaustion.
|
|
||||||
If a later failure is noneligible or the context stops the sequence, return that
|
|
||||||
failure rather than exhaustion. Count attempts actually dispatched, not waits
|
|
||||||
scheduled. For one run, the bound is (R+1)×(P+1), where P is the repair budget;
|
|
||||||
document the formula but do not compute it using overflow-prone `int` products.
|
|
||||||
|
|
||||||
### Error Contract
|
|
||||||
|
|
||||||
Add public `ErrTransport`, `ErrMalformedResponse`, and `ErrRetriesExhausted`.
|
|
||||||
Transport/malformed failures during generation also retain `ErrLLMGenerate`.
|
|
||||||
Preserve standard context identities and original transport causes for
|
|
||||||
`errors.Is`/`errors.As`. Do not expose new internal concrete error types as the
|
|
||||||
consumer contract or stringify sensitive transport errors while mapping them.
|
|
||||||
|
|
||||||
Add immutable `RetryExhaustedError` with `Attempts() uint64`, `Unwrap() error`,
|
|
||||||
`Is(error) bool` matching `ErrRetriesExhausted`, and redacted `Error`/`GoString`.
|
|
||||||
Map its cause recursively so `errors.As` still finds the existing public
|
|
||||||
`GenerationError` and status/detail accessors. Nil receivers and zero values
|
|
||||||
must be safe, consistent with existing typed errors. No stable JSON contract.
|
|
||||||
Do not let the current provider-error special case in `mapPublicError` discard
|
|
||||||
the exhaustion wrapper or outer validation/generation identities.
|
|
||||||
|
|
||||||
Injected errors retain their identities; do not guess transport or HTTP
|
|
||||||
classifications by text or structural resemblance. An injected client can use
|
|
||||||
the public sentinels deliberately. A nil successful collaborator response is
|
|
||||||
malformed generation. Completed structural rejection remains a normal result
|
|
||||||
with failed validation; operational validation errors remain errors with no
|
|
||||||
partial `RunResult`.
|
|
||||||
|
|
||||||
### Observation Values And Accounting
|
|
||||||
|
|
||||||
Define public values in `observations.go`, with separate internal equivalents
|
|
||||||
and facade conversion, not aliases. They have no stable serialized contract;
|
|
||||||
consumers translate them into application-owned records.
|
|
||||||
|
|
||||||
`ExecutionEvent` contains:
|
|
||||||
|
|
||||||
- `Kind ExecutionEventKind`: constants `ExecutionStarted`, `ExecutionAttempt`,
|
|
||||||
`ExecutionRetryDelay`, `ExecutionFinished` (string values `started`, `attempt`,
|
|
||||||
`retry_delay`, `finished`).
|
|
||||||
- `RunID string`, `Time time.Time`, and `PromptID`, `PromptVersion`, `ProfileID`,
|
|
||||||
`BackendID`, `Model` string fields. Use the same RunID as a successful result.
|
|
||||||
Start events for ordinary runs have no resolved identities; prepared runs
|
|
||||||
may supply their already-resolved snapshot. Enrich later events as resolution
|
|
||||||
succeeds. Do not label requested but unresolved values as selected values.
|
|
||||||
- `Generation uint64` (1 initial, 2 first repair), `Attempt uint64` (1-based within a
|
|
||||||
generation), and `Dispatched bool`. Non-attempt events use zero indexes unless
|
|
||||||
describing the retry delay after a specific failed attempt.
|
|
||||||
- `CapacityWait`, `GenerationDuration`, `RetryDelay`, `Duration time.Duration`.
|
|
||||||
- `Outcome ExecutionOutcome`: `success`, `invalid_output`, `error`, or empty
|
|
||||||
when not applicable; corresponding constants use the `ExecutionOutcome` prefix.
|
|
||||||
- `Failure ExecutionFailureKind`, `StatusCode int`, `RetriesExhausted bool`.
|
|
||||||
Failure constants use the `ExecutionFailure` prefix; `ExecutionFailureNone`
|
|
||||||
is the zero value empty string. Other values are `canceled`, `deadline`,
|
|
||||||
`transport`, `provider_status`,
|
|
||||||
`malformed_response`, `capacity`, `invalid_config`, `invalid_request`,
|
|
||||||
`prompt_not_found`, `profile_not_found`, `prompt_load`, `profile_load`,
|
|
||||||
`artifact_load`, `prompt_render`, `validation`, `generation`, `other`.
|
|
||||||
Missing credentials/profile selection use invalid_request. Classify context
|
|
||||||
identities first, then specific causes before broad operation categories.
|
|
||||||
Exhaustion is a separate flag, not a replacement for provider_status.
|
|
||||||
- `Usage ObservedTokenUsage` and terminal `Attempts`, `TransientRetries`,
|
|
||||||
`RepairCalls uint64`. Nonterminal aggregate counters are zero.
|
|
||||||
|
|
||||||
`ObservedTokenUsage` has the existing five token field names, each an
|
|
||||||
`ObservedTokenCount{Value int; Known bool; Complete bool}`. An attempt field is
|
|
||||||
known/complete exactly when the provider supplied that field as a non-null
|
|
||||||
integer. An aggregate Value sums known contributions, Known means at least one
|
|
||||||
known contribution, and Complete means every dispatched attempt reported that
|
|
||||||
field. For zero dispatched attempts use known complete zero totals. Preserve
|
|
||||||
reported values without deriving totals or reconciling fields. Unknown fields
|
|
||||||
have Value zero. Counts on non-dispatched attempts are unknown and excluded from
|
|
||||||
usage aggregation. This distinguishes partial accounting from complete zero.
|
|
||||||
|
|
||||||
Add optional `GenerateResponse.UsagePresence *TokenUsagePresence` with JSON tag
|
|
||||||
`usage_presence,omitempty`; `TokenUsagePresence` contains five bool fields with
|
|
||||||
the corresponding snake_case JSON names. For nil from legacy injected clients,
|
|
||||||
nonzero fields are known and zero fields are unknown: the old interface cannot
|
|
||||||
distinguish omitted zero from measured zero. An explicit presence value permits
|
|
||||||
known zeros and governs all five fields. Copy it on conversion. Existing result
|
|
||||||
accounting still sums all Usage values regardless of this observation metadata;
|
|
||||||
`TokenUsage` and existing `RunResult` JSON remain unchanged. New
|
|
||||||
GenerateResponse JSON round-trips the additive field and omits nil. The built-in
|
|
||||||
adapter always supplies explicit presence; absent/null wire counts are unknown.
|
|
||||||
|
|
||||||
Emit one attempt-completion event per acquisition attempt, including interrupted
|
|
||||||
capacity waiting with Dispatched=false. GenerationDuration covers dispatch
|
|
||||||
through response decoding/closure, excludes capacity/backoff/callback time.
|
|
||||||
CapacityWait covers acquiring a permit, and is zero for the unlimited path.
|
|
||||||
Release the active permit before invoking observers. Dispatched means invoking
|
|
||||||
the prepared HTTP attempt or injected Generate, not a proven network packet or
|
|
||||||
billable request; hidden HTTP redirects/transport retries are not observable.
|
|
||||||
|
|
||||||
Emit retry-delay completion after each actual wait, including interrupted waits,
|
|
||||||
with elapsed RetryDelay and the failed attempt indexes. No delay event when no
|
|
||||||
wait starts. Emit terminal totals once on each normal return, including errors,
|
|
||||||
after admission release. Attempts counts dispatched calls; TransientRetries
|
|
||||||
counts dispatched attempts numbered above 1; RepairCalls counts generations
|
|
||||||
above 1 with at least one dispatched attempt. Do not count scheduled work.
|
|
||||||
Terminal phase-duration fields sum their respective measured phases; Duration
|
|
||||||
is wall-clock operation time
|
|
||||||
and can additionally include preparation, validation, callbacks, and other work.
|
|
||||||
Do not add terminal usage to attempt usage. Attempt success means generation
|
|
||||||
succeeded, not that later output validation accepted it.
|
|
||||||
|
|
||||||
Synchronous callbacks are ordered per operation, may overlap across operations,
|
|
||||||
and cannot return errors. Event values contain no mutable containers or error
|
|
||||||
objects. Never include endpoint URLs, headers, keys, session values, bodies,
|
|
||||||
validation diagnostics, or provider text. Metadata identifiers can still be
|
|
||||||
sensitive. Callback panics propagate; defer cleanup before first callback after
|
|
||||||
resource acquisition. Do not promise terminal events after panics or process
|
|
||||||
termination and do not invoke further callbacks while unwinding a panic.
|
|
||||||
Callbacks must return for the operation to progress; cancellation cannot forcibly
|
|
||||||
interrupt consumer code. Warn against blocking on work requiring the same run's
|
|
||||||
admission slot. No asynchronous delivery or durable history.
|
|
||||||
|
|
||||||
## Stage 1 — Expose Missing Generation Failure Categories
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Read `errors.go`, `generation_error.go`, `llm_adapter.go`, the LLM error types,
|
|
||||||
and public error-boundary tests. Add public transport/malformed sentinels and
|
|
||||||
map current built-in errors while preserving cause identity and redacted text.
|
|
||||||
Classify actual request execution failures as transport, not unrelated request
|
|
||||||
encoding failures merely sharing the old broad ErrRequestFailed sentinel.
|
|
||||||
Normalize nil successful injected responses into the malformed category. Do not
|
|
||||||
add retries or observers yet. Retain existing provider-status and validation
|
|
||||||
behavior. Add the error GoDoc and a short consumer-guide classification update.
|
|
||||||
|
|
||||||
Verify public tests for transport causes, caller cancellation/deadline, malformed
|
|
||||||
framing/choices, nil response, provider detail redaction, and injected arbitrary
|
|
||||||
errors. Reuse current fixtures; avoid duplicating the entire transport matrix.
|
|
||||||
Run `go test ./...`. Commit when mappings and existing tests pass.
|
|
||||||
|
|
||||||
## Stage 2 — Preserve Usage Presence Through The Adapter
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Read wire response decoding, domain/public response types, facade conversion,
|
|
||||||
and JSON contract tests. Add the presence types and optional public response
|
|
||||||
field above. Convert wire usage counts to pointer-aware decoding, retaining
|
|
||||||
existing integer values and response validation behavior. Explicit null and
|
|
||||||
omission are unknown; numeric zero is known. Invalid numeric types retain current
|
|
||||||
malformed-response handling. Apply the nil-presence fallback above only to
|
|
||||||
observation metadata, never to existing successful-result accounting.
|
|
||||||
|
|
||||||
Test absent usage, partial nested usage, nulls, zero, and all five reported
|
|
||||||
fields at the decoder boundary. Test public optional-field JSON round trips and
|
|
||||||
defensive copying once at the facade. Existing successful RunResult totals and
|
|
||||||
JSON must be unchanged. Document injected-client presence semantics in GoDoc.
|
|
||||||
No observation API yet. Validate affected packages and commit.
|
|
||||||
|
|
||||||
## Stage 3 — Separate Frozen HTTP Preparation From One Attempt
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Refactor `internal/llm/OpenAICompatibleClient` without changing its one-shot
|
|
||||||
`Generate` behavior. Add an internal `PrepareGeneration` operation that validates
|
|
||||||
and encodes one request, resolves credentials once, and returns private frozen
|
|
||||||
state with an `Execute(ctx)` method. It creates a fresh body/request/deadline
|
|
||||||
on every Execute and closes every response before returning.
|
|
||||||
|
|
||||||
Return an internal attempt outcome containing response, error, HTTP status,
|
|
||||||
usage plus presence, and parsed provider not-before timing. Preserve known usage
|
|
||||||
from a strictly decoded successful envelope even if missing choices/content
|
|
||||||
subsequently make it malformed; do not salvage data from invalid framing or
|
|
||||||
non-success error bodies. Preserve current body-size limits and redaction.
|
|
||||||
The existing Generate method delegates once and returns no partial response on
|
|
||||||
error. After failed body reading/decoding, retain an observed attempt-context
|
|
||||||
cancellation/deadline instead of hiding it behind malformed-response formatting.
|
|
||||||
Timing/header data remains internal, not provider-error strings.
|
|
||||||
|
|
||||||
Implement Retry-After parsing here, but no sleeping or retry loop. Test frozen
|
|
||||||
payload/authentication across repeated Execute calls, including an initially
|
|
||||||
absent optional key; independent per-attempt timeout contexts; body closure;
|
|
||||||
usage on malformed choices; and header parsing with an explicit receipt time.
|
|
||||||
Use separate Execute contexts to verify a canceled attempt does not poison the
|
|
||||||
next call. All existing wire tests must pass. Commit this refactoring separately.
|
|
||||||
|
|
||||||
## Stage 4 — Implement The Internal Retry And Capacity Coordinator
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Create `internal/generation`, depending on domain, llm, and capacity. It implements
|
|
||||||
the existing `llm.Client` consumed by the runner and repairer. For built-in
|
|
||||||
generation it prepares once through Stage 3, then loops over capacity-controlled
|
|
||||||
Execute calls. For injected generation it dispatches once. Constructor arguments
|
|
||||||
explicitly select the path; do not infer retry eligibility from arbitrary errors.
|
|
||||||
|
|
||||||
Expose an internal Manager active-permit acquisition method returning an
|
|
||||||
idempotent release function. Reuse pool logic; never build another queue. Keep
|
|
||||||
the current capacity client as a delegating one-shot wrapper until engine wiring
|
|
||||||
moves, sharing that acquisition primitive. The coordinator defers permit release
|
|
||||||
inside each attempt; no permit survives a wait or callback. Admission remains
|
|
||||||
runner-owned. Invalid frozen-request preparation dispatches nothing.
|
|
||||||
|
|
||||||
Implement the exact delay/context/count semantics above with private test seams.
|
|
||||||
Introduce an internal exhaustion wrapper retaining the cause and attempt count.
|
|
||||||
The coordinator accepts a resolved retry count on the internal GenerateRequest;
|
|
||||||
it has no public API or engine behavior change yet. Repair propagation arrives
|
|
||||||
in Stage 5. Leave observation hooks for later stages rather than adding no-op
|
|
||||||
public callbacks now.
|
|
||||||
|
|
||||||
Tests own the retry status table, R=0, exhaustion, a noneligible later failure,
|
|
||||||
context interruption, provider minimum versus jitter, overflow-safe backoff,
|
|
||||||
frozen absent/present credentials, and independent attempt deadlines. A blocked
|
|
||||||
backoff must allow another call to acquire the only active permit. Cancellation
|
|
||||||
and panic must not leak permits. Run focused race tests and commit.
|
|
||||||
|
|
||||||
## Stage 5 — Wire Retry Configuration Through Both Execution Paths
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Add public/internal configuration and request fields. Thread the copied effective
|
|
||||||
retry count through ordinary resolution, private prepared payloads,
|
|
||||||
`executePreparedRun`, `newGenerationRequest`, and `RepairRequest`. Do not add it
|
|
||||||
to public GenerateRequest or provider settings: injected clients own their policy.
|
|
||||||
Keep one shared resolution helper for default/override/negative validation.
|
|
||||||
|
|
||||||
In `internal/engine.New`, assemble the generation coordinator around the selected
|
|
||||||
built-in or injected client and shared manager. Do not wrap it again in
|
|
||||||
capacity.NewClient: that would hold a permit over waits or deadlock reacquisition.
|
|
||||||
Initial generation and default repairer must receive the same coordinator.
|
|
||||||
Remove obsolete capacity-wrapper machinery/tests only when its distinct behavior
|
|
||||||
is covered by the coordinator and manager tests; do not retain two live schedulers.
|
|
||||||
|
|
||||||
Add the public exhaustion type and recursive error mapping. Update old tests
|
|
||||||
expecting one eligible HTTP failure to set explicit zero where their purpose is
|
|
||||||
not retry behavior, rather than making unrelated tests wait through backoff.
|
|
||||||
Document the newly active controls, default two retries, and per-attempt timeout
|
|
||||||
semantics now. Add public tests for request/engine precedence, copied pointers,
|
|
||||||
prepared freezing, repair budget multiplication, exhaustion cause access, and
|
|
||||||
injected-client single dispatch. Run full tests and focused race tests; commit.
|
|
||||||
|
|
||||||
## Stage 6 — Build The Internal Observation Recorder
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Create `internal/observation` for invocation-local counters, presence-aware
|
|
||||||
aggregation, and synchronous dispatch. Internal event/value counterparts and
|
|
||||||
the callback function type belong in domain so domain requests/payloads can
|
|
||||||
carry them without importing the recorder package. The recorder depends only
|
|
||||||
on domain/standard library, not provider or root types. No global map, history
|
|
||||||
slice, mutex held across callbacks, goroutine, or background queue is needed.
|
|
||||||
Keep outcome classification at owning boundaries; recorder consumes normalized
|
|
||||||
failure metadata, never raw error strings.
|
|
||||||
|
|
||||||
Move/reuse run-ID generation here so observation and successful results have one
|
|
||||||
identity owner. Provide start/update/attempt/delay/finish operations and a private
|
|
||||||
context key to pass the invocation-local recorder through the existing internal
|
|
||||||
client interfaces. Context propagation is solely internal plumbing, not public
|
|
||||||
control policy. A nil observer avoids event delivery, but counts needed for errors
|
|
||||||
and existing results must still work. If run-ID creation fails, report an error
|
|
||||||
terminal event with an empty RunID rather than invent a valid identifier.
|
|
||||||
|
|
||||||
Test phase sums, dispatched-only counters, partial/unknown/zero usage, no double
|
|
||||||
counting, callback order, retained-value independence, and panic propagation.
|
|
||||||
Do not promise event delivery across panics. There is no public observer or runner
|
|
||||||
wiring in this stage. Update the internal inventory when the package lands.
|
|
||||||
|
|
||||||
## Stage 7 — Instrument Attempts And Retry Delays
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Connect the coordinator to the internal recorder through context. Record measured
|
|
||||||
capacity wait even on interrupted acquisition, actual dispatch and outcome, usage
|
|
||||||
from Stage 3, and completed/canceled backoff. Release the active permit before
|
|
||||||
callback delivery. Collect injected response usage using presence from Stage 2;
|
|
||||||
arbitrary injected failures have unknown usage. Error response bodies are not
|
|
||||||
mined for usage.
|
|
||||||
|
|
||||||
Do not add an event for request freezing failure: the terminal operation will
|
|
||||||
report it without pretending a dispatch happened. A completed attempt callback
|
|
||||||
that cancels context must prevent another attempt. Measure phase end before its
|
|
||||||
callback so observer work is not reported as provider time. Feed normalized HTTP
|
|
||||||
and context classifications without exposing sensitive strings.
|
|
||||||
|
|
||||||
Tests cover retry then success, exhaustion, interrupted wait, canceled capacity,
|
|
||||||
success then later repair failure (using internal recorder plumbing), legacy
|
|
||||||
injected usage and explicit unknown fields, and callbacks outside permits/locks.
|
|
||||||
Use synchronization rather than elapsed-time thresholds. Run race tests; commit.
|
|
||||||
|
|
||||||
## Stage 8 — Add Ordinary-Run Lifecycle And Public Observation API
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Add public observation types/conversion and RunRequest.Observer. The facade adapts
|
|
||||||
the function and delegates bookkeeping to the internal recorder. Start ordinary
|
|
||||||
Run observation before request conversion and nil-engine rejection; finish after
|
|
||||||
public error mapping so the terminal classification matches consumer errors.
|
|
||||||
Use the recorder's RunID/start in the runner instead of generating another ID.
|
|
||||||
Without an observer preserve existing Result timing conventions; with an observer
|
|
||||||
its Duration includes the facade boundary and can be slightly longer.
|
|
||||||
|
|
||||||
Enrich resolved identities at the runner's resolution boundary, before admission;
|
|
||||||
do not copy unresolved input strings into selected fields. Generation indexes are
|
|
||||||
passed through initial and repair paths. Ensure final totals include completed
|
|
||||||
generation usage before validation or repair errors. Emit finished only on normal
|
|
||||||
return, after all admission cleanup; do not use a defer that calls user callbacks
|
|
||||||
during panic unwinding. Keep `Run` signatures and no-partial-result behavior.
|
|
||||||
|
|
||||||
Tests cover source/conversion failure, nil engine, admission rejection, validation
|
|
||||||
operational failure, valid/invalid/skipped validation, success followed by repair
|
|
||||||
failure, matching RunID, redaction, and observer panic cleanup. Verify callback
|
|
||||||
mutation/retention cannot alter execution state. Update public GoDoc and consumer
|
|
||||||
guidance for ordinary observations only; run root and internal race tests; commit.
|
|
||||||
|
|
||||||
## Stage 9 — Observe Prepared Execution Without Weakening Its Lifecycle
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Retain the adapted observer with the effective retries in the private prepared
|
|
||||||
payload. Claim atomically before starting observation, release the claim mutex,
|
|
||||||
install payload cleanup, then emit started. Preparation emits nothing; successful
|
|
||||||
claim followed by credential failure, capacity rejection, or cancellation still
|
|
||||||
emits finished on a normal return. Invalid/reused/foreign handles emit nothing
|
|
||||||
and must neither invoke nor discard another execution's callback.
|
|
||||||
|
|
||||||
Use the same recorder, identity enrichment, failure normalization, and terminal
|
|
||||||
accounting as ordinary execution. Keep callback work outside handle and manager
|
|
||||||
locks. Discard clears observer references without calling them. Concurrent calls
|
|
||||||
using one handle result in one executing observation sequence, not duplicate runs.
|
|
||||||
|
|
||||||
Add focused prepared public tests for frozen retry choice/observer, discarded and
|
|
||||||
foreign handles, claim races, no preparation events, later failures with usage,
|
|
||||||
and panic cleanup. Preserve existing preparation/result timing distinctions.
|
|
||||||
Document the prepared observer lifetime and exclusions. Run race tests; commit.
|
|
||||||
|
|
||||||
## Stage 10 — Verify Cross-Cutting Public Contracts
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Review assembled behavior through the public facade using controlled HTTP
|
|
||||||
transports and small synthetic fixtures. Add only missing integration evidence:
|
|
||||||
|
|
||||||
- Two profiles sharing one finite backend coordinate initial, retry, and repair
|
|
||||||
traffic. Backoff frees the active slot but not logical admission. Different
|
|
||||||
backend pools remain independent; unlimited/endpoint-only paths still work.
|
|
||||||
- A sequence of transient failure, successful invalid output, repair transient
|
|
||||||
failure, and successful valid output produces consistent counts, known partial
|
|
||||||
usage, result usage, events, and error identities. Exercise exhaustion and
|
|
||||||
cancellation as focused variants, not a Cartesian product.
|
|
||||||
- Frozen request/authentication and prepared source immutability survive retries.
|
|
||||||
Noneligible errors never replay. Injected client attempts remain opaque.
|
|
||||||
- Safe default error/event formatting and JSON never disclose credential markers,
|
|
||||||
request bodies, provider message markers, or retained callback internals.
|
|
||||||
|
|
||||||
Do not require fixed callback timing, random samples, exact error wording, or
|
|
||||||
internal helper-call sequences. Fix interaction defects in their owning layer.
|
|
||||||
Run `go test -race -vet=off ./...` and `go vet ./...`; commit the verified scope.
|
|
||||||
|
|
||||||
## Stage 11 — Add The Offline Proposal And Assessment Example
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Add `examples/go-library/observe` with embedded prompt/schema assets, two
|
|
||||||
application-owned profiles sharing a finite custom backend, and one engine.
|
|
||||||
Use a controlled RoundTripper under the built-in client: return one eligible
|
|
||||||
status before a valid structured proposal, then a structurally valid assessment
|
|
||||||
that the application can accept/reject independently. Collect metadata events
|
|
||||||
and print a compact structured summary, including actual attempt/retry counts
|
|
||||||
and assessment outcome. Do not log prompt/response bodies or provider detail.
|
|
||||||
Accept the real short default backoff in this executable demonstration; unit
|
|
||||||
tests still use controlled waiting. Do not add public timing controls for it.
|
|
||||||
|
|
||||||
Keep cancellation/exhaustion/schema-invalid branch coverage in focused tests
|
|
||||||
from earlier stages, not a large example switchboard. Add this example once to
|
|
||||||
the existing release-check script with semantic output assertions; do not run
|
|
||||||
a redundant second integration suite there. Update the development guide,
|
|
||||||
testing policy's example inventory, and internal overview accordingly. Run the
|
|
||||||
example and its relevant tests; commit.
|
|
||||||
|
|
||||||
## Stage 12 — Complete Documentation And Final Validation
|
|
||||||
|
|
||||||
**Completed:** 2026-09-23.
|
|
||||||
|
|
||||||
Reconcile GoDoc, consumer error/observer/retry guidance, provider integration
|
|
||||||
docs, and internal runner/LLM/capacity/generation/observation documentation with
|
|
||||||
the implemented design. Remove claims that no retries exist or that usage can
|
|
||||||
only be learned on success. Clearly distinguish observations from complete cost
|
|
||||||
accounting and explain nil legacy injected usage presence.
|
|
||||||
|
|
||||||
Document single-attempt migration via explicit zero, multiplication with repair
|
|
||||||
and application semantic retries, provider-delay/caller-deadline behavior,
|
|
||||||
callback lifetime/concurrency/panics, and no-partial-result compatibility. Do not
|
|
||||||
copy the same field reference into multiple documents. Link the new example.
|
|
||||||
Update Audita's wishlist status to implemented where verified. Do not invent a
|
|
||||||
release version or tag; versioned release notes and completed-roadmap retirement
|
|
||||||
belong to the later authorized release-preparation task.
|
|
||||||
|
|
||||||
Run `./scripts/release-check.sh` once for the final implementation, inspect the
|
|
||||||
diff for accidental contracts, dependencies, or artifacts, and verify no public
|
|
||||||
signature exposes internal types. Record results and any real remaining gaps.
|
|
||||||
Commit the final coherent documentation/integration work; report readiness for
|
|
||||||
review without publishing or waiting for CI.
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
None requiring a maintainer decision remain. The approved roadmap policies and
|
|
||||||
the explicit API, accounting, timing, and lifecycle choices above are the basis
|
|
||||||
for this implementation. If implementation reveals a conflict with an existing
|
|
||||||
explicit contract, report the concrete conflict rather than silently adding a
|
|
||||||
restriction or weakening that contract.
|
|
||||||
@@ -1,219 +0,0 @@
|
|||||||
# Reliable And Observable Execution
|
|
||||||
|
|
||||||
**Status:** Implemented in source; not yet released. This document records the
|
|
||||||
accepted target and approved policy choices. The companion
|
|
||||||
[implementation plan](implementation.md) records implementation steps and
|
|
||||||
concrete API and algorithm decisions. Current contracts are owned by public
|
|
||||||
GoDoc and the linked consumer and integration documentation.
|
|
||||||
|
|
||||||
## Purpose
|
|
||||||
|
|
||||||
Make Promptkit responsible for reusable provider-failure recovery and execution
|
|
||||||
accounting, including failed operations, without becoming an application workflow
|
|
||||||
or telemetry framework. Consumers should not need separate retry loops merely
|
|
||||||
to handle temporary provider failures, nor parse error strings to understand
|
|
||||||
what happened.
|
|
||||||
|
|
||||||
This scope incorporates requests 1–5 from the
|
|
||||||
[Audita wishlist](audita-feature-requests.md). Retries, error classification,
|
|
||||||
observations, shared capacity, and consumer evidence belong together because
|
|
||||||
they describe the same execution lifecycle. Licensing was handled separately.
|
|
||||||
Audita motivates the work but does not define an application-specific API.
|
|
||||||
|
|
||||||
## Foundations At Planning Time
|
|
||||||
|
|
||||||
At the time this scope was accepted, Promptkit provided structural output
|
|
||||||
validation and bounded repair, prepared execution, engine-local backend
|
|
||||||
capacity, run metadata, aggregate token usage, and typed provider-status and
|
|
||||||
capacity errors. Operational errors did not return partial run results. The
|
|
||||||
built-in path had no transient retry loop, and existing integer usage fields
|
|
||||||
did not distinguish missing reports from zero.
|
|
||||||
|
|
||||||
The implemented contracts are owned by public GoDoc, the
|
|
||||||
[consumer guide](../consumers/pkg-promptkit.md), and the
|
|
||||||
[provider integration contract](../integrations/openai-compatible-chat.md).
|
|
||||||
The [capacity document](../internal/capacity.md) and
|
|
||||||
[runner document](../internal/runner.md) describe the existing implementation.
|
|
||||||
This historical baseline does not override their current-state descriptions.
|
|
||||||
|
|
||||||
## Target End State
|
|
||||||
|
|
||||||
### Bounded Transient-Failure Retries
|
|
||||||
|
|
||||||
The built-in OpenAI-compatible client path supports configurable transient
|
|
||||||
retries. The initial policy uses two additional attempts by default and permits
|
|
||||||
an unambiguous explicit zero to disable them. Initial eligible statuses are
|
|
||||||
429, 500, 502, 503, and 504, not every unsuccessful response or every 5xx status.
|
|
||||||
|
|
||||||
Configure an engine default with a request-scoped retry-count override, using
|
|
||||||
presence to distinguish omission from zero. Keep retry policy out of profile
|
|
||||||
YAML initially. Per-request control supports mixed workflows without requiring
|
|
||||||
separate engines and splitting their capacity pools. Use a modest fixed backoff
|
|
||||||
policy without a new public timing-configuration surface.
|
|
||||||
|
|
||||||
- Retry delays use exponential backoff with jitter and honor valid provider
|
|
||||||
`Retry-After` timing as a minimum delay, subject to cancellation, rather than
|
|
||||||
truncating it and retrying too early. Invalid or absent timing falls back to
|
|
||||||
local policy.
|
|
||||||
- Caller cancellation and deadlines interrupt both waiting and active attempts.
|
|
||||||
Cancellation is never itself a reason to retry.
|
|
||||||
- Permanent request failures, malformed successful response envelopes, and
|
|
||||||
structurally invalid generated content do not enter this transient loop.
|
|
||||||
Structural invalidity remains the separate output-repair mechanism.
|
|
||||||
- Transport failures are classified and exposed without automatic replay in
|
|
||||||
this scope. A generic network error does not prove that no billable work
|
|
||||||
occurred. Conservative eligibility reduces ambiguity; even HTTP-status
|
|
||||||
retries do not promise zero duplicate billing.
|
|
||||||
- Retry one resolved generation request: retain its messages, model, endpoint,
|
|
||||||
settings, structured-output constraint, and resolved credential throughout
|
|
||||||
that retry sequence. Do not rerender or reload application sources between
|
|
||||||
attempts. Resolve environment credentials at execution, not when preparing
|
|
||||||
a handle that may remain unexecuted for a long time.
|
|
||||||
- Apply the same behavior to ordinary and prepared execution and to generation
|
|
||||||
performed for structural repair. A prepared handle remains single-use; its
|
|
||||||
execution may contain multiple internal provider attempts.
|
|
||||||
- Do not silently retry arbitrary injected `LLMClient` implementations. They
|
|
||||||
continue to own their retry policy. A custom HTTP transport under the built-in
|
|
||||||
client uses the built-in policy; consumers can disable it when their transport
|
|
||||||
already retries. Hidden attempts inside collaborators cannot be counted by
|
|
||||||
Promptkit and must not be presented as if they were observable.
|
|
||||||
|
|
||||||
Retries are not exactly-once generation. A failed or uncertain request may have
|
|
||||||
performed billable work. With R additional transient attempts per generation
|
|
||||||
and P additional structural repair generations, the maximum Promptkit-controlled
|
|
||||||
provider attempts for that execution is (R + 1) × (P + 1). Cancellation and
|
|
||||||
non-retryable failures can reduce that count. Application-owned semantic retries
|
|
||||||
are separate executions and may multiply it further.
|
|
||||||
|
|
||||||
Existing generation/HTTP timeouts remain per-attempt limits. The caller's
|
|
||||||
context bounds the complete operation, including capacity waiting, retry delays,
|
|
||||||
and repair; its deadline is never reset between attempts. Callers needing a
|
|
||||||
total time budget must provide a deadline, since multiple attempts can exceed
|
|
||||||
one attempt's timeout. An expired caller deadline ends the run rather than
|
|
||||||
starting another attempt. This preserves existing transport-setting semantics
|
|
||||||
without adding a competing total-timeout setting.
|
|
||||||
|
|
||||||
### Shared Capacity, Not A Second Scheduler
|
|
||||||
|
|
||||||
Every actual generation attempt, including retries and repair, participates in
|
|
||||||
the existing engine-local pool for its backend. Release active-generation
|
|
||||||
permits before backoff; waiting to retry must not monopolize active capacity.
|
|
||||||
Keep the logical execution's bounded admission reservation until it finishes.
|
|
||||||
Reacquiring a permit uses ordinary capacity scheduling, not a new priority lane.
|
|
||||||
|
|
||||||
Proposal and assessment calls in the representative example share one engine
|
|
||||||
and a finite backend limit. Existing unlimited custom backends and endpoint-only
|
|
||||||
profiles remain supported; do not impose a new global cap or silently change
|
|
||||||
catalog defaults. If a consumer requires a different limit, current custom
|
|
||||||
backend IDs and application-owned profiles remain the starting point.
|
|
||||||
|
|
||||||
Concurrency bounds simultaneous activity, not cumulative cost or request rate.
|
|
||||||
Independent engines, processes, and other clients do not share these limits.
|
|
||||||
|
|
||||||
### Actionable Public Failures
|
|
||||||
|
|
||||||
Extend existing public error contracts rather than replace them with an unrelated
|
|
||||||
taxonomy. Consumers must distinguish cancellation, deadlines, transport failures,
|
|
||||||
non-success provider status, malformed provider responses, retry exhaustion,
|
|
||||||
capacity rejection, and completed output-validation failure without parsing text.
|
|
||||||
|
|
||||||
Exhaustion retains the final underlying cause and actual attempt count. Existing
|
|
||||||
context identities, provider status/detail access, and `errors.Is`/`errors.As`
|
|
||||||
behavior remain useful through any added wrapping. An error can be both an
|
|
||||||
exhaustion and a particular provider failure; categories need not be exclusive.
|
|
||||||
|
|
||||||
Completed invalid output remains a validation result, not an operational error.
|
|
||||||
Operational failures continue to return no partial `RunResult`; observations
|
|
||||||
provide accounting independently. Preserve redacted default diagnostics and
|
|
||||||
deliberate access to untrusted provider detail under application disclosure policy.
|
|
||||||
|
|
||||||
### Optional Execution Observations
|
|
||||||
|
|
||||||
Provide an optional request-scoped observer through the public facade. It receives
|
|
||||||
metadata about an execution without requiring a logging or metrics framework.
|
|
||||||
Promptkit does not retain a global execution history or persist reports.
|
|
||||||
|
|
||||||
Deliver callbacks synchronously and in order within an execution. Concurrent
|
|
||||||
executions may invoke a shared callback concurrently. Do not invoke consumer
|
|
||||||
code under capacity-manager locks or add a background delivery queue. Callbacks
|
|
||||||
cannot return execution-policy errors; consumers keep them fast and handle their
|
|
||||||
own reporting failures. Panics follow ordinary injected-code behavior rather
|
|
||||||
than being silently swallowed. This keeps delivery straightforward without
|
|
||||||
buffering, dropping, or shutdown machinery; slow callbacks can still delay a run.
|
|
||||||
|
|
||||||
The observable lifecycle covers successful execution, completed invalid output,
|
|
||||||
and operational failure, including failures before any model call. Expose only
|
|
||||||
identities available at that point; do not invent selected profiles or backends
|
|
||||||
when resolution failed. Prepared execution observations distinguish execution
|
|
||||||
from earlier preparation and consumer-held delay.
|
|
||||||
|
|
||||||
Consumers can correlate the logical execution with generation attempts and learn:
|
|
||||||
|
|
||||||
- the available prompt, profile, backend, and effective model identities;
|
|
||||||
- whether generation is initial or structural repair, and which transient
|
|
||||||
attempt occurred within it;
|
|
||||||
- capacity-wait, generation, and retry-delay measurements with clear boundaries;
|
|
||||||
- classified outcomes and actual retry and repair counts; and
|
|
||||||
- provider-reported usage already received, even when a subsequent step fails.
|
|
||||||
|
|
||||||
Attempt measurements and terminal totals have explicit accounting semantics so
|
|
||||||
consumers do not add aggregates to their constituent attempts. Missing usage is
|
|
||||||
distinguished from measured zero; partial provider reporting must not become
|
|
||||||
falsely complete accounting. Do not estimate unknown usage or claim a complete
|
|
||||||
billing ledger. Preserve existing successful-result usage semantics and use a
|
|
||||||
presence-aware observation representation rather than silently redefining zero.
|
|
||||||
|
|
||||||
Observation payloads contain metadata, not credentials, request/response bodies,
|
|
||||||
raw provider messages, or arbitrary collaborator error strings. Metadata itself
|
|
||||||
may be sensitive and remains subject to application handling. Content capture
|
|
||||||
is not part of this scope. Public contracts must state event ownership, delivery
|
|
||||||
ordering, concurrency, and callback behavior without exposing internal types.
|
|
||||||
|
|
||||||
### Representative Consumer Evidence
|
|
||||||
|
|
||||||
Add a compact maintained offline example of a structured proposal followed by
|
|
||||||
an independent LLM assessment. Use embedded prompts/schemas, application-owned
|
|
||||||
profiles, synthetic content, a shared finite backend, and collected observations.
|
|
||||||
Use a controlled HTTP boundary when demonstrating built-in transient retry
|
|
||||||
behavior; injecting an already-retrying model client would not prove that path.
|
|
||||||
|
|
||||||
Show that schema validity does not imply domain acceptance. Keep transcript
|
|
||||||
processing, semantic acceptance, reports, and retained artifacts application-owned.
|
|
||||||
Do not turn the example into a miniature Audita pipeline or a failure-case menu.
|
|
||||||
|
|
||||||
Focused tests, rather than one oversized example, establish recovery, exhaustion,
|
|
||||||
disablement, retry timing, cancellation during capacity/backoff/transport, repair
|
|
||||||
interaction, peak concurrency, failure accounting, usage presence, and redaction.
|
|
||||||
Use deterministic clocks/waits or controlled transports where needed; no paid
|
|
||||||
calls, real credentials, timing-sensitive sleeps, or duplicated test matrices.
|
|
||||||
|
|
||||||
## Architecture And API Boundaries
|
|
||||||
|
|
||||||
Keep execution machinery under `internal/`; the root package remains a facade
|
|
||||||
for intentional consumer contracts. Maintain one retry-policy owner and one
|
|
||||||
capacity manager. Keep provider wire decoding in its adapter, lifecycle
|
|
||||||
coordination in the execution layer, and application reporting downstream.
|
|
||||||
|
|
||||||
Do not add a second run API solely for diagnostics. Preserve existing operation
|
|
||||||
signatures and optional-observation behavior for callers that do not use it.
|
|
||||||
Avoid public clock/sleeper abstractions introduced only for internal testing.
|
|
||||||
No profile/backend catalog schema changes are assumed by this roadmap.
|
|
||||||
|
|
||||||
## Documentation And Compatibility
|
|
||||||
|
|
||||||
Default retries change request counts, latency, and potential cost. Target a
|
|
||||||
minor release with explicit migration guidance, including how to retain
|
|
||||||
single-attempt behavior and avoid duplicate retries. Canonical GoDoc owns exact
|
|
||||||
API and error semantics; integration docs own HTTP retry behavior; consumer docs
|
|
||||||
own adoption examples. Update internal documentation and maintained-example
|
|
||||||
validation only where responsibilities actually change. Release notes summarize
|
|
||||||
and link rather than become a second contract owner.
|
|
||||||
|
|
||||||
## Non-Goals
|
|
||||||
|
|
||||||
No application pipeline, domain validator, cross-process scheduler, new task
|
|
||||||
queue, monetary budget, rate limiter, circuit breaker, automatic backend/model
|
|
||||||
failover, streaming/tool support, telemetry dependency, persistent run store,
|
|
||||||
or additional content-retention facility. Existing fallback-profile behavior
|
|
||||||
is not repurposed as runtime provider failover. Licensing is already complete
|
|
||||||
in source and is not a further implementation dependency.
|
|
||||||
@@ -1,110 +0,0 @@
|
|||||||
# Reliable Execution Review Findings
|
|
||||||
|
|
||||||
Review baseline: `cef8524`, following the completed
|
|
||||||
[implementation plan](implementation.md) and the
|
|
||||||
[reliable execution roadmap](reliable-execution.md).
|
|
||||||
|
|
||||||
The intended architecture is substantially implemented: the public facade
|
|
||||||
adapts contracts, the runner owns execution and repair, one generation
|
|
||||||
coordinator owns retries and active permits, the HTTP adapter freezes requests
|
|
||||||
and decodes responses, and an invocation-local recorder produces observations.
|
|
||||||
The two correctness gaps below have been remediated. This document preserves
|
|
||||||
the review evidence and resolution, not a new public contract.
|
|
||||||
|
|
||||||
## 1. Isolate Observation State Across Public Invocations (P2)
|
|
||||||
|
|
||||||
**Status:** Resolved.
|
|
||||||
|
|
||||||
**Location:** [Engine.Run](../../engine.go);
|
|
||||||
[Runner.Run](../../internal/usecase/runner.go);
|
|
||||||
[recorder context propagation](../../internal/observation/recorder.go).
|
|
||||||
|
|
||||||
The observer is request-scoped, but its recorder is attached to the context
|
|
||||||
passed to injected collaborators. If an injected client delegates to another
|
|
||||||
engine using that context and a request without an observer, `Engine.Run`
|
|
||||||
passes the inherited context through unchanged. The nested runner discovers
|
|
||||||
the outer recorder and treats it as its own.
|
|
||||||
|
|
||||||
A focused public-API reproduction with two engines and one nested injected
|
|
||||||
client produced identical inner and outer result RunIDs, four outer events
|
|
||||||
instead of three, and two outer attempts instead of one. The inner call had no
|
|
||||||
observer. Nested selection and usage also flow into the shared recorder;
|
|
||||||
parallel delegated calls could concurrently mutate state intended to be local
|
|
||||||
to one sequential invocation. This is not a request to add nested tracing:
|
|
||||||
injected-client work is intentionally opaque to its caller's accounting.
|
|
||||||
|
|
||||||
**Remediation direction:** Establish an observation-context boundary for each
|
|
||||||
public operation, retaining caller cancellation, deadlines, and unrelated
|
|
||||||
context values while preventing inherited Promptkit recorder/claim state from
|
|
||||||
being reused. Install only that invocation's own observation state when
|
|
||||||
appropriate. Check `RunPrepared` without a retained observer as well as
|
|
||||||
preparation operations, which must not mutate an enclosing run's selected
|
|
||||||
identities. Do not prohibit delegation or add a global recorder/locking layer.
|
|
||||||
|
|
||||||
**Regression evidence required:** Exercise delegation through the public facade
|
|
||||||
with the received context. Separate calls must have distinct result identities;
|
|
||||||
an unobserved child must not emit to or inflate its parent's observations; a
|
|
||||||
child with its own observer must have its own lifecycle. Include focused
|
|
||||||
prepared/preparation coverage where the same context plumbing is used, and
|
|
||||||
race-enabled coverage for concurrent delegation without sharing recorder state.
|
|
||||||
|
|
||||||
**Resolution:** Preparation and execution facade entry points mask inherited
|
|
||||||
recorder/claim state before installing any invocation-owned state. The context
|
|
||||||
boundary preserves caller values, deadlines, cancellation, and cancellation
|
|
||||||
causes. Public regression tests cover observed and unobserved delegated ordinary
|
|
||||||
and prepared runs, preparation isolation, independent accounting/identities,
|
|
||||||
and concurrent delegation. A focused recorder test checks context preservation
|
|
||||||
and claim-sink isolation.
|
|
||||||
|
|
||||||
## 2. Preserve Injected Failure Categories In Attempt Events (P2)
|
|
||||||
|
|
||||||
**Status:** Resolved.
|
|
||||||
|
|
||||||
**Location:** [public client adapter](../../llm_adapter.go);
|
|
||||||
[attempt classification](../../internal/generation/coordinator.go);
|
|
||||||
[terminal classification](../../observations.go).
|
|
||||||
|
|
||||||
The public adapter returns public error sentinels, including
|
|
||||||
`ErrMalformedResponse` for a successful nil response. The coordinator recognizes
|
|
||||||
only internal LLM sentinels. As a result, attempt observations report the broad
|
|
||||||
`generation` category while the terminal observation reports the specific
|
|
||||||
`malformed_response` or `transport` category for the very same failure.
|
|
||||||
|
|
||||||
Focused public-API reproductions confirmed all three cases: injected `(nil,
|
|
||||||
nil)`, an explicit public `ErrMalformedResponse`, and an explicit public
|
|
||||||
`ErrTransport`. Each emitted `generation` on the attempt and the expected
|
|
||||||
specific category on the terminal event. Existing repair-failure tests check
|
|
||||||
the terminal category but do not assert the failing attempt's category.
|
|
||||||
|
|
||||||
**Remediation direction:** Normalize deliberately supplied public failure
|
|
||||||
identities at the facade adapter boundary into metadata or internal identities
|
|
||||||
the coordinator understands, preserving the original error for `errors.Is` and
|
|
||||||
`errors.As`. Classify nil success consistently through the same boundary.
|
|
||||||
Internal packages must not import the public package. Do not infer categories
|
|
||||||
from arbitrary error text or enable retries for injected clients.
|
|
||||||
|
|
||||||
**Regression evidence required:** A compact public-boundary table covering nil
|
|
||||||
success, wrapped public malformed/transport sentinels, and an arbitrary injected
|
|
||||||
error should assert both attempt and terminal classifications, preserved error
|
|
||||||
identity, and single dispatch. Retain context-first classification for wrapped
|
|
||||||
cancellation/deadline causes.
|
|
||||||
|
|
||||||
**Resolution:** The facade adapter adds internal classification identities for
|
|
||||||
explicit public transport/malformed errors, retaining the original diagnostic
|
|
||||||
and error chain. Nil success uses the same normalization path. Public tests
|
|
||||||
check attempt/terminal agreement, `errors.Is`/`errors.As`, single dispatch even
|
|
||||||
with a positive retry budget, arbitrary errors, and context-first classification.
|
|
||||||
|
|
||||||
## Validation And Review Outcome
|
|
||||||
|
|
||||||
- `./scripts/release-check.sh` passed on the review baseline and remediated tree, including
|
|
||||||
race-enabled tests, vet, the three offline examples, formatting, documentation
|
|
||||||
links, and repository hygiene checks.
|
|
||||||
- Temporary public-API probes reproduced both findings during review. They were
|
|
||||||
removed and replaced by focused permanent regression coverage during remediation.
|
|
||||||
- The existing tests provide useful coverage of retries, repair interaction,
|
|
||||||
capacity, prepared claims, usage presence, and redaction, but did not protect
|
|
||||||
these two observation-boundary cases before remediation. The new
|
|
||||||
[boundary tests](../../observation_boundary_contract_test.go) protect them.
|
|
||||||
- No other must-fix issue was identified in this review. This does not establish
|
|
||||||
live-provider behavior; all verification was offline.
|
|
||||||
Reference in New Issue
Block a user