Coordinate retry attempts with backend permits
This commit is contained in:
27
docs/internal/generation.md
Normal file
27
docs/internal/generation.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# Internal Generation Coordination
|
||||
|
||||
`internal/generation` coordinates one model-generation request through the
|
||||
existing engine-local [capacity manager](capacity.md) and the
|
||||
[built-in HTTP client](llm.md). The current engine still uses the existing
|
||||
capacity wrapper; engine assembly moves to this coordinator when public retry
|
||||
configuration is wired.
|
||||
|
||||
The coordinator has explicit built-in and injected-client constructors. The
|
||||
built-in path prepares the provider request once, then acquires and releases an
|
||||
active backend permit around each actual HTTP attempt. Only HTTP 429, 500, 502,
|
||||
503, and 504 can lead to another attempt. The injected-client path acquires a
|
||||
permit and invokes its client once, regardless of the internal retry budget.
|
||||
The runner continues to own the logical execution's admission lease.
|
||||
|
||||
Retry waits hold no active permit. The local delay uses equal jitter between
|
||||
half and all of a capped exponential base, beginning at 500 milliseconds and
|
||||
capping at 8 seconds. A valid provider not-before time can extend that delay;
|
||||
the caller context interrupts it. The next attempt reacquires the ordinary
|
||||
FIFO permit. Preparation failures dispatch nothing. When a positive retry
|
||||
budget ends with eligible failures, the internal exhaustion error retains the
|
||||
final cause and actual dispatched-attempt count.
|
||||
|
||||
Focused coordinator tests use controlled transport and wait functions. They
|
||||
cover eligibility, exhaustion, cancellation, backoff, request freezing, permit
|
||||
release, and the injected-client path. The capacity manager tests own permit
|
||||
acquisition and idempotent release.
|
||||
Reference in New Issue
Block a user