145 lines
7.5 KiB
Markdown
145 lines
7.5 KiB
Markdown
# Internal Capacity Management
|
|
|
|
## Purpose
|
|
|
|
This document describes the implemented engine-local capacity coordination in
|
|
`internal/capacity`. The [architecture policy](../policy/architecture.md) owns
|
|
component boundaries, the [backend GoDoc](../../backends.go) owns exact public
|
|
configuration semantics, and the
|
|
[internal runner document](runner.md) owns orchestration around admission.
|
|
|
|
Capacity scheduling is outside the provider wire contract. It does not add
|
|
fields to execution targets, generated requests, prompt or profile YAML, or
|
|
stable JSON values.
|
|
|
|
## Construction And Pool Lifecycle
|
|
|
|
Each internal engine construction obtains a normalized capacity-policy snapshot from
|
|
its immutable backend registry and constructs a new `Manager`. The manager
|
|
creates one pool for each limited backend ID. It has no package-global mutable
|
|
state, background workers, shutdown protocol, or persistence, so engines with
|
|
the same registrations still have independent capacity.
|
|
|
|
Unlimited registered backends and endpoint-only profiles have no pool. Their
|
|
admission and generation calls take the unrestricted fast path. An endpoint
|
|
override does not change the selected backend ID and therefore does not change
|
|
the pool.
|
|
|
|
One pool owns immutable active and total limits plus mutex-protected admission
|
|
count, active count, and ordered waiter list. Pool state exists only for the
|
|
lifetime of its engine.
|
|
|
|
## Bounded Execution Admission
|
|
|
|
For ordinary `Run`, the runner asks the manager to admit after resolving the
|
|
prompt, profile, selected backend, effective execution target, credentials, and
|
|
output contract, but before schema loading, artifact loading, or rendering.
|
|
`PrepareExecution` performs no admission. `RunPrepared` claims its handle,
|
|
rechecks credential availability, and then asks the manager to admit the
|
|
frozen backend before generation.
|
|
|
|
Admission is immediate: a limited pool either reserves a slot or returns only
|
|
the internal `ErrCapacityExceeded` identity. The runner attaches the selected
|
|
backend identity at its use-case boundary, and the root facade translates that
|
|
typed value without treating it as an invalid request or generation failure.
|
|
|
|
The total admitted bound is the active-generation limit plus its configured
|
|
waiting capacity. The returned release function is idempotent. The runner
|
|
defers it as soon as admission succeeds. An ordinary run holds the lease across
|
|
remaining preparation, initial generation, validation, every repair attempt,
|
|
and all failure or cancellation exits. Prepared execution holds the normal
|
|
lease across generation, validation, every internal repair attempt, and all
|
|
execution exits. A repair is part of its original admission and does not
|
|
reserve another bounded slot.
|
|
|
|
## FIFO Generation Permits
|
|
|
|
The engine supplies one [generation coordinator](generation.md) to initial
|
|
generation and the default repairer. Their requests retain the same effective
|
|
backend, credential, numeric-presence metadata, structured-output settings,
|
|
and retry budget. Scheduling does not change provider omission semantics
|
|
between calls.
|
|
|
|
For each model attempt, the coordinator selects a pool from the request's
|
|
effective backend ID. An unlimited call uses the unrestricted path. A limited
|
|
call acquires an active permit and defers release around that attempt, so
|
|
ordinary returns and panic unwinding both restore capacity.
|
|
Preparation and validation never hold an active permit.
|
|
|
|
The manager's `Acquire` operation owns that active-permit acquisition and
|
|
returns an idempotent release function. The coordinator uses it around
|
|
individual attempts. The earlier one-shot wrapper still delegates to the same
|
|
operation for focused internal callers and tests, but engine assembly no
|
|
longer uses it. There is one queue and one active limit per limited backend pool.
|
|
|
|
When all active permits are occupied, calls join a mutex-protected FIFO waiter
|
|
list. Releasing a permit transfers it directly to the oldest remaining waiter
|
|
before making it generally available. Pools do not order work relative to
|
|
other backend IDs.
|
|
|
|
The coordinator does not retry injected clients; it normalizes a nil successful
|
|
response as malformed generation. The built-in HTTP client remains responsible
|
|
for provider transport behavior. The runner, rather than the
|
|
capacity layer, sums all five usage fields from the initial response and every
|
|
completed repair response into the successful run result.
|
|
|
|
An invocation-local observer receives attempt events after active permit
|
|
release, including canceled acquisition without dispatch. A retry wait holds
|
|
the logical admission lease but no active permit. Observer callbacks must not
|
|
block on work that requires the same run's admission slot.
|
|
|
|
## Smoke-Test Capacity
|
|
|
|
The engine's smoke tester shares this manager and generation coordinator with
|
|
the ordinary runner. Inference diagnostics reserve the same logical admission
|
|
slot and acquire the same backend-scoped active permits for actual model
|
|
attempts, including retries. They therefore participate in the same configured
|
|
backend limits as `Run`. Reachability and authentication HTTP probes do not
|
|
enter generation admission or consume active permits. The public
|
|
[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go) owns the caller-visible
|
|
operation and injected-client limits.
|
|
|
|
## Cancellation And Release
|
|
|
|
Admission checks the caller context before reserving a slot. A call canceled
|
|
while waiting for an active permit removes its waiter under the same pool lock
|
|
used to grant permits. If cancellation removes the waiter first, no model
|
|
attempt is dispatched. If a concurrent grant wins first, the call owns the
|
|
permit; the coordinator checks the context before dispatch and releases the
|
|
permit if cancellation has already occurred.
|
|
|
|
This grant-or-cancel decision prevents lost and double-released permits.
|
|
Admission leases and active permits are released after success, collaborator
|
|
errors, validation failures, cancellation, and panic unwinding. Canceled
|
|
waiters are unlinked so their contexts and requests are not retained by the
|
|
pool.
|
|
|
|
## Test Ownership
|
|
|
|
The [manager tests](../../internal/capacity/manager_test.go) own policy
|
|
validation, bounded admission, idempotent release, context handling, and
|
|
unlimited admission. The
|
|
[client tests](../../internal/capacity/client_test.go) own peak enforcement,
|
|
FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools,
|
|
unlimited calls, passthrough behavior, and panic release.
|
|
|
|
The [runner tests](../../internal/usecase/runner_test.go) own ordinary early
|
|
admission, lease lifetime, failure release, and shared initial/repair
|
|
scheduling. The
|
|
[prepared-execution use-case tests](../../internal/usecase/prepared_execution_test.go)
|
|
own deferred admission, credential ordering, and prepared-execution lease
|
|
release. The
|
|
[external package capacity tests](../../capacity_contract_test.go) own the
|
|
assembled public-engine behavior for configured limits, capacity errors,
|
|
endpoint identity, engine independence, and injected clients. The
|
|
[reliable execution contract tests](../../reliable_execution_contract_test.go)
|
|
cover shared and independent backend pools during retry and repair. The
|
|
[prepared-execution contract tests](../../prepared_execution_contract_test.go)
|
|
own the public prepared-capacity boundary. The
|
|
[root error-boundary tests](../../errors_internal_test.go) own preservation of
|
|
the public generation category and context identity when generation is
|
|
canceled. The assembled
|
|
[smoke diagnostic contract tests](../../smoke_integration_contract_test.go)
|
|
verify that inference diagnostics share the engine's Run admission and recover
|
|
capacity after cancellation or failure.
|