101 lines
4.9 KiB
Markdown
101 lines
4.9 KiB
Markdown
# Internal Capacity Management
|
|
|
|
## Purpose
|
|
|
|
This document describes the implemented engine-local capacity coordination in
|
|
`internal/capacity`. The [architecture policy](../policy/architecture.md) owns
|
|
component boundaries, the [backend GoDoc](../../backends.go) owns exact public
|
|
configuration semantics, and the
|
|
[internal runner document](runner.md) owns orchestration around admission.
|
|
|
|
Capacity scheduling is outside the provider wire contract. It does not add
|
|
fields to execution targets, generated requests, prompt or profile YAML, or
|
|
stable JSON values.
|
|
|
|
## Construction And Pool Lifecycle
|
|
|
|
Each root `NewEngine` call obtains a normalized capacity-policy snapshot from
|
|
its immutable backend registry and constructs a new `Manager`. The manager
|
|
creates one pool for each limited backend ID. It has no package-global mutable
|
|
state, background workers, shutdown protocol, or persistence, so engines with
|
|
the same registrations still have independent capacity.
|
|
|
|
Unlimited registered backends and endpoint-only profiles have no pool. Their
|
|
admission and generation calls take the unrestricted fast path. An endpoint
|
|
override does not change the selected backend ID and therefore does not change
|
|
the pool.
|
|
|
|
One pool owns immutable active and total limits plus mutex-protected admission
|
|
count, active count, and ordered waiter list. Pool state exists only for the
|
|
lifetime of its engine.
|
|
|
|
## Bounded Run Admission
|
|
|
|
The runner asks the manager to admit a run after resolving the prompt, profile,
|
|
selected backend, effective execution target, credentials, and output contract,
|
|
but before schema loading, artifact loading, or rendering. Admission is
|
|
immediate: a limited pool either reserves a slot or returns the internal
|
|
`ErrCapacityExceeded` identity. The root facade maps that identity to the
|
|
public error without treating it as an invalid request or generation failure.
|
|
|
|
The total admitted bound is the active-generation limit plus its configured
|
|
waiting capacity. The returned release function is idempotent. The runner
|
|
defers it as soon as admission succeeds and holds the lease across remaining
|
|
preparation, initial generation, validation, every repair attempt, and all
|
|
failure or cancellation exits. A repair is part of its original admission and
|
|
does not reserve another bounded slot.
|
|
|
|
## FIFO Generation Permits
|
|
|
|
`NewClient` wraps the engine's selected internal model client after public
|
|
client adaptation or built-in client construction. Initial generation and the
|
|
default repairer receive the same wrapper.
|
|
|
|
For each `Generate` call, the wrapper selects a pool from the request's
|
|
effective backend ID. An unlimited call passes directly to the next client. A
|
|
limited call acquires an active permit, invokes the next client, and defers
|
|
permit release so ordinary returns and panic unwinding both restore capacity.
|
|
Preparation and validation never hold an active permit.
|
|
|
|
When all active permits are occupied, calls join a mutex-protected FIFO waiter
|
|
list. Releasing a permit transfers it directly to the oldest remaining waiter
|
|
before making it generally available. Pools do not order work relative to
|
|
other backend IDs.
|
|
|
|
The wrapper passes generation requests, responses, and collaborator errors
|
|
through unchanged. It owns scheduling only; the concrete model client remains
|
|
responsible for provider transport behavior.
|
|
|
|
## Cancellation And Release
|
|
|
|
Admission checks the caller context before reserving a slot. A call canceled
|
|
while waiting for an active permit removes its waiter under the same pool lock
|
|
used to grant permits. If cancellation removes the waiter first, the wrapped
|
|
client is not invoked. If a concurrent grant wins first, the call owns the
|
|
permit and invokes the client with the original context, allowing the client
|
|
to observe cancellation normally.
|
|
|
|
This grant-or-cancel decision prevents lost and double-released permits.
|
|
Admission leases and active permits are released after success, collaborator
|
|
errors, validation failures, cancellation, and panic unwinding. Canceled
|
|
waiters are unlinked so their contexts and requests are not retained by the
|
|
pool.
|
|
|
|
## Test Ownership
|
|
|
|
The [manager tests](../../internal/capacity/manager_test.go) own policy
|
|
validation, bounded admission, idempotent release, context handling, and
|
|
unlimited admission. The
|
|
[client tests](../../internal/capacity/client_test.go) own peak enforcement,
|
|
FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools,
|
|
unlimited calls, passthrough behavior, and panic release.
|
|
|
|
The [runner tests](../../internal/usecase/runner_test.go) own early admission,
|
|
lease lifetime, failure release, and shared initial/repair scheduling. The
|
|
[external package capacity tests](../../capacity_contract_test.go) own the
|
|
assembled public-engine behavior for configured limits, capacity errors,
|
|
endpoint identity, engine independence, and injected clients. The
|
|
[root error-boundary tests](../../errors_internal_test.go) own preservation of
|
|
the public generation category and context identity when generation is
|
|
canceled.
|