4.8 KiB
Internal Capacity Management
Purpose
This document describes the implemented engine-local capacity coordination in
internal/capacity. The architecture policy owns
component boundaries, the backend GoDoc owns exact public
configuration semantics, and the
internal runner document owns orchestration around admission.
Capacity scheduling is outside the provider wire contract. It does not add fields to execution targets, generated requests, prompt or profile YAML, or stable JSON values.
Construction And Pool Lifecycle
Each root NewEngine call obtains a normalized capacity-policy snapshot from
its immutable backend registry and constructs a new Manager. The manager
creates one pool for each limited backend ID. It has no package-global mutable
state, background workers, shutdown protocol, or persistence, so engines with
the same registrations still have independent capacity.
Unlimited registered backends and endpoint-only profiles have no pool. Their admission and generation calls take the unrestricted fast path. An endpoint override does not change the selected backend ID and therefore does not change the pool.
One pool owns immutable active and total limits plus mutex-protected admission count, active count, and ordered waiter list. Pool state exists only for the lifetime of its engine.
Bounded Run Admission
The runner asks the manager to admit a run after resolving the prompt, profile,
selected backend, effective execution target, credentials, and output contract,
but before schema loading, artifact loading, or rendering. Admission is
immediate: a limited pool either reserves a slot or returns the internal
ErrCapacityExceeded identity. The root facade maps that identity to the
public error without treating it as an invalid request or generation failure.
The total admitted bound is the active-generation limit plus its configured waiting capacity. The returned release function is idempotent. The runner defers it as soon as admission succeeds and holds the lease across remaining preparation, initial generation, validation, every repair attempt, and all failure or cancellation exits. A repair is part of its original admission and does not reserve another bounded slot.
FIFO Generation Permits
NewClient wraps the engine's selected internal model client after public
client adaptation or built-in client construction. Initial generation and the
default repairer receive the same wrapper.
For each Generate call, the wrapper selects a pool from the request's
effective backend ID. An unlimited call passes directly to the next client. A
limited call acquires an active permit, invokes the next client, and defers
permit release so ordinary returns and panic unwinding both restore capacity.
Preparation and validation never hold an active permit.
When all active permits are occupied, calls join a mutex-protected FIFO waiter list. Releasing a permit transfers it directly to the oldest remaining waiter before making it generally available. Pools do not order work relative to other backend IDs.
The wrapper passes generation requests, responses, and collaborator errors through unchanged. It owns scheduling only; the concrete model client remains responsible for provider transport behavior.
Cancellation And Release
Admission checks the caller context before reserving a slot. A call canceled while waiting for an active permit removes its waiter under the same pool lock used to grant permits. If cancellation removes the waiter first, the wrapped client is not invoked. If a concurrent grant wins first, the call owns the permit and invokes the client with the original context, allowing the client to observe cancellation normally.
This grant-or-cancel decision prevents lost and double-released permits. Admission leases and active permits are released after success, collaborator errors, validation failures, cancellation, and panic unwinding. Canceled waiters are unlinked so their contexts and requests are not retained by the pool.
Test Ownership
The manager tests own policy validation, bounded admission, idempotent release, context handling, and unlimited admission. The client tests own peak enforcement, FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools, unlimited calls, passthrough behavior, and panic release.
The runner tests own early admission, lease lifetime, failure release, and shared initial/repair scheduling. The external package capacity tests own the assembled public-engine behavior for configured limits, capacity errors, endpoint identity, engine independence, cancellation, and injected clients.