114 lines
5.5 KiB
Markdown
114 lines
5.5 KiB
Markdown
# Internal Capacity Management
|
|
|
|
## Purpose
|
|
|
|
This document describes the implemented engine-local capacity coordination in
|
|
`internal/capacity`. The [architecture policy](../policy/architecture.md) owns
|
|
component boundaries, the [backend GoDoc](../../backends.go) owns exact public
|
|
configuration semantics, and the
|
|
[internal runner document](runner.md) owns orchestration around admission.
|
|
|
|
Capacity scheduling is outside the provider wire contract. It does not add
|
|
fields to execution targets, generated requests, prompt or profile YAML, or
|
|
stable JSON values.
|
|
|
|
## Construction And Pool Lifecycle
|
|
|
|
Each root `NewEngine` call obtains a normalized capacity-policy snapshot from
|
|
its immutable backend registry and constructs a new `Manager`. The manager
|
|
creates one pool for each limited backend ID. It has no package-global mutable
|
|
state, background workers, shutdown protocol, or persistence, so engines with
|
|
the same registrations still have independent capacity.
|
|
|
|
Unlimited registered backends and endpoint-only profiles have no pool. Their
|
|
admission and generation calls take the unrestricted fast path. An endpoint
|
|
override does not change the selected backend ID and therefore does not change
|
|
the pool.
|
|
|
|
One pool owns immutable active and total limits plus mutex-protected admission
|
|
count, active count, and ordered waiter list. Pool state exists only for the
|
|
lifetime of its engine.
|
|
|
|
## Bounded Execution Admission
|
|
|
|
For ordinary `Run`, the runner asks the manager to admit after resolving the
|
|
prompt, profile, selected backend, effective execution target, credentials, and
|
|
output contract, but before schema loading, artifact loading, or rendering.
|
|
`PrepareExecution` performs no admission. `RunPrepared` claims its handle,
|
|
rechecks credential availability, and then asks the manager to admit the
|
|
frozen backend before generation.
|
|
|
|
Admission is immediate: a limited pool either reserves a slot or returns the
|
|
internal `ErrCapacityExceeded` identity. The root facade maps that identity to
|
|
the public error without treating it as an invalid request or generation
|
|
failure.
|
|
|
|
The total admitted bound is the active-generation limit plus its configured
|
|
waiting capacity. The returned release function is idempotent. The runner
|
|
defers it as soon as admission succeeds. An ordinary run holds the lease across
|
|
remaining preparation, initial generation, validation, every repair attempt,
|
|
and all failure or cancellation exits. Prepared execution holds the normal
|
|
lease across generation, validation, every internal repair attempt, and all
|
|
execution exits. A repair is part of its original admission and does not
|
|
reserve another bounded slot.
|
|
|
|
## FIFO Generation Permits
|
|
|
|
`NewClient` wraps the engine's selected internal model client after public
|
|
client adaptation or built-in client construction. Initial generation and the
|
|
default repairer receive the same wrapper.
|
|
|
|
For each `Generate` call, the wrapper selects a pool from the request's
|
|
effective backend ID. An unlimited call passes directly to the next client. A
|
|
limited call acquires an active permit, invokes the next client, and defers
|
|
permit release so ordinary returns and panic unwinding both restore capacity.
|
|
Preparation and validation never hold an active permit.
|
|
|
|
When all active permits are occupied, calls join a mutex-protected FIFO waiter
|
|
list. Releasing a permit transfers it directly to the oldest remaining waiter
|
|
before making it generally available. Pools do not order work relative to
|
|
other backend IDs.
|
|
|
|
The wrapper passes generation requests, responses, and collaborator errors
|
|
through unchanged. It owns scheduling only; the concrete model client remains
|
|
responsible for provider transport behavior.
|
|
|
|
## Cancellation And Release
|
|
|
|
Admission checks the caller context before reserving a slot. A call canceled
|
|
while waiting for an active permit removes its waiter under the same pool lock
|
|
used to grant permits. If cancellation removes the waiter first, the wrapped
|
|
client is not invoked. If a concurrent grant wins first, the call owns the
|
|
permit and invokes the client with the original context, allowing the client
|
|
to observe cancellation normally.
|
|
|
|
This grant-or-cancel decision prevents lost and double-released permits.
|
|
Admission leases and active permits are released after success, collaborator
|
|
errors, validation failures, cancellation, and panic unwinding. Canceled
|
|
waiters are unlinked so their contexts and requests are not retained by the
|
|
pool.
|
|
|
|
## Test Ownership
|
|
|
|
The [manager tests](../../internal/capacity/manager_test.go) own policy
|
|
validation, bounded admission, idempotent release, context handling, and
|
|
unlimited admission. The
|
|
[client tests](../../internal/capacity/client_test.go) own peak enforcement,
|
|
FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools,
|
|
unlimited calls, passthrough behavior, and panic release.
|
|
|
|
The [runner tests](../../internal/usecase/runner_test.go) own ordinary early
|
|
admission, lease lifetime, failure release, and shared initial/repair
|
|
scheduling. The
|
|
[prepared-execution use-case tests](../../internal/usecase/prepared_execution_test.go)
|
|
own deferred admission, credential ordering, and prepared-execution lease
|
|
release. The
|
|
[external package capacity tests](../../capacity_contract_test.go) own the
|
|
assembled public-engine behavior for configured limits, capacity errors,
|
|
endpoint identity, engine independence, and injected clients. The
|
|
[prepared-execution contract tests](../../prepared_execution_contract_test.go)
|
|
own the public prepared-capacity boundary. The
|
|
[root error-boundary tests](../../errors_internal_test.go) own preservation of
|
|
the public generation category and context identity when generation is
|
|
canceled.
|