promptkit/docs/internal/capacity.md

Internal Capacity Management

Purpose

This document describes the implemented engine-local capacity coordination in internal/capacity. The architecture policy owns component boundaries, the backend GoDoc owns exact public configuration semantics, and the internal runner document owns orchestration around admission.

Capacity scheduling is outside the provider wire contract. It does not add fields to execution targets, generated requests, prompt or profile YAML, or stable JSON values.

Construction And Pool Lifecycle

Each internal engine construction obtains a normalized capacity-policy snapshot from its immutable backend registry and constructs a new Manager. The manager creates one pool for each limited backend ID. It has no package-global mutable state, background workers, shutdown protocol, or persistence, so engines with the same registrations still have independent capacity.

Unlimited registered backends and endpoint-only profiles have no pool. Their admission and generation calls take the unrestricted fast path. An endpoint override does not change the selected backend ID and therefore does not change the pool.

One pool owns immutable active and total limits plus mutex-protected admission count, active count, and ordered waiter list. Pool state exists only for the lifetime of its engine.

Bounded Execution Admission

For ordinary Run, the runner asks the manager to admit after resolving the prompt, profile, selected backend, effective execution target, credentials, and output contract, but before schema loading, artifact loading, or rendering. PrepareExecution performs no admission. RunPrepared claims its handle, rechecks credential availability, and then asks the manager to admit the frozen backend before generation.

Admission is immediate: a limited pool either reserves a slot or returns only the internal ErrCapacityExceeded identity. The runner attaches the selected backend identity at its use-case boundary, and the root facade translates that typed value without treating it as an invalid request or generation failure.

The total admitted bound is the active-generation limit plus its configured waiting capacity. The returned release function is idempotent. The runner defers it as soon as admission succeeds. An ordinary run holds the lease across remaining preparation, initial generation, validation, every repair attempt, and all failure or cancellation exits. Prepared execution holds the normal lease across generation, validation, every internal repair attempt, and all execution exits. A repair is part of its original admission and does not reserve another bounded slot.

FIFO Generation Permits

The engine supplies one generation coordinator to initial generation and the default repairer. Their requests retain the same effective backend, credential, numeric-presence metadata, structured-output settings, and retry budget. Scheduling does not change provider omission semantics between calls.

For each model attempt, the coordinator selects a pool from the request's effective backend ID. An unlimited call uses the unrestricted path. A limited call acquires an active permit and defers release around that attempt, so ordinary returns and panic unwinding both restore capacity. Preparation and validation never hold an active permit.

The manager's Acquire operation owns that active-permit acquisition and returns an idempotent release function. The coordinator uses it around individual attempts. The earlier one-shot wrapper still delegates to the same operation for focused internal callers and tests, but engine assembly no longer uses it. There is one queue and one active limit per limited backend pool.

When all active permits are occupied, calls join a mutex-protected FIFO waiter list. Releasing a permit transfers it directly to the oldest remaining waiter before making it generally available. Pools do not order work relative to other backend IDs.

The coordinator does not retry injected clients; it normalizes a nil successful response as malformed generation. The built-in HTTP client remains responsible for provider transport behavior. The runner, rather than the capacity layer, sums all five usage fields from the initial response and every completed repair response into the successful run result.

An invocation-local observer receives attempt events after active permit release, including canceled acquisition without dispatch. A retry wait holds the logical admission lease but no active permit. Observer callbacks must not block on work that requires the same run's admission slot.

Cancellation And Release

Admission checks the caller context before reserving a slot. A call canceled while waiting for an active permit removes its waiter under the same pool lock used to grant permits. If cancellation removes the waiter first, no model attempt is dispatched. If a concurrent grant wins first, the call owns the permit; the coordinator checks the context before dispatch and releases the permit if cancellation has already occurred.

This grant-or-cancel decision prevents lost and double-released permits. Admission leases and active permits are released after success, collaborator errors, validation failures, cancellation, and panic unwinding. Canceled waiters are unlinked so their contexts and requests are not retained by the pool.

Test Ownership

The manager tests own policy validation, bounded admission, idempotent release, context handling, and unlimited admission. The client tests own peak enforcement, FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools, unlimited calls, passthrough behavior, and panic release.

The runner tests own ordinary early admission, lease lifetime, failure release, and shared initial/repair scheduling. The prepared-execution use-case tests own deferred admission, credential ordering, and prepared-execution lease release. The external package capacity tests own the assembled public-engine behavior for configured limits, capacity errors, endpoint identity, engine independence, and injected clients. The reliable execution contract tests cover shared and independent backend pools during retry and repair. The prepared-execution contract tests own the public prepared-capacity boundary. The root error-boundary tests own preservation of the public generation category and context identity when generation is canceled.