Document backend capacity management

docs/internal/capacity.md

@@ -0,0 +1,97 @@
+# Internal Capacity Management
+
+## Purpose
+
+This document describes the implemented engine-local capacity coordination in
+`internal/capacity`. The [architecture policy](../policy/architecture.md) owns
+component boundaries, the [backend GoDoc](../../backends.go) owns exact public
+configuration semantics, and the
+[internal runner document](runner.md) owns orchestration around admission.
+
+Capacity scheduling is outside the provider wire contract. It does not add
+fields to execution targets, generated requests, prompt or profile YAML, or
+stable JSON values.
+
+## Construction And Pool Lifecycle
+
+Each root `NewEngine` call obtains a normalized capacity-policy snapshot from
+its immutable backend registry and constructs a new `Manager`. The manager
+creates one pool for each limited backend ID. It has no package-global mutable
+state, background workers, shutdown protocol, or persistence, so engines with
+the same registrations still have independent capacity.
+
+Unlimited registered backends and endpoint-only profiles have no pool. Their
+admission and generation calls take the unrestricted fast path. An endpoint
+override does not change the selected backend ID and therefore does not change
+the pool.
+
+One pool owns immutable active and total limits plus mutex-protected admission
+count, active count, and ordered waiter list. Pool state exists only for the
+lifetime of its engine.
+
+## Bounded Run Admission
+
+The runner asks the manager to admit a run after resolving the prompt, profile,
+selected backend, effective execution target, credentials, and output contract,
+but before schema loading, artifact loading, or rendering. Admission is
+immediate: a limited pool either reserves a slot or returns the internal
+`ErrCapacityExceeded` identity. The root facade maps that identity to the
+public error without treating it as an invalid request or generation failure.
+
+The total admitted bound is the active-generation limit plus its configured
+waiting capacity. The returned release function is idempotent. The runner
+defers it as soon as admission succeeds and holds the lease across remaining
+preparation, initial generation, validation, every repair attempt, and all
+failure or cancellation exits. A repair is part of its original admission and
+does not reserve another bounded slot.
+
+## FIFO Generation Permits
+
+`NewClient` wraps the engine's selected internal model client after public
+client adaptation or built-in client construction. Initial generation and the
+default repairer receive the same wrapper.
+
+For each `Generate` call, the wrapper selects a pool from the request's
+effective backend ID. An unlimited call passes directly to the next client. A
+limited call acquires an active permit, invokes the next client, and defers
+permit release so ordinary returns and panic unwinding both restore capacity.
+Preparation and validation never hold an active permit.
+
+When all active permits are occupied, calls join a mutex-protected FIFO waiter
+list. Releasing a permit transfers it directly to the oldest remaining waiter
+before making it generally available. Pools do not order work relative to
+other backend IDs.
+
+The wrapper passes generation requests, responses, and collaborator errors
+through unchanged. It owns scheduling only; the concrete model client remains
+responsible for provider transport behavior.
+
+## Cancellation And Release
+
+Admission checks the caller context before reserving a slot. A call canceled
+while waiting for an active permit removes its waiter under the same pool lock
+used to grant permits. If cancellation removes the waiter first, the wrapped
+client is not invoked. If a concurrent grant wins first, the call owns the
+permit and invokes the client with the original context, allowing the client
+to observe cancellation normally.
+
+This grant-or-cancel decision prevents lost and double-released permits.
+Admission leases and active permits are released after success, collaborator
+errors, validation failures, cancellation, and panic unwinding. Canceled
+waiters are unlinked so their contexts and requests are not retained by the
+pool.
+
+## Test Ownership
+
+The [manager tests](../../internal/capacity/manager_test.go) own policy
+validation, bounded admission, idempotent release, context handling, and
+unlimited admission. The
+[client tests](../../internal/capacity/client_test.go) own peak enforcement,
+FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools,
+unlimited calls, passthrough behavior, and panic release.
+
+The [runner tests](../../internal/usecase/runner_test.go) own early admission,
+lease lifetime, failure release, and shared initial/repair scheduling. The
+[external package capacity tests](../../capacity_contract_test.go) own the
+assembled public-engine behavior for configured limits, capacity errors,
+endpoint identity, engine independence, cancellation, and injected clients.

docs/internal/overview.md

@@ -11,10 +11,11 @@ contributor workflow and validation.
 
 | Component | Implemented responsibility | References |
 | --- | --- | --- |
-| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
+| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
 | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
 | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
 | `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
+| `internal/capacity` | Owns engine-local bounded run admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
 | `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
 | `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
 | `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |

docs/internal/runner.md

@@ -19,17 +19,20 @@ and override semantics consumed by the runner.
 `Runner` coordinates narrow internal interfaces for prompt definitions,
 profiles, backend resolution, artifacts, rendering, model generation, and
 validation. The root engine supplies one immutable registry containing the
-built-in backend and validated consumer additions. Schema documents are loaded
-through the validator's optional schema-loader interface. An output repairer
-can be injected internally, but the ordinary runner constructor does not
-enable one.
+built-in backend and validated consumer additions, one engine-local run
+admitter, and a model client wrapped by the same capacity manager. Schema
+documents are loaded through the validator's optional schema-loader interface.
+An output repairer can be injected internally, but the ordinary runner
+constructor does not enable one.
 
 Each invocation carries its state in request, prepared-run, and result values.
 The runner has no durable run or session store.
 
-## Preparation Flow
+## Shared Preparation Pipeline
 
-`Prepare` performs the reusable pre-generation workflow:
+`Prepare` and `Run` share one private preparation pipeline split at the point
+where a run can be assigned to its selected backend pool. The resolution phase
+performs only the work needed to validate routing and admission:
 
 1. validate the required prompt selection and normalize any direct session ID;
 2. load the prompt definition and hash the original definition;
@@ -38,13 +41,25 @@ The runner has no durable run or session store.
 5. resolve application-neutral defaults, backend defaults, profile values,
    and explicit request overrides in that order;
 6. validate endpoint, model, numeric overrides, and credential requirements;
-7. resolve the output contract and load a structured-output schema when
-   required;
-8. load and hash input artifacts;
-9. render messages, resolve the effective session ID, and hash the effective
-   rendered prompt; and
-10. return the effective settings, source identities, messages, hashes, and
-   preparation timing.
+7. resolve the effective output contract without loading its schema; and
+8. retain the definition, source identities, effective settings, output
+   contract, and preparation start time in invocation-local state.
+
+The completion phase consumes that state without reloading the prompt,
+profile, or backend:
+
+1. load structured-output schema metadata when required;
+2. load and hash input artifacts;
+3. render messages and the prompt-defined session;
+4. apply any direct session ID;
+5. hash the effective rendered prompt; and
+6. construct the prepared value and preparation timing.
+
+`Prepare` runs both phases consecutively and never performs capacity admission.
+`Run` performs backend admission between the phases. This structure preserves
+one execution-precedence and error-ordering implementation while allowing a
+full backend pool to reject work before expensive schema, artifact, and
+rendering operations.
 
 Pointer-based numeric overrides preserve an explicit zero. Invalid negative or
 out-of-range values fail as invalid requests. Endpoint overrides do not change
@@ -70,15 +85,28 @@ invocation state local.
 
 ## Run Flow
 
-`Run` calls `Prepare` rather than maintaining a second preparation path. It
-performs one initial generation call, builds the named output artifact, and
-validates that artifact. Invalid generated content remains a validation result;
-an inability to perform validation is an operational error.
+`Run` records its start time, performs the shared resolution phase, and asks
+its `RunAdmitter` to reserve capacity for the effective backend ID. A nil
+admitter is an internal unlimited fallback. After successful admission, `Run`
+immediately defers the returned release function, performs the completion
+phase, makes one initial generation call, builds the named output artifact,
+and validates that artifact. Invalid generated content remains a validation
+result; an inability to perform validation is an operational error.
+
+The admission lease covers completion-phase preparation, initial generation,
+validation, every repair, and every exit. It bounds accepted work without
+serializing preparation or validation behind the active-generation limit.
+The wrapped model client separately acquires a FIFO active permit only around
+each actual generation call.
 
 When an internal repairer is present, a JSON or JSON Schema content failure can
 trigger bounded repair attempts. Repair receives the effective execution
 target and session ID, validation errors, prior output, and structured-output
-specification. This capability remains internal and is not a public option.
+specification. The default repairer uses the same wrapped client as initial
+generation, so each repair reacquires the selected backend's active permit
+while remaining inside its original admission lease. Repair never performs a
+second bounded admission. This capability remains internal and is not a public
+option.
 
 A successful result includes the output artifact and raw output, validation
 state, effective session ID, prompt and rendered-prompt hashes, selected
@@ -94,8 +122,20 @@ Package errors distinguish invalid requests, required profile selection,
 credential failures, and prompt, profile, artifact, rendering, generation, and
 validation failures. Wrapping preserves the package identities mapped by the
 public facade and retains collaborator identities where they are part of the
-internal contract. Context cancellation propagates through the invoked
-collaborator and is classified by the owning operation.
+internal contract.
+
+Admission capacity exhaustion retains the internal capacity identity and adds
+the selected backend ID as context. It is not recategorized as an invalid
+request or generation failure, and no partial result is returned. A context
+already done at admission retains its context identity directly. Cancellation
+while waiting for an active generation permit prevents client invocation when
+it wins the grant race; the model-client boundary then preserves the context
+error through the generation-failure category. Deferred release restores the
+admission lease on preparation, generation, validation, repair, and
+cancellation failures.
+
+Other context cancellation propagates through the invoked collaborator and is
+classified by the owning operation.
 An overlong direct session is an invalid request before source loading, while
 an invalid or overlong prompt session template remains a prompt-render failure.
 An unknown selected backend, or a selected backend with no configured resolver,
@@ -104,12 +144,16 @@ is classified as a profile-load failure.
 ## Test Ownership And Changes
 
 The [runner tests](../../internal/usecase/runner_test.go) own preparation order,
-selection and override precedence, direct-session resolution,
-schema-before-generation behavior, hashing, generation and validation
-outcomes, backend propagation, bounded repair, credentials and redaction,
-error categories, artifact metadata, usage, and timing.
+selection and override precedence, the two-phase boundary, early admission,
+lease lifetime and release, direct-session resolution, schema-before-generation
+behavior, hashing, generation and validation outcomes, backend propagation,
+bounded repair, shared initial/repair capacity, credentials and redaction,
+error categories, artifact metadata, usage, and timing. The
+[capacity subsystem document](capacity.md) identifies the focused pool,
+waiter, and wrapped-client tests.
 
 Changes to orchestration should continue to use the existing package
-interfaces, keep request state local to an invocation, and preserve `Run`'s use
-of `Prepare`. Source, renderer, validator, or model-client contract changes
-belong first in their owning package and document.
+interfaces, keep request state local to an invocation, and preserve the shared
+resolution and completion pipeline. Source, renderer, validator, or
+model-client contract changes belong first in their owning package and
+document.