Document backend capacity management

doc.go

@@ -9,9 +9,11 @@
 //
 // # Concurrency and ownership
 //
-// An Engine supports concurrent Prepare and Run calls. An injected [LLMClient]
-// or [ArtifactReader] can therefore receive concurrent calls and must be safe
-// for that use.
+// An Engine supports concurrent Prepare and Run calls. Engine-local backend
+// policies bound admitted Run calls and model generations where configured,
+// while different backend pools and unlimited backends continue independently.
+// An injected [LLMClient] or [ArtifactReader] can therefore still receive
+// concurrent calls and must be safe for that use.
 //
 // NewEngine copies in-memory profiles and backend definitions. Prepare and Run
 // copy request maps, slices, pointer values, and JSON-compatible extra

docs/consumers/pkg-promptkit.md

@@ -127,7 +127,9 @@ exact normalization, precedence, error, copying, and exposure contract.
 ### Register A Custom Backend
 
 Register a reusable OpenAI-compatible connection once, then select it from a
-profile:
+profile. This local backend limits model generation to two simultaneous calls;
+because `QueueCapacity` is omitted, the engine admits up to 1024 additional
+calls waiting behind them:
 
 ```go
 engine, err := promptkit.NewEngine(promptkit.Config{
@@ -137,6 +139,7 @@ engine, err := promptkit.NewEngine(promptkit.Config{
 		ID:        "local",
 		Endpoint:  "http://localhost:8000/v1",
 		APIKeyEnv: "LOCAL_LLM_API_KEY",
+		ConcurrencyLimit: 2,
 	}),
 	promptkit.WithProfiles(promptkit.Profile{
 		ID:        "local-summary",
@@ -148,15 +151,42 @@ engine, err := promptkit.NewEngine(promptkit.Config{
 
 Registrations belong to one engine and custom IDs cannot replace built-ins.
 The [`Backend` and `WithBackend` GoDoc](../../backends.go) defines validation,
-copying, uniqueness, and request-default behavior.
+copying, uniqueness, exact concurrency-field semantics, and request-default
+behavior.
 
 Both file-backed and in-memory profiles select a registration through
 `backend` or `Profile.BackendID`. Profile and request endpoint overrides retain
-that routing identity. `PreparedRun.SelectedBackendID`,
+that routing and capacity identity. `PreparedRun.SelectedBackendID`,
 `RunResult.SelectedBackendID`, and the effective `ExecutionTarget.BackendID`
 expose it to consumers and injected model clients. Endpoint-only profiles
 remain supported and expose an empty backend ID.
 
+### Limit Backend Concurrency
+
+Set `Backend.ConcurrencyLimit` when a backend needs protection from too many
+simultaneous model calls. Leaving `QueueCapacity` nil, as in the local-backend
+example above, selects the default waiting capacity of 1024.
+
+To accept no waiting backlog beyond the active calls, provide an explicit
+zero:
+
+```go
+noWaiting := 0
+backend := promptkit.Backend{
+	ID:               "local",
+	Endpoint:         "http://localhost:8000/v1",
+	ConcurrencyLimit: 2,
+	QueueCapacity:    &noWaiting,
+}
+```
+
+The pointer distinguishes an explicit zero from omission. Keep using keyed
+`Backend` literals so additive configuration fields remain source-compatible.
+Capacity belongs to one engine and the selected backend ID; endpoint-only
+profiles and custom backends without a configured limit remain unrestricted.
+Exact validation, defaulting, ownership, and concurrency semantics belong to
+the [`Backend` GoDoc](../../backends.go).
+
 ## Credentials
 
 File-backed profiles name an environment variable; in-memory profiles can
@@ -200,6 +230,22 @@ failures. Specific request conditions may also match the broader
 documented. Invalid or duplicate backend registrations match
 `ErrInvalidConfig`; selecting an unknown backend matches `ErrProfileLoad`.
 
+When a limited backend has admitted all active and waiting calls, handle
+`ErrCapacityExceeded` separately from request errors and provider failures:
+
+```go
+result, err := engine.Run(ctx, request)
+if errors.Is(err, promptkit.ErrCapacityExceeded) {
+	// Apply application policy: shed work, report overload, or retry later.
+}
+```
+
+A rejected call returns no partial result and does not invoke the model
+client. Promptkit does not prescribe retries or map this error to an HTTP
+status; those choices remain with the consuming application. The
+[`Engine.Run` and error GoDoc](../../engine.go) owns exact error and
+cancellation identities.
+
 ## Application Boundary
 
 Promptkit is an importable library. It does not own a command, inbound HTTP

docs/formats.md

@@ -207,7 +207,10 @@ Non-empty profile strings replace backend defaults, and non-empty request
 strings replace both. Request reasoning is the exception: a nil
 `ReasoningEffort` pointer inherits the profile, a pointer to a nonblank string
 trims and replaces it, and a pointer to a blank string clears it. Backend
-identity is retained when either layer overrides the endpoint. A non-empty
+identity is retained when either layer overrides the endpoint, so the override
+also retains any engine-local capacity policy configured for that backend.
+Capacity configuration belongs to the Go
+[`Backend` API](../backends.go), not prompt or profile YAML. A non-empty
 `extra_params` map at each layer replaces the entire lower-precedence map
 rather than merging keys.
 The [outbound integration contract](integrations/openai-compatible-chat.md)

docs/internal/capacity.md

@@ -0,0 +1,97 @@
+# Internal Capacity Management
+
+## Purpose
+
+This document describes the implemented engine-local capacity coordination in
+`internal/capacity`. The [architecture policy](../policy/architecture.md) owns
+component boundaries, the [backend GoDoc](../../backends.go) owns exact public
+configuration semantics, and the
+[internal runner document](runner.md) owns orchestration around admission.
+
+Capacity scheduling is outside the provider wire contract. It does not add
+fields to execution targets, generated requests, prompt or profile YAML, or
+stable JSON values.
+
+## Construction And Pool Lifecycle
+
+Each root `NewEngine` call obtains a normalized capacity-policy snapshot from
+its immutable backend registry and constructs a new `Manager`. The manager
+creates one pool for each limited backend ID. It has no package-global mutable
+state, background workers, shutdown protocol, or persistence, so engines with
+the same registrations still have independent capacity.
+
+Unlimited registered backends and endpoint-only profiles have no pool. Their
+admission and generation calls take the unrestricted fast path. An endpoint
+override does not change the selected backend ID and therefore does not change
+the pool.
+
+One pool owns immutable active and total limits plus mutex-protected admission
+count, active count, and ordered waiter list. Pool state exists only for the
+lifetime of its engine.
+
+## Bounded Run Admission
+
+The runner asks the manager to admit a run after resolving the prompt, profile,
+selected backend, effective execution target, credentials, and output contract,
+but before schema loading, artifact loading, or rendering. Admission is
+immediate: a limited pool either reserves a slot or returns the internal
+`ErrCapacityExceeded` identity. The root facade maps that identity to the
+public error without treating it as an invalid request or generation failure.
+
+The total admitted bound is the active-generation limit plus its configured
+waiting capacity. The returned release function is idempotent. The runner
+defers it as soon as admission succeeds and holds the lease across remaining
+preparation, initial generation, validation, every repair attempt, and all
+failure or cancellation exits. A repair is part of its original admission and
+does not reserve another bounded slot.
+
+## FIFO Generation Permits
+
+`NewClient` wraps the engine's selected internal model client after public
+client adaptation or built-in client construction. Initial generation and the
+default repairer receive the same wrapper.
+
+For each `Generate` call, the wrapper selects a pool from the request's
+effective backend ID. An unlimited call passes directly to the next client. A
+limited call acquires an active permit, invokes the next client, and defers
+permit release so ordinary returns and panic unwinding both restore capacity.
+Preparation and validation never hold an active permit.
+
+When all active permits are occupied, calls join a mutex-protected FIFO waiter
+list. Releasing a permit transfers it directly to the oldest remaining waiter
+before making it generally available. Pools do not order work relative to
+other backend IDs.
+
+The wrapper passes generation requests, responses, and collaborator errors
+through unchanged. It owns scheduling only; the concrete model client remains
+responsible for provider transport behavior.
+
+## Cancellation And Release
+
+Admission checks the caller context before reserving a slot. A call canceled
+while waiting for an active permit removes its waiter under the same pool lock
+used to grant permits. If cancellation removes the waiter first, the wrapped
+client is not invoked. If a concurrent grant wins first, the call owns the
+permit and invokes the client with the original context, allowing the client
+to observe cancellation normally.
+
+This grant-or-cancel decision prevents lost and double-released permits.
+Admission leases and active permits are released after success, collaborator
+errors, validation failures, cancellation, and panic unwinding. Canceled
+waiters are unlinked so their contexts and requests are not retained by the
+pool.
+
+## Test Ownership
+
+The [manager tests](../../internal/capacity/manager_test.go) own policy
+validation, bounded admission, idempotent release, context handling, and
+unlimited admission. The
+[client tests](../../internal/capacity/client_test.go) own peak enforcement,
+FIFO transfer, canceled-waiter removal, grant/cancel races, independent pools,
+unlimited calls, passthrough behavior, and panic release.
+
+The [runner tests](../../internal/usecase/runner_test.go) own early admission,
+lease lifetime, failure release, and shared initial/repair scheduling. The
+[external package capacity tests](../../capacity_contract_test.go) own the
+assembled public-engine behavior for configured limits, capacity errors,
+endpoint identity, engine independence, cancellation, and injected clients.

docs/internal/overview.md

@@ -11,10 +11,11 @@ contributor workflow and validation.
 
 | Component | Implemented responsibility | References |
 | --- | --- | --- |
-| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, profile construction, extension interfaces, value conversion, redacted formatting, and public error mapping. | [Package GoDoc](../../doc.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
+| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request and result values, profile construction, extension interfaces, value conversion, redacted formatting, public error mapping, and engine-local assembly. | [Package GoDoc](../../doc.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
 | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
 | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
 | `internal/backend` | Constructs each engine's immutable registry from the built-in OpenRouter definition and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
+| `internal/capacity` | Owns engine-local bounded run admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
 | `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, profiles, execution targets, rendering, generation, and validation. | [Domain declarations](../../internal/domain/domain.go) |
 | `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
 | `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |

docs/internal/runner.md

@@ -19,17 +19,20 @@ and override semantics consumed by the runner.
 `Runner` coordinates narrow internal interfaces for prompt definitions,
 profiles, backend resolution, artifacts, rendering, model generation, and
 validation. The root engine supplies one immutable registry containing the
-built-in backend and validated consumer additions. Schema documents are loaded
-through the validator's optional schema-loader interface. An output repairer
-can be injected internally, but the ordinary runner constructor does not
-enable one.
+built-in backend and validated consumer additions, one engine-local run
+admitter, and a model client wrapped by the same capacity manager. Schema
+documents are loaded through the validator's optional schema-loader interface.
+An output repairer can be injected internally, but the ordinary runner
+constructor does not enable one.
 
 Each invocation carries its state in request, prepared-run, and result values.
 The runner has no durable run or session store.
 
-## Preparation Flow
+## Shared Preparation Pipeline
 
-`Prepare` performs the reusable pre-generation workflow:
+`Prepare` and `Run` share one private preparation pipeline split at the point
+where a run can be assigned to its selected backend pool. The resolution phase
+performs only the work needed to validate routing and admission:
 
 1. validate the required prompt selection and normalize any direct session ID;
 2. load the prompt definition and hash the original definition;
@@ -38,13 +41,25 @@ The runner has no durable run or session store.
 5. resolve application-neutral defaults, backend defaults, profile values,
    and explicit request overrides in that order;
 6. validate endpoint, model, numeric overrides, and credential requirements;
-7. resolve the output contract and load a structured-output schema when
-   required;
-8. load and hash input artifacts;
-9. render messages, resolve the effective session ID, and hash the effective
-   rendered prompt; and
-10. return the effective settings, source identities, messages, hashes, and
-   preparation timing.
+7. resolve the effective output contract without loading its schema; and
+8. retain the definition, source identities, effective settings, output
+   contract, and preparation start time in invocation-local state.
+
+The completion phase consumes that state without reloading the prompt,
+profile, or backend:
+
+1. load structured-output schema metadata when required;
+2. load and hash input artifacts;
+3. render messages and the prompt-defined session;
+4. apply any direct session ID;
+5. hash the effective rendered prompt; and
+6. construct the prepared value and preparation timing.
+
+`Prepare` runs both phases consecutively and never performs capacity admission.
+`Run` performs backend admission between the phases. This structure preserves
+one execution-precedence and error-ordering implementation while allowing a
+full backend pool to reject work before expensive schema, artifact, and
+rendering operations.
 
 Pointer-based numeric overrides preserve an explicit zero. Invalid negative or
 out-of-range values fail as invalid requests. Endpoint overrides do not change
@@ -70,15 +85,28 @@ invocation state local.
 
 ## Run Flow
 
-`Run` calls `Prepare` rather than maintaining a second preparation path. It
-performs one initial generation call, builds the named output artifact, and
-validates that artifact. Invalid generated content remains a validation result;
-an inability to perform validation is an operational error.
+`Run` records its start time, performs the shared resolution phase, and asks
+its `RunAdmitter` to reserve capacity for the effective backend ID. A nil
+admitter is an internal unlimited fallback. After successful admission, `Run`
+immediately defers the returned release function, performs the completion
+phase, makes one initial generation call, builds the named output artifact,
+and validates that artifact. Invalid generated content remains a validation
+result; an inability to perform validation is an operational error.
+
+The admission lease covers completion-phase preparation, initial generation,
+validation, every repair, and every exit. It bounds accepted work without
+serializing preparation or validation behind the active-generation limit.
+The wrapped model client separately acquires a FIFO active permit only around
+each actual generation call.
 
 When an internal repairer is present, a JSON or JSON Schema content failure can
 trigger bounded repair attempts. Repair receives the effective execution
 target and session ID, validation errors, prior output, and structured-output
-specification. This capability remains internal and is not a public option.
+specification. The default repairer uses the same wrapped client as initial
+generation, so each repair reacquires the selected backend's active permit
+while remaining inside its original admission lease. Repair never performs a
+second bounded admission. This capability remains internal and is not a public
+option.
 
 A successful result includes the output artifact and raw output, validation
 state, effective session ID, prompt and rendered-prompt hashes, selected
@@ -94,8 +122,20 @@ Package errors distinguish invalid requests, required profile selection,
 credential failures, and prompt, profile, artifact, rendering, generation, and
 validation failures. Wrapping preserves the package identities mapped by the
 public facade and retains collaborator identities where they are part of the
-internal contract. Context cancellation propagates through the invoked
-collaborator and is classified by the owning operation.
+internal contract.
+
+Admission capacity exhaustion retains the internal capacity identity and adds
+the selected backend ID as context. It is not recategorized as an invalid
+request or generation failure, and no partial result is returned. A context
+already done at admission retains its context identity directly. Cancellation
+while waiting for an active generation permit prevents client invocation when
+it wins the grant race; the model-client boundary then preserves the context
+error through the generation-failure category. Deferred release restores the
+admission lease on preparation, generation, validation, repair, and
+cancellation failures.
+
+Other context cancellation propagates through the invoked collaborator and is
+classified by the owning operation.
 An overlong direct session is an invalid request before source loading, while
 an invalid or overlong prompt session template remains a prompt-render failure.
 An unknown selected backend, or a selected backend with no configured resolver,
@@ -104,12 +144,16 @@ is classified as a profile-load failure.
 ## Test Ownership And Changes
 
 The [runner tests](../../internal/usecase/runner_test.go) own preparation order,
-selection and override precedence, direct-session resolution,
-schema-before-generation behavior, hashing, generation and validation
-outcomes, backend propagation, bounded repair, credentials and redaction,
-error categories, artifact metadata, usage, and timing.
+selection and override precedence, the two-phase boundary, early admission,
+lease lifetime and release, direct-session resolution, schema-before-generation
+behavior, hashing, generation and validation outcomes, backend propagation,
+bounded repair, shared initial/repair capacity, credentials and redaction,
+error categories, artifact metadata, usage, and timing. The
+[capacity subsystem document](capacity.md) identifies the focused pool,
+waiter, and wrapped-client tests.
 
 Changes to orchestration should continue to use the existing package
-interfaces, keep request state local to an invocation, and preserve `Run`'s use
-of `Prepare`. Source, renderer, validator, or model-client contract changes
-belong first in their owning package and document.
+interfaces, keep request state local to an invocation, and preserve the shared
+resolution and completion pipeline. Source, renderer, validator, or
+model-client contract changes belong first in their owning package and
+document.

docs/policy/architecture.md

@@ -23,6 +23,8 @@ The implemented internal components consist of:
   components;
 - `internal/backend`, which owns validated immutable OpenAI-compatible backend
   definitions and the built-in OpenRouter definition;
+- `internal/capacity`, which owns engine-local bounded run admission and
+  model-generation scheduling for limited backends;
 - `internal/defaults`, which owns application-neutral framework defaults and
   constructs the default execution target;
 - `internal/filecatalog`, which discovers YAML files and provides source-path
@@ -49,19 +51,21 @@ The `examples/go-library/prepare` and `examples/go-library/run` packages are
 maintained downstream consumers of the root facade. They do not expose library
 packages or participate in internal assembly.
 
-The root facade assembles one immutable backend registry, the internal
-repositories, renderer, validator, outbound client, and use-case runner while
-translating public values and errors at the library boundary. The registry
-contains built-ins plus validated engine-scoped consumer additions. The
-defaults and renderer depend on the domain model. Prompt-definition and
-profile repositories use the domain model, file catalog, and YAML decoder. The
-built-in profile repository supplies an embedded `fs.FS` to the profile
-package. Artifact reading uses the domain model and application-neutral
-defaults. Validation uses the domain model, file catalog, and JSON Schema
-implementation. The model client uses the domain model, application-neutral
-defaults, and an injected or standard-library HTTP client. The use-case runner
-depends on the narrow interfaces owned by each internal component, including
-backend lookup.
+The root facade assembles one immutable backend registry, one capacity manager,
+the internal repositories, renderer, validator, outbound client, and use-case
+runner while translating public values and errors at the library boundary. The
+registry contains built-ins plus validated engine-scoped consumer additions.
+The facade constructs the capacity manager from the registry's immutable
+policy snapshot, wraps the selected built-in or injected model client, and
+supplies bounded admission to the runner. The defaults and renderer depend on
+the domain model. Prompt-definition and profile repositories use the domain
+model, file catalog, and YAML decoder. The built-in profile repository supplies
+an embedded `fs.FS` to the profile package. Artifact reading uses the domain
+model and application-neutral defaults. Validation uses the domain model, file
+catalog, and JSON Schema implementation. The model client uses the domain
+model, application-neutral defaults, and an injected or standard-library HTTP
+client. The use-case runner depends on the narrow interfaces owned by each
+internal component, including backend lookup and run admission.
 
 The current implementation follows this dependency direction:
 
@@ -81,10 +85,12 @@ downstream consumers, including Scriptorium
 The backend registry depends on the domain model and shared JSON-value
 validation, has no mutation API after construction, and consumes the
 OpenAI-compatible reserved request-field rule owned by the model client. The
-model client does not depend on registry configuration. The facade coordinates
-internal components and adapts
-the supported public extension interfaces to narrow internal abstractions.
-Internal components must not depend on consumers or on Scriptorium.
+capacity component depends on the domain model and the narrow internal
+model-client boundary, not on provider transport implementation. The model
+client does not depend on registry or capacity configuration. The facade
+coordinates internal components and adapts the supported public extension
+interfaces to narrow internal abstractions. Internal components must not depend
+on consumers or on Scriptorium.
 
 ## Repository And Consumer Boundary
 

docs/roadmap/concurrency.md

@@ -1,6 +1,6 @@
 # Backend-Specific Concurrency Management
 
-**Status:** Accepted.
+**Status:** Complete.
 
 ## Purpose
 

docs/roadmap/implementation.md

@@ -1,6 +1,6 @@
 # Backend-Specific Concurrency Management Implementation Plan
 
-**Status:** Ready for implementation.
+**Status:** Complete.
 
 ## Purpose
 
@@ -647,7 +647,7 @@ releases admission, and existing preparation semantics remain unchanged.
 
 ## Stage 4 — Engine Assembly And Public Runtime Contract
 
-**Status:** Pending.
+**Status:** Complete.
 
 ### Goal
 
@@ -721,7 +721,7 @@ through the same wrapper.
 
 ## Stage 5 — Durable Documentation And Final Validation
 
-**Status:** Pending.
+**Status:** Complete.
 
 ### Goal
 
@@ -819,6 +819,19 @@ limits and cancellation safety, durable contracts no longer depend on roadmap
 prose, and the OpenRouter compatibility change is clearly reported for the
 next minor release.
 
+## Implementation Handoff
+
+Backend-specific capacity management is implemented and has passed the complete
+repository validation sequence. The built-in OpenRouter backend now permits 16
+active generations and a waiting capacity of 1024. Custom backends remain
+unlimited when their limit is omitted, and endpoint-only profiles remain
+unlimited.
+
+Publishing this behavior requires a pre-`v1` minor release. Its release notes
+must identify that unusually high concurrent OpenRouter use can now wait or
+return `ErrCapacityExceeded`. This implementation does not change a module
+version or create a tag.
+
 ## Open Questions
 
 None. The feature roadmap and this plan fix the public representation,

engine.go

@@ -65,7 +65,7 @@ var (
 	ErrPromptRender = errors.New("failed to render prompt")
 	// ErrCapacityExceeded identifies a Run rejected because the selected backend
 	// already admitted ConcurrencyLimit + QueueCapacity calls. It is not an
-	// invalid request, provider rate limit, or ErrLLMGenerate.
+	// invalid request, an LLM or provider rate-limit response, or ErrLLMGenerate.
 	ErrCapacityExceeded = errors.New("backend capacity exceeded")
 	// ErrLLMGenerate identifies a model-client failure or a nil successful
 	// response. Errors returned by an injected LLMClient remain available