Add roadmap and implementation plan for backend concurrency scaling

2026-07-29 20:42:45 +00:00
parent dc39562ff7
commit ffe6d261a9
3 changed files with 1067 additions and 35 deletions
--- a/docs/roadmap/concurrency.md
+++ b/docs/roadmap/concurrency.md
@@ -0,0 +1,236 @@
 # Backend-Specific Concurrency Management
 **Status:** Accepted.
 ## Purpose
 This roadmap defines the scope and target end state for engine-local,
 backend-specific concurrency management. It records the intended capability,
 consumer value, and important policy choices.
 This document is planning material, not a description of current behavior.
 Current exported contracts remain owned by Go declarations and GoDoc, backend
 registration guidance by the
 [consumer guide](../consumers/pkg-promptkit.md#register-a-custom-backend), and
 implemented orchestration by the
 [internal runner document](../internal/runner.md).
 ## Motivation
 Different model backends can sustain very different request loads. A local
 network endpoint may need a small concurrency limit, while OpenRouter can
 usually accept substantially more simultaneous work. Requiring every consumer
 to build its own semaphores and queues would duplicate routing knowledge,
 create inconsistent cancellation behavior, and make it easy for one caller to
 bypass the intended backend limit.
 Promptkit should own this coordination because it already resolves each run to
 an engine-scoped backend identity and owns every model-generation call made by
 the runner. Consumers should continue submitting ready-to-run requests through
 the synchronous API, including concurrently from multiple goroutines, without
 implementing their own backend scheduler.
 The buffered queue is a safety boundary, not an ordinary throughput
 restriction. Its primary purpose is to prevent a bug or unintended submission
 loop from creating an unbounded in-memory backlog.
 ## Scope
 The feature will add optional concurrency policy to registered backends and
 coordinate `Run` calls against independent per-backend capacity pools.
 Each policy has two distinct controls:
 - an active-generation limit, which protects the backend from too many
  simultaneous model requests; and
 - a bounded waiting capacity, which protects the process from admitting an
  unbounded backlog.
 Concurrency policy belongs to a backend registration. It is not a profile
 model parameter and cannot be overridden per run. Profiles select the policy
 through their backend ID, while a profile or request endpoint override remains
 in the selected backend's pool.
 `Prepare` does not call a model and will remain outside concurrency admission.
 ## Defaults And Configuration
 The built-in OpenRouter backend will use:
 - an active-generation limit of 16; and
 - a waiting capacity of 1024.
 The waiting default is intentionally generous. Reaching it should indicate
 abnormal submission pressure rather than normal application behavior.
 Consumer-registered backends will remain unlimited unless the consumer
 configures an active-generation limit. When a consumer enables a limit and
 does not specify waiting capacity, the waiting capacity will default to 1024.
 Consumers may configure a different bounded capacity, including zero when
 they want no admitted backlog beyond the active-limit-sized run set.
 The public representation must distinguish an omitted waiting capacity from
 an explicit zero.
 Endpoint-only profiles have no backend registration from which to obtain
 policy and will remain unlimited. A future engine-wide or endpoint-keyed
 policy can be considered separately if consumers demonstrate that need.
 Invalid limits or capacities will fail engine construction as invalid
 configuration. Policy values will be copied into engine-owned immutable state
 along with the rest of the backend registration.
 ## Admission And Execution Behavior
 `Run` remains a synchronous, wait-for-result operation. Concurrent callers may
 block inside `Run` while waiting for their selected backend, then receive the
 ordinary result or error from that invocation.
 For a configured pool, the active-generation limit plus the waiting capacity
 defines the maximum number of concurrent `Run` invocations that Promptkit will
 accept for that backend. A waiting capacity of zero therefore accepts no more
 runs than the active limit. Admission is immediate: a call either reserves one
 of those bounded slots or receives the capacity error. An accepted run may
 then wait internally for active-generation capacity.
 For a limited backend, Promptkit will bound the number of accepted runs before
 expensive artifact loading, prompt rendering, and large defensive copies where
 practical. Lightweight prompt, profile, and backend resolution may occur first
 when it is required to identify the correct capacity pool. This pre-admission
 resolution must not become a second execution-precedence path with behavior
 that can drift from `Prepare`.
 An accepted run retains its admission until it completes or fails. Every
 actual model-generation call for that run must separately observe the
 backend's active-generation limit. This includes:
 - the initial generation;
 - every output-repair generation; and
 - calls made through either the built-in or an injected model client.
 Preparation and output validation should not hold an active-generation permit.
 A repair remains part of its already-admitted run, but reacquires active
 generation capacity so repairs cannot exceed the backend limit. It must not be
 rejected merely because new runs filled the waiting queue after its initial
 generation.
 Within one backend pool, waiting generation calls should be served in FIFO
 order, subject to canceled calls being removed. Different backend pools make
 progress independently; a saturated local backend must not consume
 OpenRouter's active or waiting capacity.
 The feature will not promise ordering across backend pools or completion order
 among admitted runs.
 ## Capacity Failure And Cancellation
 When a backend's bounded waiting capacity is full, a new `Run` call will fail
 promptly rather than waiting outside the bounded admission system. The public
 API will expose a recognizable capacity-exhaustion error identity distinct
 from invalid configuration, invalid requests, and model-client failures.
 Rejected calls return no partial result and do not invoke the model client.
 Waiting within the admitted backlog or for active-generation capacity must
 honor the caller's context. Cancellation or deadline expiry will:
 - stop waiting promptly;
 - release any admission or generation capacity held by that invocation;
 - preserve the applicable context error identity; and
 - avoid invoking the model client if cancellation wins before generation
  starts.
 Capacity must also be released after preparation, generation, validation,
 repair, or collaborator failure. One failed or canceled run must not reduce
 the backend's future usable capacity.
 Elapsed `Run` timing will include time spent waiting after the call is
 accepted. `PreparedRun` timing will continue to describe preparation rather
 than queue waiting.
 ## Engine And Client Boundaries
 All pools and queued state belong to one `Engine`. Separate engines do not
 share capacity, even when they register the same backend ID or endpoint. The
 feature introduces no process-global scheduler.
 The engine will apply policy consistently to the built-in model client and an
 injected `LLMClient`. Consumers calling their own client outside Promptkit are
 outside this boundary. Injected clients remain responsible for their internal
 thread safety and cancellation behavior.
 Backend policy is keyed by the resolved backend ID rather than endpoint text.
 This preserves stable routing when a selected backend's endpoint is overridden
 and avoids accidentally combining unrelated registrations that happen to use
 the same URL.
 ## Queue Lifetime And Observability
 Admission state is buffered, ephemeral, and in-process. It is not persisted
 and has no survival guarantee across engine disposal or process termination.
 Promptkit will not introduce background job ownership or require consumers to
 start or stop workers.
 The initial feature does not require public queue-depth metrics, callbacks, or
 inspection APIs. Capacity errors and ordinary call timing provide the
 consumer-visible behavior. Operational observability can be added later
 without coupling the scheduling mechanism to an application logging or
 metrics system.
 ## Compatibility
 Consumer-registered backends and endpoint-only profiles remain unlimited
 unless concurrency is explicitly configured, preserving their existing
 behavior.
 The built-in OpenRouter backend will change from unlimited concurrency to a
 limit of 16 with a bounded waiting capacity of 1024. Ordinary synchronous
 calls remain unchanged, while unusually high concurrent use may now wait or
 return the capacity error. This behavioral change must be identified in the
 release notes for the version that publishes it.
 Adding backend policy fields and a public capacity error is otherwise
 additive. The change will use a pre-`v1` minor release under Promptkit's
 [release policy](../release.md#release-model).
 ## Non-Goals
 This scope does not include:
 - asynchronous job handles, polling, or detached result delivery;
 - durable or cross-process queues;
 - persistence or recovery across engine or process shutdown;
 - priorities, scheduling weights, or consumer-defined fairness classes;
 - automatic retries, backoff, rate-limit interpretation, or provider quota
  discovery;
 - token-per-minute or request-per-minute rate limiting;
 - dynamic reconfiguration after engine construction;
 - per-profile or per-run concurrency overrides;
 - endpoint-keyed pooling for profiles without a backend ID;
 - process-global coordination across engines;
 - application worker lifecycle, logging, tracing, or metrics policy; or
 - changes to prompt, profile, schema, or model-provider wire formats.
 ## Target End State
 This roadmap reaches its target end state when:
 - each engine independently coordinates configured backend capacity;
 - the built-in OpenRouter backend allows 16 active generations and up to 1024
  waiting runs;
 - consumer backends can opt into their own active and waiting limits while
  remaining unlimited by default;
 - endpoint overrides retain the selected backend's capacity pool and
  endpoint-only profiles remain unlimited;
 - synchronous `Run` callers wait for and receive their ordinary result;
 - admission is bounded before expensive preparation work where practical;
 - every initial and repair generation observes the backend's active limit
  without serializing preparation or validation;
 - a full waiting queue returns a recognizable capacity error without invoking
  the model client;
 - cancellation and all failure paths promptly release capacity and preserve
  context error identity;
 - built-in and injected model clients receive the same scheduling behavior;
 - pools remain ephemeral, engine-scoped, and independent across backend IDs;
  and
 - current-state GoDoc, consumer, internal, and release documentation describe
  the implemented behavior once it lands.
--- a/docs/roadmap/future.md
+++ b/docs/roadmap/future.md
@@ -33,41 +33,9 @@ consumers.
 ## Ideas
-### Backend-specific concurrency management
+No ideas are currently cataloged. Backend-specific concurrency management has
-
+been selected for active planning in the
-Extend the
+[focused concurrency roadmap](concurrency.md).
 [LLM backend registry](../consumers/pkg-promptkit.md#register-a-custom-backend)
 with optional per-backend concurrency limits and bounded, buffered admission
 queues. Promptkit could then route simultaneous
 generation requests according to backend capacity while containing accidental
 runaway submission. Downstream consumers would continue invoking synchronous
 `Run` calls, including concurrently from multiple goroutines, and each
 admitted call would wait for and return its ordinary result.
 - Scope limits to an engine instance rather than hidden process-global state.
 - Give different backend IDs independent capacity pools. A profile endpoint
  override would remain part of its selected backend's pool.
 - Configure active concurrency and waiting capacity separately. Concurrency
  protects the backend, while queue capacity protects the process from
  admitting an unbounded backlog.
 - Give queue capacity a generous, configurable bounded default intended as a
  safety ceiling for bugs or unintended loops rather than a routine
  application constraint. Select an exact default during implementation
  planning and measurement.
 - Reject a call with a recognizable capacity error when its backend queue is
  full rather than allowing it to wait outside the bounded queue.
 - Admit requests before expensive preparation and artifact copying where
  practical so queued work remains lightweight.
 - Apply a limit to each actual generation request, including repair attempts,
  without unnecessarily serializing prompt preparation.
 - Make queued and active waits respect caller cancellation and deadlines.
 - Treat concurrency as backend policy rather than a profile-level model
  setting.
 - Keep the queue ephemeral and in-process, with no survival guarantee across
  engine or process shutdown.
 - Preserve the existing execution model as far as practical. Durable jobs,
  polling, priorities, application worker lifecycle, retries, and
  cross-process coordination would be separate future capabilities.
 ## Entry Format
--- a/docs/roadmap/implementation.md
+++ b/docs/roadmap/implementation.md
@@ -0,0 +1,828 @@
 # Backend-Specific Concurrency Management Implementation Plan
 **Status:** Ready for implementation.
 ## Purpose
 This document is the decision-complete implementation plan for
 [backend-specific concurrency management](concurrency.md). It is written for a
 coding agent that will implement each stage in order.
 The feature roadmap owns the intended capability, consumer value, policy
 choices, compatibility decision, and target end state. This document owns the
 concrete API, internal representation, scheduling architecture, implementation
 sequence, test ownership, documentation updates, and completion gates.
 ## Implementation Rules
 - Complete the stages in order. Keep the repository compiling and the focused
  tests passing at every stage boundary.
 - Preserve unrelated working-tree changes. In particular, `concurrency.md` and
  the removal of its source idea from `future.md` may already be uncommitted
  when implementation begins; retain both.
 - Follow every policy under `docs/policy/`, the task-specific reading guide in
  `docs/development.md`, and the target behavior in `concurrency.md`.
 - Keep the public API in the root `promptkit` package and implementation
  details under `internal/`. Do not expose scheduler types or create another
  public package.
 - Use only the Go standard library for scheduling. Do not add a queue,
  semaphore, worker-pool, or metrics dependency.
 - Preserve synchronous, wait-for-result `Run`, unrestricted `Prepare`,
  engine-local state, endpoint-only profiles, backend-selected profiles,
  backend identity through endpoint overrides, and injected `LLMClient`
  behavior.
 - Do not broaden the work into asynchronous jobs, durable queues, retries,
  rate limiting, dynamic configuration, priorities, worker lifecycle,
  endpoint-keyed pools, or public queue observability.
 - Keep all tests deterministic, bounded, offline, and race-safe. Coordinate
  concurrent tests with channels and barriers rather than timing assumptions
  or live providers.
 - Update exact GoDoc with each exported declaration change. Update durable
  current-state documents only after the corresponding behavior is
  implemented.
 - Test configurable mechanisms with small test-owned limits. Assert the exact
  OpenRouter `16` and default queue `1024` values only at the registry contract
  that owns those operational defaults.
 - Do not create a release, change a module version, or tag a commit. The final
  implementation handoff must identify the built-in OpenRouter behavior change
  for the next pre-`v1` minor release.
 ## Fixed Design
 ### Public Backend Configuration
 Append these fields to the existing root `Backend` type in `backends.go`:
 ```go
 type Backend struct {
 	// Existing fields remain unchanged and in their current order.
 	ConcurrencyLimit int
 	QueueCapacity    *int
 }
 ```
 Use these exact semantics:
 | Public values | Meaning |
 | --- | --- |
 | `ConcurrencyLimit == 0`, `QueueCapacity == nil` | Unlimited backend; preserve current behavior. |
 | `ConcurrencyLimit > 0`, `QueueCapacity == nil` | Limit active generations and use the default waiting capacity of 1024. |
 | `ConcurrencyLimit > 0`, `QueueCapacity != nil` | Limit active generations and use the pointed-to capacity exactly, including zero. |
 | `ConcurrencyLimit < 0` | Invalid engine configuration. |
 | `QueueCapacity != nil` and `*QueueCapacity < 0` | Invalid engine configuration. |
 | `ConcurrencyLimit == 0` and `QueueCapacity != nil` | Invalid engine configuration because a queue without an active limit has no defined consumer value. |
 `ConcurrencyLimit` counts simultaneous calls to the engine-owned internal
 model-client boundary for this backend. `QueueCapacity` controls additional
 accepted `Run` invocations beyond that limit. The maximum admitted runs for a
 limited backend is therefore:
 ```text
 ConcurrencyLimit + effective QueueCapacity
 ```
 Guard that addition against integer overflow during backend validation.
 Do not impose an arbitrary upper bound beyond non-negativity and overflow
 safety.
 The `QueueCapacity` pointer exists only to distinguish omission from explicit
 zero. `WithBackend` and `NewEngine` must not retain the caller's pointer.
 `Backend` continues to have no stable JSON representation, and consumers
 remain directed to keyed literals.
 Do not add concurrency fields to `Profile`, `ExecutionTarget`,
 `ExecutionTargetOverride`, `RunRequest`, prompt or profile files, or stable
 prepared/result JSON.
 ### Built-In And Custom Defaults
 The backend registry owns these exact operational defaults:
 ```go
 const (
 	openRouterConcurrencyLimit = 16
 	defaultQueueCapacity       = 1024
 )
 ```
 The built-in `openrouter` definition has a normalized concurrency limit of 16
 and queue capacity of 1024.
 Consumer registrations remain unlimited when concurrency is omitted. For a
 consumer backend with a positive limit and omitted queue capacity, normalize
 the queue capacity to 1024. Preserve an explicitly configured zero.
 Consumers still cannot replace the reserved `openrouter` registration.
 Endpoint-only profiles have no backend policy and remain unlimited. A selected
 backend retains its pool when a profile or request overrides only its endpoint.
 ### Internal Backend Representation
 Extend `internal/domain.Backend` with scalar policy values and explicit
 presence rather than retaining a pointer:
 ```go
 type Backend struct {
 	// Existing fields...
 	ConcurrencyLimit int
 	QueueCapacity    int
 	QueueCapacitySet bool
 }
 type BackendCapacityPolicy struct {
 	ConcurrencyLimit int
 	QueueCapacity    int
 }
 ```
 `WithBackend` converts the public pointer into `QueueCapacity` plus
 `QueueCapacitySet`. Registry normalization validates the combinations above,
 fills the default, and leaves every limited stored backend with
 `QueueCapacitySet == true`. Unlimited stored backends retain zero values and
 `QueueCapacitySet == false`.
 Add this internal registry method:
 ```go
 func (r *Registry) CapacityPolicies() map[string]domain.BackendCapacityPolicy
 ```
 It returns a newly allocated map containing only limited backends. Values are
 scalars, so callers cannot mutate registry state. The built-in OpenRouter
 policy is included. `GetBackend` continues returning a defensive backend copy,
 now including normalized scalar capacity metadata.
 Capacity policy is operational registry metadata. Do not merge it into an
 execution target or expose it to injected model clients.
 ### Public Capacity Error
 Add this root sentinel beside the other run errors in `engine.go`:
 ```go
 var ErrCapacityExceeded = errors.New("backend capacity exceeded")
 ```
 Its GoDoc must state that it identifies a `Run` rejected because the selected
 backend has already admitted `ConcurrencyLimit + QueueCapacity` runs. It is
 not an invalid request, an LLM/provider rate-limit response, or an
 `ErrLLMGenerate` failure.
 The internal capacity component owns a corresponding internal
 `ErrCapacityExceeded`. Add its mapping in `publicErrorFor` before the broader
 generation and invalid-request cases. The public error must preserve the
 internal error through wrapping while matching `ErrCapacityExceeded` with
 `errors.Is`.
 A capacity rejection returns no partial result and must not invoke the
 artifact reader, renderer, schema loader, validator, or model client. Prompt,
 profile, and backend loading needed to select the pool may already have
 occurred.
 ### Internal Capacity Component
 Add `internal/capacity` as the single owner of engine-local run admission and
 active-generation permits.
 Use these package-level boundaries:
 ```go
 var ErrCapacityExceeded error
 type Manager struct {
 	// Private immutable pool map.
 }
 func NewManager(
 	policies map[string]domain.BackendCapacityPolicy,
 ) (*Manager, error)
 func (m *Manager) Admit(
 	ctx context.Context,
 	backendID string,
 ) (release func(), err error)
 func NewClient(m *Manager, next llm.Client) llm.Client
 ```
 `NewManager` copies the supplied map and creates one independent pool per
 limited backend. Defensively reject blank IDs, non-positive concurrency
 limits, negative queue capacities, or total-capacity overflow even though the
 registry normally supplies normalized values. Construction creates no worker
 goroutines.
 An absent manager, blank backend ID, or ID absent from the policy map is
 unlimited:
 - `Admit` succeeds with a non-nil no-op release function; and
 - the client wrapper calls the next client directly.
 For a limited pool, `Admit` is immediate and context-aware:
 1. return `ctx.Err()` if the context is already done;
 2. under the pool lock, compare admitted runs with
   `ConcurrencyLimit + QueueCapacity`;
 3. return an error matching internal `ErrCapacityExceeded` when full; or
 4. increment admitted runs and return an idempotent release function.
 The release function decrements admission exactly once, even if accidentally
 called more than once. It does not release an active-generation permit; those
 permits have their own lifetime.
 ### FIFO Generation Permits
 `NewClient` returns an internal `llm.Client` wrapper around either the built-in
 client or the public-client adapter. It must preserve requests, successful
 responses, nil responses, and collaborator error identities exactly.
 `next` must be non-nil; `NewEngine` and internal runner construction maintain
 that invariant. A nil manager returns `next` unchanged.
 For a configured backend ID, the wrapper:
 1. acquires one active-generation permit from the matching pool;
 2. waits in FIFO order when the active count equals `ConcurrencyLimit`;
 3. removes a canceled waiter and returns `ctx.Err()` when cancellation wins
   before the permit is granted;
 4. invokes the next client only after a permit is granted; and
 5. releases the permit with `defer` after every success, nil response,
   collaborator error, panic unwinding, or context outcome.
 Implement FIFO and cancellation explicitly with a mutex and an ordered waiter
 list. A channel used only as a counting semaphore is insufficient because it
 does not define FIFO ordering or safe removal of canceled waiters.
 Permit grant and cancellation must have one lock-protected linearization
 point. If cancellation removes the waiter first, do not invoke the next
 client. If grant wins first, invoke the next client with the caller's context;
 the next client may then observe cancellation normally. Never lose or
 double-release a permit in this race.
 Releasing a permit transfers it to the oldest non-canceled waiter before
 making it generally available. Different backend pools never share admission
 or active counts.
 The active wrapper enforces its limit even if an internal caller invokes it
 without a run admission lease. Bounded backlog is guaranteed for ordinary
 engine `Run` calls by the runner admission path; no public API exposes the
 wrapped internal client directly.
 ### Engine Assembly
 In `NewEngine`, after constructing the validated backend registry:
 1. obtain `backendRegistry.CapacityPolicies()`;
 2. construct one `capacity.Manager`;
 3. construct the selected base internal LLM client exactly as today;
 4. wrap that base client with `capacity.NewClient`; and
 5. pass both the wrapped client and manager-as-admitter to the runner.
 Every `NewEngine` call constructs a distinct manager. Do not cache managers,
 pools, or policies in package globals. The wrapper must be applied after a
 public injected client is adapted to `internal/llm.Client`, so built-in and
 injected clients receive identical scheduling behavior.
 If `NewManager` reports a defensive configuration error, make `NewEngine`
 return an error matching `ErrInvalidConfig`.
 `Prepare` does not use the manager. An injected client remains required to be
 safe for concurrent calls because different backend pools and unlimited
 backends may still invoke it concurrently.
 ### Shared Two-Phase Preparation
 Refactor `internal/usecase.Runner` so `Prepare` and `Run` share one preparation
 pipeline with two private phases. Do not duplicate prompt/profile/backend
 selection or execution precedence.
 The first phase resolves only the state required before admission:
 1. validate `PromptID`;
 2. normalize the direct session ID;
 3. load the prompt definition;
 4. hash the original prompt definition at its existing error-order position;
 5. select and load the execution profile;
 6. resolve the selected backend;
 7. resolve and validate the effective execution target and credentials; and
 8. resolve the effective output contract without loading its schema.
 Return a private state value containing the loaded definition, normalized
 direct session, prompt-definition hash, selected profile ID, effective target,
 numeric-presence metadata, effective output contract, and preparation start
 time. Keep this value private to `internal/usecase`.
 The second phase consumes that state and performs:
 1. structured-output schema loading;
 2. artifact loading and input hashing;
 3. message and prompt-session rendering;
 4. direct-session application;
 5. rendered-prompt hashing; and
 6. `PreparedRun` construction and timing.
 Preserve every existing precedence rule, error identity, direct-session
 template bypass, hash input, selected identity, copy guarantee, and timing
 field. Do not reload the prompt, profile, or backend between phases.
 `Runner.Prepare` records its start time, runs both phases consecutively, and
 never calls admission. Its behavior and error ordering remain unchanged.
 `Runner.Run` records its existing run start time, runs the first preparation
 phase, and then calls:
 ```go
 release, err := r.admitter.Admit(ctx, effectiveBackendID)
 ```
 Use a narrow use-case-owned interface with the same signature:
 ```go
 type RunAdmitter interface {
 	Admit(context.Context, string) (func(), error)
 }
 ```
 A nil admitter means unlimited behavior for internal constructors and tests.
 On successful admission, immediately `defer release()` around the remainder of
 the run. Then run the second preparation phase, initial generation,
 validation, and all repair attempts.
 If admission returns internal `capacity.ErrCapacityExceeded`, add useful
 backend context without changing its identity. If it returns `ctx.Err()`,
 preserve that identity directly rather than recategorizing it as invalid
 request or generation failure.
 This refactor intentionally replaces the current literal `Run`-calls-`Prepare`
 implementation with shared private phases. Update current-state documentation
 to describe one shared pipeline rather than retaining an inaccurate call-graph
 claim.
 ### Generation And Repair Lifetime
 The admission lease covers the entire accepted run:
 - second-phase preparation;
 - initial generation;
 - validation;
 - every repair; and
 - all failure and cancellation exits.
 Preparation and validation do not hold an active-generation permit. The
 wrapped client acquires a permit only around each actual `Generate` call.
 The runner's initial generation already carries the effective backend ID in
 `GenerateRequest.Target`. Preserve that value. `RepairRequest.Target` and the
 default repairer's generated request must continue carrying the same backend
 ID, allowing each repair to reacquire the same pool's active permit.
 When testing or constructing `NewRunnerWithRepairer`, pass the same wrapped
 client to both the runner and `NewDefaultOutputRepairer`. Do not add capacity
 state to `RepairRequest`, `ExecutionTarget`, or public generation values.
 A repair remains within its existing admission lease. It waits for a FIFO
 active permit but never performs a second bounded admission and therefore
 cannot fail merely because later runs filled the admission capacity.
 ### Error And Cancellation Semantics
 The required public outcomes are:
 | Situation | Required error identity |
 | --- | --- |
 | Admission capacity is full | `ErrCapacityExceeded` only; not `ErrInvalidRequest` or `ErrLLMGenerate`. |
 | Context is done before admission succeeds | Preserve `ctx.Err()`; do not return capacity exhaustion. |
 | Context cancels while waiting for an active permit | Preserve `ctx.Err()` through the existing `ErrLLMGenerate` generation category. |
 | Wrapped client fails after permit acquisition | Preserve existing `ErrLLMGenerate` and collaborator identities. |
 | Preparation or validation fails after admission | Preserve its existing category and release admission. |
 Maintain the existing rule that `Run` returns no partial result on any
 operational error. Do not add queue status to errors or results.
 `RunResult.Duration` continues to start at runner entry and therefore includes
 pre-admission resolution, accepted preparation, and active-permit waiting.
 `PreparedRun.DurationMS` continues to cover only its shared preparation phases;
 it does not include later generation waiting. Capacity-rejected calls have no
 result or timing value.
 ### Ownership And Concurrency Safety
 The registry, capacity policy map, pool map, and per-pool limits are immutable
 after engine construction. Only admission counts, active counts, and waiter
 lists are mutable and must be protected by the owning pool mutex.
 Do not retain public queue pointers, caller request values, contexts, or
 generation requests after their call completes. A canceled waiter must be
 unlinked so its context and request cannot remain reachable from the pool.
 Do not hold a pool mutex while:
 - loading or rendering prompts;
 - reading artifacts or schemas;
 - invoking a model client;
 - validating output;
 - closing a waiter notification channel if the implementation could re-enter
  pool code; or
 - calling consumer code.
 No scheduler operation may spawn a goroutine whose lifetime outlasts the
 calling `Run`. The zero steady-state goroutine count is part of the
 in-process/no-worker-lifecycle design.
 ## Test Ownership
 Use this ownership split and avoid repeating the full policy matrix at every
 layer:
 - `internal/backend/registry_test.go` owns normalization, validation, the exact
  OpenRouter policy, the custom default queue, explicit zero, unlimited
  omission, and policy-map copying.
 - `internal/capacity/manager_test.go` owns admission bounds, idempotent release,
  FIFO active permits, cancellation races, capacity recovery, independent
  pools, unlimited IDs, and observed peak concurrency.
 - `internal/capacity/client_test.go` owns wrapper request/response/error
  transparency and the rule that cancellation before grant does not invoke the
  next client. Combine these with manager tests if one coherent package test
  expresses the behavior more clearly.
 - `internal/usecase/runner_test.go` owns two-phase preparation parity, pool
  selection, admission before expensive work, admission release across run
  exits, `Prepare` bypass, and repair reuse of the admitted backend.
 - Root external-package tests own public configuration conversion, assembled
  engine-local behavior, endpoint-override routing, injected-client limiting,
  and public capacity/context error identities.
 - Existing model-client HTTP tests remain unchanged because scheduling does
  not alter the OpenAI-compatible wire contract.
 Concurrency tests must use test-owned limits such as one or two and
 channel-controlled blocking clients. Record observed active and peak counts
 under a mutex or atomics. Do not use `time.Sleep` to infer queue state.
 Package-internal tests may inspect a waiter list under its mutex through a
 small test helper when necessary to establish deterministic FIFO ordering; do
 not add production metrics or hooks solely for tests.
 Do not add separate tests for trivial scalar copies when registry or assembled
 behavior already protects them.
 ## Stage 1 — Backend Policy And Public Configuration
 **Status:** Pending.
 ### Goal
 Add the public and internal backend policy representation, normalize all
 configured states, and expose immutable normalized policies without changing
 runtime scheduling yet.
 ### Work
 1. Add `ConcurrencyLimit` and `QueueCapacity` to `Backend` in `backends.go`
   with exact GoDoc for unlimited, defaulted, explicit-zero, invalid, and
   engine-scoped behavior.
 2. Convert the public queue pointer into scalar value plus presence in
   `WithBackend`; do not retain the pointer.
 3. Add the internal backend policy fields and
   `BackendCapacityPolicy` to `internal/domain/domain.go`.
 4. Add the two registry-owned constants and configure the built-in OpenRouter
   definition with 16 and 1024.
 5. Extend `normalizeBackend` with the fixed validation, defaulting, explicit
   zero, and overflow rules.
 6. Add `Registry.CapacityPolicies`, returning only limited policies in a fresh
   map.
 7. Update existing backend composite literals and assertions only where the
   new fields are relevant. Continue using keyed literals.
 ### Tests
 1. Extend the exact built-in registry test with the OpenRouter limit and queue.
 2. Add one coherent table covering unlimited omission, default queue,
   explicit-zero queue, negative values, queue-without-limit, and total
   overflow.
 3. Extend the registry copy/normalization test to prove returned policy maps
   cannot mutate registry state.
 4. Add root coverage only if needed to prove the public pointer/presence
   conversion; do not reproduce registry validation cases at the facade.
 ### Focused Validation
 Run:
 ```sh
 gofmt -w backends.go internal/domain/domain.go \
  internal/backend/registry.go internal/backend/registry_test.go
 go test . ./internal/backend
 go vet . ./internal/backend
 git diff --check
 ```
 Include another touched Go test file in `gofmt` only if it actually changed.
 ### Completion Gate
 This stage is complete when every public configuration state has one normalized
 internal meaning, OpenRouter exposes exactly 16/1024, custom backends remain
 unlimited by omission, and no runtime call is scheduled yet.
 ## Stage 2 — Engine-Local Capacity Manager
 **Status:** Pending.
 ### Goal
 Implement and prove the bounded admission mechanism and FIFO active-generation
 client wrapper independently of runner orchestration.
 ### Work
 1. Add `internal/capacity/manager.go` with the manager, immutable policy copy,
   per-backend pools, internal error, immediate admission, idempotent release,
   and FIFO context-aware active permits.
 2. Add `internal/capacity/client.go` with the transparent `llm.Client` wrapper.
 3. Use mutex-protected waiter state and an ordered list; explicitly resolve
   grant-versus-cancel races.
 4. Ensure unlimited and independent-pool fast paths avoid queue allocation.
 5. Do not start workers, timers, cleanup goroutines, or process-global state.
 ### Tests
 1. Add a compact constructor-validation table for blank IDs, non-positive
   limits, negative queues, and total-capacity overflow.
 2. With a small configured policy, prove that exactly
   `limit + queueCapacity` admissions succeed, the next matches
   `ErrCapacityExceeded`, and a release permits another admission.
 3. Prove release is idempotent.
 4. Drive more blocked client calls than the active limit and assert observed
   peak concurrency never exceeds that limit.
 5. Prove FIFO order with deterministic queue-entry synchronization.
 6. Cancel the first and a middle waiter and prove they are removed, never call
   the wrapped client, and do not block later waiters.
 7. Exercise the grant/cancel race repeatedly under `go test -race`, asserting
   no permit leak or double invocation.
 8. Prove different backend IDs proceed independently and blank, unknown, or
   nil-manager paths remain unlimited.
 9. Prove request values, successful and nil responses, and collaborator errors
   pass through unchanged after permit acquisition.
 ### Focused Validation
 Run:
 ```sh
 gofmt -w internal/capacity/manager.go \
  internal/capacity/manager_test.go \
  internal/capacity/client.go \
  internal/capacity/client_test.go
 go test ./internal/capacity
 go test -race ./internal/capacity
 go vet ./internal/capacity
 git diff --check
 ```
 If tests are combined into one file, omit the nonexistent file from `gofmt`.
 ### Completion Gate
 This stage is complete when the standalone component enforces relational
 admission and active limits, FIFO cancellation is race-safe, separate pools
 are independent, and the wrapper is transparent apart from waiting.
 ## Stage 3 — Shared Preparation And Early Run Admission
 **Status:** Pending.
 ### Goal
 Refactor runner preparation into one shared two-phase pipeline and place
 bounded admission after backend resolution but before schema, artifact, and
 rendering work.
 ### Work
 1. Add the private pre-admission preparation state and split the existing
   `Prepare` logic according to the fixed design.
 2. Make `Runner.Prepare` call both phases without an admitter.
 3. Add the `RunAdmitter` interface and runner field.
 4. Update `NewRunner` and `NewRunnerWithRepairer` to accept the optional
   admitter; update internal call sites with nil until root assembly is wired.
 5. Change `Runner.Run` to use the first phase, admit by effective backend ID,
   defer the returned release, and then use the second phase.
 6. Preserve all existing error precedence, target resolution, hashes,
   metadata, session behavior, and timing.
 7. Return capacity and context errors with the fixed identities. Do not invoke
   later collaborators after rejection.
 ### Tests
 1. Keep the existing `Run`/`Prepare` parity coverage passing to prove the
   shared phases do not drift.
 2. Add a fake admitter that records backend IDs and release calls.
 3. Prove a backend-selected run admits with the selected ID even when the
   endpoint is overridden.
 4. Prove an endpoint-only run uses the unlimited/blank identity and that
   `Prepare` never calls admission.
 5. Reject admission and assert schema, artifact, renderer, validator, repairer,
   and LLM collaborators are not invoked.
 6. Prove admission is released after one successful run and representative
   second-phase, generation, and validation errors. Prefer a small table around
   the single `defer` invariant rather than duplicating every error test.
 7. Retain direct-session, backend precedence, credential, hashing, and repair
   tests unchanged except for constructor arguments.
 ### Focused Validation
 Run:
 ```sh
 gofmt -w internal/usecase/runner.go \
  internal/usecase/runner_test.go
 go test ./internal/usecase
 go test -race ./internal/usecase
 go vet ./internal/usecase
 git diff --check
 ```
 ### Completion Gate
 This stage is complete when `Prepare` remains unrestricted, `Run` admits after
 one canonical routing phase and before expensive completion work, every exit
 releases admission, and existing preparation semantics remain unchanged.
 ## Stage 4 — Engine Assembly And Public Runtime Contract
 **Status:** Pending.
 ### Goal
 Wire one manager into each engine, schedule built-in and injected clients,
 expose the capacity error, and prove assembled runtime behavior.
 ### Work
 1. Add public `ErrCapacityExceeded` and its exact GoDoc in `engine.go`.
 2. Map internal capacity exhaustion in `errors.go`.
 3. Construct the manager from the registry policy snapshot in `NewEngine`.
 4. Wrap the selected internal client after built-in or injected-client
   selection and pass the manager and wrapped client to the runner.
 5. Update `Engine`, `NewEngine`, `Run`, `WithLLMClient`, and `LLMClient` GoDoc
   only where concurrency, capacity, or cancellation statements change.
 6. Ensure manager-construction errors match `ErrInvalidConfig`.
 7. For internal repair coverage, construct the default repairer with the same
   wrapped client used by its runner and confirm repair target backend identity
   remains intact.
 ### Tests
 1. Add an external-package assembled test with a small custom limit and a
   blocking injected client; assert peak generation equals or remains below
   the configured limit.
 2. With queue capacity zero, block one accepted run before generation and
   assert the next matching-backend run returns `ErrCapacityExceeded`, does not
   match `ErrInvalidRequest` or `ErrLLMGenerate`, returns no result, and never
   reaches expensive collaborators or the client.
 3. In the same or another focused workflow, prove an endpoint override remains
   in the selected backend's pool.
 4. Prove two engines with the same backend ID have independent capacity.
 5. Prove an unlimited custom backend and an endpoint-only profile preserve
   concurrent behavior.
 6. Cancel a call waiting for an active permit; assert it matches both
   `context.Canceled` and `ErrLLMGenerate`, never invokes the injected client,
   and leaves capacity reusable.
 7. Add one internal repair workflow with concurrent runs or controlled permits
   showing initial and repair generations never exceed the same backend limit
   and repairs do not perform a second admission.
 8. Extend the public error sentinel contract test with
   `ErrCapacityExceeded`.
 Avoid a second HTTP-level concurrency suite: the capacity client tests and one
 assembled injected-client workflow already protect the shared wrapper used by
 the built-in client.
 ### Focused Validation
 Run:
 ```sh
 gofmt -w engine.go errors.go backends.go \
  internal/usecase/runner.go internal/usecase/runner_test.go \
  engine_test.go public_contract_test.go
 go test . ./internal/backend ./internal/capacity ./internal/usecase
 go test -race . ./internal/capacity ./internal/usecase
 go vet . ./internal/backend ./internal/capacity ./internal/usecase
 git diff --check
 ```
 Add any newly created capacity files to `gofmt` when they changed in this
 stage.
 ### Completion Gate
 This stage is complete when every engine has independent pools, limited runs
 are bounded and FIFO at generation, endpoint routing is correct, capacity and
 context errors are stable, repairs reuse admission, and both client kinds pass
 through the same wrapper.
 ## Stage 5 — Durable Documentation And Final Validation
 **Status:** Pending.
 ### Goal
 Move implemented contracts into their durable owners, record compatibility
 impact, and validate the complete repository.
 ### Work
 1. Review every changed exported declaration. Ensure GoDoc is the canonical
   owner of exact field types, nil/zero semantics, defaulting, error identity,
   engine scope, concurrency safety, cancellation, and source compatibility.
 2. Update `doc.go` so its concurrency summary acknowledges backend scheduling
   while continuing to require injected collaborators to be concurrency-safe.
 3. Update `docs/consumers/pkg-promptkit.md` with task-oriented examples for:
   - a limited local backend;
   - omitted queue capacity selecting 1024;
   - explicit zero queue capacity; and
   - handling `ErrCapacityExceeded`.
   Keep exact field semantics in GoDoc rather than duplicating a full table.
 4. Add `docs/internal/capacity.md` as the durable owner of pool lifecycle,
   admission, FIFO active permits, cancellation, client wrapping, and test
   ownership.
 5. Add `internal/capacity` to `docs/internal/overview.md`.
 6. Update `docs/policy/architecture.md` to include the implemented component
   and root assembly dependency without turning policy into an API reference.
 7. Update `docs/internal/runner.md` to describe the shared two-phase
   preparation pipeline, early bounded admission, lease lifetime, generation
   permits, repairs, capacity failures, and cancellation.
 8. Review `docs/formats.md`; add only a concise link or clarification if needed
   to explain that endpoint overrides preserve backend capacity identity.
   Do not add concurrency fields to YAML.
 9. Do not change the OpenAI-compatible integration contract or
   `docs/internal/llm.md` unless implementation changes their current
   statements; scheduling is outside the provider wire contract and concrete
   model-client implementation.
 10. Record in the implementation handoff that built-in OpenRouter now limits
    active generations to 16 with queue capacity 1024 and that the release
    must be a pre-`v1` minor release. Do not edit the release procedure or
    create a tag.
 11. After every check passes, set `concurrency.md`, this implementation plan,
    and each stage status to `Complete`. Do not remove the roadmaps in the
    implementation change; lifecycle retirement follows review.
 ### Full Validation
 Run the complete sequence from `docs/development.md`:
 ```sh
 go test ./...
 go test -race ./...
 go vet ./...
 go build ./...
 go run ./examples/go-library/prepare
 gofmt -l $(git ls-files '*.go')
 git diff --check
 ```
 The formatting command must produce no paths. Follow every added or changed
 Markdown link and confirm its target and heading exist.
 Also inspect:
 ```sh
 git status --short
 git diff --stat
 git diff
 ```
 Confirm that:
 - only intended backend, capacity, runner, facade, test, documentation, and
  roadmap files changed;
 - no `go.work`, `go.work.sum`, local module replacement, credential,
  generated binary, coverage output, or unrelated change was introduced;
 - the built-in OpenRouter policy is exactly 16/1024;
 - custom and endpoint-only backends remain unlimited by omission;
 - explicit queue zero is distinguishable from omission;
 - no capacity value enters execution targets, generated requests, stable JSON,
  prompt/profile YAML, or provider payloads;
 - every engine owns distinct pools with no package-global mutable state;
 - every initial and repair generation uses the active permit wrapper;
 - capacity and waiter state is released on success, error, panic unwinding,
  and cancellation;
 - concurrency tests use deterministic coordination rather than sleeps;
 - current-state documentation describes implemented behavior rather than
  referring readers to the roadmaps; and
 - the feature and implementation roadmaps contain no unresolved work marked
  complete.
 ### Completion Gate
 The implementation is complete only when every target-end-state item in
 `concurrency.md` is implemented, race-enabled tests demonstrate the configured
 limits and cancellation safety, durable contracts no longer depend on roadmap
 prose, and the OpenRouter compatibility change is clearly reported for the
 next minor release.
 ## Open Questions
 None. The feature roadmap and this plan fix the public representation,
 registry defaults, admission bound, FIFO generation behavior, early-routing
 refactor, cancellation races, error identities, engine and repair lifetimes,
 test ownership, compatibility treatment, and non-goals required for
 implementation.