promptkit/docs/roadmap/concurrency.md

Backend-Specific Concurrency Management

Status: Accepted.

Purpose

This roadmap defines the scope and target end state for engine-local, backend-specific concurrency management. It records the intended capability, consumer value, and important policy choices.

This document is planning material, not a description of current behavior. Current exported contracts remain owned by Go declarations and GoDoc, backend registration guidance by the consumer guide, and implemented orchestration by the internal runner document.

Motivation

Different model backends can sustain very different request loads. A local network endpoint may need a small concurrency limit, while OpenRouter can usually accept substantially more simultaneous work. Requiring every consumer to build its own semaphores and queues would duplicate routing knowledge, create inconsistent cancellation behavior, and make it easy for one caller to bypass the intended backend limit.

Promptkit should own this coordination because it already resolves each run to an engine-scoped backend identity and owns every model-generation call made by the runner. Consumers should continue submitting ready-to-run requests through the synchronous API, including concurrently from multiple goroutines, without implementing their own backend scheduler.

The buffered queue is a safety boundary, not an ordinary throughput restriction. Its primary purpose is to prevent a bug or unintended submission loop from creating an unbounded in-memory backlog.

Scope

The feature will add optional concurrency policy to registered backends and coordinate Run calls against independent per-backend capacity pools.

Each policy has two distinct controls:

• an active-generation limit, which protects the backend from too many simultaneous model requests; and
• a bounded waiting capacity, which protects the process from admitting an unbounded backlog.

Concurrency policy belongs to a backend registration. It is not a profile model parameter and cannot be overridden per run. Profiles select the policy through their backend ID, while a profile or request endpoint override remains in the selected backend's pool.

Prepare does not call a model and will remain outside concurrency admission.

Defaults And Configuration

The built-in OpenRouter backend will use:

• an active-generation limit of 16; and
• a waiting capacity of 1024.

The waiting default is intentionally generous. Reaching it should indicate abnormal submission pressure rather than normal application behavior.

Consumer-registered backends will remain unlimited unless the consumer configures an active-generation limit. When a consumer enables a limit and does not specify waiting capacity, the waiting capacity will default to 1024. Consumers may configure a different bounded capacity, including zero when they want no admitted backlog beyond the active-limit-sized run set.

The public representation must distinguish an omitted waiting capacity from an explicit zero.

Endpoint-only profiles have no backend registration from which to obtain policy and will remain unlimited. A future engine-wide or endpoint-keyed policy can be considered separately if consumers demonstrate that need.

Invalid limits or capacities will fail engine construction as invalid configuration. Policy values will be copied into engine-owned immutable state along with the rest of the backend registration.

Admission And Execution Behavior

Run remains a synchronous, wait-for-result operation. Concurrent callers may block inside Run while waiting for their selected backend, then receive the ordinary result or error from that invocation.

For a configured pool, the active-generation limit plus the waiting capacity defines the maximum number of concurrent Run invocations that Promptkit will accept for that backend. A waiting capacity of zero therefore accepts no more runs than the active limit. Admission is immediate: a call either reserves one of those bounded slots or receives the capacity error. An accepted run may then wait internally for active-generation capacity.

For a limited backend, Promptkit will bound the number of accepted runs before expensive artifact loading, prompt rendering, and large defensive copies where practical. Lightweight prompt, profile, and backend resolution may occur first when it is required to identify the correct capacity pool. This pre-admission resolution must not become a second execution-precedence path with behavior that can drift from Prepare.

An accepted run retains its admission until it completes or fails. Every actual model-generation call for that run must separately observe the backend's active-generation limit. This includes:

• the initial generation;
• every output-repair generation; and
• calls made through either the built-in or an injected model client.

Preparation and output validation should not hold an active-generation permit. A repair remains part of its already-admitted run, but reacquires active generation capacity so repairs cannot exceed the backend limit. It must not be rejected merely because new runs filled the waiting queue after its initial generation.

Within one backend pool, waiting generation calls should be served in FIFO order, subject to canceled calls being removed. Different backend pools make progress independently; a saturated local backend must not consume OpenRouter's active or waiting capacity.

The feature will not promise ordering across backend pools or completion order among admitted runs.

Capacity Failure And Cancellation

When a backend's bounded waiting capacity is full, a new Run call will fail promptly rather than waiting outside the bounded admission system. The public API will expose a recognizable capacity-exhaustion error identity distinct from invalid configuration, invalid requests, and model-client failures. Rejected calls return no partial result and do not invoke the model client.

Waiting within the admitted backlog or for active-generation capacity must honor the caller's context. Cancellation or deadline expiry will:

• stop waiting promptly;
• release any admission or generation capacity held by that invocation;
• preserve the applicable context error identity; and
• avoid invoking the model client if cancellation wins before generation starts.

Capacity must also be released after preparation, generation, validation, repair, or collaborator failure. One failed or canceled run must not reduce the backend's future usable capacity.

Elapsed Run timing will include time spent waiting after the call is accepted. PreparedRun timing will continue to describe preparation rather than queue waiting.

Engine And Client Boundaries

All pools and queued state belong to one Engine. Separate engines do not share capacity, even when they register the same backend ID or endpoint. The feature introduces no process-global scheduler.

The engine will apply policy consistently to the built-in model client and an injected LLMClient. Consumers calling their own client outside Promptkit are outside this boundary. Injected clients remain responsible for their internal thread safety and cancellation behavior.

Backend policy is keyed by the resolved backend ID rather than endpoint text. This preserves stable routing when a selected backend's endpoint is overridden and avoids accidentally combining unrelated registrations that happen to use the same URL.

Queue Lifetime And Observability

Admission state is buffered, ephemeral, and in-process. It is not persisted and has no survival guarantee across engine disposal or process termination. Promptkit will not introduce background job ownership or require consumers to start or stop workers.

The initial feature does not require public queue-depth metrics, callbacks, or inspection APIs. Capacity errors and ordinary call timing provide the consumer-visible behavior. Operational observability can be added later without coupling the scheduling mechanism to an application logging or metrics system.

Compatibility

Consumer-registered backends and endpoint-only profiles remain unlimited unless concurrency is explicitly configured, preserving their existing behavior.

The built-in OpenRouter backend will change from unlimited concurrency to a limit of 16 with a bounded waiting capacity of 1024. Ordinary synchronous calls remain unchanged, while unusually high concurrent use may now wait or return the capacity error. This behavioral change must be identified in the release notes for the version that publishes it.

Adding backend policy fields and a public capacity error is otherwise additive. The change will use a pre-v1 minor release under Promptkit's release policy.

Non-Goals

This scope does not include:

• asynchronous job handles, polling, or detached result delivery;
• durable or cross-process queues;
• persistence or recovery across engine or process shutdown;
• priorities, scheduling weights, or consumer-defined fairness classes;
• automatic retries, backoff, rate-limit interpretation, or provider quota discovery;
• token-per-minute or request-per-minute rate limiting;
• dynamic reconfiguration after engine construction;
• per-profile or per-run concurrency overrides;
• endpoint-keyed pooling for profiles without a backend ID;
• process-global coordination across engines;
• application worker lifecycle, logging, tracing, or metrics policy; or
• changes to prompt, profile, schema, or model-provider wire formats.

Target End State

This roadmap reaches its target end state when:

• each engine independently coordinates configured backend capacity;
• the built-in OpenRouter backend allows 16 active generations and up to 1024 waiting runs;
• consumer backends can opt into their own active and waiting limits while remaining unlimited by default;
• endpoint overrides retain the selected backend's capacity pool and endpoint-only profiles remain unlimited;
• synchronous Run callers wait for and receive their ordinary result;
• admission is bounded before expensive preparation work where practical;
• every initial and repair generation observes the backend's active limit without serializing preparation or validation;
• a full waiting queue returns a recognizable capacity error without invoking the model client;
• cancellation and all failure paths promptly release capacity and preserve context error identity;
• built-in and injected model clients receive the same scheduling behavior;
• pools remain ephemeral, engine-scoped, and independent across backend IDs; and
• current-state GoDoc, consumer, internal, and release documentation describe the implemented behavior once it lands.