diff --git a/docs/roadmap/concurrency.md b/docs/roadmap/concurrency.md new file mode 100644 index 0000000..aaa720d --- /dev/null +++ b/docs/roadmap/concurrency.md @@ -0,0 +1,236 @@ +# Backend-Specific Concurrency Management + +**Status:** Accepted. + +## Purpose + +This roadmap defines the scope and target end state for engine-local, +backend-specific concurrency management. It records the intended capability, +consumer value, and important policy choices. + +This document is planning material, not a description of current behavior. +Current exported contracts remain owned by Go declarations and GoDoc, backend +registration guidance by the +[consumer guide](../consumers/pkg-promptkit.md#register-a-custom-backend), and +implemented orchestration by the +[internal runner document](../internal/runner.md). + +## Motivation + +Different model backends can sustain very different request loads. A local +network endpoint may need a small concurrency limit, while OpenRouter can +usually accept substantially more simultaneous work. Requiring every consumer +to build its own semaphores and queues would duplicate routing knowledge, +create inconsistent cancellation behavior, and make it easy for one caller to +bypass the intended backend limit. + +Promptkit should own this coordination because it already resolves each run to +an engine-scoped backend identity and owns every model-generation call made by +the runner. Consumers should continue submitting ready-to-run requests through +the synchronous API, including concurrently from multiple goroutines, without +implementing their own backend scheduler. + +The buffered queue is a safety boundary, not an ordinary throughput +restriction. Its primary purpose is to prevent a bug or unintended submission +loop from creating an unbounded in-memory backlog. + +## Scope + +The feature will add optional concurrency policy to registered backends and +coordinate `Run` calls against independent per-backend capacity pools. + +Each policy has two distinct controls: + +- an active-generation limit, which protects the backend from too many + simultaneous model requests; and +- a bounded waiting capacity, which protects the process from admitting an + unbounded backlog. + +Concurrency policy belongs to a backend registration. It is not a profile +model parameter and cannot be overridden per run. Profiles select the policy +through their backend ID, while a profile or request endpoint override remains +in the selected backend's pool. + +`Prepare` does not call a model and will remain outside concurrency admission. + +## Defaults And Configuration + +The built-in OpenRouter backend will use: + +- an active-generation limit of 16; and +- a waiting capacity of 1024. + +The waiting default is intentionally generous. Reaching it should indicate +abnormal submission pressure rather than normal application behavior. + +Consumer-registered backends will remain unlimited unless the consumer +configures an active-generation limit. When a consumer enables a limit and +does not specify waiting capacity, the waiting capacity will default to 1024. +Consumers may configure a different bounded capacity, including zero when +they want no admitted backlog beyond the active-limit-sized run set. + +The public representation must distinguish an omitted waiting capacity from +an explicit zero. + +Endpoint-only profiles have no backend registration from which to obtain +policy and will remain unlimited. A future engine-wide or endpoint-keyed +policy can be considered separately if consumers demonstrate that need. + +Invalid limits or capacities will fail engine construction as invalid +configuration. Policy values will be copied into engine-owned immutable state +along with the rest of the backend registration. + +## Admission And Execution Behavior + +`Run` remains a synchronous, wait-for-result operation. Concurrent callers may +block inside `Run` while waiting for their selected backend, then receive the +ordinary result or error from that invocation. + +For a configured pool, the active-generation limit plus the waiting capacity +defines the maximum number of concurrent `Run` invocations that Promptkit will +accept for that backend. A waiting capacity of zero therefore accepts no more +runs than the active limit. Admission is immediate: a call either reserves one +of those bounded slots or receives the capacity error. An accepted run may +then wait internally for active-generation capacity. + +For a limited backend, Promptkit will bound the number of accepted runs before +expensive artifact loading, prompt rendering, and large defensive copies where +practical. Lightweight prompt, profile, and backend resolution may occur first +when it is required to identify the correct capacity pool. This pre-admission +resolution must not become a second execution-precedence path with behavior +that can drift from `Prepare`. + +An accepted run retains its admission until it completes or fails. Every +actual model-generation call for that run must separately observe the +backend's active-generation limit. This includes: + +- the initial generation; +- every output-repair generation; and +- calls made through either the built-in or an injected model client. + +Preparation and output validation should not hold an active-generation permit. +A repair remains part of its already-admitted run, but reacquires active +generation capacity so repairs cannot exceed the backend limit. It must not be +rejected merely because new runs filled the waiting queue after its initial +generation. + +Within one backend pool, waiting generation calls should be served in FIFO +order, subject to canceled calls being removed. Different backend pools make +progress independently; a saturated local backend must not consume +OpenRouter's active or waiting capacity. + +The feature will not promise ordering across backend pools or completion order +among admitted runs. + +## Capacity Failure And Cancellation + +When a backend's bounded waiting capacity is full, a new `Run` call will fail +promptly rather than waiting outside the bounded admission system. The public +API will expose a recognizable capacity-exhaustion error identity distinct +from invalid configuration, invalid requests, and model-client failures. +Rejected calls return no partial result and do not invoke the model client. + +Waiting within the admitted backlog or for active-generation capacity must +honor the caller's context. Cancellation or deadline expiry will: + +- stop waiting promptly; +- release any admission or generation capacity held by that invocation; +- preserve the applicable context error identity; and +- avoid invoking the model client if cancellation wins before generation + starts. + +Capacity must also be released after preparation, generation, validation, +repair, or collaborator failure. One failed or canceled run must not reduce +the backend's future usable capacity. + +Elapsed `Run` timing will include time spent waiting after the call is +accepted. `PreparedRun` timing will continue to describe preparation rather +than queue waiting. + +## Engine And Client Boundaries + +All pools and queued state belong to one `Engine`. Separate engines do not +share capacity, even when they register the same backend ID or endpoint. The +feature introduces no process-global scheduler. + +The engine will apply policy consistently to the built-in model client and an +injected `LLMClient`. Consumers calling their own client outside Promptkit are +outside this boundary. Injected clients remain responsible for their internal +thread safety and cancellation behavior. + +Backend policy is keyed by the resolved backend ID rather than endpoint text. +This preserves stable routing when a selected backend's endpoint is overridden +and avoids accidentally combining unrelated registrations that happen to use +the same URL. + +## Queue Lifetime And Observability + +Admission state is buffered, ephemeral, and in-process. It is not persisted +and has no survival guarantee across engine disposal or process termination. +Promptkit will not introduce background job ownership or require consumers to +start or stop workers. + +The initial feature does not require public queue-depth metrics, callbacks, or +inspection APIs. Capacity errors and ordinary call timing provide the +consumer-visible behavior. Operational observability can be added later +without coupling the scheduling mechanism to an application logging or +metrics system. + +## Compatibility + +Consumer-registered backends and endpoint-only profiles remain unlimited +unless concurrency is explicitly configured, preserving their existing +behavior. + +The built-in OpenRouter backend will change from unlimited concurrency to a +limit of 16 with a bounded waiting capacity of 1024. Ordinary synchronous +calls remain unchanged, while unusually high concurrent use may now wait or +return the capacity error. This behavioral change must be identified in the +release notes for the version that publishes it. + +Adding backend policy fields and a public capacity error is otherwise +additive. The change will use a pre-`v1` minor release under Promptkit's +[release policy](../release.md#release-model). + +## Non-Goals + +This scope does not include: + +- asynchronous job handles, polling, or detached result delivery; +- durable or cross-process queues; +- persistence or recovery across engine or process shutdown; +- priorities, scheduling weights, or consumer-defined fairness classes; +- automatic retries, backoff, rate-limit interpretation, or provider quota + discovery; +- token-per-minute or request-per-minute rate limiting; +- dynamic reconfiguration after engine construction; +- per-profile or per-run concurrency overrides; +- endpoint-keyed pooling for profiles without a backend ID; +- process-global coordination across engines; +- application worker lifecycle, logging, tracing, or metrics policy; or +- changes to prompt, profile, schema, or model-provider wire formats. + +## Target End State + +This roadmap reaches its target end state when: + +- each engine independently coordinates configured backend capacity; +- the built-in OpenRouter backend allows 16 active generations and up to 1024 + waiting runs; +- consumer backends can opt into their own active and waiting limits while + remaining unlimited by default; +- endpoint overrides retain the selected backend's capacity pool and + endpoint-only profiles remain unlimited; +- synchronous `Run` callers wait for and receive their ordinary result; +- admission is bounded before expensive preparation work where practical; +- every initial and repair generation observes the backend's active limit + without serializing preparation or validation; +- a full waiting queue returns a recognizable capacity error without invoking + the model client; +- cancellation and all failure paths promptly release capacity and preserve + context error identity; +- built-in and injected model clients receive the same scheduling behavior; +- pools remain ephemeral, engine-scoped, and independent across backend IDs; + and +- current-state GoDoc, consumer, internal, and release documentation describe + the implemented behavior once it lands. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 284bc4b..2fa6f82 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -33,41 +33,9 @@ consumers. ## Ideas -### Backend-specific concurrency management - -Extend the -[LLM backend registry](../consumers/pkg-promptkit.md#register-a-custom-backend) -with optional per-backend concurrency limits and bounded, buffered admission -queues. Promptkit could then route simultaneous -generation requests according to backend capacity while containing accidental -runaway submission. Downstream consumers would continue invoking synchronous -`Run` calls, including concurrently from multiple goroutines, and each -admitted call would wait for and return its ordinary result. - -- Scope limits to an engine instance rather than hidden process-global state. -- Give different backend IDs independent capacity pools. A profile endpoint - override would remain part of its selected backend's pool. -- Configure active concurrency and waiting capacity separately. Concurrency - protects the backend, while queue capacity protects the process from - admitting an unbounded backlog. -- Give queue capacity a generous, configurable bounded default intended as a - safety ceiling for bugs or unintended loops rather than a routine - application constraint. Select an exact default during implementation - planning and measurement. -- Reject a call with a recognizable capacity error when its backend queue is - full rather than allowing it to wait outside the bounded queue. -- Admit requests before expensive preparation and artifact copying where - practical so queued work remains lightweight. -- Apply a limit to each actual generation request, including repair attempts, - without unnecessarily serializing prompt preparation. -- Make queued and active waits respect caller cancellation and deadlines. -- Treat concurrency as backend policy rather than a profile-level model - setting. -- Keep the queue ephemeral and in-process, with no survival guarantee across - engine or process shutdown. -- Preserve the existing execution model as far as practical. Durable jobs, - polling, priorities, application worker lifecycle, retries, and - cross-process coordination would be separate future capabilities. +No ideas are currently cataloged. Backend-specific concurrency management has +been selected for active planning in the +[focused concurrency roadmap](concurrency.md). ## Entry Format diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..9da45b6 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,828 @@ +# Backend-Specific Concurrency Management Implementation Plan + +**Status:** Ready for implementation. + +## Purpose + +This document is the decision-complete implementation plan for +[backend-specific concurrency management](concurrency.md). It is written for a +coding agent that will implement each stage in order. + +The feature roadmap owns the intended capability, consumer value, policy +choices, compatibility decision, and target end state. This document owns the +concrete API, internal representation, scheduling architecture, implementation +sequence, test ownership, documentation updates, and completion gates. + +## Implementation Rules + +- Complete the stages in order. Keep the repository compiling and the focused + tests passing at every stage boundary. +- Preserve unrelated working-tree changes. In particular, `concurrency.md` and + the removal of its source idea from `future.md` may already be uncommitted + when implementation begins; retain both. +- Follow every policy under `docs/policy/`, the task-specific reading guide in + `docs/development.md`, and the target behavior in `concurrency.md`. +- Keep the public API in the root `promptkit` package and implementation + details under `internal/`. Do not expose scheduler types or create another + public package. +- Use only the Go standard library for scheduling. Do not add a queue, + semaphore, worker-pool, or metrics dependency. +- Preserve synchronous, wait-for-result `Run`, unrestricted `Prepare`, + engine-local state, endpoint-only profiles, backend-selected profiles, + backend identity through endpoint overrides, and injected `LLMClient` + behavior. +- Do not broaden the work into asynchronous jobs, durable queues, retries, + rate limiting, dynamic configuration, priorities, worker lifecycle, + endpoint-keyed pools, or public queue observability. +- Keep all tests deterministic, bounded, offline, and race-safe. Coordinate + concurrent tests with channels and barriers rather than timing assumptions + or live providers. +- Update exact GoDoc with each exported declaration change. Update durable + current-state documents only after the corresponding behavior is + implemented. +- Test configurable mechanisms with small test-owned limits. Assert the exact + OpenRouter `16` and default queue `1024` values only at the registry contract + that owns those operational defaults. +- Do not create a release, change a module version, or tag a commit. The final + implementation handoff must identify the built-in OpenRouter behavior change + for the next pre-`v1` minor release. + +## Fixed Design + +### Public Backend Configuration + +Append these fields to the existing root `Backend` type in `backends.go`: + +```go +type Backend struct { + // Existing fields remain unchanged and in their current order. + + ConcurrencyLimit int + QueueCapacity *int +} +``` + +Use these exact semantics: + +| Public values | Meaning | +| --- | --- | +| `ConcurrencyLimit == 0`, `QueueCapacity == nil` | Unlimited backend; preserve current behavior. | +| `ConcurrencyLimit > 0`, `QueueCapacity == nil` | Limit active generations and use the default waiting capacity of 1024. | +| `ConcurrencyLimit > 0`, `QueueCapacity != nil` | Limit active generations and use the pointed-to capacity exactly, including zero. | +| `ConcurrencyLimit < 0` | Invalid engine configuration. | +| `QueueCapacity != nil` and `*QueueCapacity < 0` | Invalid engine configuration. | +| `ConcurrencyLimit == 0` and `QueueCapacity != nil` | Invalid engine configuration because a queue without an active limit has no defined consumer value. | + +`ConcurrencyLimit` counts simultaneous calls to the engine-owned internal +model-client boundary for this backend. `QueueCapacity` controls additional +accepted `Run` invocations beyond that limit. The maximum admitted runs for a +limited backend is therefore: + +```text +ConcurrencyLimit + effective QueueCapacity +``` + +Guard that addition against integer overflow during backend validation. +Do not impose an arbitrary upper bound beyond non-negativity and overflow +safety. + +The `QueueCapacity` pointer exists only to distinguish omission from explicit +zero. `WithBackend` and `NewEngine` must not retain the caller's pointer. +`Backend` continues to have no stable JSON representation, and consumers +remain directed to keyed literals. + +Do not add concurrency fields to `Profile`, `ExecutionTarget`, +`ExecutionTargetOverride`, `RunRequest`, prompt or profile files, or stable +prepared/result JSON. + +### Built-In And Custom Defaults + +The backend registry owns these exact operational defaults: + +```go +const ( + openRouterConcurrencyLimit = 16 + defaultQueueCapacity = 1024 +) +``` + +The built-in `openrouter` definition has a normalized concurrency limit of 16 +and queue capacity of 1024. + +Consumer registrations remain unlimited when concurrency is omitted. For a +consumer backend with a positive limit and omitted queue capacity, normalize +the queue capacity to 1024. Preserve an explicitly configured zero. + +Consumers still cannot replace the reserved `openrouter` registration. +Endpoint-only profiles have no backend policy and remain unlimited. A selected +backend retains its pool when a profile or request overrides only its endpoint. + +### Internal Backend Representation + +Extend `internal/domain.Backend` with scalar policy values and explicit +presence rather than retaining a pointer: + +```go +type Backend struct { + // Existing fields... + ConcurrencyLimit int + QueueCapacity int + QueueCapacitySet bool +} + +type BackendCapacityPolicy struct { + ConcurrencyLimit int + QueueCapacity int +} +``` + +`WithBackend` converts the public pointer into `QueueCapacity` plus +`QueueCapacitySet`. Registry normalization validates the combinations above, +fills the default, and leaves every limited stored backend with +`QueueCapacitySet == true`. Unlimited stored backends retain zero values and +`QueueCapacitySet == false`. + +Add this internal registry method: + +```go +func (r *Registry) CapacityPolicies() map[string]domain.BackendCapacityPolicy +``` + +It returns a newly allocated map containing only limited backends. Values are +scalars, so callers cannot mutate registry state. The built-in OpenRouter +policy is included. `GetBackend` continues returning a defensive backend copy, +now including normalized scalar capacity metadata. + +Capacity policy is operational registry metadata. Do not merge it into an +execution target or expose it to injected model clients. + +### Public Capacity Error + +Add this root sentinel beside the other run errors in `engine.go`: + +```go +var ErrCapacityExceeded = errors.New("backend capacity exceeded") +``` + +Its GoDoc must state that it identifies a `Run` rejected because the selected +backend has already admitted `ConcurrencyLimit + QueueCapacity` runs. It is +not an invalid request, an LLM/provider rate-limit response, or an +`ErrLLMGenerate` failure. + +The internal capacity component owns a corresponding internal +`ErrCapacityExceeded`. Add its mapping in `publicErrorFor` before the broader +generation and invalid-request cases. The public error must preserve the +internal error through wrapping while matching `ErrCapacityExceeded` with +`errors.Is`. + +A capacity rejection returns no partial result and must not invoke the +artifact reader, renderer, schema loader, validator, or model client. Prompt, +profile, and backend loading needed to select the pool may already have +occurred. + +### Internal Capacity Component + +Add `internal/capacity` as the single owner of engine-local run admission and +active-generation permits. + +Use these package-level boundaries: + +```go +var ErrCapacityExceeded error + +type Manager struct { + // Private immutable pool map. +} + +func NewManager( + policies map[string]domain.BackendCapacityPolicy, +) (*Manager, error) + +func (m *Manager) Admit( + ctx context.Context, + backendID string, +) (release func(), err error) + +func NewClient(m *Manager, next llm.Client) llm.Client +``` + +`NewManager` copies the supplied map and creates one independent pool per +limited backend. Defensively reject blank IDs, non-positive concurrency +limits, negative queue capacities, or total-capacity overflow even though the +registry normally supplies normalized values. Construction creates no worker +goroutines. + +An absent manager, blank backend ID, or ID absent from the policy map is +unlimited: + +- `Admit` succeeds with a non-nil no-op release function; and +- the client wrapper calls the next client directly. + +For a limited pool, `Admit` is immediate and context-aware: + +1. return `ctx.Err()` if the context is already done; +2. under the pool lock, compare admitted runs with + `ConcurrencyLimit + QueueCapacity`; +3. return an error matching internal `ErrCapacityExceeded` when full; or +4. increment admitted runs and return an idempotent release function. + +The release function decrements admission exactly once, even if accidentally +called more than once. It does not release an active-generation permit; those +permits have their own lifetime. + +### FIFO Generation Permits + +`NewClient` returns an internal `llm.Client` wrapper around either the built-in +client or the public-client adapter. It must preserve requests, successful +responses, nil responses, and collaborator error identities exactly. +`next` must be non-nil; `NewEngine` and internal runner construction maintain +that invariant. A nil manager returns `next` unchanged. + +For a configured backend ID, the wrapper: + +1. acquires one active-generation permit from the matching pool; +2. waits in FIFO order when the active count equals `ConcurrencyLimit`; +3. removes a canceled waiter and returns `ctx.Err()` when cancellation wins + before the permit is granted; +4. invokes the next client only after a permit is granted; and +5. releases the permit with `defer` after every success, nil response, + collaborator error, panic unwinding, or context outcome. + +Implement FIFO and cancellation explicitly with a mutex and an ordered waiter +list. A channel used only as a counting semaphore is insufficient because it +does not define FIFO ordering or safe removal of canceled waiters. + +Permit grant and cancellation must have one lock-protected linearization +point. If cancellation removes the waiter first, do not invoke the next +client. If grant wins first, invoke the next client with the caller's context; +the next client may then observe cancellation normally. Never lose or +double-release a permit in this race. + +Releasing a permit transfers it to the oldest non-canceled waiter before +making it generally available. Different backend pools never share admission +or active counts. + +The active wrapper enforces its limit even if an internal caller invokes it +without a run admission lease. Bounded backlog is guaranteed for ordinary +engine `Run` calls by the runner admission path; no public API exposes the +wrapped internal client directly. + +### Engine Assembly + +In `NewEngine`, after constructing the validated backend registry: + +1. obtain `backendRegistry.CapacityPolicies()`; +2. construct one `capacity.Manager`; +3. construct the selected base internal LLM client exactly as today; +4. wrap that base client with `capacity.NewClient`; and +5. pass both the wrapped client and manager-as-admitter to the runner. + +Every `NewEngine` call constructs a distinct manager. Do not cache managers, +pools, or policies in package globals. The wrapper must be applied after a +public injected client is adapted to `internal/llm.Client`, so built-in and +injected clients receive identical scheduling behavior. + +If `NewManager` reports a defensive configuration error, make `NewEngine` +return an error matching `ErrInvalidConfig`. + +`Prepare` does not use the manager. An injected client remains required to be +safe for concurrent calls because different backend pools and unlimited +backends may still invoke it concurrently. + +### Shared Two-Phase Preparation + +Refactor `internal/usecase.Runner` so `Prepare` and `Run` share one preparation +pipeline with two private phases. Do not duplicate prompt/profile/backend +selection or execution precedence. + +The first phase resolves only the state required before admission: + +1. validate `PromptID`; +2. normalize the direct session ID; +3. load the prompt definition; +4. hash the original prompt definition at its existing error-order position; +5. select and load the execution profile; +6. resolve the selected backend; +7. resolve and validate the effective execution target and credentials; and +8. resolve the effective output contract without loading its schema. + +Return a private state value containing the loaded definition, normalized +direct session, prompt-definition hash, selected profile ID, effective target, +numeric-presence metadata, effective output contract, and preparation start +time. Keep this value private to `internal/usecase`. + +The second phase consumes that state and performs: + +1. structured-output schema loading; +2. artifact loading and input hashing; +3. message and prompt-session rendering; +4. direct-session application; +5. rendered-prompt hashing; and +6. `PreparedRun` construction and timing. + +Preserve every existing precedence rule, error identity, direct-session +template bypass, hash input, selected identity, copy guarantee, and timing +field. Do not reload the prompt, profile, or backend between phases. + +`Runner.Prepare` records its start time, runs both phases consecutively, and +never calls admission. Its behavior and error ordering remain unchanged. + +`Runner.Run` records its existing run start time, runs the first preparation +phase, and then calls: + +```go +release, err := r.admitter.Admit(ctx, effectiveBackendID) +``` + +Use a narrow use-case-owned interface with the same signature: + +```go +type RunAdmitter interface { + Admit(context.Context, string) (func(), error) +} +``` + +A nil admitter means unlimited behavior for internal constructors and tests. +On successful admission, immediately `defer release()` around the remainder of +the run. Then run the second preparation phase, initial generation, +validation, and all repair attempts. + +If admission returns internal `capacity.ErrCapacityExceeded`, add useful +backend context without changing its identity. If it returns `ctx.Err()`, +preserve that identity directly rather than recategorizing it as invalid +request or generation failure. + +This refactor intentionally replaces the current literal `Run`-calls-`Prepare` +implementation with shared private phases. Update current-state documentation +to describe one shared pipeline rather than retaining an inaccurate call-graph +claim. + +### Generation And Repair Lifetime + +The admission lease covers the entire accepted run: + +- second-phase preparation; +- initial generation; +- validation; +- every repair; and +- all failure and cancellation exits. + +Preparation and validation do not hold an active-generation permit. The +wrapped client acquires a permit only around each actual `Generate` call. + +The runner's initial generation already carries the effective backend ID in +`GenerateRequest.Target`. Preserve that value. `RepairRequest.Target` and the +default repairer's generated request must continue carrying the same backend +ID, allowing each repair to reacquire the same pool's active permit. + +When testing or constructing `NewRunnerWithRepairer`, pass the same wrapped +client to both the runner and `NewDefaultOutputRepairer`. Do not add capacity +state to `RepairRequest`, `ExecutionTarget`, or public generation values. + +A repair remains within its existing admission lease. It waits for a FIFO +active permit but never performs a second bounded admission and therefore +cannot fail merely because later runs filled the admission capacity. + +### Error And Cancellation Semantics + +The required public outcomes are: + +| Situation | Required error identity | +| --- | --- | +| Admission capacity is full | `ErrCapacityExceeded` only; not `ErrInvalidRequest` or `ErrLLMGenerate`. | +| Context is done before admission succeeds | Preserve `ctx.Err()`; do not return capacity exhaustion. | +| Context cancels while waiting for an active permit | Preserve `ctx.Err()` through the existing `ErrLLMGenerate` generation category. | +| Wrapped client fails after permit acquisition | Preserve existing `ErrLLMGenerate` and collaborator identities. | +| Preparation or validation fails after admission | Preserve its existing category and release admission. | + +Maintain the existing rule that `Run` returns no partial result on any +operational error. Do not add queue status to errors or results. + +`RunResult.Duration` continues to start at runner entry and therefore includes +pre-admission resolution, accepted preparation, and active-permit waiting. +`PreparedRun.DurationMS` continues to cover only its shared preparation phases; +it does not include later generation waiting. Capacity-rejected calls have no +result or timing value. + +### Ownership And Concurrency Safety + +The registry, capacity policy map, pool map, and per-pool limits are immutable +after engine construction. Only admission counts, active counts, and waiter +lists are mutable and must be protected by the owning pool mutex. + +Do not retain public queue pointers, caller request values, contexts, or +generation requests after their call completes. A canceled waiter must be +unlinked so its context and request cannot remain reachable from the pool. + +Do not hold a pool mutex while: + +- loading or rendering prompts; +- reading artifacts or schemas; +- invoking a model client; +- validating output; +- closing a waiter notification channel if the implementation could re-enter + pool code; or +- calling consumer code. + +No scheduler operation may spawn a goroutine whose lifetime outlasts the +calling `Run`. The zero steady-state goroutine count is part of the +in-process/no-worker-lifecycle design. + +## Test Ownership + +Use this ownership split and avoid repeating the full policy matrix at every +layer: + +- `internal/backend/registry_test.go` owns normalization, validation, the exact + OpenRouter policy, the custom default queue, explicit zero, unlimited + omission, and policy-map copying. +- `internal/capacity/manager_test.go` owns admission bounds, idempotent release, + FIFO active permits, cancellation races, capacity recovery, independent + pools, unlimited IDs, and observed peak concurrency. +- `internal/capacity/client_test.go` owns wrapper request/response/error + transparency and the rule that cancellation before grant does not invoke the + next client. Combine these with manager tests if one coherent package test + expresses the behavior more clearly. +- `internal/usecase/runner_test.go` owns two-phase preparation parity, pool + selection, admission before expensive work, admission release across run + exits, `Prepare` bypass, and repair reuse of the admitted backend. +- Root external-package tests own public configuration conversion, assembled + engine-local behavior, endpoint-override routing, injected-client limiting, + and public capacity/context error identities. +- Existing model-client HTTP tests remain unchanged because scheduling does + not alter the OpenAI-compatible wire contract. + +Concurrency tests must use test-owned limits such as one or two and +channel-controlled blocking clients. Record observed active and peak counts +under a mutex or atomics. Do not use `time.Sleep` to infer queue state. +Package-internal tests may inspect a waiter list under its mutex through a +small test helper when necessary to establish deterministic FIFO ordering; do +not add production metrics or hooks solely for tests. + +Do not add separate tests for trivial scalar copies when registry or assembled +behavior already protects them. + +## Stage 1 — Backend Policy And Public Configuration + +**Status:** Pending. + +### Goal + +Add the public and internal backend policy representation, normalize all +configured states, and expose immutable normalized policies without changing +runtime scheduling yet. + +### Work + +1. Add `ConcurrencyLimit` and `QueueCapacity` to `Backend` in `backends.go` + with exact GoDoc for unlimited, defaulted, explicit-zero, invalid, and + engine-scoped behavior. +2. Convert the public queue pointer into scalar value plus presence in + `WithBackend`; do not retain the pointer. +3. Add the internal backend policy fields and + `BackendCapacityPolicy` to `internal/domain/domain.go`. +4. Add the two registry-owned constants and configure the built-in OpenRouter + definition with 16 and 1024. +5. Extend `normalizeBackend` with the fixed validation, defaulting, explicit + zero, and overflow rules. +6. Add `Registry.CapacityPolicies`, returning only limited policies in a fresh + map. +7. Update existing backend composite literals and assertions only where the + new fields are relevant. Continue using keyed literals. + +### Tests + +1. Extend the exact built-in registry test with the OpenRouter limit and queue. +2. Add one coherent table covering unlimited omission, default queue, + explicit-zero queue, negative values, queue-without-limit, and total + overflow. +3. Extend the registry copy/normalization test to prove returned policy maps + cannot mutate registry state. +4. Add root coverage only if needed to prove the public pointer/presence + conversion; do not reproduce registry validation cases at the facade. + +### Focused Validation + +Run: + +```sh +gofmt -w backends.go internal/domain/domain.go \ + internal/backend/registry.go internal/backend/registry_test.go +go test . ./internal/backend +go vet . ./internal/backend +git diff --check +``` + +Include another touched Go test file in `gofmt` only if it actually changed. + +### Completion Gate + +This stage is complete when every public configuration state has one normalized +internal meaning, OpenRouter exposes exactly 16/1024, custom backends remain +unlimited by omission, and no runtime call is scheduled yet. + +## Stage 2 — Engine-Local Capacity Manager + +**Status:** Pending. + +### Goal + +Implement and prove the bounded admission mechanism and FIFO active-generation +client wrapper independently of runner orchestration. + +### Work + +1. Add `internal/capacity/manager.go` with the manager, immutable policy copy, + per-backend pools, internal error, immediate admission, idempotent release, + and FIFO context-aware active permits. +2. Add `internal/capacity/client.go` with the transparent `llm.Client` wrapper. +3. Use mutex-protected waiter state and an ordered list; explicitly resolve + grant-versus-cancel races. +4. Ensure unlimited and independent-pool fast paths avoid queue allocation. +5. Do not start workers, timers, cleanup goroutines, or process-global state. + +### Tests + +1. Add a compact constructor-validation table for blank IDs, non-positive + limits, negative queues, and total-capacity overflow. +2. With a small configured policy, prove that exactly + `limit + queueCapacity` admissions succeed, the next matches + `ErrCapacityExceeded`, and a release permits another admission. +3. Prove release is idempotent. +4. Drive more blocked client calls than the active limit and assert observed + peak concurrency never exceeds that limit. +5. Prove FIFO order with deterministic queue-entry synchronization. +6. Cancel the first and a middle waiter and prove they are removed, never call + the wrapped client, and do not block later waiters. +7. Exercise the grant/cancel race repeatedly under `go test -race`, asserting + no permit leak or double invocation. +8. Prove different backend IDs proceed independently and blank, unknown, or + nil-manager paths remain unlimited. +9. Prove request values, successful and nil responses, and collaborator errors + pass through unchanged after permit acquisition. + +### Focused Validation + +Run: + +```sh +gofmt -w internal/capacity/manager.go \ + internal/capacity/manager_test.go \ + internal/capacity/client.go \ + internal/capacity/client_test.go +go test ./internal/capacity +go test -race ./internal/capacity +go vet ./internal/capacity +git diff --check +``` + +If tests are combined into one file, omit the nonexistent file from `gofmt`. + +### Completion Gate + +This stage is complete when the standalone component enforces relational +admission and active limits, FIFO cancellation is race-safe, separate pools +are independent, and the wrapper is transparent apart from waiting. + +## Stage 3 — Shared Preparation And Early Run Admission + +**Status:** Pending. + +### Goal + +Refactor runner preparation into one shared two-phase pipeline and place +bounded admission after backend resolution but before schema, artifact, and +rendering work. + +### Work + +1. Add the private pre-admission preparation state and split the existing + `Prepare` logic according to the fixed design. +2. Make `Runner.Prepare` call both phases without an admitter. +3. Add the `RunAdmitter` interface and runner field. +4. Update `NewRunner` and `NewRunnerWithRepairer` to accept the optional + admitter; update internal call sites with nil until root assembly is wired. +5. Change `Runner.Run` to use the first phase, admit by effective backend ID, + defer the returned release, and then use the second phase. +6. Preserve all existing error precedence, target resolution, hashes, + metadata, session behavior, and timing. +7. Return capacity and context errors with the fixed identities. Do not invoke + later collaborators after rejection. + +### Tests + +1. Keep the existing `Run`/`Prepare` parity coverage passing to prove the + shared phases do not drift. +2. Add a fake admitter that records backend IDs and release calls. +3. Prove a backend-selected run admits with the selected ID even when the + endpoint is overridden. +4. Prove an endpoint-only run uses the unlimited/blank identity and that + `Prepare` never calls admission. +5. Reject admission and assert schema, artifact, renderer, validator, repairer, + and LLM collaborators are not invoked. +6. Prove admission is released after one successful run and representative + second-phase, generation, and validation errors. Prefer a small table around + the single `defer` invariant rather than duplicating every error test. +7. Retain direct-session, backend precedence, credential, hashing, and repair + tests unchanged except for constructor arguments. + +### Focused Validation + +Run: + +```sh +gofmt -w internal/usecase/runner.go \ + internal/usecase/runner_test.go +go test ./internal/usecase +go test -race ./internal/usecase +go vet ./internal/usecase +git diff --check +``` + +### Completion Gate + +This stage is complete when `Prepare` remains unrestricted, `Run` admits after +one canonical routing phase and before expensive completion work, every exit +releases admission, and existing preparation semantics remain unchanged. + +## Stage 4 — Engine Assembly And Public Runtime Contract + +**Status:** Pending. + +### Goal + +Wire one manager into each engine, schedule built-in and injected clients, +expose the capacity error, and prove assembled runtime behavior. + +### Work + +1. Add public `ErrCapacityExceeded` and its exact GoDoc in `engine.go`. +2. Map internal capacity exhaustion in `errors.go`. +3. Construct the manager from the registry policy snapshot in `NewEngine`. +4. Wrap the selected internal client after built-in or injected-client + selection and pass the manager and wrapped client to the runner. +5. Update `Engine`, `NewEngine`, `Run`, `WithLLMClient`, and `LLMClient` GoDoc + only where concurrency, capacity, or cancellation statements change. +6. Ensure manager-construction errors match `ErrInvalidConfig`. +7. For internal repair coverage, construct the default repairer with the same + wrapped client used by its runner and confirm repair target backend identity + remains intact. + +### Tests + +1. Add an external-package assembled test with a small custom limit and a + blocking injected client; assert peak generation equals or remains below + the configured limit. +2. With queue capacity zero, block one accepted run before generation and + assert the next matching-backend run returns `ErrCapacityExceeded`, does not + match `ErrInvalidRequest` or `ErrLLMGenerate`, returns no result, and never + reaches expensive collaborators or the client. +3. In the same or another focused workflow, prove an endpoint override remains + in the selected backend's pool. +4. Prove two engines with the same backend ID have independent capacity. +5. Prove an unlimited custom backend and an endpoint-only profile preserve + concurrent behavior. +6. Cancel a call waiting for an active permit; assert it matches both + `context.Canceled` and `ErrLLMGenerate`, never invokes the injected client, + and leaves capacity reusable. +7. Add one internal repair workflow with concurrent runs or controlled permits + showing initial and repair generations never exceed the same backend limit + and repairs do not perform a second admission. +8. Extend the public error sentinel contract test with + `ErrCapacityExceeded`. + +Avoid a second HTTP-level concurrency suite: the capacity client tests and one +assembled injected-client workflow already protect the shared wrapper used by +the built-in client. + +### Focused Validation + +Run: + +```sh +gofmt -w engine.go errors.go backends.go \ + internal/usecase/runner.go internal/usecase/runner_test.go \ + engine_test.go public_contract_test.go +go test . ./internal/backend ./internal/capacity ./internal/usecase +go test -race . ./internal/capacity ./internal/usecase +go vet . ./internal/backend ./internal/capacity ./internal/usecase +git diff --check +``` + +Add any newly created capacity files to `gofmt` when they changed in this +stage. + +### Completion Gate + +This stage is complete when every engine has independent pools, limited runs +are bounded and FIFO at generation, endpoint routing is correct, capacity and +context errors are stable, repairs reuse admission, and both client kinds pass +through the same wrapper. + +## Stage 5 — Durable Documentation And Final Validation + +**Status:** Pending. + +### Goal + +Move implemented contracts into their durable owners, record compatibility +impact, and validate the complete repository. + +### Work + +1. Review every changed exported declaration. Ensure GoDoc is the canonical + owner of exact field types, nil/zero semantics, defaulting, error identity, + engine scope, concurrency safety, cancellation, and source compatibility. +2. Update `doc.go` so its concurrency summary acknowledges backend scheduling + while continuing to require injected collaborators to be concurrency-safe. +3. Update `docs/consumers/pkg-promptkit.md` with task-oriented examples for: + - a limited local backend; + - omitted queue capacity selecting 1024; + - explicit zero queue capacity; and + - handling `ErrCapacityExceeded`. + Keep exact field semantics in GoDoc rather than duplicating a full table. +4. Add `docs/internal/capacity.md` as the durable owner of pool lifecycle, + admission, FIFO active permits, cancellation, client wrapping, and test + ownership. +5. Add `internal/capacity` to `docs/internal/overview.md`. +6. Update `docs/policy/architecture.md` to include the implemented component + and root assembly dependency without turning policy into an API reference. +7. Update `docs/internal/runner.md` to describe the shared two-phase + preparation pipeline, early bounded admission, lease lifetime, generation + permits, repairs, capacity failures, and cancellation. +8. Review `docs/formats.md`; add only a concise link or clarification if needed + to explain that endpoint overrides preserve backend capacity identity. + Do not add concurrency fields to YAML. +9. Do not change the OpenAI-compatible integration contract or + `docs/internal/llm.md` unless implementation changes their current + statements; scheduling is outside the provider wire contract and concrete + model-client implementation. +10. Record in the implementation handoff that built-in OpenRouter now limits + active generations to 16 with queue capacity 1024 and that the release + must be a pre-`v1` minor release. Do not edit the release procedure or + create a tag. +11. After every check passes, set `concurrency.md`, this implementation plan, + and each stage status to `Complete`. Do not remove the roadmaps in the + implementation change; lifecycle retirement follows review. + +### Full Validation + +Run the complete sequence from `docs/development.md`: + +```sh +go test ./... +go test -race ./... +go vet ./... +go build ./... +go run ./examples/go-library/prepare +gofmt -l $(git ls-files '*.go') +git diff --check +``` + +The formatting command must produce no paths. Follow every added or changed +Markdown link and confirm its target and heading exist. + +Also inspect: + +```sh +git status --short +git diff --stat +git diff +``` + +Confirm that: + +- only intended backend, capacity, runner, facade, test, documentation, and + roadmap files changed; +- no `go.work`, `go.work.sum`, local module replacement, credential, + generated binary, coverage output, or unrelated change was introduced; +- the built-in OpenRouter policy is exactly 16/1024; +- custom and endpoint-only backends remain unlimited by omission; +- explicit queue zero is distinguishable from omission; +- no capacity value enters execution targets, generated requests, stable JSON, + prompt/profile YAML, or provider payloads; +- every engine owns distinct pools with no package-global mutable state; +- every initial and repair generation uses the active permit wrapper; +- capacity and waiter state is released on success, error, panic unwinding, + and cancellation; +- concurrency tests use deterministic coordination rather than sleeps; +- current-state documentation describes implemented behavior rather than + referring readers to the roadmaps; and +- the feature and implementation roadmaps contain no unresolved work marked + complete. + +### Completion Gate + +The implementation is complete only when every target-end-state item in +`concurrency.md` is implemented, race-enabled tests demonstrate the configured +limits and cancellation safety, durable contracts no longer depend on roadmap +prose, and the OpenRouter compatibility change is clearly reported for the +next minor release. + +## Open Questions + +None. The feature roadmap and this plan fix the public representation, +registry defaults, admission bound, FIFO generation behavior, early-routing +refactor, cancellation races, error identities, engine and repair lifetimes, +test ownership, compatibility treatment, and non-goals required for +implementation.