promptkit/docs/roadmap/implementation.md

Backend-Specific Concurrency Management Implementation Plan

Status: Complete.

Purpose

This document is the decision-complete implementation plan for backend-specific concurrency management. It is written for a coding agent that will implement each stage in order.

The feature roadmap owns the intended capability, consumer value, policy choices, compatibility decision, and target end state. This document owns the concrete API, internal representation, scheduling architecture, implementation sequence, test ownership, documentation updates, and completion gates.

Implementation Rules

• Complete the stages in order. Keep the repository compiling and the focused tests passing at every stage boundary.
• Preserve unrelated working-tree changes. In particular, concurrency.md and the removal of its source idea from future.md may already be uncommitted when implementation begins; retain both.
• Follow every policy under docs/policy/, the task-specific reading guide in docs/development.md, and the target behavior in concurrency.md.
• Keep the public API in the root promptkit package and implementation details under internal/. Do not expose scheduler types or create another public package.
• Use only the Go standard library for scheduling. Do not add a queue, semaphore, worker-pool, or metrics dependency.
• Preserve synchronous, wait-for-result Run, unrestricted Prepare, engine-local state, endpoint-only profiles, backend-selected profiles, backend identity through endpoint overrides, and injected LLMClient behavior.
• Do not broaden the work into asynchronous jobs, durable queues, retries, rate limiting, dynamic configuration, priorities, worker lifecycle, endpoint-keyed pools, or public queue observability.
• Keep all tests deterministic, bounded, offline, and race-safe. Coordinate concurrent tests with channels and barriers rather than timing assumptions or live providers.
• Update exact GoDoc with each exported declaration change. Update durable current-state documents only after the corresponding behavior is implemented.
• Test configurable mechanisms with small test-owned limits. Assert the exact OpenRouter 16 and default queue 1024 values only at the registry contract that owns those operational defaults.
• Do not create a release, change a module version, or tag a commit. The final implementation handoff must identify the built-in OpenRouter behavior change for the next pre-v1 minor release.

Fixed Design

Public Backend Configuration

Append these fields to the existing root Backend type in backends.go:

type Backend struct {
	// Existing fields remain unchanged and in their current order.

	ConcurrencyLimit int
	QueueCapacity    *int
}

Use these exact semantics:

Public values	Meaning
ConcurrencyLimit == 0, QueueCapacity == nil	Unlimited backend; preserve current behavior.
ConcurrencyLimit > 0, QueueCapacity == nil	Limit active generations and use the default waiting capacity of 1024.
ConcurrencyLimit > 0, QueueCapacity != nil	Limit active generations and use the pointed-to capacity exactly, including zero.
ConcurrencyLimit < 0	Invalid engine configuration.
QueueCapacity != nil and *QueueCapacity < 0	Invalid engine configuration.
ConcurrencyLimit == 0 and QueueCapacity != nil	Invalid engine configuration because a queue without an active limit has no defined consumer value.

ConcurrencyLimit counts simultaneous calls to the engine-owned internal model-client boundary for this backend. QueueCapacity controls additional accepted Run invocations beyond that limit. The maximum admitted runs for a limited backend is therefore:

ConcurrencyLimit + effective QueueCapacity

Guard that addition against integer overflow during backend validation. Do not impose an arbitrary upper bound beyond non-negativity and overflow safety.

The QueueCapacity pointer exists only to distinguish omission from explicit zero. WithBackend and NewEngine must not retain the caller's pointer. Backend continues to have no stable JSON representation, and consumers remain directed to keyed literals.

Do not add concurrency fields to Profile, ExecutionTarget, ExecutionTargetOverride, RunRequest, prompt or profile files, or stable prepared/result JSON.

Built-In And Custom Defaults

The backend registry owns these exact operational defaults:

const (
	openRouterConcurrencyLimit = 16
	defaultQueueCapacity       = 1024
)

The built-in openrouter definition has a normalized concurrency limit of 16 and queue capacity of 1024.

Consumer registrations remain unlimited when concurrency is omitted. For a consumer backend with a positive limit and omitted queue capacity, normalize the queue capacity to 1024. Preserve an explicitly configured zero.

Consumers still cannot replace the reserved openrouter registration. Endpoint-only profiles have no backend policy and remain unlimited. A selected backend retains its pool when a profile or request overrides only its endpoint.

Internal Backend Representation

Extend internal/domain.Backend with scalar policy values and explicit presence rather than retaining a pointer:

type Backend struct {
	// Existing fields...
	ConcurrencyLimit int
	QueueCapacity    int
	QueueCapacitySet bool
}

type BackendCapacityPolicy struct {
	ConcurrencyLimit int
	QueueCapacity    int
}

WithBackend converts the public pointer into QueueCapacity plus QueueCapacitySet. Registry normalization validates the combinations above, fills the default, and leaves every limited stored backend with QueueCapacitySet == true. Unlimited stored backends retain zero values and QueueCapacitySet == false.

Add this internal registry method:

func (r *Registry) CapacityPolicies() map[string]domain.BackendCapacityPolicy

It returns a newly allocated map containing only limited backends. Values are scalars, so callers cannot mutate registry state. The built-in OpenRouter policy is included. GetBackend continues returning a defensive backend copy, now including normalized scalar capacity metadata.

Capacity policy is operational registry metadata. Do not merge it into an execution target or expose it to injected model clients.

Public Capacity Error

Add this root sentinel beside the other run errors in engine.go:

var ErrCapacityExceeded = errors.New("backend capacity exceeded")

Its GoDoc must state that it identifies a Run rejected because the selected backend has already admitted ConcurrencyLimit + QueueCapacity runs. It is not an invalid request, an LLM/provider rate-limit response, or an ErrLLMGenerate failure.

The internal capacity component owns a corresponding internal ErrCapacityExceeded. Add its mapping in publicErrorFor before the broader generation and invalid-request cases. The public error must preserve the internal error through wrapping while matching ErrCapacityExceeded with errors.Is.

A capacity rejection returns no partial result and must not invoke the artifact reader, renderer, schema loader, validator, or model client. Prompt, profile, and backend loading needed to select the pool may already have occurred.

Internal Capacity Component

Add internal/capacity as the single owner of engine-local run admission and active-generation permits.

Use these package-level boundaries:

var ErrCapacityExceeded error

type Manager struct {
	// Private immutable pool map.
}

func NewManager(
	policies map[string]domain.BackendCapacityPolicy,
) (*Manager, error)

func (m *Manager) Admit(
	ctx context.Context,
	backendID string,
) (release func(), err error)

func NewClient(m *Manager, next llm.Client) llm.Client

NewManager copies the supplied map and creates one independent pool per limited backend. Defensively reject blank IDs, non-positive concurrency limits, negative queue capacities, or total-capacity overflow even though the registry normally supplies normalized values. Construction creates no worker goroutines.

An absent manager, blank backend ID, or ID absent from the policy map is unlimited:

• Admit succeeds with a non-nil no-op release function; and
• the client wrapper calls the next client directly.

For a limited pool, Admit is immediate and context-aware:

1. return ctx.Err() if the context is already done;
2. under the pool lock, compare admitted runs with ConcurrencyLimit + QueueCapacity;
3. return an error matching internal ErrCapacityExceeded when full; or
4. increment admitted runs and return an idempotent release function.

The release function decrements admission exactly once, even if accidentally called more than once. It does not release an active-generation permit; those permits have their own lifetime.

FIFO Generation Permits

NewClient returns an internal llm.Client wrapper around either the built-in client or the public-client adapter. It must preserve requests, successful responses, nil responses, and collaborator error identities exactly. next must be non-nil; NewEngine and internal runner construction maintain that invariant. A nil manager returns next unchanged.

For a configured backend ID, the wrapper:

1. acquires one active-generation permit from the matching pool;
2. waits in FIFO order when the active count equals ConcurrencyLimit;
3. removes a canceled waiter and returns ctx.Err() when cancellation wins before the permit is granted;
4. invokes the next client only after a permit is granted; and
5. releases the permit with defer after every success, nil response, collaborator error, panic unwinding, or context outcome.

Implement FIFO and cancellation explicitly with a mutex and an ordered waiter list. A channel used only as a counting semaphore is insufficient because it does not define FIFO ordering or safe removal of canceled waiters.

Permit grant and cancellation must have one lock-protected linearization point. If cancellation removes the waiter first, do not invoke the next client. If grant wins first, invoke the next client with the caller's context; the next client may then observe cancellation normally. Never lose or double-release a permit in this race.

Releasing a permit transfers it to the oldest non-canceled waiter before making it generally available. Different backend pools never share admission or active counts.

The active wrapper enforces its limit even if an internal caller invokes it without a run admission lease. Bounded backlog is guaranteed for ordinary engine Run calls by the runner admission path; no public API exposes the wrapped internal client directly.

Engine Assembly

In NewEngine, after constructing the validated backend registry:

1. obtain backendRegistry.CapacityPolicies();
2. construct one capacity.Manager;
3. construct the selected base internal LLM client exactly as today;
4. wrap that base client with capacity.NewClient; and
5. pass both the wrapped client and manager-as-admitter to the runner.

Every NewEngine call constructs a distinct manager. Do not cache managers, pools, or policies in package globals. The wrapper must be applied after a public injected client is adapted to internal/llm.Client, so built-in and injected clients receive identical scheduling behavior.

If NewManager reports a defensive configuration error, make NewEngine return an error matching ErrInvalidConfig.

Prepare does not use the manager. An injected client remains required to be safe for concurrent calls because different backend pools and unlimited backends may still invoke it concurrently.

Shared Two-Phase Preparation

Refactor internal/usecase.Runner so Prepare and Run share one preparation pipeline with two private phases. Do not duplicate prompt/profile/backend selection or execution precedence.

The first phase resolves only the state required before admission:

1. validate PromptID;
2. normalize the direct session ID;
3. load the prompt definition;
4. hash the original prompt definition at its existing error-order position;
5. select and load the execution profile;
6. resolve the selected backend;
7. resolve and validate the effective execution target and credentials; and
8. resolve the effective output contract without loading its schema.

Return a private state value containing the loaded definition, normalized direct session, prompt-definition hash, selected profile ID, effective target, numeric-presence metadata, effective output contract, and preparation start time. Keep this value private to internal/usecase.

The second phase consumes that state and performs:

1. structured-output schema loading;
2. artifact loading and input hashing;
3. message and prompt-session rendering;
4. direct-session application;
5. rendered-prompt hashing; and
6. PreparedRun construction and timing.

Preserve every existing precedence rule, error identity, direct-session template bypass, hash input, selected identity, copy guarantee, and timing field. Do not reload the prompt, profile, or backend between phases.

Runner.Prepare records its start time, runs both phases consecutively, and never calls admission. Its behavior and error ordering remain unchanged.

Runner.Run records its existing run start time, runs the first preparation phase, and then calls:

release, err := r.admitter.Admit(ctx, effectiveBackendID)

Use a narrow use-case-owned interface with the same signature:

type RunAdmitter interface {
	Admit(context.Context, string) (func(), error)
}

A nil admitter means unlimited behavior for internal constructors and tests. On successful admission, immediately defer release() around the remainder of the run. Then run the second preparation phase, initial generation, validation, and all repair attempts.

If admission returns internal capacity.ErrCapacityExceeded, add useful backend context without changing its identity. If it returns ctx.Err(), preserve that identity directly rather than recategorizing it as invalid request or generation failure.

This refactor intentionally replaces the current literal Run-calls-Prepare implementation with shared private phases. Update current-state documentation to describe one shared pipeline rather than retaining an inaccurate call-graph claim.

Generation And Repair Lifetime

The admission lease covers the entire accepted run:

• second-phase preparation;
• initial generation;
• validation;
• every repair; and
• all failure and cancellation exits.

Preparation and validation do not hold an active-generation permit. The wrapped client acquires a permit only around each actual Generate call.

The runner's initial generation already carries the effective backend ID in GenerateRequest.Target. Preserve that value. RepairRequest.Target and the default repairer's generated request must continue carrying the same backend ID, allowing each repair to reacquire the same pool's active permit.

When testing or constructing NewRunnerWithRepairer, pass the same wrapped client to both the runner and NewDefaultOutputRepairer. Do not add capacity state to RepairRequest, ExecutionTarget, or public generation values.

A repair remains within its existing admission lease. It waits for a FIFO active permit but never performs a second bounded admission and therefore cannot fail merely because later runs filled the admission capacity.

Error And Cancellation Semantics

The required public outcomes are:

Situation	Required error identity
Admission capacity is full	ErrCapacityExceeded only; not ErrInvalidRequest or ErrLLMGenerate.
Context is done before admission succeeds	Preserve ctx.Err(); do not return capacity exhaustion.
Context cancels while waiting for an active permit	Preserve ctx.Err() through the existing ErrLLMGenerate generation category.
Wrapped client fails after permit acquisition	Preserve existing ErrLLMGenerate and collaborator identities.
Preparation or validation fails after admission	Preserve its existing category and release admission.

Maintain the existing rule that Run returns no partial result on any operational error. Do not add queue status to errors or results.

RunResult.Duration continues to start at runner entry and therefore includes pre-admission resolution, accepted preparation, and active-permit waiting. PreparedRun.DurationMS continues to cover only its shared preparation phases; it does not include later generation waiting. Capacity-rejected calls have no result or timing value.

Ownership And Concurrency Safety

The registry, capacity policy map, pool map, and per-pool limits are immutable after engine construction. Only admission counts, active counts, and waiter lists are mutable and must be protected by the owning pool mutex.

Do not retain public queue pointers, caller request values, contexts, or generation requests after their call completes. A canceled waiter must be unlinked so its context and request cannot remain reachable from the pool.

Do not hold a pool mutex while:

• loading or rendering prompts;
• reading artifacts or schemas;
• invoking a model client;
• validating output;
• closing a waiter notification channel if the implementation could re-enter pool code; or
• calling consumer code.

No scheduler operation may spawn a goroutine whose lifetime outlasts the calling Run. The zero steady-state goroutine count is part of the in-process/no-worker-lifecycle design.

Test Ownership

Use this ownership split and avoid repeating the full policy matrix at every layer:

• internal/backend/registry_test.go owns normalization, validation, the exact OpenRouter policy, the custom default queue, explicit zero, unlimited omission, and policy-map copying.
• internal/capacity/manager_test.go owns admission bounds, idempotent release, FIFO active permits, cancellation races, capacity recovery, independent pools, unlimited IDs, and observed peak concurrency.
• internal/capacity/client_test.go owns wrapper request/response/error transparency and the rule that cancellation before grant does not invoke the next client. Combine these with manager tests if one coherent package test expresses the behavior more clearly.
• internal/usecase/runner_test.go owns two-phase preparation parity, pool selection, admission before expensive work, admission release across run exits, Prepare bypass, and repair reuse of the admitted backend.
• Root external-package tests own public configuration conversion, assembled engine-local behavior, endpoint-override routing, injected-client limiting, and public capacity/context error identities.
• Existing model-client HTTP tests remain unchanged because scheduling does not alter the OpenAI-compatible wire contract.

Concurrency tests must use test-owned limits such as one or two and channel-controlled blocking clients. Record observed active and peak counts under a mutex or atomics. Do not use time.Sleep to infer queue state. Package-internal tests may inspect a waiter list under its mutex through a small test helper when necessary to establish deterministic FIFO ordering; do not add production metrics or hooks solely for tests.

Do not add separate tests for trivial scalar copies when registry or assembled behavior already protects them.

Stage 1 — Backend Policy And Public Configuration

Status: Complete.

Goal

Add the public and internal backend policy representation, normalize all configured states, and expose immutable normalized policies without changing runtime scheduling yet.

Work

1. Add ConcurrencyLimit and QueueCapacity to Backend in backends.go with exact GoDoc for unlimited, defaulted, explicit-zero, invalid, and engine-scoped behavior.
2. Convert the public queue pointer into scalar value plus presence in WithBackend; do not retain the pointer.
3. Add the internal backend policy fields and BackendCapacityPolicy to internal/domain/domain.go.
4. Add the two registry-owned constants and configure the built-in OpenRouter definition with 16 and 1024.
5. Extend normalizeBackend with the fixed validation, defaulting, explicit zero, and overflow rules.
6. Add Registry.CapacityPolicies, returning only limited policies in a fresh map.
7. Update existing backend composite literals and assertions only where the new fields are relevant. Continue using keyed literals.

Tests

1. Extend the exact built-in registry test with the OpenRouter limit and queue.
2. Add one coherent table covering unlimited omission, default queue, explicit-zero queue, negative values, queue-without-limit, and total overflow.
3. Extend the registry copy/normalization test to prove returned policy maps cannot mutate registry state.
4. Add root coverage only if needed to prove the public pointer/presence conversion; do not reproduce registry validation cases at the facade.

Focused Validation

Run:

gofmt -w backends.go internal/domain/domain.go \
  internal/backend/registry.go internal/backend/registry_test.go
go test . ./internal/backend
go vet . ./internal/backend
git diff --check

Include another touched Go test file in gofmt only if it actually changed.

Completion Gate

This stage is complete when every public configuration state has one normalized internal meaning, OpenRouter exposes exactly 16/1024, custom backends remain unlimited by omission, and no runtime call is scheduled yet.

Stage 2 — Engine-Local Capacity Manager

Status: Complete.

Goal

Implement and prove the bounded admission mechanism and FIFO active-generation client wrapper independently of runner orchestration.

Work

1. Add internal/capacity/manager.go with the manager, immutable policy copy, per-backend pools, internal error, immediate admission, idempotent release, and FIFO context-aware active permits.
2. Add internal/capacity/client.go with the transparent llm.Client wrapper.
3. Use mutex-protected waiter state and an ordered list; explicitly resolve grant-versus-cancel races.
4. Ensure unlimited and independent-pool fast paths avoid queue allocation.
5. Do not start workers, timers, cleanup goroutines, or process-global state.

Tests

1. Add a compact constructor-validation table for blank IDs, non-positive limits, negative queues, and total-capacity overflow.
2. With a small configured policy, prove that exactly limit + queueCapacity admissions succeed, the next matches ErrCapacityExceeded, and a release permits another admission.
3. Prove release is idempotent.
4. Drive more blocked client calls than the active limit and assert observed peak concurrency never exceeds that limit.
5. Prove FIFO order with deterministic queue-entry synchronization.
6. Cancel the first and a middle waiter and prove they are removed, never call the wrapped client, and do not block later waiters.
7. Exercise the grant/cancel race repeatedly under go test -race, asserting no permit leak or double invocation.
8. Prove different backend IDs proceed independently and blank, unknown, or nil-manager paths remain unlimited.
9. Prove request values, successful and nil responses, and collaborator errors pass through unchanged after permit acquisition.

Focused Validation

Run:

gofmt -w internal/capacity/manager.go \
  internal/capacity/manager_test.go \
  internal/capacity/client.go \
  internal/capacity/client_test.go
go test ./internal/capacity
go test -race ./internal/capacity
go vet ./internal/capacity
git diff --check

If tests are combined into one file, omit the nonexistent file from gofmt.

Completion Gate

This stage is complete when the standalone component enforces relational admission and active limits, FIFO cancellation is race-safe, separate pools are independent, and the wrapper is transparent apart from waiting.

Stage 3 — Shared Preparation And Early Run Admission

Status: Complete.

Goal

Refactor runner preparation into one shared two-phase pipeline and place bounded admission after backend resolution but before schema, artifact, and rendering work.

Work

1. Add the private pre-admission preparation state and split the existing Prepare logic according to the fixed design.
2. Make Runner.Prepare call both phases without an admitter.
3. Add the RunAdmitter interface and runner field.
4. Update NewRunner and NewRunnerWithRepairer to accept the optional admitter; update internal call sites with nil until root assembly is wired.
5. Change Runner.Run to use the first phase, admit by effective backend ID, defer the returned release, and then use the second phase.
6. Preserve all existing error precedence, target resolution, hashes, metadata, session behavior, and timing.
7. Return capacity and context errors with the fixed identities. Do not invoke later collaborators after rejection.

Tests

1. Keep the existing Run/Prepare parity coverage passing to prove the shared phases do not drift.
2. Add a fake admitter that records backend IDs and release calls.
3. Prove a backend-selected run admits with the selected ID even when the endpoint is overridden.
4. Prove an endpoint-only run uses the unlimited/blank identity and that Prepare never calls admission.
5. Reject admission and assert schema, artifact, renderer, validator, repairer, and LLM collaborators are not invoked.
6. Prove admission is released after one successful run and representative second-phase, generation, and validation errors. Prefer a small table around the single defer invariant rather than duplicating every error test.
7. Retain direct-session, backend precedence, credential, hashing, and repair tests unchanged except for constructor arguments.

Focused Validation

Run:

gofmt -w internal/usecase/runner.go \
  internal/usecase/runner_test.go
go test ./internal/usecase
go test -race ./internal/usecase
go vet ./internal/usecase
git diff --check

Completion Gate

This stage is complete when Prepare remains unrestricted, Run admits after one canonical routing phase and before expensive completion work, every exit releases admission, and existing preparation semantics remain unchanged.

Stage 4 — Engine Assembly And Public Runtime Contract

Status: Complete.

Goal

Wire one manager into each engine, schedule built-in and injected clients, expose the capacity error, and prove assembled runtime behavior.

Work

1. Add public ErrCapacityExceeded and its exact GoDoc in engine.go.
2. Map internal capacity exhaustion in errors.go.
3. Construct the manager from the registry policy snapshot in NewEngine.
4. Wrap the selected internal client after built-in or injected-client selection and pass the manager and wrapped client to the runner.
5. Update Engine, NewEngine, Run, WithLLMClient, and LLMClient GoDoc only where concurrency, capacity, or cancellation statements change.
6. Ensure manager-construction errors match ErrInvalidConfig.
7. For internal repair coverage, construct the default repairer with the same wrapped client used by its runner and confirm repair target backend identity remains intact.

Tests

1. Add an external-package assembled test with a small custom limit and a blocking injected client; assert peak generation equals or remains below the configured limit.
2. With queue capacity zero, block one accepted run before generation and assert the next matching-backend run returns ErrCapacityExceeded, does not match ErrInvalidRequest or ErrLLMGenerate, returns no result, and never reaches expensive collaborators or the client.
3. In the same or another focused workflow, prove an endpoint override remains in the selected backend's pool.
4. Prove two engines with the same backend ID have independent capacity.
5. Prove an unlimited custom backend and an endpoint-only profile preserve concurrent behavior.
6. Cancel a call waiting for an active permit; assert it matches both context.Canceled and ErrLLMGenerate, never invokes the injected client, and leaves capacity reusable.
7. Add one internal repair workflow with concurrent runs or controlled permits showing initial and repair generations never exceed the same backend limit and repairs do not perform a second admission.
8. Extend the public error sentinel contract test with ErrCapacityExceeded.

Avoid a second HTTP-level concurrency suite: the capacity client tests and one assembled injected-client workflow already protect the shared wrapper used by the built-in client.

Focused Validation

Run:

gofmt -w engine.go errors.go backends.go \
  internal/usecase/runner.go internal/usecase/runner_test.go \
  engine_test.go public_contract_test.go
go test . ./internal/backend ./internal/capacity ./internal/usecase
go test -race . ./internal/capacity ./internal/usecase
go vet . ./internal/backend ./internal/capacity ./internal/usecase
git diff --check

Add any newly created capacity files to gofmt when they changed in this stage.

Completion Gate

This stage is complete when every engine has independent pools, limited runs are bounded and FIFO at generation, endpoint routing is correct, capacity and context errors are stable, repairs reuse admission, and both client kinds pass through the same wrapper.

Stage 5 — Durable Documentation And Final Validation

Status: Complete.

Goal

Move implemented contracts into their durable owners, record compatibility impact, and validate the complete repository.

Work

1. Review every changed exported declaration. Ensure GoDoc is the canonical owner of exact field types, nil/zero semantics, defaulting, error identity, engine scope, concurrency safety, cancellation, and source compatibility.
2. Update doc.go so its concurrency summary acknowledges backend scheduling while continuing to require injected collaborators to be concurrency-safe.
3. Update docs/consumers/pkg-promptkit.md with task-oriented examples for:
   • a limited local backend;
   • omitted queue capacity selecting 1024;
   • explicit zero queue capacity; and
   • handling ErrCapacityExceeded. Keep exact field semantics in GoDoc rather than duplicating a full table.
4. Add docs/internal/capacity.md as the durable owner of pool lifecycle, admission, FIFO active permits, cancellation, client wrapping, and test ownership.
5. Add internal/capacity to docs/internal/overview.md.
6. Update docs/policy/architecture.md to include the implemented component and root assembly dependency without turning policy into an API reference.
7. Update docs/internal/runner.md to describe the shared two-phase preparation pipeline, early bounded admission, lease lifetime, generation permits, repairs, capacity failures, and cancellation.
8. Review docs/formats.md; add only a concise link or clarification if needed to explain that endpoint overrides preserve backend capacity identity. Do not add concurrency fields to YAML.
9. Do not change the OpenAI-compatible integration contract or docs/internal/llm.md unless implementation changes their current statements; scheduling is outside the provider wire contract and concrete model-client implementation.
10. Record in the implementation handoff that built-in OpenRouter now limits active generations to 16 with queue capacity 1024 and that the release must be a pre-v1 minor release. Do not edit the release procedure or create a tag.
11. After every check passes, set concurrency.md, this implementation plan, and each stage status to Complete. Do not remove the roadmaps in the implementation change; lifecycle retirement follows review.

Full Validation

Run the complete sequence from docs/development.md:

go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
gofmt -l $(git ls-files '*.go')
git diff --check

The formatting command must produce no paths. Follow every added or changed Markdown link and confirm its target and heading exist.

Also inspect:

git status --short
git diff --stat
git diff

Confirm that:

• only intended backend, capacity, runner, facade, test, documentation, and roadmap files changed;
• no go.work, go.work.sum, local module replacement, credential, generated binary, coverage output, or unrelated change was introduced;
• the built-in OpenRouter policy is exactly 16/1024;
• custom and endpoint-only backends remain unlimited by omission;
• explicit queue zero is distinguishable from omission;
• no capacity value enters execution targets, generated requests, stable JSON, prompt/profile YAML, or provider payloads;
• every engine owns distinct pools with no package-global mutable state;
• every initial and repair generation uses the active permit wrapper;
• capacity and waiter state is released on success, error, panic unwinding, and cancellation;
• concurrency tests use deterministic coordination rather than sleeps;
• current-state documentation describes implemented behavior rather than referring readers to the roadmaps; and
• the feature and implementation roadmaps contain no unresolved work marked complete.

Completion Gate

The implementation is complete only when every target-end-state item in concurrency.md is implemented, race-enabled tests demonstrate the configured limits and cancellation safety, durable contracts no longer depend on roadmap prose, and the OpenRouter compatibility change is clearly reported for the next minor release.

Implementation Handoff

Backend-specific capacity management is implemented and has passed the complete repository validation sequence. The built-in OpenRouter backend now permits 16 active generations and a waiting capacity of 1024. Custom backends remain unlimited when their limit is omitted, and endpoint-only profiles remain unlimited.

Publishing this behavior requires a pre-v1 minor release. Its release notes must identify that unusually high concurrent OpenRouter use can now wait or return ErrCapacityExceeded. This implementation does not change a module version or create a tag.

Open Questions

None. The feature roadmap and this plan fix the public representation, registry defaults, admission bound, FIFO generation behavior, early-routing refactor, cancellation races, error identities, engine and repair lifetimes, test ownership, compatibility treatment, and non-goals required for implementation.