Add roadmap and implementation plan for backend concurrency scaling
This commit is contained in:
236
docs/roadmap/concurrency.md
Normal file
236
docs/roadmap/concurrency.md
Normal file
@@ -0,0 +1,236 @@
|
||||
# Backend-Specific Concurrency Management
|
||||
|
||||
**Status:** Accepted.
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the scope and target end state for engine-local,
|
||||
backend-specific concurrency management. It records the intended capability,
|
||||
consumer value, and important policy choices.
|
||||
|
||||
This document is planning material, not a description of current behavior.
|
||||
Current exported contracts remain owned by Go declarations and GoDoc, backend
|
||||
registration guidance by the
|
||||
[consumer guide](../consumers/pkg-promptkit.md#register-a-custom-backend), and
|
||||
implemented orchestration by the
|
||||
[internal runner document](../internal/runner.md).
|
||||
|
||||
## Motivation
|
||||
|
||||
Different model backends can sustain very different request loads. A local
|
||||
network endpoint may need a small concurrency limit, while OpenRouter can
|
||||
usually accept substantially more simultaneous work. Requiring every consumer
|
||||
to build its own semaphores and queues would duplicate routing knowledge,
|
||||
create inconsistent cancellation behavior, and make it easy for one caller to
|
||||
bypass the intended backend limit.
|
||||
|
||||
Promptkit should own this coordination because it already resolves each run to
|
||||
an engine-scoped backend identity and owns every model-generation call made by
|
||||
the runner. Consumers should continue submitting ready-to-run requests through
|
||||
the synchronous API, including concurrently from multiple goroutines, without
|
||||
implementing their own backend scheduler.
|
||||
|
||||
The buffered queue is a safety boundary, not an ordinary throughput
|
||||
restriction. Its primary purpose is to prevent a bug or unintended submission
|
||||
loop from creating an unbounded in-memory backlog.
|
||||
|
||||
## Scope
|
||||
|
||||
The feature will add optional concurrency policy to registered backends and
|
||||
coordinate `Run` calls against independent per-backend capacity pools.
|
||||
|
||||
Each policy has two distinct controls:
|
||||
|
||||
- an active-generation limit, which protects the backend from too many
|
||||
simultaneous model requests; and
|
||||
- a bounded waiting capacity, which protects the process from admitting an
|
||||
unbounded backlog.
|
||||
|
||||
Concurrency policy belongs to a backend registration. It is not a profile
|
||||
model parameter and cannot be overridden per run. Profiles select the policy
|
||||
through their backend ID, while a profile or request endpoint override remains
|
||||
in the selected backend's pool.
|
||||
|
||||
`Prepare` does not call a model and will remain outside concurrency admission.
|
||||
|
||||
## Defaults And Configuration
|
||||
|
||||
The built-in OpenRouter backend will use:
|
||||
|
||||
- an active-generation limit of 16; and
|
||||
- a waiting capacity of 1024.
|
||||
|
||||
The waiting default is intentionally generous. Reaching it should indicate
|
||||
abnormal submission pressure rather than normal application behavior.
|
||||
|
||||
Consumer-registered backends will remain unlimited unless the consumer
|
||||
configures an active-generation limit. When a consumer enables a limit and
|
||||
does not specify waiting capacity, the waiting capacity will default to 1024.
|
||||
Consumers may configure a different bounded capacity, including zero when
|
||||
they want no admitted backlog beyond the active-limit-sized run set.
|
||||
|
||||
The public representation must distinguish an omitted waiting capacity from
|
||||
an explicit zero.
|
||||
|
||||
Endpoint-only profiles have no backend registration from which to obtain
|
||||
policy and will remain unlimited. A future engine-wide or endpoint-keyed
|
||||
policy can be considered separately if consumers demonstrate that need.
|
||||
|
||||
Invalid limits or capacities will fail engine construction as invalid
|
||||
configuration. Policy values will be copied into engine-owned immutable state
|
||||
along with the rest of the backend registration.
|
||||
|
||||
## Admission And Execution Behavior
|
||||
|
||||
`Run` remains a synchronous, wait-for-result operation. Concurrent callers may
|
||||
block inside `Run` while waiting for their selected backend, then receive the
|
||||
ordinary result or error from that invocation.
|
||||
|
||||
For a configured pool, the active-generation limit plus the waiting capacity
|
||||
defines the maximum number of concurrent `Run` invocations that Promptkit will
|
||||
accept for that backend. A waiting capacity of zero therefore accepts no more
|
||||
runs than the active limit. Admission is immediate: a call either reserves one
|
||||
of those bounded slots or receives the capacity error. An accepted run may
|
||||
then wait internally for active-generation capacity.
|
||||
|
||||
For a limited backend, Promptkit will bound the number of accepted runs before
|
||||
expensive artifact loading, prompt rendering, and large defensive copies where
|
||||
practical. Lightweight prompt, profile, and backend resolution may occur first
|
||||
when it is required to identify the correct capacity pool. This pre-admission
|
||||
resolution must not become a second execution-precedence path with behavior
|
||||
that can drift from `Prepare`.
|
||||
|
||||
An accepted run retains its admission until it completes or fails. Every
|
||||
actual model-generation call for that run must separately observe the
|
||||
backend's active-generation limit. This includes:
|
||||
|
||||
- the initial generation;
|
||||
- every output-repair generation; and
|
||||
- calls made through either the built-in or an injected model client.
|
||||
|
||||
Preparation and output validation should not hold an active-generation permit.
|
||||
A repair remains part of its already-admitted run, but reacquires active
|
||||
generation capacity so repairs cannot exceed the backend limit. It must not be
|
||||
rejected merely because new runs filled the waiting queue after its initial
|
||||
generation.
|
||||
|
||||
Within one backend pool, waiting generation calls should be served in FIFO
|
||||
order, subject to canceled calls being removed. Different backend pools make
|
||||
progress independently; a saturated local backend must not consume
|
||||
OpenRouter's active or waiting capacity.
|
||||
|
||||
The feature will not promise ordering across backend pools or completion order
|
||||
among admitted runs.
|
||||
|
||||
## Capacity Failure And Cancellation
|
||||
|
||||
When a backend's bounded waiting capacity is full, a new `Run` call will fail
|
||||
promptly rather than waiting outside the bounded admission system. The public
|
||||
API will expose a recognizable capacity-exhaustion error identity distinct
|
||||
from invalid configuration, invalid requests, and model-client failures.
|
||||
Rejected calls return no partial result and do not invoke the model client.
|
||||
|
||||
Waiting within the admitted backlog or for active-generation capacity must
|
||||
honor the caller's context. Cancellation or deadline expiry will:
|
||||
|
||||
- stop waiting promptly;
|
||||
- release any admission or generation capacity held by that invocation;
|
||||
- preserve the applicable context error identity; and
|
||||
- avoid invoking the model client if cancellation wins before generation
|
||||
starts.
|
||||
|
||||
Capacity must also be released after preparation, generation, validation,
|
||||
repair, or collaborator failure. One failed or canceled run must not reduce
|
||||
the backend's future usable capacity.
|
||||
|
||||
Elapsed `Run` timing will include time spent waiting after the call is
|
||||
accepted. `PreparedRun` timing will continue to describe preparation rather
|
||||
than queue waiting.
|
||||
|
||||
## Engine And Client Boundaries
|
||||
|
||||
All pools and queued state belong to one `Engine`. Separate engines do not
|
||||
share capacity, even when they register the same backend ID or endpoint. The
|
||||
feature introduces no process-global scheduler.
|
||||
|
||||
The engine will apply policy consistently to the built-in model client and an
|
||||
injected `LLMClient`. Consumers calling their own client outside Promptkit are
|
||||
outside this boundary. Injected clients remain responsible for their internal
|
||||
thread safety and cancellation behavior.
|
||||
|
||||
Backend policy is keyed by the resolved backend ID rather than endpoint text.
|
||||
This preserves stable routing when a selected backend's endpoint is overridden
|
||||
and avoids accidentally combining unrelated registrations that happen to use
|
||||
the same URL.
|
||||
|
||||
## Queue Lifetime And Observability
|
||||
|
||||
Admission state is buffered, ephemeral, and in-process. It is not persisted
|
||||
and has no survival guarantee across engine disposal or process termination.
|
||||
Promptkit will not introduce background job ownership or require consumers to
|
||||
start or stop workers.
|
||||
|
||||
The initial feature does not require public queue-depth metrics, callbacks, or
|
||||
inspection APIs. Capacity errors and ordinary call timing provide the
|
||||
consumer-visible behavior. Operational observability can be added later
|
||||
without coupling the scheduling mechanism to an application logging or
|
||||
metrics system.
|
||||
|
||||
## Compatibility
|
||||
|
||||
Consumer-registered backends and endpoint-only profiles remain unlimited
|
||||
unless concurrency is explicitly configured, preserving their existing
|
||||
behavior.
|
||||
|
||||
The built-in OpenRouter backend will change from unlimited concurrency to a
|
||||
limit of 16 with a bounded waiting capacity of 1024. Ordinary synchronous
|
||||
calls remain unchanged, while unusually high concurrent use may now wait or
|
||||
return the capacity error. This behavioral change must be identified in the
|
||||
release notes for the version that publishes it.
|
||||
|
||||
Adding backend policy fields and a public capacity error is otherwise
|
||||
additive. The change will use a pre-`v1` minor release under Promptkit's
|
||||
[release policy](../release.md#release-model).
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This scope does not include:
|
||||
|
||||
- asynchronous job handles, polling, or detached result delivery;
|
||||
- durable or cross-process queues;
|
||||
- persistence or recovery across engine or process shutdown;
|
||||
- priorities, scheduling weights, or consumer-defined fairness classes;
|
||||
- automatic retries, backoff, rate-limit interpretation, or provider quota
|
||||
discovery;
|
||||
- token-per-minute or request-per-minute rate limiting;
|
||||
- dynamic reconfiguration after engine construction;
|
||||
- per-profile or per-run concurrency overrides;
|
||||
- endpoint-keyed pooling for profiles without a backend ID;
|
||||
- process-global coordination across engines;
|
||||
- application worker lifecycle, logging, tracing, or metrics policy; or
|
||||
- changes to prompt, profile, schema, or model-provider wire formats.
|
||||
|
||||
## Target End State
|
||||
|
||||
This roadmap reaches its target end state when:
|
||||
|
||||
- each engine independently coordinates configured backend capacity;
|
||||
- the built-in OpenRouter backend allows 16 active generations and up to 1024
|
||||
waiting runs;
|
||||
- consumer backends can opt into their own active and waiting limits while
|
||||
remaining unlimited by default;
|
||||
- endpoint overrides retain the selected backend's capacity pool and
|
||||
endpoint-only profiles remain unlimited;
|
||||
- synchronous `Run` callers wait for and receive their ordinary result;
|
||||
- admission is bounded before expensive preparation work where practical;
|
||||
- every initial and repair generation observes the backend's active limit
|
||||
without serializing preparation or validation;
|
||||
- a full waiting queue returns a recognizable capacity error without invoking
|
||||
the model client;
|
||||
- cancellation and all failure paths promptly release capacity and preserve
|
||||
context error identity;
|
||||
- built-in and injected model clients receive the same scheduling behavior;
|
||||
- pools remain ephemeral, engine-scoped, and independent across backend IDs;
|
||||
and
|
||||
- current-state GoDoc, consumer, internal, and release documentation describe
|
||||
the implemented behavior once it lands.
|
||||
@@ -33,41 +33,9 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
### Backend-specific concurrency management
|
||||
|
||||
Extend the
|
||||
[LLM backend registry](../consumers/pkg-promptkit.md#register-a-custom-backend)
|
||||
with optional per-backend concurrency limits and bounded, buffered admission
|
||||
queues. Promptkit could then route simultaneous
|
||||
generation requests according to backend capacity while containing accidental
|
||||
runaway submission. Downstream consumers would continue invoking synchronous
|
||||
`Run` calls, including concurrently from multiple goroutines, and each
|
||||
admitted call would wait for and return its ordinary result.
|
||||
|
||||
- Scope limits to an engine instance rather than hidden process-global state.
|
||||
- Give different backend IDs independent capacity pools. A profile endpoint
|
||||
override would remain part of its selected backend's pool.
|
||||
- Configure active concurrency and waiting capacity separately. Concurrency
|
||||
protects the backend, while queue capacity protects the process from
|
||||
admitting an unbounded backlog.
|
||||
- Give queue capacity a generous, configurable bounded default intended as a
|
||||
safety ceiling for bugs or unintended loops rather than a routine
|
||||
application constraint. Select an exact default during implementation
|
||||
planning and measurement.
|
||||
- Reject a call with a recognizable capacity error when its backend queue is
|
||||
full rather than allowing it to wait outside the bounded queue.
|
||||
- Admit requests before expensive preparation and artifact copying where
|
||||
practical so queued work remains lightweight.
|
||||
- Apply a limit to each actual generation request, including repair attempts,
|
||||
without unnecessarily serializing prompt preparation.
|
||||
- Make queued and active waits respect caller cancellation and deadlines.
|
||||
- Treat concurrency as backend policy rather than a profile-level model
|
||||
setting.
|
||||
- Keep the queue ephemeral and in-process, with no survival guarantee across
|
||||
engine or process shutdown.
|
||||
- Preserve the existing execution model as far as practical. Durable jobs,
|
||||
polling, priorities, application worker lifecycle, retries, and
|
||||
cross-process coordination would be separate future capabilities.
|
||||
No ideas are currently cataloged. Backend-specific concurrency management has
|
||||
been selected for active planning in the
|
||||
[focused concurrency roadmap](concurrency.md).
|
||||
|
||||
## Entry Format
|
||||
|
||||
|
||||
828
docs/roadmap/implementation.md
Normal file
828
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,828 @@
|
||||
# Backend-Specific Concurrency Management Implementation Plan
|
||||
|
||||
**Status:** Ready for implementation.
|
||||
|
||||
## Purpose
|
||||
|
||||
This document is the decision-complete implementation plan for
|
||||
[backend-specific concurrency management](concurrency.md). It is written for a
|
||||
coding agent that will implement each stage in order.
|
||||
|
||||
The feature roadmap owns the intended capability, consumer value, policy
|
||||
choices, compatibility decision, and target end state. This document owns the
|
||||
concrete API, internal representation, scheduling architecture, implementation
|
||||
sequence, test ownership, documentation updates, and completion gates.
|
||||
|
||||
## Implementation Rules
|
||||
|
||||
- Complete the stages in order. Keep the repository compiling and the focused
|
||||
tests passing at every stage boundary.
|
||||
- Preserve unrelated working-tree changes. In particular, `concurrency.md` and
|
||||
the removal of its source idea from `future.md` may already be uncommitted
|
||||
when implementation begins; retain both.
|
||||
- Follow every policy under `docs/policy/`, the task-specific reading guide in
|
||||
`docs/development.md`, and the target behavior in `concurrency.md`.
|
||||
- Keep the public API in the root `promptkit` package and implementation
|
||||
details under `internal/`. Do not expose scheduler types or create another
|
||||
public package.
|
||||
- Use only the Go standard library for scheduling. Do not add a queue,
|
||||
semaphore, worker-pool, or metrics dependency.
|
||||
- Preserve synchronous, wait-for-result `Run`, unrestricted `Prepare`,
|
||||
engine-local state, endpoint-only profiles, backend-selected profiles,
|
||||
backend identity through endpoint overrides, and injected `LLMClient`
|
||||
behavior.
|
||||
- Do not broaden the work into asynchronous jobs, durable queues, retries,
|
||||
rate limiting, dynamic configuration, priorities, worker lifecycle,
|
||||
endpoint-keyed pools, or public queue observability.
|
||||
- Keep all tests deterministic, bounded, offline, and race-safe. Coordinate
|
||||
concurrent tests with channels and barriers rather than timing assumptions
|
||||
or live providers.
|
||||
- Update exact GoDoc with each exported declaration change. Update durable
|
||||
current-state documents only after the corresponding behavior is
|
||||
implemented.
|
||||
- Test configurable mechanisms with small test-owned limits. Assert the exact
|
||||
OpenRouter `16` and default queue `1024` values only at the registry contract
|
||||
that owns those operational defaults.
|
||||
- Do not create a release, change a module version, or tag a commit. The final
|
||||
implementation handoff must identify the built-in OpenRouter behavior change
|
||||
for the next pre-`v1` minor release.
|
||||
|
||||
## Fixed Design
|
||||
|
||||
### Public Backend Configuration
|
||||
|
||||
Append these fields to the existing root `Backend` type in `backends.go`:
|
||||
|
||||
```go
|
||||
type Backend struct {
|
||||
// Existing fields remain unchanged and in their current order.
|
||||
|
||||
ConcurrencyLimit int
|
||||
QueueCapacity *int
|
||||
}
|
||||
```
|
||||
|
||||
Use these exact semantics:
|
||||
|
||||
| Public values | Meaning |
|
||||
| --- | --- |
|
||||
| `ConcurrencyLimit == 0`, `QueueCapacity == nil` | Unlimited backend; preserve current behavior. |
|
||||
| `ConcurrencyLimit > 0`, `QueueCapacity == nil` | Limit active generations and use the default waiting capacity of 1024. |
|
||||
| `ConcurrencyLimit > 0`, `QueueCapacity != nil` | Limit active generations and use the pointed-to capacity exactly, including zero. |
|
||||
| `ConcurrencyLimit < 0` | Invalid engine configuration. |
|
||||
| `QueueCapacity != nil` and `*QueueCapacity < 0` | Invalid engine configuration. |
|
||||
| `ConcurrencyLimit == 0` and `QueueCapacity != nil` | Invalid engine configuration because a queue without an active limit has no defined consumer value. |
|
||||
|
||||
`ConcurrencyLimit` counts simultaneous calls to the engine-owned internal
|
||||
model-client boundary for this backend. `QueueCapacity` controls additional
|
||||
accepted `Run` invocations beyond that limit. The maximum admitted runs for a
|
||||
limited backend is therefore:
|
||||
|
||||
```text
|
||||
ConcurrencyLimit + effective QueueCapacity
|
||||
```
|
||||
|
||||
Guard that addition against integer overflow during backend validation.
|
||||
Do not impose an arbitrary upper bound beyond non-negativity and overflow
|
||||
safety.
|
||||
|
||||
The `QueueCapacity` pointer exists only to distinguish omission from explicit
|
||||
zero. `WithBackend` and `NewEngine` must not retain the caller's pointer.
|
||||
`Backend` continues to have no stable JSON representation, and consumers
|
||||
remain directed to keyed literals.
|
||||
|
||||
Do not add concurrency fields to `Profile`, `ExecutionTarget`,
|
||||
`ExecutionTargetOverride`, `RunRequest`, prompt or profile files, or stable
|
||||
prepared/result JSON.
|
||||
|
||||
### Built-In And Custom Defaults
|
||||
|
||||
The backend registry owns these exact operational defaults:
|
||||
|
||||
```go
|
||||
const (
|
||||
openRouterConcurrencyLimit = 16
|
||||
defaultQueueCapacity = 1024
|
||||
)
|
||||
```
|
||||
|
||||
The built-in `openrouter` definition has a normalized concurrency limit of 16
|
||||
and queue capacity of 1024.
|
||||
|
||||
Consumer registrations remain unlimited when concurrency is omitted. For a
|
||||
consumer backend with a positive limit and omitted queue capacity, normalize
|
||||
the queue capacity to 1024. Preserve an explicitly configured zero.
|
||||
|
||||
Consumers still cannot replace the reserved `openrouter` registration.
|
||||
Endpoint-only profiles have no backend policy and remain unlimited. A selected
|
||||
backend retains its pool when a profile or request overrides only its endpoint.
|
||||
|
||||
### Internal Backend Representation
|
||||
|
||||
Extend `internal/domain.Backend` with scalar policy values and explicit
|
||||
presence rather than retaining a pointer:
|
||||
|
||||
```go
|
||||
type Backend struct {
|
||||
// Existing fields...
|
||||
ConcurrencyLimit int
|
||||
QueueCapacity int
|
||||
QueueCapacitySet bool
|
||||
}
|
||||
|
||||
type BackendCapacityPolicy struct {
|
||||
ConcurrencyLimit int
|
||||
QueueCapacity int
|
||||
}
|
||||
```
|
||||
|
||||
`WithBackend` converts the public pointer into `QueueCapacity` plus
|
||||
`QueueCapacitySet`. Registry normalization validates the combinations above,
|
||||
fills the default, and leaves every limited stored backend with
|
||||
`QueueCapacitySet == true`. Unlimited stored backends retain zero values and
|
||||
`QueueCapacitySet == false`.
|
||||
|
||||
Add this internal registry method:
|
||||
|
||||
```go
|
||||
func (r *Registry) CapacityPolicies() map[string]domain.BackendCapacityPolicy
|
||||
```
|
||||
|
||||
It returns a newly allocated map containing only limited backends. Values are
|
||||
scalars, so callers cannot mutate registry state. The built-in OpenRouter
|
||||
policy is included. `GetBackend` continues returning a defensive backend copy,
|
||||
now including normalized scalar capacity metadata.
|
||||
|
||||
Capacity policy is operational registry metadata. Do not merge it into an
|
||||
execution target or expose it to injected model clients.
|
||||
|
||||
### Public Capacity Error
|
||||
|
||||
Add this root sentinel beside the other run errors in `engine.go`:
|
||||
|
||||
```go
|
||||
var ErrCapacityExceeded = errors.New("backend capacity exceeded")
|
||||
```
|
||||
|
||||
Its GoDoc must state that it identifies a `Run` rejected because the selected
|
||||
backend has already admitted `ConcurrencyLimit + QueueCapacity` runs. It is
|
||||
not an invalid request, an LLM/provider rate-limit response, or an
|
||||
`ErrLLMGenerate` failure.
|
||||
|
||||
The internal capacity component owns a corresponding internal
|
||||
`ErrCapacityExceeded`. Add its mapping in `publicErrorFor` before the broader
|
||||
generation and invalid-request cases. The public error must preserve the
|
||||
internal error through wrapping while matching `ErrCapacityExceeded` with
|
||||
`errors.Is`.
|
||||
|
||||
A capacity rejection returns no partial result and must not invoke the
|
||||
artifact reader, renderer, schema loader, validator, or model client. Prompt,
|
||||
profile, and backend loading needed to select the pool may already have
|
||||
occurred.
|
||||
|
||||
### Internal Capacity Component
|
||||
|
||||
Add `internal/capacity` as the single owner of engine-local run admission and
|
||||
active-generation permits.
|
||||
|
||||
Use these package-level boundaries:
|
||||
|
||||
```go
|
||||
var ErrCapacityExceeded error
|
||||
|
||||
type Manager struct {
|
||||
// Private immutable pool map.
|
||||
}
|
||||
|
||||
func NewManager(
|
||||
policies map[string]domain.BackendCapacityPolicy,
|
||||
) (*Manager, error)
|
||||
|
||||
func (m *Manager) Admit(
|
||||
ctx context.Context,
|
||||
backendID string,
|
||||
) (release func(), err error)
|
||||
|
||||
func NewClient(m *Manager, next llm.Client) llm.Client
|
||||
```
|
||||
|
||||
`NewManager` copies the supplied map and creates one independent pool per
|
||||
limited backend. Defensively reject blank IDs, non-positive concurrency
|
||||
limits, negative queue capacities, or total-capacity overflow even though the
|
||||
registry normally supplies normalized values. Construction creates no worker
|
||||
goroutines.
|
||||
|
||||
An absent manager, blank backend ID, or ID absent from the policy map is
|
||||
unlimited:
|
||||
|
||||
- `Admit` succeeds with a non-nil no-op release function; and
|
||||
- the client wrapper calls the next client directly.
|
||||
|
||||
For a limited pool, `Admit` is immediate and context-aware:
|
||||
|
||||
1. return `ctx.Err()` if the context is already done;
|
||||
2. under the pool lock, compare admitted runs with
|
||||
`ConcurrencyLimit + QueueCapacity`;
|
||||
3. return an error matching internal `ErrCapacityExceeded` when full; or
|
||||
4. increment admitted runs and return an idempotent release function.
|
||||
|
||||
The release function decrements admission exactly once, even if accidentally
|
||||
called more than once. It does not release an active-generation permit; those
|
||||
permits have their own lifetime.
|
||||
|
||||
### FIFO Generation Permits
|
||||
|
||||
`NewClient` returns an internal `llm.Client` wrapper around either the built-in
|
||||
client or the public-client adapter. It must preserve requests, successful
|
||||
responses, nil responses, and collaborator error identities exactly.
|
||||
`next` must be non-nil; `NewEngine` and internal runner construction maintain
|
||||
that invariant. A nil manager returns `next` unchanged.
|
||||
|
||||
For a configured backend ID, the wrapper:
|
||||
|
||||
1. acquires one active-generation permit from the matching pool;
|
||||
2. waits in FIFO order when the active count equals `ConcurrencyLimit`;
|
||||
3. removes a canceled waiter and returns `ctx.Err()` when cancellation wins
|
||||
before the permit is granted;
|
||||
4. invokes the next client only after a permit is granted; and
|
||||
5. releases the permit with `defer` after every success, nil response,
|
||||
collaborator error, panic unwinding, or context outcome.
|
||||
|
||||
Implement FIFO and cancellation explicitly with a mutex and an ordered waiter
|
||||
list. A channel used only as a counting semaphore is insufficient because it
|
||||
does not define FIFO ordering or safe removal of canceled waiters.
|
||||
|
||||
Permit grant and cancellation must have one lock-protected linearization
|
||||
point. If cancellation removes the waiter first, do not invoke the next
|
||||
client. If grant wins first, invoke the next client with the caller's context;
|
||||
the next client may then observe cancellation normally. Never lose or
|
||||
double-release a permit in this race.
|
||||
|
||||
Releasing a permit transfers it to the oldest non-canceled waiter before
|
||||
making it generally available. Different backend pools never share admission
|
||||
or active counts.
|
||||
|
||||
The active wrapper enforces its limit even if an internal caller invokes it
|
||||
without a run admission lease. Bounded backlog is guaranteed for ordinary
|
||||
engine `Run` calls by the runner admission path; no public API exposes the
|
||||
wrapped internal client directly.
|
||||
|
||||
### Engine Assembly
|
||||
|
||||
In `NewEngine`, after constructing the validated backend registry:
|
||||
|
||||
1. obtain `backendRegistry.CapacityPolicies()`;
|
||||
2. construct one `capacity.Manager`;
|
||||
3. construct the selected base internal LLM client exactly as today;
|
||||
4. wrap that base client with `capacity.NewClient`; and
|
||||
5. pass both the wrapped client and manager-as-admitter to the runner.
|
||||
|
||||
Every `NewEngine` call constructs a distinct manager. Do not cache managers,
|
||||
pools, or policies in package globals. The wrapper must be applied after a
|
||||
public injected client is adapted to `internal/llm.Client`, so built-in and
|
||||
injected clients receive identical scheduling behavior.
|
||||
|
||||
If `NewManager` reports a defensive configuration error, make `NewEngine`
|
||||
return an error matching `ErrInvalidConfig`.
|
||||
|
||||
`Prepare` does not use the manager. An injected client remains required to be
|
||||
safe for concurrent calls because different backend pools and unlimited
|
||||
backends may still invoke it concurrently.
|
||||
|
||||
### Shared Two-Phase Preparation
|
||||
|
||||
Refactor `internal/usecase.Runner` so `Prepare` and `Run` share one preparation
|
||||
pipeline with two private phases. Do not duplicate prompt/profile/backend
|
||||
selection or execution precedence.
|
||||
|
||||
The first phase resolves only the state required before admission:
|
||||
|
||||
1. validate `PromptID`;
|
||||
2. normalize the direct session ID;
|
||||
3. load the prompt definition;
|
||||
4. hash the original prompt definition at its existing error-order position;
|
||||
5. select and load the execution profile;
|
||||
6. resolve the selected backend;
|
||||
7. resolve and validate the effective execution target and credentials; and
|
||||
8. resolve the effective output contract without loading its schema.
|
||||
|
||||
Return a private state value containing the loaded definition, normalized
|
||||
direct session, prompt-definition hash, selected profile ID, effective target,
|
||||
numeric-presence metadata, effective output contract, and preparation start
|
||||
time. Keep this value private to `internal/usecase`.
|
||||
|
||||
The second phase consumes that state and performs:
|
||||
|
||||
1. structured-output schema loading;
|
||||
2. artifact loading and input hashing;
|
||||
3. message and prompt-session rendering;
|
||||
4. direct-session application;
|
||||
5. rendered-prompt hashing; and
|
||||
6. `PreparedRun` construction and timing.
|
||||
|
||||
Preserve every existing precedence rule, error identity, direct-session
|
||||
template bypass, hash input, selected identity, copy guarantee, and timing
|
||||
field. Do not reload the prompt, profile, or backend between phases.
|
||||
|
||||
`Runner.Prepare` records its start time, runs both phases consecutively, and
|
||||
never calls admission. Its behavior and error ordering remain unchanged.
|
||||
|
||||
`Runner.Run` records its existing run start time, runs the first preparation
|
||||
phase, and then calls:
|
||||
|
||||
```go
|
||||
release, err := r.admitter.Admit(ctx, effectiveBackendID)
|
||||
```
|
||||
|
||||
Use a narrow use-case-owned interface with the same signature:
|
||||
|
||||
```go
|
||||
type RunAdmitter interface {
|
||||
Admit(context.Context, string) (func(), error)
|
||||
}
|
||||
```
|
||||
|
||||
A nil admitter means unlimited behavior for internal constructors and tests.
|
||||
On successful admission, immediately `defer release()` around the remainder of
|
||||
the run. Then run the second preparation phase, initial generation,
|
||||
validation, and all repair attempts.
|
||||
|
||||
If admission returns internal `capacity.ErrCapacityExceeded`, add useful
|
||||
backend context without changing its identity. If it returns `ctx.Err()`,
|
||||
preserve that identity directly rather than recategorizing it as invalid
|
||||
request or generation failure.
|
||||
|
||||
This refactor intentionally replaces the current literal `Run`-calls-`Prepare`
|
||||
implementation with shared private phases. Update current-state documentation
|
||||
to describe one shared pipeline rather than retaining an inaccurate call-graph
|
||||
claim.
|
||||
|
||||
### Generation And Repair Lifetime
|
||||
|
||||
The admission lease covers the entire accepted run:
|
||||
|
||||
- second-phase preparation;
|
||||
- initial generation;
|
||||
- validation;
|
||||
- every repair; and
|
||||
- all failure and cancellation exits.
|
||||
|
||||
Preparation and validation do not hold an active-generation permit. The
|
||||
wrapped client acquires a permit only around each actual `Generate` call.
|
||||
|
||||
The runner's initial generation already carries the effective backend ID in
|
||||
`GenerateRequest.Target`. Preserve that value. `RepairRequest.Target` and the
|
||||
default repairer's generated request must continue carrying the same backend
|
||||
ID, allowing each repair to reacquire the same pool's active permit.
|
||||
|
||||
When testing or constructing `NewRunnerWithRepairer`, pass the same wrapped
|
||||
client to both the runner and `NewDefaultOutputRepairer`. Do not add capacity
|
||||
state to `RepairRequest`, `ExecutionTarget`, or public generation values.
|
||||
|
||||
A repair remains within its existing admission lease. It waits for a FIFO
|
||||
active permit but never performs a second bounded admission and therefore
|
||||
cannot fail merely because later runs filled the admission capacity.
|
||||
|
||||
### Error And Cancellation Semantics
|
||||
|
||||
The required public outcomes are:
|
||||
|
||||
| Situation | Required error identity |
|
||||
| --- | --- |
|
||||
| Admission capacity is full | `ErrCapacityExceeded` only; not `ErrInvalidRequest` or `ErrLLMGenerate`. |
|
||||
| Context is done before admission succeeds | Preserve `ctx.Err()`; do not return capacity exhaustion. |
|
||||
| Context cancels while waiting for an active permit | Preserve `ctx.Err()` through the existing `ErrLLMGenerate` generation category. |
|
||||
| Wrapped client fails after permit acquisition | Preserve existing `ErrLLMGenerate` and collaborator identities. |
|
||||
| Preparation or validation fails after admission | Preserve its existing category and release admission. |
|
||||
|
||||
Maintain the existing rule that `Run` returns no partial result on any
|
||||
operational error. Do not add queue status to errors or results.
|
||||
|
||||
`RunResult.Duration` continues to start at runner entry and therefore includes
|
||||
pre-admission resolution, accepted preparation, and active-permit waiting.
|
||||
`PreparedRun.DurationMS` continues to cover only its shared preparation phases;
|
||||
it does not include later generation waiting. Capacity-rejected calls have no
|
||||
result or timing value.
|
||||
|
||||
### Ownership And Concurrency Safety
|
||||
|
||||
The registry, capacity policy map, pool map, and per-pool limits are immutable
|
||||
after engine construction. Only admission counts, active counts, and waiter
|
||||
lists are mutable and must be protected by the owning pool mutex.
|
||||
|
||||
Do not retain public queue pointers, caller request values, contexts, or
|
||||
generation requests after their call completes. A canceled waiter must be
|
||||
unlinked so its context and request cannot remain reachable from the pool.
|
||||
|
||||
Do not hold a pool mutex while:
|
||||
|
||||
- loading or rendering prompts;
|
||||
- reading artifacts or schemas;
|
||||
- invoking a model client;
|
||||
- validating output;
|
||||
- closing a waiter notification channel if the implementation could re-enter
|
||||
pool code; or
|
||||
- calling consumer code.
|
||||
|
||||
No scheduler operation may spawn a goroutine whose lifetime outlasts the
|
||||
calling `Run`. The zero steady-state goroutine count is part of the
|
||||
in-process/no-worker-lifecycle design.
|
||||
|
||||
## Test Ownership
|
||||
|
||||
Use this ownership split and avoid repeating the full policy matrix at every
|
||||
layer:
|
||||
|
||||
- `internal/backend/registry_test.go` owns normalization, validation, the exact
|
||||
OpenRouter policy, the custom default queue, explicit zero, unlimited
|
||||
omission, and policy-map copying.
|
||||
- `internal/capacity/manager_test.go` owns admission bounds, idempotent release,
|
||||
FIFO active permits, cancellation races, capacity recovery, independent
|
||||
pools, unlimited IDs, and observed peak concurrency.
|
||||
- `internal/capacity/client_test.go` owns wrapper request/response/error
|
||||
transparency and the rule that cancellation before grant does not invoke the
|
||||
next client. Combine these with manager tests if one coherent package test
|
||||
expresses the behavior more clearly.
|
||||
- `internal/usecase/runner_test.go` owns two-phase preparation parity, pool
|
||||
selection, admission before expensive work, admission release across run
|
||||
exits, `Prepare` bypass, and repair reuse of the admitted backend.
|
||||
- Root external-package tests own public configuration conversion, assembled
|
||||
engine-local behavior, endpoint-override routing, injected-client limiting,
|
||||
and public capacity/context error identities.
|
||||
- Existing model-client HTTP tests remain unchanged because scheduling does
|
||||
not alter the OpenAI-compatible wire contract.
|
||||
|
||||
Concurrency tests must use test-owned limits such as one or two and
|
||||
channel-controlled blocking clients. Record observed active and peak counts
|
||||
under a mutex or atomics. Do not use `time.Sleep` to infer queue state.
|
||||
Package-internal tests may inspect a waiter list under its mutex through a
|
||||
small test helper when necessary to establish deterministic FIFO ordering; do
|
||||
not add production metrics or hooks solely for tests.
|
||||
|
||||
Do not add separate tests for trivial scalar copies when registry or assembled
|
||||
behavior already protects them.
|
||||
|
||||
## Stage 1 — Backend Policy And Public Configuration
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Goal
|
||||
|
||||
Add the public and internal backend policy representation, normalize all
|
||||
configured states, and expose immutable normalized policies without changing
|
||||
runtime scheduling yet.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `ConcurrencyLimit` and `QueueCapacity` to `Backend` in `backends.go`
|
||||
with exact GoDoc for unlimited, defaulted, explicit-zero, invalid, and
|
||||
engine-scoped behavior.
|
||||
2. Convert the public queue pointer into scalar value plus presence in
|
||||
`WithBackend`; do not retain the pointer.
|
||||
3. Add the internal backend policy fields and
|
||||
`BackendCapacityPolicy` to `internal/domain/domain.go`.
|
||||
4. Add the two registry-owned constants and configure the built-in OpenRouter
|
||||
definition with 16 and 1024.
|
||||
5. Extend `normalizeBackend` with the fixed validation, defaulting, explicit
|
||||
zero, and overflow rules.
|
||||
6. Add `Registry.CapacityPolicies`, returning only limited policies in a fresh
|
||||
map.
|
||||
7. Update existing backend composite literals and assertions only where the
|
||||
new fields are relevant. Continue using keyed literals.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Extend the exact built-in registry test with the OpenRouter limit and queue.
|
||||
2. Add one coherent table covering unlimited omission, default queue,
|
||||
explicit-zero queue, negative values, queue-without-limit, and total
|
||||
overflow.
|
||||
3. Extend the registry copy/normalization test to prove returned policy maps
|
||||
cannot mutate registry state.
|
||||
4. Add root coverage only if needed to prove the public pointer/presence
|
||||
conversion; do not reproduce registry validation cases at the facade.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w backends.go internal/domain/domain.go \
|
||||
internal/backend/registry.go internal/backend/registry_test.go
|
||||
go test . ./internal/backend
|
||||
go vet . ./internal/backend
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Include another touched Go test file in `gofmt` only if it actually changed.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
This stage is complete when every public configuration state has one normalized
|
||||
internal meaning, OpenRouter exposes exactly 16/1024, custom backends remain
|
||||
unlimited by omission, and no runtime call is scheduled yet.
|
||||
|
||||
## Stage 2 — Engine-Local Capacity Manager
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Goal
|
||||
|
||||
Implement and prove the bounded admission mechanism and FIFO active-generation
|
||||
client wrapper independently of runner orchestration.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `internal/capacity/manager.go` with the manager, immutable policy copy,
|
||||
per-backend pools, internal error, immediate admission, idempotent release,
|
||||
and FIFO context-aware active permits.
|
||||
2. Add `internal/capacity/client.go` with the transparent `llm.Client` wrapper.
|
||||
3. Use mutex-protected waiter state and an ordered list; explicitly resolve
|
||||
grant-versus-cancel races.
|
||||
4. Ensure unlimited and independent-pool fast paths avoid queue allocation.
|
||||
5. Do not start workers, timers, cleanup goroutines, or process-global state.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Add a compact constructor-validation table for blank IDs, non-positive
|
||||
limits, negative queues, and total-capacity overflow.
|
||||
2. With a small configured policy, prove that exactly
|
||||
`limit + queueCapacity` admissions succeed, the next matches
|
||||
`ErrCapacityExceeded`, and a release permits another admission.
|
||||
3. Prove release is idempotent.
|
||||
4. Drive more blocked client calls than the active limit and assert observed
|
||||
peak concurrency never exceeds that limit.
|
||||
5. Prove FIFO order with deterministic queue-entry synchronization.
|
||||
6. Cancel the first and a middle waiter and prove they are removed, never call
|
||||
the wrapped client, and do not block later waiters.
|
||||
7. Exercise the grant/cancel race repeatedly under `go test -race`, asserting
|
||||
no permit leak or double invocation.
|
||||
8. Prove different backend IDs proceed independently and blank, unknown, or
|
||||
nil-manager paths remain unlimited.
|
||||
9. Prove request values, successful and nil responses, and collaborator errors
|
||||
pass through unchanged after permit acquisition.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/capacity/manager.go \
|
||||
internal/capacity/manager_test.go \
|
||||
internal/capacity/client.go \
|
||||
internal/capacity/client_test.go
|
||||
go test ./internal/capacity
|
||||
go test -race ./internal/capacity
|
||||
go vet ./internal/capacity
|
||||
git diff --check
|
||||
```
|
||||
|
||||
If tests are combined into one file, omit the nonexistent file from `gofmt`.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
This stage is complete when the standalone component enforces relational
|
||||
admission and active limits, FIFO cancellation is race-safe, separate pools
|
||||
are independent, and the wrapper is transparent apart from waiting.
|
||||
|
||||
## Stage 3 — Shared Preparation And Early Run Admission
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Goal
|
||||
|
||||
Refactor runner preparation into one shared two-phase pipeline and place
|
||||
bounded admission after backend resolution but before schema, artifact, and
|
||||
rendering work.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add the private pre-admission preparation state and split the existing
|
||||
`Prepare` logic according to the fixed design.
|
||||
2. Make `Runner.Prepare` call both phases without an admitter.
|
||||
3. Add the `RunAdmitter` interface and runner field.
|
||||
4. Update `NewRunner` and `NewRunnerWithRepairer` to accept the optional
|
||||
admitter; update internal call sites with nil until root assembly is wired.
|
||||
5. Change `Runner.Run` to use the first phase, admit by effective backend ID,
|
||||
defer the returned release, and then use the second phase.
|
||||
6. Preserve all existing error precedence, target resolution, hashes,
|
||||
metadata, session behavior, and timing.
|
||||
7. Return capacity and context errors with the fixed identities. Do not invoke
|
||||
later collaborators after rejection.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Keep the existing `Run`/`Prepare` parity coverage passing to prove the
|
||||
shared phases do not drift.
|
||||
2. Add a fake admitter that records backend IDs and release calls.
|
||||
3. Prove a backend-selected run admits with the selected ID even when the
|
||||
endpoint is overridden.
|
||||
4. Prove an endpoint-only run uses the unlimited/blank identity and that
|
||||
`Prepare` never calls admission.
|
||||
5. Reject admission and assert schema, artifact, renderer, validator, repairer,
|
||||
and LLM collaborators are not invoked.
|
||||
6. Prove admission is released after one successful run and representative
|
||||
second-phase, generation, and validation errors. Prefer a small table around
|
||||
the single `defer` invariant rather than duplicating every error test.
|
||||
7. Retain direct-session, backend precedence, credential, hashing, and repair
|
||||
tests unchanged except for constructor arguments.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/runner.go \
|
||||
internal/usecase/runner_test.go
|
||||
go test ./internal/usecase
|
||||
go test -race ./internal/usecase
|
||||
go vet ./internal/usecase
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Completion Gate
|
||||
|
||||
This stage is complete when `Prepare` remains unrestricted, `Run` admits after
|
||||
one canonical routing phase and before expensive completion work, every exit
|
||||
releases admission, and existing preparation semantics remain unchanged.
|
||||
|
||||
## Stage 4 — Engine Assembly And Public Runtime Contract
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Goal
|
||||
|
||||
Wire one manager into each engine, schedule built-in and injected clients,
|
||||
expose the capacity error, and prove assembled runtime behavior.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add public `ErrCapacityExceeded` and its exact GoDoc in `engine.go`.
|
||||
2. Map internal capacity exhaustion in `errors.go`.
|
||||
3. Construct the manager from the registry policy snapshot in `NewEngine`.
|
||||
4. Wrap the selected internal client after built-in or injected-client
|
||||
selection and pass the manager and wrapped client to the runner.
|
||||
5. Update `Engine`, `NewEngine`, `Run`, `WithLLMClient`, and `LLMClient` GoDoc
|
||||
only where concurrency, capacity, or cancellation statements change.
|
||||
6. Ensure manager-construction errors match `ErrInvalidConfig`.
|
||||
7. For internal repair coverage, construct the default repairer with the same
|
||||
wrapped client used by its runner and confirm repair target backend identity
|
||||
remains intact.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Add an external-package assembled test with a small custom limit and a
|
||||
blocking injected client; assert peak generation equals or remains below
|
||||
the configured limit.
|
||||
2. With queue capacity zero, block one accepted run before generation and
|
||||
assert the next matching-backend run returns `ErrCapacityExceeded`, does not
|
||||
match `ErrInvalidRequest` or `ErrLLMGenerate`, returns no result, and never
|
||||
reaches expensive collaborators or the client.
|
||||
3. In the same or another focused workflow, prove an endpoint override remains
|
||||
in the selected backend's pool.
|
||||
4. Prove two engines with the same backend ID have independent capacity.
|
||||
5. Prove an unlimited custom backend and an endpoint-only profile preserve
|
||||
concurrent behavior.
|
||||
6. Cancel a call waiting for an active permit; assert it matches both
|
||||
`context.Canceled` and `ErrLLMGenerate`, never invokes the injected client,
|
||||
and leaves capacity reusable.
|
||||
7. Add one internal repair workflow with concurrent runs or controlled permits
|
||||
showing initial and repair generations never exceed the same backend limit
|
||||
and repairs do not perform a second admission.
|
||||
8. Extend the public error sentinel contract test with
|
||||
`ErrCapacityExceeded`.
|
||||
|
||||
Avoid a second HTTP-level concurrency suite: the capacity client tests and one
|
||||
assembled injected-client workflow already protect the shared wrapper used by
|
||||
the built-in client.
|
||||
|
||||
### Focused Validation
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w engine.go errors.go backends.go \
|
||||
internal/usecase/runner.go internal/usecase/runner_test.go \
|
||||
engine_test.go public_contract_test.go
|
||||
go test . ./internal/backend ./internal/capacity ./internal/usecase
|
||||
go test -race . ./internal/capacity ./internal/usecase
|
||||
go vet . ./internal/backend ./internal/capacity ./internal/usecase
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Add any newly created capacity files to `gofmt` when they changed in this
|
||||
stage.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
This stage is complete when every engine has independent pools, limited runs
|
||||
are bounded and FIFO at generation, endpoint routing is correct, capacity and
|
||||
context errors are stable, repairs reuse admission, and both client kinds pass
|
||||
through the same wrapper.
|
||||
|
||||
## Stage 5 — Durable Documentation And Final Validation
|
||||
|
||||
**Status:** Pending.
|
||||
|
||||
### Goal
|
||||
|
||||
Move implemented contracts into their durable owners, record compatibility
|
||||
impact, and validate the complete repository.
|
||||
|
||||
### Work
|
||||
|
||||
1. Review every changed exported declaration. Ensure GoDoc is the canonical
|
||||
owner of exact field types, nil/zero semantics, defaulting, error identity,
|
||||
engine scope, concurrency safety, cancellation, and source compatibility.
|
||||
2. Update `doc.go` so its concurrency summary acknowledges backend scheduling
|
||||
while continuing to require injected collaborators to be concurrency-safe.
|
||||
3. Update `docs/consumers/pkg-promptkit.md` with task-oriented examples for:
|
||||
- a limited local backend;
|
||||
- omitted queue capacity selecting 1024;
|
||||
- explicit zero queue capacity; and
|
||||
- handling `ErrCapacityExceeded`.
|
||||
Keep exact field semantics in GoDoc rather than duplicating a full table.
|
||||
4. Add `docs/internal/capacity.md` as the durable owner of pool lifecycle,
|
||||
admission, FIFO active permits, cancellation, client wrapping, and test
|
||||
ownership.
|
||||
5. Add `internal/capacity` to `docs/internal/overview.md`.
|
||||
6. Update `docs/policy/architecture.md` to include the implemented component
|
||||
and root assembly dependency without turning policy into an API reference.
|
||||
7. Update `docs/internal/runner.md` to describe the shared two-phase
|
||||
preparation pipeline, early bounded admission, lease lifetime, generation
|
||||
permits, repairs, capacity failures, and cancellation.
|
||||
8. Review `docs/formats.md`; add only a concise link or clarification if needed
|
||||
to explain that endpoint overrides preserve backend capacity identity.
|
||||
Do not add concurrency fields to YAML.
|
||||
9. Do not change the OpenAI-compatible integration contract or
|
||||
`docs/internal/llm.md` unless implementation changes their current
|
||||
statements; scheduling is outside the provider wire contract and concrete
|
||||
model-client implementation.
|
||||
10. Record in the implementation handoff that built-in OpenRouter now limits
|
||||
active generations to 16 with queue capacity 1024 and that the release
|
||||
must be a pre-`v1` minor release. Do not edit the release procedure or
|
||||
create a tag.
|
||||
11. After every check passes, set `concurrency.md`, this implementation plan,
|
||||
and each stage status to `Complete`. Do not remove the roadmaps in the
|
||||
implementation change; lifecycle retirement follows review.
|
||||
|
||||
### Full Validation
|
||||
|
||||
Run the complete sequence from `docs/development.md`:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
gofmt -l $(git ls-files '*.go')
|
||||
git diff --check
|
||||
```
|
||||
|
||||
The formatting command must produce no paths. Follow every added or changed
|
||||
Markdown link and confirm its target and heading exist.
|
||||
|
||||
Also inspect:
|
||||
|
||||
```sh
|
||||
git status --short
|
||||
git diff --stat
|
||||
git diff
|
||||
```
|
||||
|
||||
Confirm that:
|
||||
|
||||
- only intended backend, capacity, runner, facade, test, documentation, and
|
||||
roadmap files changed;
|
||||
- no `go.work`, `go.work.sum`, local module replacement, credential,
|
||||
generated binary, coverage output, or unrelated change was introduced;
|
||||
- the built-in OpenRouter policy is exactly 16/1024;
|
||||
- custom and endpoint-only backends remain unlimited by omission;
|
||||
- explicit queue zero is distinguishable from omission;
|
||||
- no capacity value enters execution targets, generated requests, stable JSON,
|
||||
prompt/profile YAML, or provider payloads;
|
||||
- every engine owns distinct pools with no package-global mutable state;
|
||||
- every initial and repair generation uses the active permit wrapper;
|
||||
- capacity and waiter state is released on success, error, panic unwinding,
|
||||
and cancellation;
|
||||
- concurrency tests use deterministic coordination rather than sleeps;
|
||||
- current-state documentation describes implemented behavior rather than
|
||||
referring readers to the roadmaps; and
|
||||
- the feature and implementation roadmaps contain no unresolved work marked
|
||||
complete.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
The implementation is complete only when every target-end-state item in
|
||||
`concurrency.md` is implemented, race-enabled tests demonstrate the configured
|
||||
limits and cancellation safety, durable contracts no longer depend on roadmap
|
||||
prose, and the OpenRouter compatibility change is clearly reported for the
|
||||
next minor release.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and this plan fix the public representation,
|
||||
registry defaults, admission bound, FIFO generation behavior, early-routing
|
||||
refactor, cancellation races, error identities, engine and repair lifetimes,
|
||||
test ownership, compatibility treatment, and non-goals required for
|
||||
implementation.
|
||||
Reference in New Issue
Block a user