|
|
|
|
@@ -0,0 +1,262 @@
|
|
|
|
|
# Structured Capacity Errors
|
|
|
|
|
|
|
|
|
|
**Status:** Accepted.
|
|
|
|
|
|
|
|
|
|
## Purpose
|
|
|
|
|
|
|
|
|
|
Allow consumers to identify which registered backend rejected engine
|
|
|
|
|
admission without parsing an error string, while preserving the existing
|
|
|
|
|
`ErrCapacityExceeded` classification and Promptkit's provider-neutral error
|
|
|
|
|
boundary.
|
|
|
|
|
|
|
|
|
|
This provides safe operational context requested by
|
|
|
|
|
[Notarius](notarius-promptkit-wishlist.md#priority-4-structured-capacity-errors)
|
|
|
|
|
and
|
|
|
|
|
[Weatherreporter](weatherreporter-promptkit-wishlist.md#structured-capacity-errors).
|
|
|
|
|
|
|
|
|
|
## Motivation
|
|
|
|
|
|
|
|
|
|
Promptkit currently returns an error matching `ErrCapacityExceeded` when a
|
|
|
|
|
limited backend's active and waiting admission capacity is full. That sentinel
|
|
|
|
|
lets consumers classify the failure reliably, but the selected backend ID is
|
|
|
|
|
available only in diagnostic text.
|
|
|
|
|
|
|
|
|
|
Applications using multiple backends need the stable backend ID for
|
|
|
|
|
operator-facing diagnostics, metrics labels, failure receipts, and their own
|
|
|
|
|
retry decisions. Parsing `Error()` text would turn non-contractual wording
|
|
|
|
|
into an accidental API and risks retaining more diagnostic context than the
|
|
|
|
|
application needs.
|
|
|
|
|
|
|
|
|
|
Promptkit already owns backend selection and the admission boundary. It should
|
|
|
|
|
attach the selected backend's safe identity at that boundary and translate it
|
|
|
|
|
into a small public typed error.
|
|
|
|
|
|
|
|
|
|
## Consumer Workflow
|
|
|
|
|
|
|
|
|
|
The target public workflow is:
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
result, err := engine.Run(ctx, request)
|
|
|
|
|
if err != nil {
|
|
|
|
|
var capacityErr *promptkit.CapacityError
|
|
|
|
|
if errors.As(err, &capacityErr) {
|
|
|
|
|
// Record capacityErr.BackendID or apply application retry policy.
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Existing classification remains valid:
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
if errors.Is(err, promptkit.ErrCapacityExceeded) {
|
|
|
|
|
// Handle engine admission rejection without structured context.
|
|
|
|
|
}
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The target public surface is:
|
|
|
|
|
|
|
|
|
|
```go
|
|
|
|
|
type CapacityError struct {
|
|
|
|
|
BackendID string
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
func (e *CapacityError) Error() string
|
|
|
|
|
func (e *CapacityError) Unwrap() error
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
`Unwrap` returns `ErrCapacityExceeded`. Engine-produced capacity errors
|
|
|
|
|
therefore support both `errors.As` to `*CapacityError` and
|
|
|
|
|
`errors.Is(err, ErrCapacityExceeded)`, including through additional wrapping.
|
|
|
|
|
|
|
|
|
|
The exported declaration and GoDoc will own the exact implemented contract.
|
|
|
|
|
`CapacityError` does not need a stable JSON representation, an exported
|
|
|
|
|
constructor, or fields beyond `BackendID`.
|
|
|
|
|
|
|
|
|
|
## Error Boundary
|
|
|
|
|
|
|
|
|
|
A `CapacityError` represents one specific condition: `Run` or `RunPrepared`
|
|
|
|
|
was rejected at Promptkit's engine-local bounded admission boundary because
|
|
|
|
|
the selected limited backend had no remaining admitted slot.
|
|
|
|
|
|
|
|
|
|
It does not represent:
|
|
|
|
|
|
|
|
|
|
- waiting for an active-generation permit after admission;
|
|
|
|
|
- caller cancellation or deadline expiration while acquiring or holding
|
|
|
|
|
capacity;
|
|
|
|
|
- provider rate limiting, quota exhaustion, HTTP 429 responses, or another
|
|
|
|
|
model-client failure;
|
|
|
|
|
- validation, output repair, or application-level queue rejection; or
|
|
|
|
|
- an application's decision not to submit or retry work.
|
|
|
|
|
|
|
|
|
|
Provider and injected-client errors retain their existing generation error
|
|
|
|
|
classification even when a provider describes its failure as capacity or rate
|
|
|
|
|
limiting. Promptkit must not infer an engine admission error from provider
|
|
|
|
|
status codes or error text.
|
|
|
|
|
|
|
|
|
|
Admission continues to check caller cancellation before reporting a full
|
|
|
|
|
pool. A cancellation that wins that decision remains a context-related
|
|
|
|
|
execution error and does not become a `CapacityError`.
|
|
|
|
|
|
|
|
|
|
## Backend Identity
|
|
|
|
|
|
|
|
|
|
`BackendID` is the normalized stable ID of the registered backend whose pool
|
|
|
|
|
rejected admission. It is the same effective backend identity used for
|
|
|
|
|
capacity routing and exposed by prepared and run metadata.
|
|
|
|
|
|
|
|
|
|
For ordinary `Run`, the ID comes from the backend selected by the resolved
|
|
|
|
|
prompt, profile, and execution target. For `RunPrepared`, it comes from the
|
|
|
|
|
backend identity frozen in the prepared execution.
|
|
|
|
|
|
|
|
|
|
An endpoint override does not change backend identity and therefore does not
|
|
|
|
|
change the reported `BackendID`. Endpoint-only profiles and unlimited
|
|
|
|
|
backends have no bounded admission pool and cannot produce an engine-generated
|
|
|
|
|
`CapacityError`.
|
|
|
|
|
|
|
|
|
|
Every engine-produced `CapacityError` has a nonblank registered backend ID.
|
|
|
|
|
The type does not report a profile ID, prompt ID, model, endpoint, run ID, or
|
|
|
|
|
prepared-handle identity.
|
|
|
|
|
|
|
|
|
|
## Compatibility
|
|
|
|
|
|
|
|
|
|
`ErrCapacityExceeded` remains the stable sentinel for broad classification.
|
|
|
|
|
This feature refines the error value returned by capacity-rejected `Run` and
|
|
|
|
|
`RunPrepared` calls without replacing or removing that sentinel.
|
|
|
|
|
|
|
|
|
|
The following behaviors remain unchanged:
|
|
|
|
|
|
|
|
|
|
- `errors.Is(err, ErrCapacityExceeded)` succeeds for engine admission
|
|
|
|
|
rejection;
|
|
|
|
|
- capacity rejection remains distinct from `ErrInvalidRequest`,
|
|
|
|
|
`ErrLLMGenerate`, and other public categories;
|
|
|
|
|
- `Run` returns no partial result after admission rejection;
|
|
|
|
|
- `RunPrepared` retains its existing one-attempt lifecycle, including that a
|
|
|
|
|
claimed handle is consumed when admission is rejected;
|
|
|
|
|
- admission ordering, limits, queue capacity, leases, release behavior, and
|
|
|
|
|
backend independence are unchanged; and
|
|
|
|
|
- `Prepare`, `PrepareExecution`, `InspectPrompt`, and `InspectProfile` perform
|
|
|
|
|
no admission and cannot return this error.
|
|
|
|
|
|
|
|
|
|
Direct equality with the sentinel is not introduced as a contract. Consumers
|
|
|
|
|
continue to use `errors.Is` for category checks and `errors.As` for structured
|
|
|
|
|
context.
|
|
|
|
|
|
|
|
|
|
## Sensitive Data And Diagnostics
|
|
|
|
|
|
|
|
|
|
The typed error contains only `BackendID`. It must not expose or retain:
|
|
|
|
|
|
|
|
|
|
- backend endpoints;
|
|
|
|
|
- credential environment-variable names or credential values;
|
|
|
|
|
- model, profile, prompt, session, or request identifiers;
|
|
|
|
|
- prompt, input, rendered, generated, or validation content;
|
|
|
|
|
- concurrency limits, queue capacities, current counts, or queue depth;
|
|
|
|
|
- request position or identities of other admitted work; or
|
|
|
|
|
- retry timing, provider health, or speculative availability.
|
|
|
|
|
|
|
|
|
|
`Error()` may include the quoted backend ID and a concise admission-rejection
|
|
|
|
|
description. Its wording is diagnostic and not a stable parsing contract.
|
|
|
|
|
Default formatting, wrapping, and Go string formatting must not reveal
|
|
|
|
|
anything beyond the public field and generic capacity classification.
|
|
|
|
|
|
|
|
|
|
Concurrency and queue configuration remain available where consumers define
|
|
|
|
|
their backends; repeating them on each error would add stale or unnecessary
|
|
|
|
|
operational detail. Current counts are race-prone observations rather than a
|
|
|
|
|
durable explanation of when capacity will become available.
|
|
|
|
|
|
|
|
|
|
## Retry And Application Policy
|
|
|
|
|
|
|
|
|
|
Promptkit reports that admission was rejected but does not retry, wait for a
|
|
|
|
|
new admission slot, calculate backoff, or declare an HTTP status.
|
|
|
|
|
|
|
|
|
|
A full in-process admission pool does not provide a meaningful retry delay:
|
|
|
|
|
lease duration depends on consumer inputs, model latency, validation, repair,
|
|
|
|
|
cancellation, and provider behavior. Consumers own whether to retry, which
|
|
|
|
|
backend to use, how to schedule or back off, what to expose to operators, and
|
|
|
|
|
how to translate the error into CLI or transport behavior.
|
|
|
|
|
|
|
|
|
|
The absence of retry metadata is deliberate. A future retry facility would
|
|
|
|
|
require its own roadmap and application-neutral policy rather than extending
|
|
|
|
|
this error speculatively.
|
|
|
|
|
|
|
|
|
|
## Ownership And Concurrency
|
|
|
|
|
|
|
|
|
|
Each rejected operation returns its own caller-owned `*CapacityError`. The
|
|
|
|
|
error does not reference mutable pool state or retain the request. Reading it
|
|
|
|
|
requires no engine access and remains safe after the rejected call returns.
|
|
|
|
|
|
|
|
|
|
Mutating the exported `BackendID` on a returned error affects only that error
|
|
|
|
|
value. It cannot change the backend registry, capacity manager, another
|
|
|
|
|
operation's error, or later routing and admission.
|
|
|
|
|
|
|
|
|
|
Constructing or returning the typed error requires no new process-global
|
|
|
|
|
state, cache, goroutine, or synchronization beyond existing capacity
|
|
|
|
|
management.
|
|
|
|
|
|
|
|
|
|
## Architectural Boundaries
|
|
|
|
|
|
|
|
|
|
The root `promptkit` package owns the exported error type, sentinel
|
|
|
|
|
compatibility, and public translation. The engine's internal capacity and
|
|
|
|
|
use-case components remain responsible for admission and for carrying the
|
|
|
|
|
selected backend identity to the facade.
|
|
|
|
|
|
|
|
|
|
Internal error types or capacity-manager state must not cross the public
|
|
|
|
|
boundary. The public facade must obtain structured identity through an
|
|
|
|
|
internal typed or otherwise non-textual error contract; it must not parse the
|
|
|
|
|
runner's diagnostic string.
|
|
|
|
|
|
|
|
|
|
The feature does not introduce a public capacity manager, backend registry,
|
|
|
|
|
queue, scheduler, metrics collector, or transport-specific error. It remains a
|
|
|
|
|
small refinement of the existing root error boundary.
|
|
|
|
|
|
|
|
|
|
## Documentation
|
|
|
|
|
|
|
|
|
|
When implemented, documentation should retain these ownership boundaries:
|
|
|
|
|
|
|
|
|
|
- the exported declaration and GoDoc own the exact type, field, `Error`,
|
|
|
|
|
`Unwrap`, `errors.Is`, `errors.As`, and formatting contracts;
|
|
|
|
|
- the consumer guide demonstrates broad classification and optional backend
|
|
|
|
|
extraction without duplicating every error guarantee;
|
|
|
|
|
- internal capacity and runner documentation describe where backend identity
|
|
|
|
|
is attached and translated;
|
|
|
|
|
- the backend API continues to own capacity configuration; and
|
|
|
|
|
- the architecture and documentation policies continue to own library,
|
|
|
|
|
application, and documentation boundaries.
|
|
|
|
|
|
|
|
|
|
## Non-Goals
|
|
|
|
|
|
|
|
|
|
This work does not include:
|
|
|
|
|
|
|
|
|
|
- changing concurrency limits, queue capacities, FIFO scheduling, or default
|
|
|
|
|
capacity policy;
|
|
|
|
|
- exposing configured limits, current counts, queue depth, or waiter
|
|
|
|
|
positions;
|
|
|
|
|
- reporting a retry delay, retryable boolean, HTTP status, CLI exit code, or
|
|
|
|
|
backoff policy;
|
|
|
|
|
- automatically retrying or rerouting rejected work;
|
|
|
|
|
- unifying provider rate limits or quota failures with engine admission;
|
|
|
|
|
- adding structured context to generation, validation, prompt, profile, or
|
|
|
|
|
other error categories;
|
|
|
|
|
- returning partial preparation or run results with the error;
|
|
|
|
|
- changing prepared-execution handle lifecycle;
|
|
|
|
|
- adding stable JSON for errors;
|
|
|
|
|
- exposing internal capacity or use-case types; or
|
|
|
|
|
- adding application logging, metrics emission, persistence, or failure
|
|
|
|
|
receipts.
|
|
|
|
|
|
|
|
|
|
## Target End State
|
|
|
|
|
|
|
|
|
|
After this work:
|
|
|
|
|
|
|
|
|
|
- every engine admission rejection from `Run` or `RunPrepared` returns an
|
|
|
|
|
error discoverable as `*CapacityError`;
|
|
|
|
|
- the typed error contains the normalized selected backend ID and no other
|
|
|
|
|
structured field;
|
|
|
|
|
- existing `errors.Is(err, ErrCapacityExceeded)` handling continues to work;
|
|
|
|
|
- `errors.As` obtains backend identity without parsing diagnostic text;
|
|
|
|
|
- capacity rejection remains distinct from cancellation, model-client
|
|
|
|
|
failures, provider throttling, and application scheduling;
|
|
|
|
|
- endpoints, credentials, request content, capacity metrics, and speculative
|
|
|
|
|
retry information remain absent;
|
|
|
|
|
- retry, backoff, rerouting, metrics, persistence, and transport mapping remain
|
|
|
|
|
consumer responsibilities; and
|
|
|
|
|
- the existing capacity architecture and runtime behavior are otherwise
|
|
|
|
|
unchanged.
|
