diff --git a/docs/integrations/pkg-promptkit.md b/docs/integrations/pkg-promptkit.md index ff34e8a..23647d0 100644 --- a/docs/integrations/pkg-promptkit.md +++ b/docs/integrations/pkg-promptkit.md @@ -19,7 +19,8 @@ Notarius relies on the root `promptkit` package to: - return rendered debug material, validated structured output, selected profile, backend, effective model metadata, and token usage; - distinguish structured-output validation failure from execution failure; and -- identify a missing explicit profile through `ErrProfileNotFound`. +- identify a missing explicit profile through `ErrProfileNotFound` and backend + admission exhaustion through `ErrCapacityExceeded`. Notarius does not use PromptKit's optional `ArtifactReader`. It materializes source and reference content itself and supplies owned inline artifacts at the @@ -40,6 +41,13 @@ PromptKit's stable lower-case `effective_model_params` JSON, which may include `backend_id`. Notarius production configuration does not expose user-defined PromptKit backend registration. +Notarius retains its application-wide scheduled client around the PromptKit +adapter. PromptKit may apply a narrower limit for the selected backend; +endpoint-only profiles have no such backend limit. The adapter translates +PromptKit capacity rejection into the provider-neutral Notarius +`ErrLLMCapacityExceeded` contract and leaves retries to the calling pipeline +stage. + ## Notarius Ownership [LLM Runtime Internals](../internal/llm.md) describes how Notarius mounts diff --git a/docs/internal/llm.md b/docs/internal/llm.md index d181c14..72c7dde 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -81,6 +81,13 @@ worker counts cannot exceed the configured LLM limit. The configuration field and its effective default are owned by [Configuration](../config.md#concurrency-output-cache-and-debug). +PromptKit applies a second, independent admission limit when the selected +profile names a limited backend. It sits beneath the Notarius scheduled client, +so it may narrow but cannot expand the application-wide limit. Built-in +OpenRouter profiles select PromptKit's reserved backend and its upstream +capacity policy. Endpoint-only profiles do not select a PromptKit backend and +remain limited only by the Notarius scheduler. + ## Prompt And Schema Assets An `AssetRegistry` collects prompt and schema filesystems from production module @@ -132,6 +139,14 @@ failure, empty structured body, or decode failure as material when they exist. Provider failures remain operational errors rather than output-validation failures. +When PromptKit rejects backend admission before generation, the adapter maps +`promptkit.ErrCapacityExceeded` to +`contracts.ErrLLMCapacityExceeded`, retaining prompt context and a redacted +upstream diagnostic without exposing the PromptKit sentinel as a framework +contract. A canceled caller context takes precedence. The adapter does not +retry capacity failures; the pipeline's existing binding attempt policy sees +the operational error and decides whether to rerun the complete operation. + Prompt-declared repair is executed within PromptKit’s structured-output flow. The current production D&D prompt manifests set repair attempts to zero. That setting does not replace pipeline retry behavior: a binding’s configured retry @@ -184,6 +199,8 @@ contents safe for general logging. sources, invalid asset registration, or a non-positive scheduler limit. - Preparation failures, unavailable explicit profiles, provider failures, and context cancellation propagate to the calling stage with context. +- Backend admission exhaustion is a provider-neutral operational error and is + not classified as invalid structured output or validator rejection. - Malformed or schema-invalid provider output is classified separately as invalid structured output so the module or pipeline can apply its own retry and rejection policy. diff --git a/docs/operations.md b/docs/operations.md index 077b62a..4709378 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -217,9 +217,9 @@ Provider execution settings and the generation timeout come from the selected PromptKit profile. The invocation-only **--reasoning-effort** and **--clear-reasoning-effort** controls may replace or clear that profile setting for all LLM-backed calls in one run without changing the profile. PromptKit -v0.2.0 does not add a provider retry loop; -Notarius binding retries rerun the complete module operation and validation -chain as defined by [module bindings](config.md#module-bindings-and-validators). +v0.2.0 does not add a provider retry loop. Notarius binding retries rerun the +complete module operation and validation chain as defined by +[module bindings](config.md#module-bindings-and-validators). Timeouts are layered. Caller cancellation is the outer authority. A positive effective generation timeout adds an inner request deadline, while zero @@ -228,7 +228,19 @@ transport-wide cap. Notarius does not add another timeout around PromptKit. The pinned upstream boundary and profile-format links are in [PromptKit Integration](integrations/pkg-promptkit.md). -Concurrency limits are configuration contracts; see +Concurrency has two independent layers. Notarius **total_llm** is the +application-wide provider-call limit shared by all backends, modules, retries, +and validators. PromptKit may impose a narrower admission limit for the +selected backend. The effective active-generation bound is the intersection of +both limits and can therefore be lower than **total_llm**. Built-in OpenRouter +profiles use PromptKit's upstream backend limit; endpoint-only profiles have no +PromptKit backend limit and remain bounded by Notarius. + +When a PromptKit backend has admitted all active and queued work, a new call +fails as capacity exhaustion before generation. The adapter does not retry it. +The calling stage's configured retry policy applies normally, and the run fails +if those attempts are exhausted. Caller cancellation remains authoritative. +Configuration contracts are documented under [concurrency](config.md#concurrency-output-cache-and-debug). Extract-worker limits and actual provider-call limits are independent. Notarius writes local filesystem state only; remote storage, archival, and retention automation are diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index d7f177d..22b8ae2 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -177,7 +177,8 @@ individual modules. The application-wide LLM scheduler bounds actual provider calls independently of framework worker limits. Every LLM-backed module, retry, and validator uses the single injected scheduled client, including work performed by overlapping -lanes. +lanes. Provider runtime adapters may enforce a narrower backend-specific limit +beneath this mandatory application-wide scheduler. ## Configuration And Provenance diff --git a/internal/framework/contracts/errors.go b/internal/framework/contracts/errors.go index 218641a..a2bc276 100644 --- a/internal/framework/contracts/errors.go +++ b/internal/framework/contracts/errors.go @@ -5,3 +5,7 @@ import "errors" // ErrInvalidStructuredOutput identifies a provider response that cannot satisfy // the caller's declared structured-output contract. var ErrInvalidStructuredOutput = errors.New("invalid structured output") + +// ErrLLMCapacityExceeded identifies backend admission exhaustion before model +// generation begins. +var ErrLLMCapacityExceeded = errors.New("LLM capacity exceeded") diff --git a/internal/framework/llm/promptkit_client.go b/internal/framework/llm/promptkit_client.go index c712ee9..33c935f 100644 --- a/internal/framework/llm/promptkit_client.go +++ b/internal/framework/llm/promptkit_client.go @@ -3,6 +3,7 @@ package llm import ( "context" "encoding/json" + "errors" "fmt" "net/http" "regexp" @@ -131,6 +132,14 @@ func (c *PromptKitClient) CompleteStructured(ctx context.Context, req contracts. if ctxErr := ctx.Err(); ctxErr != nil { return contracts.StructuredCompletionResponse{}, ctxErr } + if errors.Is(err, promptkit.ErrCapacityExceeded) { + return contracts.StructuredCompletionResponse{}, fmt.Errorf( + "run PromptKit prompt %q: %w: %v", + promptID, + contracts.ErrLLMCapacityExceeded, + redactPromptKitError(err), + ) + } return contracts.StructuredCompletionResponse{}, fmt.Errorf("run PromptKit prompt %q: %w", promptID, redactPromptKitError(err)) } if result == nil { diff --git a/internal/framework/llm/promptkit_client_test.go b/internal/framework/llm/promptkit_client_test.go index f2dc017..d388409 100644 --- a/internal/framework/llm/promptkit_client_test.go +++ b/internal/framework/llm/promptkit_client_test.go @@ -429,6 +429,89 @@ func TestPromptKitClientProviderFailureIncludesContextAndRedactsBearerToken(t *t } } +func TestPromptKitClientTranslatesBackendCapacityExhaustion(t *testing.T) { + queueCapacity := 0 + fake := &fakePromptKitLLM{ + err: errors.New("provider failed with Bearer secret-token"), + block: make(chan struct{}), + } + client, err := NewPromptKitClient(PromptKitClientConfig{ + Assets: newTestPromptKitAssets(t), + EngineOptions: []promptkit.Option{ + promptkit.WithBackend(promptkit.Backend{ + ID: "limited-backend", + Endpoint: "http://127.0.0.1:1/v1", + ConcurrencyLimit: 1, + QueueCapacity: &queueCapacity, + }), + promptkit.WithProfiles(promptkit.OpenAICompatibleProfile(promptkit.OpenAICompatibleProfileConfig{ + ID: "limited-profile", + BackendID: "limited-backend", + Model: "limited-model", + })), + promptkit.WithLLMClient(fake), + }, + }) + if err != nil { + t.Fatalf("NewPromptKitClient() error = %v, want nil", err) + } + defer func() { + select { + case <-fake.block: + default: + close(fake.block) + } + }() + + request := contracts.StructuredCompletionRequest{ + PromptID: "adapter.direct-session", + ProfileID: "limited-profile", + Inputs: contracts.LLMInputSet{ + "transcript": contracts.NewLLMInputMaterial("transcript", "application/json", []byte(`{"source":true}`), "", ""), + }, + Vars: map[string]any{"custom": "value"}, + } + firstResult := make(chan error, 1) + go func() { + var out map[string]any + _, callErr := client.CompleteStructured(context.Background(), request, &out) + firstResult <- callErr + }() + waitForAtomicAtLeast(t, &fake.calls, 1) + + var out map[string]any + response, capacityErr := client.CompleteStructured(context.Background(), request, &out) + if len(response.Content) != 0 { + t.Fatalf("capacity response = %#v, want empty", response) + } + if !errors.Is(capacityErr, contracts.ErrLLMCapacityExceeded) { + t.Fatalf("capacity error = %v, want ErrLLMCapacityExceeded", capacityErr) + } + if errors.Is(capacityErr, contracts.ErrInvalidStructuredOutput) { + t.Fatalf("capacity error = %v, must not be invalid structured output", capacityErr) + } + if errors.Is(capacityErr, promptkit.ErrCapacityExceeded) { + t.Fatalf("capacity error exposes PromptKit sentinel: %v", capacityErr) + } + if !strings.Contains(capacityErr.Error(), `run PromptKit prompt "adapter.direct-session"`) || + !strings.Contains(capacityErr.Error(), "backend capacity exceeded") { + t.Fatalf("capacity error = %q, want prompt context and upstream diagnostic", capacityErr) + } + if calls := atomic.LoadInt32(&fake.calls); calls != 1 { + t.Fatalf("provider calls after capacity rejection = %d, want 1", calls) + } + + close(fake.block) + firstErr := <-firstResult + if firstErr == nil || strings.Contains(firstErr.Error(), "secret-token") || + !strings.Contains(firstErr.Error(), "Bearer [REDACTED]") { + t.Fatalf("admitted provider error = %v, want redacted diagnostic", firstErr) + } + if calls := atomic.LoadInt32(&fake.calls); calls != 1 { + t.Fatalf("provider calls after release = %d, want no adapter retry", calls) + } +} + func TestPromptKitClientContextCancellationIsRespected(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel()