diff --git a/docs/roadmap/audita-feature-requests.md b/docs/roadmap/audita-feature-requests.md index 565c0d5..84451fd 100644 --- a/docs/roadmap/audita-feature-requests.md +++ b/docs/roadmap/audita-feature-requests.md @@ -102,26 +102,36 @@ carry generated output, validation findings, and partial execution observations. Promptkit need not decide Audita's recovery or correction-acceptance policy. -## 4. Aggregate and Workload Capacity Controls +## 4. Simple Per-Backend Capacity Limits -**Priority: Evaluate after the execution boundary is established.** Audita currently -needs to reason about total LLM concurrency and separate proposal/validation work, -which are not equivalent to per-backend limits. +**Priority: Adoption requirement, largely supported by existing facilities.** +Audita needs only two protections: -A useful extension would let a consumer impose an overall concurrent-generation -ceiling alongside backend limits. If the consumer need remains after redesign, -independently named workload budgets could distinguish proposal from validation -work without introducing Audita-specific concepts into Promptkit. +- For a local backend with a hard simultaneous-query limit, keep combined Audita + traffic within its assigned capacity. For example, a DGX Spark serving + gemma-4-31b-it may allow four simultaneous queries. +- For a paid API backend, use a reasonable finite concurrency cap so a large + transcript or pipeline cannot accidentally launch dozens or thousands of + simultaneous billable calls. -The desired outcome is predictable bounded provider activity across initial calls, -retries, and repair calls, with cancellation-aware waiting and understandable -capacity rejection. Delayed retries should not unnecessarily prevent unrelated -eligible work from progressing. Consumers need clear scope and interaction rules -for all selected limits. +Proposal and validation calls targeting the same backend should share its limit. +The protection must cover initial calls, transient retries, and content-repair +calls. Audita does not need separate proposal/validation budgets, named workload +limits, or an additional aggregate ceiling across backends. -This is not a request to reproduce Audita's scheduler or introduce a general task -scheduler. Determine whether existing controls and application orchestration are -sufficient before expanding the public capacity model. +Promptkit's existing engine-local, per-backend capacity controls are the intended +foundation. Audita should select finite limits for the backends it uses; adoption +must not accidentally leave paid endpoints on an unlimited path. Exact defaults +remain to be selected. No new fine-grained scheduling feature is requested. + +Verify the documented capacity scope in the consumer example, especially as retries +are added. An engine-local limit does not coordinate other processes or independent +clients using the same service; a shared local backend must have capacity allocated +accordingly or enforced at the service boundary. Cross-process coordination is not +an upstream feature request here. + +Concurrency limits bound simultaneous activity, not total token usage or spending +across a long run. A monetary budget or rate-limiting system is not requested. ## 5. Representative Consumer Example @@ -154,9 +164,10 @@ This proposal does not change existing releases or their licenses. ## Suggested Evaluation Order and Scope Evaluate retries, failure classification, and execution observations together, then -use the representative consumer example to assess readiness. Consider additional -capacity controls only after that exercise establishes a concrete remaining need. -Resolve licensing before selecting a release for Audita's migration. +use the representative consumer example to assess readiness and verify shared +per-backend limits. Existing capacity controls should suffice unless that exercise +reveals a concrete gap. Resolve licensing before selecting a release for Audita's +migration. Existing facilities for embedded sources, application-owned profiles, prepared execution, JSON-schema validation, and request-scoped credentials are useful