Simplify Audita's backend capacity requirements
This commit is contained in:
@@ -102,26 +102,36 @@ carry generated output, validation findings, and partial execution observations.
|
|||||||
|
|
||||||
Promptkit need not decide Audita's recovery or correction-acceptance policy.
|
Promptkit need not decide Audita's recovery or correction-acceptance policy.
|
||||||
|
|
||||||
## 4. Aggregate and Workload Capacity Controls
|
## 4. Simple Per-Backend Capacity Limits
|
||||||
|
|
||||||
**Priority: Evaluate after the execution boundary is established.** Audita currently
|
**Priority: Adoption requirement, largely supported by existing facilities.**
|
||||||
needs to reason about total LLM concurrency and separate proposal/validation work,
|
Audita needs only two protections:
|
||||||
which are not equivalent to per-backend limits.
|
|
||||||
|
|
||||||
A useful extension would let a consumer impose an overall concurrent-generation
|
- For a local backend with a hard simultaneous-query limit, keep combined Audita
|
||||||
ceiling alongside backend limits. If the consumer need remains after redesign,
|
traffic within its assigned capacity. For example, a DGX Spark serving
|
||||||
independently named workload budgets could distinguish proposal from validation
|
gemma-4-31b-it may allow four simultaneous queries.
|
||||||
work without introducing Audita-specific concepts into Promptkit.
|
- For a paid API backend, use a reasonable finite concurrency cap so a large
|
||||||
|
transcript or pipeline cannot accidentally launch dozens or thousands of
|
||||||
|
simultaneous billable calls.
|
||||||
|
|
||||||
The desired outcome is predictable bounded provider activity across initial calls,
|
Proposal and validation calls targeting the same backend should share its limit.
|
||||||
retries, and repair calls, with cancellation-aware waiting and understandable
|
The protection must cover initial calls, transient retries, and content-repair
|
||||||
capacity rejection. Delayed retries should not unnecessarily prevent unrelated
|
calls. Audita does not need separate proposal/validation budgets, named workload
|
||||||
eligible work from progressing. Consumers need clear scope and interaction rules
|
limits, or an additional aggregate ceiling across backends.
|
||||||
for all selected limits.
|
|
||||||
|
|
||||||
This is not a request to reproduce Audita's scheduler or introduce a general task
|
Promptkit's existing engine-local, per-backend capacity controls are the intended
|
||||||
scheduler. Determine whether existing controls and application orchestration are
|
foundation. Audita should select finite limits for the backends it uses; adoption
|
||||||
sufficient before expanding the public capacity model.
|
must not accidentally leave paid endpoints on an unlimited path. Exact defaults
|
||||||
|
remain to be selected. No new fine-grained scheduling feature is requested.
|
||||||
|
|
||||||
|
Verify the documented capacity scope in the consumer example, especially as retries
|
||||||
|
are added. An engine-local limit does not coordinate other processes or independent
|
||||||
|
clients using the same service; a shared local backend must have capacity allocated
|
||||||
|
accordingly or enforced at the service boundary. Cross-process coordination is not
|
||||||
|
an upstream feature request here.
|
||||||
|
|
||||||
|
Concurrency limits bound simultaneous activity, not total token usage or spending
|
||||||
|
across a long run. A monetary budget or rate-limiting system is not requested.
|
||||||
|
|
||||||
## 5. Representative Consumer Example
|
## 5. Representative Consumer Example
|
||||||
|
|
||||||
@@ -154,9 +164,10 @@ This proposal does not change existing releases or their licenses.
|
|||||||
## Suggested Evaluation Order and Scope
|
## Suggested Evaluation Order and Scope
|
||||||
|
|
||||||
Evaluate retries, failure classification, and execution observations together, then
|
Evaluate retries, failure classification, and execution observations together, then
|
||||||
use the representative consumer example to assess readiness. Consider additional
|
use the representative consumer example to assess readiness and verify shared
|
||||||
capacity controls only after that exercise establishes a concrete remaining need.
|
per-backend limits. Existing capacity controls should suffice unless that exercise
|
||||||
Resolve licensing before selecting a release for Audita's migration.
|
reveals a concrete gap. Resolve licensing before selecting a release for Audita's
|
||||||
|
migration.
|
||||||
|
|
||||||
Existing facilities for embedded sources, application-owned profiles, prepared
|
Existing facilities for embedded sources, application-owned profiles, prepared
|
||||||
execution, JSON-schema validation, and request-scoped credentials are useful
|
execution, JSON-schema validation, and request-scoped credentials are useful
|
||||||
|
|||||||
Reference in New Issue
Block a user