Simplify Audita's backend capacity requirements
This commit is contained in:
@@ -102,26 +102,36 @@ carry generated output, validation findings, and partial execution observations.
|
||||
|
||||
Promptkit need not decide Audita's recovery or correction-acceptance policy.
|
||||
|
||||
## 4. Aggregate and Workload Capacity Controls
|
||||
## 4. Simple Per-Backend Capacity Limits
|
||||
|
||||
**Priority: Evaluate after the execution boundary is established.** Audita currently
|
||||
needs to reason about total LLM concurrency and separate proposal/validation work,
|
||||
which are not equivalent to per-backend limits.
|
||||
**Priority: Adoption requirement, largely supported by existing facilities.**
|
||||
Audita needs only two protections:
|
||||
|
||||
A useful extension would let a consumer impose an overall concurrent-generation
|
||||
ceiling alongside backend limits. If the consumer need remains after redesign,
|
||||
independently named workload budgets could distinguish proposal from validation
|
||||
work without introducing Audita-specific concepts into Promptkit.
|
||||
- For a local backend with a hard simultaneous-query limit, keep combined Audita
|
||||
traffic within its assigned capacity. For example, a DGX Spark serving
|
||||
gemma-4-31b-it may allow four simultaneous queries.
|
||||
- For a paid API backend, use a reasonable finite concurrency cap so a large
|
||||
transcript or pipeline cannot accidentally launch dozens or thousands of
|
||||
simultaneous billable calls.
|
||||
|
||||
The desired outcome is predictable bounded provider activity across initial calls,
|
||||
retries, and repair calls, with cancellation-aware waiting and understandable
|
||||
capacity rejection. Delayed retries should not unnecessarily prevent unrelated
|
||||
eligible work from progressing. Consumers need clear scope and interaction rules
|
||||
for all selected limits.
|
||||
Proposal and validation calls targeting the same backend should share its limit.
|
||||
The protection must cover initial calls, transient retries, and content-repair
|
||||
calls. Audita does not need separate proposal/validation budgets, named workload
|
||||
limits, or an additional aggregate ceiling across backends.
|
||||
|
||||
This is not a request to reproduce Audita's scheduler or introduce a general task
|
||||
scheduler. Determine whether existing controls and application orchestration are
|
||||
sufficient before expanding the public capacity model.
|
||||
Promptkit's existing engine-local, per-backend capacity controls are the intended
|
||||
foundation. Audita should select finite limits for the backends it uses; adoption
|
||||
must not accidentally leave paid endpoints on an unlimited path. Exact defaults
|
||||
remain to be selected. No new fine-grained scheduling feature is requested.
|
||||
|
||||
Verify the documented capacity scope in the consumer example, especially as retries
|
||||
are added. An engine-local limit does not coordinate other processes or independent
|
||||
clients using the same service; a shared local backend must have capacity allocated
|
||||
accordingly or enforced at the service boundary. Cross-process coordination is not
|
||||
an upstream feature request here.
|
||||
|
||||
Concurrency limits bound simultaneous activity, not total token usage or spending
|
||||
across a long run. A monetary budget or rate-limiting system is not requested.
|
||||
|
||||
## 5. Representative Consumer Example
|
||||
|
||||
@@ -154,9 +164,10 @@ This proposal does not change existing releases or their licenses.
|
||||
## Suggested Evaluation Order and Scope
|
||||
|
||||
Evaluate retries, failure classification, and execution observations together, then
|
||||
use the representative consumer example to assess readiness. Consider additional
|
||||
capacity controls only after that exercise establishes a concrete remaining need.
|
||||
Resolve licensing before selecting a release for Audita's migration.
|
||||
use the representative consumer example to assess readiness and verify shared
|
||||
per-backend limits. Existing capacity controls should suffice unless that exercise
|
||||
reveals a concrete gap. Resolve licensing before selecting a release for Audita's
|
||||
migration.
|
||||
|
||||
Existing facilities for embedded sources, application-owned profiles, prepared
|
||||
execution, JSON-schema validation, and request-scoped credentials are useful
|
||||
|
||||
Reference in New Issue
Block a user