Simplify Audita's backend capacity requirements

This commit is contained in:
2026-09-23 15:35:50 +00:00
parent 5dd5b2f3cc
commit 9f69da716b

View File

@@ -102,26 +102,36 @@ carry generated output, validation findings, and partial execution observations.
Promptkit need not decide Audita's recovery or correction-acceptance policy. Promptkit need not decide Audita's recovery or correction-acceptance policy.
## 4. Aggregate and Workload Capacity Controls ## 4. Simple Per-Backend Capacity Limits
**Priority: Evaluate after the execution boundary is established.** Audita currently **Priority: Adoption requirement, largely supported by existing facilities.**
needs to reason about total LLM concurrency and separate proposal/validation work, Audita needs only two protections:
which are not equivalent to per-backend limits.
A useful extension would let a consumer impose an overall concurrent-generation - For a local backend with a hard simultaneous-query limit, keep combined Audita
ceiling alongside backend limits. If the consumer need remains after redesign, traffic within its assigned capacity. For example, a DGX Spark serving
independently named workload budgets could distinguish proposal from validation gemma-4-31b-it may allow four simultaneous queries.
work without introducing Audita-specific concepts into Promptkit. - For a paid API backend, use a reasonable finite concurrency cap so a large
transcript or pipeline cannot accidentally launch dozens or thousands of
simultaneous billable calls.
The desired outcome is predictable bounded provider activity across initial calls, Proposal and validation calls targeting the same backend should share its limit.
retries, and repair calls, with cancellation-aware waiting and understandable The protection must cover initial calls, transient retries, and content-repair
capacity rejection. Delayed retries should not unnecessarily prevent unrelated calls. Audita does not need separate proposal/validation budgets, named workload
eligible work from progressing. Consumers need clear scope and interaction rules limits, or an additional aggregate ceiling across backends.
for all selected limits.
This is not a request to reproduce Audita's scheduler or introduce a general task Promptkit's existing engine-local, per-backend capacity controls are the intended
scheduler. Determine whether existing controls and application orchestration are foundation. Audita should select finite limits for the backends it uses; adoption
sufficient before expanding the public capacity model. must not accidentally leave paid endpoints on an unlimited path. Exact defaults
remain to be selected. No new fine-grained scheduling feature is requested.
Verify the documented capacity scope in the consumer example, especially as retries
are added. An engine-local limit does not coordinate other processes or independent
clients using the same service; a shared local backend must have capacity allocated
accordingly or enforced at the service boundary. Cross-process coordination is not
an upstream feature request here.
Concurrency limits bound simultaneous activity, not total token usage or spending
across a long run. A monetary budget or rate-limiting system is not requested.
## 5. Representative Consumer Example ## 5. Representative Consumer Example
@@ -154,9 +164,10 @@ This proposal does not change existing releases or their licenses.
## Suggested Evaluation Order and Scope ## Suggested Evaluation Order and Scope
Evaluate retries, failure classification, and execution observations together, then Evaluate retries, failure classification, and execution observations together, then
use the representative consumer example to assess readiness. Consider additional use the representative consumer example to assess readiness and verify shared
capacity controls only after that exercise establishes a concrete remaining need. per-backend limits. Existing capacity controls should suffice unless that exercise
Resolve licensing before selecting a release for Audita's migration. reveals a concrete gap. Resolve licensing before selecting a release for Audita's
migration.
Existing facilities for embedded sources, application-owned profiles, prepared Existing facilities for embedded sources, application-owned profiles, prepared
execution, JSON-schema validation, and request-scoped credentials are useful execution, JSON-schema validation, and request-scoped credentials are useful