Simplify Audita's backend capacity requirements

This commit is contained in:
2026-09-23 15:35:50 +00:00
parent 5dd5b2f3cc
commit 9f69da716b

View File

@@ -102,26 +102,36 @@ carry generated output, validation findings, and partial execution observations.
Promptkit need not decide Audita's recovery or correction-acceptance policy.
## 4. Aggregate and Workload Capacity Controls
## 4. Simple Per-Backend Capacity Limits
**Priority: Evaluate after the execution boundary is established.** Audita currently
needs to reason about total LLM concurrency and separate proposal/validation work,
which are not equivalent to per-backend limits.
**Priority: Adoption requirement, largely supported by existing facilities.**
Audita needs only two protections:
A useful extension would let a consumer impose an overall concurrent-generation
ceiling alongside backend limits. If the consumer need remains after redesign,
independently named workload budgets could distinguish proposal from validation
work without introducing Audita-specific concepts into Promptkit.
- For a local backend with a hard simultaneous-query limit, keep combined Audita
traffic within its assigned capacity. For example, a DGX Spark serving
gemma-4-31b-it may allow four simultaneous queries.
- For a paid API backend, use a reasonable finite concurrency cap so a large
transcript or pipeline cannot accidentally launch dozens or thousands of
simultaneous billable calls.
The desired outcome is predictable bounded provider activity across initial calls,
retries, and repair calls, with cancellation-aware waiting and understandable
capacity rejection. Delayed retries should not unnecessarily prevent unrelated
eligible work from progressing. Consumers need clear scope and interaction rules
for all selected limits.
Proposal and validation calls targeting the same backend should share its limit.
The protection must cover initial calls, transient retries, and content-repair
calls. Audita does not need separate proposal/validation budgets, named workload
limits, or an additional aggregate ceiling across backends.
This is not a request to reproduce Audita's scheduler or introduce a general task
scheduler. Determine whether existing controls and application orchestration are
sufficient before expanding the public capacity model.
Promptkit's existing engine-local, per-backend capacity controls are the intended
foundation. Audita should select finite limits for the backends it uses; adoption
must not accidentally leave paid endpoints on an unlimited path. Exact defaults
remain to be selected. No new fine-grained scheduling feature is requested.
Verify the documented capacity scope in the consumer example, especially as retries
are added. An engine-local limit does not coordinate other processes or independent
clients using the same service; a shared local backend must have capacity allocated
accordingly or enforced at the service boundary. Cross-process coordination is not
an upstream feature request here.
Concurrency limits bound simultaneous activity, not total token usage or spending
across a long run. A monetary budget or rate-limiting system is not requested.
## 5. Representative Consumer Example
@@ -154,9 +164,10 @@ This proposal does not change existing releases or their licenses.
## Suggested Evaluation Order and Scope
Evaluate retries, failure classification, and execution observations together, then
use the representative consumer example to assess readiness. Consider additional
capacity controls only after that exercise establishes a concrete remaining need.
Resolve licensing before selecting a release for Audita's migration.
use the representative consumer example to assess readiness and verify shared
per-backend limits. Existing capacity controls should suffice unless that exercise
reveals a concrete gap. Resolve licensing before selecting a release for Audita's
migration.
Existing facilities for embedded sources, application-owned profiles, prepared
execution, JSON-schema validation, and request-scoped credentials are useful