promptkit/docs/roadmap/audita-feature-requests.md

Audita Consumer Feature Requests

Last updated: 2026-09-23

Status: Requests 1–6 are implemented in source but not yet released. These requests originated from a review of Promptkit v0.10.0. They retain Audita's consumer context and priorities, not a current API specification or a delivery commitment for Audita adoption.

Requests 1–5 were resolved through the reliable-execution roadmap and its implementation plan. The consumer guide and public GoDoc own current behavior. No release or Audita adoption is claimed here.

Consumer Context

Audita polishes transcripts through ordered correction modules. It makes LLM calls both to propose corrections and to assess proposed changes, alongside deterministic domain validation. It needs bounded execution, understandable failures, and useful run diagnostics without disclosing credentials or transcript content unnecessarily.

Audita would like Promptkit to own shared prompt and LLM execution capabilities, while Audita retains transcript processing, module orchestration, domain acceptance, correction application, reports, and diagnostic artifact retention. Suite-wide consistency is the adoption goal; these requests do not ask Promptkit to reproduce Audita's existing APIs or internal architecture.

The consumer guide, format reference, and provider integration contract remain the owners of implemented behavior. This document specifies desired consumer outcomes only; API shape and implementation are left to Promptkit.

1. Transient-Failure Retries

Implemented in source; not yet released. The built-in HTTP path owns the selected transient-status policy; injected clients keep their own policy.

Priority: High. Audita should not need its own general-purpose provider retry policy to recover from temporary rate limiting or service failures.

Requested behavior:

• Retry HTTP 429 and selected transient server failures. Suggested initial defaults cover 500, 502, 503, and 504; retrying every 5xx status is not a requirement.
• Supply reasonable documented defaults. Two additional attempts is a proposed starting point, with a configurable maximum and an unambiguous explicit zero that disables retries.
• Delay retries to avoid immediate repeated requests, honor applicable provider retry timing, and bound the operation by the caller's cancellation/deadline. Cancellation must interrupt waiting as well as active requests.
• Define which transport failures are eligible. Document that an uncertain response can mean the provider already performed billable work; retries do not imply exactly-once generation.
• Keep retry attempts tied to the same prepared generation request. An automatic retry must not silently select different prompt content, model settings, or credential identity.
• Distinguish transient-failure retries from content-repair calls. Document how their budgets combine so consumers can reason about maximum request counts, latency, and potential cost.
• State whether the policy applies to built-in and injected clients. Consumers must be able to avoid duplicate retries when another collaborator already owns retry behavior.

Expected consumer evidence includes a transient failure followed by success, exhaustion, explicit retry disablement, cancellation during a delay, and interaction with content repair. Permanent request failures must remain distinguishable.

2. Execution Observations, Including Failed Runs

Implemented in source; not yet released. Optional metadata callbacks cover ordinary and claimed prepared execution, including normal error returns.

Priority: High. Audita needs to explain the time and calls consumed by a logical operation, including when that operation never produces a successful result.

Requested observations include:

• Correlation between the logical operation and its individual attempts, selected prompt, profile, and backend.
• Attempt outcomes and enough timing information to distinguish capacity waiting, generation, and retry delays.
• Separate counts for transient-failure retries and content-repair calls.
• Available provider-reported token usage, including completed calls preceding a later failure, with unknown usage distinguishable from measured zero.
• Clear accounting boundaries so consumers do not double-count aggregate and individual-attempt measurements.

Observations should be usable by application-owned reports, logging, and metrics without requiring a particular telemetry system or artifact layout. Default observations should contain metadata rather than credentials or prompt/response bodies. Any content disclosure should be an explicit consumer choice, with its sensitivity clearly documented.

Current result metadata and usage remain useful; the request is to make execution and failure accounting sufficiently complete for downstream diagnostics.

3. Actionable Failure Classification

Implemented in source; not yet released. Public sentinels and typed errors preserve provider status, retry exhaustion, transport, malformed-response, capacity, validation, and context distinctions.

Priority: High; closely related to retries and observations. Audita needs to choose between failing a run, rejecting affected work, and recording a warning without interpreting human-readable error messages.

Build on existing public error and validation-result contracts so consumers can reliably distinguish cancellation, deadline expiration, transport failure, non-success provider status, malformed provider response, exhausted retries, capacity rejection, and completed output-validation failure.

On retry exhaustion, retain the underlying failure classification and attempt count. Preserve safe default diagnostics while allowing deliberate access to provider detail under the consumer's disclosure policy. Document which outcomes carry generated output, validation findings, and partial execution observations.

Promptkit need not decide Audita's recovery or correction-acceptance policy.

4. Simple Per-Backend Capacity Limits

Implemented in source; not yet released. One engine-local backend pool coordinates initial, retry, and repair attempts across profiles selecting it. Audita must still configure finite limits for the paid backends it adopts.

Priority: Adoption requirement, largely supported by existing facilities. Audita needs only two protections:

• For a local backend with a hard simultaneous-query limit, keep combined Audita traffic within its assigned capacity. For example, a DGX Spark serving gemma-4-31b-it may allow four simultaneous queries.
• For a paid API backend, use a reasonable finite concurrency cap so a large transcript or pipeline cannot accidentally launch dozens or thousands of simultaneous billable calls.

Proposal and validation calls targeting the same backend should share its limit. The protection must cover initial calls, transient retries, and content-repair calls. Audita does not need separate proposal/validation budgets, named workload limits, or an additional aggregate ceiling across backends.

Promptkit's existing engine-local, per-backend capacity controls are the intended foundation. Audita should select finite limits for the backends it uses; adoption must not accidentally leave paid endpoints on an unlimited path. Exact defaults remain to be selected. No new fine-grained scheduling feature is requested.

Verify the documented capacity scope in the consumer example, especially as retries are added. An engine-local limit does not coordinate other processes or independent clients using the same service; a shared local backend must have capacity allocated accordingly or enforced at the service boundary. Cross-process coordination is not an upstream feature request here.

Concurrency limits bound simultaneous activity, not total token usage or spending across a long run. A monetary budget or rate-limiting system is not requested.

5. Representative Consumer Example

Implemented in source; not yet released. The offline example demonstrates the built-in retry, structured proposal and assessment, metadata collection, and separate application acceptance decision. Focused tests, not example switches, cover cancellation and invalid-output branches.

Priority: High as adoption evidence. Provide a maintained offline example of a structured proposal call followed by an independent LLM validation call, using embedded prompts/schemas and separate application-owned execution profiles.

Demonstrate transient failure handling, schema-invalid output, cancellation, and application collection of execution observations. Use synthetic inputs and controlled model responses without credentials or paid provider calls.

Keep schema validity distinct from domain acceptance: a structurally valid response may still be rejected by Audita's deterministic or semantic checks. The example should show how an application uses Promptkit's public contracts, not duplicate Audita's processing pipeline.

6. Licensing Recommendation

Implemented in source; not yet released. The maintainer authorized LGPL-3.0-or-later. See the license summary, LGPL notice and text, and accompanying GPL text. Existing release tags remain unchanged. Audita must select a subsequent published release containing this change and satisfy the applicable distribution obligations.

Suggested Evaluation Order and Scope

The implemented retry, classification, observation, and per-backend capacity contracts can now be evaluated together with the representative example. Audita still owns its migration decision and must select a later published release before adopting this source state.

Existing facilities for embedded sources, application-owned profiles, prepared execution, JSON-schema validation, and request-scoped credentials are useful adoption foundations rather than new requests. Audita can bridge its application configuration and credentials into Promptkit's public API; a dependency between Promptkit and ConfigKit is not requested.

Transcript normalization, chunking, module ordering, semantic validators, correction application, correction-ledger assembly, and filesystem retention remain Audita's responsibility. No requested feature requires Promptkit to own those concerns.