201 lines
10 KiB
Markdown
201 lines
10 KiB
Markdown
# Audita Consumer Feature Requests
|
||
|
||
**Last updated:** 2026-09-23
|
||
|
||
**Status:** Requests 1–6 are implemented in source but not yet released.
|
||
These requests originated from a review of Promptkit v0.10.0. They retain
|
||
Audita's consumer context and priorities, not a current API specification or
|
||
a delivery commitment for Audita adoption.
|
||
|
||
Requests 1–5 were resolved through the
|
||
[reliable-execution roadmap](reliable-execution.md) and its
|
||
[implementation plan](implementation.md). The [consumer guide](../consumers/pkg-promptkit.md)
|
||
and public GoDoc own current behavior. No release or Audita adoption is
|
||
claimed here.
|
||
|
||
## Consumer Context
|
||
|
||
Audita polishes transcripts through ordered correction modules. It makes LLM calls
|
||
both to propose corrections and to assess proposed changes, alongside deterministic
|
||
domain validation. It needs bounded execution, understandable failures, and useful
|
||
run diagnostics without disclosing credentials or transcript content unnecessarily.
|
||
|
||
Audita would like Promptkit to own shared prompt and LLM execution capabilities,
|
||
while Audita retains transcript processing, module orchestration, domain acceptance,
|
||
correction application, reports, and diagnostic artifact retention. Suite-wide
|
||
consistency is the adoption goal; these requests do not ask Promptkit to reproduce
|
||
Audita's existing APIs or internal architecture.
|
||
|
||
The [consumer guide](../consumers/pkg-promptkit.md),
|
||
[format reference](../formats.md), and
|
||
[provider integration contract](../integrations/openai-compatible-chat.md)
|
||
remain the owners of implemented behavior. This document specifies desired
|
||
consumer outcomes only; API shape and implementation are left to Promptkit.
|
||
|
||
## 1. Transient-Failure Retries
|
||
|
||
**Implemented in source; not yet released.** The built-in HTTP path owns the
|
||
selected transient-status policy; injected clients keep their own policy.
|
||
|
||
**Priority: High.** Audita should not need its own general-purpose provider retry
|
||
policy to recover from temporary rate limiting or service failures.
|
||
|
||
Requested behavior:
|
||
|
||
- Retry HTTP 429 and selected transient server failures. Suggested initial defaults
|
||
cover 500, 502, 503, and 504; retrying every 5xx status is not a requirement.
|
||
- Supply reasonable documented defaults. Two additional attempts is a proposed
|
||
starting point, with a configurable maximum and an unambiguous explicit zero
|
||
that disables retries.
|
||
- Delay retries to avoid immediate repeated requests, honor applicable provider
|
||
retry timing, and bound the operation by the caller's cancellation/deadline.
|
||
Cancellation must interrupt waiting as well as active requests.
|
||
- Define which transport failures are eligible. Document that an uncertain response
|
||
can mean the provider already performed billable work; retries do not imply
|
||
exactly-once generation.
|
||
- Keep retry attempts tied to the same prepared generation request. An automatic
|
||
retry must not silently select different prompt content, model settings, or
|
||
credential identity.
|
||
- Distinguish transient-failure retries from content-repair calls. Document how
|
||
their budgets combine so consumers can reason about maximum request counts,
|
||
latency, and potential cost.
|
||
- State whether the policy applies to built-in and injected clients. Consumers
|
||
must be able to avoid duplicate retries when another collaborator already owns
|
||
retry behavior.
|
||
|
||
Expected consumer evidence includes a transient failure followed by success,
|
||
exhaustion, explicit retry disablement, cancellation during a delay, and interaction
|
||
with content repair. Permanent request failures must remain distinguishable.
|
||
|
||
## 2. Execution Observations, Including Failed Runs
|
||
|
||
**Implemented in source; not yet released.** Optional metadata callbacks cover
|
||
ordinary and claimed prepared execution, including normal error returns.
|
||
|
||
**Priority: High.** Audita needs to explain the time and calls consumed by a logical
|
||
operation, including when that operation never produces a successful result.
|
||
|
||
Requested observations include:
|
||
|
||
- Correlation between the logical operation and its individual attempts, selected
|
||
prompt, profile, and backend.
|
||
- Attempt outcomes and enough timing information to distinguish capacity waiting,
|
||
generation, and retry delays.
|
||
- Separate counts for transient-failure retries and content-repair calls.
|
||
- Available provider-reported token usage, including completed calls preceding a
|
||
later failure, with unknown usage distinguishable from measured zero.
|
||
- Clear accounting boundaries so consumers do not double-count aggregate and
|
||
individual-attempt measurements.
|
||
|
||
Observations should be usable by application-owned reports, logging, and metrics
|
||
without requiring a particular telemetry system or artifact layout. Default
|
||
observations should contain metadata rather than credentials or prompt/response
|
||
bodies. Any content disclosure should be an explicit consumer choice, with its
|
||
sensitivity clearly documented.
|
||
|
||
Current result metadata and usage remain useful; the request is to make execution
|
||
and failure accounting sufficiently complete for downstream diagnostics.
|
||
|
||
## 3. Actionable Failure Classification
|
||
|
||
**Implemented in source; not yet released.** Public sentinels and typed errors
|
||
preserve provider status, retry exhaustion, transport, malformed-response,
|
||
capacity, validation, and context distinctions.
|
||
|
||
**Priority: High; closely related to retries and observations.** Audita needs to
|
||
choose between failing a run, rejecting affected work, and recording a warning
|
||
without interpreting human-readable error messages.
|
||
|
||
Build on existing public error and validation-result contracts so consumers can
|
||
reliably distinguish cancellation, deadline expiration, transport failure,
|
||
non-success provider status, malformed provider response, exhausted retries,
|
||
capacity rejection, and completed output-validation failure.
|
||
|
||
On retry exhaustion, retain the underlying failure classification and attempt
|
||
count. Preserve safe default diagnostics while allowing deliberate access to
|
||
provider detail under the consumer's disclosure policy. Document which outcomes
|
||
carry generated output, validation findings, and partial execution observations.
|
||
|
||
Promptkit need not decide Audita's recovery or correction-acceptance policy.
|
||
|
||
## 4. Simple Per-Backend Capacity Limits
|
||
|
||
**Implemented in source; not yet released.** One engine-local backend pool
|
||
coordinates initial, retry, and repair attempts across profiles selecting it.
|
||
Audita must still configure finite limits for the paid backends it adopts.
|
||
|
||
**Priority: Adoption requirement, largely supported by existing facilities.**
|
||
Audita needs only two protections:
|
||
|
||
- For a local backend with a hard simultaneous-query limit, keep combined Audita
|
||
traffic within its assigned capacity. For example, a DGX Spark serving
|
||
gemma-4-31b-it may allow four simultaneous queries.
|
||
- For a paid API backend, use a reasonable finite concurrency cap so a large
|
||
transcript or pipeline cannot accidentally launch dozens or thousands of
|
||
simultaneous billable calls.
|
||
|
||
Proposal and validation calls targeting the same backend should share its limit.
|
||
The protection must cover initial calls, transient retries, and content-repair
|
||
calls. Audita does not need separate proposal/validation budgets, named workload
|
||
limits, or an additional aggregate ceiling across backends.
|
||
|
||
Promptkit's existing engine-local, per-backend capacity controls are the intended
|
||
foundation. Audita should select finite limits for the backends it uses; adoption
|
||
must not accidentally leave paid endpoints on an unlimited path. Exact defaults
|
||
remain to be selected. No new fine-grained scheduling feature is requested.
|
||
|
||
Verify the documented capacity scope in the consumer example, especially as retries
|
||
are added. An engine-local limit does not coordinate other processes or independent
|
||
clients using the same service; a shared local backend must have capacity allocated
|
||
accordingly or enforced at the service boundary. Cross-process coordination is not
|
||
an upstream feature request here.
|
||
|
||
Concurrency limits bound simultaneous activity, not total token usage or spending
|
||
across a long run. A monetary budget or rate-limiting system is not requested.
|
||
|
||
## 5. Representative Consumer Example
|
||
|
||
**Implemented in source; not yet released.** The
|
||
[offline example](../../examples/go-library/observe/main.go) demonstrates the
|
||
built-in retry, structured proposal and assessment, metadata collection, and
|
||
separate application acceptance decision. Focused tests, not example switches,
|
||
cover cancellation and invalid-output branches.
|
||
|
||
**Priority: High as adoption evidence.** Provide a maintained offline example of a
|
||
structured proposal call followed by an independent LLM validation call, using
|
||
embedded prompts/schemas and separate application-owned execution profiles.
|
||
|
||
Demonstrate transient failure handling, schema-invalid output, cancellation, and
|
||
application collection of execution observations. Use synthetic inputs and
|
||
controlled model responses without credentials or paid provider calls.
|
||
|
||
Keep schema validity distinct from domain acceptance: a structurally valid response
|
||
may still be rejected by Audita's deterministic or semantic checks. The example
|
||
should show how an application uses Promptkit's public contracts, not duplicate
|
||
Audita's processing pipeline.
|
||
|
||
## 6. Licensing Recommendation
|
||
|
||
**Implemented in source; not yet released.** The maintainer authorized
|
||
`LGPL-3.0-or-later`. See the [license summary](../../README.md#license),
|
||
[LGPL notice and text](../../LICENSE), and [accompanying GPL text](../../COPYING).
|
||
Existing release tags remain unchanged. Audita must select a subsequent published
|
||
release containing this change and satisfy the applicable distribution obligations.
|
||
|
||
## Suggested Evaluation Order and Scope
|
||
|
||
The implemented retry, classification, observation, and per-backend capacity
|
||
contracts can now be evaluated together with the representative example.
|
||
Audita still owns its migration decision and must select a later published
|
||
release before adopting this source state.
|
||
|
||
Existing facilities for embedded sources, application-owned profiles, prepared
|
||
execution, JSON-schema validation, and request-scoped credentials are useful
|
||
adoption foundations rather than new requests. Audita can bridge its application
|
||
configuration and credentials into Promptkit's public API; a dependency between
|
||
Promptkit and ConfigKit is not requested.
|
||
|
||
Transcript normalization, chunking, module ordering, semantic validators, correction
|
||
application, correction-ledger assembly, and filesystem retention remain Audita's
|
||
responsibility. No requested feature requires Promptkit to own those concerns.
|