diff --git a/docs/roadmap/audita-feature-requests.md b/docs/roadmap/audita-feature-requests.md new file mode 100644 index 0000000..565c0d5 --- /dev/null +++ b/docs/roadmap/audita-feature-requests.md @@ -0,0 +1,169 @@ +# Audita Consumer Feature Requests + +**Last updated:** 2026-09-23 + +**Status:** Proposed consumer requests, based on a review of Promptkit v0.10.0. +These are not approved features, implementation plans, delivery commitments, or +claims about current behavior. Priorities reflect Audita's prospective adoption; +Promptkit should evaluate them against its broader consumer needs. + +## Consumer Context + +Audita polishes transcripts through ordered correction modules. It makes LLM calls +both to propose corrections and to assess proposed changes, alongside deterministic +domain validation. It needs bounded execution, understandable failures, and useful +run diagnostics without disclosing credentials or transcript content unnecessarily. + +Audita would like Promptkit to own shared prompt and LLM execution capabilities, +while Audita retains transcript processing, module orchestration, domain acceptance, +correction application, reports, and diagnostic artifact retention. Suite-wide +consistency is the adoption goal; these requests do not ask Promptkit to reproduce +Audita's existing APIs or internal architecture. + +The [consumer guide](../consumers/pkg-promptkit.md), +[format reference](../formats.md), and +[provider integration contract](../integrations/openai-compatible-chat.md) +remain the owners of implemented behavior. This document specifies desired +consumer outcomes only; API shape and implementation are left to Promptkit. + +## 1. Transient-Failure Retries + +**Priority: High.** Audita should not need its own general-purpose provider retry +policy to recover from temporary rate limiting or service failures. + +Requested behavior: + +- Retry HTTP 429 and selected transient server failures. Suggested initial defaults + cover 500, 502, 503, and 504; retrying every 5xx status is not a requirement. +- Supply reasonable documented defaults. Two additional attempts is a proposed + starting point, with a configurable maximum and an unambiguous explicit zero + that disables retries. +- Delay retries to avoid immediate repeated requests, honor applicable provider + retry timing, and bound the operation by the caller's cancellation/deadline. + Cancellation must interrupt waiting as well as active requests. +- Define which transport failures are eligible. Document that an uncertain response + can mean the provider already performed billable work; retries do not imply + exactly-once generation. +- Keep retry attempts tied to the same prepared generation request. An automatic + retry must not silently select different prompt content, model settings, or + credential identity. +- Distinguish transient-failure retries from content-repair calls. Document how + their budgets combine so consumers can reason about maximum request counts, + latency, and potential cost. +- State whether the policy applies to built-in and injected clients. Consumers + must be able to avoid duplicate retries when another collaborator already owns + retry behavior. + +Expected consumer evidence includes a transient failure followed by success, +exhaustion, explicit retry disablement, cancellation during a delay, and interaction +with content repair. Permanent request failures must remain distinguishable. + +## 2. Execution Observations, Including Failed Runs + +**Priority: High.** Audita needs to explain the time and calls consumed by a logical +operation, including when that operation never produces a successful result. + +Requested observations include: + +- Correlation between the logical operation and its individual attempts, selected + prompt, profile, and backend. +- Attempt outcomes and enough timing information to distinguish capacity waiting, + generation, and retry delays. +- Separate counts for transient-failure retries and content-repair calls. +- Available provider-reported token usage, including completed calls preceding a + later failure, with unknown usage distinguishable from measured zero. +- Clear accounting boundaries so consumers do not double-count aggregate and + individual-attempt measurements. + +Observations should be usable by application-owned reports, logging, and metrics +without requiring a particular telemetry system or artifact layout. Default +observations should contain metadata rather than credentials or prompt/response +bodies. Any content disclosure should be an explicit consumer choice, with its +sensitivity clearly documented. + +Current result metadata and usage remain useful; the request is to make execution +and failure accounting sufficiently complete for downstream diagnostics. + +## 3. Actionable Failure Classification + +**Priority: High; closely related to retries and observations.** Audita needs to +choose between failing a run, rejecting affected work, and recording a warning +without interpreting human-readable error messages. + +Build on existing public error and validation-result contracts so consumers can +reliably distinguish cancellation, deadline expiration, transport failure, +non-success provider status, malformed provider response, exhausted retries, +capacity rejection, and completed output-validation failure. + +On retry exhaustion, retain the underlying failure classification and attempt +count. Preserve safe default diagnostics while allowing deliberate access to +provider detail under the consumer's disclosure policy. Document which outcomes +carry generated output, validation findings, and partial execution observations. + +Promptkit need not decide Audita's recovery or correction-acceptance policy. + +## 4. Aggregate and Workload Capacity Controls + +**Priority: Evaluate after the execution boundary is established.** Audita currently +needs to reason about total LLM concurrency and separate proposal/validation work, +which are not equivalent to per-backend limits. + +A useful extension would let a consumer impose an overall concurrent-generation +ceiling alongside backend limits. If the consumer need remains after redesign, +independently named workload budgets could distinguish proposal from validation +work without introducing Audita-specific concepts into Promptkit. + +The desired outcome is predictable bounded provider activity across initial calls, +retries, and repair calls, with cancellation-aware waiting and understandable +capacity rejection. Delayed retries should not unnecessarily prevent unrelated +eligible work from progressing. Consumers need clear scope and interaction rules +for all selected limits. + +This is not a request to reproduce Audita's scheduler or introduce a general task +scheduler. Determine whether existing controls and application orchestration are +sufficient before expanding the public capacity model. + +## 5. Representative Consumer Example + +**Priority: High as adoption evidence.** Provide a maintained offline example of a +structured proposal call followed by an independent LLM validation call, using +embedded prompts/schemas and separate application-owned execution profiles. + +Demonstrate transient failure handling, schema-invalid output, cancellation, and +application collection of execution observations. Use synthetic inputs and +controlled model responses without credentials or paid provider calls. + +Keep schema validity distinct from domain acceptance: a structurally valid response +may still be rejected by Audita's deterministic or semantic checks. The example +should show how an application uses Promptkit's public contracts, not duplicate +Audita's processing pipeline. + +## 6. Licensing Recommendation + +Recommend offering a future Promptkit release under LGPL 3.0, subject to the +maintainer confirming the relevant rights and dependency terms. Audita's intent is +to retain its BSD-3-Clause application licensing while adopting a shared library +whose modifications can remain subject to library copyleft. + +The requested outcome is clear, consistent license notices and consumer guidance +for the published library and its dependencies, including the obligations relevant +to statically linked Go binaries. LGPL is not a waiver of source, notice, or +recombination/relinking requirements; see [LGPLv3 section 4](https://www.gnu.org/licenses/lgpl-3.0.en.html). +This proposal does not change existing releases or their licenses. + +## Suggested Evaluation Order and Scope + +Evaluate retries, failure classification, and execution observations together, then +use the representative consumer example to assess readiness. Consider additional +capacity controls only after that exercise establishes a concrete remaining need. +Resolve licensing before selecting a release for Audita's migration. + +Existing facilities for embedded sources, application-owned profiles, prepared +execution, JSON-schema validation, and request-scoped credentials are useful +adoption foundations rather than new requests. Audita can bridge its application +configuration and credentials into Promptkit's public API; a dependency between +Promptkit and ConfigKit is not requested. + +Transcript normalization, chunking, module ordering, semantic validators, correction +application, correction-ledger assembly, and filesystem retention remain Audita's +responsibility. No requested feature requires Promptkit to own those concerns.