Record Audita's requested Promptkit capabilities

docs/roadmap/audita-feature-requests.md

@@ -0,0 +1,169 @@
+# Audita Consumer Feature Requests
+
+**Last updated:** 2026-09-23
+
+**Status:** Proposed consumer requests, based on a review of Promptkit v0.10.0.
+These are not approved features, implementation plans, delivery commitments, or
+claims about current behavior. Priorities reflect Audita's prospective adoption;
+Promptkit should evaluate them against its broader consumer needs.
+
+## Consumer Context
+
+Audita polishes transcripts through ordered correction modules. It makes LLM calls
+both to propose corrections and to assess proposed changes, alongside deterministic
+domain validation. It needs bounded execution, understandable failures, and useful
+run diagnostics without disclosing credentials or transcript content unnecessarily.
+
+Audita would like Promptkit to own shared prompt and LLM execution capabilities,
+while Audita retains transcript processing, module orchestration, domain acceptance,
+correction application, reports, and diagnostic artifact retention. Suite-wide
+consistency is the adoption goal; these requests do not ask Promptkit to reproduce
+Audita's existing APIs or internal architecture.
+
+The [consumer guide](../consumers/pkg-promptkit.md),
+[format reference](../formats.md), and
+[provider integration contract](../integrations/openai-compatible-chat.md)
+remain the owners of implemented behavior. This document specifies desired
+consumer outcomes only; API shape and implementation are left to Promptkit.
+
+## 1. Transient-Failure Retries
+
+**Priority: High.** Audita should not need its own general-purpose provider retry
+policy to recover from temporary rate limiting or service failures.
+
+Requested behavior:
+
+- Retry HTTP 429 and selected transient server failures. Suggested initial defaults
+  cover 500, 502, 503, and 504; retrying every 5xx status is not a requirement.
+- Supply reasonable documented defaults. Two additional attempts is a proposed
+  starting point, with a configurable maximum and an unambiguous explicit zero
+  that disables retries.
+- Delay retries to avoid immediate repeated requests, honor applicable provider
+  retry timing, and bound the operation by the caller's cancellation/deadline.
+  Cancellation must interrupt waiting as well as active requests.
+- Define which transport failures are eligible. Document that an uncertain response
+  can mean the provider already performed billable work; retries do not imply
+  exactly-once generation.
+- Keep retry attempts tied to the same prepared generation request. An automatic
+  retry must not silently select different prompt content, model settings, or
+  credential identity.
+- Distinguish transient-failure retries from content-repair calls. Document how
+  their budgets combine so consumers can reason about maximum request counts,
+  latency, and potential cost.
+- State whether the policy applies to built-in and injected clients. Consumers
+  must be able to avoid duplicate retries when another collaborator already owns
+  retry behavior.
+
+Expected consumer evidence includes a transient failure followed by success,
+exhaustion, explicit retry disablement, cancellation during a delay, and interaction
+with content repair. Permanent request failures must remain distinguishable.
+
+## 2. Execution Observations, Including Failed Runs
+
+**Priority: High.** Audita needs to explain the time and calls consumed by a logical
+operation, including when that operation never produces a successful result.
+
+Requested observations include:
+
+- Correlation between the logical operation and its individual attempts, selected
+  prompt, profile, and backend.
+- Attempt outcomes and enough timing information to distinguish capacity waiting,
+  generation, and retry delays.
+- Separate counts for transient-failure retries and content-repair calls.
+- Available provider-reported token usage, including completed calls preceding a
+  later failure, with unknown usage distinguishable from measured zero.
+- Clear accounting boundaries so consumers do not double-count aggregate and
+  individual-attempt measurements.
+
+Observations should be usable by application-owned reports, logging, and metrics
+without requiring a particular telemetry system or artifact layout. Default
+observations should contain metadata rather than credentials or prompt/response
+bodies. Any content disclosure should be an explicit consumer choice, with its
+sensitivity clearly documented.
+
+Current result metadata and usage remain useful; the request is to make execution
+and failure accounting sufficiently complete for downstream diagnostics.
+
+## 3. Actionable Failure Classification
+
+**Priority: High; closely related to retries and observations.** Audita needs to
+choose between failing a run, rejecting affected work, and recording a warning
+without interpreting human-readable error messages.
+
+Build on existing public error and validation-result contracts so consumers can
+reliably distinguish cancellation, deadline expiration, transport failure,
+non-success provider status, malformed provider response, exhausted retries,
+capacity rejection, and completed output-validation failure.
+
+On retry exhaustion, retain the underlying failure classification and attempt
+count. Preserve safe default diagnostics while allowing deliberate access to
+provider detail under the consumer's disclosure policy. Document which outcomes
+carry generated output, validation findings, and partial execution observations.
+
+Promptkit need not decide Audita's recovery or correction-acceptance policy.
+
+## 4. Aggregate and Workload Capacity Controls
+
+**Priority: Evaluate after the execution boundary is established.** Audita currently
+needs to reason about total LLM concurrency and separate proposal/validation work,
+which are not equivalent to per-backend limits.
+
+A useful extension would let a consumer impose an overall concurrent-generation
+ceiling alongside backend limits. If the consumer need remains after redesign,
+independently named workload budgets could distinguish proposal from validation
+work without introducing Audita-specific concepts into Promptkit.
+
+The desired outcome is predictable bounded provider activity across initial calls,
+retries, and repair calls, with cancellation-aware waiting and understandable
+capacity rejection. Delayed retries should not unnecessarily prevent unrelated
+eligible work from progressing. Consumers need clear scope and interaction rules
+for all selected limits.
+
+This is not a request to reproduce Audita's scheduler or introduce a general task
+scheduler. Determine whether existing controls and application orchestration are
+sufficient before expanding the public capacity model.
+
+## 5. Representative Consumer Example
+
+**Priority: High as adoption evidence.** Provide a maintained offline example of a
+structured proposal call followed by an independent LLM validation call, using
+embedded prompts/schemas and separate application-owned execution profiles.
+
+Demonstrate transient failure handling, schema-invalid output, cancellation, and
+application collection of execution observations. Use synthetic inputs and
+controlled model responses without credentials or paid provider calls.
+
+Keep schema validity distinct from domain acceptance: a structurally valid response
+may still be rejected by Audita's deterministic or semantic checks. The example
+should show how an application uses Promptkit's public contracts, not duplicate
+Audita's processing pipeline.
+
+## 6. Licensing Recommendation
+
+Recommend offering a future Promptkit release under LGPL 3.0, subject to the
+maintainer confirming the relevant rights and dependency terms. Audita's intent is
+to retain its BSD-3-Clause application licensing while adopting a shared library
+whose modifications can remain subject to library copyleft.
+
+The requested outcome is clear, consistent license notices and consumer guidance
+for the published library and its dependencies, including the obligations relevant
+to statically linked Go binaries. LGPL is not a waiver of source, notice, or
+recombination/relinking requirements; see [LGPLv3 section 4](https://www.gnu.org/licenses/lgpl-3.0.en.html).
+This proposal does not change existing releases or their licenses.
+
+## Suggested Evaluation Order and Scope
+
+Evaluate retries, failure classification, and execution observations together, then
+use the representative consumer example to assess readiness. Consider additional
+capacity controls only after that exercise establishes a concrete remaining need.
+Resolve licensing before selecting a release for Audita's migration.
+
+Existing facilities for embedded sources, application-owned profiles, prepared
+execution, JSON-schema validation, and request-scoped credentials are useful
+adoption foundations rather than new requests. Audita can bridge its application
+configuration and credentials into Promptkit's public API; a dependency between
+Promptkit and ConfigKit is not requested.
+
+Transcript normalization, chunking, module ordering, semantic validators, correction
+application, correction-ledger assembly, and filesystem retention remain Audita's
+responsibility. No requested feature requires Promptkit to own those concerns.