Reconcile reliable execution documentation
This commit is contained in:
@@ -2,16 +2,16 @@
|
||||
|
||||
**Last updated:** 2026-09-23
|
||||
|
||||
**Status:** Proposed consumer requests, based on a review of Promptkit v0.10.0.
|
||||
These are not approved features, implementation plans, delivery commitments, or
|
||||
claims about current behavior. Priorities reflect Audita's prospective adoption;
|
||||
Promptkit should evaluate them against its broader consumer needs.
|
||||
**Status:** Requests 1–6 are implemented in source but not yet released.
|
||||
These requests originated from a review of Promptkit v0.10.0. They retain
|
||||
Audita's consumer context and priorities, not a current API specification or
|
||||
a delivery commitment for Audita adoption.
|
||||
|
||||
Requests 1–5 are now incorporated into the active
|
||||
[reliable-execution roadmap](reliable-execution.md), which owns the selected
|
||||
Promptkit-native scope and remaining decisions. The requests below retain the
|
||||
consumer context, not a separate implementation specification. Request 6 is
|
||||
implemented in source as noted below.
|
||||
Requests 1–5 were resolved through the
|
||||
[reliable-execution roadmap](reliable-execution.md) and its
|
||||
[implementation plan](implementation.md). The [consumer guide](../consumers/pkg-promptkit.md)
|
||||
and public GoDoc own current behavior. No release or Audita adoption is
|
||||
claimed here.
|
||||
|
||||
## Consumer Context
|
||||
|
||||
@@ -34,6 +34,9 @@ consumer outcomes only; API shape and implementation are left to Promptkit.
|
||||
|
||||
## 1. Transient-Failure Retries
|
||||
|
||||
**Implemented in source; not yet released.** The built-in HTTP path owns the
|
||||
selected transient-status policy; injected clients keep their own policy.
|
||||
|
||||
**Priority: High.** Audita should not need its own general-purpose provider retry
|
||||
policy to recover from temporary rate limiting or service failures.
|
||||
|
||||
@@ -66,6 +69,9 @@ with content repair. Permanent request failures must remain distinguishable.
|
||||
|
||||
## 2. Execution Observations, Including Failed Runs
|
||||
|
||||
**Implemented in source; not yet released.** Optional metadata callbacks cover
|
||||
ordinary and claimed prepared execution, including normal error returns.
|
||||
|
||||
**Priority: High.** Audita needs to explain the time and calls consumed by a logical
|
||||
operation, including when that operation never produces a successful result.
|
||||
|
||||
@@ -92,6 +98,10 @@ and failure accounting sufficiently complete for downstream diagnostics.
|
||||
|
||||
## 3. Actionable Failure Classification
|
||||
|
||||
**Implemented in source; not yet released.** Public sentinels and typed errors
|
||||
preserve provider status, retry exhaustion, transport, malformed-response,
|
||||
capacity, validation, and context distinctions.
|
||||
|
||||
**Priority: High; closely related to retries and observations.** Audita needs to
|
||||
choose between failing a run, rejecting affected work, and recording a warning
|
||||
without interpreting human-readable error messages.
|
||||
@@ -110,6 +120,10 @@ Promptkit need not decide Audita's recovery or correction-acceptance policy.
|
||||
|
||||
## 4. Simple Per-Backend Capacity Limits
|
||||
|
||||
**Implemented in source; not yet released.** One engine-local backend pool
|
||||
coordinates initial, retry, and repair attempts across profiles selecting it.
|
||||
Audita must still configure finite limits for the paid backends it adopts.
|
||||
|
||||
**Priority: Adoption requirement, largely supported by existing facilities.**
|
||||
Audita needs only two protections:
|
||||
|
||||
@@ -141,6 +155,12 @@ across a long run. A monetary budget or rate-limiting system is not requested.
|
||||
|
||||
## 5. Representative Consumer Example
|
||||
|
||||
**Implemented in source; not yet released.** The
|
||||
[offline example](../../examples/go-library/observe/main.go) demonstrates the
|
||||
built-in retry, structured proposal and assessment, metadata collection, and
|
||||
separate application acceptance decision. Focused tests, not example switches,
|
||||
cover cancellation and invalid-output branches.
|
||||
|
||||
**Priority: High as adoption evidence.** Provide a maintained offline example of a
|
||||
structured proposal call followed by an independent LLM validation call, using
|
||||
embedded prompts/schemas and separate application-owned execution profiles.
|
||||
@@ -164,11 +184,10 @@ release containing this change and satisfy the applicable distribution obligatio
|
||||
|
||||
## Suggested Evaluation Order and Scope
|
||||
|
||||
Evaluate retries, failure classification, and execution observations together, then
|
||||
use the representative consumer example to assess readiness and verify shared
|
||||
per-backend limits. Existing capacity controls should suffice unless that exercise
|
||||
reveals a concrete gap. Resolve licensing before selecting a release for Audita's
|
||||
migration.
|
||||
The implemented retry, classification, observation, and per-backend capacity
|
||||
contracts can now be evaluated together with the representative example.
|
||||
Audita still owns its migration decision and must select a later published
|
||||
release before adopting this source state.
|
||||
|
||||
Existing facilities for embedded sources, application-owned profiles, prepared
|
||||
execution, JSON-schema validation, and request-scoped credentials are useful
|
||||
|
||||
@@ -472,6 +472,8 @@ example and its relevant tests; commit.
|
||||
|
||||
## Stage 12 — Complete Documentation And Final Validation
|
||||
|
||||
**Completed:** 2026-09-23.
|
||||
|
||||
Reconcile GoDoc, consumer error/observer/retry guidance, provider integration
|
||||
docs, and internal runner/LLM/capacity/generation/observation documentation with
|
||||
the implemented design. Remove claims that no retries exist or that usage can
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# Reliable And Observable Execution
|
||||
|
||||
**Status:** Accepted feature scope; not implemented. This document owns the
|
||||
target end state and approved policy choices. The companion
|
||||
[implementation plan](implementation.md) owns implementation steps and concrete
|
||||
API and algorithm decisions.
|
||||
**Status:** Implemented in source; not yet released. This document records the
|
||||
accepted target and approved policy choices. The companion
|
||||
[implementation plan](implementation.md) records implementation steps and
|
||||
concrete API and algorithm decisions. Current contracts are owned by public
|
||||
GoDoc and the linked consumer and integration documentation.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -19,20 +20,21 @@ observations, shared capacity, and consumer evidence belong together because
|
||||
they describe the same execution lifecycle. Licensing was handled separately.
|
||||
Audita motivates the work but does not define an application-specific API.
|
||||
|
||||
## Existing Foundations
|
||||
## Foundations At Planning Time
|
||||
|
||||
Promptkit already provides structural output validation and bounded repair,
|
||||
prepared execution, engine-local backend capacity, run metadata, aggregate token
|
||||
usage, and typed provider-status and capacity errors. Operational errors do not
|
||||
return partial run results. The built-in client has no transient retry loop,
|
||||
and existing integer usage fields do not distinguish missing reports from zero.
|
||||
At the time this scope was accepted, Promptkit provided structural output
|
||||
validation and bounded repair, prepared execution, engine-local backend
|
||||
capacity, run metadata, aggregate token usage, and typed provider-status and
|
||||
capacity errors. Operational errors did not return partial run results. The
|
||||
built-in path had no transient retry loop, and existing integer usage fields
|
||||
did not distinguish missing reports from zero.
|
||||
|
||||
The current contracts remain owned by public GoDoc, the
|
||||
The implemented contracts are owned by public GoDoc, the
|
||||
[consumer guide](../consumers/pkg-promptkit.md), and the
|
||||
[provider integration contract](../integrations/openai-compatible-chat.md).
|
||||
The [capacity document](../internal/capacity.md) and
|
||||
[runner document](../internal/runner.md) describe the existing implementation.
|
||||
This roadmap does not redefine those documents as already implementing new work.
|
||||
This historical baseline does not override their current-state descriptions.
|
||||
|
||||
## Target End State
|
||||
|
||||
|
||||
Reference in New Issue
Block a user