Plan staged implementation of reliable execution
This commit is contained in:
479
docs/roadmap/implementation.md
Normal file
479
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,479 @@
|
||||
# Reliable Execution Implementation Plan
|
||||
|
||||
## Scope And Working Rules
|
||||
|
||||
Implement the target in [reliable-execution.md](reliable-execution.md). This plan
|
||||
is for sequential execution by a coding agent, one numbered stage per prompt.
|
||||
Stages intentionally separate transport refactoring, retry behavior, lifecycle
|
||||
instrumentation, public integration, and consumer evidence. Do not merge stages
|
||||
merely because their production changes look small.
|
||||
|
||||
Read all [repository policies](../development.md#initial-orientation), especially
|
||||
the [agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||||
[documentation](../policy/documentation.md), [testing](../policy/testing.md), and
|
||||
[release](../policy/release.md) policies. Preserve unrelated changes. Commit each
|
||||
completed, validated stage with a plain-English message and report remaining
|
||||
work. Do not push, tag, publish, rewrite history, or change dependencies. Licensing
|
||||
is complete. No new live-provider checks or release gates are needed.
|
||||
|
||||
Use focused tests at each stage; run race tests for concurrency changes. Keep
|
||||
intermediate commits building, and expose public controls only when their
|
||||
behavior works. Update owning documentation alongside implemented behavior,
|
||||
not ahead of it. Exact internal helper names may change without approval; the
|
||||
consumer semantics and explicit choices below must not drift silently.
|
||||
|
||||
## Decisions Shared By All Stages
|
||||
|
||||
### Public Controls And Prepared Execution
|
||||
|
||||
- Add `Config.TransientRetries *int` and `RunRequest.TransientRetries *int`.
|
||||
Nil selects two additional attempts at construction or inherits the engine
|
||||
value on a request. Zero disables retries. Reject negative values with
|
||||
`ErrInvalidConfig` or `ErrInvalidRequest`, respectively. Do not invent a small
|
||||
maximum; count actual attempts with uint64 and convert the nonnegative budget
|
||||
before adding the initial attempt, avoiding int overflow at MaxInt. Use
|
||||
overflow-safe duration arithmetic. Copy pointed values.
|
||||
- Validate request retry values in `Prepare`, `PrepareExecution`, and `Run`.
|
||||
`Prepare` performs no retries; `PrepareExecution` freezes the effective retry
|
||||
count privately. Do not add retry settings to profile YAML, execution targets,
|
||||
provider payloads, prepared metadata, or prompt hashes.
|
||||
- Add `RunRequest.Observer ExecutionObserver`, with
|
||||
`type ExecutionObserver func(ExecutionEvent)`. It has no error return and is
|
||||
excluded from JSON. Ordinary `Run` observes that invocation. Successful
|
||||
`PrepareExecution` retains the function in its private payload for the owning
|
||||
`RunPrepared`; `Prepare` and `PrepareExecution` do not emit execution events.
|
||||
No new `RunPrepared` parameters or parallel execution API are introduced.
|
||||
- `RunPrepared` observation starts only after successful atomic claim. Invalid,
|
||||
foreign, reused, discarded, or nil handles cannot supply an execution observer
|
||||
and emit no events. Preparation failures return their existing errors without
|
||||
execution events. Ordinary `Run` does observe pre-generation errors, including
|
||||
facade conversion errors and a nil engine if an observer was supplied.
|
||||
- Clear retained observers and private execution settings on claim completion
|
||||
or discard along with the existing private payload. Never invoke observers
|
||||
while holding the prepared-handle mutex. A callback is retained by identity;
|
||||
its closure is not cloned. Document caller synchronization responsibilities.
|
||||
- Injected `LLMClient` values are never automatically retried, even when configured
|
||||
retry values are positive. Values are still validated. A custom HTTP client
|
||||
used by the built-in implementation does receive built-in retries; explicit
|
||||
zero disables them when a transport already owns retries.
|
||||
|
||||
### Retry Semantics
|
||||
|
||||
Only HTTP 429, 500, 502, 503, and 504 permit a retry. Transport failures, timeout,
|
||||
malformed envelopes, invalid requests, and nil collaborator responses do not.
|
||||
Content repair remains separate. Each repair generation gets the same configured
|
||||
transient budget as initial generation, never a budget shared across the run.
|
||||
|
||||
For retry ordinal k starting at 1, use equal jitter in [base/2, base], where
|
||||
base is min(500 milliseconds × 2^(k-1), 8 seconds). Cap before doubling to avoid
|
||||
overflow. These are documented defaults, not public clock/backoff options.
|
||||
Internal tests can supply a clock, jitter function, and cancellable wait function
|
||||
through private constructors; do not change package globals or sleep in tests.
|
||||
|
||||
Parse `Retry-After` as unsigned decimal seconds or an HTTP date via
|
||||
`http.ParseTime`, following [RFC 9110 section 10.2.3](https://www.rfc-editor.org/rfc/rfc9110.html#section-10.2.3).
|
||||
Use the first header value; trim whitespace; negative, fractional, or malformed
|
||||
values use local backoff. Past dates supply zero additional minimum. Measure a
|
||||
valid delay from response receipt, not from the end of body decoding; wait until
|
||||
the later of that not-before time and the local backoff deadline. Never apply
|
||||
the 8-second local cap to the provider minimum. Saturate unrepresentably large
|
||||
valid delays safely rather than wrapping or treating them as permission to
|
||||
retry early. Cancellation interrupts the wait; it must not busy-loop.
|
||||
|
||||
Check the caller context before waiting, before permit acquisition, and before
|
||||
dispatch. Caller cancellation/deadline wins when observed at those boundaries.
|
||||
Do not proactively reject long provider delays; wait cancellably. A context
|
||||
without a deadline can wait a long time, which must be documented.
|
||||
|
||||
Existing HTTP-client and target generation timeouts apply separately to each
|
||||
actual attempt after capacity admission. Retry waiting and capacity waiting use
|
||||
the original caller context, not a previously canceled attempt context. No
|
||||
new overall timeout is imposed. Never reset the caller's deadline.
|
||||
|
||||
Freeze built-in request payload, URL, and resolved authentication once per
|
||||
generation sequence, before its first attempt; create a fresh HTTP request/body
|
||||
reader and timeout context for every attempt. An absent optional key must remain
|
||||
absent if the environment changes during retries. Repair is a new generation
|
||||
sequence and may resolve the environment again, consistent with existing calls.
|
||||
Do not freeze environment values at `PrepareExecution` time.
|
||||
|
||||
When R is zero, return the ordinary first failure, not retry exhaustion. When
|
||||
R is positive and all R+1 attempts return eligible statuses, return exhaustion.
|
||||
If a later failure is noneligible or the context stops the sequence, return that
|
||||
failure rather than exhaustion. Count attempts actually dispatched, not waits
|
||||
scheduled. For one run, the bound is (R+1)×(P+1), where P is the repair budget;
|
||||
document the formula but do not compute it using overflow-prone `int` products.
|
||||
|
||||
### Error Contract
|
||||
|
||||
Add public `ErrTransport`, `ErrMalformedResponse`, and `ErrRetriesExhausted`.
|
||||
Transport/malformed failures during generation also retain `ErrLLMGenerate`.
|
||||
Preserve standard context identities and original transport causes for
|
||||
`errors.Is`/`errors.As`. Do not expose new internal concrete error types as the
|
||||
consumer contract or stringify sensitive transport errors while mapping them.
|
||||
|
||||
Add immutable `RetryExhaustedError` with `Attempts() uint64`, `Unwrap() error`,
|
||||
`Is(error) bool` matching `ErrRetriesExhausted`, and redacted `Error`/`GoString`.
|
||||
Map its cause recursively so `errors.As` still finds the existing public
|
||||
`GenerationError` and status/detail accessors. Nil receivers and zero values
|
||||
must be safe, consistent with existing typed errors. No stable JSON contract.
|
||||
Do not let the current provider-error special case in `mapPublicError` discard
|
||||
the exhaustion wrapper or outer validation/generation identities.
|
||||
|
||||
Injected errors retain their identities; do not guess transport or HTTP
|
||||
classifications by text or structural resemblance. An injected client can use
|
||||
the public sentinels deliberately. A nil successful collaborator response is
|
||||
malformed generation. Completed structural rejection remains a normal result
|
||||
with failed validation; operational validation errors remain errors with no
|
||||
partial `RunResult`.
|
||||
|
||||
### Observation Values And Accounting
|
||||
|
||||
Define public values in `observations.go`, with separate internal equivalents
|
||||
and facade conversion, not aliases. They have no stable serialized contract;
|
||||
consumers translate them into application-owned records.
|
||||
|
||||
`ExecutionEvent` contains:
|
||||
|
||||
- `Kind ExecutionEventKind`: constants `ExecutionStarted`, `ExecutionAttempt`,
|
||||
`ExecutionRetryDelay`, `ExecutionFinished` (string values `started`, `attempt`,
|
||||
`retry_delay`, `finished`).
|
||||
- `RunID string`, `Time time.Time`, and `PromptID`, `PromptVersion`, `ProfileID`,
|
||||
`BackendID`, `Model` string fields. Use the same RunID as a successful result.
|
||||
Start events for ordinary runs have no resolved identities; prepared runs
|
||||
may supply their already-resolved snapshot. Enrich later events as resolution
|
||||
succeeds. Do not label requested but unresolved values as selected values.
|
||||
- `Generation uint64` (1 initial, 2 first repair), `Attempt uint64` (1-based within a
|
||||
generation), and `Dispatched bool`. Non-attempt events use zero indexes unless
|
||||
describing the retry delay after a specific failed attempt.
|
||||
- `CapacityWait`, `GenerationDuration`, `RetryDelay`, `Duration time.Duration`.
|
||||
- `Outcome ExecutionOutcome`: `success`, `invalid_output`, `error`, or empty
|
||||
when not applicable; corresponding constants use the `ExecutionOutcome` prefix.
|
||||
- `Failure ExecutionFailureKind`, `StatusCode int`, `RetriesExhausted bool`.
|
||||
Failure constants use the `ExecutionFailure` prefix; `ExecutionFailureNone`
|
||||
is the zero value empty string. Other values are `canceled`, `deadline`,
|
||||
`transport`, `provider_status`,
|
||||
`malformed_response`, `capacity`, `invalid_config`, `invalid_request`,
|
||||
`prompt_not_found`, `profile_not_found`, `prompt_load`, `profile_load`,
|
||||
`artifact_load`, `prompt_render`, `validation`, `generation`, `other`.
|
||||
Missing credentials/profile selection use invalid_request. Classify context
|
||||
identities first, then specific causes before broad operation categories.
|
||||
Exhaustion is a separate flag, not a replacement for provider_status.
|
||||
- `Usage ObservedTokenUsage` and terminal `Attempts`, `TransientRetries`,
|
||||
`RepairCalls uint64`. Nonterminal aggregate counters are zero.
|
||||
|
||||
`ObservedTokenUsage` has the existing five token field names, each an
|
||||
`ObservedTokenCount{Value int; Known bool; Complete bool}`. An attempt field is
|
||||
known/complete exactly when the provider supplied that field as a non-null
|
||||
integer. An aggregate Value sums known contributions, Known means at least one
|
||||
known contribution, and Complete means every dispatched attempt reported that
|
||||
field. For zero dispatched attempts use known complete zero totals. Preserve
|
||||
reported values without deriving totals or reconciling fields. Unknown fields
|
||||
have Value zero. Counts on non-dispatched attempts are unknown and excluded from
|
||||
usage aggregation. This distinguishes partial accounting from complete zero.
|
||||
|
||||
Add optional `GenerateResponse.UsagePresence *TokenUsagePresence` with JSON tag
|
||||
`usage_presence,omitempty`; `TokenUsagePresence` contains five bool fields with
|
||||
the corresponding snake_case JSON names. For nil from legacy injected clients,
|
||||
nonzero fields are known and zero fields are unknown: the old interface cannot
|
||||
distinguish omitted zero from measured zero. An explicit presence value permits
|
||||
known zeros and governs all five fields. Copy it on conversion. Existing result
|
||||
accounting still sums all Usage values regardless of this observation metadata;
|
||||
`TokenUsage` and existing `RunResult` JSON remain unchanged. New
|
||||
GenerateResponse JSON round-trips the additive field and omits nil. The built-in
|
||||
adapter always supplies explicit presence; absent/null wire counts are unknown.
|
||||
|
||||
Emit one attempt-completion event per acquisition attempt, including interrupted
|
||||
capacity waiting with Dispatched=false. GenerationDuration covers dispatch
|
||||
through response decoding/closure, excludes capacity/backoff/callback time.
|
||||
CapacityWait covers acquiring a permit, and is zero for the unlimited path.
|
||||
Release the active permit before invoking observers. Dispatched means invoking
|
||||
the prepared HTTP attempt or injected Generate, not a proven network packet or
|
||||
billable request; hidden HTTP redirects/transport retries are not observable.
|
||||
|
||||
Emit retry-delay completion after each actual wait, including interrupted waits,
|
||||
with elapsed RetryDelay and the failed attempt indexes. No delay event when no
|
||||
wait starts. Emit terminal totals once on each normal return, including errors,
|
||||
after admission release. Attempts counts dispatched calls; TransientRetries
|
||||
counts dispatched attempts numbered above 1; RepairCalls counts generations
|
||||
above 1 with at least one dispatched attempt. Do not count scheduled work.
|
||||
Terminal phase-duration fields sum their respective measured phases; Duration
|
||||
is wall-clock operation time
|
||||
and can additionally include preparation, validation, callbacks, and other work.
|
||||
Do not add terminal usage to attempt usage. Attempt success means generation
|
||||
succeeded, not that later output validation accepted it.
|
||||
|
||||
Synchronous callbacks are ordered per operation, may overlap across operations,
|
||||
and cannot return errors. Event values contain no mutable containers or error
|
||||
objects. Never include endpoint URLs, headers, keys, session values, bodies,
|
||||
validation diagnostics, or provider text. Metadata identifiers can still be
|
||||
sensitive. Callback panics propagate; defer cleanup before first callback after
|
||||
resource acquisition. Do not promise terminal events after panics or process
|
||||
termination and do not invoke further callbacks while unwinding a panic.
|
||||
Callbacks must return for the operation to progress; cancellation cannot forcibly
|
||||
interrupt consumer code. Warn against blocking on work requiring the same run's
|
||||
admission slot. No asynchronous delivery or durable history.
|
||||
|
||||
## Stage 1 — Expose Missing Generation Failure Categories
|
||||
|
||||
Read `errors.go`, `generation_error.go`, `llm_adapter.go`, the LLM error types,
|
||||
and public error-boundary tests. Add public transport/malformed sentinels and
|
||||
map current built-in errors while preserving cause identity and redacted text.
|
||||
Classify actual request execution failures as transport, not unrelated request
|
||||
encoding failures merely sharing the old broad ErrRequestFailed sentinel.
|
||||
Normalize nil successful injected responses into the malformed category. Do not
|
||||
add retries or observers yet. Retain existing provider-status and validation
|
||||
behavior. Add the error GoDoc and a short consumer-guide classification update.
|
||||
|
||||
Verify public tests for transport causes, caller cancellation/deadline, malformed
|
||||
framing/choices, nil response, provider detail redaction, and injected arbitrary
|
||||
errors. Reuse current fixtures; avoid duplicating the entire transport matrix.
|
||||
Run `go test ./...`. Commit when mappings and existing tests pass.
|
||||
|
||||
## Stage 2 — Preserve Usage Presence Through The Adapter
|
||||
|
||||
Read wire response decoding, domain/public response types, facade conversion,
|
||||
and JSON contract tests. Add the presence types and optional public response
|
||||
field above. Convert wire usage counts to pointer-aware decoding, retaining
|
||||
existing integer values and response validation behavior. Explicit null and
|
||||
omission are unknown; numeric zero is known. Invalid numeric types retain current
|
||||
malformed-response handling. Apply the nil-presence fallback above only to
|
||||
observation metadata, never to existing successful-result accounting.
|
||||
|
||||
Test absent usage, partial nested usage, nulls, zero, and all five reported
|
||||
fields at the decoder boundary. Test public optional-field JSON round trips and
|
||||
defensive copying once at the facade. Existing successful RunResult totals and
|
||||
JSON must be unchanged. Document injected-client presence semantics in GoDoc.
|
||||
No observation API yet. Validate affected packages and commit.
|
||||
|
||||
## Stage 3 — Separate Frozen HTTP Preparation From One Attempt
|
||||
|
||||
Refactor `internal/llm/OpenAICompatibleClient` without changing its one-shot
|
||||
`Generate` behavior. Add an internal `PrepareGeneration` operation that validates
|
||||
and encodes one request, resolves credentials once, and returns private frozen
|
||||
state with an `Execute(ctx)` method. It creates a fresh body/request/deadline
|
||||
on every Execute and closes every response before returning.
|
||||
|
||||
Return an internal attempt outcome containing response, error, HTTP status,
|
||||
usage plus presence, and parsed provider not-before timing. Preserve known usage
|
||||
from a strictly decoded successful envelope even if missing choices/content
|
||||
subsequently make it malformed; do not salvage data from invalid framing or
|
||||
non-success error bodies. Preserve current body-size limits and redaction.
|
||||
The existing Generate method delegates once and returns no partial response on
|
||||
error. After failed body reading/decoding, retain an observed attempt-context
|
||||
cancellation/deadline instead of hiding it behind malformed-response formatting.
|
||||
Timing/header data remains internal, not provider-error strings.
|
||||
|
||||
Implement Retry-After parsing here, but no sleeping or retry loop. Test frozen
|
||||
payload/authentication across repeated Execute calls, including an initially
|
||||
absent optional key; independent per-attempt timeout contexts; body closure;
|
||||
usage on malformed choices; and header parsing with an explicit receipt time.
|
||||
Use separate Execute contexts to verify a canceled attempt does not poison the
|
||||
next call. All existing wire tests must pass. Commit this refactoring separately.
|
||||
|
||||
## Stage 4 — Implement The Internal Retry And Capacity Coordinator
|
||||
|
||||
Create `internal/generation`, depending on domain, llm, and capacity. It implements
|
||||
the existing `llm.Client` consumed by the runner and repairer. For built-in
|
||||
generation it prepares once through Stage 3, then loops over capacity-controlled
|
||||
Execute calls. For injected generation it dispatches once. Constructor arguments
|
||||
explicitly select the path; do not infer retry eligibility from arbitrary errors.
|
||||
|
||||
Expose an internal Manager active-permit acquisition method returning an
|
||||
idempotent release function. Reuse pool logic; never build another queue. Keep
|
||||
the current capacity client as a delegating one-shot wrapper until engine wiring
|
||||
moves, sharing that acquisition primitive. The coordinator defers permit release
|
||||
inside each attempt; no permit survives a wait or callback. Admission remains
|
||||
runner-owned. Invalid frozen-request preparation dispatches nothing.
|
||||
|
||||
Implement the exact delay/context/count semantics above with private test seams.
|
||||
Introduce an internal exhaustion wrapper retaining the cause and attempt count.
|
||||
The coordinator accepts a resolved retry count on the internal GenerateRequest;
|
||||
it has no public API or engine behavior change yet. Repair propagation arrives
|
||||
in Stage 5. Leave observation hooks for later stages rather than adding no-op
|
||||
public callbacks now.
|
||||
|
||||
Tests own the retry status table, R=0, exhaustion, a noneligible later failure,
|
||||
context interruption, provider minimum versus jitter, overflow-safe backoff,
|
||||
frozen absent/present credentials, and independent attempt deadlines. A blocked
|
||||
backoff must allow another call to acquire the only active permit. Cancellation
|
||||
and panic must not leak permits. Run focused race tests and commit.
|
||||
|
||||
## Stage 5 — Wire Retry Configuration Through Both Execution Paths
|
||||
|
||||
Add public/internal configuration and request fields. Thread the copied effective
|
||||
retry count through ordinary resolution, private prepared payloads,
|
||||
`executePreparedRun`, `newGenerationRequest`, and `RepairRequest`. Do not add it
|
||||
to public GenerateRequest or provider settings: injected clients own their policy.
|
||||
Keep one shared resolution helper for default/override/negative validation.
|
||||
|
||||
In `internal/engine.New`, assemble the generation coordinator around the selected
|
||||
built-in or injected client and shared manager. Do not wrap it again in
|
||||
capacity.NewClient: that would hold a permit over waits or deadlock reacquisition.
|
||||
Initial generation and default repairer must receive the same coordinator.
|
||||
Remove obsolete capacity-wrapper machinery/tests only when its distinct behavior
|
||||
is covered by the coordinator and manager tests; do not retain two live schedulers.
|
||||
|
||||
Add the public exhaustion type and recursive error mapping. Update old tests
|
||||
expecting one eligible HTTP failure to set explicit zero where their purpose is
|
||||
not retry behavior, rather than making unrelated tests wait through backoff.
|
||||
Document the newly active controls, default two retries, and per-attempt timeout
|
||||
semantics now. Add public tests for request/engine precedence, copied pointers,
|
||||
prepared freezing, repair budget multiplication, exhaustion cause access, and
|
||||
injected-client single dispatch. Run full tests and focused race tests; commit.
|
||||
|
||||
## Stage 6 — Build The Internal Observation Recorder
|
||||
|
||||
Create `internal/observation` for invocation-local counters, presence-aware
|
||||
aggregation, and synchronous dispatch. Internal event/value counterparts and
|
||||
the callback function type belong in domain so domain requests/payloads can
|
||||
carry them without importing the recorder package. The recorder depends only
|
||||
on domain/standard library, not provider or root types. No global map, history
|
||||
slice, mutex held across callbacks, goroutine, or background queue is needed.
|
||||
Keep outcome classification at owning boundaries; recorder consumes normalized
|
||||
failure metadata, never raw error strings.
|
||||
|
||||
Move/reuse run-ID generation here so observation and successful results have one
|
||||
identity owner. Provide start/update/attempt/delay/finish operations and a private
|
||||
context key to pass the invocation-local recorder through the existing internal
|
||||
client interfaces. Context propagation is solely internal plumbing, not public
|
||||
control policy. A nil observer avoids event delivery, but counts needed for errors
|
||||
and existing results must still work. If run-ID creation fails, report an error
|
||||
terminal event with an empty RunID rather than invent a valid identifier.
|
||||
|
||||
Test phase sums, dispatched-only counters, partial/unknown/zero usage, no double
|
||||
counting, callback order, retained-value independence, and panic propagation.
|
||||
Do not promise event delivery across panics. There is no public observer or runner
|
||||
wiring in this stage. Update the internal inventory when the package lands.
|
||||
|
||||
## Stage 7 — Instrument Attempts And Retry Delays
|
||||
|
||||
Connect the coordinator to the internal recorder through context. Record measured
|
||||
capacity wait even on interrupted acquisition, actual dispatch and outcome, usage
|
||||
from Stage 3, and completed/canceled backoff. Release the active permit before
|
||||
callback delivery. Collect injected response usage using presence from Stage 2;
|
||||
arbitrary injected failures have unknown usage. Error response bodies are not
|
||||
mined for usage.
|
||||
|
||||
Do not add an event for request freezing failure: the terminal operation will
|
||||
report it without pretending a dispatch happened. A completed attempt callback
|
||||
that cancels context must prevent another attempt. Measure phase end before its
|
||||
callback so observer work is not reported as provider time. Feed normalized HTTP
|
||||
and context classifications without exposing sensitive strings.
|
||||
|
||||
Tests cover retry then success, exhaustion, interrupted wait, canceled capacity,
|
||||
success then later repair failure (using internal recorder plumbing), legacy
|
||||
injected usage and explicit unknown fields, and callbacks outside permits/locks.
|
||||
Use synchronization rather than elapsed-time thresholds. Run race tests; commit.
|
||||
|
||||
## Stage 8 — Add Ordinary-Run Lifecycle And Public Observation API
|
||||
|
||||
Add public observation types/conversion and RunRequest.Observer. The facade adapts
|
||||
the function and delegates bookkeeping to the internal recorder. Start ordinary
|
||||
Run observation before request conversion and nil-engine rejection; finish after
|
||||
public error mapping so the terminal classification matches consumer errors.
|
||||
Use the recorder's RunID/start in the runner instead of generating another ID.
|
||||
Without an observer preserve existing Result timing conventions; with an observer
|
||||
its Duration includes the facade boundary and can be slightly longer.
|
||||
|
||||
Enrich resolved identities at the runner's resolution boundary, before admission;
|
||||
do not copy unresolved input strings into selected fields. Generation indexes are
|
||||
passed through initial and repair paths. Ensure final totals include completed
|
||||
generation usage before validation or repair errors. Emit finished only on normal
|
||||
return, after all admission cleanup; do not use a defer that calls user callbacks
|
||||
during panic unwinding. Keep `Run` signatures and no-partial-result behavior.
|
||||
|
||||
Tests cover source/conversion failure, nil engine, admission rejection, validation
|
||||
operational failure, valid/invalid/skipped validation, success followed by repair
|
||||
failure, matching RunID, redaction, and observer panic cleanup. Verify callback
|
||||
mutation/retention cannot alter execution state. Update public GoDoc and consumer
|
||||
guidance for ordinary observations only; run root and internal race tests; commit.
|
||||
|
||||
## Stage 9 — Observe Prepared Execution Without Weakening Its Lifecycle
|
||||
|
||||
Retain the adapted observer with the effective retries in the private prepared
|
||||
payload. Claim atomically before starting observation, release the claim mutex,
|
||||
install payload cleanup, then emit started. Preparation emits nothing; successful
|
||||
claim followed by credential failure, capacity rejection, or cancellation still
|
||||
emits finished on a normal return. Invalid/reused/foreign handles emit nothing
|
||||
and must neither invoke nor discard another execution's callback.
|
||||
|
||||
Use the same recorder, identity enrichment, failure normalization, and terminal
|
||||
accounting as ordinary execution. Keep callback work outside handle and manager
|
||||
locks. Discard clears observer references without calling them. Concurrent calls
|
||||
using one handle result in one executing observation sequence, not duplicate runs.
|
||||
|
||||
Add focused prepared public tests for frozen retry choice/observer, discarded and
|
||||
foreign handles, claim races, no preparation events, later failures with usage,
|
||||
and panic cleanup. Preserve existing preparation/result timing distinctions.
|
||||
Document the prepared observer lifetime and exclusions. Run race tests; commit.
|
||||
|
||||
## Stage 10 — Verify Cross-Cutting Public Contracts
|
||||
|
||||
Review assembled behavior through the public facade using controlled HTTP
|
||||
transports and small synthetic fixtures. Add only missing integration evidence:
|
||||
|
||||
- Two profiles sharing one finite backend coordinate initial, retry, and repair
|
||||
traffic. Backoff frees the active slot but not logical admission. Different
|
||||
backend pools remain independent; unlimited/endpoint-only paths still work.
|
||||
- A sequence of transient failure, successful invalid output, repair transient
|
||||
failure, and successful valid output produces consistent counts, known partial
|
||||
usage, result usage, events, and error identities. Exercise exhaustion and
|
||||
cancellation as focused variants, not a Cartesian product.
|
||||
- Frozen request/authentication and prepared source immutability survive retries.
|
||||
Noneligible errors never replay. Injected client attempts remain opaque.
|
||||
- Safe default error/event formatting and JSON never disclose credential markers,
|
||||
request bodies, provider message markers, or retained callback internals.
|
||||
|
||||
Do not require fixed callback timing, random samples, exact error wording, or
|
||||
internal helper-call sequences. Fix interaction defects in their owning layer.
|
||||
Run `go test -race -vet=off ./...` and `go vet ./...`; commit the verified scope.
|
||||
|
||||
## Stage 11 — Add The Offline Proposal And Assessment Example
|
||||
|
||||
Add `examples/go-library/observe` with embedded prompt/schema assets, two
|
||||
application-owned profiles sharing a finite custom backend, and one engine.
|
||||
Use a controlled RoundTripper under the built-in client: return one eligible
|
||||
status before a valid structured proposal, then a structurally valid assessment
|
||||
that the application can accept/reject independently. Collect metadata events
|
||||
and print a compact structured summary, including actual attempt/retry counts
|
||||
and assessment outcome. Do not log prompt/response bodies or provider detail.
|
||||
Accept the real short default backoff in this executable demonstration; unit
|
||||
tests still use controlled waiting. Do not add public timing controls for it.
|
||||
|
||||
Keep cancellation/exhaustion/schema-invalid branch coverage in focused tests
|
||||
from earlier stages, not a large example switchboard. Add this example once to
|
||||
the existing release-check script with semantic output assertions; do not run
|
||||
a redundant second integration suite there. Update the development guide,
|
||||
testing policy's example inventory, and internal overview accordingly. Run the
|
||||
example and its relevant tests; commit.
|
||||
|
||||
## Stage 12 — Complete Documentation And Final Validation
|
||||
|
||||
Reconcile GoDoc, consumer error/observer/retry guidance, provider integration
|
||||
docs, and internal runner/LLM/capacity/generation/observation documentation with
|
||||
the implemented design. Remove claims that no retries exist or that usage can
|
||||
only be learned on success. Clearly distinguish observations from complete cost
|
||||
accounting and explain nil legacy injected usage presence.
|
||||
|
||||
Document single-attempt migration via explicit zero, multiplication with repair
|
||||
and application semantic retries, provider-delay/caller-deadline behavior,
|
||||
callback lifetime/concurrency/panics, and no-partial-result compatibility. Do not
|
||||
copy the same field reference into multiple documents. Link the new example.
|
||||
Update Audita's wishlist status to implemented where verified. Do not invent a
|
||||
release version or tag; versioned release notes and completed-roadmap retirement
|
||||
belong to the later authorized release-preparation task.
|
||||
|
||||
Run `./scripts/release-check.sh` once for the final implementation, inspect the
|
||||
diff for accidental contracts, dependencies, or artifacts, and verify no public
|
||||
signature exposes internal types. Record results and any real remaining gaps.
|
||||
Commit the final coherent documentation/integration work; report readiness for
|
||||
review without publishing or waiting for CI.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None requiring a maintainer decision remain. The approved roadmap policies and
|
||||
the explicit API, accounting, timing, and lifecycle choices above are the basis
|
||||
for this implementation. If implementation reveals a conflict with an existing
|
||||
explicit contract, report the concrete conflict rather than silently adding a
|
||||
restriction or weakening that contract.
|
||||
@@ -1,10 +1,9 @@
|
||||
# Reliable And Observable Execution
|
||||
|
||||
**Status:** Feature roadmap; not implemented. This document defines the target
|
||||
end state. A separate implementation plan will supply sequential stages and
|
||||
prompt-level steps. The four policy decisions reviewed with the maintainer are
|
||||
settled below; exact API spelling, event field layout, and backoff timing values
|
||||
remain implementation-plan details.
|
||||
**Status:** Accepted feature scope; not implemented. This document owns the
|
||||
target end state and approved policy choices. The companion
|
||||
[implementation plan](implementation.md) owns implementation steps and concrete
|
||||
API and algorithm decisions.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -48,8 +47,7 @@ Configure an engine default with a request-scoped retry-count override, using
|
||||
presence to distinguish omission from zero. Keep retry policy out of profile
|
||||
YAML initially. Per-request control supports mixed workflows without requiring
|
||||
separate engines and splitting their capacity pools. Use a modest fixed backoff
|
||||
policy; concrete timing values belong in the implementation plan, not a new
|
||||
public timing-configuration surface.
|
||||
policy without a new public timing-configuration surface.
|
||||
|
||||
- Retry delays use exponential backoff with jitter and honor valid provider
|
||||
`Retry-After` timing as a minimum delay, subject to cancellation, rather than
|
||||
@@ -193,7 +191,6 @@ Keep execution machinery under `internal/`; the root package remains a facade
|
||||
for intentional consumer contracts. Maintain one retry-policy owner and one
|
||||
capacity manager. Keep provider wire decoding in its adapter, lifecycle
|
||||
coordination in the execution layer, and application reporting downstream.
|
||||
Exact internal package arrangement belongs in the implementation plan.
|
||||
|
||||
Do not add a second run API solely for diagnostics. Preserve existing operation
|
||||
signatures and optional-observation behavior for callers that do not use it.
|
||||
|
||||
Reference in New Issue
Block a user