From 115fe8ba58de891df7d240d873755ca82a86af53 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 25 Aug 2026 03:33:33 +0000 Subject: [PATCH] Plan bounded output repair --- docs/roadmap/future.md | 24 +-- docs/roadmap/output-repair.md | 361 ++++++++++++++++++++++++++++++++++ 2 files changed, 363 insertions(+), 22 deletions(-) create mode 100644 docs/roadmap/output-repair.md diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 6e10ebe..911748d 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -38,28 +38,8 @@ consumers. ## Ideas -### Public bounded output repair - -Promptkit should make its bounded output-repair capability available through -the public engine. A consumer should be able to request a limited number of -corrective generation attempts when JSON or JSON Schema output fails content -validation, without having to reproduce Promptkit's generation, validation, -capacity, and result-accounting orchestration. - -- Repair is validation recovery, not a general provider retry, failover, or - backoff policy. Transport failures, cancellation, and operational schema or - validation errors must retain their ordinary error behavior. -- Repair must stop after the first valid result or the configured attempt - bound. Exhausting the bound should preserve the final invalid result and its - validation diagnostics rather than inventing success. -- Initial generation and every repair attempt must use the same resolved - backend, effective execution settings and presence semantics, session, - credential boundary, structured-output contract, and backend-capacity - policy. -- Results should report the number of repair attempts and cumulative usage for - every model call made by the run. -- Ordinary and prepared execution should expose coherent behavior, including - cancellation, frozen prepared state, error identity, and capacity lifetime. +No ideas are currently awaiting selection. Active feature work belongs in its +focused roadmap rather than this catalog. ## Entry Format diff --git a/docs/roadmap/output-repair.md b/docs/roadmap/output-repair.md new file mode 100644 index 0000000..15080ff --- /dev/null +++ b/docs/roadmap/output-repair.md @@ -0,0 +1,361 @@ +# Public Bounded Output Repair Roadmap + +## Purpose + +Make Promptkit's existing bounded output-repair capability available through +the public engine. Consumers that require nonempty text, JSON, or JSON Schema +output should be able to ask Promptkit to make a limited number of corrective +model calls after content validation fails, without rebuilding Promptkit's +generation, validation, capacity, credential, session, and result-accounting +workflow. + +The feature improves basic content and structural reliability, not domain +correctness. Promptkit can determine whether output is nonempty, whether it is +syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It +cannot determine whether otherwise valid content is factual, complete, useful, +or semantically correct for a downstream application. + +## Target End State + +- The existing `repair_attempts` prompt field and + `OutputContract.RepairAttempts` request field control public repair behavior. + Zero remains the default and makes execution single-pass. +- A repair budget may be one, two, or three corrective calls. Values above + three are invalid rather than silently clamped, providing a framework-level + bound on accidental cost and latency. +- A positive repair budget is eligible when the effective validation mode is + `basic`, `json`, or `json_schema` and the latest completed content validation + has failed. `none` explicitly permits empty content, and pairing it with a + positive repair budget is an invalid output contract. +- `Engine.Run` and `Engine.RunPrepared` provide the same repair behavior for + the built-in OpenAI-compatible client and an injected `LLMClient`. +- Provider-native structured-output metadata remains the first line of + defense for JSON Schema generation. Repair applies only after the resulting + content still fails Promptkit's own validation. +- Promptkit stops at the first contract-valid response or after the + requested bound is exhausted. It never reports an invalid response as valid. +- A completed exhausted run returns the final invalid candidate and its final + validation diagnostics in the ordinary `RunResult`; exhaustion is not an + operational error. +- Results report the number of corrective calls actually attempted and the + cumulative token usage reported by the initial generation and every + completed repair generation. +- Repair remains internal orchestration. Consumers do not need to construct or + register a repairer, and this feature does not add a public repair strategy + interface. + +## Eligibility And Contract Semantics + +Repair is opt-in for each effective output contract. The prompt definition may +declare a budget, and a request-level `OutputContract` may replace that whole +contract under the existing replacement semantics. Preparation validates and +freezes the effective budget along with the validation mode and schema plan. + +Only a completed `ValidationFailed` result under `basic`, `json`, or +`json_schema` can start or continue repair. The following do not trigger +repair: + +- `none` validation; +- an initially valid response; +- schema loading, decoding, registration, or compilation failures; +- an operational inability to execute validation; +- invalid requests, provider failures, transport failures, or cancellation; + and +- capacity-admission failures. + +Selecting `none` is the explicit way for a caller to allow empty output and +requires a zero repair budget. Rejecting a positive budget with `none` avoids a +contradictory configuration whose requested corrective calls could never be +eligible. Under `basic`, an empty or whitespace-only candidate fails validation +and becomes repairable. Under `json` and `json_schema`, empty content is +already invalid JSON and follows the same repair path as other JSON failures. + +The requested budget must be between zero and three. Output-contract validation +rejects negative values and values above three at the same source-neutral +boundary for prompt definitions, request replacements, ordinary preparation, +and prepared execution. The effective value is never silently clamped. + +`repair_attempts` counts additional model calls after the initial generation, +not total calls. `ValidationResult.RepairAttempts` reports calls actually +started. A response that validates on the first corrective call therefore +reports one repair attempt. + +## Repair Request Behavior + +Each corrective call extends the original rendered conversation rather than +replacing it with a standalone repair prompt. For a nonempty candidate, +Promptkit constructs the request from: + +1. the complete original rendered messages in their original order; +2. one additional `assistant` message containing the latest invalid candidate; + and +3. one additional `user` message containing Promptkit's application-neutral + correction instruction and the latest validation diagnostics. + +For an empty or whitespace-only candidate, Promptkit omits the additional +`assistant` message and appends only the corrective `user` message. This avoids +sending an empty assistant message that a compatible provider may reject and +does not invent placeholder model content. The corrective message states that +the previous response was empty. + +The corrective user message always: + +- identifies the effective validation mode and the current attempt number; +- supplies the current validation diagnostics as data; +- asks the model to preserve valid values and change only what is necessary to + satisfy the effective output contract; and +- uses clear boundaries so provider-controlled diagnostics are not confused + with Promptkit's instructions. + +The mode-specific instruction reflects the contract Promptkit can actually +enforce: + +- for `basic`, it asks for a nonempty response that satisfies the original + request; and +- for `json` and `json_schema`, it asks for only corrected JSON, without + Markdown fences or explanation. + +Representing the candidate as an `assistant` message keeps model output +separate from Promptkit's corrective `user` instruction and gives the model +access to the original task and source material when a missing or invalid +field must be regenerated. Promptkit does not flatten the original messages or +embed the candidate into the corrective message. + +Each attempt repairs only the latest candidate. Promptkit does not append the +entire history of earlier candidates, which would increase cost without adding +equivalent corrective value and could make the active correction ambiguous. +Every attempt starts again from the original messages and appends only the +latest nonempty candidate, when present, and latest diagnostics. + +JSON Schema mode retains the prepared structured-output specification, so a +compatible client receives the same provider-native schema contract on the +initial call and every corrective call. This preventive constraint remains +the first line of defense even though Promptkit independently validates the +returned content and can repair failures from providers that ignore, partially +implement, or fail to satisfy it. + +Plain `json` validation continues to permit any JSON value, including an +object, array, string, number, boolean, or `null`. Promptkit must not send a +provider-native JSON-object constraint for that mode because doing so would +silently narrow the declared validation contract. A future contract may add +provider-native object generation if consumers need that distinct behavior. + +Resending the original messages may allow a provider to reuse cached prompt +prefixes, especially when the original cache-control metadata is preserved. +Promptkit does not assume, require, measure, or promise provider caching, and +usage and cost remain whatever the selected client reports or charges. + +When it is not empty or whitespace-only, the latest candidate is included in +full and is never truncated or rewritten by Promptkit. The built-in client +already bounds successful provider response bodies; consumers that inject an +`LLMClient` own that client's response-size policy. Repair adds no smaller +candidate limit or silent size-based skip. + +Validation diagnostics included in the corrective message are limited to +65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the +diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a +bounded notice that additional diagnostic text was omitted. This prompt-safety +limit does not alter the complete diagnostics returned in the final +`ValidationResult`. + +## Empty Content And Provider Responses + +Content presence belongs to validation when the provider has returned an +otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible +client therefore distinguishes these cases: + +- a first choice with an explicitly present string `content` value is a + completed generation candidate even when that string is empty or consists + only of whitespace; and +- no choices, a missing `content` field, `null` content, or non-string content + remains a malformed provider response and returns a generation error without + entering content repair. + +This distinction keeps the built-in and injected-client paths coherent. An +explicit empty candidate reaches `none`, `basic`, `json`, or `json_schema` +validation under the same rules regardless of which client produced it. A +malformed response envelope is not reclassified as invalid model content and +does not consume the repair budget. + +The initial and corrective calls use the same: + +- resolved backend identity and endpoint; +- model and effective execution settings, including presence-aware numeric and + reasoning values; +- direct credential or environment lookup boundary; +- effective session ID; +- structured-output specification; and +- capacity manager and selected backend pool. + +Repair does not reload prompt, profile, backend, input, or schema sources and +does not rerender the original prompt. It reuses the already rendered message +snapshot, including its roles, contents, and cache-control metadata. A +prepared execution uses its retained messages, validation plan, and frozen +effective target throughout repair. + +## Results And Failure Behavior + +After every successful corrective generation, Promptkit rebuilds the output +artifact from that candidate and validates it with the operation's existing +validation plan. The final `RawOutput`, artifact, validation result, and repair +attempt count all describe the same final candidate. + +Token usage is accumulated field by field across the initial response and each +completed corrective response. Timing covers the entire operation, including +repair. Prepared execution remains one-shot, and its handle is consumed by the +single execution even when that execution performs several generation calls. + +A failure while making a corrective model call retains the same public error +category and underlying identity it would have had during initial generation: + +- invalid model requests match `ErrInvalidRequest`; +- provider and transport generation failures match `ErrLLMGenerate`, including + any available `GenerationError` details from the built-in client; and +- cancellation and deadlines remain discoverable through the error chain. + +An operational failure while validating a corrective response continues to +match `ErrValidation`. These failures return no partial `RunResult` or partial +usage, consistent with existing operation failure behavior. Merely exhausting +the content-repair budget is different: validation completed successfully, so +the engine returns the final failed validation result rather than an error. + +## Capacity And Concurrency + +The ordinary run-admission lease spans preparation completion, initial +generation, validation, all corrective calls, and every exit. Repair never +performs a second admission. Each corrective generation uses the same +capacity-wrapped model client and independently reacquires an active-generation +permit for its backend, allowing unrelated accepted work to use the pool while +the repairing run validates or waits. + +Cancellation while waiting for a repair-generation permit prevents that model +call when cancellation wins the grant race. All admission and generation +permits must be released on successful repair, exhaustion, provider failure, +validation failure, and cancellation. + +## Architecture And Ownership + +The root facade installs Promptkit's default output repairer when assembling +the runner. It supplies the same capacity-wrapped `llm.Client` used for initial +generation, whether that client ultimately delegates to Promptkit's built-in +transport or a consumer-injected `LLMClient`. + +`internal/usecase` continues to own the bounded repair state machine, attempt +accounting, final-result selection, cumulative usage, error categorization, +and coordination with prepared validation. The default repairer owns only the +construction of the augmented conversation and the corrective generation +call. `internal/validate` remains the sole authority for structural validity, +while `internal/llm` retains responsibility for distinguishing explicit empty +content from an absent or malformed completion and `internal/capacity` retains +its scheduling responsibility. + +No repair-specific type needs to enter `internal/domain` beyond the existing +output-contract and validation-result state, and no public package or mutable +engine registry is introduced. + +## Public And Documentation Surface + +No new public method or option is required. The feature activates the existing +public fields and updates their contracts: + +- `OutputContract.RepairAttempts` describes the requested additional-call + budget, its zero-to-three range, and eligible validation modes; +- `ValidationResult.RepairAttempts` describes the attempts actually made; and +- `Run`, `RunPrepared`, and related result GoDoc describe exhaustion, failure, + usage, cancellation, and prepared-execution behavior. + +Once implemented, the framework format reference owns the exact +`repair_attempts` YAML semantics and the meaning of `none`, `basic`, `json`, +and `json_schema`. Consumer guidance should contain one concise JSON Schema +example, note that `basic` can repair empty output, and emphasize that content +presence or structural validity does not establish domain correctness. The +internal runner document owns orchestration and test ownership. The +OpenAI-compatible integration contract owns the explicit-empty versus missing +content distinction. Architecture policy needs to change only if implementation +changes its existing package boundaries. + +Current-state documentation must not describe public repair as available until +the public engine actually installs it. Completed roadmap material should be +retired after the implementation and canonical documentation land. + +## Verification Expectations + +Testing should preserve the repository's lean ownership model and build on the +existing internal repair coverage: + +- domain and source-boundary tests own accepted repair budgets and output- + contract validation, including the upper bound and invalid `none` pairing; +- use-case tests own eligibility, bounded progression, latest-candidate + feedback, original-message preservation, non-accumulation of earlier + candidates, stop-on-valid behavior, exhaustion, cumulative usage, error + categories, prepared-plan reuse, cancellation, and capacity lifetime; +- default-repairer tests own the stable behavioral requirements of its prompt + construction, mode-appropriate correction, empty-candidate behavior, + diagnostic-feedback bound, and role boundaries without snapshotting + incidental prose; +- model-client tests own explicit empty and whitespace string content as + successful candidates and retain malformed-response coverage for no choices, + missing, null, and non-string content; +- root external-package tests prove representative public ordinary and + prepared repair workflows, including an injected client, without repeating + the internal state-machine matrix; and +- existing single-pass tests are revised to protect zero-budget and ineligible- + mode behavior rather than obsolete public-engine wiring. + +All tests remain deterministic, offline, parallel-safe, and independent of +real credentials or provider services. Maintainer validation continues to use +the canonical workflow in the +[development guide](../development.md#maintainer-validation). + +## Acceptance Criteria + +- A public-engine run with failed basic, JSON, or JSON Schema validation and a + positive budget makes no more than the permitted number of corrective calls. +- Output contracts accept zero through three repair attempts, reject larger or + negative values, and reject a positive budget under `none` validation. +- The first contract-valid candidate is returned and no later call is + made. +- Exhaustion returns the final invalid candidate with final diagnostics and + the actual attempt count. +- Zero-budget, initially successful, and `none` cases remain single-pass. +- Empty and whitespace-only candidates repair under `basic`, `json`, and + `json_schema` when the budget is positive, while `none` returns them without + repair. +- The built-in client sends explicit empty string content to validation but + retains generation errors for missing or malformed completion content. +- Ordinary and prepared execution preserve the same effective target, session, + credential, original rendered messages, structured-output, validation-plan, + capacity, and error semantics across every attempt. +- Every corrective request contains only the original messages, latest invalid + assistant candidate when nonempty, and latest corrective user message; + failed-attempt history does not accumulate and empty assistant messages are + not fabricated. +- Repair sends the complete candidate, bounds only diagnostic prompt feedback, + and leaves final validation diagnostics intact. +- Usage is cumulative across all completed generations, and no partial result + is returned after an operational error. +- Public GoDoc and canonical format, consumer, and internal documentation agree + with the implemented behavior and do not claim semantic correctness. + +## Non-Goals + +- Domain-specific, factual, qualitative, or business-rule validation. +- A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge + extension point. +- Provider retries for HTTP failures, transport failures, rate limits, or + malformed successful response envelopes. +- Backoff, failover, alternate-profile selection, backend routing changes, or + a durable task queue. +- Repair for `none` validation or content requirements beyond `basic` + nonemptiness and the declared JSON contracts. +- A provider-native JSON-object constraint for plain `json` validation. +- Local stripping, extraction, normalization, or heuristic rewriting of model + output before validation. +- Persisting candidate history, returning every candidate, or exposing repair + prompts in public results. +- Depending on provider prompt caching or promising a particular reduction in + billed or reported input tokens. +- Reopening sources, changing the selected profile, or changing execution + settings between attempts. +- Guaranteeing a valid result when the model cannot satisfy the effective + output contract within the configured bound.