promptkit/docs/roadmap/output-repair.md

Public Bounded Output Repair Roadmap

Purpose

Make Promptkit's existing bounded output-repair capability available through the public engine. Consumers that require nonempty text, JSON, or JSON Schema output should be able to ask Promptkit to make a limited number of corrective model calls after content validation fails, without rebuilding Promptkit's generation, validation, capacity, credential, session, and result-accounting workflow.

The feature improves basic content and structural reliability, not domain correctness. Promptkit can determine whether output is nonempty, whether it is syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It cannot determine whether otherwise valid content is factual, complete, useful, or semantically correct for a downstream application.

Target End State

• The existing repair_attempts prompt field and OutputContract.RepairAttempts request field control public repair behavior. Zero remains the default and makes execution single-pass.
• A repair budget may be one, two, or three corrective calls. Values above three are invalid rather than silently clamped, providing a framework-level bound on accidental cost and latency.
• A positive repair budget is eligible when the effective validation mode is basic, json, or json_schema and the latest completed content validation has failed. none explicitly permits empty content, and pairing it with a positive repair budget is an invalid output contract.
• Engine.Run and Engine.RunPrepared provide the same repair behavior for the built-in OpenAI-compatible client and an injected LLMClient.
• Provider-native structured-output metadata remains the first line of defense for JSON Schema generation. Repair applies only after the resulting content still fails Promptkit's own validation.
• Promptkit stops at the first contract-valid response or after the requested bound is exhausted. It never reports an invalid response as valid.
• A completed exhausted run returns the final invalid candidate and its final validation diagnostics in the ordinary RunResult; exhaustion is not an operational error.
• Results report the number of corrective calls actually attempted and the cumulative token usage reported by the initial generation and every completed repair generation.
• Repair remains internal orchestration. Consumers do not need to construct or register a repairer, and this feature does not add a public repair strategy interface.

Eligibility And Contract Semantics

Repair is opt-in for each effective output contract. The prompt definition may declare a budget, and a request-level OutputContract may replace that whole contract under the existing replacement semantics. Preparation validates and freezes the effective budget along with the validation mode and schema plan.

Only a completed ValidationFailed result under basic, json, or json_schema can start or continue repair. The following do not trigger repair:

• none validation;
• an initially valid response;
• schema loading, decoding, registration, or compilation failures;
• an operational inability to execute validation;
• invalid requests, provider failures, transport failures, or cancellation; and
• capacity-admission failures.

Selecting none is the explicit way for a caller to allow empty output and requires a zero repair budget. Rejecting a positive budget with none avoids a contradictory configuration whose requested corrective calls could never be eligible. Under basic, an empty or whitespace-only candidate fails validation and becomes repairable. Under json and json_schema, empty content is already invalid JSON and follows the same repair path as other JSON failures.

The requested budget must be between zero and three. Output-contract validation rejects negative values and values above three at the same source-neutral boundary for prompt definitions, request replacements, ordinary preparation, and prepared execution. The effective value is never silently clamped.

repair_attempts counts additional model calls after the initial generation, not total calls. ValidationResult.RepairAttempts reports calls actually started. A response that validates on the first corrective call therefore reports one repair attempt.

Repair Request Behavior

Each corrective call extends the original rendered conversation rather than replacing it with a standalone repair prompt. For a nonempty candidate, Promptkit constructs the request from:

1. the complete original rendered messages in their original order;
2. one additional assistant message containing the latest invalid candidate; and
3. one additional user message containing Promptkit's application-neutral correction instruction and the latest validation diagnostics.

For an empty or whitespace-only candidate, Promptkit omits the additional assistant message and appends only the corrective user message. This avoids sending an empty assistant message that a compatible provider may reject and does not invent placeholder model content. The corrective message states that the previous response was empty.

The corrective user message always:

• identifies the effective validation mode and the current attempt number;
• supplies the current validation diagnostics as data;
• asks the model to preserve valid values and change only what is necessary to satisfy the effective output contract; and
• uses clear boundaries so provider-controlled diagnostics are not confused with Promptkit's instructions.

The mode-specific instruction reflects the contract Promptkit can actually enforce:

• for basic, it asks for a nonempty response that satisfies the original request; and
• for json and json_schema, it asks for only corrected JSON, without Markdown fences or explanation.

Representing the candidate as an assistant message keeps model output separate from Promptkit's corrective user instruction and gives the model access to the original task and source material when a missing or invalid field must be regenerated. Promptkit does not flatten the original messages or embed the candidate into the corrective message.

Each attempt repairs only the latest candidate. Promptkit does not append the entire history of earlier candidates, which would increase cost without adding equivalent corrective value and could make the active correction ambiguous. Every attempt starts again from the original messages and appends only the latest nonempty candidate, when present, and latest diagnostics.

JSON Schema mode retains the prepared structured-output specification, so a compatible client receives the same provider-native schema contract on the initial call and every corrective call. This preventive constraint remains the first line of defense even though Promptkit independently validates the returned content and can repair failures from providers that ignore, partially implement, or fail to satisfy it.

Plain json validation continues to permit any JSON value, including an object, array, string, number, boolean, or null. Promptkit must not send a provider-native JSON-object constraint for that mode because doing so would silently narrow the declared validation contract. A future contract may add provider-native object generation if consumers need that distinct behavior.

Resending the original messages may allow a provider to reuse cached prompt prefixes, especially when the original cache-control metadata is preserved. Promptkit does not assume, require, measure, or promise provider caching, and usage and cost remain whatever the selected client reports or charges.

When it is not empty or whitespace-only, the latest candidate is included in full and is never truncated or rewritten by Promptkit. The built-in client already bounds successful provider response bodies; consumers that inject an LLMClient own that client's response-size policy. Repair adds no smaller candidate limit or silent size-based skip.

Validation diagnostics included in the corrective message are limited to 65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a bounded notice that additional diagnostic text was omitted. This prompt-safety limit does not alter the complete diagnostics returned in the final ValidationResult.

Empty Content And Provider Responses

Content presence belongs to validation when the provider has returned an otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible client therefore distinguishes these cases:

• a first choice with an explicitly present string content value is a completed generation candidate even when that string is empty or consists only of whitespace; and
• no choices, a missing content field, null content, or non-string content remains a malformed provider response and returns a generation error without entering content repair.

This distinction keeps the built-in and injected-client paths coherent. An explicit empty candidate reaches none, basic, json, or json_schema validation under the same rules regardless of which client produced it. A malformed response envelope is not reclassified as invalid model content and does not consume the repair budget.

The initial and corrective calls use the same:

• resolved backend identity and endpoint;
• model and effective execution settings, including presence-aware numeric and reasoning values;
• direct credential or environment lookup boundary;
• effective session ID;
• structured-output specification; and
• capacity manager and selected backend pool.

Repair does not reload prompt, profile, backend, input, or schema sources and does not rerender the original prompt. It reuses the already rendered message snapshot, including its roles, contents, and cache-control metadata. A prepared execution uses its retained messages, validation plan, and frozen effective target throughout repair.

Results And Failure Behavior

After every successful corrective generation, Promptkit rebuilds the output artifact from that candidate and validates it with the operation's existing validation plan. The final RawOutput, artifact, validation result, and repair attempt count all describe the same final candidate.

Token usage is accumulated field by field across the initial response and each completed corrective response. Timing covers the entire operation, including repair. Prepared execution remains one-shot, and its handle is consumed by the single execution even when that execution performs several generation calls.

A failure while making a corrective model call retains the same public error category and underlying identity it would have had during initial generation:

• invalid model requests match ErrInvalidRequest;
• provider and transport generation failures match ErrLLMGenerate, including any available GenerationError details from the built-in client; and
• cancellation and deadlines remain discoverable through the error chain.

An operational failure while validating a corrective response continues to match ErrValidation. These failures return no partial RunResult or partial usage, consistent with existing operation failure behavior. Merely exhausting the content-repair budget is different: validation completed successfully, so the engine returns the final failed validation result rather than an error.

Capacity And Concurrency

The ordinary run-admission lease spans preparation completion, initial generation, validation, all corrective calls, and every exit. Repair never performs a second admission. Each corrective generation uses the same capacity-wrapped model client and independently reacquires an active-generation permit for its backend, allowing unrelated accepted work to use the pool while the repairing run validates or waits.

Cancellation while waiting for a repair-generation permit prevents that model call when cancellation wins the grant race. All admission and generation permits must be released on successful repair, exhaustion, provider failure, validation failure, and cancellation.

Architecture And Ownership

The root facade installs Promptkit's default output repairer when assembling the runner. It supplies the same capacity-wrapped llm.Client used for initial generation, whether that client ultimately delegates to Promptkit's built-in transport or a consumer-injected LLMClient.

internal/usecase continues to own the bounded repair state machine, attempt accounting, final-result selection, cumulative usage, error categorization, and coordination with prepared validation. The default repairer owns only the construction of the augmented conversation and the corrective generation call. internal/validate remains the sole authority for structural validity, while internal/llm retains responsibility for distinguishing explicit empty content from an absent or malformed completion and internal/capacity retains its scheduling responsibility.

No repair-specific type needs to enter internal/domain beyond the existing output-contract and validation-result state, and no public package or mutable engine registry is introduced.

Public And Documentation Surface

No new public method or option is required. The feature activates the existing public fields and updates their contracts:

• OutputContract.RepairAttempts describes the requested additional-call budget, its zero-to-three range, and eligible validation modes;
• ValidationResult.RepairAttempts describes the attempts actually made; and
• Run, RunPrepared, and related result GoDoc describe exhaustion, failure, usage, cancellation, and prepared-execution behavior.

Once implemented, the framework format reference owns the exact repair_attempts YAML semantics and the meaning of none, basic, json, and json_schema. Consumer guidance should contain one concise JSON Schema example, note that basic can repair empty output, and emphasize that content presence or structural validity does not establish domain correctness. The internal runner document owns orchestration and test ownership. The OpenAI-compatible integration contract owns the explicit-empty versus missing content distinction. Architecture policy needs to change only if implementation changes its existing package boundaries.

Current-state documentation must not describe public repair as available until the public engine actually installs it. Completed roadmap material should be retired after the implementation and canonical documentation land.

Verification Expectations

Testing should preserve the repository's lean ownership model and build on the existing internal repair coverage:

• domain and source-boundary tests own accepted repair budgets and output- contract validation, including the upper bound and invalid none pairing;
• use-case tests own eligibility, bounded progression, latest-candidate feedback, original-message preservation, non-accumulation of earlier candidates, stop-on-valid behavior, exhaustion, cumulative usage, error categories, prepared-plan reuse, cancellation, and capacity lifetime;
• default-repairer tests own the stable behavioral requirements of its prompt construction, mode-appropriate correction, empty-candidate behavior, diagnostic-feedback bound, and role boundaries without snapshotting incidental prose;
• model-client tests own explicit empty and whitespace string content as successful candidates and retain malformed-response coverage for no choices, missing, null, and non-string content;
• root external-package tests prove representative public ordinary and prepared repair workflows, including an injected client, without repeating the internal state-machine matrix; and
• existing single-pass tests are revised to protect zero-budget and ineligible- mode behavior rather than obsolete public-engine wiring.

All tests remain deterministic, offline, parallel-safe, and independent of real credentials or provider services. Maintainer validation continues to use the canonical workflow in the development guide.

Acceptance Criteria

• A public-engine run with failed basic, JSON, or JSON Schema validation and a positive budget makes no more than the permitted number of corrective calls.
• Output contracts accept zero through three repair attempts, reject larger or negative values, and reject a positive budget under none validation.
• The first contract-valid candidate is returned and no later call is made.
• Exhaustion returns the final invalid candidate with final diagnostics and the actual attempt count.
• Zero-budget, initially successful, and none cases remain single-pass.
• Empty and whitespace-only candidates repair under basic, json, and json_schema when the budget is positive, while none returns them without repair.
• The built-in client sends explicit empty string content to validation but retains generation errors for missing or malformed completion content.
• Ordinary and prepared execution preserve the same effective target, session, credential, original rendered messages, structured-output, validation-plan, capacity, and error semantics across every attempt.
• Every corrective request contains only the original messages, latest invalid assistant candidate when nonempty, and latest corrective user message; failed-attempt history does not accumulate and empty assistant messages are not fabricated.
• Repair sends the complete candidate, bounds only diagnostic prompt feedback, and leaves final validation diagnostics intact.
• Usage is cumulative across all completed generations, and no partial result is returned after an operational error.
• Public GoDoc and canonical format, consumer, and internal documentation agree with the implemented behavior and do not claim semantic correctness.

Non-Goals

• Domain-specific, factual, qualitative, or business-rule validation.
• A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge extension point.
• Provider retries for HTTP failures, transport failures, rate limits, or malformed successful response envelopes.
• Backoff, failover, alternate-profile selection, backend routing changes, or a durable task queue.
• Repair for none validation or content requirements beyond basic nonemptiness and the declared JSON contracts.
• A provider-native JSON-object constraint for plain json validation.
• Local stripping, extraction, normalization, or heuristic rewriting of model output before validation.
• Persisting candidate history, returning every candidate, or exposing repair prompts in public results.
• Depending on provider prompt caching or promising a particular reduction in billed or reported input tokens.
• Reopening sources, changing the selected profile, or changing execution settings between attempts.
• Guaranteeing a valid result when the model cannot satisfy the effective output contract within the configured bound.