promptkit/docs/roadmap/output-repair.md

# Public Bounded Output Repair Roadmap

## Purpose

Make Promptkit's existing bounded output-repair capability available through
the public engine. Consumers that require nonempty text, JSON, or JSON Schema
output should be able to ask Promptkit to make a limited number of corrective
model calls after content validation fails, without rebuilding Promptkit's
generation, validation, capacity, credential, session, and result-accounting
workflow.

The feature improves basic content and structural reliability, not domain
correctness. Promptkit can determine whether output is nonempty, whether it is
syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It
cannot determine whether otherwise valid content is factual, complete, useful,
or semantically correct for a downstream application.

## Target End State

- The existing `repair_attempts` prompt field and
  `OutputContract.RepairAttempts` request field control public repair behavior.
  Zero remains the default and makes execution single-pass.
- A repair budget may be one, two, or three corrective calls. Values above
  three are invalid rather than silently clamped, providing a framework-level
  bound on accidental cost and latency.
- A positive repair budget is eligible when the effective validation mode is
  `basic`, `json`, or `json_schema` and the latest completed content validation
  has failed. `none` explicitly permits empty content, and pairing it with a
  positive repair budget is an invalid output contract.
- `Engine.Run` and `Engine.RunPrepared` provide the same repair behavior for
  the built-in OpenAI-compatible client and an injected `LLMClient`.
- Provider-native structured-output metadata remains the first line of
  defense for JSON Schema generation. Repair applies only after the resulting
  content still fails Promptkit's own validation.
- Promptkit stops at the first contract-valid response or after the
  requested bound is exhausted. It never reports an invalid response as valid.
- A completed exhausted run returns the final invalid candidate and its final
  validation diagnostics in the ordinary `RunResult`; exhaustion is not an
  operational error.
- Results report the number of corrective calls actually attempted and the
  cumulative token usage reported by the initial generation and every
  completed repair generation.
- Repair remains internal orchestration. Consumers do not need to construct or
  register a repairer, and this feature does not add a public repair strategy
  interface.

## Eligibility And Contract Semantics

Repair is opt-in for each effective output contract. The prompt definition may
declare a budget, and a request-level `OutputContract` may replace that whole
contract under the existing replacement semantics. Preparation validates and
freezes the effective budget along with the validation mode and schema plan.

Only a completed `ValidationFailed` result under `basic`, `json`, or
`json_schema` can start or continue repair. The following do not trigger
repair:

- `none` validation;
- an initially valid response;
- schema loading, decoding, registration, or compilation failures;
- an operational inability to execute validation;
- invalid requests, provider failures, transport failures, or cancellation;
  and
- capacity-admission failures.

Selecting `none` is the explicit way for a caller to allow empty output and
requires a zero repair budget. Rejecting a positive budget with `none` avoids a
contradictory configuration whose requested corrective calls could never be
eligible. Under `basic`, an empty or whitespace-only candidate fails validation
and becomes repairable. Under `json` and `json_schema`, empty content is
already invalid JSON and follows the same repair path as other JSON failures.

The requested budget must be between zero and three. Output-contract validation
rejects negative values and values above three at the same source-neutral
boundary for prompt definitions, request replacements, ordinary preparation,
and prepared execution. The effective value is never silently clamped.

`repair_attempts` counts additional model calls after the initial generation,
not total calls. `ValidationResult.RepairAttempts` reports calls actually
started. A response that validates on the first corrective call therefore
reports one repair attempt.

## Repair Request Behavior

Each corrective call extends the original rendered conversation rather than
replacing it with a standalone repair prompt. For a nonempty candidate,
Promptkit constructs the request from:

1. the complete original rendered messages in their original order;
2. one additional `assistant` message containing the latest invalid candidate;
   and
3. one additional `user` message containing Promptkit's application-neutral
   correction instruction and the latest validation diagnostics.

For an empty or whitespace-only candidate, Promptkit omits the additional
`assistant` message and appends only the corrective `user` message. This avoids
sending an empty assistant message that a compatible provider may reject and
does not invent placeholder model content. The corrective message states that
the previous response was empty.

The corrective user message always:

- identifies the effective validation mode and the current attempt number;
- supplies the current validation diagnostics as data;
- asks the model to preserve valid values and change only what is necessary to
  satisfy the effective output contract; and
- uses clear boundaries so provider-controlled diagnostics are not confused
  with Promptkit's instructions.

The mode-specific instruction reflects the contract Promptkit can actually
enforce:

- for `basic`, it asks for a nonempty response that satisfies the original
  request; and
- for `json` and `json_schema`, it asks for only corrected JSON, without
  Markdown fences or explanation.

Representing the candidate as an `assistant` message keeps model output
separate from Promptkit's corrective `user` instruction and gives the model
access to the original task and source material when a missing or invalid
field must be regenerated. Promptkit does not flatten the original messages or
embed the candidate into the corrective message.

Each attempt repairs only the latest candidate. Promptkit does not append the
entire history of earlier candidates, which would increase cost without adding
equivalent corrective value and could make the active correction ambiguous.
Every attempt starts again from the original messages and appends only the
latest nonempty candidate, when present, and latest diagnostics.

JSON Schema mode retains the prepared structured-output specification, so a
compatible client receives the same provider-native schema contract on the
initial call and every corrective call. This preventive constraint remains
the first line of defense even though Promptkit independently validates the
returned content and can repair failures from providers that ignore, partially
implement, or fail to satisfy it.

Plain `json` validation continues to permit any JSON value, including an
object, array, string, number, boolean, or `null`. Promptkit must not send a
provider-native JSON-object constraint for that mode because doing so would
silently narrow the declared validation contract. A future contract may add
provider-native object generation if consumers need that distinct behavior.

Resending the original messages may allow a provider to reuse cached prompt
prefixes, especially when the original cache-control metadata is preserved.
Promptkit does not assume, require, measure, or promise provider caching, and
usage and cost remain whatever the selected client reports or charges.

When it is not empty or whitespace-only, the latest candidate is included in
full and is never truncated or rewritten by Promptkit. The built-in client
already bounds successful provider response bodies; consumers that inject an
`LLMClient` own that client's response-size policy. Repair adds no smaller
candidate limit or silent size-based skip.

Validation diagnostics included in the corrective message are limited to
65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the
diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a
bounded notice that additional diagnostic text was omitted. This prompt-safety
limit does not alter the complete diagnostics returned in the final
`ValidationResult`.

## Empty Content And Provider Responses

Content presence belongs to validation when the provider has returned an
otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible
client therefore distinguishes these cases:

- a first choice with an explicitly present string `content` value is a
  completed generation candidate even when that string is empty or consists
  only of whitespace; and
- no choices, a missing `content` field, `null` content, or non-string content
  remains a malformed provider response and returns a generation error without
  entering content repair.

This distinction keeps the built-in and injected-client paths coherent. An
explicit empty candidate reaches `none`, `basic`, `json`, or `json_schema`
validation under the same rules regardless of which client produced it. A
malformed response envelope is not reclassified as invalid model content and
does not consume the repair budget.

The initial and corrective calls use the same:

- resolved backend identity and endpoint;
- model and effective execution settings, including presence-aware numeric and
  reasoning values;
- direct credential or environment lookup boundary;
- effective session ID;
- structured-output specification; and
- capacity manager and selected backend pool.

Repair does not reload prompt, profile, backend, input, or schema sources and
does not rerender the original prompt. It reuses the already rendered message
snapshot, including its roles, contents, and cache-control metadata. A
prepared execution uses its retained messages, validation plan, and frozen
effective target throughout repair.

## Results And Failure Behavior

After every successful corrective generation, Promptkit rebuilds the output
artifact from that candidate and validates it with the operation's existing
validation plan. The final `RawOutput`, artifact, validation result, and repair
attempt count all describe the same final candidate.

Token usage is accumulated field by field across the initial response and each
completed corrective response. Timing covers the entire operation, including
repair. Prepared execution remains one-shot, and its handle is consumed by the
single execution even when that execution performs several generation calls.

A failure while making a corrective model call retains the same public error
category and underlying identity it would have had during initial generation:

- invalid model requests match `ErrInvalidRequest`;
- provider and transport generation failures match `ErrLLMGenerate`, including
  any available `GenerationError` details from the built-in client; and
- cancellation and deadlines remain discoverable through the error chain.

An operational failure while validating a corrective response continues to
match `ErrValidation`. These failures return no partial `RunResult` or partial
usage, consistent with existing operation failure behavior. Merely exhausting
the content-repair budget is different: validation completed successfully, so
the engine returns the final failed validation result rather than an error.

## Capacity And Concurrency

The ordinary run-admission lease spans preparation completion, initial
generation, validation, all corrective calls, and every exit. Repair never
performs a second admission. Each corrective generation uses the same
capacity-wrapped model client and independently reacquires an active-generation
permit for its backend, allowing unrelated accepted work to use the pool while
the repairing run validates or waits.

Cancellation while waiting for a repair-generation permit prevents that model
call when cancellation wins the grant race. All admission and generation
permits must be released on successful repair, exhaustion, provider failure,
validation failure, and cancellation.

## Architecture And Ownership

The root facade installs Promptkit's default output repairer when assembling
the runner. It supplies the same capacity-wrapped `llm.Client` used for initial
generation, whether that client ultimately delegates to Promptkit's built-in
transport or a consumer-injected `LLMClient`.

`internal/usecase` continues to own the bounded repair state machine, attempt
accounting, final-result selection, cumulative usage, error categorization,
and coordination with prepared validation. The default repairer owns only the
construction of the augmented conversation and the corrective generation
call. `internal/validate` remains the sole authority for structural validity,
while `internal/llm` retains responsibility for distinguishing explicit empty
content from an absent or malformed completion and `internal/capacity` retains
its scheduling responsibility.

No repair-specific type needs to enter `internal/domain` beyond the existing
output-contract and validation-result state, and no public package or mutable
engine registry is introduced.

## Public And Documentation Surface

No new public method or option is required. The feature activates the existing
public fields and updates their contracts:

- `OutputContract.RepairAttempts` describes the requested additional-call
  budget, its zero-to-three range, and eligible validation modes;
- `ValidationResult.RepairAttempts` describes the attempts actually made; and
- `Run`, `RunPrepared`, and related result GoDoc describe exhaustion, failure,
  usage, cancellation, and prepared-execution behavior.

Once implemented, the framework format reference owns the exact
`repair_attempts` YAML semantics and the meaning of `none`, `basic`, `json`,
and `json_schema`. Consumer guidance should contain one concise JSON Schema
example, note that `basic` can repair empty output, and emphasize that content
presence or structural validity does not establish domain correctness. The
internal runner document owns orchestration and test ownership. The
OpenAI-compatible integration contract owns the explicit-empty versus missing
content distinction. Architecture policy needs to change only if implementation
changes its existing package boundaries.

Current-state documentation must not describe public repair as available until
the public engine actually installs it. Completed roadmap material should be
retired after the implementation and canonical documentation land.

## Verification Expectations

Testing should preserve the repository's lean ownership model and build on the
existing internal repair coverage:

- domain and source-boundary tests own accepted repair budgets and output-
  contract validation, including the upper bound and invalid `none` pairing;
- use-case tests own eligibility, bounded progression, latest-candidate
  feedback, original-message preservation, non-accumulation of earlier
  candidates, stop-on-valid behavior, exhaustion, cumulative usage, error
  categories, prepared-plan reuse, cancellation, and capacity lifetime;
- default-repairer tests own the stable behavioral requirements of its prompt
  construction, mode-appropriate correction, empty-candidate behavior,
  diagnostic-feedback bound, and role boundaries without snapshotting
  incidental prose;
- model-client tests own explicit empty and whitespace string content as
  successful candidates and retain malformed-response coverage for no choices,
  missing, null, and non-string content;
- root external-package tests prove representative public ordinary and
  prepared repair workflows, including an injected client, without repeating
  the internal state-machine matrix; and
- existing single-pass tests are revised to protect zero-budget and ineligible-
  mode behavior rather than obsolete public-engine wiring.

All tests remain deterministic, offline, parallel-safe, and independent of
real credentials or provider services. Maintainer validation continues to use
the canonical workflow in the
[development guide](../development.md#maintainer-validation).

## Acceptance Criteria

- A public-engine run with failed basic, JSON, or JSON Schema validation and a
  positive budget makes no more than the permitted number of corrective calls.
- Output contracts accept zero through three repair attempts, reject larger or
  negative values, and reject a positive budget under `none` validation.
- The first contract-valid candidate is returned and no later call is
  made.
- Exhaustion returns the final invalid candidate with final diagnostics and
  the actual attempt count.
- Zero-budget, initially successful, and `none` cases remain single-pass.
- Empty and whitespace-only candidates repair under `basic`, `json`, and
  `json_schema` when the budget is positive, while `none` returns them without
  repair.
- The built-in client sends explicit empty string content to validation but
  retains generation errors for missing or malformed completion content.
- Ordinary and prepared execution preserve the same effective target, session,
  credential, original rendered messages, structured-output, validation-plan,
  capacity, and error semantics across every attempt.
- Every corrective request contains only the original messages, latest invalid
  assistant candidate when nonempty, and latest corrective user message;
  failed-attempt history does not accumulate and empty assistant messages are
  not fabricated.
- Repair sends the complete candidate, bounds only diagnostic prompt feedback,
  and leaves final validation diagnostics intact.
- Usage is cumulative across all completed generations, and no partial result
  is returned after an operational error.
- Public GoDoc and canonical format, consumer, and internal documentation agree
  with the implemented behavior and do not claim semantic correctness.

## Non-Goals

- Domain-specific, factual, qualitative, or business-rule validation.
- A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge
  extension point.
- Provider retries for HTTP failures, transport failures, rate limits, or
  malformed successful response envelopes.
- Backoff, failover, alternate-profile selection, backend routing changes, or
  a durable task queue.
- Repair for `none` validation or content requirements beyond `basic`
  nonemptiness and the declared JSON contracts.
- A provider-native JSON-object constraint for plain `json` validation.
- Local stripping, extraction, normalization, or heuristic rewriting of model
  output before validation.
- Persisting candidate history, returning every candidate, or exposing repair
  prompts in public results.
- Depending on provider prompt caching or promising a particular reduction in
  billed or reported input tokens.
- Reopening sources, changing the selected profile, or changing execution
  settings between attempts.
- Guaranteeing a valid result when the model cannot satisfy the effective
  output contract within the configured bound.