Plan bounded output repair

docs/roadmap/future.md

@@ -38,28 +38,8 @@ consumers.
 
 ## Ideas
 
-### Public bounded output repair
-
-Promptkit should make its bounded output-repair capability available through
-the public engine. A consumer should be able to request a limited number of
-corrective generation attempts when JSON or JSON Schema output fails content
-validation, without having to reproduce Promptkit's generation, validation,
-capacity, and result-accounting orchestration.
-
-- Repair is validation recovery, not a general provider retry, failover, or
-  backoff policy. Transport failures, cancellation, and operational schema or
-  validation errors must retain their ordinary error behavior.
-- Repair must stop after the first valid result or the configured attempt
-  bound. Exhausting the bound should preserve the final invalid result and its
-  validation diagnostics rather than inventing success.
-- Initial generation and every repair attempt must use the same resolved
-  backend, effective execution settings and presence semantics, session,
-  credential boundary, structured-output contract, and backend-capacity
-  policy.
-- Results should report the number of repair attempts and cumulative usage for
-  every model call made by the run.
-- Ordinary and prepared execution should expose coherent behavior, including
-  cancellation, frozen prepared state, error identity, and capacity lifetime.
+No ideas are currently awaiting selection. Active feature work belongs in its
+focused roadmap rather than this catalog.
 
 ## Entry Format
 

docs/roadmap/output-repair.md

@@ -0,0 +1,361 @@
+# Public Bounded Output Repair Roadmap
+
+## Purpose
+
+Make Promptkit's existing bounded output-repair capability available through
+the public engine. Consumers that require nonempty text, JSON, or JSON Schema
+output should be able to ask Promptkit to make a limited number of corrective
+model calls after content validation fails, without rebuilding Promptkit's
+generation, validation, capacity, credential, session, and result-accounting
+workflow.
+
+The feature improves basic content and structural reliability, not domain
+correctness. Promptkit can determine whether output is nonempty, whether it is
+syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It
+cannot determine whether otherwise valid content is factual, complete, useful,
+or semantically correct for a downstream application.
+
+## Target End State
+
+- The existing `repair_attempts` prompt field and
+  `OutputContract.RepairAttempts` request field control public repair behavior.
+  Zero remains the default and makes execution single-pass.
+- A repair budget may be one, two, or three corrective calls. Values above
+  three are invalid rather than silently clamped, providing a framework-level
+  bound on accidental cost and latency.
+- A positive repair budget is eligible when the effective validation mode is
+  `basic`, `json`, or `json_schema` and the latest completed content validation
+  has failed. `none` explicitly permits empty content, and pairing it with a
+  positive repair budget is an invalid output contract.
+- `Engine.Run` and `Engine.RunPrepared` provide the same repair behavior for
+  the built-in OpenAI-compatible client and an injected `LLMClient`.
+- Provider-native structured-output metadata remains the first line of
+  defense for JSON Schema generation. Repair applies only after the resulting
+  content still fails Promptkit's own validation.
+- Promptkit stops at the first contract-valid response or after the
+  requested bound is exhausted. It never reports an invalid response as valid.
+- A completed exhausted run returns the final invalid candidate and its final
+  validation diagnostics in the ordinary `RunResult`; exhaustion is not an
+  operational error.
+- Results report the number of corrective calls actually attempted and the
+  cumulative token usage reported by the initial generation and every
+  completed repair generation.
+- Repair remains internal orchestration. Consumers do not need to construct or
+  register a repairer, and this feature does not add a public repair strategy
+  interface.
+
+## Eligibility And Contract Semantics
+
+Repair is opt-in for each effective output contract. The prompt definition may
+declare a budget, and a request-level `OutputContract` may replace that whole
+contract under the existing replacement semantics. Preparation validates and
+freezes the effective budget along with the validation mode and schema plan.
+
+Only a completed `ValidationFailed` result under `basic`, `json`, or
+`json_schema` can start or continue repair. The following do not trigger
+repair:
+
+- `none` validation;
+- an initially valid response;
+- schema loading, decoding, registration, or compilation failures;
+- an operational inability to execute validation;
+- invalid requests, provider failures, transport failures, or cancellation;
+  and
+- capacity-admission failures.
+
+Selecting `none` is the explicit way for a caller to allow empty output and
+requires a zero repair budget. Rejecting a positive budget with `none` avoids a
+contradictory configuration whose requested corrective calls could never be
+eligible. Under `basic`, an empty or whitespace-only candidate fails validation
+and becomes repairable. Under `json` and `json_schema`, empty content is
+already invalid JSON and follows the same repair path as other JSON failures.
+
+The requested budget must be between zero and three. Output-contract validation
+rejects negative values and values above three at the same source-neutral
+boundary for prompt definitions, request replacements, ordinary preparation,
+and prepared execution. The effective value is never silently clamped.
+
+`repair_attempts` counts additional model calls after the initial generation,
+not total calls. `ValidationResult.RepairAttempts` reports calls actually
+started. A response that validates on the first corrective call therefore
+reports one repair attempt.
+
+## Repair Request Behavior
+
+Each corrective call extends the original rendered conversation rather than
+replacing it with a standalone repair prompt. For a nonempty candidate,
+Promptkit constructs the request from:
+
+1. the complete original rendered messages in their original order;
+2. one additional `assistant` message containing the latest invalid candidate;
+   and
+3. one additional `user` message containing Promptkit's application-neutral
+   correction instruction and the latest validation diagnostics.
+
+For an empty or whitespace-only candidate, Promptkit omits the additional
+`assistant` message and appends only the corrective `user` message. This avoids
+sending an empty assistant message that a compatible provider may reject and
+does not invent placeholder model content. The corrective message states that
+the previous response was empty.
+
+The corrective user message always:
+
+- identifies the effective validation mode and the current attempt number;
+- supplies the current validation diagnostics as data;
+- asks the model to preserve valid values and change only what is necessary to
+  satisfy the effective output contract; and
+- uses clear boundaries so provider-controlled diagnostics are not confused
+  with Promptkit's instructions.
+
+The mode-specific instruction reflects the contract Promptkit can actually
+enforce:
+
+- for `basic`, it asks for a nonempty response that satisfies the original
+  request; and
+- for `json` and `json_schema`, it asks for only corrected JSON, without
+  Markdown fences or explanation.
+
+Representing the candidate as an `assistant` message keeps model output
+separate from Promptkit's corrective `user` instruction and gives the model
+access to the original task and source material when a missing or invalid
+field must be regenerated. Promptkit does not flatten the original messages or
+embed the candidate into the corrective message.
+
+Each attempt repairs only the latest candidate. Promptkit does not append the
+entire history of earlier candidates, which would increase cost without adding
+equivalent corrective value and could make the active correction ambiguous.
+Every attempt starts again from the original messages and appends only the
+latest nonempty candidate, when present, and latest diagnostics.
+
+JSON Schema mode retains the prepared structured-output specification, so a
+compatible client receives the same provider-native schema contract on the
+initial call and every corrective call. This preventive constraint remains
+the first line of defense even though Promptkit independently validates the
+returned content and can repair failures from providers that ignore, partially
+implement, or fail to satisfy it.
+
+Plain `json` validation continues to permit any JSON value, including an
+object, array, string, number, boolean, or `null`. Promptkit must not send a
+provider-native JSON-object constraint for that mode because doing so would
+silently narrow the declared validation contract. A future contract may add
+provider-native object generation if consumers need that distinct behavior.
+
+Resending the original messages may allow a provider to reuse cached prompt
+prefixes, especially when the original cache-control metadata is preserved.
+Promptkit does not assume, require, measure, or promise provider caching, and
+usage and cost remain whatever the selected client reports or charges.
+
+When it is not empty or whitespace-only, the latest candidate is included in
+full and is never truncated or rewritten by Promptkit. The built-in client
+already bounds successful provider response bodies; consumers that inject an
+`LLMClient` own that client's response-size policy. Repair adds no smaller
+candidate limit or silent size-based skip.
+
+Validation diagnostics included in the corrective message are limited to
+65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the
+diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a
+bounded notice that additional diagnostic text was omitted. This prompt-safety
+limit does not alter the complete diagnostics returned in the final
+`ValidationResult`.
+
+## Empty Content And Provider Responses
+
+Content presence belongs to validation when the provider has returned an
+otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible
+client therefore distinguishes these cases:
+
+- a first choice with an explicitly present string `content` value is a
+  completed generation candidate even when that string is empty or consists
+  only of whitespace; and
+- no choices, a missing `content` field, `null` content, or non-string content
+  remains a malformed provider response and returns a generation error without
+  entering content repair.
+
+This distinction keeps the built-in and injected-client paths coherent. An
+explicit empty candidate reaches `none`, `basic`, `json`, or `json_schema`
+validation under the same rules regardless of which client produced it. A
+malformed response envelope is not reclassified as invalid model content and
+does not consume the repair budget.
+
+The initial and corrective calls use the same:
+
+- resolved backend identity and endpoint;
+- model and effective execution settings, including presence-aware numeric and
+  reasoning values;
+- direct credential or environment lookup boundary;
+- effective session ID;
+- structured-output specification; and
+- capacity manager and selected backend pool.
+
+Repair does not reload prompt, profile, backend, input, or schema sources and
+does not rerender the original prompt. It reuses the already rendered message
+snapshot, including its roles, contents, and cache-control metadata. A
+prepared execution uses its retained messages, validation plan, and frozen
+effective target throughout repair.
+
+## Results And Failure Behavior
+
+After every successful corrective generation, Promptkit rebuilds the output
+artifact from that candidate and validates it with the operation's existing
+validation plan. The final `RawOutput`, artifact, validation result, and repair
+attempt count all describe the same final candidate.
+
+Token usage is accumulated field by field across the initial response and each
+completed corrective response. Timing covers the entire operation, including
+repair. Prepared execution remains one-shot, and its handle is consumed by the
+single execution even when that execution performs several generation calls.
+
+A failure while making a corrective model call retains the same public error
+category and underlying identity it would have had during initial generation:
+
+- invalid model requests match `ErrInvalidRequest`;
+- provider and transport generation failures match `ErrLLMGenerate`, including
+  any available `GenerationError` details from the built-in client; and
+- cancellation and deadlines remain discoverable through the error chain.
+
+An operational failure while validating a corrective response continues to
+match `ErrValidation`. These failures return no partial `RunResult` or partial
+usage, consistent with existing operation failure behavior. Merely exhausting
+the content-repair budget is different: validation completed successfully, so
+the engine returns the final failed validation result rather than an error.
+
+## Capacity And Concurrency
+
+The ordinary run-admission lease spans preparation completion, initial
+generation, validation, all corrective calls, and every exit. Repair never
+performs a second admission. Each corrective generation uses the same
+capacity-wrapped model client and independently reacquires an active-generation
+permit for its backend, allowing unrelated accepted work to use the pool while
+the repairing run validates or waits.
+
+Cancellation while waiting for a repair-generation permit prevents that model
+call when cancellation wins the grant race. All admission and generation
+permits must be released on successful repair, exhaustion, provider failure,
+validation failure, and cancellation.
+
+## Architecture And Ownership
+
+The root facade installs Promptkit's default output repairer when assembling
+the runner. It supplies the same capacity-wrapped `llm.Client` used for initial
+generation, whether that client ultimately delegates to Promptkit's built-in
+transport or a consumer-injected `LLMClient`.
+
+`internal/usecase` continues to own the bounded repair state machine, attempt
+accounting, final-result selection, cumulative usage, error categorization,
+and coordination with prepared validation. The default repairer owns only the
+construction of the augmented conversation and the corrective generation
+call. `internal/validate` remains the sole authority for structural validity,
+while `internal/llm` retains responsibility for distinguishing explicit empty
+content from an absent or malformed completion and `internal/capacity` retains
+its scheduling responsibility.
+
+No repair-specific type needs to enter `internal/domain` beyond the existing
+output-contract and validation-result state, and no public package or mutable
+engine registry is introduced.
+
+## Public And Documentation Surface
+
+No new public method or option is required. The feature activates the existing
+public fields and updates their contracts:
+
+- `OutputContract.RepairAttempts` describes the requested additional-call
+  budget, its zero-to-three range, and eligible validation modes;
+- `ValidationResult.RepairAttempts` describes the attempts actually made; and
+- `Run`, `RunPrepared`, and related result GoDoc describe exhaustion, failure,
+  usage, cancellation, and prepared-execution behavior.
+
+Once implemented, the framework format reference owns the exact
+`repair_attempts` YAML semantics and the meaning of `none`, `basic`, `json`,
+and `json_schema`. Consumer guidance should contain one concise JSON Schema
+example, note that `basic` can repair empty output, and emphasize that content
+presence or structural validity does not establish domain correctness. The
+internal runner document owns orchestration and test ownership. The
+OpenAI-compatible integration contract owns the explicit-empty versus missing
+content distinction. Architecture policy needs to change only if implementation
+changes its existing package boundaries.
+
+Current-state documentation must not describe public repair as available until
+the public engine actually installs it. Completed roadmap material should be
+retired after the implementation and canonical documentation land.
+
+## Verification Expectations
+
+Testing should preserve the repository's lean ownership model and build on the
+existing internal repair coverage:
+
+- domain and source-boundary tests own accepted repair budgets and output-
+  contract validation, including the upper bound and invalid `none` pairing;
+- use-case tests own eligibility, bounded progression, latest-candidate
+  feedback, original-message preservation, non-accumulation of earlier
+  candidates, stop-on-valid behavior, exhaustion, cumulative usage, error
+  categories, prepared-plan reuse, cancellation, and capacity lifetime;
+- default-repairer tests own the stable behavioral requirements of its prompt
+  construction, mode-appropriate correction, empty-candidate behavior,
+  diagnostic-feedback bound, and role boundaries without snapshotting
+  incidental prose;
+- model-client tests own explicit empty and whitespace string content as
+  successful candidates and retain malformed-response coverage for no choices,
+  missing, null, and non-string content;
+- root external-package tests prove representative public ordinary and
+  prepared repair workflows, including an injected client, without repeating
+  the internal state-machine matrix; and
+- existing single-pass tests are revised to protect zero-budget and ineligible-
+  mode behavior rather than obsolete public-engine wiring.
+
+All tests remain deterministic, offline, parallel-safe, and independent of
+real credentials or provider services. Maintainer validation continues to use
+the canonical workflow in the
+[development guide](../development.md#maintainer-validation).
+
+## Acceptance Criteria
+
+- A public-engine run with failed basic, JSON, or JSON Schema validation and a
+  positive budget makes no more than the permitted number of corrective calls.
+- Output contracts accept zero through three repair attempts, reject larger or
+  negative values, and reject a positive budget under `none` validation.
+- The first contract-valid candidate is returned and no later call is
+  made.
+- Exhaustion returns the final invalid candidate with final diagnostics and
+  the actual attempt count.
+- Zero-budget, initially successful, and `none` cases remain single-pass.
+- Empty and whitespace-only candidates repair under `basic`, `json`, and
+  `json_schema` when the budget is positive, while `none` returns them without
+  repair.
+- The built-in client sends explicit empty string content to validation but
+  retains generation errors for missing or malformed completion content.
+- Ordinary and prepared execution preserve the same effective target, session,
+  credential, original rendered messages, structured-output, validation-plan,
+  capacity, and error semantics across every attempt.
+- Every corrective request contains only the original messages, latest invalid
+  assistant candidate when nonempty, and latest corrective user message;
+  failed-attempt history does not accumulate and empty assistant messages are
+  not fabricated.
+- Repair sends the complete candidate, bounds only diagnostic prompt feedback,
+  and leaves final validation diagnostics intact.
+- Usage is cumulative across all completed generations, and no partial result
+  is returned after an operational error.
+- Public GoDoc and canonical format, consumer, and internal documentation agree
+  with the implemented behavior and do not claim semantic correctness.
+
+## Non-Goals
+
+- Domain-specific, factual, qualitative, or business-rule validation.
+- A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge
+  extension point.
+- Provider retries for HTTP failures, transport failures, rate limits, or
+  malformed successful response envelopes.
+- Backoff, failover, alternate-profile selection, backend routing changes, or
+  a durable task queue.
+- Repair for `none` validation or content requirements beyond `basic`
+  nonemptiness and the declared JSON contracts.
+- A provider-native JSON-object constraint for plain `json` validation.
+- Local stripping, extraction, normalization, or heuristic rewriting of model
+  output before validation.
+- Persisting candidate history, returning every candidate, or exposing repair
+  prompts in public results.
+- Depending on provider prompt caching or promising a particular reduction in
+  billed or reported input tokens.
+- Reopening sources, changing the selected profile, or changing execution
+  settings between attempts.
+- Guaranteeing a valid result when the model cannot satisfy the effective
+  output contract within the configured bound.