From 465dc1389d8be07e6f51b448d112f20b2c4e6837 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 25 Aug 2026 09:58:10 +0000 Subject: [PATCH] Document bounded output repair --- docs/consumers/pkg-promptkit.md | 23 ++++++++++++ docs/formats.md | 14 +++++--- docs/integrations/openai-compatible-chat.md | 13 ++++--- docs/internal/llm.md | 6 ++++ docs/internal/overview.md | 4 +-- docs/internal/runner.md | 40 +++++++++++++-------- docs/roadmap/implementation.md | 2 ++ 7 files changed, 76 insertions(+), 26 deletions(-) diff --git a/docs/consumers/pkg-promptkit.md b/docs/consumers/pkg-promptkit.md index 4dca6ad..c67de18 100644 --- a/docs/consumers/pkg-promptkit.md +++ b/docs/consumers/pkg-promptkit.md @@ -173,6 +173,29 @@ semantics. The [OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md) owns the built-in client's outbound HTTP behavior. +### Repair A Structured Result + +Set a small additional-call budget when a structurally invalid result can be +corrected automatically: + +```go +request.Validation = &promptkit.OutputContract{ + Format: promptkit.FormatJSON, + ValidationMode: promptkit.ValidationJSONSchema, + SchemaPath: "events.schema.json", + RepairAttempts: 1, +} +``` + +Each repair attempt is another model call, so it can increase latency and +usage; `RunResult.Usage` is cumulative and `Validation.RepairAttempts` reports +calls actually started. Exhaustion still returns the final failed validation +result. `basic` validation can also repair an empty candidate, but structural +validity is not evidence of factual or domain correctness. See the +[output-contract format reference](../formats.md#output-contract) and +[`OutputContract` GoDoc](../../types.go) for the exact budget and eligibility +rules. + ## Inputs, Profiles, And Overrides Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request diff --git a/docs/formats.md b/docs/formats.md index 570efae..56544d7 100644 --- a/docs/formats.md +++ b/docs/formats.md @@ -126,7 +126,7 @@ outbound integration determines its wire representation. | `format` | yes | `text`, `markdown`, or `json`. | | `validation_mode` | yes | `none`, `basic`, `json`, or `json_schema`. | | `schema_path` | for `json_schema` | Path to a schema in the configured schema source. | -| `repair_attempts` | no | Integer zero or greater; omitted means zero. | +| `repair_attempts` | no | Integer from zero through three; omitted means zero. A positive value requires `basic`, `json`, or `json_schema` validation. | The validation modes behave as follows: @@ -136,9 +136,15 @@ The validation modes behave as follows: - `json_schema` requires valid JSON that satisfies the selected schema. `format` controls output artifact metadata. JSON Schema mode also supplies the -schema to compatible model clients as structured-output metadata. The public -engine does not install an output repairer, so its validation is single-pass -even when a positive `repair_attempts` value is present. +schema to compatible model clients as structured-output metadata. Plain `json` +validation accepts every valid JSON value and does not request a provider-native +JSON-object constraint. + +`repair_attempts` counts additional generation calls after a failed validation. +Zero is single-pass. With a positive eligible budget, Promptkit stops at the +first valid candidate. If the budget is exhausted, it returns the final +candidate and its complete failed validation result; generation and operational +validation failures remain errors. `none` never permits repair. A request-level `OutputContract` replaces the complete prompt output contract. It does not merge individual fields. If its format is empty, Promptkit uses diff --git a/docs/integrations/openai-compatible-chat.md b/docs/integrations/openai-compatible-chat.md index 77e84cf..ee3eec6 100644 --- a/docs/integrations/openai-compatible-chat.md +++ b/docs/integrations/openai-compatible-chat.md @@ -67,7 +67,8 @@ The client conditionally includes: - non-empty `service_tier` and effective `reasoning_effort`; an explicitly disabled reasoning setting is empty and therefore omitted; and - `response_format` for JSON Schema structured output, including its name, - strict flag, and schema document. + strict flag, and schema document. Plain JSON validation does not add an + object-only response constraint. The engine resolves backend, profile, and request extra-parameter maps by whole-map replacement rather than key merging. The resulting effective map is @@ -99,10 +100,12 @@ drained. The bounded body must contain exactly one OpenAI-compatible JSON response object followed only by JSON whitespace and EOF. The client returns the first -choice's non-empty message content and maps prompt, completion, total, cached, -and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace -data, a second JSON value, absent choices, empty first-choice content, and size -overflow are malformed responses and return no partial result. +choice's explicitly present string message content, including an empty or +whitespace-only string, and maps prompt, completion, total, cached, and +cache-write token counts. Invalid or truncated JSON, trailing non-whitespace +data, a second JSON value, absent choices, missing content, `null` content, +non-string content, and size overflow are malformed responses and return no +partial result. For a non-2xx status, Promptkit recognizes one JSON document with a top-level object-valued `error` member. Its optional `message` and `type` fields must be diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 8277c88..1803ff6 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -89,6 +89,12 @@ truncation, malformed JSON, trailing data, and a second value are malformed responses with no partial result or provider content in the error. Every body is closed, and an unbounded oversized stream is not drained. +After framing succeeds, the first choice must contain an explicitly present +string `message.content`. The string is returned exactly, including empty or +whitespace-only content. Missing choices, missing or `null` content, and +non-string content are malformed responses. Output validation and correction +eligibility remain outside this package. + For a non-success response, `ProviderHTTPError` retains the HTTP status and only normalized detail from the bounded recognized envelope. It retains `ErrUnexpectedStatus` through unwrapping. The client owns response closure; diff --git a/docs/internal/overview.md b/docs/internal/overview.md index d844bdd..064da3a 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -11,7 +11,7 @@ contributor workflow and validation. | Component | Implemented responsibility | References | | --- | --- | --- | -| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, and engine-local profile-source assembly including application fallbacks. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) | +| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, engine-local profile-source assembly including application fallbacks, and bounded output-repair assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) | | `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) | | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) | | `internal/backend` | Constructs each engine's immutable registry from the maintained built-in definitions and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) | @@ -27,7 +27,7 @@ contributor workflow and validation. | `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) | | `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) | | `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) | -| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) | +| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) | The root package assembles these internal components without exposing their representations. Consumers depend only on the root facade. diff --git a/docs/internal/runner.md b/docs/internal/runner.md index 6977c09..9acdd81 100644 --- a/docs/internal/runner.md +++ b/docs/internal/runner.md @@ -23,8 +23,10 @@ built-in backend and validated consumer additions, one engine-local run admitter, and a model client wrapped by the same capacity manager. Validation plans and provider-facing schema metadata come from the validator's preparation interface. -An output repairer can be injected internally, but the ordinary runner -constructor does not enable one. +The root engine supplies one default output repairer through the explicit +runner constructor, using the same capacity-wrapped client as initial +generation. The no-repair runner constructor remains available for focused +internal callers and tests. Each invocation carries its state in request, prepared-run, and result values. The runner has no durable run or session store. @@ -125,8 +127,8 @@ admitter is an internal unlimited fallback. After successful admission, `Run` immediately defers the returned release function, performs the completion phase, makes one initial generation call, builds the named output artifact, and validates that artifact with the plan compiled during completion. Invalid -generated content remains a validation result; an inability to perform -validation is an operational error. +generated content remains a validation result; an inability to generate or +perform validation is an operational error. Validation preparation and execution honor cancellation at every Promptkit-controlled boundary and do not publish a partial plan or result. @@ -142,17 +144,25 @@ serializing preparation or validation behind the active-generation limit. The wrapped model client separately acquires a FIFO active permit only around each actual generation call. -When an internal repairer is present, a JSON or JSON Schema content failure can -trigger bounded repair attempts. Repair receives the effective execution -target, explicit numeric-presence bits, credential, backend identity, session -ID, validation errors, prior output, and structured-output specification. One -request constructor supplies those common fields to initial and repair -generation while their rendered prompts remain intentionally distinct. The -default repairer uses the same wrapped client as initial generation, so each -repair reacquires the selected backend's active permit while remaining inside -its original admission lease. Repair never performs a second bounded -admission, and repaired outputs use the operation's existing validation plan. -This capability remains internal and is not a public option. +After a failed `basic`, JSON, or JSON Schema validation with a positive frozen +budget, the installed repairer can make a bounded corrective call. Each request +starts with a fresh copy of the complete original rendered messages, includes +only the latest nonempty candidate as an assistant message, and appends one +corrective user message. Empty candidates omit that assistant message. The +correction carries validation diagnostics as JSON data bounded to 64 KiB; the +full diagnostics remain in the validation result. + +Repair receives the effective execution target, explicit numeric-presence bits, +credential, backend identity, session ID, and structured-output specification. +The same request constructor supplies those common fields to initial and repair +generation. The default repairer uses the same wrapped client as initial +generation, so each repair reacquires the selected backend's active permit +while remaining inside its original admission lease. Repair never performs a +second bounded admission, and repaired outputs use the operation's existing +validation plan. The runner stops at the first valid candidate, sums completed +generation usage, reports calls actually started, and returns the final failed +validation result on exhaustion. A repair generation failure follows the +ordinary generation-error category rather than becoming a validation error. A successful result includes the output artifact and raw output, validation state, effective session ID, prompt and rendered-prompt hashes, selected diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 72d1167..9ff948d 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -595,6 +595,8 @@ consumer guidance, integration contracts, and internal documentation agree; all canonical validation commands pass; and the working tree contains only the intended feature implementation and roadmap changes. +**Status:** Complete. + ## Open Questions None. The feature roadmap and the fixed decisions above define all behavior