Document bounded output repair
This commit is contained in:
@@ -173,6 +173,29 @@ semantics. The
|
||||
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
||||
owns the built-in client's outbound HTTP behavior.
|
||||
|
||||
### Repair A Structured Result
|
||||
|
||||
Set a small additional-call budget when a structurally invalid result can be
|
||||
corrected automatically:
|
||||
|
||||
```go
|
||||
request.Validation = &promptkit.OutputContract{
|
||||
Format: promptkit.FormatJSON,
|
||||
ValidationMode: promptkit.ValidationJSONSchema,
|
||||
SchemaPath: "events.schema.json",
|
||||
RepairAttempts: 1,
|
||||
}
|
||||
```
|
||||
|
||||
Each repair attempt is another model call, so it can increase latency and
|
||||
usage; `RunResult.Usage` is cumulative and `Validation.RepairAttempts` reports
|
||||
calls actually started. Exhaustion still returns the final failed validation
|
||||
result. `basic` validation can also repair an empty candidate, but structural
|
||||
validity is not evidence of factual or domain correctness. See the
|
||||
[output-contract format reference](../formats.md#output-contract) and
|
||||
[`OutputContract` GoDoc](../../types.go) for the exact budget and eligibility
|
||||
rules.
|
||||
|
||||
## Inputs, Profiles, And Overrides
|
||||
|
||||
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
|
||||
|
||||
@@ -126,7 +126,7 @@ outbound integration determines its wire representation.
|
||||
| `format` | yes | `text`, `markdown`, or `json`. |
|
||||
| `validation_mode` | yes | `none`, `basic`, `json`, or `json_schema`. |
|
||||
| `schema_path` | for `json_schema` | Path to a schema in the configured schema source. |
|
||||
| `repair_attempts` | no | Integer zero or greater; omitted means zero. |
|
||||
| `repair_attempts` | no | Integer from zero through three; omitted means zero. A positive value requires `basic`, `json`, or `json_schema` validation. |
|
||||
|
||||
The validation modes behave as follows:
|
||||
|
||||
@@ -136,9 +136,15 @@ The validation modes behave as follows:
|
||||
- `json_schema` requires valid JSON that satisfies the selected schema.
|
||||
|
||||
`format` controls output artifact metadata. JSON Schema mode also supplies the
|
||||
schema to compatible model clients as structured-output metadata. The public
|
||||
engine does not install an output repairer, so its validation is single-pass
|
||||
even when a positive `repair_attempts` value is present.
|
||||
schema to compatible model clients as structured-output metadata. Plain `json`
|
||||
validation accepts every valid JSON value and does not request a provider-native
|
||||
JSON-object constraint.
|
||||
|
||||
`repair_attempts` counts additional generation calls after a failed validation.
|
||||
Zero is single-pass. With a positive eligible budget, Promptkit stops at the
|
||||
first valid candidate. If the budget is exhausted, it returns the final
|
||||
candidate and its complete failed validation result; generation and operational
|
||||
validation failures remain errors. `none` never permits repair.
|
||||
|
||||
A request-level `OutputContract` replaces the complete prompt output contract.
|
||||
It does not merge individual fields. If its format is empty, Promptkit uses
|
||||
|
||||
@@ -67,7 +67,8 @@ The client conditionally includes:
|
||||
- non-empty `service_tier` and effective `reasoning_effort`; an explicitly
|
||||
disabled reasoning setting is empty and therefore omitted; and
|
||||
- `response_format` for JSON Schema structured output, including its name,
|
||||
strict flag, and schema document.
|
||||
strict flag, and schema document. Plain JSON validation does not add an
|
||||
object-only response constraint.
|
||||
|
||||
The engine resolves backend, profile, and request extra-parameter maps by
|
||||
whole-map replacement rather than key merging. The resulting effective map is
|
||||
@@ -99,10 +100,12 @@ drained.
|
||||
|
||||
The bounded body must contain exactly one OpenAI-compatible JSON response
|
||||
object followed only by JSON whitespace and EOF. The client returns the first
|
||||
choice's non-empty message content and maps prompt, completion, total, cached,
|
||||
and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
|
||||
data, a second JSON value, absent choices, empty first-choice content, and size
|
||||
overflow are malformed responses and return no partial result.
|
||||
choice's explicitly present string message content, including an empty or
|
||||
whitespace-only string, and maps prompt, completion, total, cached, and
|
||||
cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
|
||||
data, a second JSON value, absent choices, missing content, `null` content,
|
||||
non-string content, and size overflow are malformed responses and return no
|
||||
partial result.
|
||||
|
||||
For a non-2xx status, Promptkit recognizes one JSON document with a top-level
|
||||
object-valued `error` member. Its optional `message` and `type` fields must be
|
||||
|
||||
@@ -89,6 +89,12 @@ truncation, malformed JSON, trailing data, and a second value are malformed
|
||||
responses with no partial result or provider content in the error. Every body
|
||||
is closed, and an unbounded oversized stream is not drained.
|
||||
|
||||
After framing succeeds, the first choice must contain an explicitly present
|
||||
string `message.content`. The string is returned exactly, including empty or
|
||||
whitespace-only content. Missing choices, missing or `null` content, and
|
||||
non-string content are malformed responses. Output validation and correction
|
||||
eligibility remain outside this package.
|
||||
|
||||
For a non-success response, `ProviderHTTPError` retains the HTTP status and
|
||||
only normalized detail from the bounded recognized envelope. It retains
|
||||
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;
|
||||
|
||||
@@ -11,7 +11,7 @@ contributor workflow and validation.
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, and engine-local profile-source assembly including application fallbacks. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, engine-local profile-source assembly including application fallbacks, and bounded output-repair assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||
| `internal/backend` | Constructs each engine's immutable registry from the maintained built-in definitions and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
||||
@@ -27,7 +27,7 @@ contributor workflow and validation.
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
|
||||
The root package assembles these internal components without exposing their
|
||||
representations. Consumers depend only on the root facade.
|
||||
|
||||
@@ -23,8 +23,10 @@ built-in backend and validated consumer additions, one engine-local run
|
||||
admitter, and a model client wrapped by the same capacity manager. Validation
|
||||
plans and provider-facing schema metadata come from the validator's preparation
|
||||
interface.
|
||||
An output repairer can be injected internally, but the ordinary runner
|
||||
constructor does not enable one.
|
||||
The root engine supplies one default output repairer through the explicit
|
||||
runner constructor, using the same capacity-wrapped client as initial
|
||||
generation. The no-repair runner constructor remains available for focused
|
||||
internal callers and tests.
|
||||
|
||||
Each invocation carries its state in request, prepared-run, and result values.
|
||||
The runner has no durable run or session store.
|
||||
@@ -125,8 +127,8 @@ admitter is an internal unlimited fallback. After successful admission, `Run`
|
||||
immediately defers the returned release function, performs the completion
|
||||
phase, makes one initial generation call, builds the named output artifact,
|
||||
and validates that artifact with the plan compiled during completion. Invalid
|
||||
generated content remains a validation result; an inability to perform
|
||||
validation is an operational error.
|
||||
generated content remains a validation result; an inability to generate or
|
||||
perform validation is an operational error.
|
||||
|
||||
Validation preparation and execution honor cancellation at every
|
||||
Promptkit-controlled boundary and do not publish a partial plan or result.
|
||||
@@ -142,17 +144,25 @@ serializing preparation or validation behind the active-generation limit.
|
||||
The wrapped model client separately acquires a FIFO active permit only around
|
||||
each actual generation call.
|
||||
|
||||
When an internal repairer is present, a JSON or JSON Schema content failure can
|
||||
trigger bounded repair attempts. Repair receives the effective execution
|
||||
target, explicit numeric-presence bits, credential, backend identity, session
|
||||
ID, validation errors, prior output, and structured-output specification. One
|
||||
request constructor supplies those common fields to initial and repair
|
||||
generation while their rendered prompts remain intentionally distinct. The
|
||||
default repairer uses the same wrapped client as initial generation, so each
|
||||
repair reacquires the selected backend's active permit while remaining inside
|
||||
its original admission lease. Repair never performs a second bounded
|
||||
admission, and repaired outputs use the operation's existing validation plan.
|
||||
This capability remains internal and is not a public option.
|
||||
After a failed `basic`, JSON, or JSON Schema validation with a positive frozen
|
||||
budget, the installed repairer can make a bounded corrective call. Each request
|
||||
starts with a fresh copy of the complete original rendered messages, includes
|
||||
only the latest nonempty candidate as an assistant message, and appends one
|
||||
corrective user message. Empty candidates omit that assistant message. The
|
||||
correction carries validation diagnostics as JSON data bounded to 64 KiB; the
|
||||
full diagnostics remain in the validation result.
|
||||
|
||||
Repair receives the effective execution target, explicit numeric-presence bits,
|
||||
credential, backend identity, session ID, and structured-output specification.
|
||||
The same request constructor supplies those common fields to initial and repair
|
||||
generation. The default repairer uses the same wrapped client as initial
|
||||
generation, so each repair reacquires the selected backend's active permit
|
||||
while remaining inside its original admission lease. Repair never performs a
|
||||
second bounded admission, and repaired outputs use the operation's existing
|
||||
validation plan. The runner stops at the first valid candidate, sums completed
|
||||
generation usage, reports calls actually started, and returns the final failed
|
||||
validation result on exhaustion. A repair generation failure follows the
|
||||
ordinary generation-error category rather than becoming a validation error.
|
||||
|
||||
A successful result includes the output artifact and raw output, validation
|
||||
state, effective session ID, prompt and rendered-prompt hashes, selected
|
||||
|
||||
@@ -595,6 +595,8 @@ consumer guidance, integration contracts, and internal documentation agree;
|
||||
all canonical validation commands pass; and the working tree contains only the
|
||||
intended feature implementation and roadmap changes.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and the fixed decisions above define all behavior
|
||||
|
||||
Reference in New Issue
Block a user