10 Commits

29 changed files with 1193 additions and 157 deletions

View File

@@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve.
## Release Guidance
Consumers upgrading from `v0.6.0` to `v0.7.0` should read the
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
Consumers upgrading from `v0.7.0` to `v0.8.0` should read the
[v0.8.0 changelog and migration guide](docs/releases/v0.8.0.md).
Earlier adopters can consult the
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
Consumers upgrading from `v0.5.0` to `v0.6.0` can consult the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the

View File

@@ -173,6 +173,29 @@ semantics. The
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
owns the built-in client's outbound HTTP behavior.
### Repair A Structured Result
Set a small additional-call budget when a structurally invalid result can be
corrected automatically:
```go
request.Validation = &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSONSchema,
SchemaPath: "events.schema.json",
RepairAttempts: 1,
}
```
Each repair attempt is another model call, so it can increase latency and
usage; `RunResult.Usage` is cumulative and `Validation.RepairAttempts` reports
calls actually started. Exhaustion still returns the final failed validation
result. `basic` validation can also repair an empty candidate, but structural
validity is not evidence of factual or domain correctness. See the
[output-contract format reference](../formats.md#output-contract) and
[`OutputContract` GoDoc](../../types.go) for the exact budget and eligibility
rules.
## Inputs, Profiles, And Overrides
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request

View File

@@ -126,7 +126,7 @@ outbound integration determines its wire representation.
| `format` | yes | `text`, `markdown`, or `json`. |
| `validation_mode` | yes | `none`, `basic`, `json`, or `json_schema`. |
| `schema_path` | for `json_schema` | Path to a schema in the configured schema source. |
| `repair_attempts` | no | Integer zero or greater; omitted means zero. |
| `repair_attempts` | no | Integer from zero through three; omitted means zero. A positive value requires `basic`, `json`, or `json_schema` validation. |
The validation modes behave as follows:
@@ -136,9 +136,15 @@ The validation modes behave as follows:
- `json_schema` requires valid JSON that satisfies the selected schema.
`format` controls output artifact metadata. JSON Schema mode also supplies the
schema to compatible model clients as structured-output metadata. The public
engine does not install an output repairer, so its validation is single-pass
even when a positive `repair_attempts` value is present.
schema to compatible model clients as structured-output metadata. Plain `json`
validation accepts every valid JSON value and does not request a provider-native
JSON-object constraint.
`repair_attempts` counts additional generation calls after a failed validation.
Zero is single-pass. With a positive eligible budget, Promptkit stops at the
first valid candidate. If the budget is exhausted, it returns the final
candidate and its complete failed validation result; generation and operational
validation failures remain errors. `none` never permits repair.
A request-level `OutputContract` replaces the complete prompt output contract.
It does not merge individual fields. If its format is empty, Promptkit uses

View File

@@ -67,7 +67,8 @@ The client conditionally includes:
- non-empty `service_tier` and effective `reasoning_effort`; an explicitly
disabled reasoning setting is empty and therefore omitted; and
- `response_format` for JSON Schema structured output, including its name,
strict flag, and schema document.
strict flag, and schema document. Plain JSON validation does not add an
object-only response constraint.
The engine resolves backend, profile, and request extra-parameter maps by
whole-map replacement rather than key merging. The resulting effective map is
@@ -99,10 +100,12 @@ drained.
The bounded body must contain exactly one OpenAI-compatible JSON response
object followed only by JSON whitespace and EOF. The client returns the first
choice's non-empty message content and maps prompt, completion, total, cached,
and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
data, a second JSON value, absent choices, empty first-choice content, and size
overflow are malformed responses and return no partial result.
choice's explicitly present string message content, including an empty or
whitespace-only string, and maps prompt, completion, total, cached, and
cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
data, a second JSON value, absent choices, missing content, `null` content,
non-string content, and size overflow are malformed responses and return no
partial result.
For a non-2xx status, Promptkit recognizes one JSON document with a top-level
object-valued `error` member. Its optional `message` and `type` fields must be

View File

@@ -89,6 +89,12 @@ truncation, malformed JSON, trailing data, and a second value are malformed
responses with no partial result or provider content in the error. Every body
is closed, and an unbounded oversized stream is not drained.
After framing succeeds, the first choice must contain an explicitly present
string `message.content`. The string is returned exactly, including empty or
whitespace-only content. Missing choices, missing or `null` content, and
non-string content are malformed responses. Output validation and correction
eligibility remain outside this package.
For a non-success response, `ProviderHTTPError` retains the HTTP status and
only normalized detail from the bounded recognized envelope. It retains
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;

View File

@@ -11,7 +11,7 @@ contributor workflow and validation.
| Component | Implemented responsibility | References |
| --- | --- | --- |
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, and engine-local profile-source assembly including application fallbacks. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, engine-local profile-source assembly including application fallbacks, and bounded output-repair assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
| `internal/backend` | Constructs each engine's immutable registry from the maintained built-in definitions and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
@@ -27,7 +27,7 @@ contributor workflow and validation.
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
The root package assembles these internal components without exposing their
representations. Consumers depend only on the root facade.

View File

@@ -23,8 +23,10 @@ built-in backend and validated consumer additions, one engine-local run
admitter, and a model client wrapped by the same capacity manager. Validation
plans and provider-facing schema metadata come from the validator's preparation
interface.
An output repairer can be injected internally, but the ordinary runner
constructor does not enable one.
The root engine supplies one default output repairer through the explicit
runner constructor, using the same capacity-wrapped client as initial
generation. The no-repair runner constructor remains available for focused
internal callers and tests.
Each invocation carries its state in request, prepared-run, and result values.
The runner has no durable run or session store.
@@ -125,8 +127,8 @@ admitter is an internal unlimited fallback. After successful admission, `Run`
immediately defers the returned release function, performs the completion
phase, makes one initial generation call, builds the named output artifact,
and validates that artifact with the plan compiled during completion. Invalid
generated content remains a validation result; an inability to perform
validation is an operational error.
generated content remains a validation result; an inability to generate or
perform validation is an operational error.
Validation preparation and execution honor cancellation at every
Promptkit-controlled boundary and do not publish a partial plan or result.
@@ -142,17 +144,25 @@ serializing preparation or validation behind the active-generation limit.
The wrapped model client separately acquires a FIFO active permit only around
each actual generation call.
When an internal repairer is present, a JSON or JSON Schema content failure can
trigger bounded repair attempts. Repair receives the effective execution
target, explicit numeric-presence bits, credential, backend identity, session
ID, validation errors, prior output, and structured-output specification. One
request constructor supplies those common fields to initial and repair
generation while their rendered prompts remain intentionally distinct. The
default repairer uses the same wrapped client as initial generation, so each
repair reacquires the selected backend's active permit while remaining inside
its original admission lease. Repair never performs a second bounded
admission, and repaired outputs use the operation's existing validation plan.
This capability remains internal and is not a public option.
After a failed `basic`, JSON, or JSON Schema validation with a positive frozen
budget, the installed repairer can make a bounded corrective call. Each request
starts with a fresh copy of the complete original rendered messages, includes
only the latest nonempty candidate as an assistant message, and appends one
corrective user message. Empty candidates omit that assistant message. The
correction carries validation diagnostics as JSON data bounded to 64 KiB; the
full diagnostics remain in the validation result.
Repair receives the effective execution target, explicit numeric-presence bits,
credential, backend identity, session ID, and structured-output specification.
The same request constructor supplies those common fields to initial and repair
generation. The default repairer uses the same wrapped client as initial
generation, so each repair reacquires the selected backend's active permit
while remaining inside its original admission lease. Repair never performs a
second bounded admission, and repaired outputs use the operation's existing
validation plan. The runner stops at the first valid candidate, sums completed
generation usage, reports calls actually started, and returns the final failed
validation result on exhaustion. A repair generation failure follows the
ordinary generation-error category rather than becoming a validation error.
A successful result includes the output artifact and raw output, validation
state, effective session ID, prompt and rendered-prompt hashes, selected

109
docs/releases/v0.8.0.md Normal file
View File

@@ -0,0 +1,109 @@
# Promptkit v0.8.0
This supplemental changelog and migration guide summarizes the consumer-facing
changes from `v0.7.0` to `v0.8.0`. The annotated `v0.8.0` tag is the
authoritative release record. Exact current contracts belong to the linked
GoDoc and durable documentation.
## Summary
`v0.8.0` activates Promptkit's bounded output-repair workflow:
- failed nonempty-text, JSON, and JSON Schema validation can make a limited
number of corrective model calls;
- corrective calls preserve the original rendered conversation, effective
target, session, structured-output contract, and backend capacity policy;
- results report cumulative usage and the number of corrective calls actually
made; and
- explicitly empty OpenAI-compatible response content now reaches output
validation instead of being classified as a malformed provider envelope.
## Compatibility
This release adds no public declarations or fields and removes none. Existing
source code remains source-compatible.
The behavior of the existing `OutputContract.RepairAttempts` field and prompt
YAML `repair_attempts` field has changed. A positive value now authorizes real
additional model calls after eligible validation failures; earlier releases
accepted the field but the public engine remained single-pass. Consumers that
set a positive value should expect additional latency, token usage, and
provider cost when repair is needed.
Repair budgets must now be between zero and three. A positive budget requires
`basic`, `json`, or `json_schema` validation. Values above three and a positive
budget paired with `none` are invalid contracts rather than ignored settings.
An explicitly present empty or whitespace-only string returned by the built-in
OpenAI-compatible client is now a completed generation candidate. `none`
validation permits it, while `basic`, `json`, and `json_schema` classify it
under their ordinary validation rules and may repair it when configured.
Missing, `null`, or non-string content remains a malformed provider response.
## Upgrade
Update the module dependency with:
```sh
go get gitea.maximumdirect.net/eric/promptkit@v0.8.0
go mod tidy
```
Review every prompt definition and request override that sets a positive repair
budget. Use zero or omit the field to retain single-pass execution. Ensure each
positive budget is no greater than three and uses an eligible validation mode,
then run the consuming project's ordinary and race-enabled tests.
## Bounded Output Repair
`repair_attempts` counts corrective calls in addition to the initial model
call. Promptkit validates each completed candidate, stops at the first valid
one, and never exceeds the configured bound. If every candidate remains
invalid, the run completes successfully with the final candidate and its
failed validation result rather than returning an operational error.
Each correction starts from the original rendered messages and includes only
the latest invalid candidate and latest validation diagnostics. JSON Schema
mode retains the provider-native structured-output request as its first line of
defense. Promptkit performs only deterministic structural validation; a valid
response is not necessarily factual or correct for an application's domain.
Usage in the final result is cumulative across the initial response and every
completed corrective response. `ValidationResult.RepairAttempts` reports the
number of corrective calls actually made. Corrective generation failures use
the same public generation-error categories and structured provider details as
an initial generation failure.
See the [output-contract format reference](../formats.md#output-contract), the
[consumer repair example](../consumers/pkg-promptkit.md#repair-a-structured-result),
and the [`OutputContract` and `ValidationResult` GoDoc](../../types.go) for the
current contracts.
## Explicit Empty Content
The built-in OpenAI-compatible client now distinguishes an explicitly present
empty string from a missing or malformed `content` field. This aligns built-in
and injected clients by letting the selected output contract decide whether an
empty candidate is acceptable, invalid, or eligible for repair.
See the
[OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling)
for the exact envelope behavior.
## Public API Changes
None. This release activates and tightens the documented behavior of existing
fields.
## Consumer Action
- Remove or set `repair_attempts` to zero where execution must remain
single-pass.
- Keep every positive repair budget at three or fewer and pair it with
`basic`, `json`, or `json_schema` validation.
- Account for additional latency, usage, and provider cost when enabling
repair.
- Continue checking the returned validation status because bounded repair can
exhaust without producing a valid candidate.
- Review workflows that previously treated explicit empty provider content as
a generation error.

View File

@@ -38,28 +38,8 @@ consumers.
## Ideas
### Public bounded output repair
Promptkit should make its bounded output-repair capability available through
the public engine. A consumer should be able to request a limited number of
corrective generation attempts when JSON or JSON Schema output fails content
validation, without having to reproduce Promptkit's generation, validation,
capacity, and result-accounting orchestration.
- Repair is validation recovery, not a general provider retry, failover, or
backoff policy. Transport failures, cancellation, and operational schema or
validation errors must retain their ordinary error behavior.
- Repair must stop after the first valid result or the configured attempt
bound. Exhausting the bound should preserve the final invalid result and its
validation diagnostics rather than inventing success.
- Initial generation and every repair attempt must use the same resolved
backend, effective execution settings and presence semantics, session,
credential boundary, structured-output contract, and backend-capacity
policy.
- Results should report the number of repair attempts and cumulative usage for
every model call made by the run.
- Ordinary and prepared execution should expose coherent behavior, including
cancellation, frozen prepared state, error identity, and capacity lifetime.
No ideas are currently awaiting selection. Active feature work belongs in its
focused roadmap rather than this catalog.
## Entry Format

View File

@@ -430,7 +430,7 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
}
return &Engine{
runner: usecase.NewRunner(
runner: usecase.NewRunnerWithRepairer(
promptDefs,
profiles,
backendRegistry,
@@ -438,6 +438,7 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
prompt.NewGoRenderer(),
llmClient,
validator,
usecase.NewDefaultOutputRepairer(llmClient),
capacityManager,
),
}, nil
@@ -639,10 +640,12 @@ func (e *Engine) PrepareExecution(ctx context.Context, req RunRequest) (*Prepare
// generated output.
//
// A content-validation failure is a successful run whose
// RunResult.Validation has Status ValidationFailed. An inability to perform
// validation returns an error matching ErrValidation and no partial result.
// The public Engine does not perform output repair, so validation is
// single-pass even when OutputContract.RepairAttempts is positive.
// RunResult.Validation has Status ValidationFailed. When its output contract
// has a positive repair budget, a failed eligible validation can make bounded
// additional model calls and stops at the first valid candidate. Exhaustion
// returns the final failed validation result with cumulative usage and actual
// repair attempts. An inability to generate or validate returns an error and
// no partial result.
//
// Run can return every error category documented by [Engine.Prepare], plus
// ErrCapacityExceeded and ErrLLMGenerate. An engine admission rejection is
@@ -686,17 +689,17 @@ func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
//
// The supplied context governs this execution attempt independently of the
// preparation context. It covers credential revalidation, admission,
// generation, validation, and any internal repair. Result timing begins after
// the claim and excludes preparation and consumer-held delay.
// generation, validation, and any bounded output repair. Result timing begins
// after the claim and excludes preparation and consumer-held delay.
//
// RunPrepared can return ErrInvalidRequest, ErrAPIKeyEnvMissing,
// ErrCapacityExceeded, ErrLLMGenerate, or ErrValidation as applicable while
// preserving documented collaborator and context identities. An engine
// admission rejection is discoverable as [CapacityError] and still matches
// ErrCapacityExceeded. A built-in OpenAI-compatible non-2xx response is
// discoverable as [GenerationError]. A completed content-validation rejection
// is returned in RunResult, not as an operational error. An operational error
// returns no partial RunResult.
// discoverable as [GenerationError]. A completed content-validation rejection,
// including repair exhaustion, is returned in RunResult, not as an operational
// error. An operational error returns no partial RunResult.
func (e *Engine) RunPrepared(ctx context.Context, prepared *PreparedExecution) (*RunResult, error) {
if e == nil || e.runner == nil {
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)

View File

@@ -3486,9 +3486,10 @@ extra_params:
}
type fakeLLMClient struct {
response *promptkit.GenerateResponse
err error
requests []promptkit.GenerateRequest
response *promptkit.GenerateResponse
responses []*promptkit.GenerateResponse
err error
requests []promptkit.GenerateRequest
}
type recordingArtifactReader struct {
@@ -3664,5 +3665,12 @@ func (f *fakeLLMClient) Generate(_ context.Context, req promptkit.GenerateReques
if f.err != nil {
return nil, f.err
}
if len(f.responses) > 0 {
index := len(f.requests) - 1
if index >= len(f.responses) {
return nil, fmt.Errorf("no response configured for generation %d", index+1)
}
return f.responses[index], nil
}
return f.response, nil
}

View File

@@ -39,6 +39,44 @@ func TestBuiltInGenerationError(t *testing.T) {
assertGenerationError(t, err, http.StatusServiceUnavailable, "", "", "")
}
func TestBuiltInRepairGenerationError(t *testing.T) {
const (
codeMarker = "repair-code-marker"
typeMarker = "repair-type-marker"
messageMarker = "repair-message-marker"
)
calls := 0
config := contractConfig(frameworkSchemaDir)
config.HTTPClient = &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
calls++
if calls == 1 {
body := `{"choices":[{"message":{"content":"not-json"}}]}`
return &http.Response{StatusCode: http.StatusOK, ContentLength: int64(len(body)), Body: io.NopCloser(strings.NewReader(body))}, nil
}
body := `{"error":{"code":"` + codeMarker + `","type":"` + typeMarker + `","message":"` + messageMarker + `"}}`
return &http.Response{StatusCode: http.StatusUnprocessableEntity, ContentLength: int64(len(body)), Body: io.NopCloser(strings.NewReader(body))}, nil
})}
engine, err := promptkit.NewEngine(config)
if err != nil {
t.Fatalf("NewEngine: %v", err)
}
req := generationErrorRunRequest()
req.Validation = &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSON,
RepairAttempts: 1,
}
result, err := engine.Run(context.Background(), req)
if result != nil {
t.Fatalf("Run result = %#v, want nil", result)
}
if calls != 2 {
t.Fatalf("provider calls = %d, want 2", calls)
}
assertGenerationError(t, err, http.StatusUnprocessableEntity, codeMarker, typeMarker, messageMarker)
}
func assertGenerationError(t *testing.T, err error, statusCode int, code, providerType, message string) {
t.Helper()

View File

@@ -6,6 +6,8 @@ import (
"strings"
)
const maxOutputRepairAttempts = 3
// ValidateOutputContract validates source-neutral output-contract invariants.
func ValidateOutputContract(contract OutputContract) error {
switch contract.Format {
@@ -26,5 +28,11 @@ func ValidateOutputContract(contract OutputContract) error {
if contract.RepairAttempts < 0 {
return errors.New("repair_attempts must be greater than or equal to 0")
}
if contract.RepairAttempts > maxOutputRepairAttempts {
return fmt.Errorf("repair_attempts must be less than or equal to %d", maxOutputRepairAttempts)
}
if contract.ValidationMode == ValidationNone && contract.RepairAttempts > 0 {
return errors.New("repair_attempts requires basic, json, or json_schema validation")
}
return nil
}

View File

@@ -30,9 +30,28 @@ func TestValidateOutputContract(t *testing.T) {
}},
{name: "empty validation mode", change: func(c *OutputContract) { c.ValidationMode = "" }, wantErr: "validation mode"},
{name: "unsupported validation mode", change: func(c *OutputContract) { c.ValidationMode = ValidationMode("unknown") }, wantErr: "validation mode"},
{name: "negative repair attempts", change: func(c *OutputContract) { c.RepairAttempts = -1 }, wantErr: "repair_attempts"},
{name: "negative repair attempts", change: func(c *OutputContract) {
c.ValidationMode = ValidationBasic
c.RepairAttempts = -1
}, wantErr: "repair_attempts"},
{name: "zero repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 0 }},
{name: "positive repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 1 }},
{name: "one repair attempt", change: func(c *OutputContract) {
c.ValidationMode = ValidationBasic
c.RepairAttempts = 1
}},
{name: "maximum repair attempts", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSON
c.RepairAttempts = 3
}},
{name: "too many repair attempts", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSONSchema
c.SchemaPath = "schema.json"
c.RepairAttempts = 4
}, wantErr: "repair_attempts"},
{name: "none validation with repair attempts", change: func(c *OutputContract) {
c.ValidationMode = ValidationNone
c.RepairAttempts = 1
}, wantErr: "repair_attempts"},
{name: "json schema empty path", change: func(c *OutputContract) {
c.ValidationMode = ValidationJSONSchema
c.SchemaPath = ""

View File

@@ -179,12 +179,12 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
return nil, fmt.Errorf("%w: no choices returned", ErrMalformedResponse)
}
content := wireResp.Choices[0].Message.Content
if content == "" {
return nil, fmt.Errorf("%w: first choice has empty message content", ErrMalformedResponse)
if content == nil {
return nil, fmt.Errorf("%w: first choice has missing message content", ErrMalformedResponse)
}
return &domain.GenerateResponse{
Content: content,
Content: *content,
Usage: domain.TokenUsage{
PromptTokens: wireResp.Usage.PromptTokens,
CompletionTokens: wireResp.Usage.CompletionTokens,
@@ -379,8 +379,8 @@ type openAICacheControl struct {
}
type openAIChatResponseMessage struct {
Role string `json:"role"`
Content string `json:"content"`
Role string `json:"role"`
Content *string `json:"content"`
}
type openAIChatResponse struct {

View File

@@ -765,6 +765,7 @@ func TestOpenAICompatibleClientResponseFraming(t *testing.T) {
run func(*testing.T)
}{
{name: "usage mapping", run: checkCacheUsageMapping},
{name: "content presence", run: checkContentPresence},
{name: "common response failures", run: checkCommonResponseFailures},
{name: "successful response byte boundary", run: checkSuccessfulResponseByteBoundary},
{name: "continuing oversized response", run: checkContinuingOversizedResponse},
@@ -776,6 +777,53 @@ func TestOpenAICompatibleClientResponseFraming(t *testing.T) {
}
}
func checkContentPresence(t *testing.T) {
tests := []struct {
name string
content string
}{
{name: "explicit empty string", content: ""},
{name: "whitespace string", content: " \n\t "},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
provider := newRecordingProvider(t)
provider.respond(http.StatusOK, `{
"choices": [{"message": {"content": `+strconv.Quote(tc.content)+`}}],
"usage": {
"prompt_tokens": 10,
"completion_tokens": 20,
"total_tokens": 30,
"prompt_tokens_details": {"cached_tokens": 4},
"cache_write_tokens": 5
}
}`)
client := newProviderClient(t, provider, OpenAICompatibleConfig{Model: "model"})
response, err := client.Generate(context.Background(), ordinaryGenerateRequest())
if err != nil {
t.Fatalf("generate: %v", err)
}
if response == nil {
t.Fatal("expected response")
}
if response.Content != tc.content {
t.Fatalf("content = %q, want %q", response.Content, tc.content)
}
if response.Usage != (domain.TokenUsage{
PromptTokens: 10,
CompletionTokens: 20,
TotalTokens: 30,
CachedTokens: 4,
CacheWriteTokens: 5,
}) {
t.Fatalf("usage = %+v", response.Usage)
}
})
}
}
func checkCacheUsageMapping(t *testing.T) {
provider := newRecordingProvider(t)
provider.respond(http.StatusOK, `{
@@ -1129,6 +1177,9 @@ func checkCommonResponseFailures(t *testing.T) {
},
{name: "invalid JSON", statusCode: http.StatusOK, body: `{not valid json`, wantErr: ErrMalformedResponse},
{name: "missing choices", statusCode: http.StatusOK, body: `{"choices": []}`, wantErr: ErrMalformedResponse},
{name: "missing content", statusCode: http.StatusOK, body: `{"choices": [{"message": {}}]}`, wantErr: ErrMalformedResponse},
{name: "null content", statusCode: http.StatusOK, body: `{"choices": [{"message": {"content": null}}]}`, wantErr: ErrMalformedResponse},
{name: "non-string content", statusCode: http.StatusOK, body: `{"choices": [{"message": {"content": 1}}]}`, wantErr: ErrMalformedResponse},
}
for _, tc := range tests {

View File

@@ -892,6 +892,38 @@ output:
format: text
validation_mode: none
repair_attempts: -1
`,
wantErr: true,
wantDiagnostic: "repair_attempts",
},
{
name: "repair attempts above maximum",
definition: `
id: normalization-rule
version: "1"
messages:
- role: user
content: test
output:
format: text
validation_mode: basic
repair_attempts: 4
`,
wantErr: true,
wantDiagnostic: "repair_attempts",
},
{
name: "none validation with repair attempts",
definition: `
id: normalization-rule
version: "1"
messages:
- role: user
content: test
output:
format: text
validation_mode: none
repair_attempts: 1
`,
wantErr: true,
wantDiagnostic: "repair_attempts",

View File

@@ -106,6 +106,8 @@ func TestRunnerPreparationRejectsInvalidOutputContractsBeforeCompletion(t *testi
{name: "unsupported format", override: domain.OutputContract{Format: "binary", ValidationMode: domain.ValidationNone}},
{name: "empty validation mode", override: domain.OutputContract{Format: domain.FormatText}},
{name: "negative repair attempts", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone, RepairAttempts: -1}},
{name: "repair attempts above maximum", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationBasic, RepairAttempts: 4}},
{name: "none validation with repair attempts", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone, RepairAttempts: 1}},
{name: "json schema without path", override: domain.OutputContract{Format: domain.FormatJSON, ValidationMode: domain.ValidationJSONSchema}},
}

View File

@@ -3,10 +3,12 @@ package usecase
import (
"context"
"errors"
"fmt"
"reflect"
"testing"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
)
@@ -385,19 +387,20 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
admitter := &fakeRunAdmitter{}
reader := defaultArtifactReader()
renderer := defaultRenderer()
client := &sequenceLLM{responses: []*domain.GenerateResponse{{
Content: `{"broken":true}`,
Usage: domain.TokenUsage{
PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19,
CachedTokens: 23, CacheWriteTokens: 29,
},
}}}
runner := NewRunnerWithRepairer(
&fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)},
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
nil,
reader,
renderer,
&fakeLLM{resp: &domain.GenerateResponse{
Content: `{"broken":true}`,
Usage: domain.TokenUsage{
PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19,
CachedTokens: 23, CacheWriteTokens: 29,
},
}},
client,
validator,
repairer,
admitter,
@@ -421,6 +424,10 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
if validator.directValidateCalls != 0 || repairer.calls != 1 {
t.Fatalf("validation/repair calls=(direct=%d repair=%d), want (0, 1)", validator.directValidateCalls, repairer.calls)
}
if len(client.requests) != 1 || len(repairer.reqs) != 1 ||
!reflect.DeepEqual(repairer.reqs[0].OriginalMessages, client.requests[0].Prompt.Messages) {
t.Fatalf("initial and repair messages = (%#v, %#v)", client.requests, repairer.reqs)
}
if result.Validation.Status != domain.ValidationPassed || result.Validation.RepairAttempts != 1 {
t.Fatalf("unexpected repaired validation result: %+v", result.Validation)
}
@@ -443,6 +450,7 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
generationFailure := errors.New("generation failed")
validationFailure := errors.New("validation failed")
repairFailure := errors.New("repair failed")
repairInvalidRequest := fmt.Errorf("repair request: %w", llm.ErrInvalidRequest)
tests := []struct {
name string
@@ -450,6 +458,7 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
validation *recordingPreparedValidation
repairer *fakeRepairer
wantError error
wantSource error
}{
{
name: "generation failure",
@@ -474,8 +483,51 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
IsValid: false,
}},
},
repairer: &fakeRepairer{err: repairFailure},
wantError: ErrValidation,
repairer: &fakeRepairer{err: repairFailure},
wantError: ErrLLMGenerate,
wantSource: repairFailure,
},
{
name: "repair invalid request",
validation: &recordingPreparedValidation{
results: []domain.ValidationResult{{
Status: domain.ValidationFailed,
Mode: domain.ValidationJSON,
Errors: []string{"invalid"},
IsValid: false,
}},
},
repairer: &fakeRepairer{err: repairInvalidRequest},
wantError: ErrInvalidRequest,
wantSource: repairInvalidRequest,
},
{
name: "repair cancellation",
validation: &recordingPreparedValidation{
results: []domain.ValidationResult{{
Status: domain.ValidationFailed,
Mode: domain.ValidationJSON,
Errors: []string{"invalid"},
IsValid: false,
}},
},
repairer: &fakeRepairer{err: context.Canceled},
wantError: ErrLLMGenerate,
wantSource: context.Canceled,
},
{
name: "repair deadline",
validation: &recordingPreparedValidation{
results: []domain.ValidationResult{{
Status: domain.ValidationFailed,
Mode: domain.ValidationJSON,
Errors: []string{"invalid"},
IsValid: false,
}},
},
repairer: &fakeRepairer{err: context.DeadlineExceeded},
wantError: ErrLLMGenerate,
wantSource: context.DeadlineExceeded,
},
}
@@ -514,6 +566,9 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
if result != nil || !errors.Is(err, test.wantError) {
t.Fatalf("run prepared=(%+v, %v), want %v", result, err, test.wantError)
}
if test.wantSource != nil && !errors.Is(err, test.wantSource) {
t.Fatalf("run prepared error = %v, want source %v", err, test.wantSource)
}
if len(admitter.backendIDs) != 1 || admitter.releaseCalls != 1 {
t.Fatalf(
"admission calls=%#v releases=%d, want one each",

View File

@@ -2,19 +2,29 @@ package usecase
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"unicode/utf8"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
)
const (
maxRepairDiagnosticBytes = 64 * 1024
omittedRepairDiagnostics = "additional validation diagnostics were omitted"
)
// OutputRepairer generates a corrected candidate after validation fails.
type OutputRepairer interface {
Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error)
}
// RepairRequest contains the immutable execution state needed for one correction.
type RepairRequest struct {
OriginalMessages []domain.RenderedMessage
PreviousOutput string
ValidationErrors []string
SessionID string
@@ -30,6 +40,7 @@ type defaultOutputRepairer struct {
llm llm.Client
}
// NewDefaultOutputRepairer constructs the standard internal output repairer.
func NewDefaultOutputRepairer(llmClient llm.Client) OutputRepairer {
return &defaultOutputRepairer{llm: llmClient}
}
@@ -39,33 +50,42 @@ func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (
return nil, errors.New("llm client is required for repair")
}
errs := "(none provided)"
if len(req.ValidationErrors) > 0 {
errs = strings.Join(req.ValidationErrors, "\n")
guidance, err := repairGuidance(req.Mode)
if err != nil {
return nil, err
}
prompt := domain.RenderedPrompt{
Messages: []domain.RenderedMessage{
{
Role: "system",
Content: "You repair invalid JSON output. Return only corrected JSON. Do not include explanations or markdown code fences.",
},
{
Role: "user",
Content: fmt.Sprintf(
"Repair attempt %d of %d for validation mode %s.\n\nValidation errors:\n%s\n\nPrevious output:\n%s\n\nReturn only corrected JSON.",
req.Attempt,
req.MaxAttempts,
req.Mode,
errs,
req.PreviousOutput,
),
},
},
messages := make([]domain.RenderedMessage, len(req.OriginalMessages), len(req.OriginalMessages)+2)
copy(messages, req.OriginalMessages)
if strings.TrimSpace(req.PreviousOutput) != "" {
messages = append(messages, domain.RenderedMessage{
Role: "assistant",
Content: req.PreviousOutput,
})
}
previousResponse := "The previous response was empty."
if strings.TrimSpace(req.PreviousOutput) != "" {
previousResponse = "The previous response is included immediately before this instruction."
}
messages = append(messages, domain.RenderedMessage{
Role: "user",
Content: fmt.Sprintf(
"Repair attempt %d of %d for validation mode %s.\n"+
"Preserve valid values and change only what is necessary.\n"+
"%s\n%s\n"+
"Validation diagnostics (data):\n%s",
req.Attempt,
req.MaxAttempts,
req.Mode,
previousResponse,
guidance,
formatRepairDiagnostics(req.ValidationErrors),
),
})
resp, err := r.llm.Generate(ctx, newGenerationRequest(
prompt,
domain.RenderedPrompt{Messages: messages},
req.SessionID,
req.Target,
req.TargetPresence,
@@ -80,3 +100,90 @@ func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (
return resp, nil
}
func repairGuidance(mode domain.ValidationMode) (string, error) {
switch mode {
case domain.ValidationBasic:
return "Return a nonempty response satisfying the original request.", nil
case domain.ValidationJSON, domain.ValidationJSONSchema:
return "Return only corrected JSON, with no explanation or Markdown fences.", nil
default:
return "", fmt.Errorf("unsupported validation mode for repair: %q", mode)
}
}
func formatRepairDiagnostics(errors []string) string {
diagnostics := make([]string, len(errors))
for index, diagnostic := range errors {
diagnostics[index] = strings.ToValidUTF8(diagnostic, "\uFFFD")
}
complete, _ := json.Marshal(diagnostics)
if len(complete) <= maxRepairDiagnosticBytes {
return string(complete)
}
omission, _ := json.Marshal(omittedRepairDiagnostics)
encoded := make([]byte, 0, maxRepairDiagnosticBytes)
encoded = append(encoded, '[')
for _, diagnostic := range diagnostics {
entry, _ := json.Marshal(diagnostic)
separator := 0
if len(encoded) > 1 {
separator = 1
}
available := maxRepairDiagnosticBytes - len(encoded) - separator - 1 - len(omission) - 1
if len(entry) <= available {
if separator != 0 {
encoded = append(encoded, ',')
}
encoded = append(encoded, entry...)
continue
}
if available < len(`""`) {
break
}
if separator != 0 {
encoded = append(encoded, ',')
}
encoded = append(encoded, truncateDiagnosticJSONValue(diagnostic, available)...)
break
}
if len(encoded) > 1 {
encoded = append(encoded, ',')
}
encoded = append(encoded, omission...)
encoded = append(encoded, ']')
return string(encoded)
}
func truncateDiagnosticJSONValue(value string, maxBytes int) []byte {
if maxBytes < len(`""`) {
return nil
}
boundaries := []int{0}
for end := 0; end < len(value); {
_, size := utf8.DecodeRuneInString(value[end:])
end += size
if end+len(`""`) > maxBytes {
break
}
boundaries = append(boundaries, end)
}
low, high := 0, len(boundaries)-1
best := []byte(`""`)
for low <= high {
mid := low + (high-low)/2
candidate, _ := json.Marshal(value[:boundaries[mid]])
if len(candidate) <= maxBytes {
best = candidate
low = mid + 1
continue
}
high = mid - 1
}
return best
}

View File

@@ -0,0 +1,349 @@
package usecase
import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"strings"
"testing"
"unicode/utf8"
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
)
type recordingRepairClient struct {
requests []domain.GenerateRequest
response *domain.GenerateResponse
err error
}
func (c *recordingRepairClient) Generate(_ context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) {
c.requests = append(c.requests, req)
return c.response, c.err
}
func TestDefaultOutputRepairerBuildsFullContextRequest(t *testing.T) {
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
repairer := NewDefaultOutputRepairer(client)
original := []domain.RenderedMessage{
{Role: "system", Content: "Follow the task.", CacheControl: &domain.CacheControl{Type: domain.CacheControlEphemeral, TTL: "1h"}},
{Role: "user", Content: "Summarize the report."},
}
before := append([]domain.RenderedMessage(nil), original...)
previous := strings.Repeat("candidate ", 12_000)
target := domain.ExecutionTarget{BackendID: "backend", Endpoint: "https://provider.example/v1", Model: "model"}
presence := domain.ExecutionTargetPresence{Temperature: true, TopP: true}
structured := &domain.StructuredOutputSpec{}
response, err := repairer.Repair(context.Background(), RepairRequest{
OriginalMessages: original,
PreviousOutput: previous,
ValidationErrors: []string{"invalid JSON"},
SessionID: "session",
Target: target,
TargetPresence: presence,
StructuredOutput: structured,
Attempt: 1,
MaxAttempts: 3,
Mode: domain.ValidationJSONSchema,
})
if err != nil {
t.Fatalf("repair: %v", err)
}
if response == nil || response.Content != "corrected" {
t.Fatalf("response = %+v", response)
}
if !reflect.DeepEqual(original, before) {
t.Fatalf("original messages changed: got %#v, want %#v", original, before)
}
if len(client.requests) != 1 {
t.Fatalf("generation requests = %d, want 1", len(client.requests))
}
request := client.requests[0]
if request.Prompt.SessionID != "session" || !reflect.DeepEqual(request.Target, target) || request.TargetPresence != presence || request.StructuredOutput != structured {
t.Fatalf("generation request fields = %+v", request)
}
if len(request.Prompt.Messages) != len(original)+2 {
t.Fatalf("message count = %d, want %d", len(request.Prompt.Messages), len(original)+2)
}
if !reflect.DeepEqual(request.Prompt.Messages[:len(original)], original) {
t.Fatalf("original messages = %#v, want %#v", request.Prompt.Messages[:len(original)], original)
}
assistant := request.Prompt.Messages[len(original)]
if assistant.Role != "assistant" || assistant.Content != previous {
t.Fatalf("assistant candidate = %+v", assistant)
}
correction := request.Prompt.Messages[len(original)+1]
if correction.Role != "user" || !strings.Contains(correction.Content, "Repair attempt 1 of 3") ||
!strings.Contains(correction.Content, "Preserve valid values") ||
!strings.Contains(correction.Content, "Return only corrected JSON") {
t.Fatalf("correction message = %q", correction.Content)
}
if diagnostics := repairDiagnosticsFromMessage(t, correction.Content); !reflect.DeepEqual(diagnostics, []string{"invalid JSON"}) {
t.Fatalf("diagnostics = %#v", diagnostics)
}
}
func TestDefaultOutputRepairerDoesNotAccumulateCandidates(t *testing.T) {
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
repairer := NewDefaultOutputRepairer(client)
backing := make([]domain.RenderedMessage, 1, 4)
backing[0] = domain.RenderedMessage{Role: "user", Content: "Original task"}
before := append([]domain.RenderedMessage(nil), backing...)
for _, candidate := range []string{"first invalid", "second invalid"} {
_, err := repairer.Repair(context.Background(), RepairRequest{
OriginalMessages: backing,
PreviousOutput: candidate,
Attempt: 1,
MaxAttempts: 3,
Mode: domain.ValidationJSON,
})
if err != nil {
t.Fatalf("repair %q: %v", candidate, err)
}
}
if !reflect.DeepEqual(backing, before) {
t.Fatalf("caller messages changed: got %#v, want %#v", backing, before)
}
if len(client.requests) != 2 {
t.Fatalf("generation requests = %d, want 2", len(client.requests))
}
for index, request := range client.requests {
messages := request.Prompt.Messages
if len(messages) != 3 || messages[0] != backing[0] || messages[1].Role != "assistant" || messages[1].Content != []string{"first invalid", "second invalid"}[index] {
t.Fatalf("request %d messages = %#v", index, messages)
}
}
}
func TestDefaultOutputRepairerOmitsEmptyCandidateMessage(t *testing.T) {
for _, tc := range []struct {
name string
candidate string
}{
{name: "empty", candidate: ""},
{name: "whitespace", candidate: " \n\t "},
} {
t.Run(tc.name, func(t *testing.T) {
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
repairer := NewDefaultOutputRepairer(client)
_, err := repairer.Repair(context.Background(), RepairRequest{
OriginalMessages: []domain.RenderedMessage{{Role: "user", Content: "Original task"}},
PreviousOutput: tc.candidate,
Attempt: 1,
MaxAttempts: 1,
Mode: domain.ValidationBasic,
})
if err != nil {
t.Fatalf("repair: %v", err)
}
messages := client.requests[0].Prompt.Messages
if len(messages) != 2 || messages[1].Role != "user" || !strings.Contains(messages[1].Content, "previous response was empty") {
t.Fatalf("messages = %#v", messages)
}
if strings.Contains(messages[1].Content, tc.candidate) && tc.candidate != "" {
t.Fatalf("correction message repeated whitespace candidate: %q", messages[1].Content)
}
})
}
}
func TestDefaultOutputRepairerUsesModeSpecificGuidance(t *testing.T) {
tests := []struct {
mode domain.ValidationMode
want string
}{
{mode: domain.ValidationBasic, want: "Return a nonempty response"},
{mode: domain.ValidationJSON, want: "Return only corrected JSON"},
{mode: domain.ValidationJSONSchema, want: "Return only corrected JSON"},
}
for _, tc := range tests {
t.Run(string(tc.mode), func(t *testing.T) {
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
_, err := NewDefaultOutputRepairer(client).Repair(context.Background(), RepairRequest{Mode: tc.mode, Attempt: 1, MaxAttempts: 1})
if err != nil {
t.Fatalf("repair: %v", err)
}
if !strings.Contains(client.requests[0].Prompt.Messages[0].Content, tc.want) {
t.Fatalf("correction message = %q", client.requests[0].Prompt.Messages[0].Content)
}
})
}
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "unexpected"}}
_, err := NewDefaultOutputRepairer(client).Repair(context.Background(), RepairRequest{Mode: domain.ValidationNone})
if err == nil || !strings.Contains(err.Error(), "unsupported validation mode") {
t.Fatalf("unsupported mode error = %v", err)
}
if len(client.requests) != 0 {
t.Fatalf("generation requests = %d, want 0", len(client.requests))
}
}
func TestFormatRepairDiagnosticsBoundsAndPreservesData(t *testing.T) {
tests := []struct {
name string
errors []string
wantOmission bool
check func(*testing.T, []string)
}{
{
name: "below limit",
errors: []string{"first", "second"},
check: func(t *testing.T, got []string) {
t.Helper()
if !reflect.DeepEqual(got, []string{"first", "second"}) {
t.Fatalf("diagnostics = %#v", got)
}
},
},
{
name: "at limit",
errors: []string{strings.Repeat("x", maxRepairDiagnosticBytes-4)},
check: func(t *testing.T, got []string) {
t.Helper()
if len(got) != 1 || len(got[0]) != maxRepairDiagnosticBytes-4 {
t.Fatalf("diagnostics lengths = %#v", got)
}
},
},
{
name: "multibyte truncation preserves prior entries",
errors: []string{"first", strings.Repeat("界", maxRepairDiagnosticBytes)},
wantOmission: true,
check: func(t *testing.T, got []string) {
t.Helper()
if len(got) != 3 || got[0] != "first" || !strings.HasPrefix(strings.Repeat("界", maxRepairDiagnosticBytes), got[1]) {
t.Fatalf("diagnostics = %#v", got)
}
},
},
{
name: "many diagnostics",
errors: manyRepairDiagnostics(),
wantOmission: true,
check: func(t *testing.T, got []string) {
t.Helper()
if len(got) < 2 || got[0] != manyRepairDiagnostics()[0] {
t.Fatalf("diagnostics = %#v", got)
}
},
},
{
name: "no room for partial diagnostic",
errors: func() []string {
omission, _ := json.Marshal(omittedRepairDiagnostics)
return []string{
strings.Repeat("x", maxRepairDiagnosticBytes-len(omission)-5),
strings.Repeat("y", 128),
}
}(),
wantOmission: true,
check: func(t *testing.T, got []string) {
t.Helper()
if len(got) != 2 || got[1] != omittedRepairDiagnostics {
t.Fatalf("diagnostics = %#v", got)
}
},
},
{
name: "invalid UTF-8",
errors: []string{"broken\xffinput"},
check: func(t *testing.T, got []string) {
t.Helper()
if !reflect.DeepEqual(got, []string{"broken\uFFFDinput"}) {
t.Fatalf("diagnostics = %#v", got)
}
},
},
{
name: "one huge diagnostic",
errors: []string{strings.Repeat("x", maxRepairDiagnosticBytes*2)},
wantOmission: true,
check: func(t *testing.T, got []string) {
t.Helper()
if len(got) != 2 || !strings.HasPrefix(strings.Repeat("x", maxRepairDiagnosticBytes*2), got[0]) {
t.Fatalf("diagnostics = %#v", got)
}
},
},
}
for _, tc := range tests {
t.Run(tc.name, func(t *testing.T) {
before := append([]string(nil), tc.errors...)
encoded := formatRepairDiagnostics(tc.errors)
if len(encoded) > maxRepairDiagnosticBytes || !utf8.ValidString(encoded) {
t.Fatalf("encoded diagnostic length/UTF-8 = (%d, %v)", len(encoded), utf8.ValidString(encoded))
}
var got []string
if err := json.Unmarshal([]byte(encoded), &got); err != nil {
t.Fatalf("decode diagnostics: %v; encoded=%q", err, encoded)
}
if !reflect.DeepEqual(tc.errors, before) {
t.Fatalf("input diagnostics changed: got %#v, want %#v", tc.errors, before)
}
if hasOmission := len(got) > 0 && got[len(got)-1] == omittedRepairDiagnostics; hasOmission != tc.wantOmission {
t.Fatalf("omission = %v, want %v; diagnostics=%#v", hasOmission, tc.wantOmission, got)
}
tc.check(t, got)
})
}
}
func manyRepairDiagnostics() []string {
diagnostics := make([]string, 1_000)
for index := range diagnostics {
diagnostics[index] = fmt.Sprintf("diagnostic %04d %s", index, strings.Repeat("x", 128))
}
return diagnostics
}
func TestDefaultOutputRepairerPropagatesGenerationFailures(t *testing.T) {
expected := errors.New("generation failed")
for _, tc := range []struct {
name string
client *recordingRepairClient
want error
}{
{name: "nil client", want: nil},
{name: "generation error", client: &recordingRepairClient{err: expected}, want: expected},
{name: "nil response", client: &recordingRepairClient{}, want: nil},
} {
t.Run(tc.name, func(t *testing.T) {
var repairer OutputRepairer
if tc.client == nil {
repairer = NewDefaultOutputRepairer(nil)
} else {
repairer = NewDefaultOutputRepairer(tc.client)
}
response, err := repairer.Repair(context.Background(), RepairRequest{Mode: domain.ValidationJSON})
if response != nil || err == nil {
t.Fatalf("response/error = (%+v, %v)", response, err)
}
if tc.want != nil && !errors.Is(err, tc.want) {
t.Fatalf("error = %v, want %v", err, tc.want)
}
})
}
}
func repairDiagnosticsFromMessage(t *testing.T, message string) []string {
t.Helper()
const marker = "Validation diagnostics (data):\n"
index := strings.Index(message, marker)
if index < 0 {
t.Fatalf("missing diagnostics marker in %q", message)
}
encoded := message[index+len(marker):]
var diagnostics []string
if err := json.Unmarshal([]byte(encoded), &diagnostics); err != nil {
t.Fatalf("decode diagnostics: %v", err)
}
return diagnostics
}

View File

@@ -184,10 +184,10 @@ func (r *Runner) executePreparedRun(
prepared.StructuredOutput,
))
if err != nil {
if errors.Is(err, llm.ErrInvalidRequest) {
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
return nil, fmt.Errorf("%w: %w", ErrLLMGenerate, err)
return nil, wrapGenerationError(err)
}
if genResp == nil {
return nil, fmt.Errorf("%w: model returned nil response", ErrLLMGenerate)
}
usage := genResp.Usage
@@ -203,6 +203,7 @@ func (r *Runner) executePreparedRun(
attemptsUsed++
repairResp, repairErr := r.repairer.Repair(ctx, RepairRequest{
OriginalMessages: prepared.Messages,
PreviousOutput: genResp.Content,
ValidationErrors: validationResult.Errors,
SessionID: prepared.SessionID,
@@ -214,10 +215,10 @@ func (r *Runner) executePreparedRun(
Mode: prepared.OutputContract.ValidationMode,
})
if repairErr != nil {
return nil, fmt.Errorf("%w: %w", ErrValidation, repairErr)
return nil, wrapGenerationError(repairErr)
}
if repairResp == nil {
return nil, fmt.Errorf("%w: repairer returned nil response", ErrValidation)
return nil, fmt.Errorf("%w: repairer returned nil response", ErrLLMGenerate)
}
genResp = repairResp
@@ -257,6 +258,13 @@ func (r *Runner) executePreparedRun(
}, nil
}
func wrapGenerationError(err error) error {
if errors.Is(err, llm.ErrInvalidRequest) {
return fmt.Errorf("%w: %w", ErrInvalidRequest, err)
}
return fmt.Errorf("%w: %w", ErrLLMGenerate, err)
}
func addTokenUsage(total, next domain.TokenUsage) domain.TokenUsage {
return domain.TokenUsage{
PromptTokens: total.PromptTokens + next.PromptTokens,
@@ -524,7 +532,12 @@ func (r *Runner) shouldAttemptRepair(contract domain.OutputContract, validationR
if validationResult.Status != domain.ValidationFailed {
return false
}
return contract.ValidationMode == domain.ValidationJSON || contract.ValidationMode == domain.ValidationJSONSchema
switch contract.ValidationMode {
case domain.ValidationBasic, domain.ValidationJSON, domain.ValidationJSONSchema:
return true
default:
return false
}
}
func mergeExecutionTarget(base domain.ExecutionTarget, override domain.ExecutionTarget) domain.ExecutionTarget {

View File

@@ -295,8 +295,10 @@ func (c *controlledRepairLLM) Generate(
ctx context.Context,
req domain.GenerateRequest,
) (*domain.GenerateResponse, error) {
isRepair := len(req.Prompt.Messages) > 0 &&
strings.HasPrefix(req.Prompt.Messages[0].Content, "You repair invalid JSON")
messages := req.Prompt.Messages
isRepair := len(messages) >= 2 &&
messages[len(messages)-2].Role == "assistant" &&
messages[len(messages)-1].Role == "user"
c.mu.Lock()
c.calls++
c.active++
@@ -2143,6 +2145,8 @@ func TestRunnerRepairStateMachine(t *testing.T) {
TopP: true,
TimeoutSeconds: true,
}
emptyThenValid := responses(2)
emptyThenValid[0].Content = " \t "
tests := []struct {
name string
@@ -2165,12 +2169,13 @@ func TestRunnerRepairStateMachine(t *testing.T) {
wantStatus: domain.ValidationPassed,
},
{
name: "basic failure is ineligible despite budget",
name: "empty basic output repairs successfully",
mode: domain.ValidationBasic,
budget: 3,
validationResults: []domain.ValidationResult{failed(domain.ValidationBasic, "empty output")},
responses: responses(1),
wantStatus: domain.ValidationFailed,
validationResults: []domain.ValidationResult{failed(domain.ValidationBasic, "empty output"), passed(domain.ValidationBasic)},
responses: emptyThenValid,
wantRepairs: 1,
wantStatus: domain.ValidationPassed,
},
{
name: "inherited numeric values remain absent",
@@ -2202,7 +2207,7 @@ func TestRunnerRepairStateMachine(t *testing.T) {
{
name: "successful repair stops below larger budget",
mode: domain.ValidationJSON,
budget: 4,
budget: 3,
validationResults: []domain.ValidationResult{
failed(domain.ValidationJSON, "candidate zero"),
failed(domain.ValidationJSON, "candidate one"),
@@ -2305,6 +2310,9 @@ func TestRunnerRepairStateMachine(t *testing.T) {
!reflect.DeepEqual(req.ValidationErrors, tc.validationResults[index].Errors) {
t.Fatalf("repair request %d prior state = %+v", index, req)
}
if !reflect.DeepEqual(req.OriginalMessages, initialRequest.Prompt.Messages) {
t.Fatalf("repair request %d original messages drifted: %#v", index, req.OriginalMessages)
}
if req.TargetPresence != tc.wantPresence || !reflect.DeepEqual(req.Target, initialRequest.Target) ||
req.SessionID != initialRequest.Prompt.SessionID ||
!reflect.DeepEqual(req.StructuredOutput, initialRequest.StructuredOutput) {
@@ -2318,8 +2326,19 @@ func TestRunnerRepairStateMachine(t *testing.T) {
!reflect.DeepEqual(generated.StructuredOutput, initialRequest.StructuredOutput) {
t.Fatalf("repair generation request %d common fields drifted: %+v", index, generated)
}
if reflect.DeepEqual(generated.Prompt.Messages, initialRequest.Prompt.Messages) {
t.Fatalf("repair generation request %d reused the initial prompt", index)
expectedMessages := len(initialRequest.Prompt.Messages) + 1
if strings.TrimSpace(tc.responses[index].Content) != "" {
expectedMessages++
}
if len(generated.Prompt.Messages) != expectedMessages ||
generated.Prompt.Messages[len(generated.Prompt.Messages)-1].Role != "user" {
t.Fatalf("repair generation request %d messages = %#v", index, generated.Prompt.Messages)
}
if strings.TrimSpace(tc.responses[index].Content) != "" {
assistant := generated.Prompt.Messages[len(generated.Prompt.Messages)-2]
if assistant.Role != "assistant" || assistant.Content != tc.responses[index].Content {
t.Fatalf("repair generation request %d candidate = %+v", index, assistant)
}
}
}

View File

@@ -174,9 +174,8 @@ func validateArtifact(ctx context.Context, artifact *domain.Artifact, contract d
}
res := domain.ValidationResult{
Mode: contract.ValidationMode,
SchemaPath: contract.SchemaPath,
RepairAttempts: contract.RepairAttempts,
Mode: contract.ValidationMode,
SchemaPath: contract.SchemaPath,
}
if artifact == nil {

View File

@@ -64,6 +64,43 @@ func TestStandardValidatorBasicFailureEmpty(t *testing.T) {
}
}
func TestStandardValidatorReportsZeroRepairAttempts(t *testing.T) {
v := NewStandardValidator("")
for _, tc := range []struct {
name string
body string
contract domain.OutputContract
}{
{
name: "passed basic validation",
body: "answer",
contract: domain.OutputContract{
ValidationMode: domain.ValidationBasic,
RepairAttempts: 3,
},
},
{
name: "failed JSON validation",
body: `{"answer":`,
contract: domain.OutputContract{
ValidationMode: domain.ValidationJSON,
RepairAttempts: 3,
},
},
} {
t.Run(tc.name, func(t *testing.T) {
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(tc.body)}, tc.contract)
if err != nil {
t.Fatalf("validate: %v", err)
}
if res.RepairAttempts != 0 {
t.Fatalf("repair attempts = %d, want 0", res.RepairAttempts)
}
})
}
}
func TestStandardValidatorJSONSuccess(t *testing.T) {
v := NewStandardValidator("")

View File

@@ -22,27 +22,52 @@ func TestPreparationRejectsInvalidOutputContractWithPublicError(t *testing.T) {
t.Fatalf("construct engine: %v", err)
}
req := promptkit.RunRequest{
PromptID: "prompt",
Validation: &promptkit.OutputContract{
Format: promptkit.OutputFormat("binary"),
ValidationMode: promptkit.ValidationNone,
for _, tc := range []struct {
name string
contract promptkit.OutputContract
}{
{
name: "unsupported format",
contract: promptkit.OutputContract{
Format: promptkit.OutputFormat("binary"),
ValidationMode: promptkit.ValidationNone,
},
},
}
{
name: "repair attempts above maximum",
contract: promptkit.OutputContract{
Format: promptkit.FormatText,
ValidationMode: promptkit.ValidationBasic,
RepairAttempts: 4,
},
},
{
name: "none validation with repair attempts",
contract: promptkit.OutputContract{
Format: promptkit.FormatText,
ValidationMode: promptkit.ValidationNone,
RepairAttempts: 1,
},
},
} {
t.Run(tc.name, func(t *testing.T) {
req := promptkit.RunRequest{PromptID: "prompt", Validation: &tc.contract}
prepared, err := engine.Prepare(context.Background(), req)
if prepared != nil {
t.Fatalf("expected no partial prepared run, got %+v", prepared)
}
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("prepare error = %v, want ErrInvalidRequest", err)
}
prepared, err := engine.Prepare(context.Background(), req)
if prepared != nil {
t.Fatalf("expected no partial prepared run, got %+v", prepared)
}
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("prepare error = %v, want ErrInvalidRequest", err)
}
preparedExecution, err := engine.PrepareExecution(context.Background(), req)
if preparedExecution != nil {
t.Fatalf("expected no partial prepared execution, got %+v", preparedExecution)
}
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("prepare execution error = %v, want ErrInvalidRequest", err)
preparedExecution, err := engine.PrepareExecution(context.Background(), req)
if preparedExecution != nil {
t.Fatalf("expected no partial prepared execution, got %+v", preparedExecution)
}
if !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("prepare execution error = %v, want ErrInvalidRequest", err)
}
})
}
}

View File

@@ -256,6 +256,50 @@ func TestPreparedExecutionLifecycleAndEngineBinding(t *testing.T) {
}
}
func TestPreparedExecutionRepairsEmptyBasicOutput(t *testing.T) {
client := &preparedRecordingClient{responses: []*promptkit.GenerateResponse{
{
Content: "",
Usage: promptkit.TokenUsage{PromptTokens: 3, CompletionTokens: 5, TotalTokens: 8},
},
{
Content: "Corrected summary.",
Usage: promptkit.TokenUsage{PromptTokens: 7, CompletionTokens: 11, TotalTokens: 18},
},
}}
engine := newPreparedContractEngine(t, client, "Summarize the source.")
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{
PromptID: "prepared",
Validation: &promptkit.OutputContract{
Format: promptkit.FormatMarkdown,
ValidationMode: promptkit.ValidationBasic,
RepairAttempts: 1,
},
})
if err != nil {
t.Fatalf("prepare execution: %v", err)
}
details := prepared.Details()
result, err := engine.RunPrepared(context.Background(), prepared)
if err != nil {
t.Fatalf("run prepared: %v", err)
}
if result.RawOutput != "Corrected summary." || result.Validation.Status != promptkit.ValidationPassed ||
result.Validation.RepairAttempts != 1 || result.Usage != (promptkit.TokenUsage{PromptTokens: 10, CompletionTokens: 16, TotalTokens: 26}) {
t.Fatalf("repaired result = %+v", result)
}
requests := client.snapshot()
if len(requests) != 2 || len(requests[1].Prompt.Messages) != len(details.Messages)+1 ||
!reflect.DeepEqual(requests[1].Prompt.Messages[:len(details.Messages)], details.Messages) ||
requests[1].Prompt.Messages[len(requests[1].Prompt.Messages)-1].Role != "user" {
t.Fatalf("prepared repair requests = %#v", requests)
}
if _, err := engine.RunPrepared(context.Background(), prepared); !errors.Is(err, promptkit.ErrInvalidRequest) {
t.Fatalf("second RunPrepared error = %v, want ErrInvalidRequest", err)
}
}
func TestPreparedExecutionConcurrentClaimAllowsOneGeneration(t *testing.T) {
release := make(chan struct{})
client := &preparedRecordingClient{
@@ -671,12 +715,13 @@ func (r *mutablePreparedArtifactReader) callCount() int {
}
type preparedRecordingClient struct {
mu sync.Mutex
response *promptkit.GenerateResponse
err error
requests []promptkit.GenerateRequest
started chan struct{}
release <-chan struct{}
mu sync.Mutex
response *promptkit.GenerateResponse
responses []*promptkit.GenerateResponse
err error
requests []promptkit.GenerateRequest
started chan struct{}
release <-chan struct{}
}
func (c *preparedRecordingClient) Generate(
@@ -700,6 +745,12 @@ func (c *preparedRecordingClient) Generate(
if c.err != nil {
return nil, c.err
}
if len(c.responses) > 0 {
if index := len(c.requests) - 1; index < len(c.responses) {
return c.responses[index], nil
}
return nil, fmt.Errorf("no response configured for generation %d", len(c.requests))
}
return c.response, nil
}

View File

@@ -844,7 +844,7 @@ func TestBackendExtraParamsAreDeeplyCopiedAtConstructionAndLookup(t *testing.T)
}
}
func TestEngineValidationIsSinglePass(t *testing.T) {
func TestEngineValidationWithZeroRepairBudgetIsSinglePass(t *testing.T) {
client := &fakeLLMClient{
response: &promptkit.GenerateResponse{Content: "not-json"},
}
@@ -859,7 +859,7 @@ func TestEngineValidationIsSinglePass(t *testing.T) {
Validation: &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSON,
RepairAttempts: 3,
RepairAttempts: 0,
},
})
if err != nil {
@@ -874,6 +874,87 @@ func TestEngineValidationIsSinglePass(t *testing.T) {
}
}
func TestEngineRunRepairsJSONSchemaOutput(t *testing.T) {
client := &fakeLLMClient{responses: []*promptkit.GenerateResponse{
{
Content: "{}",
Usage: promptkit.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5, CachedTokens: 7, CacheWriteTokens: 11},
},
{
Content: `{"events":[{"title":"Repaired event"}]}`,
Usage: promptkit.TokenUsage{PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19, CachedTokens: 23, CacheWriteTokens: 29},
},
}}
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: frameworkStructuredEventsPromptID,
SessionID: " repair-session ",
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
Validation: &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSONSchema,
SchemaPath: "structured_events.schema.json",
RepairAttempts: 1,
},
})
if err != nil {
t.Fatalf("run: %v", err)
}
if result.RawOutput != client.responses[1].Content || result.Validation.Status != promptkit.ValidationPassed ||
result.Validation.RepairAttempts != 1 {
t.Fatalf("repaired result = %+v", result)
}
wantUsage := promptkit.TokenUsage{PromptTokens: 15, CompletionTokens: 20, TotalTokens: 24, CachedTokens: 30, CacheWriteTokens: 40}
if result.Usage != wantUsage {
t.Fatalf("usage = %+v, want %+v", result.Usage, wantUsage)
}
if len(client.requests) != 2 {
t.Fatalf("generation calls = %d, want 2", len(client.requests))
}
initial, repaired := client.requests[0], client.requests[1]
if initial.Prompt.SessionID != "repair-session" || repaired.Prompt.SessionID != initial.Prompt.SessionID ||
!reflect.DeepEqual(repaired.Target, initial.Target) || repaired.TargetPresence != initial.TargetPresence ||
!reflect.DeepEqual(repaired.StructuredOutput, initial.StructuredOutput) {
t.Fatalf("generation request state drifted: initial=%+v repaired=%+v", initial, repaired)
}
if initial.StructuredOutput == nil || initial.StructuredOutput.JSONSchema == nil {
t.Fatalf("expected structured output on initial request: %+v", initial)
}
}
func TestEngineRunReturnsFinalResultAfterRepairExhaustion(t *testing.T) {
client := &fakeLLMClient{responses: []*promptkit.GenerateResponse{
{Content: "not-json", Usage: promptkit.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5}},
{Content: "still-not-json", Usage: promptkit.TokenUsage{PromptTokens: 7, CompletionTokens: 11, TotalTokens: 18}},
}}
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
result, err := engine.Run(context.Background(), promptkit.RunRequest{
PromptID: frameworkMarkdownSummaryPromptID,
Inputs: map[string]promptkit.ArtifactRef{
"transcript": promptkit.Inline("Rin opens the gate."),
"glossary": promptkit.Inline("gate: A guarded passage."),
},
Validation: &promptkit.OutputContract{
Format: promptkit.FormatJSON,
ValidationMode: promptkit.ValidationJSON,
RepairAttempts: 1,
},
})
if err != nil || result == nil {
t.Fatalf("run = (%+v, %v), want exhausted result", result, err)
}
if result.RawOutput != "still-not-json" || result.Validation.Status != promptkit.ValidationFailed ||
result.Validation.RepairAttempts != 1 || len(result.Validation.Errors) == 0 ||
result.Usage != (promptkit.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23}) {
t.Fatalf("exhausted result = %+v", result)
}
}
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}

View File

@@ -565,8 +565,8 @@ type ExecutionTargetPresence struct {
// JSON representation.
//
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
// does not merge fields. The public Engine validates generated output once and
// does not install an output repairer.
// does not merge fields. The public Engine performs bounded correction after a
// failed eligible validation when RepairAttempts is positive.
type OutputContract struct {
// Format selects generated artifact metadata. An empty value in a non-nil
// request replacement defaults to FormatText.
@@ -577,9 +577,9 @@ type OutputContract struct {
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
// ignored by other modes.
SchemaPath string `json:"schema_path"`
// RepairAttempts is a non-negative requested repair limit. Zero requests no
// repairs. The public Engine performs no repairs even when this value is
// positive, so its runs report zero attempts used.
// RepairAttempts is an additional generation-call budget from zero through
// three. Zero is single-pass. A positive value is valid only with basic,
// json, or json_schema validation.
RepairAttempts int `json:"repair_attempts"`
}
@@ -595,8 +595,8 @@ type ValidationResult struct {
Errors []string `json:"errors,omitempty"`
// SchemaPath is the effective schema path for JSON Schema validation.
SchemaPath string `json:"schema_path,omitempty"`
// RepairAttempts is the number of repairs actually attempted. It is always
// zero for the public Engine.
// RepairAttempts is the number of corrective generation calls actually
// started for this result.
RepairAttempts int `json:"repair_attempts"`
// IsValid is true for ValidationPassed and ValidationSkipped and false for
// ValidationFailed.
@@ -714,9 +714,8 @@ type GenerateRequest struct {
// GenerateResponse is returned by an injected LLM client and has a stable JSON
// representation.
type GenerateResponse struct {
// Content is the generated output. It must be non-empty when using the
// built-in client; injected clients may return empty content for Promptkit
// validation to classify.
// Content is the generated output. It may be explicitly empty; Promptkit
// applies the effective output contract to classify it.
Content string `json:"content"`
// Usage is the client's token accounting.
Usage TokenUsage `json:"usage"`