Compare commits
10 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 2e76003fd5 | |||
| 36ce5a5099 | |||
| 465dc1389d | |||
| ae6f1a9865 | |||
| ee99dc9478 | |||
| 00ee5893e9 | |||
| 64d1cffd89 | |||
| f9e8afa2c3 | |||
| e827631d8c | |||
| 115fe8ba58 |
@@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve.
|
|||||||
|
|
||||||
## Release Guidance
|
## Release Guidance
|
||||||
|
|
||||||
Consumers upgrading from `v0.6.0` to `v0.7.0` should read the
|
Consumers upgrading from `v0.7.0` to `v0.8.0` should read the
|
||||||
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
|
[v0.8.0 changelog and migration guide](docs/releases/v0.8.0.md).
|
||||||
|
|
||||||
Earlier adopters can consult the
|
Earlier adopters can consult the
|
||||||
|
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
|
||||||
|
|
||||||
|
Consumers upgrading from `v0.5.0` to `v0.6.0` can consult the
|
||||||
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
|
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
|
||||||
|
|
||||||
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the
|
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the
|
||||||
|
|||||||
@@ -173,6 +173,29 @@ semantics. The
|
|||||||
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
[OpenAI-compatible integration contract](../integrations/openai-compatible-chat.md)
|
||||||
owns the built-in client's outbound HTTP behavior.
|
owns the built-in client's outbound HTTP behavior.
|
||||||
|
|
||||||
|
### Repair A Structured Result
|
||||||
|
|
||||||
|
Set a small additional-call budget when a structurally invalid result can be
|
||||||
|
corrected automatically:
|
||||||
|
|
||||||
|
```go
|
||||||
|
request.Validation = &promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatJSON,
|
||||||
|
ValidationMode: promptkit.ValidationJSONSchema,
|
||||||
|
SchemaPath: "events.schema.json",
|
||||||
|
RepairAttempts: 1,
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Each repair attempt is another model call, so it can increase latency and
|
||||||
|
usage; `RunResult.Usage` is cumulative and `Validation.RepairAttempts` reports
|
||||||
|
calls actually started. Exhaustion still returns the final failed validation
|
||||||
|
result. `basic` validation can also repair an empty candidate, but structural
|
||||||
|
validity is not evidence of factual or domain correctness. See the
|
||||||
|
[output-contract format reference](../formats.md#output-contract) and
|
||||||
|
[`OutputContract` GoDoc](../../types.go) for the exact budget and eligibility
|
||||||
|
rules.
|
||||||
|
|
||||||
## Inputs, Profiles, And Overrides
|
## Inputs, Profiles, And Overrides
|
||||||
|
|
||||||
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
|
Use `File`, `Inline`, or `InlineWithURI` to construct request inputs. A request
|
||||||
|
|||||||
@@ -126,7 +126,7 @@ outbound integration determines its wire representation.
|
|||||||
| `format` | yes | `text`, `markdown`, or `json`. |
|
| `format` | yes | `text`, `markdown`, or `json`. |
|
||||||
| `validation_mode` | yes | `none`, `basic`, `json`, or `json_schema`. |
|
| `validation_mode` | yes | `none`, `basic`, `json`, or `json_schema`. |
|
||||||
| `schema_path` | for `json_schema` | Path to a schema in the configured schema source. |
|
| `schema_path` | for `json_schema` | Path to a schema in the configured schema source. |
|
||||||
| `repair_attempts` | no | Integer zero or greater; omitted means zero. |
|
| `repair_attempts` | no | Integer from zero through three; omitted means zero. A positive value requires `basic`, `json`, or `json_schema` validation. |
|
||||||
|
|
||||||
The validation modes behave as follows:
|
The validation modes behave as follows:
|
||||||
|
|
||||||
@@ -136,9 +136,15 @@ The validation modes behave as follows:
|
|||||||
- `json_schema` requires valid JSON that satisfies the selected schema.
|
- `json_schema` requires valid JSON that satisfies the selected schema.
|
||||||
|
|
||||||
`format` controls output artifact metadata. JSON Schema mode also supplies the
|
`format` controls output artifact metadata. JSON Schema mode also supplies the
|
||||||
schema to compatible model clients as structured-output metadata. The public
|
schema to compatible model clients as structured-output metadata. Plain `json`
|
||||||
engine does not install an output repairer, so its validation is single-pass
|
validation accepts every valid JSON value and does not request a provider-native
|
||||||
even when a positive `repair_attempts` value is present.
|
JSON-object constraint.
|
||||||
|
|
||||||
|
`repair_attempts` counts additional generation calls after a failed validation.
|
||||||
|
Zero is single-pass. With a positive eligible budget, Promptkit stops at the
|
||||||
|
first valid candidate. If the budget is exhausted, it returns the final
|
||||||
|
candidate and its complete failed validation result; generation and operational
|
||||||
|
validation failures remain errors. `none` never permits repair.
|
||||||
|
|
||||||
A request-level `OutputContract` replaces the complete prompt output contract.
|
A request-level `OutputContract` replaces the complete prompt output contract.
|
||||||
It does not merge individual fields. If its format is empty, Promptkit uses
|
It does not merge individual fields. If its format is empty, Promptkit uses
|
||||||
|
|||||||
@@ -67,7 +67,8 @@ The client conditionally includes:
|
|||||||
- non-empty `service_tier` and effective `reasoning_effort`; an explicitly
|
- non-empty `service_tier` and effective `reasoning_effort`; an explicitly
|
||||||
disabled reasoning setting is empty and therefore omitted; and
|
disabled reasoning setting is empty and therefore omitted; and
|
||||||
- `response_format` for JSON Schema structured output, including its name,
|
- `response_format` for JSON Schema structured output, including its name,
|
||||||
strict flag, and schema document.
|
strict flag, and schema document. Plain JSON validation does not add an
|
||||||
|
object-only response constraint.
|
||||||
|
|
||||||
The engine resolves backend, profile, and request extra-parameter maps by
|
The engine resolves backend, profile, and request extra-parameter maps by
|
||||||
whole-map replacement rather than key merging. The resulting effective map is
|
whole-map replacement rather than key merging. The resulting effective map is
|
||||||
@@ -99,10 +100,12 @@ drained.
|
|||||||
|
|
||||||
The bounded body must contain exactly one OpenAI-compatible JSON response
|
The bounded body must contain exactly one OpenAI-compatible JSON response
|
||||||
object followed only by JSON whitespace and EOF. The client returns the first
|
object followed only by JSON whitespace and EOF. The client returns the first
|
||||||
choice's non-empty message content and maps prompt, completion, total, cached,
|
choice's explicitly present string message content, including an empty or
|
||||||
and cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
|
whitespace-only string, and maps prompt, completion, total, cached, and
|
||||||
data, a second JSON value, absent choices, empty first-choice content, and size
|
cache-write token counts. Invalid or truncated JSON, trailing non-whitespace
|
||||||
overflow are malformed responses and return no partial result.
|
data, a second JSON value, absent choices, missing content, `null` content,
|
||||||
|
non-string content, and size overflow are malformed responses and return no
|
||||||
|
partial result.
|
||||||
|
|
||||||
For a non-2xx status, Promptkit recognizes one JSON document with a top-level
|
For a non-2xx status, Promptkit recognizes one JSON document with a top-level
|
||||||
object-valued `error` member. Its optional `message` and `type` fields must be
|
object-valued `error` member. Its optional `message` and `type` fields must be
|
||||||
|
|||||||
@@ -89,6 +89,12 @@ truncation, malformed JSON, trailing data, and a second value are malformed
|
|||||||
responses with no partial result or provider content in the error. Every body
|
responses with no partial result or provider content in the error. Every body
|
||||||
is closed, and an unbounded oversized stream is not drained.
|
is closed, and an unbounded oversized stream is not drained.
|
||||||
|
|
||||||
|
After framing succeeds, the first choice must contain an explicitly present
|
||||||
|
string `message.content`. The string is returned exactly, including empty or
|
||||||
|
whitespace-only content. Missing choices, missing or `null` content, and
|
||||||
|
non-string content are malformed responses. Output validation and correction
|
||||||
|
eligibility remain outside this package.
|
||||||
|
|
||||||
For a non-success response, `ProviderHTTPError` retains the HTTP status and
|
For a non-success response, `ProviderHTTPError` retains the HTTP status and
|
||||||
only normalized detail from the bounded recognized envelope. It retains
|
only normalized detail from the bounded recognized envelope. It retains
|
||||||
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;
|
`ErrUnexpectedStatus` through unwrapping. The client owns response closure;
|
||||||
|
|||||||
@@ -11,7 +11,7 @@ contributor workflow and validation.
|
|||||||
|
|
||||||
| Component | Implemented responsibility | References |
|
| Component | Implemented responsibility | References |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, and engine-local profile-source assembly including application fallbacks. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
| Root `promptkit` package | Provides the supported engine facade, source, backend-registration, and injection options, public request, result, prompt-inspection, and profile-inspection values, opaque prepared-execution handles, profile construction, extension interfaces, value conversion, redacted formatting, typed capacity and generation error mapping, engine-local profile-source assembly including application fallbacks, and bounded output-repair assembly. | [Package GoDoc](../../doc.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine assembly](../../engine.go) |
|
||||||
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
| `examples/go-library/prepare` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, and `Prepare`. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, in-memory profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||||
| `internal/backend` | Constructs each engine's immutable registry from the maintained built-in definitions and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
| `internal/backend` | Constructs each engine's immutable registry from the maintained built-in definitions and consumer additions, validates and defensively copies definitions through the shared JSON-value package, and consumes the LLM-owned OpenAI-compatible reserved request-field rule. | [Backend registry](../../internal/backend/registry.go) |
|
||||||
@@ -27,7 +27,7 @@ contributor workflow and validation.
|
|||||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||||
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and optional repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
| `internal/usecase` | Resolves prompt definitions and hashes, profiles, backends, and targets for exact inspection and request settings for preparation, and coordinates ordinary execution and one-attempt prepared execution across internal sources, rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||||
|
|
||||||
The root package assembles these internal components without exposing their
|
The root package assembles these internal components without exposing their
|
||||||
representations. Consumers depend only on the root facade.
|
representations. Consumers depend only on the root facade.
|
||||||
|
|||||||
@@ -23,8 +23,10 @@ built-in backend and validated consumer additions, one engine-local run
|
|||||||
admitter, and a model client wrapped by the same capacity manager. Validation
|
admitter, and a model client wrapped by the same capacity manager. Validation
|
||||||
plans and provider-facing schema metadata come from the validator's preparation
|
plans and provider-facing schema metadata come from the validator's preparation
|
||||||
interface.
|
interface.
|
||||||
An output repairer can be injected internally, but the ordinary runner
|
The root engine supplies one default output repairer through the explicit
|
||||||
constructor does not enable one.
|
runner constructor, using the same capacity-wrapped client as initial
|
||||||
|
generation. The no-repair runner constructor remains available for focused
|
||||||
|
internal callers and tests.
|
||||||
|
|
||||||
Each invocation carries its state in request, prepared-run, and result values.
|
Each invocation carries its state in request, prepared-run, and result values.
|
||||||
The runner has no durable run or session store.
|
The runner has no durable run or session store.
|
||||||
@@ -125,8 +127,8 @@ admitter is an internal unlimited fallback. After successful admission, `Run`
|
|||||||
immediately defers the returned release function, performs the completion
|
immediately defers the returned release function, performs the completion
|
||||||
phase, makes one initial generation call, builds the named output artifact,
|
phase, makes one initial generation call, builds the named output artifact,
|
||||||
and validates that artifact with the plan compiled during completion. Invalid
|
and validates that artifact with the plan compiled during completion. Invalid
|
||||||
generated content remains a validation result; an inability to perform
|
generated content remains a validation result; an inability to generate or
|
||||||
validation is an operational error.
|
perform validation is an operational error.
|
||||||
|
|
||||||
Validation preparation and execution honor cancellation at every
|
Validation preparation and execution honor cancellation at every
|
||||||
Promptkit-controlled boundary and do not publish a partial plan or result.
|
Promptkit-controlled boundary and do not publish a partial plan or result.
|
||||||
@@ -142,17 +144,25 @@ serializing preparation or validation behind the active-generation limit.
|
|||||||
The wrapped model client separately acquires a FIFO active permit only around
|
The wrapped model client separately acquires a FIFO active permit only around
|
||||||
each actual generation call.
|
each actual generation call.
|
||||||
|
|
||||||
When an internal repairer is present, a JSON or JSON Schema content failure can
|
After a failed `basic`, JSON, or JSON Schema validation with a positive frozen
|
||||||
trigger bounded repair attempts. Repair receives the effective execution
|
budget, the installed repairer can make a bounded corrective call. Each request
|
||||||
target, explicit numeric-presence bits, credential, backend identity, session
|
starts with a fresh copy of the complete original rendered messages, includes
|
||||||
ID, validation errors, prior output, and structured-output specification. One
|
only the latest nonempty candidate as an assistant message, and appends one
|
||||||
request constructor supplies those common fields to initial and repair
|
corrective user message. Empty candidates omit that assistant message. The
|
||||||
generation while their rendered prompts remain intentionally distinct. The
|
correction carries validation diagnostics as JSON data bounded to 64 KiB; the
|
||||||
default repairer uses the same wrapped client as initial generation, so each
|
full diagnostics remain in the validation result.
|
||||||
repair reacquires the selected backend's active permit while remaining inside
|
|
||||||
its original admission lease. Repair never performs a second bounded
|
Repair receives the effective execution target, explicit numeric-presence bits,
|
||||||
admission, and repaired outputs use the operation's existing validation plan.
|
credential, backend identity, session ID, and structured-output specification.
|
||||||
This capability remains internal and is not a public option.
|
The same request constructor supplies those common fields to initial and repair
|
||||||
|
generation. The default repairer uses the same wrapped client as initial
|
||||||
|
generation, so each repair reacquires the selected backend's active permit
|
||||||
|
while remaining inside its original admission lease. Repair never performs a
|
||||||
|
second bounded admission, and repaired outputs use the operation's existing
|
||||||
|
validation plan. The runner stops at the first valid candidate, sums completed
|
||||||
|
generation usage, reports calls actually started, and returns the final failed
|
||||||
|
validation result on exhaustion. A repair generation failure follows the
|
||||||
|
ordinary generation-error category rather than becoming a validation error.
|
||||||
|
|
||||||
A successful result includes the output artifact and raw output, validation
|
A successful result includes the output artifact and raw output, validation
|
||||||
state, effective session ID, prompt and rendered-prompt hashes, selected
|
state, effective session ID, prompt and rendered-prompt hashes, selected
|
||||||
|
|||||||
109
docs/releases/v0.8.0.md
Normal file
109
docs/releases/v0.8.0.md
Normal file
@@ -0,0 +1,109 @@
|
|||||||
|
# Promptkit v0.8.0
|
||||||
|
|
||||||
|
This supplemental changelog and migration guide summarizes the consumer-facing
|
||||||
|
changes from `v0.7.0` to `v0.8.0`. The annotated `v0.8.0` tag is the
|
||||||
|
authoritative release record. Exact current contracts belong to the linked
|
||||||
|
GoDoc and durable documentation.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
`v0.8.0` activates Promptkit's bounded output-repair workflow:
|
||||||
|
|
||||||
|
- failed nonempty-text, JSON, and JSON Schema validation can make a limited
|
||||||
|
number of corrective model calls;
|
||||||
|
- corrective calls preserve the original rendered conversation, effective
|
||||||
|
target, session, structured-output contract, and backend capacity policy;
|
||||||
|
- results report cumulative usage and the number of corrective calls actually
|
||||||
|
made; and
|
||||||
|
- explicitly empty OpenAI-compatible response content now reaches output
|
||||||
|
validation instead of being classified as a malformed provider envelope.
|
||||||
|
|
||||||
|
## Compatibility
|
||||||
|
|
||||||
|
This release adds no public declarations or fields and removes none. Existing
|
||||||
|
source code remains source-compatible.
|
||||||
|
|
||||||
|
The behavior of the existing `OutputContract.RepairAttempts` field and prompt
|
||||||
|
YAML `repair_attempts` field has changed. A positive value now authorizes real
|
||||||
|
additional model calls after eligible validation failures; earlier releases
|
||||||
|
accepted the field but the public engine remained single-pass. Consumers that
|
||||||
|
set a positive value should expect additional latency, token usage, and
|
||||||
|
provider cost when repair is needed.
|
||||||
|
|
||||||
|
Repair budgets must now be between zero and three. A positive budget requires
|
||||||
|
`basic`, `json`, or `json_schema` validation. Values above three and a positive
|
||||||
|
budget paired with `none` are invalid contracts rather than ignored settings.
|
||||||
|
|
||||||
|
An explicitly present empty or whitespace-only string returned by the built-in
|
||||||
|
OpenAI-compatible client is now a completed generation candidate. `none`
|
||||||
|
validation permits it, while `basic`, `json`, and `json_schema` classify it
|
||||||
|
under their ordinary validation rules and may repair it when configured.
|
||||||
|
Missing, `null`, or non-string content remains a malformed provider response.
|
||||||
|
|
||||||
|
## Upgrade
|
||||||
|
|
||||||
|
Update the module dependency with:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
go get gitea.maximumdirect.net/eric/promptkit@v0.8.0
|
||||||
|
go mod tidy
|
||||||
|
```
|
||||||
|
|
||||||
|
Review every prompt definition and request override that sets a positive repair
|
||||||
|
budget. Use zero or omit the field to retain single-pass execution. Ensure each
|
||||||
|
positive budget is no greater than three and uses an eligible validation mode,
|
||||||
|
then run the consuming project's ordinary and race-enabled tests.
|
||||||
|
|
||||||
|
## Bounded Output Repair
|
||||||
|
|
||||||
|
`repair_attempts` counts corrective calls in addition to the initial model
|
||||||
|
call. Promptkit validates each completed candidate, stops at the first valid
|
||||||
|
one, and never exceeds the configured bound. If every candidate remains
|
||||||
|
invalid, the run completes successfully with the final candidate and its
|
||||||
|
failed validation result rather than returning an operational error.
|
||||||
|
|
||||||
|
Each correction starts from the original rendered messages and includes only
|
||||||
|
the latest invalid candidate and latest validation diagnostics. JSON Schema
|
||||||
|
mode retains the provider-native structured-output request as its first line of
|
||||||
|
defense. Promptkit performs only deterministic structural validation; a valid
|
||||||
|
response is not necessarily factual or correct for an application's domain.
|
||||||
|
|
||||||
|
Usage in the final result is cumulative across the initial response and every
|
||||||
|
completed corrective response. `ValidationResult.RepairAttempts` reports the
|
||||||
|
number of corrective calls actually made. Corrective generation failures use
|
||||||
|
the same public generation-error categories and structured provider details as
|
||||||
|
an initial generation failure.
|
||||||
|
|
||||||
|
See the [output-contract format reference](../formats.md#output-contract), the
|
||||||
|
[consumer repair example](../consumers/pkg-promptkit.md#repair-a-structured-result),
|
||||||
|
and the [`OutputContract` and `ValidationResult` GoDoc](../../types.go) for the
|
||||||
|
current contracts.
|
||||||
|
|
||||||
|
## Explicit Empty Content
|
||||||
|
|
||||||
|
The built-in OpenAI-compatible client now distinguishes an explicitly present
|
||||||
|
empty string from a missing or malformed `content` field. This aligns built-in
|
||||||
|
and injected clients by letting the selected output contract decide whether an
|
||||||
|
empty candidate is acceptable, invalid, or eligible for repair.
|
||||||
|
|
||||||
|
See the
|
||||||
|
[OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling)
|
||||||
|
for the exact envelope behavior.
|
||||||
|
|
||||||
|
## Public API Changes
|
||||||
|
|
||||||
|
None. This release activates and tightens the documented behavior of existing
|
||||||
|
fields.
|
||||||
|
|
||||||
|
## Consumer Action
|
||||||
|
|
||||||
|
- Remove or set `repair_attempts` to zero where execution must remain
|
||||||
|
single-pass.
|
||||||
|
- Keep every positive repair budget at three or fewer and pair it with
|
||||||
|
`basic`, `json`, or `json_schema` validation.
|
||||||
|
- Account for additional latency, usage, and provider cost when enabling
|
||||||
|
repair.
|
||||||
|
- Continue checking the returned validation status because bounded repair can
|
||||||
|
exhaust without producing a valid candidate.
|
||||||
|
- Review workflows that previously treated explicit empty provider content as
|
||||||
|
a generation error.
|
||||||
@@ -38,28 +38,8 @@ consumers.
|
|||||||
|
|
||||||
## Ideas
|
## Ideas
|
||||||
|
|
||||||
### Public bounded output repair
|
No ideas are currently awaiting selection. Active feature work belongs in its
|
||||||
|
focused roadmap rather than this catalog.
|
||||||
Promptkit should make its bounded output-repair capability available through
|
|
||||||
the public engine. A consumer should be able to request a limited number of
|
|
||||||
corrective generation attempts when JSON or JSON Schema output fails content
|
|
||||||
validation, without having to reproduce Promptkit's generation, validation,
|
|
||||||
capacity, and result-accounting orchestration.
|
|
||||||
|
|
||||||
- Repair is validation recovery, not a general provider retry, failover, or
|
|
||||||
backoff policy. Transport failures, cancellation, and operational schema or
|
|
||||||
validation errors must retain their ordinary error behavior.
|
|
||||||
- Repair must stop after the first valid result or the configured attempt
|
|
||||||
bound. Exhausting the bound should preserve the final invalid result and its
|
|
||||||
validation diagnostics rather than inventing success.
|
|
||||||
- Initial generation and every repair attempt must use the same resolved
|
|
||||||
backend, effective execution settings and presence semantics, session,
|
|
||||||
credential boundary, structured-output contract, and backend-capacity
|
|
||||||
policy.
|
|
||||||
- Results should report the number of repair attempts and cumulative usage for
|
|
||||||
every model call made by the run.
|
|
||||||
- Ordinary and prepared execution should expose coherent behavior, including
|
|
||||||
cancellation, frozen prepared state, error identity, and capacity lifetime.
|
|
||||||
|
|
||||||
## Entry Format
|
## Entry Format
|
||||||
|
|
||||||
|
|||||||
23
engine.go
23
engine.go
@@ -430,7 +430,7 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
return &Engine{
|
return &Engine{
|
||||||
runner: usecase.NewRunner(
|
runner: usecase.NewRunnerWithRepairer(
|
||||||
promptDefs,
|
promptDefs,
|
||||||
profiles,
|
profiles,
|
||||||
backendRegistry,
|
backendRegistry,
|
||||||
@@ -438,6 +438,7 @@ func NewEngine(cfg Config, opts ...Option) (*Engine, error) {
|
|||||||
prompt.NewGoRenderer(),
|
prompt.NewGoRenderer(),
|
||||||
llmClient,
|
llmClient,
|
||||||
validator,
|
validator,
|
||||||
|
usecase.NewDefaultOutputRepairer(llmClient),
|
||||||
capacityManager,
|
capacityManager,
|
||||||
),
|
),
|
||||||
}, nil
|
}, nil
|
||||||
@@ -639,10 +640,12 @@ func (e *Engine) PrepareExecution(ctx context.Context, req RunRequest) (*Prepare
|
|||||||
// generated output.
|
// generated output.
|
||||||
//
|
//
|
||||||
// A content-validation failure is a successful run whose
|
// A content-validation failure is a successful run whose
|
||||||
// RunResult.Validation has Status ValidationFailed. An inability to perform
|
// RunResult.Validation has Status ValidationFailed. When its output contract
|
||||||
// validation returns an error matching ErrValidation and no partial result.
|
// has a positive repair budget, a failed eligible validation can make bounded
|
||||||
// The public Engine does not perform output repair, so validation is
|
// additional model calls and stops at the first valid candidate. Exhaustion
|
||||||
// single-pass even when OutputContract.RepairAttempts is positive.
|
// returns the final failed validation result with cumulative usage and actual
|
||||||
|
// repair attempts. An inability to generate or validate returns an error and
|
||||||
|
// no partial result.
|
||||||
//
|
//
|
||||||
// Run can return every error category documented by [Engine.Prepare], plus
|
// Run can return every error category documented by [Engine.Prepare], plus
|
||||||
// ErrCapacityExceeded and ErrLLMGenerate. An engine admission rejection is
|
// ErrCapacityExceeded and ErrLLMGenerate. An engine admission rejection is
|
||||||
@@ -686,17 +689,17 @@ func (e *Engine) Run(ctx context.Context, req RunRequest) (*RunResult, error) {
|
|||||||
//
|
//
|
||||||
// The supplied context governs this execution attempt independently of the
|
// The supplied context governs this execution attempt independently of the
|
||||||
// preparation context. It covers credential revalidation, admission,
|
// preparation context. It covers credential revalidation, admission,
|
||||||
// generation, validation, and any internal repair. Result timing begins after
|
// generation, validation, and any bounded output repair. Result timing begins
|
||||||
// the claim and excludes preparation and consumer-held delay.
|
// after the claim and excludes preparation and consumer-held delay.
|
||||||
//
|
//
|
||||||
// RunPrepared can return ErrInvalidRequest, ErrAPIKeyEnvMissing,
|
// RunPrepared can return ErrInvalidRequest, ErrAPIKeyEnvMissing,
|
||||||
// ErrCapacityExceeded, ErrLLMGenerate, or ErrValidation as applicable while
|
// ErrCapacityExceeded, ErrLLMGenerate, or ErrValidation as applicable while
|
||||||
// preserving documented collaborator and context identities. An engine
|
// preserving documented collaborator and context identities. An engine
|
||||||
// admission rejection is discoverable as [CapacityError] and still matches
|
// admission rejection is discoverable as [CapacityError] and still matches
|
||||||
// ErrCapacityExceeded. A built-in OpenAI-compatible non-2xx response is
|
// ErrCapacityExceeded. A built-in OpenAI-compatible non-2xx response is
|
||||||
// discoverable as [GenerationError]. A completed content-validation rejection
|
// discoverable as [GenerationError]. A completed content-validation rejection,
|
||||||
// is returned in RunResult, not as an operational error. An operational error
|
// including repair exhaustion, is returned in RunResult, not as an operational
|
||||||
// returns no partial RunResult.
|
// error. An operational error returns no partial RunResult.
|
||||||
func (e *Engine) RunPrepared(ctx context.Context, prepared *PreparedExecution) (*RunResult, error) {
|
func (e *Engine) RunPrepared(ctx context.Context, prepared *PreparedExecution) (*RunResult, error) {
|
||||||
if e == nil || e.runner == nil {
|
if e == nil || e.runner == nil {
|
||||||
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
return nil, fmt.Errorf("%w: engine is nil", ErrInvalidConfig)
|
||||||
|
|||||||
@@ -3486,9 +3486,10 @@ extra_params:
|
|||||||
}
|
}
|
||||||
|
|
||||||
type fakeLLMClient struct {
|
type fakeLLMClient struct {
|
||||||
response *promptkit.GenerateResponse
|
response *promptkit.GenerateResponse
|
||||||
err error
|
responses []*promptkit.GenerateResponse
|
||||||
requests []promptkit.GenerateRequest
|
err error
|
||||||
|
requests []promptkit.GenerateRequest
|
||||||
}
|
}
|
||||||
|
|
||||||
type recordingArtifactReader struct {
|
type recordingArtifactReader struct {
|
||||||
@@ -3664,5 +3665,12 @@ func (f *fakeLLMClient) Generate(_ context.Context, req promptkit.GenerateReques
|
|||||||
if f.err != nil {
|
if f.err != nil {
|
||||||
return nil, f.err
|
return nil, f.err
|
||||||
}
|
}
|
||||||
|
if len(f.responses) > 0 {
|
||||||
|
index := len(f.requests) - 1
|
||||||
|
if index >= len(f.responses) {
|
||||||
|
return nil, fmt.Errorf("no response configured for generation %d", index+1)
|
||||||
|
}
|
||||||
|
return f.responses[index], nil
|
||||||
|
}
|
||||||
return f.response, nil
|
return f.response, nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -39,6 +39,44 @@ func TestBuiltInGenerationError(t *testing.T) {
|
|||||||
assertGenerationError(t, err, http.StatusServiceUnavailable, "", "", "")
|
assertGenerationError(t, err, http.StatusServiceUnavailable, "", "", "")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestBuiltInRepairGenerationError(t *testing.T) {
|
||||||
|
const (
|
||||||
|
codeMarker = "repair-code-marker"
|
||||||
|
typeMarker = "repair-type-marker"
|
||||||
|
messageMarker = "repair-message-marker"
|
||||||
|
)
|
||||||
|
calls := 0
|
||||||
|
config := contractConfig(frameworkSchemaDir)
|
||||||
|
config.HTTPClient = &http.Client{Transport: roundTripFunc(func(*http.Request) (*http.Response, error) {
|
||||||
|
calls++
|
||||||
|
if calls == 1 {
|
||||||
|
body := `{"choices":[{"message":{"content":"not-json"}}]}`
|
||||||
|
return &http.Response{StatusCode: http.StatusOK, ContentLength: int64(len(body)), Body: io.NopCloser(strings.NewReader(body))}, nil
|
||||||
|
}
|
||||||
|
body := `{"error":{"code":"` + codeMarker + `","type":"` + typeMarker + `","message":"` + messageMarker + `"}}`
|
||||||
|
return &http.Response{StatusCode: http.StatusUnprocessableEntity, ContentLength: int64(len(body)), Body: io.NopCloser(strings.NewReader(body))}, nil
|
||||||
|
})}
|
||||||
|
engine, err := promptkit.NewEngine(config)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewEngine: %v", err)
|
||||||
|
}
|
||||||
|
req := generationErrorRunRequest()
|
||||||
|
req.Validation = &promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatJSON,
|
||||||
|
ValidationMode: promptkit.ValidationJSON,
|
||||||
|
RepairAttempts: 1,
|
||||||
|
}
|
||||||
|
|
||||||
|
result, err := engine.Run(context.Background(), req)
|
||||||
|
if result != nil {
|
||||||
|
t.Fatalf("Run result = %#v, want nil", result)
|
||||||
|
}
|
||||||
|
if calls != 2 {
|
||||||
|
t.Fatalf("provider calls = %d, want 2", calls)
|
||||||
|
}
|
||||||
|
assertGenerationError(t, err, http.StatusUnprocessableEntity, codeMarker, typeMarker, messageMarker)
|
||||||
|
}
|
||||||
|
|
||||||
func assertGenerationError(t *testing.T, err error, statusCode int, code, providerType, message string) {
|
func assertGenerationError(t *testing.T, err error, statusCode int, code, providerType, message string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
|
|||||||
@@ -6,6 +6,8 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
const maxOutputRepairAttempts = 3
|
||||||
|
|
||||||
// ValidateOutputContract validates source-neutral output-contract invariants.
|
// ValidateOutputContract validates source-neutral output-contract invariants.
|
||||||
func ValidateOutputContract(contract OutputContract) error {
|
func ValidateOutputContract(contract OutputContract) error {
|
||||||
switch contract.Format {
|
switch contract.Format {
|
||||||
@@ -26,5 +28,11 @@ func ValidateOutputContract(contract OutputContract) error {
|
|||||||
if contract.RepairAttempts < 0 {
|
if contract.RepairAttempts < 0 {
|
||||||
return errors.New("repair_attempts must be greater than or equal to 0")
|
return errors.New("repair_attempts must be greater than or equal to 0")
|
||||||
}
|
}
|
||||||
|
if contract.RepairAttempts > maxOutputRepairAttempts {
|
||||||
|
return fmt.Errorf("repair_attempts must be less than or equal to %d", maxOutputRepairAttempts)
|
||||||
|
}
|
||||||
|
if contract.ValidationMode == ValidationNone && contract.RepairAttempts > 0 {
|
||||||
|
return errors.New("repair_attempts requires basic, json, or json_schema validation")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -30,9 +30,28 @@ func TestValidateOutputContract(t *testing.T) {
|
|||||||
}},
|
}},
|
||||||
{name: "empty validation mode", change: func(c *OutputContract) { c.ValidationMode = "" }, wantErr: "validation mode"},
|
{name: "empty validation mode", change: func(c *OutputContract) { c.ValidationMode = "" }, wantErr: "validation mode"},
|
||||||
{name: "unsupported validation mode", change: func(c *OutputContract) { c.ValidationMode = ValidationMode("unknown") }, wantErr: "validation mode"},
|
{name: "unsupported validation mode", change: func(c *OutputContract) { c.ValidationMode = ValidationMode("unknown") }, wantErr: "validation mode"},
|
||||||
{name: "negative repair attempts", change: func(c *OutputContract) { c.RepairAttempts = -1 }, wantErr: "repair_attempts"},
|
{name: "negative repair attempts", change: func(c *OutputContract) {
|
||||||
|
c.ValidationMode = ValidationBasic
|
||||||
|
c.RepairAttempts = -1
|
||||||
|
}, wantErr: "repair_attempts"},
|
||||||
{name: "zero repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 0 }},
|
{name: "zero repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 0 }},
|
||||||
{name: "positive repair attempts", change: func(c *OutputContract) { c.RepairAttempts = 1 }},
|
{name: "one repair attempt", change: func(c *OutputContract) {
|
||||||
|
c.ValidationMode = ValidationBasic
|
||||||
|
c.RepairAttempts = 1
|
||||||
|
}},
|
||||||
|
{name: "maximum repair attempts", change: func(c *OutputContract) {
|
||||||
|
c.ValidationMode = ValidationJSON
|
||||||
|
c.RepairAttempts = 3
|
||||||
|
}},
|
||||||
|
{name: "too many repair attempts", change: func(c *OutputContract) {
|
||||||
|
c.ValidationMode = ValidationJSONSchema
|
||||||
|
c.SchemaPath = "schema.json"
|
||||||
|
c.RepairAttempts = 4
|
||||||
|
}, wantErr: "repair_attempts"},
|
||||||
|
{name: "none validation with repair attempts", change: func(c *OutputContract) {
|
||||||
|
c.ValidationMode = ValidationNone
|
||||||
|
c.RepairAttempts = 1
|
||||||
|
}, wantErr: "repair_attempts"},
|
||||||
{name: "json schema empty path", change: func(c *OutputContract) {
|
{name: "json schema empty path", change: func(c *OutputContract) {
|
||||||
c.ValidationMode = ValidationJSONSchema
|
c.ValidationMode = ValidationJSONSchema
|
||||||
c.SchemaPath = ""
|
c.SchemaPath = ""
|
||||||
|
|||||||
@@ -179,12 +179,12 @@ func (c *OpenAICompatibleClient) Generate(ctx context.Context, req domain.Genera
|
|||||||
return nil, fmt.Errorf("%w: no choices returned", ErrMalformedResponse)
|
return nil, fmt.Errorf("%w: no choices returned", ErrMalformedResponse)
|
||||||
}
|
}
|
||||||
content := wireResp.Choices[0].Message.Content
|
content := wireResp.Choices[0].Message.Content
|
||||||
if content == "" {
|
if content == nil {
|
||||||
return nil, fmt.Errorf("%w: first choice has empty message content", ErrMalformedResponse)
|
return nil, fmt.Errorf("%w: first choice has missing message content", ErrMalformedResponse)
|
||||||
}
|
}
|
||||||
|
|
||||||
return &domain.GenerateResponse{
|
return &domain.GenerateResponse{
|
||||||
Content: content,
|
Content: *content,
|
||||||
Usage: domain.TokenUsage{
|
Usage: domain.TokenUsage{
|
||||||
PromptTokens: wireResp.Usage.PromptTokens,
|
PromptTokens: wireResp.Usage.PromptTokens,
|
||||||
CompletionTokens: wireResp.Usage.CompletionTokens,
|
CompletionTokens: wireResp.Usage.CompletionTokens,
|
||||||
@@ -379,8 +379,8 @@ type openAICacheControl struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type openAIChatResponseMessage struct {
|
type openAIChatResponseMessage struct {
|
||||||
Role string `json:"role"`
|
Role string `json:"role"`
|
||||||
Content string `json:"content"`
|
Content *string `json:"content"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type openAIChatResponse struct {
|
type openAIChatResponse struct {
|
||||||
|
|||||||
@@ -765,6 +765,7 @@ func TestOpenAICompatibleClientResponseFraming(t *testing.T) {
|
|||||||
run func(*testing.T)
|
run func(*testing.T)
|
||||||
}{
|
}{
|
||||||
{name: "usage mapping", run: checkCacheUsageMapping},
|
{name: "usage mapping", run: checkCacheUsageMapping},
|
||||||
|
{name: "content presence", run: checkContentPresence},
|
||||||
{name: "common response failures", run: checkCommonResponseFailures},
|
{name: "common response failures", run: checkCommonResponseFailures},
|
||||||
{name: "successful response byte boundary", run: checkSuccessfulResponseByteBoundary},
|
{name: "successful response byte boundary", run: checkSuccessfulResponseByteBoundary},
|
||||||
{name: "continuing oversized response", run: checkContinuingOversizedResponse},
|
{name: "continuing oversized response", run: checkContinuingOversizedResponse},
|
||||||
@@ -776,6 +777,53 @@ func TestOpenAICompatibleClientResponseFraming(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func checkContentPresence(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
content string
|
||||||
|
}{
|
||||||
|
{name: "explicit empty string", content: ""},
|
||||||
|
{name: "whitespace string", content: " \n\t "},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
provider := newRecordingProvider(t)
|
||||||
|
provider.respond(http.StatusOK, `{
|
||||||
|
"choices": [{"message": {"content": `+strconv.Quote(tc.content)+`}}],
|
||||||
|
"usage": {
|
||||||
|
"prompt_tokens": 10,
|
||||||
|
"completion_tokens": 20,
|
||||||
|
"total_tokens": 30,
|
||||||
|
"prompt_tokens_details": {"cached_tokens": 4},
|
||||||
|
"cache_write_tokens": 5
|
||||||
|
}
|
||||||
|
}`)
|
||||||
|
client := newProviderClient(t, provider, OpenAICompatibleConfig{Model: "model"})
|
||||||
|
|
||||||
|
response, err := client.Generate(context.Background(), ordinaryGenerateRequest())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("generate: %v", err)
|
||||||
|
}
|
||||||
|
if response == nil {
|
||||||
|
t.Fatal("expected response")
|
||||||
|
}
|
||||||
|
if response.Content != tc.content {
|
||||||
|
t.Fatalf("content = %q, want %q", response.Content, tc.content)
|
||||||
|
}
|
||||||
|
if response.Usage != (domain.TokenUsage{
|
||||||
|
PromptTokens: 10,
|
||||||
|
CompletionTokens: 20,
|
||||||
|
TotalTokens: 30,
|
||||||
|
CachedTokens: 4,
|
||||||
|
CacheWriteTokens: 5,
|
||||||
|
}) {
|
||||||
|
t.Fatalf("usage = %+v", response.Usage)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func checkCacheUsageMapping(t *testing.T) {
|
func checkCacheUsageMapping(t *testing.T) {
|
||||||
provider := newRecordingProvider(t)
|
provider := newRecordingProvider(t)
|
||||||
provider.respond(http.StatusOK, `{
|
provider.respond(http.StatusOK, `{
|
||||||
@@ -1129,6 +1177,9 @@ func checkCommonResponseFailures(t *testing.T) {
|
|||||||
},
|
},
|
||||||
{name: "invalid JSON", statusCode: http.StatusOK, body: `{not valid json`, wantErr: ErrMalformedResponse},
|
{name: "invalid JSON", statusCode: http.StatusOK, body: `{not valid json`, wantErr: ErrMalformedResponse},
|
||||||
{name: "missing choices", statusCode: http.StatusOK, body: `{"choices": []}`, wantErr: ErrMalformedResponse},
|
{name: "missing choices", statusCode: http.StatusOK, body: `{"choices": []}`, wantErr: ErrMalformedResponse},
|
||||||
|
{name: "missing content", statusCode: http.StatusOK, body: `{"choices": [{"message": {}}]}`, wantErr: ErrMalformedResponse},
|
||||||
|
{name: "null content", statusCode: http.StatusOK, body: `{"choices": [{"message": {"content": null}}]}`, wantErr: ErrMalformedResponse},
|
||||||
|
{name: "non-string content", statusCode: http.StatusOK, body: `{"choices": [{"message": {"content": 1}}]}`, wantErr: ErrMalformedResponse},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, tc := range tests {
|
for _, tc := range tests {
|
||||||
|
|||||||
@@ -892,6 +892,38 @@ output:
|
|||||||
format: text
|
format: text
|
||||||
validation_mode: none
|
validation_mode: none
|
||||||
repair_attempts: -1
|
repair_attempts: -1
|
||||||
|
`,
|
||||||
|
wantErr: true,
|
||||||
|
wantDiagnostic: "repair_attempts",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "repair attempts above maximum",
|
||||||
|
definition: `
|
||||||
|
id: normalization-rule
|
||||||
|
version: "1"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: test
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: basic
|
||||||
|
repair_attempts: 4
|
||||||
|
`,
|
||||||
|
wantErr: true,
|
||||||
|
wantDiagnostic: "repair_attempts",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "none validation with repair attempts",
|
||||||
|
definition: `
|
||||||
|
id: normalization-rule
|
||||||
|
version: "1"
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: test
|
||||||
|
output:
|
||||||
|
format: text
|
||||||
|
validation_mode: none
|
||||||
|
repair_attempts: 1
|
||||||
`,
|
`,
|
||||||
wantErr: true,
|
wantErr: true,
|
||||||
wantDiagnostic: "repair_attempts",
|
wantDiagnostic: "repair_attempts",
|
||||||
|
|||||||
@@ -106,6 +106,8 @@ func TestRunnerPreparationRejectsInvalidOutputContractsBeforeCompletion(t *testi
|
|||||||
{name: "unsupported format", override: domain.OutputContract{Format: "binary", ValidationMode: domain.ValidationNone}},
|
{name: "unsupported format", override: domain.OutputContract{Format: "binary", ValidationMode: domain.ValidationNone}},
|
||||||
{name: "empty validation mode", override: domain.OutputContract{Format: domain.FormatText}},
|
{name: "empty validation mode", override: domain.OutputContract{Format: domain.FormatText}},
|
||||||
{name: "negative repair attempts", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone, RepairAttempts: -1}},
|
{name: "negative repair attempts", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone, RepairAttempts: -1}},
|
||||||
|
{name: "repair attempts above maximum", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationBasic, RepairAttempts: 4}},
|
||||||
|
{name: "none validation with repair attempts", override: domain.OutputContract{Format: domain.FormatText, ValidationMode: domain.ValidationNone, RepairAttempts: 1}},
|
||||||
{name: "json schema without path", override: domain.OutputContract{Format: domain.FormatJSON, ValidationMode: domain.ValidationJSONSchema}},
|
{name: "json schema without path", override: domain.OutputContract{Format: domain.FormatJSON, ValidationMode: domain.ValidationJSONSchema}},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -3,10 +3,12 @@ package usecase
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"reflect"
|
"reflect"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
|
||||||
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
|
"gitea.maximumdirect.net/eric/promptkit/internal/validate"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -385,19 +387,20 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
|
|||||||
admitter := &fakeRunAdmitter{}
|
admitter := &fakeRunAdmitter{}
|
||||||
reader := defaultArtifactReader()
|
reader := defaultArtifactReader()
|
||||||
renderer := defaultRenderer()
|
renderer := defaultRenderer()
|
||||||
|
client := &sequenceLLM{responses: []*domain.GenerateResponse{{
|
||||||
|
Content: `{"broken":true}`,
|
||||||
|
Usage: domain.TokenUsage{
|
||||||
|
PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19,
|
||||||
|
CachedTokens: 23, CacheWriteTokens: 29,
|
||||||
|
},
|
||||||
|
}}}
|
||||||
runner := NewRunnerWithRepairer(
|
runner := NewRunnerWithRepairer(
|
||||||
&fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)},
|
&fakePromptRepo{def: promptDef(domain.FormatJSON, domain.ValidationJSON, 1)},
|
||||||
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
|
&fakeExecutionProfileRepo{profiles: map[string]*domain.ExecutionProfile{"exec": defaultExecutionProfile()}},
|
||||||
nil,
|
nil,
|
||||||
reader,
|
reader,
|
||||||
renderer,
|
renderer,
|
||||||
&fakeLLM{resp: &domain.GenerateResponse{
|
client,
|
||||||
Content: `{"broken":true}`,
|
|
||||||
Usage: domain.TokenUsage{
|
|
||||||
PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19,
|
|
||||||
CachedTokens: 23, CacheWriteTokens: 29,
|
|
||||||
},
|
|
||||||
}},
|
|
||||||
validator,
|
validator,
|
||||||
repairer,
|
repairer,
|
||||||
admitter,
|
admitter,
|
||||||
@@ -421,6 +424,10 @@ func TestRunnerRunPreparedUsesFrozenValidationForInitialAndRepairOutputs(t *test
|
|||||||
if validator.directValidateCalls != 0 || repairer.calls != 1 {
|
if validator.directValidateCalls != 0 || repairer.calls != 1 {
|
||||||
t.Fatalf("validation/repair calls=(direct=%d repair=%d), want (0, 1)", validator.directValidateCalls, repairer.calls)
|
t.Fatalf("validation/repair calls=(direct=%d repair=%d), want (0, 1)", validator.directValidateCalls, repairer.calls)
|
||||||
}
|
}
|
||||||
|
if len(client.requests) != 1 || len(repairer.reqs) != 1 ||
|
||||||
|
!reflect.DeepEqual(repairer.reqs[0].OriginalMessages, client.requests[0].Prompt.Messages) {
|
||||||
|
t.Fatalf("initial and repair messages = (%#v, %#v)", client.requests, repairer.reqs)
|
||||||
|
}
|
||||||
if result.Validation.Status != domain.ValidationPassed || result.Validation.RepairAttempts != 1 {
|
if result.Validation.Status != domain.ValidationPassed || result.Validation.RepairAttempts != 1 {
|
||||||
t.Fatalf("unexpected repaired validation result: %+v", result.Validation)
|
t.Fatalf("unexpected repaired validation result: %+v", result.Validation)
|
||||||
}
|
}
|
||||||
@@ -443,6 +450,7 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
|
|||||||
generationFailure := errors.New("generation failed")
|
generationFailure := errors.New("generation failed")
|
||||||
validationFailure := errors.New("validation failed")
|
validationFailure := errors.New("validation failed")
|
||||||
repairFailure := errors.New("repair failed")
|
repairFailure := errors.New("repair failed")
|
||||||
|
repairInvalidRequest := fmt.Errorf("repair request: %w", llm.ErrInvalidRequest)
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -450,6 +458,7 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
|
|||||||
validation *recordingPreparedValidation
|
validation *recordingPreparedValidation
|
||||||
repairer *fakeRepairer
|
repairer *fakeRepairer
|
||||||
wantError error
|
wantError error
|
||||||
|
wantSource error
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
name: "generation failure",
|
name: "generation failure",
|
||||||
@@ -474,8 +483,51 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
|
|||||||
IsValid: false,
|
IsValid: false,
|
||||||
}},
|
}},
|
||||||
},
|
},
|
||||||
repairer: &fakeRepairer{err: repairFailure},
|
repairer: &fakeRepairer{err: repairFailure},
|
||||||
wantError: ErrValidation,
|
wantError: ErrLLMGenerate,
|
||||||
|
wantSource: repairFailure,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "repair invalid request",
|
||||||
|
validation: &recordingPreparedValidation{
|
||||||
|
results: []domain.ValidationResult{{
|
||||||
|
Status: domain.ValidationFailed,
|
||||||
|
Mode: domain.ValidationJSON,
|
||||||
|
Errors: []string{"invalid"},
|
||||||
|
IsValid: false,
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
repairer: &fakeRepairer{err: repairInvalidRequest},
|
||||||
|
wantError: ErrInvalidRequest,
|
||||||
|
wantSource: repairInvalidRequest,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "repair cancellation",
|
||||||
|
validation: &recordingPreparedValidation{
|
||||||
|
results: []domain.ValidationResult{{
|
||||||
|
Status: domain.ValidationFailed,
|
||||||
|
Mode: domain.ValidationJSON,
|
||||||
|
Errors: []string{"invalid"},
|
||||||
|
IsValid: false,
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
repairer: &fakeRepairer{err: context.Canceled},
|
||||||
|
wantError: ErrLLMGenerate,
|
||||||
|
wantSource: context.Canceled,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "repair deadline",
|
||||||
|
validation: &recordingPreparedValidation{
|
||||||
|
results: []domain.ValidationResult{{
|
||||||
|
Status: domain.ValidationFailed,
|
||||||
|
Mode: domain.ValidationJSON,
|
||||||
|
Errors: []string{"invalid"},
|
||||||
|
IsValid: false,
|
||||||
|
}},
|
||||||
|
},
|
||||||
|
repairer: &fakeRepairer{err: context.DeadlineExceeded},
|
||||||
|
wantError: ErrLLMGenerate,
|
||||||
|
wantSource: context.DeadlineExceeded,
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -514,6 +566,9 @@ func TestRunnerRunPreparedReleasesAdmissionAcrossExecutionErrors(t *testing.T) {
|
|||||||
if result != nil || !errors.Is(err, test.wantError) {
|
if result != nil || !errors.Is(err, test.wantError) {
|
||||||
t.Fatalf("run prepared=(%+v, %v), want %v", result, err, test.wantError)
|
t.Fatalf("run prepared=(%+v, %v), want %v", result, err, test.wantError)
|
||||||
}
|
}
|
||||||
|
if test.wantSource != nil && !errors.Is(err, test.wantSource) {
|
||||||
|
t.Fatalf("run prepared error = %v, want source %v", err, test.wantSource)
|
||||||
|
}
|
||||||
if len(admitter.backendIDs) != 1 || admitter.releaseCalls != 1 {
|
if len(admitter.backendIDs) != 1 || admitter.releaseCalls != 1 {
|
||||||
t.Fatalf(
|
t.Fatalf(
|
||||||
"admission calls=%#v releases=%d, want one each",
|
"admission calls=%#v releases=%d, want one each",
|
||||||
|
|||||||
@@ -2,19 +2,29 @@ package usecase
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||||
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
|
"gitea.maximumdirect.net/eric/promptkit/internal/llm"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
maxRepairDiagnosticBytes = 64 * 1024
|
||||||
|
omittedRepairDiagnostics = "additional validation diagnostics were omitted"
|
||||||
|
)
|
||||||
|
|
||||||
|
// OutputRepairer generates a corrected candidate after validation fails.
|
||||||
type OutputRepairer interface {
|
type OutputRepairer interface {
|
||||||
Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error)
|
Repair(ctx context.Context, req RepairRequest) (*domain.GenerateResponse, error)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RepairRequest contains the immutable execution state needed for one correction.
|
||||||
type RepairRequest struct {
|
type RepairRequest struct {
|
||||||
|
OriginalMessages []domain.RenderedMessage
|
||||||
PreviousOutput string
|
PreviousOutput string
|
||||||
ValidationErrors []string
|
ValidationErrors []string
|
||||||
SessionID string
|
SessionID string
|
||||||
@@ -30,6 +40,7 @@ type defaultOutputRepairer struct {
|
|||||||
llm llm.Client
|
llm llm.Client
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// NewDefaultOutputRepairer constructs the standard internal output repairer.
|
||||||
func NewDefaultOutputRepairer(llmClient llm.Client) OutputRepairer {
|
func NewDefaultOutputRepairer(llmClient llm.Client) OutputRepairer {
|
||||||
return &defaultOutputRepairer{llm: llmClient}
|
return &defaultOutputRepairer{llm: llmClient}
|
||||||
}
|
}
|
||||||
@@ -39,33 +50,42 @@ func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (
|
|||||||
return nil, errors.New("llm client is required for repair")
|
return nil, errors.New("llm client is required for repair")
|
||||||
}
|
}
|
||||||
|
|
||||||
errs := "(none provided)"
|
guidance, err := repairGuidance(req.Mode)
|
||||||
if len(req.ValidationErrors) > 0 {
|
if err != nil {
|
||||||
errs = strings.Join(req.ValidationErrors, "\n")
|
return nil, err
|
||||||
}
|
}
|
||||||
|
|
||||||
prompt := domain.RenderedPrompt{
|
messages := make([]domain.RenderedMessage, len(req.OriginalMessages), len(req.OriginalMessages)+2)
|
||||||
Messages: []domain.RenderedMessage{
|
copy(messages, req.OriginalMessages)
|
||||||
{
|
if strings.TrimSpace(req.PreviousOutput) != "" {
|
||||||
Role: "system",
|
messages = append(messages, domain.RenderedMessage{
|
||||||
Content: "You repair invalid JSON output. Return only corrected JSON. Do not include explanations or markdown code fences.",
|
Role: "assistant",
|
||||||
},
|
Content: req.PreviousOutput,
|
||||||
{
|
})
|
||||||
Role: "user",
|
|
||||||
Content: fmt.Sprintf(
|
|
||||||
"Repair attempt %d of %d for validation mode %s.\n\nValidation errors:\n%s\n\nPrevious output:\n%s\n\nReturn only corrected JSON.",
|
|
||||||
req.Attempt,
|
|
||||||
req.MaxAttempts,
|
|
||||||
req.Mode,
|
|
||||||
errs,
|
|
||||||
req.PreviousOutput,
|
|
||||||
),
|
|
||||||
},
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
previousResponse := "The previous response was empty."
|
||||||
|
if strings.TrimSpace(req.PreviousOutput) != "" {
|
||||||
|
previousResponse = "The previous response is included immediately before this instruction."
|
||||||
|
}
|
||||||
|
messages = append(messages, domain.RenderedMessage{
|
||||||
|
Role: "user",
|
||||||
|
Content: fmt.Sprintf(
|
||||||
|
"Repair attempt %d of %d for validation mode %s.\n"+
|
||||||
|
"Preserve valid values and change only what is necessary.\n"+
|
||||||
|
"%s\n%s\n"+
|
||||||
|
"Validation diagnostics (data):\n%s",
|
||||||
|
req.Attempt,
|
||||||
|
req.MaxAttempts,
|
||||||
|
req.Mode,
|
||||||
|
previousResponse,
|
||||||
|
guidance,
|
||||||
|
formatRepairDiagnostics(req.ValidationErrors),
|
||||||
|
),
|
||||||
|
})
|
||||||
|
|
||||||
resp, err := r.llm.Generate(ctx, newGenerationRequest(
|
resp, err := r.llm.Generate(ctx, newGenerationRequest(
|
||||||
prompt,
|
domain.RenderedPrompt{Messages: messages},
|
||||||
req.SessionID,
|
req.SessionID,
|
||||||
req.Target,
|
req.Target,
|
||||||
req.TargetPresence,
|
req.TargetPresence,
|
||||||
@@ -80,3 +100,90 @@ func (r *defaultOutputRepairer) Repair(ctx context.Context, req RepairRequest) (
|
|||||||
|
|
||||||
return resp, nil
|
return resp, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func repairGuidance(mode domain.ValidationMode) (string, error) {
|
||||||
|
switch mode {
|
||||||
|
case domain.ValidationBasic:
|
||||||
|
return "Return a nonempty response satisfying the original request.", nil
|
||||||
|
case domain.ValidationJSON, domain.ValidationJSONSchema:
|
||||||
|
return "Return only corrected JSON, with no explanation or Markdown fences.", nil
|
||||||
|
default:
|
||||||
|
return "", fmt.Errorf("unsupported validation mode for repair: %q", mode)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func formatRepairDiagnostics(errors []string) string {
|
||||||
|
diagnostics := make([]string, len(errors))
|
||||||
|
for index, diagnostic := range errors {
|
||||||
|
diagnostics[index] = strings.ToValidUTF8(diagnostic, "\uFFFD")
|
||||||
|
}
|
||||||
|
|
||||||
|
complete, _ := json.Marshal(diagnostics)
|
||||||
|
if len(complete) <= maxRepairDiagnosticBytes {
|
||||||
|
return string(complete)
|
||||||
|
}
|
||||||
|
|
||||||
|
omission, _ := json.Marshal(omittedRepairDiagnostics)
|
||||||
|
encoded := make([]byte, 0, maxRepairDiagnosticBytes)
|
||||||
|
encoded = append(encoded, '[')
|
||||||
|
for _, diagnostic := range diagnostics {
|
||||||
|
entry, _ := json.Marshal(diagnostic)
|
||||||
|
separator := 0
|
||||||
|
if len(encoded) > 1 {
|
||||||
|
separator = 1
|
||||||
|
}
|
||||||
|
available := maxRepairDiagnosticBytes - len(encoded) - separator - 1 - len(omission) - 1
|
||||||
|
if len(entry) <= available {
|
||||||
|
if separator != 0 {
|
||||||
|
encoded = append(encoded, ',')
|
||||||
|
}
|
||||||
|
encoded = append(encoded, entry...)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if available < len(`""`) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if separator != 0 {
|
||||||
|
encoded = append(encoded, ',')
|
||||||
|
}
|
||||||
|
encoded = append(encoded, truncateDiagnosticJSONValue(diagnostic, available)...)
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if len(encoded) > 1 {
|
||||||
|
encoded = append(encoded, ',')
|
||||||
|
}
|
||||||
|
encoded = append(encoded, omission...)
|
||||||
|
encoded = append(encoded, ']')
|
||||||
|
return string(encoded)
|
||||||
|
}
|
||||||
|
|
||||||
|
func truncateDiagnosticJSONValue(value string, maxBytes int) []byte {
|
||||||
|
if maxBytes < len(`""`) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
boundaries := []int{0}
|
||||||
|
for end := 0; end < len(value); {
|
||||||
|
_, size := utf8.DecodeRuneInString(value[end:])
|
||||||
|
end += size
|
||||||
|
if end+len(`""`) > maxBytes {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
boundaries = append(boundaries, end)
|
||||||
|
}
|
||||||
|
|
||||||
|
low, high := 0, len(boundaries)-1
|
||||||
|
best := []byte(`""`)
|
||||||
|
for low <= high {
|
||||||
|
mid := low + (high-low)/2
|
||||||
|
candidate, _ := json.Marshal(value[:boundaries[mid]])
|
||||||
|
if len(candidate) <= maxBytes {
|
||||||
|
best = candidate
|
||||||
|
low = mid + 1
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
high = mid - 1
|
||||||
|
}
|
||||||
|
return best
|
||||||
|
}
|
||||||
|
|||||||
349
internal/usecase/repairer_test.go
Normal file
349
internal/usecase/repairer_test.go
Normal file
@@ -0,0 +1,349 @@
|
|||||||
|
package usecase
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"reflect"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"gitea.maximumdirect.net/eric/promptkit/internal/domain"
|
||||||
|
)
|
||||||
|
|
||||||
|
type recordingRepairClient struct {
|
||||||
|
requests []domain.GenerateRequest
|
||||||
|
response *domain.GenerateResponse
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
|
func (c *recordingRepairClient) Generate(_ context.Context, req domain.GenerateRequest) (*domain.GenerateResponse, error) {
|
||||||
|
c.requests = append(c.requests, req)
|
||||||
|
return c.response, c.err
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDefaultOutputRepairerBuildsFullContextRequest(t *testing.T) {
|
||||||
|
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
|
||||||
|
repairer := NewDefaultOutputRepairer(client)
|
||||||
|
original := []domain.RenderedMessage{
|
||||||
|
{Role: "system", Content: "Follow the task.", CacheControl: &domain.CacheControl{Type: domain.CacheControlEphemeral, TTL: "1h"}},
|
||||||
|
{Role: "user", Content: "Summarize the report."},
|
||||||
|
}
|
||||||
|
before := append([]domain.RenderedMessage(nil), original...)
|
||||||
|
previous := strings.Repeat("candidate ", 12_000)
|
||||||
|
target := domain.ExecutionTarget{BackendID: "backend", Endpoint: "https://provider.example/v1", Model: "model"}
|
||||||
|
presence := domain.ExecutionTargetPresence{Temperature: true, TopP: true}
|
||||||
|
structured := &domain.StructuredOutputSpec{}
|
||||||
|
|
||||||
|
response, err := repairer.Repair(context.Background(), RepairRequest{
|
||||||
|
OriginalMessages: original,
|
||||||
|
PreviousOutput: previous,
|
||||||
|
ValidationErrors: []string{"invalid JSON"},
|
||||||
|
SessionID: "session",
|
||||||
|
Target: target,
|
||||||
|
TargetPresence: presence,
|
||||||
|
StructuredOutput: structured,
|
||||||
|
Attempt: 1,
|
||||||
|
MaxAttempts: 3,
|
||||||
|
Mode: domain.ValidationJSONSchema,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("repair: %v", err)
|
||||||
|
}
|
||||||
|
if response == nil || response.Content != "corrected" {
|
||||||
|
t.Fatalf("response = %+v", response)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(original, before) {
|
||||||
|
t.Fatalf("original messages changed: got %#v, want %#v", original, before)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 1 {
|
||||||
|
t.Fatalf("generation requests = %d, want 1", len(client.requests))
|
||||||
|
}
|
||||||
|
|
||||||
|
request := client.requests[0]
|
||||||
|
if request.Prompt.SessionID != "session" || !reflect.DeepEqual(request.Target, target) || request.TargetPresence != presence || request.StructuredOutput != structured {
|
||||||
|
t.Fatalf("generation request fields = %+v", request)
|
||||||
|
}
|
||||||
|
if len(request.Prompt.Messages) != len(original)+2 {
|
||||||
|
t.Fatalf("message count = %d, want %d", len(request.Prompt.Messages), len(original)+2)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(request.Prompt.Messages[:len(original)], original) {
|
||||||
|
t.Fatalf("original messages = %#v, want %#v", request.Prompt.Messages[:len(original)], original)
|
||||||
|
}
|
||||||
|
assistant := request.Prompt.Messages[len(original)]
|
||||||
|
if assistant.Role != "assistant" || assistant.Content != previous {
|
||||||
|
t.Fatalf("assistant candidate = %+v", assistant)
|
||||||
|
}
|
||||||
|
correction := request.Prompt.Messages[len(original)+1]
|
||||||
|
if correction.Role != "user" || !strings.Contains(correction.Content, "Repair attempt 1 of 3") ||
|
||||||
|
!strings.Contains(correction.Content, "Preserve valid values") ||
|
||||||
|
!strings.Contains(correction.Content, "Return only corrected JSON") {
|
||||||
|
t.Fatalf("correction message = %q", correction.Content)
|
||||||
|
}
|
||||||
|
if diagnostics := repairDiagnosticsFromMessage(t, correction.Content); !reflect.DeepEqual(diagnostics, []string{"invalid JSON"}) {
|
||||||
|
t.Fatalf("diagnostics = %#v", diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDefaultOutputRepairerDoesNotAccumulateCandidates(t *testing.T) {
|
||||||
|
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
|
||||||
|
repairer := NewDefaultOutputRepairer(client)
|
||||||
|
backing := make([]domain.RenderedMessage, 1, 4)
|
||||||
|
backing[0] = domain.RenderedMessage{Role: "user", Content: "Original task"}
|
||||||
|
before := append([]domain.RenderedMessage(nil), backing...)
|
||||||
|
|
||||||
|
for _, candidate := range []string{"first invalid", "second invalid"} {
|
||||||
|
_, err := repairer.Repair(context.Background(), RepairRequest{
|
||||||
|
OriginalMessages: backing,
|
||||||
|
PreviousOutput: candidate,
|
||||||
|
Attempt: 1,
|
||||||
|
MaxAttempts: 3,
|
||||||
|
Mode: domain.ValidationJSON,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("repair %q: %v", candidate, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(backing, before) {
|
||||||
|
t.Fatalf("caller messages changed: got %#v, want %#v", backing, before)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 2 {
|
||||||
|
t.Fatalf("generation requests = %d, want 2", len(client.requests))
|
||||||
|
}
|
||||||
|
for index, request := range client.requests {
|
||||||
|
messages := request.Prompt.Messages
|
||||||
|
if len(messages) != 3 || messages[0] != backing[0] || messages[1].Role != "assistant" || messages[1].Content != []string{"first invalid", "second invalid"}[index] {
|
||||||
|
t.Fatalf("request %d messages = %#v", index, messages)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDefaultOutputRepairerOmitsEmptyCandidateMessage(t *testing.T) {
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
candidate string
|
||||||
|
}{
|
||||||
|
{name: "empty", candidate: ""},
|
||||||
|
{name: "whitespace", candidate: " \n\t "},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
|
||||||
|
repairer := NewDefaultOutputRepairer(client)
|
||||||
|
_, err := repairer.Repair(context.Background(), RepairRequest{
|
||||||
|
OriginalMessages: []domain.RenderedMessage{{Role: "user", Content: "Original task"}},
|
||||||
|
PreviousOutput: tc.candidate,
|
||||||
|
Attempt: 1,
|
||||||
|
MaxAttempts: 1,
|
||||||
|
Mode: domain.ValidationBasic,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("repair: %v", err)
|
||||||
|
}
|
||||||
|
messages := client.requests[0].Prompt.Messages
|
||||||
|
if len(messages) != 2 || messages[1].Role != "user" || !strings.Contains(messages[1].Content, "previous response was empty") {
|
||||||
|
t.Fatalf("messages = %#v", messages)
|
||||||
|
}
|
||||||
|
if strings.Contains(messages[1].Content, tc.candidate) && tc.candidate != "" {
|
||||||
|
t.Fatalf("correction message repeated whitespace candidate: %q", messages[1].Content)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDefaultOutputRepairerUsesModeSpecificGuidance(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
mode domain.ValidationMode
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{mode: domain.ValidationBasic, want: "Return a nonempty response"},
|
||||||
|
{mode: domain.ValidationJSON, want: "Return only corrected JSON"},
|
||||||
|
{mode: domain.ValidationJSONSchema, want: "Return only corrected JSON"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(string(tc.mode), func(t *testing.T) {
|
||||||
|
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "corrected"}}
|
||||||
|
_, err := NewDefaultOutputRepairer(client).Repair(context.Background(), RepairRequest{Mode: tc.mode, Attempt: 1, MaxAttempts: 1})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("repair: %v", err)
|
||||||
|
}
|
||||||
|
if !strings.Contains(client.requests[0].Prompt.Messages[0].Content, tc.want) {
|
||||||
|
t.Fatalf("correction message = %q", client.requests[0].Prompt.Messages[0].Content)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
client := &recordingRepairClient{response: &domain.GenerateResponse{Content: "unexpected"}}
|
||||||
|
_, err := NewDefaultOutputRepairer(client).Repair(context.Background(), RepairRequest{Mode: domain.ValidationNone})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "unsupported validation mode") {
|
||||||
|
t.Fatalf("unsupported mode error = %v", err)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 0 {
|
||||||
|
t.Fatalf("generation requests = %d, want 0", len(client.requests))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFormatRepairDiagnosticsBoundsAndPreservesData(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
errors []string
|
||||||
|
wantOmission bool
|
||||||
|
check func(*testing.T, []string)
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "below limit",
|
||||||
|
errors: []string{"first", "second"},
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if !reflect.DeepEqual(got, []string{"first", "second"}) {
|
||||||
|
t.Fatalf("diagnostics = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "at limit",
|
||||||
|
errors: []string{strings.Repeat("x", maxRepairDiagnosticBytes-4)},
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if len(got) != 1 || len(got[0]) != maxRepairDiagnosticBytes-4 {
|
||||||
|
t.Fatalf("diagnostics lengths = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "multibyte truncation preserves prior entries",
|
||||||
|
errors: []string{"first", strings.Repeat("界", maxRepairDiagnosticBytes)},
|
||||||
|
wantOmission: true,
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if len(got) != 3 || got[0] != "first" || !strings.HasPrefix(strings.Repeat("界", maxRepairDiagnosticBytes), got[1]) {
|
||||||
|
t.Fatalf("diagnostics = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "many diagnostics",
|
||||||
|
errors: manyRepairDiagnostics(),
|
||||||
|
wantOmission: true,
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if len(got) < 2 || got[0] != manyRepairDiagnostics()[0] {
|
||||||
|
t.Fatalf("diagnostics = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "no room for partial diagnostic",
|
||||||
|
errors: func() []string {
|
||||||
|
omission, _ := json.Marshal(omittedRepairDiagnostics)
|
||||||
|
return []string{
|
||||||
|
strings.Repeat("x", maxRepairDiagnosticBytes-len(omission)-5),
|
||||||
|
strings.Repeat("y", 128),
|
||||||
|
}
|
||||||
|
}(),
|
||||||
|
wantOmission: true,
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if len(got) != 2 || got[1] != omittedRepairDiagnostics {
|
||||||
|
t.Fatalf("diagnostics = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid UTF-8",
|
||||||
|
errors: []string{"broken\xffinput"},
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if !reflect.DeepEqual(got, []string{"broken\uFFFDinput"}) {
|
||||||
|
t.Fatalf("diagnostics = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "one huge diagnostic",
|
||||||
|
errors: []string{strings.Repeat("x", maxRepairDiagnosticBytes*2)},
|
||||||
|
wantOmission: true,
|
||||||
|
check: func(t *testing.T, got []string) {
|
||||||
|
t.Helper()
|
||||||
|
if len(got) != 2 || !strings.HasPrefix(strings.Repeat("x", maxRepairDiagnosticBytes*2), got[0]) {
|
||||||
|
t.Fatalf("diagnostics = %#v", got)
|
||||||
|
}
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range tests {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
before := append([]string(nil), tc.errors...)
|
||||||
|
encoded := formatRepairDiagnostics(tc.errors)
|
||||||
|
if len(encoded) > maxRepairDiagnosticBytes || !utf8.ValidString(encoded) {
|
||||||
|
t.Fatalf("encoded diagnostic length/UTF-8 = (%d, %v)", len(encoded), utf8.ValidString(encoded))
|
||||||
|
}
|
||||||
|
var got []string
|
||||||
|
if err := json.Unmarshal([]byte(encoded), &got); err != nil {
|
||||||
|
t.Fatalf("decode diagnostics: %v; encoded=%q", err, encoded)
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(tc.errors, before) {
|
||||||
|
t.Fatalf("input diagnostics changed: got %#v, want %#v", tc.errors, before)
|
||||||
|
}
|
||||||
|
if hasOmission := len(got) > 0 && got[len(got)-1] == omittedRepairDiagnostics; hasOmission != tc.wantOmission {
|
||||||
|
t.Fatalf("omission = %v, want %v; diagnostics=%#v", hasOmission, tc.wantOmission, got)
|
||||||
|
}
|
||||||
|
tc.check(t, got)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func manyRepairDiagnostics() []string {
|
||||||
|
diagnostics := make([]string, 1_000)
|
||||||
|
for index := range diagnostics {
|
||||||
|
diagnostics[index] = fmt.Sprintf("diagnostic %04d %s", index, strings.Repeat("x", 128))
|
||||||
|
}
|
||||||
|
return diagnostics
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDefaultOutputRepairerPropagatesGenerationFailures(t *testing.T) {
|
||||||
|
expected := errors.New("generation failed")
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
client *recordingRepairClient
|
||||||
|
want error
|
||||||
|
}{
|
||||||
|
{name: "nil client", want: nil},
|
||||||
|
{name: "generation error", client: &recordingRepairClient{err: expected}, want: expected},
|
||||||
|
{name: "nil response", client: &recordingRepairClient{}, want: nil},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
var repairer OutputRepairer
|
||||||
|
if tc.client == nil {
|
||||||
|
repairer = NewDefaultOutputRepairer(nil)
|
||||||
|
} else {
|
||||||
|
repairer = NewDefaultOutputRepairer(tc.client)
|
||||||
|
}
|
||||||
|
response, err := repairer.Repair(context.Background(), RepairRequest{Mode: domain.ValidationJSON})
|
||||||
|
if response != nil || err == nil {
|
||||||
|
t.Fatalf("response/error = (%+v, %v)", response, err)
|
||||||
|
}
|
||||||
|
if tc.want != nil && !errors.Is(err, tc.want) {
|
||||||
|
t.Fatalf("error = %v, want %v", err, tc.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func repairDiagnosticsFromMessage(t *testing.T, message string) []string {
|
||||||
|
t.Helper()
|
||||||
|
const marker = "Validation diagnostics (data):\n"
|
||||||
|
index := strings.Index(message, marker)
|
||||||
|
if index < 0 {
|
||||||
|
t.Fatalf("missing diagnostics marker in %q", message)
|
||||||
|
}
|
||||||
|
encoded := message[index+len(marker):]
|
||||||
|
var diagnostics []string
|
||||||
|
if err := json.Unmarshal([]byte(encoded), &diagnostics); err != nil {
|
||||||
|
t.Fatalf("decode diagnostics: %v", err)
|
||||||
|
}
|
||||||
|
return diagnostics
|
||||||
|
}
|
||||||
@@ -184,10 +184,10 @@ func (r *Runner) executePreparedRun(
|
|||||||
prepared.StructuredOutput,
|
prepared.StructuredOutput,
|
||||||
))
|
))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
if errors.Is(err, llm.ErrInvalidRequest) {
|
return nil, wrapGenerationError(err)
|
||||||
return nil, fmt.Errorf("%w: %w", ErrInvalidRequest, err)
|
}
|
||||||
}
|
if genResp == nil {
|
||||||
return nil, fmt.Errorf("%w: %w", ErrLLMGenerate, err)
|
return nil, fmt.Errorf("%w: model returned nil response", ErrLLMGenerate)
|
||||||
}
|
}
|
||||||
usage := genResp.Usage
|
usage := genResp.Usage
|
||||||
|
|
||||||
@@ -203,6 +203,7 @@ func (r *Runner) executePreparedRun(
|
|||||||
attemptsUsed++
|
attemptsUsed++
|
||||||
|
|
||||||
repairResp, repairErr := r.repairer.Repair(ctx, RepairRequest{
|
repairResp, repairErr := r.repairer.Repair(ctx, RepairRequest{
|
||||||
|
OriginalMessages: prepared.Messages,
|
||||||
PreviousOutput: genResp.Content,
|
PreviousOutput: genResp.Content,
|
||||||
ValidationErrors: validationResult.Errors,
|
ValidationErrors: validationResult.Errors,
|
||||||
SessionID: prepared.SessionID,
|
SessionID: prepared.SessionID,
|
||||||
@@ -214,10 +215,10 @@ func (r *Runner) executePreparedRun(
|
|||||||
Mode: prepared.OutputContract.ValidationMode,
|
Mode: prepared.OutputContract.ValidationMode,
|
||||||
})
|
})
|
||||||
if repairErr != nil {
|
if repairErr != nil {
|
||||||
return nil, fmt.Errorf("%w: %w", ErrValidation, repairErr)
|
return nil, wrapGenerationError(repairErr)
|
||||||
}
|
}
|
||||||
if repairResp == nil {
|
if repairResp == nil {
|
||||||
return nil, fmt.Errorf("%w: repairer returned nil response", ErrValidation)
|
return nil, fmt.Errorf("%w: repairer returned nil response", ErrLLMGenerate)
|
||||||
}
|
}
|
||||||
|
|
||||||
genResp = repairResp
|
genResp = repairResp
|
||||||
@@ -257,6 +258,13 @@ func (r *Runner) executePreparedRun(
|
|||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func wrapGenerationError(err error) error {
|
||||||
|
if errors.Is(err, llm.ErrInvalidRequest) {
|
||||||
|
return fmt.Errorf("%w: %w", ErrInvalidRequest, err)
|
||||||
|
}
|
||||||
|
return fmt.Errorf("%w: %w", ErrLLMGenerate, err)
|
||||||
|
}
|
||||||
|
|
||||||
func addTokenUsage(total, next domain.TokenUsage) domain.TokenUsage {
|
func addTokenUsage(total, next domain.TokenUsage) domain.TokenUsage {
|
||||||
return domain.TokenUsage{
|
return domain.TokenUsage{
|
||||||
PromptTokens: total.PromptTokens + next.PromptTokens,
|
PromptTokens: total.PromptTokens + next.PromptTokens,
|
||||||
@@ -524,7 +532,12 @@ func (r *Runner) shouldAttemptRepair(contract domain.OutputContract, validationR
|
|||||||
if validationResult.Status != domain.ValidationFailed {
|
if validationResult.Status != domain.ValidationFailed {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
return contract.ValidationMode == domain.ValidationJSON || contract.ValidationMode == domain.ValidationJSONSchema
|
switch contract.ValidationMode {
|
||||||
|
case domain.ValidationBasic, domain.ValidationJSON, domain.ValidationJSONSchema:
|
||||||
|
return true
|
||||||
|
default:
|
||||||
|
return false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func mergeExecutionTarget(base domain.ExecutionTarget, override domain.ExecutionTarget) domain.ExecutionTarget {
|
func mergeExecutionTarget(base domain.ExecutionTarget, override domain.ExecutionTarget) domain.ExecutionTarget {
|
||||||
|
|||||||
@@ -295,8 +295,10 @@ func (c *controlledRepairLLM) Generate(
|
|||||||
ctx context.Context,
|
ctx context.Context,
|
||||||
req domain.GenerateRequest,
|
req domain.GenerateRequest,
|
||||||
) (*domain.GenerateResponse, error) {
|
) (*domain.GenerateResponse, error) {
|
||||||
isRepair := len(req.Prompt.Messages) > 0 &&
|
messages := req.Prompt.Messages
|
||||||
strings.HasPrefix(req.Prompt.Messages[0].Content, "You repair invalid JSON")
|
isRepair := len(messages) >= 2 &&
|
||||||
|
messages[len(messages)-2].Role == "assistant" &&
|
||||||
|
messages[len(messages)-1].Role == "user"
|
||||||
c.mu.Lock()
|
c.mu.Lock()
|
||||||
c.calls++
|
c.calls++
|
||||||
c.active++
|
c.active++
|
||||||
@@ -2143,6 +2145,8 @@ func TestRunnerRepairStateMachine(t *testing.T) {
|
|||||||
TopP: true,
|
TopP: true,
|
||||||
TimeoutSeconds: true,
|
TimeoutSeconds: true,
|
||||||
}
|
}
|
||||||
|
emptyThenValid := responses(2)
|
||||||
|
emptyThenValid[0].Content = " \t "
|
||||||
|
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
name string
|
name string
|
||||||
@@ -2165,12 +2169,13 @@ func TestRunnerRepairStateMachine(t *testing.T) {
|
|||||||
wantStatus: domain.ValidationPassed,
|
wantStatus: domain.ValidationPassed,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "basic failure is ineligible despite budget",
|
name: "empty basic output repairs successfully",
|
||||||
mode: domain.ValidationBasic,
|
mode: domain.ValidationBasic,
|
||||||
budget: 3,
|
budget: 3,
|
||||||
validationResults: []domain.ValidationResult{failed(domain.ValidationBasic, "empty output")},
|
validationResults: []domain.ValidationResult{failed(domain.ValidationBasic, "empty output"), passed(domain.ValidationBasic)},
|
||||||
responses: responses(1),
|
responses: emptyThenValid,
|
||||||
wantStatus: domain.ValidationFailed,
|
wantRepairs: 1,
|
||||||
|
wantStatus: domain.ValidationPassed,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "inherited numeric values remain absent",
|
name: "inherited numeric values remain absent",
|
||||||
@@ -2202,7 +2207,7 @@ func TestRunnerRepairStateMachine(t *testing.T) {
|
|||||||
{
|
{
|
||||||
name: "successful repair stops below larger budget",
|
name: "successful repair stops below larger budget",
|
||||||
mode: domain.ValidationJSON,
|
mode: domain.ValidationJSON,
|
||||||
budget: 4,
|
budget: 3,
|
||||||
validationResults: []domain.ValidationResult{
|
validationResults: []domain.ValidationResult{
|
||||||
failed(domain.ValidationJSON, "candidate zero"),
|
failed(domain.ValidationJSON, "candidate zero"),
|
||||||
failed(domain.ValidationJSON, "candidate one"),
|
failed(domain.ValidationJSON, "candidate one"),
|
||||||
@@ -2305,6 +2310,9 @@ func TestRunnerRepairStateMachine(t *testing.T) {
|
|||||||
!reflect.DeepEqual(req.ValidationErrors, tc.validationResults[index].Errors) {
|
!reflect.DeepEqual(req.ValidationErrors, tc.validationResults[index].Errors) {
|
||||||
t.Fatalf("repair request %d prior state = %+v", index, req)
|
t.Fatalf("repair request %d prior state = %+v", index, req)
|
||||||
}
|
}
|
||||||
|
if !reflect.DeepEqual(req.OriginalMessages, initialRequest.Prompt.Messages) {
|
||||||
|
t.Fatalf("repair request %d original messages drifted: %#v", index, req.OriginalMessages)
|
||||||
|
}
|
||||||
if req.TargetPresence != tc.wantPresence || !reflect.DeepEqual(req.Target, initialRequest.Target) ||
|
if req.TargetPresence != tc.wantPresence || !reflect.DeepEqual(req.Target, initialRequest.Target) ||
|
||||||
req.SessionID != initialRequest.Prompt.SessionID ||
|
req.SessionID != initialRequest.Prompt.SessionID ||
|
||||||
!reflect.DeepEqual(req.StructuredOutput, initialRequest.StructuredOutput) {
|
!reflect.DeepEqual(req.StructuredOutput, initialRequest.StructuredOutput) {
|
||||||
@@ -2318,8 +2326,19 @@ func TestRunnerRepairStateMachine(t *testing.T) {
|
|||||||
!reflect.DeepEqual(generated.StructuredOutput, initialRequest.StructuredOutput) {
|
!reflect.DeepEqual(generated.StructuredOutput, initialRequest.StructuredOutput) {
|
||||||
t.Fatalf("repair generation request %d common fields drifted: %+v", index, generated)
|
t.Fatalf("repair generation request %d common fields drifted: %+v", index, generated)
|
||||||
}
|
}
|
||||||
if reflect.DeepEqual(generated.Prompt.Messages, initialRequest.Prompt.Messages) {
|
expectedMessages := len(initialRequest.Prompt.Messages) + 1
|
||||||
t.Fatalf("repair generation request %d reused the initial prompt", index)
|
if strings.TrimSpace(tc.responses[index].Content) != "" {
|
||||||
|
expectedMessages++
|
||||||
|
}
|
||||||
|
if len(generated.Prompt.Messages) != expectedMessages ||
|
||||||
|
generated.Prompt.Messages[len(generated.Prompt.Messages)-1].Role != "user" {
|
||||||
|
t.Fatalf("repair generation request %d messages = %#v", index, generated.Prompt.Messages)
|
||||||
|
}
|
||||||
|
if strings.TrimSpace(tc.responses[index].Content) != "" {
|
||||||
|
assistant := generated.Prompt.Messages[len(generated.Prompt.Messages)-2]
|
||||||
|
if assistant.Role != "assistant" || assistant.Content != tc.responses[index].Content {
|
||||||
|
t.Fatalf("repair generation request %d candidate = %+v", index, assistant)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -174,9 +174,8 @@ func validateArtifact(ctx context.Context, artifact *domain.Artifact, contract d
|
|||||||
}
|
}
|
||||||
|
|
||||||
res := domain.ValidationResult{
|
res := domain.ValidationResult{
|
||||||
Mode: contract.ValidationMode,
|
Mode: contract.ValidationMode,
|
||||||
SchemaPath: contract.SchemaPath,
|
SchemaPath: contract.SchemaPath,
|
||||||
RepairAttempts: contract.RepairAttempts,
|
|
||||||
}
|
}
|
||||||
|
|
||||||
if artifact == nil {
|
if artifact == nil {
|
||||||
|
|||||||
@@ -64,6 +64,43 @@ func TestStandardValidatorBasicFailureEmpty(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestStandardValidatorReportsZeroRepairAttempts(t *testing.T) {
|
||||||
|
v := NewStandardValidator("")
|
||||||
|
|
||||||
|
for _, tc := range []struct {
|
||||||
|
name string
|
||||||
|
body string
|
||||||
|
contract domain.OutputContract
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "passed basic validation",
|
||||||
|
body: "answer",
|
||||||
|
contract: domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationBasic,
|
||||||
|
RepairAttempts: 3,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "failed JSON validation",
|
||||||
|
body: `{"answer":`,
|
||||||
|
contract: domain.OutputContract{
|
||||||
|
ValidationMode: domain.ValidationJSON,
|
||||||
|
RepairAttempts: 3,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
res, err := v.Validate(context.Background(), &domain.Artifact{Body: []byte(tc.body)}, tc.contract)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("validate: %v", err)
|
||||||
|
}
|
||||||
|
if res.RepairAttempts != 0 {
|
||||||
|
t.Fatalf("repair attempts = %d, want 0", res.RepairAttempts)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestStandardValidatorJSONSuccess(t *testing.T) {
|
func TestStandardValidatorJSONSuccess(t *testing.T) {
|
||||||
v := NewStandardValidator("")
|
v := NewStandardValidator("")
|
||||||
|
|
||||||
|
|||||||
@@ -22,27 +22,52 @@ func TestPreparationRejectsInvalidOutputContractWithPublicError(t *testing.T) {
|
|||||||
t.Fatalf("construct engine: %v", err)
|
t.Fatalf("construct engine: %v", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
req := promptkit.RunRequest{
|
for _, tc := range []struct {
|
||||||
PromptID: "prompt",
|
name string
|
||||||
Validation: &promptkit.OutputContract{
|
contract promptkit.OutputContract
|
||||||
Format: promptkit.OutputFormat("binary"),
|
}{
|
||||||
ValidationMode: promptkit.ValidationNone,
|
{
|
||||||
|
name: "unsupported format",
|
||||||
|
contract: promptkit.OutputContract{
|
||||||
|
Format: promptkit.OutputFormat("binary"),
|
||||||
|
ValidationMode: promptkit.ValidationNone,
|
||||||
|
},
|
||||||
},
|
},
|
||||||
}
|
{
|
||||||
|
name: "repair attempts above maximum",
|
||||||
|
contract: promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatText,
|
||||||
|
ValidationMode: promptkit.ValidationBasic,
|
||||||
|
RepairAttempts: 4,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "none validation with repair attempts",
|
||||||
|
contract: promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatText,
|
||||||
|
ValidationMode: promptkit.ValidationNone,
|
||||||
|
RepairAttempts: 1,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
} {
|
||||||
|
t.Run(tc.name, func(t *testing.T) {
|
||||||
|
req := promptkit.RunRequest{PromptID: "prompt", Validation: &tc.contract}
|
||||||
|
|
||||||
prepared, err := engine.Prepare(context.Background(), req)
|
prepared, err := engine.Prepare(context.Background(), req)
|
||||||
if prepared != nil {
|
if prepared != nil {
|
||||||
t.Fatalf("expected no partial prepared run, got %+v", prepared)
|
t.Fatalf("expected no partial prepared run, got %+v", prepared)
|
||||||
}
|
}
|
||||||
if !errors.Is(err, promptkit.ErrInvalidRequest) {
|
if !errors.Is(err, promptkit.ErrInvalidRequest) {
|
||||||
t.Fatalf("prepare error = %v, want ErrInvalidRequest", err)
|
t.Fatalf("prepare error = %v, want ErrInvalidRequest", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
preparedExecution, err := engine.PrepareExecution(context.Background(), req)
|
preparedExecution, err := engine.PrepareExecution(context.Background(), req)
|
||||||
if preparedExecution != nil {
|
if preparedExecution != nil {
|
||||||
t.Fatalf("expected no partial prepared execution, got %+v", preparedExecution)
|
t.Fatalf("expected no partial prepared execution, got %+v", preparedExecution)
|
||||||
}
|
}
|
||||||
if !errors.Is(err, promptkit.ErrInvalidRequest) {
|
if !errors.Is(err, promptkit.ErrInvalidRequest) {
|
||||||
t.Fatalf("prepare execution error = %v, want ErrInvalidRequest", err)
|
t.Fatalf("prepare execution error = %v, want ErrInvalidRequest", err)
|
||||||
|
}
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -256,6 +256,50 @@ func TestPreparedExecutionLifecycleAndEngineBinding(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPreparedExecutionRepairsEmptyBasicOutput(t *testing.T) {
|
||||||
|
client := &preparedRecordingClient{responses: []*promptkit.GenerateResponse{
|
||||||
|
{
|
||||||
|
Content: "",
|
||||||
|
Usage: promptkit.TokenUsage{PromptTokens: 3, CompletionTokens: 5, TotalTokens: 8},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Content: "Corrected summary.",
|
||||||
|
Usage: promptkit.TokenUsage{PromptTokens: 7, CompletionTokens: 11, TotalTokens: 18},
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
engine := newPreparedContractEngine(t, client, "Summarize the source.")
|
||||||
|
prepared, err := engine.PrepareExecution(context.Background(), promptkit.RunRequest{
|
||||||
|
PromptID: "prepared",
|
||||||
|
Validation: &promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatMarkdown,
|
||||||
|
ValidationMode: promptkit.ValidationBasic,
|
||||||
|
RepairAttempts: 1,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("prepare execution: %v", err)
|
||||||
|
}
|
||||||
|
details := prepared.Details()
|
||||||
|
|
||||||
|
result, err := engine.RunPrepared(context.Background(), prepared)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("run prepared: %v", err)
|
||||||
|
}
|
||||||
|
if result.RawOutput != "Corrected summary." || result.Validation.Status != promptkit.ValidationPassed ||
|
||||||
|
result.Validation.RepairAttempts != 1 || result.Usage != (promptkit.TokenUsage{PromptTokens: 10, CompletionTokens: 16, TotalTokens: 26}) {
|
||||||
|
t.Fatalf("repaired result = %+v", result)
|
||||||
|
}
|
||||||
|
requests := client.snapshot()
|
||||||
|
if len(requests) != 2 || len(requests[1].Prompt.Messages) != len(details.Messages)+1 ||
|
||||||
|
!reflect.DeepEqual(requests[1].Prompt.Messages[:len(details.Messages)], details.Messages) ||
|
||||||
|
requests[1].Prompt.Messages[len(requests[1].Prompt.Messages)-1].Role != "user" {
|
||||||
|
t.Fatalf("prepared repair requests = %#v", requests)
|
||||||
|
}
|
||||||
|
if _, err := engine.RunPrepared(context.Background(), prepared); !errors.Is(err, promptkit.ErrInvalidRequest) {
|
||||||
|
t.Fatalf("second RunPrepared error = %v, want ErrInvalidRequest", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestPreparedExecutionConcurrentClaimAllowsOneGeneration(t *testing.T) {
|
func TestPreparedExecutionConcurrentClaimAllowsOneGeneration(t *testing.T) {
|
||||||
release := make(chan struct{})
|
release := make(chan struct{})
|
||||||
client := &preparedRecordingClient{
|
client := &preparedRecordingClient{
|
||||||
@@ -671,12 +715,13 @@ func (r *mutablePreparedArtifactReader) callCount() int {
|
|||||||
}
|
}
|
||||||
|
|
||||||
type preparedRecordingClient struct {
|
type preparedRecordingClient struct {
|
||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
response *promptkit.GenerateResponse
|
response *promptkit.GenerateResponse
|
||||||
err error
|
responses []*promptkit.GenerateResponse
|
||||||
requests []promptkit.GenerateRequest
|
err error
|
||||||
started chan struct{}
|
requests []promptkit.GenerateRequest
|
||||||
release <-chan struct{}
|
started chan struct{}
|
||||||
|
release <-chan struct{}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (c *preparedRecordingClient) Generate(
|
func (c *preparedRecordingClient) Generate(
|
||||||
@@ -700,6 +745,12 @@ func (c *preparedRecordingClient) Generate(
|
|||||||
if c.err != nil {
|
if c.err != nil {
|
||||||
return nil, c.err
|
return nil, c.err
|
||||||
}
|
}
|
||||||
|
if len(c.responses) > 0 {
|
||||||
|
if index := len(c.requests) - 1; index < len(c.responses) {
|
||||||
|
return c.responses[index], nil
|
||||||
|
}
|
||||||
|
return nil, fmt.Errorf("no response configured for generation %d", len(c.requests))
|
||||||
|
}
|
||||||
return c.response, nil
|
return c.response, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -844,7 +844,7 @@ func TestBackendExtraParamsAreDeeplyCopiedAtConstructionAndLookup(t *testing.T)
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestEngineValidationIsSinglePass(t *testing.T) {
|
func TestEngineValidationWithZeroRepairBudgetIsSinglePass(t *testing.T) {
|
||||||
client := &fakeLLMClient{
|
client := &fakeLLMClient{
|
||||||
response: &promptkit.GenerateResponse{Content: "not-json"},
|
response: &promptkit.GenerateResponse{Content: "not-json"},
|
||||||
}
|
}
|
||||||
@@ -859,7 +859,7 @@ func TestEngineValidationIsSinglePass(t *testing.T) {
|
|||||||
Validation: &promptkit.OutputContract{
|
Validation: &promptkit.OutputContract{
|
||||||
Format: promptkit.FormatJSON,
|
Format: promptkit.FormatJSON,
|
||||||
ValidationMode: promptkit.ValidationJSON,
|
ValidationMode: promptkit.ValidationJSON,
|
||||||
RepairAttempts: 3,
|
RepairAttempts: 0,
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -874,6 +874,87 @@ func TestEngineValidationIsSinglePass(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEngineRunRepairsJSONSchemaOutput(t *testing.T) {
|
||||||
|
client := &fakeLLMClient{responses: []*promptkit.GenerateResponse{
|
||||||
|
{
|
||||||
|
Content: "{}",
|
||||||
|
Usage: promptkit.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5, CachedTokens: 7, CacheWriteTokens: 11},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
Content: `{"events":[{"title":"Repaired event"}]}`,
|
||||||
|
Usage: promptkit.TokenUsage{PromptTokens: 13, CompletionTokens: 17, TotalTokens: 19, CachedTokens: 23, CacheWriteTokens: 29},
|
||||||
|
},
|
||||||
|
}}
|
||||||
|
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
|
||||||
|
|
||||||
|
result, err := engine.Run(context.Background(), promptkit.RunRequest{
|
||||||
|
PromptID: frameworkStructuredEventsPromptID,
|
||||||
|
SessionID: " repair-session ",
|
||||||
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
|
"transcript": promptkit.Inline("Rin opens the gate."),
|
||||||
|
"glossary": promptkit.Inline("gate: A guarded passage."),
|
||||||
|
},
|
||||||
|
Validation: &promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatJSON,
|
||||||
|
ValidationMode: promptkit.ValidationJSONSchema,
|
||||||
|
SchemaPath: "structured_events.schema.json",
|
||||||
|
RepairAttempts: 1,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("run: %v", err)
|
||||||
|
}
|
||||||
|
if result.RawOutput != client.responses[1].Content || result.Validation.Status != promptkit.ValidationPassed ||
|
||||||
|
result.Validation.RepairAttempts != 1 {
|
||||||
|
t.Fatalf("repaired result = %+v", result)
|
||||||
|
}
|
||||||
|
wantUsage := promptkit.TokenUsage{PromptTokens: 15, CompletionTokens: 20, TotalTokens: 24, CachedTokens: 30, CacheWriteTokens: 40}
|
||||||
|
if result.Usage != wantUsage {
|
||||||
|
t.Fatalf("usage = %+v, want %+v", result.Usage, wantUsage)
|
||||||
|
}
|
||||||
|
if len(client.requests) != 2 {
|
||||||
|
t.Fatalf("generation calls = %d, want 2", len(client.requests))
|
||||||
|
}
|
||||||
|
initial, repaired := client.requests[0], client.requests[1]
|
||||||
|
if initial.Prompt.SessionID != "repair-session" || repaired.Prompt.SessionID != initial.Prompt.SessionID ||
|
||||||
|
!reflect.DeepEqual(repaired.Target, initial.Target) || repaired.TargetPresence != initial.TargetPresence ||
|
||||||
|
!reflect.DeepEqual(repaired.StructuredOutput, initial.StructuredOutput) {
|
||||||
|
t.Fatalf("generation request state drifted: initial=%+v repaired=%+v", initial, repaired)
|
||||||
|
}
|
||||||
|
if initial.StructuredOutput == nil || initial.StructuredOutput.JSONSchema == nil {
|
||||||
|
t.Fatalf("expected structured output on initial request: %+v", initial)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEngineRunReturnsFinalResultAfterRepairExhaustion(t *testing.T) {
|
||||||
|
client := &fakeLLMClient{responses: []*promptkit.GenerateResponse{
|
||||||
|
{Content: "not-json", Usage: promptkit.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5}},
|
||||||
|
{Content: "still-not-json", Usage: promptkit.TokenUsage{PromptTokens: 7, CompletionTokens: 11, TotalTokens: 18}},
|
||||||
|
}}
|
||||||
|
engine := newContractEngineWithOptions(t, frameworkSchemaDir, promptkit.WithLLMClient(client))
|
||||||
|
|
||||||
|
result, err := engine.Run(context.Background(), promptkit.RunRequest{
|
||||||
|
PromptID: frameworkMarkdownSummaryPromptID,
|
||||||
|
Inputs: map[string]promptkit.ArtifactRef{
|
||||||
|
"transcript": promptkit.Inline("Rin opens the gate."),
|
||||||
|
"glossary": promptkit.Inline("gate: A guarded passage."),
|
||||||
|
},
|
||||||
|
Validation: &promptkit.OutputContract{
|
||||||
|
Format: promptkit.FormatJSON,
|
||||||
|
ValidationMode: promptkit.ValidationJSON,
|
||||||
|
RepairAttempts: 1,
|
||||||
|
},
|
||||||
|
})
|
||||||
|
if err != nil || result == nil {
|
||||||
|
t.Fatalf("run = (%+v, %v), want exhausted result", result, err)
|
||||||
|
}
|
||||||
|
if result.RawOutput != "still-not-json" || result.Validation.Status != promptkit.ValidationFailed ||
|
||||||
|
result.Validation.RepairAttempts != 1 || len(result.Validation.Errors) == 0 ||
|
||||||
|
result.Usage != (promptkit.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23}) {
|
||||||
|
t.Fatalf("exhausted result = %+v", result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
|
func TestRepeatedOptionsUseLastValueInEachCategory(t *testing.T) {
|
||||||
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
|
profile := promptkit.Profile{ID: "profile", Endpoint: "http://example.test/v1", Model: "model"}
|
||||||
|
|
||||||
|
|||||||
19
types.go
19
types.go
@@ -565,8 +565,8 @@ type ExecutionTargetPresence struct {
|
|||||||
// JSON representation.
|
// JSON representation.
|
||||||
//
|
//
|
||||||
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
|
// A non-nil RunRequest.Validation replaces the complete prompt contract. It
|
||||||
// does not merge fields. The public Engine validates generated output once and
|
// does not merge fields. The public Engine performs bounded correction after a
|
||||||
// does not install an output repairer.
|
// failed eligible validation when RepairAttempts is positive.
|
||||||
type OutputContract struct {
|
type OutputContract struct {
|
||||||
// Format selects generated artifact metadata. An empty value in a non-nil
|
// Format selects generated artifact metadata. An empty value in a non-nil
|
||||||
// request replacement defaults to FormatText.
|
// request replacement defaults to FormatText.
|
||||||
@@ -577,9 +577,9 @@ type OutputContract struct {
|
|||||||
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
|
// SchemaPath is required when ValidationMode is ValidationJSONSchema and is
|
||||||
// ignored by other modes.
|
// ignored by other modes.
|
||||||
SchemaPath string `json:"schema_path"`
|
SchemaPath string `json:"schema_path"`
|
||||||
// RepairAttempts is a non-negative requested repair limit. Zero requests no
|
// RepairAttempts is an additional generation-call budget from zero through
|
||||||
// repairs. The public Engine performs no repairs even when this value is
|
// three. Zero is single-pass. A positive value is valid only with basic,
|
||||||
// positive, so its runs report zero attempts used.
|
// json, or json_schema validation.
|
||||||
RepairAttempts int `json:"repair_attempts"`
|
RepairAttempts int `json:"repair_attempts"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -595,8 +595,8 @@ type ValidationResult struct {
|
|||||||
Errors []string `json:"errors,omitempty"`
|
Errors []string `json:"errors,omitempty"`
|
||||||
// SchemaPath is the effective schema path for JSON Schema validation.
|
// SchemaPath is the effective schema path for JSON Schema validation.
|
||||||
SchemaPath string `json:"schema_path,omitempty"`
|
SchemaPath string `json:"schema_path,omitempty"`
|
||||||
// RepairAttempts is the number of repairs actually attempted. It is always
|
// RepairAttempts is the number of corrective generation calls actually
|
||||||
// zero for the public Engine.
|
// started for this result.
|
||||||
RepairAttempts int `json:"repair_attempts"`
|
RepairAttempts int `json:"repair_attempts"`
|
||||||
// IsValid is true for ValidationPassed and ValidationSkipped and false for
|
// IsValid is true for ValidationPassed and ValidationSkipped and false for
|
||||||
// ValidationFailed.
|
// ValidationFailed.
|
||||||
@@ -714,9 +714,8 @@ type GenerateRequest struct {
|
|||||||
// GenerateResponse is returned by an injected LLM client and has a stable JSON
|
// GenerateResponse is returned by an injected LLM client and has a stable JSON
|
||||||
// representation.
|
// representation.
|
||||||
type GenerateResponse struct {
|
type GenerateResponse struct {
|
||||||
// Content is the generated output. It must be non-empty when using the
|
// Content is the generated output. It may be explicitly empty; Promptkit
|
||||||
// built-in client; injected clients may return empty content for Promptkit
|
// applies the effective output contract to classify it.
|
||||||
// validation to classify.
|
|
||||||
Content string `json:"content"`
|
Content string `json:"content"`
|
||||||
// Usage is the client's token accounting.
|
// Usage is the client's token accounting.
|
||||||
Usage TokenUsage `json:"usage"`
|
Usage TokenUsage `json:"usage"`
|
||||||
|
|||||||
Reference in New Issue
Block a user