Prepare documentation for the v0.8.0 release
This commit is contained in:
109
docs/releases/v0.8.0.md
Normal file
109
docs/releases/v0.8.0.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Promptkit v0.8.0
|
||||
|
||||
This supplemental changelog and migration guide summarizes the consumer-facing
|
||||
changes from `v0.7.0` to `v0.8.0`. The annotated `v0.8.0` tag is the
|
||||
authoritative release record. Exact current contracts belong to the linked
|
||||
GoDoc and durable documentation.
|
||||
|
||||
## Summary
|
||||
|
||||
`v0.8.0` activates Promptkit's bounded output-repair workflow:
|
||||
|
||||
- failed nonempty-text, JSON, and JSON Schema validation can make a limited
|
||||
number of corrective model calls;
|
||||
- corrective calls preserve the original rendered conversation, effective
|
||||
target, session, structured-output contract, and backend capacity policy;
|
||||
- results report cumulative usage and the number of corrective calls actually
|
||||
made; and
|
||||
- explicitly empty OpenAI-compatible response content now reaches output
|
||||
validation instead of being classified as a malformed provider envelope.
|
||||
|
||||
## Compatibility
|
||||
|
||||
This release adds no public declarations or fields and removes none. Existing
|
||||
source code remains source-compatible.
|
||||
|
||||
The behavior of the existing `OutputContract.RepairAttempts` field and prompt
|
||||
YAML `repair_attempts` field has changed. A positive value now authorizes real
|
||||
additional model calls after eligible validation failures; earlier releases
|
||||
accepted the field but the public engine remained single-pass. Consumers that
|
||||
set a positive value should expect additional latency, token usage, and
|
||||
provider cost when repair is needed.
|
||||
|
||||
Repair budgets must now be between zero and three. A positive budget requires
|
||||
`basic`, `json`, or `json_schema` validation. Values above three and a positive
|
||||
budget paired with `none` are invalid contracts rather than ignored settings.
|
||||
|
||||
An explicitly present empty or whitespace-only string returned by the built-in
|
||||
OpenAI-compatible client is now a completed generation candidate. `none`
|
||||
validation permits it, while `basic`, `json`, and `json_schema` classify it
|
||||
under their ordinary validation rules and may repair it when configured.
|
||||
Missing, `null`, or non-string content remains a malformed provider response.
|
||||
|
||||
## Upgrade
|
||||
|
||||
Update the module dependency with:
|
||||
|
||||
```sh
|
||||
go get gitea.maximumdirect.net/eric/promptkit@v0.8.0
|
||||
go mod tidy
|
||||
```
|
||||
|
||||
Review every prompt definition and request override that sets a positive repair
|
||||
budget. Use zero or omit the field to retain single-pass execution. Ensure each
|
||||
positive budget is no greater than three and uses an eligible validation mode,
|
||||
then run the consuming project's ordinary and race-enabled tests.
|
||||
|
||||
## Bounded Output Repair
|
||||
|
||||
`repair_attempts` counts corrective calls in addition to the initial model
|
||||
call. Promptkit validates each completed candidate, stops at the first valid
|
||||
one, and never exceeds the configured bound. If every candidate remains
|
||||
invalid, the run completes successfully with the final candidate and its
|
||||
failed validation result rather than returning an operational error.
|
||||
|
||||
Each correction starts from the original rendered messages and includes only
|
||||
the latest invalid candidate and latest validation diagnostics. JSON Schema
|
||||
mode retains the provider-native structured-output request as its first line of
|
||||
defense. Promptkit performs only deterministic structural validation; a valid
|
||||
response is not necessarily factual or correct for an application's domain.
|
||||
|
||||
Usage in the final result is cumulative across the initial response and every
|
||||
completed corrective response. `ValidationResult.RepairAttempts` reports the
|
||||
number of corrective calls actually made. Corrective generation failures use
|
||||
the same public generation-error categories and structured provider details as
|
||||
an initial generation failure.
|
||||
|
||||
See the [output-contract format reference](../formats.md#output-contract), the
|
||||
[consumer repair example](../consumers/pkg-promptkit.md#repair-a-structured-result),
|
||||
and the [`OutputContract` and `ValidationResult` GoDoc](../../types.go) for the
|
||||
current contracts.
|
||||
|
||||
## Explicit Empty Content
|
||||
|
||||
The built-in OpenAI-compatible client now distinguishes an explicitly present
|
||||
empty string from a missing or malformed `content` field. This aligns built-in
|
||||
and injected clients by letting the selected output contract decide whether an
|
||||
empty candidate is acceptable, invalid, or eligible for repair.
|
||||
|
||||
See the
|
||||
[OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling)
|
||||
for the exact envelope behavior.
|
||||
|
||||
## Public API Changes
|
||||
|
||||
None. This release activates and tightens the documented behavior of existing
|
||||
fields.
|
||||
|
||||
## Consumer Action
|
||||
|
||||
- Remove or set `repair_attempts` to zero where execution must remain
|
||||
single-pass.
|
||||
- Keep every positive repair budget at three or fewer and pair it with
|
||||
`basic`, `json`, or `json_schema` validation.
|
||||
- Account for additional latency, usage, and provider cost when enabling
|
||||
repair.
|
||||
- Continue checking the returned validation status because bounded repair can
|
||||
exhaust without producing a valid candidate.
|
||||
- Review workflows that previously treated explicit empty provider content as
|
||||
a generation error.
|
||||
@@ -1,603 +0,0 @@
|
||||
# Public Bounded Output Repair Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement the feature defined by the
|
||||
[public bounded output repair roadmap](output-repair.md). This document is the
|
||||
decision-complete execution sequence for a coding agent. Follow the repository
|
||||
[architecture](../policy/architecture.md), [testing](../policy/testing.md), and
|
||||
[documentation](../policy/documentation.md) policies and the task-specific
|
||||
reading guide in the [development guide](../development.md) throughout the
|
||||
work.
|
||||
|
||||
## Target Outcome
|
||||
|
||||
Promptkit's public `Engine.Run` and `Engine.RunPrepared` workflows honor the
|
||||
existing repair budget in an effective output contract. A failed `basic`,
|
||||
`json`, or `json_schema` validation may make at most three additional calls
|
||||
through the same resolved target, session, credential boundary, prepared
|
||||
schema, and capacity-wrapped client. Each correction receives the original
|
||||
rendered conversation, the latest nonempty candidate, and bounded diagnostics.
|
||||
|
||||
The first valid candidate ends the operation. Exhaustion returns the final
|
||||
invalid candidate and complete final diagnostics as an ordinary result, while
|
||||
generation, validation, capacity, and cancellation failures retain their
|
||||
existing public error categories and identities. Zero remains single-pass, no
|
||||
public repair interface is added, and an explicitly present empty provider
|
||||
string reaches validation instead of being mistaken for a malformed envelope.
|
||||
|
||||
## Fixed Decisions
|
||||
|
||||
- `OutputContract.RepairAttempts` and prompt YAML `repair_attempts` are the
|
||||
only consumer controls. The value counts additional generation calls and
|
||||
must be in the inclusive range zero through three.
|
||||
- A positive budget with `none` validation is invalid. Positive budgets are
|
||||
eligible only after a completed failed validation under `basic`, `json`, or
|
||||
`json_schema`.
|
||||
- The public engine installs one internal default repairer. There is no new
|
||||
public option, interface, callback, prompt override, or mutable registry.
|
||||
- Every corrective call starts with a fresh copy of the complete original
|
||||
rendered messages. It appends only the latest nonempty candidate as an
|
||||
`assistant` message and one corrective `user` message. Earlier failed
|
||||
candidates and corrective messages do not accumulate.
|
||||
- Empty and whitespace-only candidates omit the additional assistant message.
|
||||
The corrective user message states that the prior response was empty.
|
||||
- The corrective instruction is application-neutral. `basic` requests a
|
||||
nonempty answer to the original task; `json` and `json_schema` request only
|
||||
corrected JSON without explanation or Markdown fences.
|
||||
- Validation diagnostics are represented as bounded data in the correction
|
||||
message. Their encoded feedback is at most 65,536 UTF-8 bytes, preserves
|
||||
order, and includes an in-bound omission notice when truncated. This bound
|
||||
never alters the complete diagnostics retained by validation or returned in
|
||||
the final result.
|
||||
- The latest nonempty candidate is included verbatim and in full. Promptkit
|
||||
adds no repair-specific candidate-size limit.
|
||||
- JSON Schema repair retains the exact prepared structured-output
|
||||
specification. Plain JSON does not gain a provider-native JSON-object
|
||||
constraint because its contract permits every JSON value.
|
||||
- Explicitly present string content from the built-in OpenAI-compatible client
|
||||
is a successful generation candidate even when empty or whitespace-only.
|
||||
Missing content, `null`, non-string content, and absent choices remain
|
||||
malformed provider responses and never enter repair.
|
||||
- The initial generation and every correction use the same capacity-wrapped
|
||||
`llm.Client`. One run-admission lease spans the complete operation, while
|
||||
each model call independently acquires and releases the selected backend's
|
||||
active-generation permit.
|
||||
- `ValidationResult.RepairAttempts` is execution outcome state and reports the
|
||||
number of corrective calls actually started. It is not initialized from the
|
||||
requested budget by the validation package.
|
||||
- Usage is the field-by-field sum of every completed generation. An
|
||||
operational error returns no partial `RunResult`; repair exhaustion is not
|
||||
an operational error.
|
||||
- A corrective generation failure is categorized exactly like an initial
|
||||
generation failure: invalid model requests match `ErrInvalidRequest`, other
|
||||
provider or transport failures match `ErrLLMGenerate`, typed generation
|
||||
details remain available, and cancellation and deadline identities remain
|
||||
discoverable.
|
||||
- No release notes, roadmap retirement, version changes, commits, tags, or
|
||||
pushes are part of these stages.
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- Complete the stages in numerical order. Each stage is sized for one
|
||||
gpt-5.6-terra implementation prompt and must finish its focused verification
|
||||
before the next begins.
|
||||
- At the start of each stage, inspect the working tree and preserve unrelated
|
||||
changes. The feature roadmap and this plan are intentional planning state.
|
||||
- Treat intermediate stages as an unreleased partial implementation. Durable
|
||||
current-state documentation changes belong to Stage 6, after public engine
|
||||
assembly and behavior are complete.
|
||||
- Keep source-neutral contract legality in `internal/domain`, structural
|
||||
validity in `internal/validate`, conversation construction in the default
|
||||
repairer, orchestration and error categorization in `internal/usecase`, wire
|
||||
envelope interpretation in `internal/llm`, and public assembly in the root
|
||||
package.
|
||||
- Do not move repair behavior into the root facade, validator, capacity
|
||||
manager, public client adapter, or provider transport merely to simplify a
|
||||
test.
|
||||
- Reuse the existing prepared validation plan and generation request
|
||||
constructor. Do not reopen prompt, profile, backend, artifact, or schema
|
||||
sources and do not rerender during correction.
|
||||
- Update existing test owners when a contract changes. Add only representative
|
||||
public workflow tests; do not duplicate the internal state-machine matrix at
|
||||
the root package.
|
||||
- Tests must be deterministic, offline, parallel-safe, race-safe, and
|
||||
independent of real credentials or provider services. Use injected fakes,
|
||||
synthetic filesystems, and `httptest` where provider-wire behavior matters.
|
||||
- Use `apply_patch` for edits, format every changed Go file, review the full
|
||||
stage diff, and run `git diff --check` before moving to the next stage.
|
||||
|
||||
## Stage 1: Enforce The Repair Contract At Its Source-Neutral Boundary
|
||||
|
||||
### Objective
|
||||
|
||||
Make every supported output-contract source agree on the zero-to-three budget
|
||||
and invalid `none` pairing, and establish that actual attempt accounting belongs
|
||||
to execution rather than validation. This stage changes legality and result
|
||||
ownership but does not activate repair in the public engine.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/domain/output_contract.go`, add one unexported
|
||||
`maxOutputRepairAttempts = 3` constant and extend
|
||||
`ValidateOutputContract` to reject:
|
||||
- negative `RepairAttempts`;
|
||||
- values greater than three; and
|
||||
- a positive value when `ValidationMode` is `ValidationNone`.
|
||||
2. Keep the existing format/mode/schema checks in the same helper. Do not clamp
|
||||
invalid budgets and do not add the maximum as a root-package public
|
||||
constant; exact public semantics will be documented through existing fields.
|
||||
3. Preserve the current normalization and replacement paths so prompt files,
|
||||
request-level `OutputContract` replacements, `Prepare`, and
|
||||
`PrepareExecution` all reach the shared domain helper. Do not add parallel
|
||||
range checks in the root facade or prompt repository.
|
||||
4. In `internal/validate/standard_validator.go`, stop copying the configured
|
||||
`contract.RepairAttempts` into a newly created `ValidationResult`. A direct
|
||||
validation result starts with zero attempts. Retain the existing use-case
|
||||
wrappers that assign the actual `attemptsUsed` value after each completed
|
||||
validation.
|
||||
5. Preserve output-contract whole-value replacement behavior and all existing
|
||||
validation modes, schema preparation, and error wrapping apart from the new
|
||||
invalid cases.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Extend `internal/domain/output_contract_test.go` with a focused boundary
|
||||
table covering `-1`, `0`, `1`, `3`, and `4`, plus `none` with zero and with
|
||||
a positive value. Use representative valid basic/JSON contracts so unrelated
|
||||
validation errors do not mask the intended assertion.
|
||||
2. Extend the existing prompt-definition repository tests with representative
|
||||
YAML cases proving a budget above three and a positive `none` pairing are
|
||||
rejected through the repository's existing invalid-prompt identity. Do not
|
||||
duplicate the full domain table there.
|
||||
3. Update `internal/usecase/output_contract_test.go` to prove a request
|
||||
replacement with either new invalid condition fails before completion
|
||||
collaborators or admission. Cover both ordinary and prepared preparation
|
||||
through the existing shared test structure instead of creating a second
|
||||
matrix.
|
||||
4. Extend the root output-contract contract test only as needed to preserve the
|
||||
established public error mapping for invalid request replacements. Source
|
||||
file failures retain their existing prompt-load category.
|
||||
5. Update validator tests to assert a direct failed or passed validation reports
|
||||
zero attempts even when the input contract carries a positive budget. Keep
|
||||
execution-level attempt assertions in use-case tests.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/domain/output_contract.go internal/domain/output_contract_test.go
|
||||
gofmt -w internal/validate/standard_validator.go internal/validate/standard_validator_test.go
|
||||
gofmt -w internal/promptdef/repository_test.go internal/usecase/output_contract_test.go
|
||||
gofmt -w output_contract_contract_test.go
|
||||
go test ./internal/domain ./internal/validate ./internal/promptdef ./internal/usecase
|
||||
go test ./... -run 'OutputContract|ValidationResult'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 1 is complete when every input path accepts only a coherent zero-to-three
|
||||
budget, invalid contracts fail before model work, and validators no longer
|
||||
misreport a configured budget as completed repair work.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 2: Distinguish Explicit Empty Content From A Malformed Envelope
|
||||
|
||||
### Objective
|
||||
|
||||
Allow the built-in OpenAI-compatible client to return an explicitly present
|
||||
empty or whitespace string as a generation candidate while retaining strict
|
||||
malformed-response handling for absent or wrongly typed content.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/llm/openai_compatible_client.go`, make
|
||||
`openAIChatResponseMessage.Content` presence-aware by decoding it as a
|
||||
`*string`.
|
||||
2. After response framing and JSON decoding succeed, require at least one
|
||||
choice and a non-nil first-choice content pointer. Dereference and return
|
||||
the string exactly as received without trimming or rejecting an empty
|
||||
value.
|
||||
3. Treat a missing field and explicit `null` as malformed because both decode
|
||||
to nil. Let a non-string value fail JSON decoding and remain wrapped with
|
||||
`ErrMalformedResponse`. Preserve the existing no-choice, trailing-data,
|
||||
response-size, usage, provider-error, and body-ownership behavior.
|
||||
4. Do not perform output validation or decide repair eligibility in the model
|
||||
client. This stage changes only the interpretation of an otherwise
|
||||
successful completion envelope.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Add focused cases to `internal/llm/openai_compatible_client_test.go` proving
|
||||
explicit `""` and whitespace-only string content return successful
|
||||
`GenerateResponse` values with the exact content and mapped usage.
|
||||
2. Add or retain distinct malformed-response cases for no choices, missing
|
||||
content, `null` content, and non-string content. Each must return no response
|
||||
and match `ErrMalformedResponse`.
|
||||
3. Keep these tests at the model-client owner. Do not use a root engine or
|
||||
duplicate validation-mode behavior in this stage.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/llm/openai_compatible_client.go internal/llm/openai_compatible_client_test.go
|
||||
go test ./internal/llm -run 'OpenAICompatibleClient|Response'
|
||||
go test ./internal/llm
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 2 is complete when content presence is distinguishable from content
|
||||
emptiness and all malformed successful-envelope cases retain their prior error
|
||||
identity.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 3: Build Full-Context, Prompt-Safe Corrective Requests
|
||||
|
||||
### Objective
|
||||
|
||||
Refactor the dormant default repairer so it constructs one correction from the
|
||||
original rendered conversation and latest validation state, supports basic and
|
||||
JSON modes, and safely bounds only diagnostic feedback.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Extend the internal `RepairRequest` in `internal/usecase/repairer.go` with
|
||||
`OriginalMessages []domain.RenderedMessage`. Keep the type internal and do
|
||||
not add repair fields to `internal/domain` or the public package.
|
||||
2. Replace the standalone system/user repair prompt with a freshly allocated
|
||||
message slice on every call:
|
||||
- copy every original rendered message in order without changing role,
|
||||
content, or cache-control metadata;
|
||||
- if `strings.TrimSpace(PreviousOutput)` is nonempty, append one `assistant`
|
||||
message whose content is the exact, full `PreviousOutput` string;
|
||||
- otherwise append no assistant message; and
|
||||
- append one corrective `user` message after the original conversation and
|
||||
optional candidate.
|
||||
Allocate independently from `OriginalMessages` so `append` cannot mutate a
|
||||
prepared snapshot's backing array.
|
||||
3. Construct one stable application-neutral corrective message that includes
|
||||
the validation mode, `Attempt` and `MaxAttempts`, an instruction to preserve
|
||||
valid values and change only what is necessary, and diagnostics clearly
|
||||
labeled as data rather than instructions. For an empty/whitespace candidate,
|
||||
explicitly state that the previous response was empty and do not reproduce
|
||||
its whitespace.
|
||||
4. Use mode-specific terminal guidance:
|
||||
- `ValidationBasic`: return a nonempty response satisfying the original
|
||||
request; and
|
||||
- `ValidationJSON` or `ValidationJSONSchema`: return only corrected JSON,
|
||||
with no explanation or Markdown fence.
|
||||
Return a defensive internal error for any other mode; the runner must never
|
||||
call the repairer for it after Stage 4.
|
||||
5. Add an unexported `maxRepairDiagnosticBytes = 64 * 1024` constant and one
|
||||
focused diagnostic formatter. Encode the ordered diagnostics as a JSON array
|
||||
within the corrective message so provider-controlled text remains visibly
|
||||
data. The formatter must:
|
||||
- convert malformed input strings to valid UTF-8 for prompt transport;
|
||||
- return the complete encoded array when it is within the limit;
|
||||
- when over the limit, preserve complete earlier entries, include as much
|
||||
of the next entry as fits at a rune boundary, omit all remaining text, and
|
||||
append a stable final array entry stating that additional validation
|
||||
diagnostics were omitted;
|
||||
- account for JSON quoting, delimiters, and the omission entry when enforcing
|
||||
the 65,536-byte maximum; and
|
||||
- return a valid JSON array at or below the limit for every input, including
|
||||
one oversized error, many errors, control characters, multibyte runes, and
|
||||
an empty list.
|
||||
This limit applies only to the encoded diagnostic block, not fixed repair
|
||||
instructions, the previous candidate, or `ValidationResult.Errors`.
|
||||
6. Continue to call `newGenerationRequest` with the request's exact session,
|
||||
target, target-presence metadata, and structured-output pointer. Preserve
|
||||
the existing nil-client and nil-response defenses. Do not locally strip,
|
||||
parse, normalize, or repair the candidate.
|
||||
|
||||
### Tests
|
||||
|
||||
Create `internal/usecase/repairer_test.go` as the narrow owner of default
|
||||
repairer behavior:
|
||||
|
||||
1. Supply original system and user messages, including cache-control metadata,
|
||||
and assert the generated request contains those values unchanged followed
|
||||
by the exact assistant candidate and one corrective user message.
|
||||
2. Prove separate correction calls begin from the same original messages and
|
||||
do not mutate the caller's slice or accumulate an earlier candidate or
|
||||
corrective message.
|
||||
3. Cover empty and whitespace-only candidates: no assistant message is added,
|
||||
the user message identifies the empty response, and no placeholder model
|
||||
content is invented.
|
||||
4. Cover basic, JSON, and JSON Schema guidance by stable behavioral phrases and
|
||||
role boundaries rather than snapshotting the full incidental prose. Assert
|
||||
an unsupported mode returns an error without calling the client.
|
||||
5. Prove the exact target, presence flags, session ID, and structured-output
|
||||
specification reach the model client and that an arbitrarily large nonempty
|
||||
candidate is not truncated.
|
||||
6. Exercise diagnostics below, at, and above the limit, including ordered
|
||||
entries, multibyte UTF-8 near the boundary, invalid UTF-8, and one huge
|
||||
diagnostic. Assert the encoded block is valid JSON, valid UTF-8, no more
|
||||
than 65,536 bytes, ordered, and contains the omission notice only when
|
||||
needed. Also assert the input error slice and strings remain unchanged.
|
||||
7. Retain focused nil-client, generation-error passthrough, and nil-response
|
||||
tests without duplicating runner error categorization.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/repairer.go internal/usecase/repairer_test.go
|
||||
go test ./internal/usecase -run 'Repairer|RepairRequest'
|
||||
go test ./internal/usecase
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 3 is complete when the default repairer produces bounded, mode-correct,
|
||||
full-context requests without mutating prepared state or retaining failed
|
||||
history.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 4: Complete The Internal Repair State Machine And Error Semantics
|
||||
|
||||
### Objective
|
||||
|
||||
Connect the improved repairer contract to runner execution, make `basic`
|
||||
eligible, and ensure corrective generation failures share the initial
|
||||
generation error path while preserving bounded state, usage, prepared
|
||||
validation, admission, and capacity behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/usecase/runner.go`, add one private generation-error wrapper
|
||||
used by both the initial `llm.Generate` call and every repair call:
|
||||
- if the error matches `llm.ErrInvalidRequest`, wrap it with
|
||||
`ErrInvalidRequest`; and
|
||||
- otherwise wrap it with `ErrLLMGenerate`.
|
||||
Use `%w` wrapping so provider details, cancellation, and deadlines remain
|
||||
discoverable. Do not classify a failed corrective model call as
|
||||
`ErrValidation`.
|
||||
2. Treat a nil repair response without an error as a generation-side failure
|
||||
matching `ErrLLMGenerate`. Continue returning no partial result for that or
|
||||
any other operational error.
|
||||
3. Pass `prepared.Messages` as `RepairRequest.OriginalMessages` on every
|
||||
attempt. Continue passing the latest response content and latest complete
|
||||
validation error slice, so each attempt repairs only the newest candidate.
|
||||
4. Extend `shouldAttemptRepair` to include `ValidationBasic`. Retain all other
|
||||
gates: a configured repairer, positive budget, and
|
||||
`ValidationFailed`. `ValidationNone`, passed/skipped validation, and
|
||||
operational validator errors never reach the repairer.
|
||||
5. Preserve the existing loop shape and attempt semantics: increment the
|
||||
attempt count immediately before starting a corrective call, validate every
|
||||
completed response with the retained operation-local plan, stop on the first
|
||||
valid response, and make no more calls than the frozen budget.
|
||||
6. Preserve final-result coherence and cumulative usage. A successful or
|
||||
exhausted run uses the same final candidate for `RawOutput`, artifact, and
|
||||
validation; sums all five token-usage fields across completed generations;
|
||||
reports actual calls in `ValidationResult.RepairAttempts`; and returns the
|
||||
complete latest diagnostics on exhaustion.
|
||||
7. Do not change admission or capacity architecture. The admission release
|
||||
remains deferred across the full execution, and the default repairer must
|
||||
continue using the same capacity-wrapped client so each correction acquires
|
||||
the normal active-generation permit without a second run admission.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Refactor `TestRunnerRepairStateMachine` rather than adding a parallel
|
||||
matrix:
|
||||
- change the obsolete “basic failure is ineligible” case into an empty/basic
|
||||
repair success or bounded exhaustion case;
|
||||
- keep initial success, stop-on-valid, exact exhaustion, JSON, JSON Schema,
|
||||
target-presence, structured-output, session, direct credential, final
|
||||
candidate, and cumulative-usage assertions;
|
||||
- replace the now-invalid budget-four fixture with a maximum-three fixture;
|
||||
- assert each `RepairRequest` receives the original rendered messages and
|
||||
only the latest output/errors; and
|
||||
- update repair-call detection helpers that currently recognize the old
|
||||
standalone system prompt so they recognize the appended assistant/user
|
||||
shape without depending on exact correction prose.
|
||||
2. Add a zero-budget failed-validation case if the existing public test is its
|
||||
only owner, ensuring the internal runner remains single-pass without invoking
|
||||
a configured repairer. A positive `none` case belongs to Stage 1 contract
|
||||
validation, not this state machine.
|
||||
3. Update prepared-execution tests to prove initial and repaired candidates use
|
||||
the same frozen validation plan and messages, sources are not reopened, and
|
||||
the handle remains one-shot.
|
||||
4. Update execution-error tables so a repairer generation failure expects
|
||||
`ErrLLMGenerate`; add a repair error matching `llm.ErrInvalidRequest` and
|
||||
assert it maps to `ErrInvalidRequest`. Preserve underlying sentinel,
|
||||
cancellation, and deadline identity and confirm all operational failures
|
||||
return no partial result.
|
||||
5. Retain and adapt the shared-backend-pool test to prove two repaired runs
|
||||
each receive one admission, all initial and repair calls use the same
|
||||
backend pool, peak active generation does not exceed the configured limit,
|
||||
and every lease/permit is released.
|
||||
6. Add a focused cancellation case only if existing capacity coverage does not
|
||||
already prove cancellation while a corrective call waits for its active
|
||||
permit prevents that provider call and releases the run admission. Avoid
|
||||
duplicating `internal/capacity`'s scheduler matrix.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/runner.go internal/usecase/runner_test.go
|
||||
gofmt -w internal/usecase/prepared_execution_test.go
|
||||
go test ./internal/usecase -run 'Repair|Prepared|Admission|Capacity|Cancellation'
|
||||
go test -race ./internal/usecase -run 'Repair|Capacity|Cancellation'
|
||||
go test ./internal/usecase
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 4 is complete when the internal state machine satisfies every bounded
|
||||
success, exhaustion, error, prepared-state, and concurrency invariant without
|
||||
being publicly activated yet.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 5: Activate Repair Through Public Engine Assembly
|
||||
|
||||
### Objective
|
||||
|
||||
Make the completed behavior available through the existing public API for both
|
||||
built-in and injected clients, and align exported Go contracts with the now
|
||||
active behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `engine.go`, after constructing or adapting the selected `llm.Client` and
|
||||
wrapping it with `capacity.NewClient`, construct
|
||||
`usecase.NewDefaultOutputRepairer(llmClient)` and assemble the engine runner
|
||||
with `usecase.NewRunnerWithRepairer`. Pass that exact same wrapped client as
|
||||
the runner's initial-generation client and the repairer's client.
|
||||
2. Do not add a public engine option or expose the internal repairer. Keep
|
||||
`usecase.NewRunner` as the no-repair convenience constructor for focused
|
||||
internal tests and callers; the root facade deliberately uses the explicit
|
||||
repairer constructor.
|
||||
3. Update exported GoDoc in `types.go`:
|
||||
- `OutputContract.RepairAttempts` is an additional-call budget from zero
|
||||
through three and is valid only with `basic`, `json`, or `json_schema` when
|
||||
positive;
|
||||
- `ValidationResult.RepairAttempts` is the actual number of corrective calls
|
||||
started and may be nonzero in public results; and
|
||||
- `LLMResponse.Content` may be explicitly empty and will be interpreted by
|
||||
the effective output contract rather than rejected by Promptkit's public
|
||||
adapter.
|
||||
4. Update `Engine.Run`, `Engine.RunPrepared`, and related prepared/result GoDoc
|
||||
in `engine.go` to describe opt-in bounded correction, cumulative usage,
|
||||
exhaustion as a returned failed validation result, operational errors as no
|
||||
result, cancellation, and one-shot prepared execution. Remove statements
|
||||
that the public engine never repairs or is always single-pass.
|
||||
5. Keep the public JSON representation and public type set unchanged. Do not
|
||||
expose correction messages, candidate history, a semantic-validity claim, or
|
||||
a new error sentinel.
|
||||
|
||||
### Tests
|
||||
|
||||
Use root external-package contract tests for representative assembly behavior,
|
||||
reusing existing public fakes where possible:
|
||||
|
||||
1. Replace the obsolete public “positive budget is still single-pass” assertion
|
||||
with a zero-budget failed-validation assertion that remains exactly one
|
||||
generation and reports zero attempts.
|
||||
2. Add one ordinary `Engine.Run` workflow with an injected `LLMClient` that
|
||||
returns invalid JSON and then valid JSON. Assert two calls, the final valid
|
||||
candidate, one actual repair, cumulative usage, the same target/session and
|
||||
structured-output values, and no new configuration surface.
|
||||
3. Add one `PrepareExecution`/`RunPrepared` workflow using a different eligible
|
||||
mode, preferably `basic` with an explicit empty initial candidate, to prove
|
||||
public prepared parity, frozen messages/settings, and one-shot behavior.
|
||||
4. Add one bounded exhaustion assertion that returns a non-nil result, nil
|
||||
error, final invalid candidate, complete final diagnostics, exact actual
|
||||
attempt count, and cumulative usage. Keep the full progression matrix in
|
||||
`internal/usecase`.
|
||||
5. Use an offline `httptest` OpenAI-compatible endpoint for one built-in-client
|
||||
integration path: return a structurally valid completion envelope whose
|
||||
content fails validation, then a provider error on correction. Assert no
|
||||
partial result, `ErrLLMGenerate`, available public `GenerationError` details,
|
||||
and preserved underlying cancellation/deadline behavior where applicable.
|
||||
Do not duplicate every provider failure shape already owned by
|
||||
`internal/llm`.
|
||||
6. Ensure existing capacity tests for injected clients still pass with the
|
||||
repairer installed, especially engine-local backend limits and unlimited
|
||||
backend behavior when repair is not requested.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w engine.go types.go output_contract_contract_test.go public_contract_test.go
|
||||
gofmt -w prepared_execution_contract_test.go engine_test.go
|
||||
go test ./... -run 'Repair|OutputContract|RunPrepared|GenerationError|Capacity'
|
||||
go test -race ./... -run 'Repair|RunPrepared|Capacity'
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
If a named root test file does not require a change, omit it from `gofmt`
|
||||
rather than touching it mechanically. Stage 5 is complete when ordinary and
|
||||
prepared public workflows repair through either client path with unchanged API
|
||||
shape and correct public result and error semantics.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Stage 6: Publish Canonical Documentation And Run Maintainer Validation
|
||||
|
||||
### Objective
|
||||
|
||||
Make every canonical documentation owner describe the implemented feature,
|
||||
remove obsolete dormant/single-pass claims, and validate the complete
|
||||
cross-cutting change using the repository's canonical workflow.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Update `docs/formats.md`, the exact prompt-format owner, to document:
|
||||
- `repair_attempts` as an integer from zero through three;
|
||||
- zero as the single-pass default and positive values as additional calls;
|
||||
- eligible `basic`, `json`, and `json_schema` modes;
|
||||
- invalid positive budgets with `none`;
|
||||
- stop-on-valid and exhausted-final-result behavior; and
|
||||
- plain JSON's acceptance of every JSON value without an object-only wire
|
||||
constraint.
|
||||
2. Update `docs/consumers/pkg-promptkit.md` with one concise JSON Schema repair
|
||||
example using the existing public field or prompt YAML. Explain cumulative
|
||||
usage, actual attempt reporting, the possibility of a final failed
|
||||
validation after exhaustion, and the cost/latency implications of extra
|
||||
calls. Note briefly that `basic` can repair empty output and that structural
|
||||
validity is not factual or domain correctness.
|
||||
3. Update `docs/integrations/openai-compatible-chat.md` to own the provider-wire
|
||||
distinction: an explicitly present string `content`, including empty or
|
||||
whitespace-only, is a candidate; absent choices, missing content, `null`,
|
||||
and non-string content are malformed. Retain provider-native JSON Schema as
|
||||
the first defense and do not claim an object-only constraint for plain JSON.
|
||||
4. Update `docs/internal/llm.md` with the same internal response-decoding
|
||||
boundary without duplicating public repair orchestration.
|
||||
5. Update `docs/internal/runner.md` to describe the installed default repairer,
|
||||
full-original-message/latest-candidate request shape, empty-candidate role
|
||||
behavior, diagnostic bound, eligible modes, attempt/usage accounting,
|
||||
exhaustion, generation-versus-validation error categories, prepared-plan
|
||||
reuse, and one-admission/shared-capacity behavior. Remove the claim that the
|
||||
ordinary public runner omits a repairer.
|
||||
6. Update `docs/internal/overview.md` so root assembly and `internal/usecase`
|
||||
ownership reflect public bounded repair. Review `docs/internal/capacity.md`
|
||||
and `docs/internal/sources.md`; edit only statements that are stale or
|
||||
incomplete after implementation, preserving their narrow ownership.
|
||||
7. Do not put canonical behavior in a release note or roadmap. Do not update
|
||||
the architecture policy unless implementation actually crossed the package
|
||||
boundaries fixed by this plan. Do not retire `output-repair.md` or this plan
|
||||
in this implementation sequence.
|
||||
8. Check every changed Markdown link and heading fragment with the canonical
|
||||
offline link checker in the development guide. Review the final diff for
|
||||
consistent terminology: “repair attempt” always means an additional model
|
||||
call, “exhaustion” returns a completed failed validation result, and no text
|
||||
promises semantic correctness or provider caching.
|
||||
|
||||
### Tests And Final Verification
|
||||
|
||||
Run the complete
|
||||
[maintainer validation workflow](../development.md#maintainer-validation) from
|
||||
the repository root, including both maintained offline examples:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
go run ./examples/go-library/run
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Also run the development guide's tracked-Go formatting check and local
|
||||
Markdown-link checker exactly as documented. Review example JSON output for
|
||||
the invariants listed there; neither example may contact a real provider or
|
||||
require credentials.
|
||||
|
||||
Stage 6 is complete when code, tests, exported GoDoc, format documentation,
|
||||
consumer guidance, integration contracts, and internal documentation agree;
|
||||
all canonical validation commands pass; and the working tree contains only the
|
||||
intended feature implementation and roadmap changes.
|
||||
|
||||
**Status:** Complete.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and the fixed decisions above define all behavior
|
||||
needed for implementation.
|
||||
@@ -1,361 +0,0 @@
|
||||
# Public Bounded Output Repair Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
Make Promptkit's existing bounded output-repair capability available through
|
||||
the public engine. Consumers that require nonempty text, JSON, or JSON Schema
|
||||
output should be able to ask Promptkit to make a limited number of corrective
|
||||
model calls after content validation fails, without rebuilding Promptkit's
|
||||
generation, validation, capacity, credential, session, and result-accounting
|
||||
workflow.
|
||||
|
||||
The feature improves basic content and structural reliability, not domain
|
||||
correctness. Promptkit can determine whether output is nonempty, whether it is
|
||||
syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It
|
||||
cannot determine whether otherwise valid content is factual, complete, useful,
|
||||
or semantically correct for a downstream application.
|
||||
|
||||
## Target End State
|
||||
|
||||
- The existing `repair_attempts` prompt field and
|
||||
`OutputContract.RepairAttempts` request field control public repair behavior.
|
||||
Zero remains the default and makes execution single-pass.
|
||||
- A repair budget may be one, two, or three corrective calls. Values above
|
||||
three are invalid rather than silently clamped, providing a framework-level
|
||||
bound on accidental cost and latency.
|
||||
- A positive repair budget is eligible when the effective validation mode is
|
||||
`basic`, `json`, or `json_schema` and the latest completed content validation
|
||||
has failed. `none` explicitly permits empty content, and pairing it with a
|
||||
positive repair budget is an invalid output contract.
|
||||
- `Engine.Run` and `Engine.RunPrepared` provide the same repair behavior for
|
||||
the built-in OpenAI-compatible client and an injected `LLMClient`.
|
||||
- Provider-native structured-output metadata remains the first line of
|
||||
defense for JSON Schema generation. Repair applies only after the resulting
|
||||
content still fails Promptkit's own validation.
|
||||
- Promptkit stops at the first contract-valid response or after the
|
||||
requested bound is exhausted. It never reports an invalid response as valid.
|
||||
- A completed exhausted run returns the final invalid candidate and its final
|
||||
validation diagnostics in the ordinary `RunResult`; exhaustion is not an
|
||||
operational error.
|
||||
- Results report the number of corrective calls actually attempted and the
|
||||
cumulative token usage reported by the initial generation and every
|
||||
completed repair generation.
|
||||
- Repair remains internal orchestration. Consumers do not need to construct or
|
||||
register a repairer, and this feature does not add a public repair strategy
|
||||
interface.
|
||||
|
||||
## Eligibility And Contract Semantics
|
||||
|
||||
Repair is opt-in for each effective output contract. The prompt definition may
|
||||
declare a budget, and a request-level `OutputContract` may replace that whole
|
||||
contract under the existing replacement semantics. Preparation validates and
|
||||
freezes the effective budget along with the validation mode and schema plan.
|
||||
|
||||
Only a completed `ValidationFailed` result under `basic`, `json`, or
|
||||
`json_schema` can start or continue repair. The following do not trigger
|
||||
repair:
|
||||
|
||||
- `none` validation;
|
||||
- an initially valid response;
|
||||
- schema loading, decoding, registration, or compilation failures;
|
||||
- an operational inability to execute validation;
|
||||
- invalid requests, provider failures, transport failures, or cancellation;
|
||||
and
|
||||
- capacity-admission failures.
|
||||
|
||||
Selecting `none` is the explicit way for a caller to allow empty output and
|
||||
requires a zero repair budget. Rejecting a positive budget with `none` avoids a
|
||||
contradictory configuration whose requested corrective calls could never be
|
||||
eligible. Under `basic`, an empty or whitespace-only candidate fails validation
|
||||
and becomes repairable. Under `json` and `json_schema`, empty content is
|
||||
already invalid JSON and follows the same repair path as other JSON failures.
|
||||
|
||||
The requested budget must be between zero and three. Output-contract validation
|
||||
rejects negative values and values above three at the same source-neutral
|
||||
boundary for prompt definitions, request replacements, ordinary preparation,
|
||||
and prepared execution. The effective value is never silently clamped.
|
||||
|
||||
`repair_attempts` counts additional model calls after the initial generation,
|
||||
not total calls. `ValidationResult.RepairAttempts` reports calls actually
|
||||
started. A response that validates on the first corrective call therefore
|
||||
reports one repair attempt.
|
||||
|
||||
## Repair Request Behavior
|
||||
|
||||
Each corrective call extends the original rendered conversation rather than
|
||||
replacing it with a standalone repair prompt. For a nonempty candidate,
|
||||
Promptkit constructs the request from:
|
||||
|
||||
1. the complete original rendered messages in their original order;
|
||||
2. one additional `assistant` message containing the latest invalid candidate;
|
||||
and
|
||||
3. one additional `user` message containing Promptkit's application-neutral
|
||||
correction instruction and the latest validation diagnostics.
|
||||
|
||||
For an empty or whitespace-only candidate, Promptkit omits the additional
|
||||
`assistant` message and appends only the corrective `user` message. This avoids
|
||||
sending an empty assistant message that a compatible provider may reject and
|
||||
does not invent placeholder model content. The corrective message states that
|
||||
the previous response was empty.
|
||||
|
||||
The corrective user message always:
|
||||
|
||||
- identifies the effective validation mode and the current attempt number;
|
||||
- supplies the current validation diagnostics as data;
|
||||
- asks the model to preserve valid values and change only what is necessary to
|
||||
satisfy the effective output contract; and
|
||||
- uses clear boundaries so provider-controlled diagnostics are not confused
|
||||
with Promptkit's instructions.
|
||||
|
||||
The mode-specific instruction reflects the contract Promptkit can actually
|
||||
enforce:
|
||||
|
||||
- for `basic`, it asks for a nonempty response that satisfies the original
|
||||
request; and
|
||||
- for `json` and `json_schema`, it asks for only corrected JSON, without
|
||||
Markdown fences or explanation.
|
||||
|
||||
Representing the candidate as an `assistant` message keeps model output
|
||||
separate from Promptkit's corrective `user` instruction and gives the model
|
||||
access to the original task and source material when a missing or invalid
|
||||
field must be regenerated. Promptkit does not flatten the original messages or
|
||||
embed the candidate into the corrective message.
|
||||
|
||||
Each attempt repairs only the latest candidate. Promptkit does not append the
|
||||
entire history of earlier candidates, which would increase cost without adding
|
||||
equivalent corrective value and could make the active correction ambiguous.
|
||||
Every attempt starts again from the original messages and appends only the
|
||||
latest nonempty candidate, when present, and latest diagnostics.
|
||||
|
||||
JSON Schema mode retains the prepared structured-output specification, so a
|
||||
compatible client receives the same provider-native schema contract on the
|
||||
initial call and every corrective call. This preventive constraint remains
|
||||
the first line of defense even though Promptkit independently validates the
|
||||
returned content and can repair failures from providers that ignore, partially
|
||||
implement, or fail to satisfy it.
|
||||
|
||||
Plain `json` validation continues to permit any JSON value, including an
|
||||
object, array, string, number, boolean, or `null`. Promptkit must not send a
|
||||
provider-native JSON-object constraint for that mode because doing so would
|
||||
silently narrow the declared validation contract. A future contract may add
|
||||
provider-native object generation if consumers need that distinct behavior.
|
||||
|
||||
Resending the original messages may allow a provider to reuse cached prompt
|
||||
prefixes, especially when the original cache-control metadata is preserved.
|
||||
Promptkit does not assume, require, measure, or promise provider caching, and
|
||||
usage and cost remain whatever the selected client reports or charges.
|
||||
|
||||
When it is not empty or whitespace-only, the latest candidate is included in
|
||||
full and is never truncated or rewritten by Promptkit. The built-in client
|
||||
already bounds successful provider response bodies; consumers that inject an
|
||||
`LLMClient` own that client's response-size policy. Repair adds no smaller
|
||||
candidate limit or silent size-based skip.
|
||||
|
||||
Validation diagnostics included in the corrective message are limited to
|
||||
65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the
|
||||
diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a
|
||||
bounded notice that additional diagnostic text was omitted. This prompt-safety
|
||||
limit does not alter the complete diagnostics returned in the final
|
||||
`ValidationResult`.
|
||||
|
||||
## Empty Content And Provider Responses
|
||||
|
||||
Content presence belongs to validation when the provider has returned an
|
||||
otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible
|
||||
client therefore distinguishes these cases:
|
||||
|
||||
- a first choice with an explicitly present string `content` value is a
|
||||
completed generation candidate even when that string is empty or consists
|
||||
only of whitespace; and
|
||||
- no choices, a missing `content` field, `null` content, or non-string content
|
||||
remains a malformed provider response and returns a generation error without
|
||||
entering content repair.
|
||||
|
||||
This distinction keeps the built-in and injected-client paths coherent. An
|
||||
explicit empty candidate reaches `none`, `basic`, `json`, or `json_schema`
|
||||
validation under the same rules regardless of which client produced it. A
|
||||
malformed response envelope is not reclassified as invalid model content and
|
||||
does not consume the repair budget.
|
||||
|
||||
The initial and corrective calls use the same:
|
||||
|
||||
- resolved backend identity and endpoint;
|
||||
- model and effective execution settings, including presence-aware numeric and
|
||||
reasoning values;
|
||||
- direct credential or environment lookup boundary;
|
||||
- effective session ID;
|
||||
- structured-output specification; and
|
||||
- capacity manager and selected backend pool.
|
||||
|
||||
Repair does not reload prompt, profile, backend, input, or schema sources and
|
||||
does not rerender the original prompt. It reuses the already rendered message
|
||||
snapshot, including its roles, contents, and cache-control metadata. A
|
||||
prepared execution uses its retained messages, validation plan, and frozen
|
||||
effective target throughout repair.
|
||||
|
||||
## Results And Failure Behavior
|
||||
|
||||
After every successful corrective generation, Promptkit rebuilds the output
|
||||
artifact from that candidate and validates it with the operation's existing
|
||||
validation plan. The final `RawOutput`, artifact, validation result, and repair
|
||||
attempt count all describe the same final candidate.
|
||||
|
||||
Token usage is accumulated field by field across the initial response and each
|
||||
completed corrective response. Timing covers the entire operation, including
|
||||
repair. Prepared execution remains one-shot, and its handle is consumed by the
|
||||
single execution even when that execution performs several generation calls.
|
||||
|
||||
A failure while making a corrective model call retains the same public error
|
||||
category and underlying identity it would have had during initial generation:
|
||||
|
||||
- invalid model requests match `ErrInvalidRequest`;
|
||||
- provider and transport generation failures match `ErrLLMGenerate`, including
|
||||
any available `GenerationError` details from the built-in client; and
|
||||
- cancellation and deadlines remain discoverable through the error chain.
|
||||
|
||||
An operational failure while validating a corrective response continues to
|
||||
match `ErrValidation`. These failures return no partial `RunResult` or partial
|
||||
usage, consistent with existing operation failure behavior. Merely exhausting
|
||||
the content-repair budget is different: validation completed successfully, so
|
||||
the engine returns the final failed validation result rather than an error.
|
||||
|
||||
## Capacity And Concurrency
|
||||
|
||||
The ordinary run-admission lease spans preparation completion, initial
|
||||
generation, validation, all corrective calls, and every exit. Repair never
|
||||
performs a second admission. Each corrective generation uses the same
|
||||
capacity-wrapped model client and independently reacquires an active-generation
|
||||
permit for its backend, allowing unrelated accepted work to use the pool while
|
||||
the repairing run validates or waits.
|
||||
|
||||
Cancellation while waiting for a repair-generation permit prevents that model
|
||||
call when cancellation wins the grant race. All admission and generation
|
||||
permits must be released on successful repair, exhaustion, provider failure,
|
||||
validation failure, and cancellation.
|
||||
|
||||
## Architecture And Ownership
|
||||
|
||||
The root facade installs Promptkit's default output repairer when assembling
|
||||
the runner. It supplies the same capacity-wrapped `llm.Client` used for initial
|
||||
generation, whether that client ultimately delegates to Promptkit's built-in
|
||||
transport or a consumer-injected `LLMClient`.
|
||||
|
||||
`internal/usecase` continues to own the bounded repair state machine, attempt
|
||||
accounting, final-result selection, cumulative usage, error categorization,
|
||||
and coordination with prepared validation. The default repairer owns only the
|
||||
construction of the augmented conversation and the corrective generation
|
||||
call. `internal/validate` remains the sole authority for structural validity,
|
||||
while `internal/llm` retains responsibility for distinguishing explicit empty
|
||||
content from an absent or malformed completion and `internal/capacity` retains
|
||||
its scheduling responsibility.
|
||||
|
||||
No repair-specific type needs to enter `internal/domain` beyond the existing
|
||||
output-contract and validation-result state, and no public package or mutable
|
||||
engine registry is introduced.
|
||||
|
||||
## Public And Documentation Surface
|
||||
|
||||
No new public method or option is required. The feature activates the existing
|
||||
public fields and updates their contracts:
|
||||
|
||||
- `OutputContract.RepairAttempts` describes the requested additional-call
|
||||
budget, its zero-to-three range, and eligible validation modes;
|
||||
- `ValidationResult.RepairAttempts` describes the attempts actually made; and
|
||||
- `Run`, `RunPrepared`, and related result GoDoc describe exhaustion, failure,
|
||||
usage, cancellation, and prepared-execution behavior.
|
||||
|
||||
The framework format reference owns the exact
|
||||
`repair_attempts` YAML semantics and the meaning of `none`, `basic`, `json`,
|
||||
and `json_schema`. Consumer guidance contains one concise JSON Schema
|
||||
example, note that `basic` can repair empty output, and emphasize that content
|
||||
presence or structural validity does not establish domain correctness. The
|
||||
internal runner document owns orchestration and test ownership. The
|
||||
OpenAI-compatible integration contract owns the explicit-empty versus missing
|
||||
content distinction. Architecture policy needs to change only if implementation
|
||||
changes its existing package boundaries.
|
||||
|
||||
Canonical current-state documentation describes implemented behavior in
|
||||
accordance with the documentation policy. Roadmap material remains temporary
|
||||
and is retired when it no longer describes future work.
|
||||
|
||||
## Verification Expectations
|
||||
|
||||
Testing should preserve the repository's lean ownership model and build on the
|
||||
existing internal repair coverage:
|
||||
|
||||
- domain and source-boundary tests own accepted repair budgets and output-
|
||||
contract validation, including the upper bound and invalid `none` pairing;
|
||||
- use-case tests own eligibility, bounded progression, latest-candidate
|
||||
feedback, original-message preservation, non-accumulation of earlier
|
||||
candidates, stop-on-valid behavior, exhaustion, cumulative usage, error
|
||||
categories, prepared-plan reuse, cancellation, and capacity lifetime;
|
||||
- default-repairer tests own the stable behavioral requirements of its prompt
|
||||
construction, mode-appropriate correction, empty-candidate behavior,
|
||||
diagnostic-feedback bound, and role boundaries without snapshotting
|
||||
incidental prose;
|
||||
- model-client tests own explicit empty and whitespace string content as
|
||||
successful candidates and retain malformed-response coverage for no choices,
|
||||
missing, null, and non-string content;
|
||||
- root external-package tests prove representative public ordinary and
|
||||
prepared repair workflows, including an injected client, without repeating
|
||||
the internal state-machine matrix; and
|
||||
- existing single-pass tests are revised to protect zero-budget and ineligible-
|
||||
mode behavior rather than obsolete public-engine wiring.
|
||||
|
||||
All tests remain deterministic, offline, parallel-safe, and independent of
|
||||
real credentials or provider services. Maintainer validation continues to use
|
||||
the canonical workflow in the
|
||||
[development guide](../development.md#maintainer-validation).
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- A public-engine run with failed basic, JSON, or JSON Schema validation and a
|
||||
positive budget makes no more than the permitted number of corrective calls.
|
||||
- Output contracts accept zero through three repair attempts, reject larger or
|
||||
negative values, and reject a positive budget under `none` validation.
|
||||
- The first contract-valid candidate is returned and no later call is
|
||||
made.
|
||||
- Exhaustion returns the final invalid candidate with final diagnostics and
|
||||
the actual attempt count.
|
||||
- Zero-budget, initially successful, and `none` cases remain single-pass.
|
||||
- Empty and whitespace-only candidates repair under `basic`, `json`, and
|
||||
`json_schema` when the budget is positive, while `none` returns them without
|
||||
repair.
|
||||
- The built-in client sends explicit empty string content to validation but
|
||||
retains generation errors for missing or malformed completion content.
|
||||
- Ordinary and prepared execution preserve the same effective target, session,
|
||||
credential, original rendered messages, structured-output, validation-plan,
|
||||
capacity, and error semantics across every attempt.
|
||||
- Every corrective request contains only the original messages, latest invalid
|
||||
assistant candidate when nonempty, and latest corrective user message;
|
||||
failed-attempt history does not accumulate and empty assistant messages are
|
||||
not fabricated.
|
||||
- Repair sends the complete candidate, bounds only diagnostic prompt feedback,
|
||||
and leaves final validation diagnostics intact.
|
||||
- Usage is cumulative across all completed generations, and no partial result
|
||||
is returned after an operational error.
|
||||
- Public GoDoc and canonical format, consumer, and internal documentation agree
|
||||
with the implemented behavior and do not claim semantic correctness.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- Domain-specific, factual, qualitative, or business-rule validation.
|
||||
- A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge
|
||||
extension point.
|
||||
- Provider retries for HTTP failures, transport failures, rate limits, or
|
||||
malformed successful response envelopes.
|
||||
- Backoff, failover, alternate-profile selection, backend routing changes, or
|
||||
a durable task queue.
|
||||
- Repair for `none` validation or content requirements beyond `basic`
|
||||
nonemptiness and the declared JSON contracts.
|
||||
- A provider-native JSON-object constraint for plain `json` validation.
|
||||
- Local stripping, extraction, normalization, or heuristic rewriting of model
|
||||
output before validation.
|
||||
- Persisting candidate history, returning every candidate, or exposing repair
|
||||
prompts in public results.
|
||||
- Depending on provider prompt caching or promising a particular reduction in
|
||||
billed or reported input tokens.
|
||||
- Reopening sources, changing the selected profile, or changing execution
|
||||
settings between attempts.
|
||||
- Guaranteeing a valid result when the model cannot satisfy the effective
|
||||
output contract within the configured bound.
|
||||
Reference in New Issue
Block a user