eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Prepare documentation for the v0.8.0 release

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-25 12:19:17 +00:00
parent 36ce5a5099
commit a6a8564caa
4 changed files with 114 additions and 966 deletions
Download Patch File Download Diff File Expand all files Collapse all files

7
README.md
View File

@@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve.
## Release Guidance
Consumers upgrading from `v0.6.0` to `v0.7.0` should read the
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
Consumers upgrading from `v0.7.0` to `v0.8.0` should read the
[v0.8.0 changelog and migration guide](docs/releases/v0.8.0.md).
Earlier adopters can consult the
[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md).
Consumers upgrading from `v0.5.0` to `v0.6.0` can consult the
[v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md).
Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the

109
docs/releases/v0.8.0.md Normal file
View File

@@ -0,0 +1,109 @@
# Promptkit v0.8.0
This supplemental changelog and migration guide summarizes the consumer-facing
changes from `v0.7.0` to `v0.8.0`. The annotated `v0.8.0` tag is the
authoritative release record. Exact current contracts belong to the linked
GoDoc and durable documentation.
## Summary
`v0.8.0` activates Promptkit's bounded output-repair workflow:
- failed nonempty-text, JSON, and JSON Schema validation can make a limited
number of corrective model calls;
- corrective calls preserve the original rendered conversation, effective
target, session, structured-output contract, and backend capacity policy;
- results report cumulative usage and the number of corrective calls actually
made; and
- explicitly empty OpenAI-compatible response content now reaches output
validation instead of being classified as a malformed provider envelope.
## Compatibility
This release adds no public declarations or fields and removes none. Existing
source code remains source-compatible.
The behavior of the existing `OutputContract.RepairAttempts` field and prompt
YAML `repair_attempts` field has changed. A positive value now authorizes real
additional model calls after eligible validation failures; earlier releases
accepted the field but the public engine remained single-pass. Consumers that
set a positive value should expect additional latency, token usage, and
provider cost when repair is needed.
Repair budgets must now be between zero and three. A positive budget requires
`basic`, `json`, or `json_schema` validation. Values above three and a positive
budget paired with `none` are invalid contracts rather than ignored settings.
An explicitly present empty or whitespace-only string returned by the built-in
OpenAI-compatible client is now a completed generation candidate. `none`
validation permits it, while `basic`, `json`, and `json_schema` classify it
under their ordinary validation rules and may repair it when configured.
Missing, `null`, or non-string content remains a malformed provider response.
## Upgrade
Update the module dependency with:
```sh
go get gitea.maximumdirect.net/eric/promptkit@v0.8.0
go mod tidy
```
Review every prompt definition and request override that sets a positive repair
budget. Use zero or omit the field to retain single-pass execution. Ensure each
positive budget is no greater than three and uses an eligible validation mode,
then run the consuming project's ordinary and race-enabled tests.
## Bounded Output Repair
`repair_attempts` counts corrective calls in addition to the initial model
call. Promptkit validates each completed candidate, stops at the first valid
one, and never exceeds the configured bound. If every candidate remains
invalid, the run completes successfully with the final candidate and its
failed validation result rather than returning an operational error.
Each correction starts from the original rendered messages and includes only
the latest invalid candidate and latest validation diagnostics. JSON Schema
mode retains the provider-native structured-output request as its first line of
defense. Promptkit performs only deterministic structural validation; a valid
response is not necessarily factual or correct for an application's domain.
Usage in the final result is cumulative across the initial response and every
completed corrective response. `ValidationResult.RepairAttempts` reports the
number of corrective calls actually made. Corrective generation failures use
the same public generation-error categories and structured provider details as
an initial generation failure.
See the [output-contract format reference](../formats.md#output-contract), the
[consumer repair example](../consumers/pkg-promptkit.md#repair-a-structured-result),
and the [`OutputContract` and `ValidationResult` GoDoc](../../types.go) for the
current contracts.
## Explicit Empty Content
The built-in OpenAI-compatible client now distinguishes an explicitly present
empty string from a missing or malformed `content` field. This aligns built-in
and injected clients by letting the selected output contract decide whether an
empty candidate is acceptable, invalid, or eligible for repair.
See the
[OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling)
for the exact envelope behavior.
## Public API Changes
None. This release activates and tightens the documented behavior of existing
fields.
## Consumer Action
- Remove or set `repair_attempts` to zero where execution must remain
single-pass.
- Keep every positive repair budget at three or fewer and pair it with
`basic`, `json`, or `json_schema` validation.
- Account for additional latency, usage, and provider cost when enabling
repair.
- Continue checking the returned validation status because bounded repair can
exhaust without producing a valid candidate.
- Review workflows that previously treated explicit empty provider content as
a generation error.

603
docs/roadmap/implementation.md
View File

@@ -1,603 +0,0 @@
# Public Bounded Output Repair Implementation Plan
## Purpose
Implement the feature defined by the
[public bounded output repair roadmap](output-repair.md). This document is the
decision-complete execution sequence for a coding agent. Follow the repository
[architecture](../policy/architecture.md), [testing](../policy/testing.md), and
[documentation](../policy/documentation.md) policies and the task-specific
reading guide in the [development guide](../development.md) throughout the
work.
## Target Outcome
Promptkit's public `Engine.Run` and `Engine.RunPrepared` workflows honor the
existing repair budget in an effective output contract. A failed `basic`,
`json`, or `json_schema` validation may make at most three additional calls
through the same resolved target, session, credential boundary, prepared
schema, and capacity-wrapped client. Each correction receives the original
rendered conversation, the latest nonempty candidate, and bounded diagnostics.
The first valid candidate ends the operation. Exhaustion returns the final
invalid candidate and complete final diagnostics as an ordinary result, while
generation, validation, capacity, and cancellation failures retain their
existing public error categories and identities. Zero remains single-pass, no
public repair interface is added, and an explicitly present empty provider
string reaches validation instead of being mistaken for a malformed envelope.
## Fixed Decisions
- `OutputContract.RepairAttempts` and prompt YAML `repair_attempts` are the
only consumer controls. The value counts additional generation calls and
must be in the inclusive range zero through three.
- A positive budget with `none` validation is invalid. Positive budgets are
eligible only after a completed failed validation under `basic`, `json`, or
`json_schema`.
- The public engine installs one internal default repairer. There is no new
public option, interface, callback, prompt override, or mutable registry.
- Every corrective call starts with a fresh copy of the complete original
rendered messages. It appends only the latest nonempty candidate as an
`assistant` message and one corrective `user` message. Earlier failed
candidates and corrective messages do not accumulate.
- Empty and whitespace-only candidates omit the additional assistant message.
The corrective user message states that the prior response was empty.
- The corrective instruction is application-neutral. `basic` requests a
nonempty answer to the original task; `json` and `json_schema` request only
corrected JSON without explanation or Markdown fences.
- Validation diagnostics are represented as bounded data in the correction
message. Their encoded feedback is at most 65,536 UTF-8 bytes, preserves
order, and includes an in-bound omission notice when truncated. This bound
never alters the complete diagnostics retained by validation or returned in
the final result.
- The latest nonempty candidate is included verbatim and in full. Promptkit
adds no repair-specific candidate-size limit.
- JSON Schema repair retains the exact prepared structured-output
specification. Plain JSON does not gain a provider-native JSON-object
constraint because its contract permits every JSON value.
- Explicitly present string content from the built-in OpenAI-compatible client
is a successful generation candidate even when empty or whitespace-only.
Missing content, `null`, non-string content, and absent choices remain
malformed provider responses and never enter repair.
- The initial generation and every correction use the same capacity-wrapped
`llm.Client`. One run-admission lease spans the complete operation, while
each model call independently acquires and releases the selected backend's
active-generation permit.
- `ValidationResult.RepairAttempts` is execution outcome state and reports the
number of corrective calls actually started. It is not initialized from the
requested budget by the validation package.
- Usage is the field-by-field sum of every completed generation. An
operational error returns no partial `RunResult`; repair exhaustion is not
an operational error.
- A corrective generation failure is categorized exactly like an initial
generation failure: invalid model requests match `ErrInvalidRequest`, other
provider or transport failures match `ErrLLMGenerate`, typed generation
details remain available, and cancellation and deadline identities remain
discoverable.
- No release notes, roadmap retirement, version changes, commits, tags, or
pushes are part of these stages.
## Execution Rules
- Complete the stages in numerical order. Each stage is sized for one
gpt-5.6-terra implementation prompt and must finish its focused verification
before the next begins.
- At the start of each stage, inspect the working tree and preserve unrelated
changes. The feature roadmap and this plan are intentional planning state.
- Treat intermediate stages as an unreleased partial implementation. Durable
current-state documentation changes belong to Stage 6, after public engine
assembly and behavior are complete.
- Keep source-neutral contract legality in `internal/domain`, structural
validity in `internal/validate`, conversation construction in the default
repairer, orchestration and error categorization in `internal/usecase`, wire
envelope interpretation in `internal/llm`, and public assembly in the root
package.
- Do not move repair behavior into the root facade, validator, capacity
manager, public client adapter, or provider transport merely to simplify a
test.
- Reuse the existing prepared validation plan and generation request
constructor. Do not reopen prompt, profile, backend, artifact, or schema
sources and do not rerender during correction.
- Update existing test owners when a contract changes. Add only representative
public workflow tests; do not duplicate the internal state-machine matrix at
the root package.
- Tests must be deterministic, offline, parallel-safe, race-safe, and
independent of real credentials or provider services. Use injected fakes,
synthetic filesystems, and `httptest` where provider-wire behavior matters.
- Use `apply_patch` for edits, format every changed Go file, review the full
stage diff, and run `git diff --check` before moving to the next stage.
## Stage 1: Enforce The Repair Contract At Its Source-Neutral Boundary
### Objective
Make every supported output-contract source agree on the zero-to-three budget
and invalid `none` pairing, and establish that actual attempt accounting belongs
to execution rather than validation. This stage changes legality and result
ownership but does not activate repair in the public engine.
### Implementation
1. In `internal/domain/output_contract.go`, add one unexported
`maxOutputRepairAttempts = 3` constant and extend
`ValidateOutputContract` to reject:
- negative `RepairAttempts`;
- values greater than three; and
- a positive value when `ValidationMode` is `ValidationNone`.
2. Keep the existing format/mode/schema checks in the same helper. Do not clamp
invalid budgets and do not add the maximum as a root-package public
constant; exact public semantics will be documented through existing fields.
3. Preserve the current normalization and replacement paths so prompt files,
request-level `OutputContract` replacements, `Prepare`, and
`PrepareExecution` all reach the shared domain helper. Do not add parallel
range checks in the root facade or prompt repository.
4. In `internal/validate/standard_validator.go`, stop copying the configured
`contract.RepairAttempts` into a newly created `ValidationResult`. A direct
validation result starts with zero attempts. Retain the existing use-case
wrappers that assign the actual `attemptsUsed` value after each completed
validation.
5. Preserve output-contract whole-value replacement behavior and all existing
validation modes, schema preparation, and error wrapping apart from the new
invalid cases.
### Tests
1. Extend `internal/domain/output_contract_test.go` with a focused boundary
table covering `-1`, `0`, `1`, `3`, and `4`, plus `none` with zero and with
a positive value. Use representative valid basic/JSON contracts so unrelated
validation errors do not mask the intended assertion.
2. Extend the existing prompt-definition repository tests with representative
YAML cases proving a budget above three and a positive `none` pairing are
rejected through the repository's existing invalid-prompt identity. Do not
duplicate the full domain table there.
3. Update `internal/usecase/output_contract_test.go` to prove a request
replacement with either new invalid condition fails before completion
collaborators or admission. Cover both ordinary and prepared preparation
through the existing shared test structure instead of creating a second
matrix.
4. Extend the root output-contract contract test only as needed to preserve the
established public error mapping for invalid request replacements. Source
file failures retain their existing prompt-load category.
5. Update validator tests to assert a direct failed or passed validation reports
zero attempts even when the input contract carries a positive budget. Keep
execution-level attempt assertions in use-case tests.
### Verification
```sh
gofmt -w internal/domain/output_contract.go internal/domain/output_contract_test.go
gofmt -w internal/validate/standard_validator.go internal/validate/standard_validator_test.go
gofmt -w internal/promptdef/repository_test.go internal/usecase/output_contract_test.go
gofmt -w output_contract_contract_test.go
go test ./internal/domain ./internal/validate ./internal/promptdef ./internal/usecase
go test ./... -run 'OutputContract|ValidationResult'
git diff --check
```
Stage 1 is complete when every input path accepts only a coherent zero-to-three
budget, invalid contracts fail before model work, and validators no longer
misreport a configured budget as completed repair work.
**Status:** Complete.
## Stage 2: Distinguish Explicit Empty Content From A Malformed Envelope
### Objective
Allow the built-in OpenAI-compatible client to return an explicitly present
empty or whitespace string as a generation candidate while retaining strict
malformed-response handling for absent or wrongly typed content.
### Implementation
1. In `internal/llm/openai_compatible_client.go`, make
`openAIChatResponseMessage.Content` presence-aware by decoding it as a
`*string`.
2. After response framing and JSON decoding succeed, require at least one
choice and a non-nil first-choice content pointer. Dereference and return
the string exactly as received without trimming or rejecting an empty
value.
3. Treat a missing field and explicit `null` as malformed because both decode
to nil. Let a non-string value fail JSON decoding and remain wrapped with
`ErrMalformedResponse`. Preserve the existing no-choice, trailing-data,
response-size, usage, provider-error, and body-ownership behavior.
4. Do not perform output validation or decide repair eligibility in the model
client. This stage changes only the interpretation of an otherwise
successful completion envelope.
### Tests
1. Add focused cases to `internal/llm/openai_compatible_client_test.go` proving
explicit `""` and whitespace-only string content return successful
`GenerateResponse` values with the exact content and mapped usage.
2. Add or retain distinct malformed-response cases for no choices, missing
content, `null` content, and non-string content. Each must return no response
and match `ErrMalformedResponse`.
3. Keep these tests at the model-client owner. Do not use a root engine or
duplicate validation-mode behavior in this stage.
### Verification
```sh
gofmt -w internal/llm/openai_compatible_client.go internal/llm/openai_compatible_client_test.go
go test ./internal/llm -run 'OpenAICompatibleClient|Response'
go test ./internal/llm
go test ./...
git diff --check
```
Stage 2 is complete when content presence is distinguishable from content
emptiness and all malformed successful-envelope cases retain their prior error
identity.
**Status:** Complete.
## Stage 3: Build Full-Context, Prompt-Safe Corrective Requests
### Objective
Refactor the dormant default repairer so it constructs one correction from the
original rendered conversation and latest validation state, supports basic and
JSON modes, and safely bounds only diagnostic feedback.
### Implementation
1. Extend the internal `RepairRequest` in `internal/usecase/repairer.go` with
`OriginalMessages []domain.RenderedMessage`. Keep the type internal and do
not add repair fields to `internal/domain` or the public package.
2. Replace the standalone system/user repair prompt with a freshly allocated
message slice on every call:
- copy every original rendered message in order without changing role,
content, or cache-control metadata;
- if `strings.TrimSpace(PreviousOutput)` is nonempty, append one `assistant`
message whose content is the exact, full `PreviousOutput` string;
- otherwise append no assistant message; and
- append one corrective `user` message after the original conversation and
optional candidate.
Allocate independently from `OriginalMessages` so `append` cannot mutate a
prepared snapshot's backing array.
3. Construct one stable application-neutral corrective message that includes
the validation mode, `Attempt` and `MaxAttempts`, an instruction to preserve
valid values and change only what is necessary, and diagnostics clearly
labeled as data rather than instructions. For an empty/whitespace candidate,
explicitly state that the previous response was empty and do not reproduce
its whitespace.
4. Use mode-specific terminal guidance:
- `ValidationBasic`: return a nonempty response satisfying the original
request; and
- `ValidationJSON` or `ValidationJSONSchema`: return only corrected JSON,
with no explanation or Markdown fence.
Return a defensive internal error for any other mode; the runner must never
call the repairer for it after Stage 4.
5. Add an unexported `maxRepairDiagnosticBytes = 64 * 1024` constant and one
focused diagnostic formatter. Encode the ordered diagnostics as a JSON array
within the corrective message so provider-controlled text remains visibly
data. The formatter must:
- convert malformed input strings to valid UTF-8 for prompt transport;
- return the complete encoded array when it is within the limit;
- when over the limit, preserve complete earlier entries, include as much
of the next entry as fits at a rune boundary, omit all remaining text, and
append a stable final array entry stating that additional validation
diagnostics were omitted;
- account for JSON quoting, delimiters, and the omission entry when enforcing
the 65,536-byte maximum; and
- return a valid JSON array at or below the limit for every input, including
one oversized error, many errors, control characters, multibyte runes, and
an empty list.
This limit applies only to the encoded diagnostic block, not fixed repair
instructions, the previous candidate, or `ValidationResult.Errors`.
6. Continue to call `newGenerationRequest` with the request's exact session,
target, target-presence metadata, and structured-output pointer. Preserve
the existing nil-client and nil-response defenses. Do not locally strip,
parse, normalize, or repair the candidate.
### Tests
Create `internal/usecase/repairer_test.go` as the narrow owner of default
repairer behavior:
1. Supply original system and user messages, including cache-control metadata,
and assert the generated request contains those values unchanged followed
by the exact assistant candidate and one corrective user message.
2. Prove separate correction calls begin from the same original messages and
do not mutate the caller's slice or accumulate an earlier candidate or
corrective message.
3. Cover empty and whitespace-only candidates: no assistant message is added,
the user message identifies the empty response, and no placeholder model
content is invented.
4. Cover basic, JSON, and JSON Schema guidance by stable behavioral phrases and
role boundaries rather than snapshotting the full incidental prose. Assert
an unsupported mode returns an error without calling the client.
5. Prove the exact target, presence flags, session ID, and structured-output
specification reach the model client and that an arbitrarily large nonempty
candidate is not truncated.
6. Exercise diagnostics below, at, and above the limit, including ordered
entries, multibyte UTF-8 near the boundary, invalid UTF-8, and one huge
diagnostic. Assert the encoded block is valid JSON, valid UTF-8, no more
than 65,536 bytes, ordered, and contains the omission notice only when
needed. Also assert the input error slice and strings remain unchanged.
7. Retain focused nil-client, generation-error passthrough, and nil-response
tests without duplicating runner error categorization.
### Verification
```sh
gofmt -w internal/usecase/repairer.go internal/usecase/repairer_test.go
go test ./internal/usecase -run 'Repairer|RepairRequest'
go test ./internal/usecase
go test ./...
git diff --check
```
Stage 3 is complete when the default repairer produces bounded, mode-correct,
full-context requests without mutating prepared state or retaining failed
history.
**Status:** Complete.
## Stage 4: Complete The Internal Repair State Machine And Error Semantics
### Objective
Connect the improved repairer contract to runner execution, make `basic`
eligible, and ensure corrective generation failures share the initial
generation error path while preserving bounded state, usage, prepared
validation, admission, and capacity behavior.
### Implementation
1. In `internal/usecase/runner.go`, add one private generation-error wrapper
used by both the initial `llm.Generate` call and every repair call:
- if the error matches `llm.ErrInvalidRequest`, wrap it with
`ErrInvalidRequest`; and
- otherwise wrap it with `ErrLLMGenerate`.
Use `%w` wrapping so provider details, cancellation, and deadlines remain
discoverable. Do not classify a failed corrective model call as
`ErrValidation`.
2. Treat a nil repair response without an error as a generation-side failure
matching `ErrLLMGenerate`. Continue returning no partial result for that or
any other operational error.
3. Pass `prepared.Messages` as `RepairRequest.OriginalMessages` on every
attempt. Continue passing the latest response content and latest complete
validation error slice, so each attempt repairs only the newest candidate.
4. Extend `shouldAttemptRepair` to include `ValidationBasic`. Retain all other
gates: a configured repairer, positive budget, and
`ValidationFailed`. `ValidationNone`, passed/skipped validation, and
operational validator errors never reach the repairer.
5. Preserve the existing loop shape and attempt semantics: increment the
attempt count immediately before starting a corrective call, validate every
completed response with the retained operation-local plan, stop on the first
valid response, and make no more calls than the frozen budget.
6. Preserve final-result coherence and cumulative usage. A successful or
exhausted run uses the same final candidate for `RawOutput`, artifact, and
validation; sums all five token-usage fields across completed generations;
reports actual calls in `ValidationResult.RepairAttempts`; and returns the
complete latest diagnostics on exhaustion.
7. Do not change admission or capacity architecture. The admission release
remains deferred across the full execution, and the default repairer must
continue using the same capacity-wrapped client so each correction acquires
the normal active-generation permit without a second run admission.
### Tests
1. Refactor `TestRunnerRepairStateMachine` rather than adding a parallel
matrix:
- change the obsolete “basic failure is ineligible” case into an empty/basic
repair success or bounded exhaustion case;
- keep initial success, stop-on-valid, exact exhaustion, JSON, JSON Schema,
target-presence, structured-output, session, direct credential, final
candidate, and cumulative-usage assertions;
- replace the now-invalid budget-four fixture with a maximum-three fixture;
- assert each `RepairRequest` receives the original rendered messages and
only the latest output/errors; and
- update repair-call detection helpers that currently recognize the old
standalone system prompt so they recognize the appended assistant/user
shape without depending on exact correction prose.
2. Add a zero-budget failed-validation case if the existing public test is its
only owner, ensuring the internal runner remains single-pass without invoking
a configured repairer. A positive `none` case belongs to Stage 1 contract
validation, not this state machine.
3. Update prepared-execution tests to prove initial and repaired candidates use
the same frozen validation plan and messages, sources are not reopened, and
the handle remains one-shot.
4. Update execution-error tables so a repairer generation failure expects
`ErrLLMGenerate`; add a repair error matching `llm.ErrInvalidRequest` and
assert it maps to `ErrInvalidRequest`. Preserve underlying sentinel,
cancellation, and deadline identity and confirm all operational failures
return no partial result.
5. Retain and adapt the shared-backend-pool test to prove two repaired runs
each receive one admission, all initial and repair calls use the same
backend pool, peak active generation does not exceed the configured limit,
and every lease/permit is released.
6. Add a focused cancellation case only if existing capacity coverage does not
already prove cancellation while a corrective call waits for its active
permit prevents that provider call and releases the run admission. Avoid
duplicating `internal/capacity`'s scheduler matrix.
### Verification
```sh
gofmt -w internal/usecase/runner.go internal/usecase/runner_test.go
gofmt -w internal/usecase/prepared_execution_test.go
go test ./internal/usecase -run 'Repair|Prepared|Admission|Capacity|Cancellation'
go test -race ./internal/usecase -run 'Repair|Capacity|Cancellation'
go test ./internal/usecase
go test ./...
git diff --check
```
Stage 4 is complete when the internal state machine satisfies every bounded
success, exhaustion, error, prepared-state, and concurrency invariant without
being publicly activated yet.
**Status:** Complete.
## Stage 5: Activate Repair Through Public Engine Assembly
### Objective
Make the completed behavior available through the existing public API for both
built-in and injected clients, and align exported Go contracts with the now
active behavior.
### Implementation
1. In `engine.go`, after constructing or adapting the selected `llm.Client` and
wrapping it with `capacity.NewClient`, construct
`usecase.NewDefaultOutputRepairer(llmClient)` and assemble the engine runner
with `usecase.NewRunnerWithRepairer`. Pass that exact same wrapped client as
the runner's initial-generation client and the repairer's client.
2. Do not add a public engine option or expose the internal repairer. Keep
`usecase.NewRunner` as the no-repair convenience constructor for focused
internal tests and callers; the root facade deliberately uses the explicit
repairer constructor.
3. Update exported GoDoc in `types.go`:
- `OutputContract.RepairAttempts` is an additional-call budget from zero
through three and is valid only with `basic`, `json`, or `json_schema` when
positive;
- `ValidationResult.RepairAttempts` is the actual number of corrective calls
started and may be nonzero in public results; and
- `LLMResponse.Content` may be explicitly empty and will be interpreted by
the effective output contract rather than rejected by Promptkit's public
adapter.
4. Update `Engine.Run`, `Engine.RunPrepared`, and related prepared/result GoDoc
in `engine.go` to describe opt-in bounded correction, cumulative usage,
exhaustion as a returned failed validation result, operational errors as no
result, cancellation, and one-shot prepared execution. Remove statements
that the public engine never repairs or is always single-pass.
5. Keep the public JSON representation and public type set unchanged. Do not
expose correction messages, candidate history, a semantic-validity claim, or
a new error sentinel.
### Tests
Use root external-package contract tests for representative assembly behavior,
reusing existing public fakes where possible:
1. Replace the obsolete public “positive budget is still single-pass” assertion
with a zero-budget failed-validation assertion that remains exactly one
generation and reports zero attempts.
2. Add one ordinary `Engine.Run` workflow with an injected `LLMClient` that
returns invalid JSON and then valid JSON. Assert two calls, the final valid
candidate, one actual repair, cumulative usage, the same target/session and
structured-output values, and no new configuration surface.
3. Add one `PrepareExecution`/`RunPrepared` workflow using a different eligible
mode, preferably `basic` with an explicit empty initial candidate, to prove
public prepared parity, frozen messages/settings, and one-shot behavior.
4. Add one bounded exhaustion assertion that returns a non-nil result, nil
error, final invalid candidate, complete final diagnostics, exact actual
attempt count, and cumulative usage. Keep the full progression matrix in
`internal/usecase`.
5. Use an offline `httptest` OpenAI-compatible endpoint for one built-in-client
integration path: return a structurally valid completion envelope whose
content fails validation, then a provider error on correction. Assert no
partial result, `ErrLLMGenerate`, available public `GenerationError` details,
and preserved underlying cancellation/deadline behavior where applicable.
Do not duplicate every provider failure shape already owned by
`internal/llm`.
6. Ensure existing capacity tests for injected clients still pass with the
repairer installed, especially engine-local backend limits and unlimited
backend behavior when repair is not requested.
### Verification
```sh
gofmt -w engine.go types.go output_contract_contract_test.go public_contract_test.go
gofmt -w prepared_execution_contract_test.go engine_test.go
go test ./... -run 'Repair|OutputContract|RunPrepared|GenerationError|Capacity'
go test -race ./... -run 'Repair|RunPrepared|Capacity'
go test ./...
git diff --check
```
If a named root test file does not require a change, omit it from `gofmt`
rather than touching it mechanically. Stage 5 is complete when ordinary and
prepared public workflows repair through either client path with unchanged API
shape and correct public result and error semantics.
**Status:** Complete.
## Stage 6: Publish Canonical Documentation And Run Maintainer Validation
### Objective
Make every canonical documentation owner describe the implemented feature,
remove obsolete dormant/single-pass claims, and validate the complete
cross-cutting change using the repository's canonical workflow.
### Implementation
1. Update `docs/formats.md`, the exact prompt-format owner, to document:
- `repair_attempts` as an integer from zero through three;
- zero as the single-pass default and positive values as additional calls;
- eligible `basic`, `json`, and `json_schema` modes;
- invalid positive budgets with `none`;
- stop-on-valid and exhausted-final-result behavior; and
- plain JSON's acceptance of every JSON value without an object-only wire
constraint.
2. Update `docs/consumers/pkg-promptkit.md` with one concise JSON Schema repair
example using the existing public field or prompt YAML. Explain cumulative
usage, actual attempt reporting, the possibility of a final failed
validation after exhaustion, and the cost/latency implications of extra
calls. Note briefly that `basic` can repair empty output and that structural
validity is not factual or domain correctness.
3. Update `docs/integrations/openai-compatible-chat.md` to own the provider-wire
distinction: an explicitly present string `content`, including empty or
whitespace-only, is a candidate; absent choices, missing content, `null`,
and non-string content are malformed. Retain provider-native JSON Schema as
the first defense and do not claim an object-only constraint for plain JSON.
4. Update `docs/internal/llm.md` with the same internal response-decoding
boundary without duplicating public repair orchestration.
5. Update `docs/internal/runner.md` to describe the installed default repairer,
full-original-message/latest-candidate request shape, empty-candidate role
behavior, diagnostic bound, eligible modes, attempt/usage accounting,
exhaustion, generation-versus-validation error categories, prepared-plan
reuse, and one-admission/shared-capacity behavior. Remove the claim that the
ordinary public runner omits a repairer.
6. Update `docs/internal/overview.md` so root assembly and `internal/usecase`
ownership reflect public bounded repair. Review `docs/internal/capacity.md`
and `docs/internal/sources.md`; edit only statements that are stale or
incomplete after implementation, preserving their narrow ownership.
7. Do not put canonical behavior in a release note or roadmap. Do not update
the architecture policy unless implementation actually crossed the package
boundaries fixed by this plan. Do not retire `output-repair.md` or this plan
in this implementation sequence.
8. Check every changed Markdown link and heading fragment with the canonical
offline link checker in the development guide. Review the final diff for
consistent terminology: “repair attempt” always means an additional model
call, “exhaustion” returns a completed failed validation result, and no text
promises semantic correctness or provider caching.
### Tests And Final Verification
Run the complete
[maintainer validation workflow](../development.md#maintainer-validation) from
the repository root, including both maintained offline examples:
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
go run ./examples/go-library/run
git diff --check
git status --short
```
Also run the development guide's tracked-Go formatting check and local
Markdown-link checker exactly as documented. Review example JSON output for
the invariants listed there; neither example may contact a real provider or
require credentials.
Stage 6 is complete when code, tests, exported GoDoc, format documentation,
consumer guidance, integration contracts, and internal documentation agree;
all canonical validation commands pass; and the working tree contains only the
intended feature implementation and roadmap changes.
**Status:** Complete.
## Open Questions
None. The feature roadmap and the fixed decisions above define all behavior
needed for implementation.

361
docs/roadmap/output-repair.md
View File

@@ -1,361 +0,0 @@
# Public Bounded Output Repair Roadmap
## Purpose
Make Promptkit's existing bounded output-repair capability available through
the public engine. Consumers that require nonempty text, JSON, or JSON Schema
output should be able to ask Promptkit to make a limited number of corrective
model calls after content validation fails, without rebuilding Promptkit's
generation, validation, capacity, credential, session, and result-accounting
workflow.
The feature improves basic content and structural reliability, not domain
correctness. Promptkit can determine whether output is nonempty, whether it is
syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It
cannot determine whether otherwise valid content is factual, complete, useful,
or semantically correct for a downstream application.
## Target End State
- The existing `repair_attempts` prompt field and
`OutputContract.RepairAttempts` request field control public repair behavior.
Zero remains the default and makes execution single-pass.
- A repair budget may be one, two, or three corrective calls. Values above
three are invalid rather than silently clamped, providing a framework-level
bound on accidental cost and latency.
- A positive repair budget is eligible when the effective validation mode is
`basic`, `json`, or `json_schema` and the latest completed content validation
has failed. `none` explicitly permits empty content, and pairing it with a
positive repair budget is an invalid output contract.
- `Engine.Run` and `Engine.RunPrepared` provide the same repair behavior for
the built-in OpenAI-compatible client and an injected `LLMClient`.
- Provider-native structured-output metadata remains the first line of
defense for JSON Schema generation. Repair applies only after the resulting
content still fails Promptkit's own validation.
- Promptkit stops at the first contract-valid response or after the
requested bound is exhausted. It never reports an invalid response as valid.
- A completed exhausted run returns the final invalid candidate and its final
validation diagnostics in the ordinary `RunResult`; exhaustion is not an
operational error.
- Results report the number of corrective calls actually attempted and the
cumulative token usage reported by the initial generation and every
completed repair generation.
- Repair remains internal orchestration. Consumers do not need to construct or
register a repairer, and this feature does not add a public repair strategy
interface.
## Eligibility And Contract Semantics
Repair is opt-in for each effective output contract. The prompt definition may
declare a budget, and a request-level `OutputContract` may replace that whole
contract under the existing replacement semantics. Preparation validates and
freezes the effective budget along with the validation mode and schema plan.
Only a completed `ValidationFailed` result under `basic`, `json`, or
`json_schema` can start or continue repair. The following do not trigger
repair:
- `none` validation;
- an initially valid response;
- schema loading, decoding, registration, or compilation failures;
- an operational inability to execute validation;
- invalid requests, provider failures, transport failures, or cancellation;
and
- capacity-admission failures.
Selecting `none` is the explicit way for a caller to allow empty output and
requires a zero repair budget. Rejecting a positive budget with `none` avoids a
contradictory configuration whose requested corrective calls could never be
eligible. Under `basic`, an empty or whitespace-only candidate fails validation
and becomes repairable. Under `json` and `json_schema`, empty content is
already invalid JSON and follows the same repair path as other JSON failures.
The requested budget must be between zero and three. Output-contract validation
rejects negative values and values above three at the same source-neutral
boundary for prompt definitions, request replacements, ordinary preparation,
and prepared execution. The effective value is never silently clamped.
`repair_attempts` counts additional model calls after the initial generation,
not total calls. `ValidationResult.RepairAttempts` reports calls actually
started. A response that validates on the first corrective call therefore
reports one repair attempt.
## Repair Request Behavior
Each corrective call extends the original rendered conversation rather than
replacing it with a standalone repair prompt. For a nonempty candidate,
Promptkit constructs the request from:
1. the complete original rendered messages in their original order;
2. one additional `assistant` message containing the latest invalid candidate;
and
3. one additional `user` message containing Promptkit's application-neutral
correction instruction and the latest validation diagnostics.
For an empty or whitespace-only candidate, Promptkit omits the additional
`assistant` message and appends only the corrective `user` message. This avoids
sending an empty assistant message that a compatible provider may reject and
does not invent placeholder model content. The corrective message states that
the previous response was empty.
The corrective user message always:
- identifies the effective validation mode and the current attempt number;
- supplies the current validation diagnostics as data;
- asks the model to preserve valid values and change only what is necessary to
satisfy the effective output contract; and
- uses clear boundaries so provider-controlled diagnostics are not confused
with Promptkit's instructions.
The mode-specific instruction reflects the contract Promptkit can actually
enforce:
- for `basic`, it asks for a nonempty response that satisfies the original
request; and
- for `json` and `json_schema`, it asks for only corrected JSON, without
Markdown fences or explanation.
Representing the candidate as an `assistant` message keeps model output
separate from Promptkit's corrective `user` instruction and gives the model
access to the original task and source material when a missing or invalid
field must be regenerated. Promptkit does not flatten the original messages or
embed the candidate into the corrective message.
Each attempt repairs only the latest candidate. Promptkit does not append the
entire history of earlier candidates, which would increase cost without adding
equivalent corrective value and could make the active correction ambiguous.
Every attempt starts again from the original messages and appends only the
latest nonempty candidate, when present, and latest diagnostics.
JSON Schema mode retains the prepared structured-output specification, so a
compatible client receives the same provider-native schema contract on the
initial call and every corrective call. This preventive constraint remains
the first line of defense even though Promptkit independently validates the
returned content and can repair failures from providers that ignore, partially
implement, or fail to satisfy it.
Plain `json` validation continues to permit any JSON value, including an
object, array, string, number, boolean, or `null`. Promptkit must not send a
provider-native JSON-object constraint for that mode because doing so would
silently narrow the declared validation contract. A future contract may add
provider-native object generation if consumers need that distinct behavior.
Resending the original messages may allow a provider to reuse cached prompt
prefixes, especially when the original cache-control metadata is preserved.
Promptkit does not assume, require, measure, or promise provider caching, and
usage and cost remain whatever the selected client reports or charges.
When it is not empty or whitespace-only, the latest candidate is included in
full and is never truncated or rewritten by Promptkit. The built-in client
already bounds successful provider response bodies; consumers that inject an
`LLMClient` own that client's response-size policy. Repair adds no smaller
candidate limit or silent size-based skip.
Validation diagnostics included in the corrective message are limited to
65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the
diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a
bounded notice that additional diagnostic text was omitted. This prompt-safety
limit does not alter the complete diagnostics returned in the final
`ValidationResult`.
## Empty Content And Provider Responses
Content presence belongs to validation when the provider has returned an
otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible
client therefore distinguishes these cases:
- a first choice with an explicitly present string `content` value is a
completed generation candidate even when that string is empty or consists
only of whitespace; and
- no choices, a missing `content` field, `null` content, or non-string content
remains a malformed provider response and returns a generation error without
entering content repair.
This distinction keeps the built-in and injected-client paths coherent. An
explicit empty candidate reaches `none`, `basic`, `json`, or `json_schema`
validation under the same rules regardless of which client produced it. A
malformed response envelope is not reclassified as invalid model content and
does not consume the repair budget.
The initial and corrective calls use the same:
- resolved backend identity and endpoint;
- model and effective execution settings, including presence-aware numeric and
reasoning values;
- direct credential or environment lookup boundary;
- effective session ID;
- structured-output specification; and
- capacity manager and selected backend pool.
Repair does not reload prompt, profile, backend, input, or schema sources and
does not rerender the original prompt. It reuses the already rendered message
snapshot, including its roles, contents, and cache-control metadata. A
prepared execution uses its retained messages, validation plan, and frozen
effective target throughout repair.
## Results And Failure Behavior
After every successful corrective generation, Promptkit rebuilds the output
artifact from that candidate and validates it with the operation's existing
validation plan. The final `RawOutput`, artifact, validation result, and repair
attempt count all describe the same final candidate.
Token usage is accumulated field by field across the initial response and each
completed corrective response. Timing covers the entire operation, including
repair. Prepared execution remains one-shot, and its handle is consumed by the
single execution even when that execution performs several generation calls.
A failure while making a corrective model call retains the same public error
category and underlying identity it would have had during initial generation:
- invalid model requests match `ErrInvalidRequest`;
- provider and transport generation failures match `ErrLLMGenerate`, including
any available `GenerationError` details from the built-in client; and
- cancellation and deadlines remain discoverable through the error chain.
An operational failure while validating a corrective response continues to
match `ErrValidation`. These failures return no partial `RunResult` or partial
usage, consistent with existing operation failure behavior. Merely exhausting
the content-repair budget is different: validation completed successfully, so
the engine returns the final failed validation result rather than an error.
## Capacity And Concurrency
The ordinary run-admission lease spans preparation completion, initial
generation, validation, all corrective calls, and every exit. Repair never
performs a second admission. Each corrective generation uses the same
capacity-wrapped model client and independently reacquires an active-generation
permit for its backend, allowing unrelated accepted work to use the pool while
the repairing run validates or waits.
Cancellation while waiting for a repair-generation permit prevents that model
call when cancellation wins the grant race. All admission and generation
permits must be released on successful repair, exhaustion, provider failure,
validation failure, and cancellation.
## Architecture And Ownership
The root facade installs Promptkit's default output repairer when assembling
the runner. It supplies the same capacity-wrapped `llm.Client` used for initial
generation, whether that client ultimately delegates to Promptkit's built-in
transport or a consumer-injected `LLMClient`.
`internal/usecase` continues to own the bounded repair state machine, attempt
accounting, final-result selection, cumulative usage, error categorization,
and coordination with prepared validation. The default repairer owns only the
construction of the augmented conversation and the corrective generation
call. `internal/validate` remains the sole authority for structural validity,
while `internal/llm` retains responsibility for distinguishing explicit empty
content from an absent or malformed completion and `internal/capacity` retains
its scheduling responsibility.
No repair-specific type needs to enter `internal/domain` beyond the existing
output-contract and validation-result state, and no public package or mutable
engine registry is introduced.
## Public And Documentation Surface
No new public method or option is required. The feature activates the existing
public fields and updates their contracts:
- `OutputContract.RepairAttempts` describes the requested additional-call
budget, its zero-to-three range, and eligible validation modes;
- `ValidationResult.RepairAttempts` describes the attempts actually made; and
- `Run`, `RunPrepared`, and related result GoDoc describe exhaustion, failure,
usage, cancellation, and prepared-execution behavior.
The framework format reference owns the exact
`repair_attempts` YAML semantics and the meaning of `none`, `basic`, `json`,
and `json_schema`. Consumer guidance contains one concise JSON Schema
example, note that `basic` can repair empty output, and emphasize that content
presence or structural validity does not establish domain correctness. The
internal runner document owns orchestration and test ownership. The
OpenAI-compatible integration contract owns the explicit-empty versus missing
content distinction. Architecture policy needs to change only if implementation
changes its existing package boundaries.
Canonical current-state documentation describes implemented behavior in
accordance with the documentation policy. Roadmap material remains temporary
and is retired when it no longer describes future work.
## Verification Expectations
Testing should preserve the repository's lean ownership model and build on the
existing internal repair coverage:
- domain and source-boundary tests own accepted repair budgets and output-
contract validation, including the upper bound and invalid `none` pairing;
- use-case tests own eligibility, bounded progression, latest-candidate
feedback, original-message preservation, non-accumulation of earlier
candidates, stop-on-valid behavior, exhaustion, cumulative usage, error
categories, prepared-plan reuse, cancellation, and capacity lifetime;
- default-repairer tests own the stable behavioral requirements of its prompt
construction, mode-appropriate correction, empty-candidate behavior,
diagnostic-feedback bound, and role boundaries without snapshotting
incidental prose;
- model-client tests own explicit empty and whitespace string content as
successful candidates and retain malformed-response coverage for no choices,
missing, null, and non-string content;
- root external-package tests prove representative public ordinary and
prepared repair workflows, including an injected client, without repeating
the internal state-machine matrix; and
- existing single-pass tests are revised to protect zero-budget and ineligible-
mode behavior rather than obsolete public-engine wiring.
All tests remain deterministic, offline, parallel-safe, and independent of
real credentials or provider services. Maintainer validation continues to use
the canonical workflow in the
[development guide](../development.md#maintainer-validation).
## Acceptance Criteria
- A public-engine run with failed basic, JSON, or JSON Schema validation and a
positive budget makes no more than the permitted number of corrective calls.
- Output contracts accept zero through three repair attempts, reject larger or
negative values, and reject a positive budget under `none` validation.
- The first contract-valid candidate is returned and no later call is
made.
- Exhaustion returns the final invalid candidate with final diagnostics and
the actual attempt count.
- Zero-budget, initially successful, and `none` cases remain single-pass.
- Empty and whitespace-only candidates repair under `basic`, `json`, and
`json_schema` when the budget is positive, while `none` returns them without
repair.
- The built-in client sends explicit empty string content to validation but
retains generation errors for missing or malformed completion content.
- Ordinary and prepared execution preserve the same effective target, session,
credential, original rendered messages, structured-output, validation-plan,
capacity, and error semantics across every attempt.
- Every corrective request contains only the original messages, latest invalid
assistant candidate when nonempty, and latest corrective user message;
failed-attempt history does not accumulate and empty assistant messages are
not fabricated.
- Repair sends the complete candidate, bounds only diagnostic prompt feedback,
and leaves final validation diagnostics intact.
- Usage is cumulative across all completed generations, and no partial result
is returned after an operational error.
- Public GoDoc and canonical format, consumer, and internal documentation agree
with the implemented behavior and do not claim semantic correctness.
## Non-Goals
- Domain-specific, factual, qualitative, or business-rule validation.
- A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge
extension point.
- Provider retries for HTTP failures, transport failures, rate limits, or
malformed successful response envelopes.
- Backoff, failover, alternate-profile selection, backend routing changes, or
a durable task queue.
- Repair for `none` validation or content requirements beyond `basic`
nonemptiness and the declared JSON contracts.
- A provider-native JSON-object constraint for plain `json` validation.
- Local stripping, extraction, normalization, or heuristic rewriting of model
output before validation.
- Persisting candidate history, returning every candidate, or exposing repair
prompts in public results.
- Depending on provider prompt caching or promising a particular reduction in
billed or reported input tokens.
- Reopening sources, changing the selected profile, or changing execution
settings between attempts.
- Guaranteeing a valid result when the model cannot satisfy the effective
output contract within the configured bound.