598 lines
31 KiB
Markdown
598 lines
31 KiB
Markdown
# Public Bounded Output Repair Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
Implement the feature defined by the
|
|
[public bounded output repair roadmap](output-repair.md). This document is the
|
|
decision-complete execution sequence for a coding agent. Follow the repository
|
|
[architecture](../policy/architecture.md), [testing](../policy/testing.md), and
|
|
[documentation](../policy/documentation.md) policies and the task-specific
|
|
reading guide in the [development guide](../development.md) throughout the
|
|
work.
|
|
|
|
## Target Outcome
|
|
|
|
Promptkit's public `Engine.Run` and `Engine.RunPrepared` workflows honor the
|
|
existing repair budget in an effective output contract. A failed `basic`,
|
|
`json`, or `json_schema` validation may make at most three additional calls
|
|
through the same resolved target, session, credential boundary, prepared
|
|
schema, and capacity-wrapped client. Each correction receives the original
|
|
rendered conversation, the latest nonempty candidate, and bounded diagnostics.
|
|
|
|
The first valid candidate ends the operation. Exhaustion returns the final
|
|
invalid candidate and complete final diagnostics as an ordinary result, while
|
|
generation, validation, capacity, and cancellation failures retain their
|
|
existing public error categories and identities. Zero remains single-pass, no
|
|
public repair interface is added, and an explicitly present empty provider
|
|
string reaches validation instead of being mistaken for a malformed envelope.
|
|
|
|
## Fixed Decisions
|
|
|
|
- `OutputContract.RepairAttempts` and prompt YAML `repair_attempts` are the
|
|
only consumer controls. The value counts additional generation calls and
|
|
must be in the inclusive range zero through three.
|
|
- A positive budget with `none` validation is invalid. Positive budgets are
|
|
eligible only after a completed failed validation under `basic`, `json`, or
|
|
`json_schema`.
|
|
- The public engine installs one internal default repairer. There is no new
|
|
public option, interface, callback, prompt override, or mutable registry.
|
|
- Every corrective call starts with a fresh copy of the complete original
|
|
rendered messages. It appends only the latest nonempty candidate as an
|
|
`assistant` message and one corrective `user` message. Earlier failed
|
|
candidates and corrective messages do not accumulate.
|
|
- Empty and whitespace-only candidates omit the additional assistant message.
|
|
The corrective user message states that the prior response was empty.
|
|
- The corrective instruction is application-neutral. `basic` requests a
|
|
nonempty answer to the original task; `json` and `json_schema` request only
|
|
corrected JSON without explanation or Markdown fences.
|
|
- Validation diagnostics are represented as bounded data in the correction
|
|
message. Their encoded feedback is at most 65,536 UTF-8 bytes, preserves
|
|
order, and includes an in-bound omission notice when truncated. This bound
|
|
never alters the complete diagnostics retained by validation or returned in
|
|
the final result.
|
|
- The latest nonempty candidate is included verbatim and in full. Promptkit
|
|
adds no repair-specific candidate-size limit.
|
|
- JSON Schema repair retains the exact prepared structured-output
|
|
specification. Plain JSON does not gain a provider-native JSON-object
|
|
constraint because its contract permits every JSON value.
|
|
- Explicitly present string content from the built-in OpenAI-compatible client
|
|
is a successful generation candidate even when empty or whitespace-only.
|
|
Missing content, `null`, non-string content, and absent choices remain
|
|
malformed provider responses and never enter repair.
|
|
- The initial generation and every correction use the same capacity-wrapped
|
|
`llm.Client`. One run-admission lease spans the complete operation, while
|
|
each model call independently acquires and releases the selected backend's
|
|
active-generation permit.
|
|
- `ValidationResult.RepairAttempts` is execution outcome state and reports the
|
|
number of corrective calls actually started. It is not initialized from the
|
|
requested budget by the validation package.
|
|
- Usage is the field-by-field sum of every completed generation. An
|
|
operational error returns no partial `RunResult`; repair exhaustion is not
|
|
an operational error.
|
|
- A corrective generation failure is categorized exactly like an initial
|
|
generation failure: invalid model requests match `ErrInvalidRequest`, other
|
|
provider or transport failures match `ErrLLMGenerate`, typed generation
|
|
details remain available, and cancellation and deadline identities remain
|
|
discoverable.
|
|
- No release notes, roadmap retirement, version changes, commits, tags, or
|
|
pushes are part of these stages.
|
|
|
|
## Execution Rules
|
|
|
|
- Complete the stages in numerical order. Each stage is sized for one
|
|
gpt-5.6-terra implementation prompt and must finish its focused verification
|
|
before the next begins.
|
|
- At the start of each stage, inspect the working tree and preserve unrelated
|
|
changes. The feature roadmap and this plan are intentional planning state.
|
|
- Treat intermediate stages as an unreleased partial implementation. Durable
|
|
current-state documentation changes belong to Stage 6, after public engine
|
|
assembly and behavior are complete.
|
|
- Keep source-neutral contract legality in `internal/domain`, structural
|
|
validity in `internal/validate`, conversation construction in the default
|
|
repairer, orchestration and error categorization in `internal/usecase`, wire
|
|
envelope interpretation in `internal/llm`, and public assembly in the root
|
|
package.
|
|
- Do not move repair behavior into the root facade, validator, capacity
|
|
manager, public client adapter, or provider transport merely to simplify a
|
|
test.
|
|
- Reuse the existing prepared validation plan and generation request
|
|
constructor. Do not reopen prompt, profile, backend, artifact, or schema
|
|
sources and do not rerender during correction.
|
|
- Update existing test owners when a contract changes. Add only representative
|
|
public workflow tests; do not duplicate the internal state-machine matrix at
|
|
the root package.
|
|
- Tests must be deterministic, offline, parallel-safe, race-safe, and
|
|
independent of real credentials or provider services. Use injected fakes,
|
|
synthetic filesystems, and `httptest` where provider-wire behavior matters.
|
|
- Use `apply_patch` for edits, format every changed Go file, review the full
|
|
stage diff, and run `git diff --check` before moving to the next stage.
|
|
|
|
## Stage 1: Enforce The Repair Contract At Its Source-Neutral Boundary
|
|
|
|
### Objective
|
|
|
|
Make every supported output-contract source agree on the zero-to-three budget
|
|
and invalid `none` pairing, and establish that actual attempt accounting belongs
|
|
to execution rather than validation. This stage changes legality and result
|
|
ownership but does not activate repair in the public engine.
|
|
|
|
### Implementation
|
|
|
|
1. In `internal/domain/output_contract.go`, add one unexported
|
|
`maxOutputRepairAttempts = 3` constant and extend
|
|
`ValidateOutputContract` to reject:
|
|
- negative `RepairAttempts`;
|
|
- values greater than three; and
|
|
- a positive value when `ValidationMode` is `ValidationNone`.
|
|
2. Keep the existing format/mode/schema checks in the same helper. Do not clamp
|
|
invalid budgets and do not add the maximum as a root-package public
|
|
constant; exact public semantics will be documented through existing fields.
|
|
3. Preserve the current normalization and replacement paths so prompt files,
|
|
request-level `OutputContract` replacements, `Prepare`, and
|
|
`PrepareExecution` all reach the shared domain helper. Do not add parallel
|
|
range checks in the root facade or prompt repository.
|
|
4. In `internal/validate/standard_validator.go`, stop copying the configured
|
|
`contract.RepairAttempts` into a newly created `ValidationResult`. A direct
|
|
validation result starts with zero attempts. Retain the existing use-case
|
|
wrappers that assign the actual `attemptsUsed` value after each completed
|
|
validation.
|
|
5. Preserve output-contract whole-value replacement behavior and all existing
|
|
validation modes, schema preparation, and error wrapping apart from the new
|
|
invalid cases.
|
|
|
|
### Tests
|
|
|
|
1. Extend `internal/domain/output_contract_test.go` with a focused boundary
|
|
table covering `-1`, `0`, `1`, `3`, and `4`, plus `none` with zero and with
|
|
a positive value. Use representative valid basic/JSON contracts so unrelated
|
|
validation errors do not mask the intended assertion.
|
|
2. Extend the existing prompt-definition repository tests with representative
|
|
YAML cases proving a budget above three and a positive `none` pairing are
|
|
rejected through the repository's existing invalid-prompt identity. Do not
|
|
duplicate the full domain table there.
|
|
3. Update `internal/usecase/output_contract_test.go` to prove a request
|
|
replacement with either new invalid condition fails before completion
|
|
collaborators or admission. Cover both ordinary and prepared preparation
|
|
through the existing shared test structure instead of creating a second
|
|
matrix.
|
|
4. Extend the root output-contract contract test only as needed to preserve the
|
|
established public error mapping for invalid request replacements. Source
|
|
file failures retain their existing prompt-load category.
|
|
5. Update validator tests to assert a direct failed or passed validation reports
|
|
zero attempts even when the input contract carries a positive budget. Keep
|
|
execution-level attempt assertions in use-case tests.
|
|
|
|
### Verification
|
|
|
|
```sh
|
|
gofmt -w internal/domain/output_contract.go internal/domain/output_contract_test.go
|
|
gofmt -w internal/validate/standard_validator.go internal/validate/standard_validator_test.go
|
|
gofmt -w internal/promptdef/repository_test.go internal/usecase/output_contract_test.go
|
|
gofmt -w output_contract_contract_test.go
|
|
go test ./internal/domain ./internal/validate ./internal/promptdef ./internal/usecase
|
|
go test ./... -run 'OutputContract|ValidationResult'
|
|
git diff --check
|
|
```
|
|
|
|
Stage 1 is complete when every input path accepts only a coherent zero-to-three
|
|
budget, invalid contracts fail before model work, and validators no longer
|
|
misreport a configured budget as completed repair work.
|
|
|
|
**Status:** Complete.
|
|
|
|
## Stage 2: Distinguish Explicit Empty Content From A Malformed Envelope
|
|
|
|
### Objective
|
|
|
|
Allow the built-in OpenAI-compatible client to return an explicitly present
|
|
empty or whitespace string as a generation candidate while retaining strict
|
|
malformed-response handling for absent or wrongly typed content.
|
|
|
|
### Implementation
|
|
|
|
1. In `internal/llm/openai_compatible_client.go`, make
|
|
`openAIChatResponseMessage.Content` presence-aware by decoding it as a
|
|
`*string`.
|
|
2. After response framing and JSON decoding succeed, require at least one
|
|
choice and a non-nil first-choice content pointer. Dereference and return
|
|
the string exactly as received without trimming or rejecting an empty
|
|
value.
|
|
3. Treat a missing field and explicit `null` as malformed because both decode
|
|
to nil. Let a non-string value fail JSON decoding and remain wrapped with
|
|
`ErrMalformedResponse`. Preserve the existing no-choice, trailing-data,
|
|
response-size, usage, provider-error, and body-ownership behavior.
|
|
4. Do not perform output validation or decide repair eligibility in the model
|
|
client. This stage changes only the interpretation of an otherwise
|
|
successful completion envelope.
|
|
|
|
### Tests
|
|
|
|
1. Add focused cases to `internal/llm/openai_compatible_client_test.go` proving
|
|
explicit `""` and whitespace-only string content return successful
|
|
`GenerateResponse` values with the exact content and mapped usage.
|
|
2. Add or retain distinct malformed-response cases for no choices, missing
|
|
content, `null` content, and non-string content. Each must return no response
|
|
and match `ErrMalformedResponse`.
|
|
3. Keep these tests at the model-client owner. Do not use a root engine or
|
|
duplicate validation-mode behavior in this stage.
|
|
|
|
### Verification
|
|
|
|
```sh
|
|
gofmt -w internal/llm/openai_compatible_client.go internal/llm/openai_compatible_client_test.go
|
|
go test ./internal/llm -run 'OpenAICompatibleClient|Response'
|
|
go test ./internal/llm
|
|
go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
Stage 2 is complete when content presence is distinguishable from content
|
|
emptiness and all malformed successful-envelope cases retain their prior error
|
|
identity.
|
|
|
|
**Status:** Complete.
|
|
|
|
## Stage 3: Build Full-Context, Prompt-Safe Corrective Requests
|
|
|
|
### Objective
|
|
|
|
Refactor the dormant default repairer so it constructs one correction from the
|
|
original rendered conversation and latest validation state, supports basic and
|
|
JSON modes, and safely bounds only diagnostic feedback.
|
|
|
|
### Implementation
|
|
|
|
1. Extend the internal `RepairRequest` in `internal/usecase/repairer.go` with
|
|
`OriginalMessages []domain.RenderedMessage`. Keep the type internal and do
|
|
not add repair fields to `internal/domain` or the public package.
|
|
2. Replace the standalone system/user repair prompt with a freshly allocated
|
|
message slice on every call:
|
|
- copy every original rendered message in order without changing role,
|
|
content, or cache-control metadata;
|
|
- if `strings.TrimSpace(PreviousOutput)` is nonempty, append one `assistant`
|
|
message whose content is the exact, full `PreviousOutput` string;
|
|
- otherwise append no assistant message; and
|
|
- append one corrective `user` message after the original conversation and
|
|
optional candidate.
|
|
Allocate independently from `OriginalMessages` so `append` cannot mutate a
|
|
prepared snapshot's backing array.
|
|
3. Construct one stable application-neutral corrective message that includes
|
|
the validation mode, `Attempt` and `MaxAttempts`, an instruction to preserve
|
|
valid values and change only what is necessary, and diagnostics clearly
|
|
labeled as data rather than instructions. For an empty/whitespace candidate,
|
|
explicitly state that the previous response was empty and do not reproduce
|
|
its whitespace.
|
|
4. Use mode-specific terminal guidance:
|
|
- `ValidationBasic`: return a nonempty response satisfying the original
|
|
request; and
|
|
- `ValidationJSON` or `ValidationJSONSchema`: return only corrected JSON,
|
|
with no explanation or Markdown fence.
|
|
Return a defensive internal error for any other mode; the runner must never
|
|
call the repairer for it after Stage 4.
|
|
5. Add an unexported `maxRepairDiagnosticBytes = 64 * 1024` constant and one
|
|
focused diagnostic formatter. Encode the ordered diagnostics as a JSON array
|
|
within the corrective message so provider-controlled text remains visibly
|
|
data. The formatter must:
|
|
- convert malformed input strings to valid UTF-8 for prompt transport;
|
|
- return the complete encoded array when it is within the limit;
|
|
- when over the limit, preserve complete earlier entries, include as much
|
|
of the next entry as fits at a rune boundary, omit all remaining text, and
|
|
append a stable final array entry stating that additional validation
|
|
diagnostics were omitted;
|
|
- account for JSON quoting, delimiters, and the omission entry when enforcing
|
|
the 65,536-byte maximum; and
|
|
- return a valid JSON array at or below the limit for every input, including
|
|
one oversized error, many errors, control characters, multibyte runes, and
|
|
an empty list.
|
|
This limit applies only to the encoded diagnostic block, not fixed repair
|
|
instructions, the previous candidate, or `ValidationResult.Errors`.
|
|
6. Continue to call `newGenerationRequest` with the request's exact session,
|
|
target, target-presence metadata, and structured-output pointer. Preserve
|
|
the existing nil-client and nil-response defenses. Do not locally strip,
|
|
parse, normalize, or repair the candidate.
|
|
|
|
### Tests
|
|
|
|
Create `internal/usecase/repairer_test.go` as the narrow owner of default
|
|
repairer behavior:
|
|
|
|
1. Supply original system and user messages, including cache-control metadata,
|
|
and assert the generated request contains those values unchanged followed
|
|
by the exact assistant candidate and one corrective user message.
|
|
2. Prove separate correction calls begin from the same original messages and
|
|
do not mutate the caller's slice or accumulate an earlier candidate or
|
|
corrective message.
|
|
3. Cover empty and whitespace-only candidates: no assistant message is added,
|
|
the user message identifies the empty response, and no placeholder model
|
|
content is invented.
|
|
4. Cover basic, JSON, and JSON Schema guidance by stable behavioral phrases and
|
|
role boundaries rather than snapshotting the full incidental prose. Assert
|
|
an unsupported mode returns an error without calling the client.
|
|
5. Prove the exact target, presence flags, session ID, and structured-output
|
|
specification reach the model client and that an arbitrarily large nonempty
|
|
candidate is not truncated.
|
|
6. Exercise diagnostics below, at, and above the limit, including ordered
|
|
entries, multibyte UTF-8 near the boundary, invalid UTF-8, and one huge
|
|
diagnostic. Assert the encoded block is valid JSON, valid UTF-8, no more
|
|
than 65,536 bytes, ordered, and contains the omission notice only when
|
|
needed. Also assert the input error slice and strings remain unchanged.
|
|
7. Retain focused nil-client, generation-error passthrough, and nil-response
|
|
tests without duplicating runner error categorization.
|
|
|
|
### Verification
|
|
|
|
```sh
|
|
gofmt -w internal/usecase/repairer.go internal/usecase/repairer_test.go
|
|
go test ./internal/usecase -run 'Repairer|RepairRequest'
|
|
go test ./internal/usecase
|
|
go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
Stage 3 is complete when the default repairer produces bounded, mode-correct,
|
|
full-context requests without mutating prepared state or retaining failed
|
|
history.
|
|
|
|
**Status:** Complete.
|
|
|
|
## Stage 4: Complete The Internal Repair State Machine And Error Semantics
|
|
|
|
### Objective
|
|
|
|
Connect the improved repairer contract to runner execution, make `basic`
|
|
eligible, and ensure corrective generation failures share the initial
|
|
generation error path while preserving bounded state, usage, prepared
|
|
validation, admission, and capacity behavior.
|
|
|
|
### Implementation
|
|
|
|
1. In `internal/usecase/runner.go`, add one private generation-error wrapper
|
|
used by both the initial `llm.Generate` call and every repair call:
|
|
- if the error matches `llm.ErrInvalidRequest`, wrap it with
|
|
`ErrInvalidRequest`; and
|
|
- otherwise wrap it with `ErrLLMGenerate`.
|
|
Use `%w` wrapping so provider details, cancellation, and deadlines remain
|
|
discoverable. Do not classify a failed corrective model call as
|
|
`ErrValidation`.
|
|
2. Treat a nil repair response without an error as a generation-side failure
|
|
matching `ErrLLMGenerate`. Continue returning no partial result for that or
|
|
any other operational error.
|
|
3. Pass `prepared.Messages` as `RepairRequest.OriginalMessages` on every
|
|
attempt. Continue passing the latest response content and latest complete
|
|
validation error slice, so each attempt repairs only the newest candidate.
|
|
4. Extend `shouldAttemptRepair` to include `ValidationBasic`. Retain all other
|
|
gates: a configured repairer, positive budget, and
|
|
`ValidationFailed`. `ValidationNone`, passed/skipped validation, and
|
|
operational validator errors never reach the repairer.
|
|
5. Preserve the existing loop shape and attempt semantics: increment the
|
|
attempt count immediately before starting a corrective call, validate every
|
|
completed response with the retained operation-local plan, stop on the first
|
|
valid response, and make no more calls than the frozen budget.
|
|
6. Preserve final-result coherence and cumulative usage. A successful or
|
|
exhausted run uses the same final candidate for `RawOutput`, artifact, and
|
|
validation; sums all five token-usage fields across completed generations;
|
|
reports actual calls in `ValidationResult.RepairAttempts`; and returns the
|
|
complete latest diagnostics on exhaustion.
|
|
7. Do not change admission or capacity architecture. The admission release
|
|
remains deferred across the full execution, and the default repairer must
|
|
continue using the same capacity-wrapped client so each correction acquires
|
|
the normal active-generation permit without a second run admission.
|
|
|
|
### Tests
|
|
|
|
1. Refactor `TestRunnerRepairStateMachine` rather than adding a parallel
|
|
matrix:
|
|
- change the obsolete “basic failure is ineligible” case into an empty/basic
|
|
repair success or bounded exhaustion case;
|
|
- keep initial success, stop-on-valid, exact exhaustion, JSON, JSON Schema,
|
|
target-presence, structured-output, session, direct credential, final
|
|
candidate, and cumulative-usage assertions;
|
|
- replace the now-invalid budget-four fixture with a maximum-three fixture;
|
|
- assert each `RepairRequest` receives the original rendered messages and
|
|
only the latest output/errors; and
|
|
- update repair-call detection helpers that currently recognize the old
|
|
standalone system prompt so they recognize the appended assistant/user
|
|
shape without depending on exact correction prose.
|
|
2. Add a zero-budget failed-validation case if the existing public test is its
|
|
only owner, ensuring the internal runner remains single-pass without invoking
|
|
a configured repairer. A positive `none` case belongs to Stage 1 contract
|
|
validation, not this state machine.
|
|
3. Update prepared-execution tests to prove initial and repaired candidates use
|
|
the same frozen validation plan and messages, sources are not reopened, and
|
|
the handle remains one-shot.
|
|
4. Update execution-error tables so a repairer generation failure expects
|
|
`ErrLLMGenerate`; add a repair error matching `llm.ErrInvalidRequest` and
|
|
assert it maps to `ErrInvalidRequest`. Preserve underlying sentinel,
|
|
cancellation, and deadline identity and confirm all operational failures
|
|
return no partial result.
|
|
5. Retain and adapt the shared-backend-pool test to prove two repaired runs
|
|
each receive one admission, all initial and repair calls use the same
|
|
backend pool, peak active generation does not exceed the configured limit,
|
|
and every lease/permit is released.
|
|
6. Add a focused cancellation case only if existing capacity coverage does not
|
|
already prove cancellation while a corrective call waits for its active
|
|
permit prevents that provider call and releases the run admission. Avoid
|
|
duplicating `internal/capacity`'s scheduler matrix.
|
|
|
|
### Verification
|
|
|
|
```sh
|
|
gofmt -w internal/usecase/runner.go internal/usecase/runner_test.go
|
|
gofmt -w internal/usecase/prepared_execution_test.go
|
|
go test ./internal/usecase -run 'Repair|Prepared|Admission|Capacity|Cancellation'
|
|
go test -race ./internal/usecase -run 'Repair|Capacity|Cancellation'
|
|
go test ./internal/usecase
|
|
go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
Stage 4 is complete when the internal state machine satisfies every bounded
|
|
success, exhaustion, error, prepared-state, and concurrency invariant without
|
|
being publicly activated yet.
|
|
|
|
## Stage 5: Activate Repair Through Public Engine Assembly
|
|
|
|
### Objective
|
|
|
|
Make the completed behavior available through the existing public API for both
|
|
built-in and injected clients, and align exported Go contracts with the now
|
|
active behavior.
|
|
|
|
### Implementation
|
|
|
|
1. In `engine.go`, after constructing or adapting the selected `llm.Client` and
|
|
wrapping it with `capacity.NewClient`, construct
|
|
`usecase.NewDefaultOutputRepairer(llmClient)` and assemble the engine runner
|
|
with `usecase.NewRunnerWithRepairer`. Pass that exact same wrapped client as
|
|
the runner's initial-generation client and the repairer's client.
|
|
2. Do not add a public engine option or expose the internal repairer. Keep
|
|
`usecase.NewRunner` as the no-repair convenience constructor for focused
|
|
internal tests and callers; the root facade deliberately uses the explicit
|
|
repairer constructor.
|
|
3. Update exported GoDoc in `types.go`:
|
|
- `OutputContract.RepairAttempts` is an additional-call budget from zero
|
|
through three and is valid only with `basic`, `json`, or `json_schema` when
|
|
positive;
|
|
- `ValidationResult.RepairAttempts` is the actual number of corrective calls
|
|
started and may be nonzero in public results; and
|
|
- `LLMResponse.Content` may be explicitly empty and will be interpreted by
|
|
the effective output contract rather than rejected by Promptkit's public
|
|
adapter.
|
|
4. Update `Engine.Run`, `Engine.RunPrepared`, and related prepared/result GoDoc
|
|
in `engine.go` to describe opt-in bounded correction, cumulative usage,
|
|
exhaustion as a returned failed validation result, operational errors as no
|
|
result, cancellation, and one-shot prepared execution. Remove statements
|
|
that the public engine never repairs or is always single-pass.
|
|
5. Keep the public JSON representation and public type set unchanged. Do not
|
|
expose correction messages, candidate history, a semantic-validity claim, or
|
|
a new error sentinel.
|
|
|
|
### Tests
|
|
|
|
Use root external-package contract tests for representative assembly behavior,
|
|
reusing existing public fakes where possible:
|
|
|
|
1. Replace the obsolete public “positive budget is still single-pass” assertion
|
|
with a zero-budget failed-validation assertion that remains exactly one
|
|
generation and reports zero attempts.
|
|
2. Add one ordinary `Engine.Run` workflow with an injected `LLMClient` that
|
|
returns invalid JSON and then valid JSON. Assert two calls, the final valid
|
|
candidate, one actual repair, cumulative usage, the same target/session and
|
|
structured-output values, and no new configuration surface.
|
|
3. Add one `PrepareExecution`/`RunPrepared` workflow using a different eligible
|
|
mode, preferably `basic` with an explicit empty initial candidate, to prove
|
|
public prepared parity, frozen messages/settings, and one-shot behavior.
|
|
4. Add one bounded exhaustion assertion that returns a non-nil result, nil
|
|
error, final invalid candidate, complete final diagnostics, exact actual
|
|
attempt count, and cumulative usage. Keep the full progression matrix in
|
|
`internal/usecase`.
|
|
5. Use an offline `httptest` OpenAI-compatible endpoint for one built-in-client
|
|
integration path: return a structurally valid completion envelope whose
|
|
content fails validation, then a provider error on correction. Assert no
|
|
partial result, `ErrLLMGenerate`, available public `GenerationError` details,
|
|
and preserved underlying cancellation/deadline behavior where applicable.
|
|
Do not duplicate every provider failure shape already owned by
|
|
`internal/llm`.
|
|
6. Ensure existing capacity tests for injected clients still pass with the
|
|
repairer installed, especially engine-local backend limits and unlimited
|
|
backend behavior when repair is not requested.
|
|
|
|
### Verification
|
|
|
|
```sh
|
|
gofmt -w engine.go types.go output_contract_contract_test.go public_contract_test.go
|
|
gofmt -w prepared_execution_contract_test.go engine_test.go
|
|
go test ./... -run 'Repair|OutputContract|RunPrepared|GenerationError|Capacity'
|
|
go test -race ./... -run 'Repair|RunPrepared|Capacity'
|
|
go test ./...
|
|
git diff --check
|
|
```
|
|
|
|
If a named root test file does not require a change, omit it from `gofmt`
|
|
rather than touching it mechanically. Stage 5 is complete when ordinary and
|
|
prepared public workflows repair through either client path with unchanged API
|
|
shape and correct public result and error semantics.
|
|
|
|
## Stage 6: Publish Canonical Documentation And Run Maintainer Validation
|
|
|
|
### Objective
|
|
|
|
Make every canonical documentation owner describe the implemented feature,
|
|
remove obsolete dormant/single-pass claims, and validate the complete
|
|
cross-cutting change using the repository's canonical workflow.
|
|
|
|
### Implementation
|
|
|
|
1. Update `docs/formats.md`, the exact prompt-format owner, to document:
|
|
- `repair_attempts` as an integer from zero through three;
|
|
- zero as the single-pass default and positive values as additional calls;
|
|
- eligible `basic`, `json`, and `json_schema` modes;
|
|
- invalid positive budgets with `none`;
|
|
- stop-on-valid and exhausted-final-result behavior; and
|
|
- plain JSON's acceptance of every JSON value without an object-only wire
|
|
constraint.
|
|
2. Update `docs/consumers/pkg-promptkit.md` with one concise JSON Schema repair
|
|
example using the existing public field or prompt YAML. Explain cumulative
|
|
usage, actual attempt reporting, the possibility of a final failed
|
|
validation after exhaustion, and the cost/latency implications of extra
|
|
calls. Note briefly that `basic` can repair empty output and that structural
|
|
validity is not factual or domain correctness.
|
|
3. Update `docs/integrations/openai-compatible-chat.md` to own the provider-wire
|
|
distinction: an explicitly present string `content`, including empty or
|
|
whitespace-only, is a candidate; absent choices, missing content, `null`,
|
|
and non-string content are malformed. Retain provider-native JSON Schema as
|
|
the first defense and do not claim an object-only constraint for plain JSON.
|
|
4. Update `docs/internal/llm.md` with the same internal response-decoding
|
|
boundary without duplicating public repair orchestration.
|
|
5. Update `docs/internal/runner.md` to describe the installed default repairer,
|
|
full-original-message/latest-candidate request shape, empty-candidate role
|
|
behavior, diagnostic bound, eligible modes, attempt/usage accounting,
|
|
exhaustion, generation-versus-validation error categories, prepared-plan
|
|
reuse, and one-admission/shared-capacity behavior. Remove the claim that the
|
|
ordinary public runner omits a repairer.
|
|
6. Update `docs/internal/overview.md` so root assembly and `internal/usecase`
|
|
ownership reflect public bounded repair. Review `docs/internal/capacity.md`
|
|
and `docs/internal/sources.md`; edit only statements that are stale or
|
|
incomplete after implementation, preserving their narrow ownership.
|
|
7. Do not put canonical behavior in a release note or roadmap. Do not update
|
|
the architecture policy unless implementation actually crossed the package
|
|
boundaries fixed by this plan. Do not retire `output-repair.md` or this plan
|
|
in this implementation sequence.
|
|
8. Check every changed Markdown link and heading fragment with the canonical
|
|
offline link checker in the development guide. Review the final diff for
|
|
consistent terminology: “repair attempt” always means an additional model
|
|
call, “exhaustion” returns a completed failed validation result, and no text
|
|
promises semantic correctness or provider caching.
|
|
|
|
### Tests And Final Verification
|
|
|
|
Run the complete
|
|
[maintainer validation workflow](../development.md#maintainer-validation) from
|
|
the repository root, including both maintained offline examples:
|
|
|
|
```sh
|
|
go test ./...
|
|
go test -race ./...
|
|
go vet ./...
|
|
go build ./...
|
|
go run ./examples/go-library/prepare
|
|
go run ./examples/go-library/run
|
|
git diff --check
|
|
git status --short
|
|
```
|
|
|
|
Also run the development guide's tracked-Go formatting check and local
|
|
Markdown-link checker exactly as documented. Review example JSON output for
|
|
the invariants listed there; neither example may contact a real provider or
|
|
require credentials.
|
|
|
|
Stage 6 is complete when code, tests, exported GoDoc, format documentation,
|
|
consumer guidance, integration contracts, and internal documentation agree;
|
|
all canonical validation commands pass; and the working tree contains only the
|
|
intended feature implementation and roadmap changes.
|
|
|
|
## Open Questions
|
|
|
|
None. The feature roadmap and the fixed decisions above define all behavior
|
|
needed for implementation.
|