Plan the public output repair implementation
This commit is contained in:
591
docs/roadmap/implementation.md
Normal file
591
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,591 @@
|
||||
# Public Bounded Output Repair Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement the feature defined by the
|
||||
[public bounded output repair roadmap](output-repair.md). This document is the
|
||||
decision-complete execution sequence for a coding agent. Follow the repository
|
||||
[architecture](../policy/architecture.md), [testing](../policy/testing.md), and
|
||||
[documentation](../policy/documentation.md) policies and the task-specific
|
||||
reading guide in the [development guide](../development.md) throughout the
|
||||
work.
|
||||
|
||||
## Target Outcome
|
||||
|
||||
Promptkit's public `Engine.Run` and `Engine.RunPrepared` workflows honor the
|
||||
existing repair budget in an effective output contract. A failed `basic`,
|
||||
`json`, or `json_schema` validation may make at most three additional calls
|
||||
through the same resolved target, session, credential boundary, prepared
|
||||
schema, and capacity-wrapped client. Each correction receives the original
|
||||
rendered conversation, the latest nonempty candidate, and bounded diagnostics.
|
||||
|
||||
The first valid candidate ends the operation. Exhaustion returns the final
|
||||
invalid candidate and complete final diagnostics as an ordinary result, while
|
||||
generation, validation, capacity, and cancellation failures retain their
|
||||
existing public error categories and identities. Zero remains single-pass, no
|
||||
public repair interface is added, and an explicitly present empty provider
|
||||
string reaches validation instead of being mistaken for a malformed envelope.
|
||||
|
||||
## Fixed Decisions
|
||||
|
||||
- `OutputContract.RepairAttempts` and prompt YAML `repair_attempts` are the
|
||||
only consumer controls. The value counts additional generation calls and
|
||||
must be in the inclusive range zero through three.
|
||||
- A positive budget with `none` validation is invalid. Positive budgets are
|
||||
eligible only after a completed failed validation under `basic`, `json`, or
|
||||
`json_schema`.
|
||||
- The public engine installs one internal default repairer. There is no new
|
||||
public option, interface, callback, prompt override, or mutable registry.
|
||||
- Every corrective call starts with a fresh copy of the complete original
|
||||
rendered messages. It appends only the latest nonempty candidate as an
|
||||
`assistant` message and one corrective `user` message. Earlier failed
|
||||
candidates and corrective messages do not accumulate.
|
||||
- Empty and whitespace-only candidates omit the additional assistant message.
|
||||
The corrective user message states that the prior response was empty.
|
||||
- The corrective instruction is application-neutral. `basic` requests a
|
||||
nonempty answer to the original task; `json` and `json_schema` request only
|
||||
corrected JSON without explanation or Markdown fences.
|
||||
- Validation diagnostics are represented as bounded data in the correction
|
||||
message. Their encoded feedback is at most 65,536 UTF-8 bytes, preserves
|
||||
order, and includes an in-bound omission notice when truncated. This bound
|
||||
never alters the complete diagnostics retained by validation or returned in
|
||||
the final result.
|
||||
- The latest nonempty candidate is included verbatim and in full. Promptkit
|
||||
adds no repair-specific candidate-size limit.
|
||||
- JSON Schema repair retains the exact prepared structured-output
|
||||
specification. Plain JSON does not gain a provider-native JSON-object
|
||||
constraint because its contract permits every JSON value.
|
||||
- Explicitly present string content from the built-in OpenAI-compatible client
|
||||
is a successful generation candidate even when empty or whitespace-only.
|
||||
Missing content, `null`, non-string content, and absent choices remain
|
||||
malformed provider responses and never enter repair.
|
||||
- The initial generation and every correction use the same capacity-wrapped
|
||||
`llm.Client`. One run-admission lease spans the complete operation, while
|
||||
each model call independently acquires and releases the selected backend's
|
||||
active-generation permit.
|
||||
- `ValidationResult.RepairAttempts` is execution outcome state and reports the
|
||||
number of corrective calls actually started. It is not initialized from the
|
||||
requested budget by the validation package.
|
||||
- Usage is the field-by-field sum of every completed generation. An
|
||||
operational error returns no partial `RunResult`; repair exhaustion is not
|
||||
an operational error.
|
||||
- A corrective generation failure is categorized exactly like an initial
|
||||
generation failure: invalid model requests match `ErrInvalidRequest`, other
|
||||
provider or transport failures match `ErrLLMGenerate`, typed generation
|
||||
details remain available, and cancellation and deadline identities remain
|
||||
discoverable.
|
||||
- No release notes, roadmap retirement, version changes, commits, tags, or
|
||||
pushes are part of these stages.
|
||||
|
||||
## Execution Rules
|
||||
|
||||
- Complete the stages in numerical order. Each stage is sized for one
|
||||
gpt-5.6-terra implementation prompt and must finish its focused verification
|
||||
before the next begins.
|
||||
- At the start of each stage, inspect the working tree and preserve unrelated
|
||||
changes. The feature roadmap and this plan are intentional planning state.
|
||||
- Treat intermediate stages as an unreleased partial implementation. Durable
|
||||
current-state documentation changes belong to Stage 6, after public engine
|
||||
assembly and behavior are complete.
|
||||
- Keep source-neutral contract legality in `internal/domain`, structural
|
||||
validity in `internal/validate`, conversation construction in the default
|
||||
repairer, orchestration and error categorization in `internal/usecase`, wire
|
||||
envelope interpretation in `internal/llm`, and public assembly in the root
|
||||
package.
|
||||
- Do not move repair behavior into the root facade, validator, capacity
|
||||
manager, public client adapter, or provider transport merely to simplify a
|
||||
test.
|
||||
- Reuse the existing prepared validation plan and generation request
|
||||
constructor. Do not reopen prompt, profile, backend, artifact, or schema
|
||||
sources and do not rerender during correction.
|
||||
- Update existing test owners when a contract changes. Add only representative
|
||||
public workflow tests; do not duplicate the internal state-machine matrix at
|
||||
the root package.
|
||||
- Tests must be deterministic, offline, parallel-safe, race-safe, and
|
||||
independent of real credentials or provider services. Use injected fakes,
|
||||
synthetic filesystems, and `httptest` where provider-wire behavior matters.
|
||||
- Use `apply_patch` for edits, format every changed Go file, review the full
|
||||
stage diff, and run `git diff --check` before moving to the next stage.
|
||||
|
||||
## Stage 1: Enforce The Repair Contract At Its Source-Neutral Boundary
|
||||
|
||||
### Objective
|
||||
|
||||
Make every supported output-contract source agree on the zero-to-three budget
|
||||
and invalid `none` pairing, and establish that actual attempt accounting belongs
|
||||
to execution rather than validation. This stage changes legality and result
|
||||
ownership but does not activate repair in the public engine.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/domain/output_contract.go`, add one unexported
|
||||
`maxOutputRepairAttempts = 3` constant and extend
|
||||
`ValidateOutputContract` to reject:
|
||||
- negative `RepairAttempts`;
|
||||
- values greater than three; and
|
||||
- a positive value when `ValidationMode` is `ValidationNone`.
|
||||
2. Keep the existing format/mode/schema checks in the same helper. Do not clamp
|
||||
invalid budgets and do not add the maximum as a root-package public
|
||||
constant; exact public semantics will be documented through existing fields.
|
||||
3. Preserve the current normalization and replacement paths so prompt files,
|
||||
request-level `OutputContract` replacements, `Prepare`, and
|
||||
`PrepareExecution` all reach the shared domain helper. Do not add parallel
|
||||
range checks in the root facade or prompt repository.
|
||||
4. In `internal/validate/standard_validator.go`, stop copying the configured
|
||||
`contract.RepairAttempts` into a newly created `ValidationResult`. A direct
|
||||
validation result starts with zero attempts. Retain the existing use-case
|
||||
wrappers that assign the actual `attemptsUsed` value after each completed
|
||||
validation.
|
||||
5. Preserve output-contract whole-value replacement behavior and all existing
|
||||
validation modes, schema preparation, and error wrapping apart from the new
|
||||
invalid cases.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Extend `internal/domain/output_contract_test.go` with a focused boundary
|
||||
table covering `-1`, `0`, `1`, `3`, and `4`, plus `none` with zero and with
|
||||
a positive value. Use representative valid basic/JSON contracts so unrelated
|
||||
validation errors do not mask the intended assertion.
|
||||
2. Extend the existing prompt-definition repository tests with representative
|
||||
YAML cases proving a budget above three and a positive `none` pairing are
|
||||
rejected through the repository's existing invalid-prompt identity. Do not
|
||||
duplicate the full domain table there.
|
||||
3. Update `internal/usecase/output_contract_test.go` to prove a request
|
||||
replacement with either new invalid condition fails before completion
|
||||
collaborators or admission. Cover both ordinary and prepared preparation
|
||||
through the existing shared test structure instead of creating a second
|
||||
matrix.
|
||||
4. Extend the root output-contract contract test only as needed to preserve the
|
||||
established public error mapping for invalid request replacements. Source
|
||||
file failures retain their existing prompt-load category.
|
||||
5. Update validator tests to assert a direct failed or passed validation reports
|
||||
zero attempts even when the input contract carries a positive budget. Keep
|
||||
execution-level attempt assertions in use-case tests.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/domain/output_contract.go internal/domain/output_contract_test.go
|
||||
gofmt -w internal/validate/standard_validator.go internal/validate/standard_validator_test.go
|
||||
gofmt -w internal/promptdef/repository_test.go internal/usecase/output_contract_test.go
|
||||
gofmt -w output_contract_contract_test.go
|
||||
go test ./internal/domain ./internal/validate ./internal/promptdef ./internal/usecase
|
||||
go test ./... -run 'OutputContract|ValidationResult'
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 1 is complete when every input path accepts only a coherent zero-to-three
|
||||
budget, invalid contracts fail before model work, and validators no longer
|
||||
misreport a configured budget as completed repair work.
|
||||
|
||||
## Stage 2: Distinguish Explicit Empty Content From A Malformed Envelope
|
||||
|
||||
### Objective
|
||||
|
||||
Allow the built-in OpenAI-compatible client to return an explicitly present
|
||||
empty or whitespace string as a generation candidate while retaining strict
|
||||
malformed-response handling for absent or wrongly typed content.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/llm/openai_compatible_client.go`, make
|
||||
`openAIChatResponseMessage.Content` presence-aware by decoding it as a
|
||||
`*string`.
|
||||
2. After response framing and JSON decoding succeed, require at least one
|
||||
choice and a non-nil first-choice content pointer. Dereference and return
|
||||
the string exactly as received without trimming or rejecting an empty
|
||||
value.
|
||||
3. Treat a missing field and explicit `null` as malformed because both decode
|
||||
to nil. Let a non-string value fail JSON decoding and remain wrapped with
|
||||
`ErrMalformedResponse`. Preserve the existing no-choice, trailing-data,
|
||||
response-size, usage, provider-error, and body-ownership behavior.
|
||||
4. Do not perform output validation or decide repair eligibility in the model
|
||||
client. This stage changes only the interpretation of an otherwise
|
||||
successful completion envelope.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Add focused cases to `internal/llm/openai_compatible_client_test.go` proving
|
||||
explicit `""` and whitespace-only string content return successful
|
||||
`GenerateResponse` values with the exact content and mapped usage.
|
||||
2. Add or retain distinct malformed-response cases for no choices, missing
|
||||
content, `null` content, and non-string content. Each must return no response
|
||||
and match `ErrMalformedResponse`.
|
||||
3. Keep these tests at the model-client owner. Do not use a root engine or
|
||||
duplicate validation-mode behavior in this stage.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/llm/openai_compatible_client.go internal/llm/openai_compatible_client_test.go
|
||||
go test ./internal/llm -run 'OpenAICompatibleClient|Response'
|
||||
go test ./internal/llm
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 2 is complete when content presence is distinguishable from content
|
||||
emptiness and all malformed successful-envelope cases retain their prior error
|
||||
identity.
|
||||
|
||||
## Stage 3: Build Full-Context, Prompt-Safe Corrective Requests
|
||||
|
||||
### Objective
|
||||
|
||||
Refactor the dormant default repairer so it constructs one correction from the
|
||||
original rendered conversation and latest validation state, supports basic and
|
||||
JSON modes, and safely bounds only diagnostic feedback.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Extend the internal `RepairRequest` in `internal/usecase/repairer.go` with
|
||||
`OriginalMessages []domain.RenderedMessage`. Keep the type internal and do
|
||||
not add repair fields to `internal/domain` or the public package.
|
||||
2. Replace the standalone system/user repair prompt with a freshly allocated
|
||||
message slice on every call:
|
||||
- copy every original rendered message in order without changing role,
|
||||
content, or cache-control metadata;
|
||||
- if `strings.TrimSpace(PreviousOutput)` is nonempty, append one `assistant`
|
||||
message whose content is the exact, full `PreviousOutput` string;
|
||||
- otherwise append no assistant message; and
|
||||
- append one corrective `user` message after the original conversation and
|
||||
optional candidate.
|
||||
Allocate independently from `OriginalMessages` so `append` cannot mutate a
|
||||
prepared snapshot's backing array.
|
||||
3. Construct one stable application-neutral corrective message that includes
|
||||
the validation mode, `Attempt` and `MaxAttempts`, an instruction to preserve
|
||||
valid values and change only what is necessary, and diagnostics clearly
|
||||
labeled as data rather than instructions. For an empty/whitespace candidate,
|
||||
explicitly state that the previous response was empty and do not reproduce
|
||||
its whitespace.
|
||||
4. Use mode-specific terminal guidance:
|
||||
- `ValidationBasic`: return a nonempty response satisfying the original
|
||||
request; and
|
||||
- `ValidationJSON` or `ValidationJSONSchema`: return only corrected JSON,
|
||||
with no explanation or Markdown fence.
|
||||
Return a defensive internal error for any other mode; the runner must never
|
||||
call the repairer for it after Stage 4.
|
||||
5. Add an unexported `maxRepairDiagnosticBytes = 64 * 1024` constant and one
|
||||
focused diagnostic formatter. Encode the ordered diagnostics as a JSON array
|
||||
within the corrective message so provider-controlled text remains visibly
|
||||
data. The formatter must:
|
||||
- convert malformed input strings to valid UTF-8 for prompt transport;
|
||||
- return the complete encoded array when it is within the limit;
|
||||
- when over the limit, preserve complete earlier entries, include as much
|
||||
of the next entry as fits at a rune boundary, omit all remaining text, and
|
||||
append a stable final array entry stating that additional validation
|
||||
diagnostics were omitted;
|
||||
- account for JSON quoting, delimiters, and the omission entry when enforcing
|
||||
the 65,536-byte maximum; and
|
||||
- return a valid JSON array at or below the limit for every input, including
|
||||
one oversized error, many errors, control characters, multibyte runes, and
|
||||
an empty list.
|
||||
This limit applies only to the encoded diagnostic block, not fixed repair
|
||||
instructions, the previous candidate, or `ValidationResult.Errors`.
|
||||
6. Continue to call `newGenerationRequest` with the request's exact session,
|
||||
target, target-presence metadata, and structured-output pointer. Preserve
|
||||
the existing nil-client and nil-response defenses. Do not locally strip,
|
||||
parse, normalize, or repair the candidate.
|
||||
|
||||
### Tests
|
||||
|
||||
Create `internal/usecase/repairer_test.go` as the narrow owner of default
|
||||
repairer behavior:
|
||||
|
||||
1. Supply original system and user messages, including cache-control metadata,
|
||||
and assert the generated request contains those values unchanged followed
|
||||
by the exact assistant candidate and one corrective user message.
|
||||
2. Prove separate correction calls begin from the same original messages and
|
||||
do not mutate the caller's slice or accumulate an earlier candidate or
|
||||
corrective message.
|
||||
3. Cover empty and whitespace-only candidates: no assistant message is added,
|
||||
the user message identifies the empty response, and no placeholder model
|
||||
content is invented.
|
||||
4. Cover basic, JSON, and JSON Schema guidance by stable behavioral phrases and
|
||||
role boundaries rather than snapshotting the full incidental prose. Assert
|
||||
an unsupported mode returns an error without calling the client.
|
||||
5. Prove the exact target, presence flags, session ID, and structured-output
|
||||
specification reach the model client and that an arbitrarily large nonempty
|
||||
candidate is not truncated.
|
||||
6. Exercise diagnostics below, at, and above the limit, including ordered
|
||||
entries, multibyte UTF-8 near the boundary, invalid UTF-8, and one huge
|
||||
diagnostic. Assert the encoded block is valid JSON, valid UTF-8, no more
|
||||
than 65,536 bytes, ordered, and contains the omission notice only when
|
||||
needed. Also assert the input error slice and strings remain unchanged.
|
||||
7. Retain focused nil-client, generation-error passthrough, and nil-response
|
||||
tests without duplicating runner error categorization.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/repairer.go internal/usecase/repairer_test.go
|
||||
go test ./internal/usecase -run 'Repairer|RepairRequest'
|
||||
go test ./internal/usecase
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 3 is complete when the default repairer produces bounded, mode-correct,
|
||||
full-context requests without mutating prepared state or retaining failed
|
||||
history.
|
||||
|
||||
## Stage 4: Complete The Internal Repair State Machine And Error Semantics
|
||||
|
||||
### Objective
|
||||
|
||||
Connect the improved repairer contract to runner execution, make `basic`
|
||||
eligible, and ensure corrective generation failures share the initial
|
||||
generation error path while preserving bounded state, usage, prepared
|
||||
validation, admission, and capacity behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `internal/usecase/runner.go`, add one private generation-error wrapper
|
||||
used by both the initial `llm.Generate` call and every repair call:
|
||||
- if the error matches `llm.ErrInvalidRequest`, wrap it with
|
||||
`ErrInvalidRequest`; and
|
||||
- otherwise wrap it with `ErrLLMGenerate`.
|
||||
Use `%w` wrapping so provider details, cancellation, and deadlines remain
|
||||
discoverable. Do not classify a failed corrective model call as
|
||||
`ErrValidation`.
|
||||
2. Treat a nil repair response without an error as a generation-side failure
|
||||
matching `ErrLLMGenerate`. Continue returning no partial result for that or
|
||||
any other operational error.
|
||||
3. Pass `prepared.Messages` as `RepairRequest.OriginalMessages` on every
|
||||
attempt. Continue passing the latest response content and latest complete
|
||||
validation error slice, so each attempt repairs only the newest candidate.
|
||||
4. Extend `shouldAttemptRepair` to include `ValidationBasic`. Retain all other
|
||||
gates: a configured repairer, positive budget, and
|
||||
`ValidationFailed`. `ValidationNone`, passed/skipped validation, and
|
||||
operational validator errors never reach the repairer.
|
||||
5. Preserve the existing loop shape and attempt semantics: increment the
|
||||
attempt count immediately before starting a corrective call, validate every
|
||||
completed response with the retained operation-local plan, stop on the first
|
||||
valid response, and make no more calls than the frozen budget.
|
||||
6. Preserve final-result coherence and cumulative usage. A successful or
|
||||
exhausted run uses the same final candidate for `RawOutput`, artifact, and
|
||||
validation; sums all five token-usage fields across completed generations;
|
||||
reports actual calls in `ValidationResult.RepairAttempts`; and returns the
|
||||
complete latest diagnostics on exhaustion.
|
||||
7. Do not change admission or capacity architecture. The admission release
|
||||
remains deferred across the full execution, and the default repairer must
|
||||
continue using the same capacity-wrapped client so each correction acquires
|
||||
the normal active-generation permit without a second run admission.
|
||||
|
||||
### Tests
|
||||
|
||||
1. Refactor `TestRunnerRepairStateMachine` rather than adding a parallel
|
||||
matrix:
|
||||
- change the obsolete “basic failure is ineligible” case into an empty/basic
|
||||
repair success or bounded exhaustion case;
|
||||
- keep initial success, stop-on-valid, exact exhaustion, JSON, JSON Schema,
|
||||
target-presence, structured-output, session, direct credential, final
|
||||
candidate, and cumulative-usage assertions;
|
||||
- replace the now-invalid budget-four fixture with a maximum-three fixture;
|
||||
- assert each `RepairRequest` receives the original rendered messages and
|
||||
only the latest output/errors; and
|
||||
- update repair-call detection helpers that currently recognize the old
|
||||
standalone system prompt so they recognize the appended assistant/user
|
||||
shape without depending on exact correction prose.
|
||||
2. Add a zero-budget failed-validation case if the existing public test is its
|
||||
only owner, ensuring the internal runner remains single-pass without invoking
|
||||
a configured repairer. A positive `none` case belongs to Stage 1 contract
|
||||
validation, not this state machine.
|
||||
3. Update prepared-execution tests to prove initial and repaired candidates use
|
||||
the same frozen validation plan and messages, sources are not reopened, and
|
||||
the handle remains one-shot.
|
||||
4. Update execution-error tables so a repairer generation failure expects
|
||||
`ErrLLMGenerate`; add a repair error matching `llm.ErrInvalidRequest` and
|
||||
assert it maps to `ErrInvalidRequest`. Preserve underlying sentinel,
|
||||
cancellation, and deadline identity and confirm all operational failures
|
||||
return no partial result.
|
||||
5. Retain and adapt the shared-backend-pool test to prove two repaired runs
|
||||
each receive one admission, all initial and repair calls use the same
|
||||
backend pool, peak active generation does not exceed the configured limit,
|
||||
and every lease/permit is released.
|
||||
6. Add a focused cancellation case only if existing capacity coverage does not
|
||||
already prove cancellation while a corrective call waits for its active
|
||||
permit prevents that provider call and releases the run admission. Avoid
|
||||
duplicating `internal/capacity`'s scheduler matrix.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w internal/usecase/runner.go internal/usecase/runner_test.go
|
||||
gofmt -w internal/usecase/prepared_execution_test.go
|
||||
go test ./internal/usecase -run 'Repair|Prepared|Admission|Capacity|Cancellation'
|
||||
go test -race ./internal/usecase -run 'Repair|Capacity|Cancellation'
|
||||
go test ./internal/usecase
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Stage 4 is complete when the internal state machine satisfies every bounded
|
||||
success, exhaustion, error, prepared-state, and concurrency invariant without
|
||||
being publicly activated yet.
|
||||
|
||||
## Stage 5: Activate Repair Through Public Engine Assembly
|
||||
|
||||
### Objective
|
||||
|
||||
Make the completed behavior available through the existing public API for both
|
||||
built-in and injected clients, and align exported Go contracts with the now
|
||||
active behavior.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. In `engine.go`, after constructing or adapting the selected `llm.Client` and
|
||||
wrapping it with `capacity.NewClient`, construct
|
||||
`usecase.NewDefaultOutputRepairer(llmClient)` and assemble the engine runner
|
||||
with `usecase.NewRunnerWithRepairer`. Pass that exact same wrapped client as
|
||||
the runner's initial-generation client and the repairer's client.
|
||||
2. Do not add a public engine option or expose the internal repairer. Keep
|
||||
`usecase.NewRunner` as the no-repair convenience constructor for focused
|
||||
internal tests and callers; the root facade deliberately uses the explicit
|
||||
repairer constructor.
|
||||
3. Update exported GoDoc in `types.go`:
|
||||
- `OutputContract.RepairAttempts` is an additional-call budget from zero
|
||||
through three and is valid only with `basic`, `json`, or `json_schema` when
|
||||
positive;
|
||||
- `ValidationResult.RepairAttempts` is the actual number of corrective calls
|
||||
started and may be nonzero in public results; and
|
||||
- `LLMResponse.Content` may be explicitly empty and will be interpreted by
|
||||
the effective output contract rather than rejected by Promptkit's public
|
||||
adapter.
|
||||
4. Update `Engine.Run`, `Engine.RunPrepared`, and related prepared/result GoDoc
|
||||
in `engine.go` to describe opt-in bounded correction, cumulative usage,
|
||||
exhaustion as a returned failed validation result, operational errors as no
|
||||
result, cancellation, and one-shot prepared execution. Remove statements
|
||||
that the public engine never repairs or is always single-pass.
|
||||
5. Keep the public JSON representation and public type set unchanged. Do not
|
||||
expose correction messages, candidate history, a semantic-validity claim, or
|
||||
a new error sentinel.
|
||||
|
||||
### Tests
|
||||
|
||||
Use root external-package contract tests for representative assembly behavior,
|
||||
reusing existing public fakes where possible:
|
||||
|
||||
1. Replace the obsolete public “positive budget is still single-pass” assertion
|
||||
with a zero-budget failed-validation assertion that remains exactly one
|
||||
generation and reports zero attempts.
|
||||
2. Add one ordinary `Engine.Run` workflow with an injected `LLMClient` that
|
||||
returns invalid JSON and then valid JSON. Assert two calls, the final valid
|
||||
candidate, one actual repair, cumulative usage, the same target/session and
|
||||
structured-output values, and no new configuration surface.
|
||||
3. Add one `PrepareExecution`/`RunPrepared` workflow using a different eligible
|
||||
mode, preferably `basic` with an explicit empty initial candidate, to prove
|
||||
public prepared parity, frozen messages/settings, and one-shot behavior.
|
||||
4. Add one bounded exhaustion assertion that returns a non-nil result, nil
|
||||
error, final invalid candidate, complete final diagnostics, exact actual
|
||||
attempt count, and cumulative usage. Keep the full progression matrix in
|
||||
`internal/usecase`.
|
||||
5. Use an offline `httptest` OpenAI-compatible endpoint for one built-in-client
|
||||
integration path: return a structurally valid completion envelope whose
|
||||
content fails validation, then a provider error on correction. Assert no
|
||||
partial result, `ErrLLMGenerate`, available public `GenerationError` details,
|
||||
and preserved underlying cancellation/deadline behavior where applicable.
|
||||
Do not duplicate every provider failure shape already owned by
|
||||
`internal/llm`.
|
||||
6. Ensure existing capacity tests for injected clients still pass with the
|
||||
repairer installed, especially engine-local backend limits and unlimited
|
||||
backend behavior when repair is not requested.
|
||||
|
||||
### Verification
|
||||
|
||||
```sh
|
||||
gofmt -w engine.go types.go output_contract_contract_test.go public_contract_test.go
|
||||
gofmt -w prepared_execution_contract_test.go engine_test.go
|
||||
go test ./... -run 'Repair|OutputContract|RunPrepared|GenerationError|Capacity'
|
||||
go test -race ./... -run 'Repair|RunPrepared|Capacity'
|
||||
go test ./...
|
||||
git diff --check
|
||||
```
|
||||
|
||||
If a named root test file does not require a change, omit it from `gofmt`
|
||||
rather than touching it mechanically. Stage 5 is complete when ordinary and
|
||||
prepared public workflows repair through either client path with unchanged API
|
||||
shape and correct public result and error semantics.
|
||||
|
||||
## Stage 6: Publish Canonical Documentation And Run Maintainer Validation
|
||||
|
||||
### Objective
|
||||
|
||||
Make every canonical documentation owner describe the implemented feature,
|
||||
remove obsolete dormant/single-pass claims, and validate the complete
|
||||
cross-cutting change using the repository's canonical workflow.
|
||||
|
||||
### Implementation
|
||||
|
||||
1. Update `docs/formats.md`, the exact prompt-format owner, to document:
|
||||
- `repair_attempts` as an integer from zero through three;
|
||||
- zero as the single-pass default and positive values as additional calls;
|
||||
- eligible `basic`, `json`, and `json_schema` modes;
|
||||
- invalid positive budgets with `none`;
|
||||
- stop-on-valid and exhausted-final-result behavior; and
|
||||
- plain JSON's acceptance of every JSON value without an object-only wire
|
||||
constraint.
|
||||
2. Update `docs/consumers/pkg-promptkit.md` with one concise JSON Schema repair
|
||||
example using the existing public field or prompt YAML. Explain cumulative
|
||||
usage, actual attempt reporting, the possibility of a final failed
|
||||
validation after exhaustion, and the cost/latency implications of extra
|
||||
calls. Note briefly that `basic` can repair empty output and that structural
|
||||
validity is not factual or domain correctness.
|
||||
3. Update `docs/integrations/openai-compatible-chat.md` to own the provider-wire
|
||||
distinction: an explicitly present string `content`, including empty or
|
||||
whitespace-only, is a candidate; absent choices, missing content, `null`,
|
||||
and non-string content are malformed. Retain provider-native JSON Schema as
|
||||
the first defense and do not claim an object-only constraint for plain JSON.
|
||||
4. Update `docs/internal/llm.md` with the same internal response-decoding
|
||||
boundary without duplicating public repair orchestration.
|
||||
5. Update `docs/internal/runner.md` to describe the installed default repairer,
|
||||
full-original-message/latest-candidate request shape, empty-candidate role
|
||||
behavior, diagnostic bound, eligible modes, attempt/usage accounting,
|
||||
exhaustion, generation-versus-validation error categories, prepared-plan
|
||||
reuse, and one-admission/shared-capacity behavior. Remove the claim that the
|
||||
ordinary public runner omits a repairer.
|
||||
6. Update `docs/internal/overview.md` so root assembly and `internal/usecase`
|
||||
ownership reflect public bounded repair. Review `docs/internal/capacity.md`
|
||||
and `docs/internal/sources.md`; edit only statements that are stale or
|
||||
incomplete after implementation, preserving their narrow ownership.
|
||||
7. Do not put canonical behavior in a release note or roadmap. Do not update
|
||||
the architecture policy unless implementation actually crossed the package
|
||||
boundaries fixed by this plan. Do not retire `output-repair.md` or this plan
|
||||
in this implementation sequence.
|
||||
8. Check every changed Markdown link and heading fragment with the canonical
|
||||
offline link checker in the development guide. Review the final diff for
|
||||
consistent terminology: “repair attempt” always means an additional model
|
||||
call, “exhaustion” returns a completed failed validation result, and no text
|
||||
promises semantic correctness or provider caching.
|
||||
|
||||
### Tests And Final Verification
|
||||
|
||||
Run the complete
|
||||
[maintainer validation workflow](../development.md#maintainer-validation) from
|
||||
the repository root, including both maintained offline examples:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
go run ./examples/go-library/run
|
||||
git diff --check
|
||||
git status --short
|
||||
```
|
||||
|
||||
Also run the development guide's tracked-Go formatting check and local
|
||||
Markdown-link checker exactly as documented. Review example JSON output for
|
||||
the invariants listed there; neither example may contact a real provider or
|
||||
require credentials.
|
||||
|
||||
Stage 6 is complete when code, tests, exported GoDoc, format documentation,
|
||||
consumer guidance, integration contracts, and internal documentation agree;
|
||||
all canonical validation commands pass; and the working tree contains only the
|
||||
intended feature implementation and roadmap changes.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and the fixed decisions above define all behavior
|
||||
needed for implementation.
|
||||
Reference in New Issue
Block a user