promptkit/docs/roadmap/implementation.md

Public Bounded Output Repair Implementation Plan

Purpose

Implement the feature defined by the public bounded output repair roadmap. This document is the decision-complete execution sequence for a coding agent. Follow the repository architecture, testing, and documentation policies and the task-specific reading guide in the development guide throughout the work.

Target Outcome

Promptkit's public Engine.Run and Engine.RunPrepared workflows honor the existing repair budget in an effective output contract. A failed basic, json, or json_schema validation may make at most three additional calls through the same resolved target, session, credential boundary, prepared schema, and capacity-wrapped client. Each correction receives the original rendered conversation, the latest nonempty candidate, and bounded diagnostics.

The first valid candidate ends the operation. Exhaustion returns the final invalid candidate and complete final diagnostics as an ordinary result, while generation, validation, capacity, and cancellation failures retain their existing public error categories and identities. Zero remains single-pass, no public repair interface is added, and an explicitly present empty provider string reaches validation instead of being mistaken for a malformed envelope.

Fixed Decisions

• OutputContract.RepairAttempts and prompt YAML repair_attempts are the only consumer controls. The value counts additional generation calls and must be in the inclusive range zero through three.
• A positive budget with none validation is invalid. Positive budgets are eligible only after a completed failed validation under basic, json, or json_schema.
• The public engine installs one internal default repairer. There is no new public option, interface, callback, prompt override, or mutable registry.
• Every corrective call starts with a fresh copy of the complete original rendered messages. It appends only the latest nonempty candidate as an assistant message and one corrective user message. Earlier failed candidates and corrective messages do not accumulate.
• Empty and whitespace-only candidates omit the additional assistant message. The corrective user message states that the prior response was empty.
• The corrective instruction is application-neutral. basic requests a nonempty answer to the original task; json and json_schema request only corrected JSON without explanation or Markdown fences.
• Validation diagnostics are represented as bounded data in the correction message. Their encoded feedback is at most 65,536 UTF-8 bytes, preserves order, and includes an in-bound omission notice when truncated. This bound never alters the complete diagnostics retained by validation or returned in the final result.
• The latest nonempty candidate is included verbatim and in full. Promptkit adds no repair-specific candidate-size limit.
• JSON Schema repair retains the exact prepared structured-output specification. Plain JSON does not gain a provider-native JSON-object constraint because its contract permits every JSON value.
• Explicitly present string content from the built-in OpenAI-compatible client is a successful generation candidate even when empty or whitespace-only. Missing content, null, non-string content, and absent choices remain malformed provider responses and never enter repair.
• The initial generation and every correction use the same capacity-wrapped llm.Client. One run-admission lease spans the complete operation, while each model call independently acquires and releases the selected backend's active-generation permit.
• ValidationResult.RepairAttempts is execution outcome state and reports the number of corrective calls actually started. It is not initialized from the requested budget by the validation package.
• Usage is the field-by-field sum of every completed generation. An operational error returns no partial RunResult; repair exhaustion is not an operational error.
• A corrective generation failure is categorized exactly like an initial generation failure: invalid model requests match ErrInvalidRequest, other provider or transport failures match ErrLLMGenerate, typed generation details remain available, and cancellation and deadline identities remain discoverable.
• No release notes, roadmap retirement, version changes, commits, tags, or pushes are part of these stages.

Execution Rules

• Complete the stages in numerical order. Each stage is sized for one gpt-5.6-terra implementation prompt and must finish its focused verification before the next begins.
• At the start of each stage, inspect the working tree and preserve unrelated changes. The feature roadmap and this plan are intentional planning state.
• Treat intermediate stages as an unreleased partial implementation. Durable current-state documentation changes belong to Stage 6, after public engine assembly and behavior are complete.
• Keep source-neutral contract legality in internal/domain, structural validity in internal/validate, conversation construction in the default repairer, orchestration and error categorization in internal/usecase, wire envelope interpretation in internal/llm, and public assembly in the root package.
• Do not move repair behavior into the root facade, validator, capacity manager, public client adapter, or provider transport merely to simplify a test.
• Reuse the existing prepared validation plan and generation request constructor. Do not reopen prompt, profile, backend, artifact, or schema sources and do not rerender during correction.
• Update existing test owners when a contract changes. Add only representative public workflow tests; do not duplicate the internal state-machine matrix at the root package.
• Tests must be deterministic, offline, parallel-safe, race-safe, and independent of real credentials or provider services. Use injected fakes, synthetic filesystems, and httptest where provider-wire behavior matters.
• Use apply_patch for edits, format every changed Go file, review the full stage diff, and run git diff --check before moving to the next stage.

Stage 1: Enforce The Repair Contract At Its Source-Neutral Boundary

Objective

Make every supported output-contract source agree on the zero-to-three budget and invalid none pairing, and establish that actual attempt accounting belongs to execution rather than validation. This stage changes legality and result ownership but does not activate repair in the public engine.

Implementation

1. In internal/domain/output_contract.go, add one unexported maxOutputRepairAttempts = 3 constant and extend ValidateOutputContract to reject:
   • negative RepairAttempts;
   • values greater than three; and
   • a positive value when ValidationMode is ValidationNone.
2. Keep the existing format/mode/schema checks in the same helper. Do not clamp invalid budgets and do not add the maximum as a root-package public constant; exact public semantics will be documented through existing fields.
3. Preserve the current normalization and replacement paths so prompt files, request-level OutputContract replacements, Prepare, and PrepareExecution all reach the shared domain helper. Do not add parallel range checks in the root facade or prompt repository.
4. In internal/validate/standard_validator.go, stop copying the configured contract.RepairAttempts into a newly created ValidationResult. A direct validation result starts with zero attempts. Retain the existing use-case wrappers that assign the actual attemptsUsed value after each completed validation.
5. Preserve output-contract whole-value replacement behavior and all existing validation modes, schema preparation, and error wrapping apart from the new invalid cases.

Tests

1. Extend internal/domain/output_contract_test.go with a focused boundary table covering -1, 0, 1, 3, and 4, plus none with zero and with a positive value. Use representative valid basic/JSON contracts so unrelated validation errors do not mask the intended assertion.
2. Extend the existing prompt-definition repository tests with representative YAML cases proving a budget above three and a positive none pairing are rejected through the repository's existing invalid-prompt identity. Do not duplicate the full domain table there.
3. Update internal/usecase/output_contract_test.go to prove a request replacement with either new invalid condition fails before completion collaborators or admission. Cover both ordinary and prepared preparation through the existing shared test structure instead of creating a second matrix.
4. Extend the root output-contract contract test only as needed to preserve the established public error mapping for invalid request replacements. Source file failures retain their existing prompt-load category.
5. Update validator tests to assert a direct failed or passed validation reports zero attempts even when the input contract carries a positive budget. Keep execution-level attempt assertions in use-case tests.

Verification

gofmt -w internal/domain/output_contract.go internal/domain/output_contract_test.go
gofmt -w internal/validate/standard_validator.go internal/validate/standard_validator_test.go
gofmt -w internal/promptdef/repository_test.go internal/usecase/output_contract_test.go
gofmt -w output_contract_contract_test.go
go test ./internal/domain ./internal/validate ./internal/promptdef ./internal/usecase
go test ./... -run 'OutputContract|ValidationResult'
git diff --check

Stage 1 is complete when every input path accepts only a coherent zero-to-three budget, invalid contracts fail before model work, and validators no longer misreport a configured budget as completed repair work.

Status: Complete.

Stage 2: Distinguish Explicit Empty Content From A Malformed Envelope

Objective

Allow the built-in OpenAI-compatible client to return an explicitly present empty or whitespace string as a generation candidate while retaining strict malformed-response handling for absent or wrongly typed content.

Implementation

1. In internal/llm/openai_compatible_client.go, make openAIChatResponseMessage.Content presence-aware by decoding it as a *string.
2. After response framing and JSON decoding succeed, require at least one choice and a non-nil first-choice content pointer. Dereference and return the string exactly as received without trimming or rejecting an empty value.
3. Treat a missing field and explicit null as malformed because both decode to nil. Let a non-string value fail JSON decoding and remain wrapped with ErrMalformedResponse. Preserve the existing no-choice, trailing-data, response-size, usage, provider-error, and body-ownership behavior.
4. Do not perform output validation or decide repair eligibility in the model client. This stage changes only the interpretation of an otherwise successful completion envelope.

Tests

1. Add focused cases to internal/llm/openai_compatible_client_test.go proving explicit "" and whitespace-only string content return successful GenerateResponse values with the exact content and mapped usage.
2. Add or retain distinct malformed-response cases for no choices, missing content, null content, and non-string content. Each must return no response and match ErrMalformedResponse.
3. Keep these tests at the model-client owner. Do not use a root engine or duplicate validation-mode behavior in this stage.

Verification

gofmt -w internal/llm/openai_compatible_client.go internal/llm/openai_compatible_client_test.go
go test ./internal/llm -run 'OpenAICompatibleClient|Response'
go test ./internal/llm
go test ./...
git diff --check

Stage 2 is complete when content presence is distinguishable from content emptiness and all malformed successful-envelope cases retain their prior error identity.

Stage 3: Build Full-Context, Prompt-Safe Corrective Requests

Objective

Refactor the dormant default repairer so it constructs one correction from the original rendered conversation and latest validation state, supports basic and JSON modes, and safely bounds only diagnostic feedback.

Implementation

1. Extend the internal RepairRequest in internal/usecase/repairer.go with OriginalMessages []domain.RenderedMessage. Keep the type internal and do not add repair fields to internal/domain or the public package.
2. Replace the standalone system/user repair prompt with a freshly allocated message slice on every call:
   • copy every original rendered message in order without changing role, content, or cache-control metadata;
   • if strings.TrimSpace(PreviousOutput) is nonempty, append one assistant message whose content is the exact, full PreviousOutput string;
   • otherwise append no assistant message; and
   • append one corrective user message after the original conversation and optional candidate. Allocate independently from OriginalMessages so append cannot mutate a prepared snapshot's backing array.
3. Construct one stable application-neutral corrective message that includes the validation mode, Attempt and MaxAttempts, an instruction to preserve valid values and change only what is necessary, and diagnostics clearly labeled as data rather than instructions. For an empty/whitespace candidate, explicitly state that the previous response was empty and do not reproduce its whitespace.
4. Use mode-specific terminal guidance:
   • ValidationBasic: return a nonempty response satisfying the original request; and
   • ValidationJSON or ValidationJSONSchema: return only corrected JSON, with no explanation or Markdown fence. Return a defensive internal error for any other mode; the runner must never call the repairer for it after Stage 4.
5. Add an unexported maxRepairDiagnosticBytes = 64 * 1024 constant and one focused diagnostic formatter. Encode the ordered diagnostics as a JSON array within the corrective message so provider-controlled text remains visibly data. The formatter must:
   • convert malformed input strings to valid UTF-8 for prompt transport;
   • return the complete encoded array when it is within the limit;
   • when over the limit, preserve complete earlier entries, include as much of the next entry as fits at a rune boundary, omit all remaining text, and append a stable final array entry stating that additional validation diagnostics were omitted;
   • account for JSON quoting, delimiters, and the omission entry when enforcing the 65,536-byte maximum; and
   • return a valid JSON array at or below the limit for every input, including one oversized error, many errors, control characters, multibyte runes, and an empty list. This limit applies only to the encoded diagnostic block, not fixed repair instructions, the previous candidate, or ValidationResult.Errors.
6. Continue to call newGenerationRequest with the request's exact session, target, target-presence metadata, and structured-output pointer. Preserve the existing nil-client and nil-response defenses. Do not locally strip, parse, normalize, or repair the candidate.

Tests

Create internal/usecase/repairer_test.go as the narrow owner of default repairer behavior:

1. Supply original system and user messages, including cache-control metadata, and assert the generated request contains those values unchanged followed by the exact assistant candidate and one corrective user message.
2. Prove separate correction calls begin from the same original messages and do not mutate the caller's slice or accumulate an earlier candidate or corrective message.
3. Cover empty and whitespace-only candidates: no assistant message is added, the user message identifies the empty response, and no placeholder model content is invented.
4. Cover basic, JSON, and JSON Schema guidance by stable behavioral phrases and role boundaries rather than snapshotting the full incidental prose. Assert an unsupported mode returns an error without calling the client.
5. Prove the exact target, presence flags, session ID, and structured-output specification reach the model client and that an arbitrarily large nonempty candidate is not truncated.
6. Exercise diagnostics below, at, and above the limit, including ordered entries, multibyte UTF-8 near the boundary, invalid UTF-8, and one huge diagnostic. Assert the encoded block is valid JSON, valid UTF-8, no more than 65,536 bytes, ordered, and contains the omission notice only when needed. Also assert the input error slice and strings remain unchanged.
7. Retain focused nil-client, generation-error passthrough, and nil-response tests without duplicating runner error categorization.

Verification

gofmt -w internal/usecase/repairer.go internal/usecase/repairer_test.go
go test ./internal/usecase -run 'Repairer|RepairRequest'
go test ./internal/usecase
go test ./...
git diff --check

Stage 3 is complete when the default repairer produces bounded, mode-correct, full-context requests without mutating prepared state or retaining failed history.

Stage 4: Complete The Internal Repair State Machine And Error Semantics

Objective

Connect the improved repairer contract to runner execution, make basic eligible, and ensure corrective generation failures share the initial generation error path while preserving bounded state, usage, prepared validation, admission, and capacity behavior.

Implementation

1. In internal/usecase/runner.go, add one private generation-error wrapper used by both the initial llm.Generate call and every repair call:
   • if the error matches llm.ErrInvalidRequest, wrap it with ErrInvalidRequest; and
   • otherwise wrap it with ErrLLMGenerate. Use %w wrapping so provider details, cancellation, and deadlines remain discoverable. Do not classify a failed corrective model call as ErrValidation.
2. Treat a nil repair response without an error as a generation-side failure matching ErrLLMGenerate. Continue returning no partial result for that or any other operational error.
3. Pass prepared.Messages as RepairRequest.OriginalMessages on every attempt. Continue passing the latest response content and latest complete validation error slice, so each attempt repairs only the newest candidate.
4. Extend shouldAttemptRepair to include ValidationBasic. Retain all other gates: a configured repairer, positive budget, and ValidationFailed. ValidationNone, passed/skipped validation, and operational validator errors never reach the repairer.
5. Preserve the existing loop shape and attempt semantics: increment the attempt count immediately before starting a corrective call, validate every completed response with the retained operation-local plan, stop on the first valid response, and make no more calls than the frozen budget.
6. Preserve final-result coherence and cumulative usage. A successful or exhausted run uses the same final candidate for RawOutput, artifact, and validation; sums all five token-usage fields across completed generations; reports actual calls in ValidationResult.RepairAttempts; and returns the complete latest diagnostics on exhaustion.
7. Do not change admission or capacity architecture. The admission release remains deferred across the full execution, and the default repairer must continue using the same capacity-wrapped client so each correction acquires the normal active-generation permit without a second run admission.

Tests

1. Refactor TestRunnerRepairStateMachine rather than adding a parallel matrix:
   • change the obsolete “basic failure is ineligible” case into an empty/basic repair success or bounded exhaustion case;
   • keep initial success, stop-on-valid, exact exhaustion, JSON, JSON Schema, target-presence, structured-output, session, direct credential, final candidate, and cumulative-usage assertions;
   • replace the now-invalid budget-four fixture with a maximum-three fixture;
   • assert each RepairRequest receives the original rendered messages and only the latest output/errors; and
   • update repair-call detection helpers that currently recognize the old standalone system prompt so they recognize the appended assistant/user shape without depending on exact correction prose.
2. Add a zero-budget failed-validation case if the existing public test is its only owner, ensuring the internal runner remains single-pass without invoking a configured repairer. A positive none case belongs to Stage 1 contract validation, not this state machine.
3. Update prepared-execution tests to prove initial and repaired candidates use the same frozen validation plan and messages, sources are not reopened, and the handle remains one-shot.
4. Update execution-error tables so a repairer generation failure expects ErrLLMGenerate; add a repair error matching llm.ErrInvalidRequest and assert it maps to ErrInvalidRequest. Preserve underlying sentinel, cancellation, and deadline identity and confirm all operational failures return no partial result.
5. Retain and adapt the shared-backend-pool test to prove two repaired runs each receive one admission, all initial and repair calls use the same backend pool, peak active generation does not exceed the configured limit, and every lease/permit is released.
6. Add a focused cancellation case only if existing capacity coverage does not already prove cancellation while a corrective call waits for its active permit prevents that provider call and releases the run admission. Avoid duplicating internal/capacity's scheduler matrix.

Verification

gofmt -w internal/usecase/runner.go internal/usecase/runner_test.go
gofmt -w internal/usecase/prepared_execution_test.go
go test ./internal/usecase -run 'Repair|Prepared|Admission|Capacity|Cancellation'
go test -race ./internal/usecase -run 'Repair|Capacity|Cancellation'
go test ./internal/usecase
go test ./...
git diff --check

Stage 4 is complete when the internal state machine satisfies every bounded success, exhaustion, error, prepared-state, and concurrency invariant without being publicly activated yet.

Stage 5: Activate Repair Through Public Engine Assembly

Objective

Make the completed behavior available through the existing public API for both built-in and injected clients, and align exported Go contracts with the now active behavior.

Implementation

1. In engine.go, after constructing or adapting the selected llm.Client and wrapping it with capacity.NewClient, construct usecase.NewDefaultOutputRepairer(llmClient) and assemble the engine runner with usecase.NewRunnerWithRepairer. Pass that exact same wrapped client as the runner's initial-generation client and the repairer's client.
2. Do not add a public engine option or expose the internal repairer. Keep usecase.NewRunner as the no-repair convenience constructor for focused internal tests and callers; the root facade deliberately uses the explicit repairer constructor.
3. Update exported GoDoc in types.go:
   • OutputContract.RepairAttempts is an additional-call budget from zero through three and is valid only with basic, json, or json_schema when positive;
   • ValidationResult.RepairAttempts is the actual number of corrective calls started and may be nonzero in public results; and
   • LLMResponse.Content may be explicitly empty and will be interpreted by the effective output contract rather than rejected by Promptkit's public adapter.
4. Update Engine.Run, Engine.RunPrepared, and related prepared/result GoDoc in engine.go to describe opt-in bounded correction, cumulative usage, exhaustion as a returned failed validation result, operational errors as no result, cancellation, and one-shot prepared execution. Remove statements that the public engine never repairs or is always single-pass.
5. Keep the public JSON representation and public type set unchanged. Do not expose correction messages, candidate history, a semantic-validity claim, or a new error sentinel.

Tests

Use root external-package contract tests for representative assembly behavior, reusing existing public fakes where possible:

1. Replace the obsolete public “positive budget is still single-pass” assertion with a zero-budget failed-validation assertion that remains exactly one generation and reports zero attempts.
2. Add one ordinary Engine.Run workflow with an injected LLMClient that returns invalid JSON and then valid JSON. Assert two calls, the final valid candidate, one actual repair, cumulative usage, the same target/session and structured-output values, and no new configuration surface.
3. Add one PrepareExecution/RunPrepared workflow using a different eligible mode, preferably basic with an explicit empty initial candidate, to prove public prepared parity, frozen messages/settings, and one-shot behavior.
4. Add one bounded exhaustion assertion that returns a non-nil result, nil error, final invalid candidate, complete final diagnostics, exact actual attempt count, and cumulative usage. Keep the full progression matrix in internal/usecase.
5. Use an offline httptest OpenAI-compatible endpoint for one built-in-client integration path: return a structurally valid completion envelope whose content fails validation, then a provider error on correction. Assert no partial result, ErrLLMGenerate, available public GenerationError details, and preserved underlying cancellation/deadline behavior where applicable. Do not duplicate every provider failure shape already owned by internal/llm.
6. Ensure existing capacity tests for injected clients still pass with the repairer installed, especially engine-local backend limits and unlimited backend behavior when repair is not requested.

Verification

gofmt -w engine.go types.go output_contract_contract_test.go public_contract_test.go
gofmt -w prepared_execution_contract_test.go engine_test.go
go test ./... -run 'Repair|OutputContract|RunPrepared|GenerationError|Capacity'
go test -race ./... -run 'Repair|RunPrepared|Capacity'
go test ./...
git diff --check

If a named root test file does not require a change, omit it from gofmt rather than touching it mechanically. Stage 5 is complete when ordinary and prepared public workflows repair through either client path with unchanged API shape and correct public result and error semantics.

Stage 6: Publish Canonical Documentation And Run Maintainer Validation

Objective

Make every canonical documentation owner describe the implemented feature, remove obsolete dormant/single-pass claims, and validate the complete cross-cutting change using the repository's canonical workflow.

Implementation

1. Update docs/formats.md, the exact prompt-format owner, to document:
   • repair_attempts as an integer from zero through three;
   • zero as the single-pass default and positive values as additional calls;
   • eligible basic, json, and json_schema modes;
   • invalid positive budgets with none;
   • stop-on-valid and exhausted-final-result behavior; and
   • plain JSON's acceptance of every JSON value without an object-only wire constraint.
2. Update docs/consumers/pkg-promptkit.md with one concise JSON Schema repair example using the existing public field or prompt YAML. Explain cumulative usage, actual attempt reporting, the possibility of a final failed validation after exhaustion, and the cost/latency implications of extra calls. Note briefly that basic can repair empty output and that structural validity is not factual or domain correctness.
3. Update docs/integrations/openai-compatible-chat.md to own the provider-wire distinction: an explicitly present string content, including empty or whitespace-only, is a candidate; absent choices, missing content, null, and non-string content are malformed. Retain provider-native JSON Schema as the first defense and do not claim an object-only constraint for plain JSON.
4. Update docs/internal/llm.md with the same internal response-decoding boundary without duplicating public repair orchestration.
5. Update docs/internal/runner.md to describe the installed default repairer, full-original-message/latest-candidate request shape, empty-candidate role behavior, diagnostic bound, eligible modes, attempt/usage accounting, exhaustion, generation-versus-validation error categories, prepared-plan reuse, and one-admission/shared-capacity behavior. Remove the claim that the ordinary public runner omits a repairer.
6. Update docs/internal/overview.md so root assembly and internal/usecase ownership reflect public bounded repair. Review docs/internal/capacity.md and docs/internal/sources.md; edit only statements that are stale or incomplete after implementation, preserving their narrow ownership.
7. Do not put canonical behavior in a release note or roadmap. Do not update the architecture policy unless implementation actually crossed the package boundaries fixed by this plan. Do not retire output-repair.md or this plan in this implementation sequence.
8. Check every changed Markdown link and heading fragment with the canonical offline link checker in the development guide. Review the final diff for consistent terminology: “repair attempt” always means an additional model call, “exhaustion” returns a completed failed validation result, and no text promises semantic correctness or provider caching.

Tests And Final Verification

Run the complete maintainer validation workflow from the repository root, including both maintained offline examples:

go test ./...
go test -race ./...
go vet ./...
go build ./...
go run ./examples/go-library/prepare
go run ./examples/go-library/run
git diff --check
git status --short

Also run the development guide's tracked-Go formatting check and local Markdown-link checker exactly as documented. Review example JSON output for the invariants listed there; neither example may contact a real provider or require credentials.

Stage 6 is complete when code, tests, exported GoDoc, format documentation, consumer guidance, integration contracts, and internal documentation agree; all canonical validation commands pass; and the working tree contains only the intended feature implementation and roadmap changes.

Open Questions

None. The feature roadmap and the fixed decisions above define all behavior needed for implementation.