From 2e76003fd55ad14c3f60be2deb759faec29c6aa3 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 25 Aug 2026 12:19:17 +0000 Subject: [PATCH] Prepare documentation for the v0.8.0 release --- README.md | 7 +- docs/releases/v0.8.0.md | 109 ++++++ docs/roadmap/implementation.md | 603 --------------------------------- docs/roadmap/output-repair.md | 361 -------------------- 4 files changed, 114 insertions(+), 966 deletions(-) create mode 100644 docs/releases/v0.8.0.md delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/output-repair.md diff --git a/README.md b/README.md index 84ee51c..1a7164e 100644 --- a/README.md +++ b/README.md @@ -33,10 +33,13 @@ boundary and constraints that framework work must preserve. ## Release Guidance -Consumers upgrading from `v0.6.0` to `v0.7.0` should read the -[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md). +Consumers upgrading from `v0.7.0` to `v0.8.0` should read the +[v0.8.0 changelog and migration guide](docs/releases/v0.8.0.md). Earlier adopters can consult the +[v0.7.0 changelog and migration guide](docs/releases/v0.7.0.md). + +Consumers upgrading from `v0.5.0` to `v0.6.0` can consult the [v0.6.0 changelog and migration guide](docs/releases/v0.6.0.md). Consumers upgrading from `v0.4.0` to `v0.5.0` can consult the diff --git a/docs/releases/v0.8.0.md b/docs/releases/v0.8.0.md new file mode 100644 index 0000000..c01d804 --- /dev/null +++ b/docs/releases/v0.8.0.md @@ -0,0 +1,109 @@ +# Promptkit v0.8.0 + +This supplemental changelog and migration guide summarizes the consumer-facing +changes from `v0.7.0` to `v0.8.0`. The annotated `v0.8.0` tag is the +authoritative release record. Exact current contracts belong to the linked +GoDoc and durable documentation. + +## Summary + +`v0.8.0` activates Promptkit's bounded output-repair workflow: + +- failed nonempty-text, JSON, and JSON Schema validation can make a limited + number of corrective model calls; +- corrective calls preserve the original rendered conversation, effective + target, session, structured-output contract, and backend capacity policy; +- results report cumulative usage and the number of corrective calls actually + made; and +- explicitly empty OpenAI-compatible response content now reaches output + validation instead of being classified as a malformed provider envelope. + +## Compatibility + +This release adds no public declarations or fields and removes none. Existing +source code remains source-compatible. + +The behavior of the existing `OutputContract.RepairAttempts` field and prompt +YAML `repair_attempts` field has changed. A positive value now authorizes real +additional model calls after eligible validation failures; earlier releases +accepted the field but the public engine remained single-pass. Consumers that +set a positive value should expect additional latency, token usage, and +provider cost when repair is needed. + +Repair budgets must now be between zero and three. A positive budget requires +`basic`, `json`, or `json_schema` validation. Values above three and a positive +budget paired with `none` are invalid contracts rather than ignored settings. + +An explicitly present empty or whitespace-only string returned by the built-in +OpenAI-compatible client is now a completed generation candidate. `none` +validation permits it, while `basic`, `json`, and `json_schema` classify it +under their ordinary validation rules and may repair it when configured. +Missing, `null`, or non-string content remains a malformed provider response. + +## Upgrade + +Update the module dependency with: + +```sh +go get gitea.maximumdirect.net/eric/promptkit@v0.8.0 +go mod tidy +``` + +Review every prompt definition and request override that sets a positive repair +budget. Use zero or omit the field to retain single-pass execution. Ensure each +positive budget is no greater than three and uses an eligible validation mode, +then run the consuming project's ordinary and race-enabled tests. + +## Bounded Output Repair + +`repair_attempts` counts corrective calls in addition to the initial model +call. Promptkit validates each completed candidate, stops at the first valid +one, and never exceeds the configured bound. If every candidate remains +invalid, the run completes successfully with the final candidate and its +failed validation result rather than returning an operational error. + +Each correction starts from the original rendered messages and includes only +the latest invalid candidate and latest validation diagnostics. JSON Schema +mode retains the provider-native structured-output request as its first line of +defense. Promptkit performs only deterministic structural validation; a valid +response is not necessarily factual or correct for an application's domain. + +Usage in the final result is cumulative across the initial response and every +completed corrective response. `ValidationResult.RepairAttempts` reports the +number of corrective calls actually made. Corrective generation failures use +the same public generation-error categories and structured provider details as +an initial generation failure. + +See the [output-contract format reference](../formats.md#output-contract), the +[consumer repair example](../consumers/pkg-promptkit.md#repair-a-structured-result), +and the [`OutputContract` and `ValidationResult` GoDoc](../../types.go) for the +current contracts. + +## Explicit Empty Content + +The built-in OpenAI-compatible client now distinguishes an explicitly present +empty string from a missing or malformed `content` field. This aligns built-in +and injected clients by letting the selected output contract decide whether an +empty candidate is acceptable, invalid, or eligible for repair. + +See the +[OpenAI-compatible response contract](../integrations/openai-compatible-chat.md#response-handling) +for the exact envelope behavior. + +## Public API Changes + +None. This release activates and tightens the documented behavior of existing +fields. + +## Consumer Action + +- Remove or set `repair_attempts` to zero where execution must remain + single-pass. +- Keep every positive repair budget at three or fewer and pair it with + `basic`, `json`, or `json_schema` validation. +- Account for additional latency, usage, and provider cost when enabling + repair. +- Continue checking the returned validation status because bounded repair can + exhaust without producing a valid candidate. +- Review workflows that previously treated explicit empty provider content as + a generation error. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 9ff948d..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,603 +0,0 @@ -# Public Bounded Output Repair Implementation Plan - -## Purpose - -Implement the feature defined by the -[public bounded output repair roadmap](output-repair.md). This document is the -decision-complete execution sequence for a coding agent. Follow the repository -[architecture](../policy/architecture.md), [testing](../policy/testing.md), and -[documentation](../policy/documentation.md) policies and the task-specific -reading guide in the [development guide](../development.md) throughout the -work. - -## Target Outcome - -Promptkit's public `Engine.Run` and `Engine.RunPrepared` workflows honor the -existing repair budget in an effective output contract. A failed `basic`, -`json`, or `json_schema` validation may make at most three additional calls -through the same resolved target, session, credential boundary, prepared -schema, and capacity-wrapped client. Each correction receives the original -rendered conversation, the latest nonempty candidate, and bounded diagnostics. - -The first valid candidate ends the operation. Exhaustion returns the final -invalid candidate and complete final diagnostics as an ordinary result, while -generation, validation, capacity, and cancellation failures retain their -existing public error categories and identities. Zero remains single-pass, no -public repair interface is added, and an explicitly present empty provider -string reaches validation instead of being mistaken for a malformed envelope. - -## Fixed Decisions - -- `OutputContract.RepairAttempts` and prompt YAML `repair_attempts` are the - only consumer controls. The value counts additional generation calls and - must be in the inclusive range zero through three. -- A positive budget with `none` validation is invalid. Positive budgets are - eligible only after a completed failed validation under `basic`, `json`, or - `json_schema`. -- The public engine installs one internal default repairer. There is no new - public option, interface, callback, prompt override, or mutable registry. -- Every corrective call starts with a fresh copy of the complete original - rendered messages. It appends only the latest nonempty candidate as an - `assistant` message and one corrective `user` message. Earlier failed - candidates and corrective messages do not accumulate. -- Empty and whitespace-only candidates omit the additional assistant message. - The corrective user message states that the prior response was empty. -- The corrective instruction is application-neutral. `basic` requests a - nonempty answer to the original task; `json` and `json_schema` request only - corrected JSON without explanation or Markdown fences. -- Validation diagnostics are represented as bounded data in the correction - message. Their encoded feedback is at most 65,536 UTF-8 bytes, preserves - order, and includes an in-bound omission notice when truncated. This bound - never alters the complete diagnostics retained by validation or returned in - the final result. -- The latest nonempty candidate is included verbatim and in full. Promptkit - adds no repair-specific candidate-size limit. -- JSON Schema repair retains the exact prepared structured-output - specification. Plain JSON does not gain a provider-native JSON-object - constraint because its contract permits every JSON value. -- Explicitly present string content from the built-in OpenAI-compatible client - is a successful generation candidate even when empty or whitespace-only. - Missing content, `null`, non-string content, and absent choices remain - malformed provider responses and never enter repair. -- The initial generation and every correction use the same capacity-wrapped - `llm.Client`. One run-admission lease spans the complete operation, while - each model call independently acquires and releases the selected backend's - active-generation permit. -- `ValidationResult.RepairAttempts` is execution outcome state and reports the - number of corrective calls actually started. It is not initialized from the - requested budget by the validation package. -- Usage is the field-by-field sum of every completed generation. An - operational error returns no partial `RunResult`; repair exhaustion is not - an operational error. -- A corrective generation failure is categorized exactly like an initial - generation failure: invalid model requests match `ErrInvalidRequest`, other - provider or transport failures match `ErrLLMGenerate`, typed generation - details remain available, and cancellation and deadline identities remain - discoverable. -- No release notes, roadmap retirement, version changes, commits, tags, or - pushes are part of these stages. - -## Execution Rules - -- Complete the stages in numerical order. Each stage is sized for one - gpt-5.6-terra implementation prompt and must finish its focused verification - before the next begins. -- At the start of each stage, inspect the working tree and preserve unrelated - changes. The feature roadmap and this plan are intentional planning state. -- Treat intermediate stages as an unreleased partial implementation. Durable - current-state documentation changes belong to Stage 6, after public engine - assembly and behavior are complete. -- Keep source-neutral contract legality in `internal/domain`, structural - validity in `internal/validate`, conversation construction in the default - repairer, orchestration and error categorization in `internal/usecase`, wire - envelope interpretation in `internal/llm`, and public assembly in the root - package. -- Do not move repair behavior into the root facade, validator, capacity - manager, public client adapter, or provider transport merely to simplify a - test. -- Reuse the existing prepared validation plan and generation request - constructor. Do not reopen prompt, profile, backend, artifact, or schema - sources and do not rerender during correction. -- Update existing test owners when a contract changes. Add only representative - public workflow tests; do not duplicate the internal state-machine matrix at - the root package. -- Tests must be deterministic, offline, parallel-safe, race-safe, and - independent of real credentials or provider services. Use injected fakes, - synthetic filesystems, and `httptest` where provider-wire behavior matters. -- Use `apply_patch` for edits, format every changed Go file, review the full - stage diff, and run `git diff --check` before moving to the next stage. - -## Stage 1: Enforce The Repair Contract At Its Source-Neutral Boundary - -### Objective - -Make every supported output-contract source agree on the zero-to-three budget -and invalid `none` pairing, and establish that actual attempt accounting belongs -to execution rather than validation. This stage changes legality and result -ownership but does not activate repair in the public engine. - -### Implementation - -1. In `internal/domain/output_contract.go`, add one unexported - `maxOutputRepairAttempts = 3` constant and extend - `ValidateOutputContract` to reject: - - negative `RepairAttempts`; - - values greater than three; and - - a positive value when `ValidationMode` is `ValidationNone`. -2. Keep the existing format/mode/schema checks in the same helper. Do not clamp - invalid budgets and do not add the maximum as a root-package public - constant; exact public semantics will be documented through existing fields. -3. Preserve the current normalization and replacement paths so prompt files, - request-level `OutputContract` replacements, `Prepare`, and - `PrepareExecution` all reach the shared domain helper. Do not add parallel - range checks in the root facade or prompt repository. -4. In `internal/validate/standard_validator.go`, stop copying the configured - `contract.RepairAttempts` into a newly created `ValidationResult`. A direct - validation result starts with zero attempts. Retain the existing use-case - wrappers that assign the actual `attemptsUsed` value after each completed - validation. -5. Preserve output-contract whole-value replacement behavior and all existing - validation modes, schema preparation, and error wrapping apart from the new - invalid cases. - -### Tests - -1. Extend `internal/domain/output_contract_test.go` with a focused boundary - table covering `-1`, `0`, `1`, `3`, and `4`, plus `none` with zero and with - a positive value. Use representative valid basic/JSON contracts so unrelated - validation errors do not mask the intended assertion. -2. Extend the existing prompt-definition repository tests with representative - YAML cases proving a budget above three and a positive `none` pairing are - rejected through the repository's existing invalid-prompt identity. Do not - duplicate the full domain table there. -3. Update `internal/usecase/output_contract_test.go` to prove a request - replacement with either new invalid condition fails before completion - collaborators or admission. Cover both ordinary and prepared preparation - through the existing shared test structure instead of creating a second - matrix. -4. Extend the root output-contract contract test only as needed to preserve the - established public error mapping for invalid request replacements. Source - file failures retain their existing prompt-load category. -5. Update validator tests to assert a direct failed or passed validation reports - zero attempts even when the input contract carries a positive budget. Keep - execution-level attempt assertions in use-case tests. - -### Verification - -```sh -gofmt -w internal/domain/output_contract.go internal/domain/output_contract_test.go -gofmt -w internal/validate/standard_validator.go internal/validate/standard_validator_test.go -gofmt -w internal/promptdef/repository_test.go internal/usecase/output_contract_test.go -gofmt -w output_contract_contract_test.go -go test ./internal/domain ./internal/validate ./internal/promptdef ./internal/usecase -go test ./... -run 'OutputContract|ValidationResult' -git diff --check -``` - -Stage 1 is complete when every input path accepts only a coherent zero-to-three -budget, invalid contracts fail before model work, and validators no longer -misreport a configured budget as completed repair work. - -**Status:** Complete. - -## Stage 2: Distinguish Explicit Empty Content From A Malformed Envelope - -### Objective - -Allow the built-in OpenAI-compatible client to return an explicitly present -empty or whitespace string as a generation candidate while retaining strict -malformed-response handling for absent or wrongly typed content. - -### Implementation - -1. In `internal/llm/openai_compatible_client.go`, make - `openAIChatResponseMessage.Content` presence-aware by decoding it as a - `*string`. -2. After response framing and JSON decoding succeed, require at least one - choice and a non-nil first-choice content pointer. Dereference and return - the string exactly as received without trimming or rejecting an empty - value. -3. Treat a missing field and explicit `null` as malformed because both decode - to nil. Let a non-string value fail JSON decoding and remain wrapped with - `ErrMalformedResponse`. Preserve the existing no-choice, trailing-data, - response-size, usage, provider-error, and body-ownership behavior. -4. Do not perform output validation or decide repair eligibility in the model - client. This stage changes only the interpretation of an otherwise - successful completion envelope. - -### Tests - -1. Add focused cases to `internal/llm/openai_compatible_client_test.go` proving - explicit `""` and whitespace-only string content return successful - `GenerateResponse` values with the exact content and mapped usage. -2. Add or retain distinct malformed-response cases for no choices, missing - content, `null` content, and non-string content. Each must return no response - and match `ErrMalformedResponse`. -3. Keep these tests at the model-client owner. Do not use a root engine or - duplicate validation-mode behavior in this stage. - -### Verification - -```sh -gofmt -w internal/llm/openai_compatible_client.go internal/llm/openai_compatible_client_test.go -go test ./internal/llm -run 'OpenAICompatibleClient|Response' -go test ./internal/llm -go test ./... -git diff --check -``` - -Stage 2 is complete when content presence is distinguishable from content -emptiness and all malformed successful-envelope cases retain their prior error -identity. - -**Status:** Complete. - -## Stage 3: Build Full-Context, Prompt-Safe Corrective Requests - -### Objective - -Refactor the dormant default repairer so it constructs one correction from the -original rendered conversation and latest validation state, supports basic and -JSON modes, and safely bounds only diagnostic feedback. - -### Implementation - -1. Extend the internal `RepairRequest` in `internal/usecase/repairer.go` with - `OriginalMessages []domain.RenderedMessage`. Keep the type internal and do - not add repair fields to `internal/domain` or the public package. -2. Replace the standalone system/user repair prompt with a freshly allocated - message slice on every call: - - copy every original rendered message in order without changing role, - content, or cache-control metadata; - - if `strings.TrimSpace(PreviousOutput)` is nonempty, append one `assistant` - message whose content is the exact, full `PreviousOutput` string; - - otherwise append no assistant message; and - - append one corrective `user` message after the original conversation and - optional candidate. - Allocate independently from `OriginalMessages` so `append` cannot mutate a - prepared snapshot's backing array. -3. Construct one stable application-neutral corrective message that includes - the validation mode, `Attempt` and `MaxAttempts`, an instruction to preserve - valid values and change only what is necessary, and diagnostics clearly - labeled as data rather than instructions. For an empty/whitespace candidate, - explicitly state that the previous response was empty and do not reproduce - its whitespace. -4. Use mode-specific terminal guidance: - - `ValidationBasic`: return a nonempty response satisfying the original - request; and - - `ValidationJSON` or `ValidationJSONSchema`: return only corrected JSON, - with no explanation or Markdown fence. - Return a defensive internal error for any other mode; the runner must never - call the repairer for it after Stage 4. -5. Add an unexported `maxRepairDiagnosticBytes = 64 * 1024` constant and one - focused diagnostic formatter. Encode the ordered diagnostics as a JSON array - within the corrective message so provider-controlled text remains visibly - data. The formatter must: - - convert malformed input strings to valid UTF-8 for prompt transport; - - return the complete encoded array when it is within the limit; - - when over the limit, preserve complete earlier entries, include as much - of the next entry as fits at a rune boundary, omit all remaining text, and - append a stable final array entry stating that additional validation - diagnostics were omitted; - - account for JSON quoting, delimiters, and the omission entry when enforcing - the 65,536-byte maximum; and - - return a valid JSON array at or below the limit for every input, including - one oversized error, many errors, control characters, multibyte runes, and - an empty list. - This limit applies only to the encoded diagnostic block, not fixed repair - instructions, the previous candidate, or `ValidationResult.Errors`. -6. Continue to call `newGenerationRequest` with the request's exact session, - target, target-presence metadata, and structured-output pointer. Preserve - the existing nil-client and nil-response defenses. Do not locally strip, - parse, normalize, or repair the candidate. - -### Tests - -Create `internal/usecase/repairer_test.go` as the narrow owner of default -repairer behavior: - -1. Supply original system and user messages, including cache-control metadata, - and assert the generated request contains those values unchanged followed - by the exact assistant candidate and one corrective user message. -2. Prove separate correction calls begin from the same original messages and - do not mutate the caller's slice or accumulate an earlier candidate or - corrective message. -3. Cover empty and whitespace-only candidates: no assistant message is added, - the user message identifies the empty response, and no placeholder model - content is invented. -4. Cover basic, JSON, and JSON Schema guidance by stable behavioral phrases and - role boundaries rather than snapshotting the full incidental prose. Assert - an unsupported mode returns an error without calling the client. -5. Prove the exact target, presence flags, session ID, and structured-output - specification reach the model client and that an arbitrarily large nonempty - candidate is not truncated. -6. Exercise diagnostics below, at, and above the limit, including ordered - entries, multibyte UTF-8 near the boundary, invalid UTF-8, and one huge - diagnostic. Assert the encoded block is valid JSON, valid UTF-8, no more - than 65,536 bytes, ordered, and contains the omission notice only when - needed. Also assert the input error slice and strings remain unchanged. -7. Retain focused nil-client, generation-error passthrough, and nil-response - tests without duplicating runner error categorization. - -### Verification - -```sh -gofmt -w internal/usecase/repairer.go internal/usecase/repairer_test.go -go test ./internal/usecase -run 'Repairer|RepairRequest' -go test ./internal/usecase -go test ./... -git diff --check -``` - -Stage 3 is complete when the default repairer produces bounded, mode-correct, -full-context requests without mutating prepared state or retaining failed -history. - -**Status:** Complete. - -## Stage 4: Complete The Internal Repair State Machine And Error Semantics - -### Objective - -Connect the improved repairer contract to runner execution, make `basic` -eligible, and ensure corrective generation failures share the initial -generation error path while preserving bounded state, usage, prepared -validation, admission, and capacity behavior. - -### Implementation - -1. In `internal/usecase/runner.go`, add one private generation-error wrapper - used by both the initial `llm.Generate` call and every repair call: - - if the error matches `llm.ErrInvalidRequest`, wrap it with - `ErrInvalidRequest`; and - - otherwise wrap it with `ErrLLMGenerate`. - Use `%w` wrapping so provider details, cancellation, and deadlines remain - discoverable. Do not classify a failed corrective model call as - `ErrValidation`. -2. Treat a nil repair response without an error as a generation-side failure - matching `ErrLLMGenerate`. Continue returning no partial result for that or - any other operational error. -3. Pass `prepared.Messages` as `RepairRequest.OriginalMessages` on every - attempt. Continue passing the latest response content and latest complete - validation error slice, so each attempt repairs only the newest candidate. -4. Extend `shouldAttemptRepair` to include `ValidationBasic`. Retain all other - gates: a configured repairer, positive budget, and - `ValidationFailed`. `ValidationNone`, passed/skipped validation, and - operational validator errors never reach the repairer. -5. Preserve the existing loop shape and attempt semantics: increment the - attempt count immediately before starting a corrective call, validate every - completed response with the retained operation-local plan, stop on the first - valid response, and make no more calls than the frozen budget. -6. Preserve final-result coherence and cumulative usage. A successful or - exhausted run uses the same final candidate for `RawOutput`, artifact, and - validation; sums all five token-usage fields across completed generations; - reports actual calls in `ValidationResult.RepairAttempts`; and returns the - complete latest diagnostics on exhaustion. -7. Do not change admission or capacity architecture. The admission release - remains deferred across the full execution, and the default repairer must - continue using the same capacity-wrapped client so each correction acquires - the normal active-generation permit without a second run admission. - -### Tests - -1. Refactor `TestRunnerRepairStateMachine` rather than adding a parallel - matrix: - - change the obsolete “basic failure is ineligible” case into an empty/basic - repair success or bounded exhaustion case; - - keep initial success, stop-on-valid, exact exhaustion, JSON, JSON Schema, - target-presence, structured-output, session, direct credential, final - candidate, and cumulative-usage assertions; - - replace the now-invalid budget-four fixture with a maximum-three fixture; - - assert each `RepairRequest` receives the original rendered messages and - only the latest output/errors; and - - update repair-call detection helpers that currently recognize the old - standalone system prompt so they recognize the appended assistant/user - shape without depending on exact correction prose. -2. Add a zero-budget failed-validation case if the existing public test is its - only owner, ensuring the internal runner remains single-pass without invoking - a configured repairer. A positive `none` case belongs to Stage 1 contract - validation, not this state machine. -3. Update prepared-execution tests to prove initial and repaired candidates use - the same frozen validation plan and messages, sources are not reopened, and - the handle remains one-shot. -4. Update execution-error tables so a repairer generation failure expects - `ErrLLMGenerate`; add a repair error matching `llm.ErrInvalidRequest` and - assert it maps to `ErrInvalidRequest`. Preserve underlying sentinel, - cancellation, and deadline identity and confirm all operational failures - return no partial result. -5. Retain and adapt the shared-backend-pool test to prove two repaired runs - each receive one admission, all initial and repair calls use the same - backend pool, peak active generation does not exceed the configured limit, - and every lease/permit is released. -6. Add a focused cancellation case only if existing capacity coverage does not - already prove cancellation while a corrective call waits for its active - permit prevents that provider call and releases the run admission. Avoid - duplicating `internal/capacity`'s scheduler matrix. - -### Verification - -```sh -gofmt -w internal/usecase/runner.go internal/usecase/runner_test.go -gofmt -w internal/usecase/prepared_execution_test.go -go test ./internal/usecase -run 'Repair|Prepared|Admission|Capacity|Cancellation' -go test -race ./internal/usecase -run 'Repair|Capacity|Cancellation' -go test ./internal/usecase -go test ./... -git diff --check -``` - -Stage 4 is complete when the internal state machine satisfies every bounded -success, exhaustion, error, prepared-state, and concurrency invariant without -being publicly activated yet. - -**Status:** Complete. - -## Stage 5: Activate Repair Through Public Engine Assembly - -### Objective - -Make the completed behavior available through the existing public API for both -built-in and injected clients, and align exported Go contracts with the now -active behavior. - -### Implementation - -1. In `engine.go`, after constructing or adapting the selected `llm.Client` and - wrapping it with `capacity.NewClient`, construct - `usecase.NewDefaultOutputRepairer(llmClient)` and assemble the engine runner - with `usecase.NewRunnerWithRepairer`. Pass that exact same wrapped client as - the runner's initial-generation client and the repairer's client. -2. Do not add a public engine option or expose the internal repairer. Keep - `usecase.NewRunner` as the no-repair convenience constructor for focused - internal tests and callers; the root facade deliberately uses the explicit - repairer constructor. -3. Update exported GoDoc in `types.go`: - - `OutputContract.RepairAttempts` is an additional-call budget from zero - through three and is valid only with `basic`, `json`, or `json_schema` when - positive; - - `ValidationResult.RepairAttempts` is the actual number of corrective calls - started and may be nonzero in public results; and - - `LLMResponse.Content` may be explicitly empty and will be interpreted by - the effective output contract rather than rejected by Promptkit's public - adapter. -4. Update `Engine.Run`, `Engine.RunPrepared`, and related prepared/result GoDoc - in `engine.go` to describe opt-in bounded correction, cumulative usage, - exhaustion as a returned failed validation result, operational errors as no - result, cancellation, and one-shot prepared execution. Remove statements - that the public engine never repairs or is always single-pass. -5. Keep the public JSON representation and public type set unchanged. Do not - expose correction messages, candidate history, a semantic-validity claim, or - a new error sentinel. - -### Tests - -Use root external-package contract tests for representative assembly behavior, -reusing existing public fakes where possible: - -1. Replace the obsolete public “positive budget is still single-pass” assertion - with a zero-budget failed-validation assertion that remains exactly one - generation and reports zero attempts. -2. Add one ordinary `Engine.Run` workflow with an injected `LLMClient` that - returns invalid JSON and then valid JSON. Assert two calls, the final valid - candidate, one actual repair, cumulative usage, the same target/session and - structured-output values, and no new configuration surface. -3. Add one `PrepareExecution`/`RunPrepared` workflow using a different eligible - mode, preferably `basic` with an explicit empty initial candidate, to prove - public prepared parity, frozen messages/settings, and one-shot behavior. -4. Add one bounded exhaustion assertion that returns a non-nil result, nil - error, final invalid candidate, complete final diagnostics, exact actual - attempt count, and cumulative usage. Keep the full progression matrix in - `internal/usecase`. -5. Use an offline `httptest` OpenAI-compatible endpoint for one built-in-client - integration path: return a structurally valid completion envelope whose - content fails validation, then a provider error on correction. Assert no - partial result, `ErrLLMGenerate`, available public `GenerationError` details, - and preserved underlying cancellation/deadline behavior where applicable. - Do not duplicate every provider failure shape already owned by - `internal/llm`. -6. Ensure existing capacity tests for injected clients still pass with the - repairer installed, especially engine-local backend limits and unlimited - backend behavior when repair is not requested. - -### Verification - -```sh -gofmt -w engine.go types.go output_contract_contract_test.go public_contract_test.go -gofmt -w prepared_execution_contract_test.go engine_test.go -go test ./... -run 'Repair|OutputContract|RunPrepared|GenerationError|Capacity' -go test -race ./... -run 'Repair|RunPrepared|Capacity' -go test ./... -git diff --check -``` - -If a named root test file does not require a change, omit it from `gofmt` -rather than touching it mechanically. Stage 5 is complete when ordinary and -prepared public workflows repair through either client path with unchanged API -shape and correct public result and error semantics. - -**Status:** Complete. - -## Stage 6: Publish Canonical Documentation And Run Maintainer Validation - -### Objective - -Make every canonical documentation owner describe the implemented feature, -remove obsolete dormant/single-pass claims, and validate the complete -cross-cutting change using the repository's canonical workflow. - -### Implementation - -1. Update `docs/formats.md`, the exact prompt-format owner, to document: - - `repair_attempts` as an integer from zero through three; - - zero as the single-pass default and positive values as additional calls; - - eligible `basic`, `json`, and `json_schema` modes; - - invalid positive budgets with `none`; - - stop-on-valid and exhausted-final-result behavior; and - - plain JSON's acceptance of every JSON value without an object-only wire - constraint. -2. Update `docs/consumers/pkg-promptkit.md` with one concise JSON Schema repair - example using the existing public field or prompt YAML. Explain cumulative - usage, actual attempt reporting, the possibility of a final failed - validation after exhaustion, and the cost/latency implications of extra - calls. Note briefly that `basic` can repair empty output and that structural - validity is not factual or domain correctness. -3. Update `docs/integrations/openai-compatible-chat.md` to own the provider-wire - distinction: an explicitly present string `content`, including empty or - whitespace-only, is a candidate; absent choices, missing content, `null`, - and non-string content are malformed. Retain provider-native JSON Schema as - the first defense and do not claim an object-only constraint for plain JSON. -4. Update `docs/internal/llm.md` with the same internal response-decoding - boundary without duplicating public repair orchestration. -5. Update `docs/internal/runner.md` to describe the installed default repairer, - full-original-message/latest-candidate request shape, empty-candidate role - behavior, diagnostic bound, eligible modes, attempt/usage accounting, - exhaustion, generation-versus-validation error categories, prepared-plan - reuse, and one-admission/shared-capacity behavior. Remove the claim that the - ordinary public runner omits a repairer. -6. Update `docs/internal/overview.md` so root assembly and `internal/usecase` - ownership reflect public bounded repair. Review `docs/internal/capacity.md` - and `docs/internal/sources.md`; edit only statements that are stale or - incomplete after implementation, preserving their narrow ownership. -7. Do not put canonical behavior in a release note or roadmap. Do not update - the architecture policy unless implementation actually crossed the package - boundaries fixed by this plan. Do not retire `output-repair.md` or this plan - in this implementation sequence. -8. Check every changed Markdown link and heading fragment with the canonical - offline link checker in the development guide. Review the final diff for - consistent terminology: “repair attempt” always means an additional model - call, “exhaustion” returns a completed failed validation result, and no text - promises semantic correctness or provider caching. - -### Tests And Final Verification - -Run the complete -[maintainer validation workflow](../development.md#maintainer-validation) from -the repository root, including both maintained offline examples: - -```sh -go test ./... -go test -race ./... -go vet ./... -go build ./... -go run ./examples/go-library/prepare -go run ./examples/go-library/run -git diff --check -git status --short -``` - -Also run the development guide's tracked-Go formatting check and local -Markdown-link checker exactly as documented. Review example JSON output for -the invariants listed there; neither example may contact a real provider or -require credentials. - -Stage 6 is complete when code, tests, exported GoDoc, format documentation, -consumer guidance, integration contracts, and internal documentation agree; -all canonical validation commands pass; and the working tree contains only the -intended feature implementation and roadmap changes. - -**Status:** Complete. - -## Open Questions - -None. The feature roadmap and the fixed decisions above define all behavior -needed for implementation. diff --git a/docs/roadmap/output-repair.md b/docs/roadmap/output-repair.md deleted file mode 100644 index b113838..0000000 --- a/docs/roadmap/output-repair.md +++ /dev/null @@ -1,361 +0,0 @@ -# Public Bounded Output Repair Roadmap - -## Purpose - -Make Promptkit's existing bounded output-repair capability available through -the public engine. Consumers that require nonempty text, JSON, or JSON Schema -output should be able to ask Promptkit to make a limited number of corrective -model calls after content validation fails, without rebuilding Promptkit's -generation, validation, capacity, credential, session, and result-accounting -workflow. - -The feature improves basic content and structural reliability, not domain -correctness. Promptkit can determine whether output is nonempty, whether it is -syntactically valid JSON, and whether it satisfies a supplied JSON Schema. It -cannot determine whether otherwise valid content is factual, complete, useful, -or semantically correct for a downstream application. - -## Target End State - -- The existing `repair_attempts` prompt field and - `OutputContract.RepairAttempts` request field control public repair behavior. - Zero remains the default and makes execution single-pass. -- A repair budget may be one, two, or three corrective calls. Values above - three are invalid rather than silently clamped, providing a framework-level - bound on accidental cost and latency. -- A positive repair budget is eligible when the effective validation mode is - `basic`, `json`, or `json_schema` and the latest completed content validation - has failed. `none` explicitly permits empty content, and pairing it with a - positive repair budget is an invalid output contract. -- `Engine.Run` and `Engine.RunPrepared` provide the same repair behavior for - the built-in OpenAI-compatible client and an injected `LLMClient`. -- Provider-native structured-output metadata remains the first line of - defense for JSON Schema generation. Repair applies only after the resulting - content still fails Promptkit's own validation. -- Promptkit stops at the first contract-valid response or after the - requested bound is exhausted. It never reports an invalid response as valid. -- A completed exhausted run returns the final invalid candidate and its final - validation diagnostics in the ordinary `RunResult`; exhaustion is not an - operational error. -- Results report the number of corrective calls actually attempted and the - cumulative token usage reported by the initial generation and every - completed repair generation. -- Repair remains internal orchestration. Consumers do not need to construct or - register a repairer, and this feature does not add a public repair strategy - interface. - -## Eligibility And Contract Semantics - -Repair is opt-in for each effective output contract. The prompt definition may -declare a budget, and a request-level `OutputContract` may replace that whole -contract under the existing replacement semantics. Preparation validates and -freezes the effective budget along with the validation mode and schema plan. - -Only a completed `ValidationFailed` result under `basic`, `json`, or -`json_schema` can start or continue repair. The following do not trigger -repair: - -- `none` validation; -- an initially valid response; -- schema loading, decoding, registration, or compilation failures; -- an operational inability to execute validation; -- invalid requests, provider failures, transport failures, or cancellation; - and -- capacity-admission failures. - -Selecting `none` is the explicit way for a caller to allow empty output and -requires a zero repair budget. Rejecting a positive budget with `none` avoids a -contradictory configuration whose requested corrective calls could never be -eligible. Under `basic`, an empty or whitespace-only candidate fails validation -and becomes repairable. Under `json` and `json_schema`, empty content is -already invalid JSON and follows the same repair path as other JSON failures. - -The requested budget must be between zero and three. Output-contract validation -rejects negative values and values above three at the same source-neutral -boundary for prompt definitions, request replacements, ordinary preparation, -and prepared execution. The effective value is never silently clamped. - -`repair_attempts` counts additional model calls after the initial generation, -not total calls. `ValidationResult.RepairAttempts` reports calls actually -started. A response that validates on the first corrective call therefore -reports one repair attempt. - -## Repair Request Behavior - -Each corrective call extends the original rendered conversation rather than -replacing it with a standalone repair prompt. For a nonempty candidate, -Promptkit constructs the request from: - -1. the complete original rendered messages in their original order; -2. one additional `assistant` message containing the latest invalid candidate; - and -3. one additional `user` message containing Promptkit's application-neutral - correction instruction and the latest validation diagnostics. - -For an empty or whitespace-only candidate, Promptkit omits the additional -`assistant` message and appends only the corrective `user` message. This avoids -sending an empty assistant message that a compatible provider may reject and -does not invent placeholder model content. The corrective message states that -the previous response was empty. - -The corrective user message always: - -- identifies the effective validation mode and the current attempt number; -- supplies the current validation diagnostics as data; -- asks the model to preserve valid values and change only what is necessary to - satisfy the effective output contract; and -- uses clear boundaries so provider-controlled diagnostics are not confused - with Promptkit's instructions. - -The mode-specific instruction reflects the contract Promptkit can actually -enforce: - -- for `basic`, it asks for a nonempty response that satisfies the original - request; and -- for `json` and `json_schema`, it asks for only corrected JSON, without - Markdown fences or explanation. - -Representing the candidate as an `assistant` message keeps model output -separate from Promptkit's corrective `user` instruction and gives the model -access to the original task and source material when a missing or invalid -field must be regenerated. Promptkit does not flatten the original messages or -embed the candidate into the corrective message. - -Each attempt repairs only the latest candidate. Promptkit does not append the -entire history of earlier candidates, which would increase cost without adding -equivalent corrective value and could make the active correction ambiguous. -Every attempt starts again from the original messages and appends only the -latest nonempty candidate, when present, and latest diagnostics. - -JSON Schema mode retains the prepared structured-output specification, so a -compatible client receives the same provider-native schema contract on the -initial call and every corrective call. This preventive constraint remains -the first line of defense even though Promptkit independently validates the -returned content and can repair failures from providers that ignore, partially -implement, or fail to satisfy it. - -Plain `json` validation continues to permit any JSON value, including an -object, array, string, number, boolean, or `null`. Promptkit must not send a -provider-native JSON-object constraint for that mode because doing so would -silently narrow the declared validation contract. A future contract may add -provider-native object generation if consumers need that distinct behavior. - -Resending the original messages may allow a provider to reuse cached prompt -prefixes, especially when the original cache-control metadata is preserved. -Promptkit does not assume, require, measure, or promise provider caching, and -usage and cost remain whatever the selected client reports or charges. - -When it is not empty or whitespace-only, the latest candidate is included in -full and is never truncated or rewritten by Promptkit. The built-in client -already bounds successful provider response bodies; consumers that inject an -`LLMClient` own that client's response-size policy. Repair adds no smaller -candidate limit or silent size-based skip. - -Validation diagnostics included in the corrective message are limited to -65,536 UTF-8 bytes. Promptkit retains diagnostics in order, truncates only the -diagnostic feedback when necessary at a valid UTF-8 boundary, and includes a -bounded notice that additional diagnostic text was omitted. This prompt-safety -limit does not alter the complete diagnostics returned in the final -`ValidationResult`. - -## Empty Content And Provider Responses - -Content presence belongs to validation when the provider has returned an -otherwise valid completion envelope. Promptkit's built-in OpenAI-compatible -client therefore distinguishes these cases: - -- a first choice with an explicitly present string `content` value is a - completed generation candidate even when that string is empty or consists - only of whitespace; and -- no choices, a missing `content` field, `null` content, or non-string content - remains a malformed provider response and returns a generation error without - entering content repair. - -This distinction keeps the built-in and injected-client paths coherent. An -explicit empty candidate reaches `none`, `basic`, `json`, or `json_schema` -validation under the same rules regardless of which client produced it. A -malformed response envelope is not reclassified as invalid model content and -does not consume the repair budget. - -The initial and corrective calls use the same: - -- resolved backend identity and endpoint; -- model and effective execution settings, including presence-aware numeric and - reasoning values; -- direct credential or environment lookup boundary; -- effective session ID; -- structured-output specification; and -- capacity manager and selected backend pool. - -Repair does not reload prompt, profile, backend, input, or schema sources and -does not rerender the original prompt. It reuses the already rendered message -snapshot, including its roles, contents, and cache-control metadata. A -prepared execution uses its retained messages, validation plan, and frozen -effective target throughout repair. - -## Results And Failure Behavior - -After every successful corrective generation, Promptkit rebuilds the output -artifact from that candidate and validates it with the operation's existing -validation plan. The final `RawOutput`, artifact, validation result, and repair -attempt count all describe the same final candidate. - -Token usage is accumulated field by field across the initial response and each -completed corrective response. Timing covers the entire operation, including -repair. Prepared execution remains one-shot, and its handle is consumed by the -single execution even when that execution performs several generation calls. - -A failure while making a corrective model call retains the same public error -category and underlying identity it would have had during initial generation: - -- invalid model requests match `ErrInvalidRequest`; -- provider and transport generation failures match `ErrLLMGenerate`, including - any available `GenerationError` details from the built-in client; and -- cancellation and deadlines remain discoverable through the error chain. - -An operational failure while validating a corrective response continues to -match `ErrValidation`. These failures return no partial `RunResult` or partial -usage, consistent with existing operation failure behavior. Merely exhausting -the content-repair budget is different: validation completed successfully, so -the engine returns the final failed validation result rather than an error. - -## Capacity And Concurrency - -The ordinary run-admission lease spans preparation completion, initial -generation, validation, all corrective calls, and every exit. Repair never -performs a second admission. Each corrective generation uses the same -capacity-wrapped model client and independently reacquires an active-generation -permit for its backend, allowing unrelated accepted work to use the pool while -the repairing run validates or waits. - -Cancellation while waiting for a repair-generation permit prevents that model -call when cancellation wins the grant race. All admission and generation -permits must be released on successful repair, exhaustion, provider failure, -validation failure, and cancellation. - -## Architecture And Ownership - -The root facade installs Promptkit's default output repairer when assembling -the runner. It supplies the same capacity-wrapped `llm.Client` used for initial -generation, whether that client ultimately delegates to Promptkit's built-in -transport or a consumer-injected `LLMClient`. - -`internal/usecase` continues to own the bounded repair state machine, attempt -accounting, final-result selection, cumulative usage, error categorization, -and coordination with prepared validation. The default repairer owns only the -construction of the augmented conversation and the corrective generation -call. `internal/validate` remains the sole authority for structural validity, -while `internal/llm` retains responsibility for distinguishing explicit empty -content from an absent or malformed completion and `internal/capacity` retains -its scheduling responsibility. - -No repair-specific type needs to enter `internal/domain` beyond the existing -output-contract and validation-result state, and no public package or mutable -engine registry is introduced. - -## Public And Documentation Surface - -No new public method or option is required. The feature activates the existing -public fields and updates their contracts: - -- `OutputContract.RepairAttempts` describes the requested additional-call - budget, its zero-to-three range, and eligible validation modes; -- `ValidationResult.RepairAttempts` describes the attempts actually made; and -- `Run`, `RunPrepared`, and related result GoDoc describe exhaustion, failure, - usage, cancellation, and prepared-execution behavior. - -The framework format reference owns the exact -`repair_attempts` YAML semantics and the meaning of `none`, `basic`, `json`, -and `json_schema`. Consumer guidance contains one concise JSON Schema -example, note that `basic` can repair empty output, and emphasize that content -presence or structural validity does not establish domain correctness. The -internal runner document owns orchestration and test ownership. The -OpenAI-compatible integration contract owns the explicit-empty versus missing -content distinction. Architecture policy needs to change only if implementation -changes its existing package boundaries. - -Canonical current-state documentation describes implemented behavior in -accordance with the documentation policy. Roadmap material remains temporary -and is retired when it no longer describes future work. - -## Verification Expectations - -Testing should preserve the repository's lean ownership model and build on the -existing internal repair coverage: - -- domain and source-boundary tests own accepted repair budgets and output- - contract validation, including the upper bound and invalid `none` pairing; -- use-case tests own eligibility, bounded progression, latest-candidate - feedback, original-message preservation, non-accumulation of earlier - candidates, stop-on-valid behavior, exhaustion, cumulative usage, error - categories, prepared-plan reuse, cancellation, and capacity lifetime; -- default-repairer tests own the stable behavioral requirements of its prompt - construction, mode-appropriate correction, empty-candidate behavior, - diagnostic-feedback bound, and role boundaries without snapshotting - incidental prose; -- model-client tests own explicit empty and whitespace string content as - successful candidates and retain malformed-response coverage for no choices, - missing, null, and non-string content; -- root external-package tests prove representative public ordinary and - prepared repair workflows, including an injected client, without repeating - the internal state-machine matrix; and -- existing single-pass tests are revised to protect zero-budget and ineligible- - mode behavior rather than obsolete public-engine wiring. - -All tests remain deterministic, offline, parallel-safe, and independent of -real credentials or provider services. Maintainer validation continues to use -the canonical workflow in the -[development guide](../development.md#maintainer-validation). - -## Acceptance Criteria - -- A public-engine run with failed basic, JSON, or JSON Schema validation and a - positive budget makes no more than the permitted number of corrective calls. -- Output contracts accept zero through three repair attempts, reject larger or - negative values, and reject a positive budget under `none` validation. -- The first contract-valid candidate is returned and no later call is - made. -- Exhaustion returns the final invalid candidate with final diagnostics and - the actual attempt count. -- Zero-budget, initially successful, and `none` cases remain single-pass. -- Empty and whitespace-only candidates repair under `basic`, `json`, and - `json_schema` when the budget is positive, while `none` returns them without - repair. -- The built-in client sends explicit empty string content to validation but - retains generation errors for missing or malformed completion content. -- Ordinary and prepared execution preserve the same effective target, session, - credential, original rendered messages, structured-output, validation-plan, - capacity, and error semantics across every attempt. -- Every corrective request contains only the original messages, latest invalid - assistant candidate when nonempty, and latest corrective user message; - failed-attempt history does not accumulate and empty assistant messages are - not fabricated. -- Repair sends the complete candidate, bounds only diagnostic prompt feedback, - and leaves final validation diagnostics intact. -- Usage is cumulative across all completed generations, and no partial result - is returned after an operational error. -- Public GoDoc and canonical format, consumer, and internal documentation agree - with the implemented behavior and do not claim semantic correctness. - -## Non-Goals - -- Domain-specific, factual, qualitative, or business-rule validation. -- A consumer-supplied validator, repair prompt, repairer, or LLM-as-judge - extension point. -- Provider retries for HTTP failures, transport failures, rate limits, or - malformed successful response envelopes. -- Backoff, failover, alternate-profile selection, backend routing changes, or - a durable task queue. -- Repair for `none` validation or content requirements beyond `basic` - nonemptiness and the declared JSON contracts. -- A provider-native JSON-object constraint for plain `json` validation. -- Local stripping, extraction, normalization, or heuristic rewriting of model - output before validation. -- Persisting candidate history, returning every candidate, or exposing repair - prompts in public results. -- Depending on provider prompt caching or promising a particular reduction in - billed or reported input tokens. -- Reopening sources, changing the selected profile, or changing execution - settings between attempts. -- Guaranteeing a valid result when the model cannot satisfy the effective - output contract within the configured bound.