diff --git a/audit.md b/docs/roadmap/audit.md similarity index 100% rename from audit.md rename to docs/roadmap/audit.md diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 1e4ab9e..1bfaf79 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -38,7 +38,33 @@ consumers. ## Ideas -No ideas currently await selection. +### Public bounded output repair + +After the codebase-audit remediations are complete, Promptkit should make its +bounded output-repair capability available through the public engine. A +consumer should be able to request a limited number of corrective generation +attempts when JSON or JSON Schema output fails content validation, without +having to reproduce Promptkit's generation, validation, capacity, and result- +accounting orchestration. + +- Repair is validation recovery, not a general provider retry, failover, or + backoff policy. Transport failures, cancellation, and operational schema or + validation errors must retain their ordinary error behavior. +- Repair must stop after the first valid result or the configured attempt + bound. Exhausting the bound should preserve the final invalid result and its + validation diagnostics rather than inventing success. +- Initial generation and every repair attempt must use the same resolved + backend, effective execution settings and presence semantics, session, + credential boundary, structured-output contract, and backend-capacity + policy. +- Results should report the number of repair attempts and cumulative usage for + every model call made by the run. +- Ordinary and prepared execution should expose coherent behavior, including + cancellation, frozen prepared state, error identity, and capacity lifetime. + +Select this work only after the accepted audit findings affecting shared +execution invariants, validation, orchestration, transport, and repair +internals have been remediated. ## Entry Format diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..bb54583 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,840 @@ +# Audit Remediation Implementation Plan + +## Purpose + +This document is the decision-complete implementation plan for the accepted +findings in the [codebase audit](audit.md). It is written for a +`gpt-5.6-terra` coding agent that will implement one numbered stage per prompt, +in order. + +The audit remains the evidence and rationale for each finding. This plan owns +implementation order, selected policy decisions, required code and test work, +and stage gates. It does not activate the future public output-repair feature +described in the [future feature catalog](future.md); it only corrects and +protects the retained internal repair machinery on which that later feature +may build. + +## Implementation Policies + +Every stage must follow the [development guide](../development.md), +[architecture policy](../policy/architecture.md), +[testing policy](../policy/testing.md), and +[documentation policy](../policy/documentation.md). Before changing a +subsystem, read the focused current-state documents identified by the +development guide and inspect the exact implementation and tests named by the +stage. + +Apply these rules throughout: + +- Implement exactly one stage per agent prompt. Do not combine stages or begin + a later stage early. +- Inspect the working tree before editing and preserve unrelated changes. +- Use the code knowledge graph to locate symbols, callers, and dependency + paths; confirm important conclusions against source. +- Keep the root package as the public facade and implementation under + `internal/`. Do not expose internal representations or add a public package. +- Put source-neutral invariants in their assigned internal owner while + preserving source-specific normalization, error classification, and public + translation at existing boundaries. +- Add regression protection at the narrowest stable owner in the same stage + as a behavioral fix. Retain only representative integration coverage at + higher layers. +- Do not add tests to raise coverage percentages. Do not preserve tests that + assert an incidental algorithm, private constant, dormant serialization + shape, or duplicated lower-layer truth table. +- Keep all tests deterministic, offline, race-safe, credential-free, and free + of fixed-port or mutable-service assumptions. +- Do not add engine-wide caches, generic facade abstractions, scheduler + changes, provider retry policy, or new consumer configuration unless a stage + explicitly requires it. +- Update canonical GoDoc and current-state documents in the same stage as the + behavior they describe. Do not describe a later stage as already + implemented. +- Format changed Go files. Run the stage's focused commands, then at least + `go test ./...` and `go vet ./...`. Run focused race tests wherever the + stage changes ownership, cancellation, shared state, or lifecycle behavior. +- Do not commit, push, tag, or publish unless separately instructed. + +## Decisions Fixed By This Plan + +The implementing agent must not reopen these choices: + +1. **JSON-compatible numbers:** accept every value Go can faithfully encode as + a JSON number: every signed and unsigned integer width, finite `float32` and + `float64` values, and a `json.Number` whose text is valid JSON-number syntax. + Do not impose the current IEEE-754 safe-integer restriction. Reject NaN, + infinities, and malformed `json.Number` text. Preserve supported concrete + numeric types when copying. +2. **JSON-shaped traversal bounds:** allow at most 100 JSON container levels + and 100,000 produced JSON value nodes per `Copy` or `CopyMap` operation. + Count the root, each map/slice/array container, and every produced child + value; map keys are not separate nodes. Pointer and interface indirection + do not add JSON depth or an extra node. Repeated appearances of an acyclic + shared value count each produced occurrence. Continue rejecting active-path + cycles and return deterministic, path-aware validation errors on either + bound. +3. **Execution timeout bound:** a positive `TimeoutSeconds` must fit in + `time.Duration` after multiplication by `time.Second`. Derive the maximum + from `math.MaxInt64` and `time.Second`; do not duplicate its numeric literal + in tests or documentation. +4. **Output contracts:** the only valid formats are `text`, `markdown`, and + `json`; the only valid validation modes are `none`, `basic`, `json`, and + `json_schema`; repair attempts are non-negative; and `json_schema` requires + a nonblank schema path. A non-nil request replacement defaults an empty + format to `text` before shared validation. It does not default an empty + validation mode. +5. **Prompt content paths:** every `content_file` is an exact, relative path + resolved from its prompt file and contained by the configured prompt source + root. Directory, `fs.FS`, and single-file sources all reject absolute and + escaping paths. A single-file source's root is the containing directory of + that selected prompt file. Trimming determines only whether a value is + blank; it must not change the path opened. OS containment must account for + symlinks; containment inside an injected `fs.FS` remains expressed in that + filesystem's namespace. +6. **Profile IDs:** normalize file-backed IDs with `strings.TrimSpace` once, + just as in-memory IDs are normalized. Use the normalized value for + selection, duplicate detection, results, and diagnostics. A whitespace-only + ID is invalid, and IDs that become equal after normalization are + duplicates. +7. **Ordinary artifact files:** the built-in `File` reader supports regular + files, including symlinks whose targets are regular files. It rejects + directories, FIFOs, devices, sockets, and other non-regular targets before + consuming them. It remains unrestricted by an application root and does + not introduce an application-specific byte limit. +8. **Validation cancellation:** do not return early by abandoning goroutines + around `fs.FS` or the JSON Schema dependency. Promptkit must check + cancellation before, between, and after work it controls; read opened files + in context-checked chunks; and let a canceled context win before publishing + a result after synchronous decode, compile, or validation calls. Go's + `fs.FS` and the current JSON Schema library expose no general mechanism to + preempt a blocked `Open`, `Read`, compile, or validation method, so canonical + documentation must describe this synchronous limitation rather than claim + impossible asynchronous interruption. +9. **Successful provider-response limit:** the built-in OpenAI-compatible + client accepts at most 16 MiB (`16 << 20` bytes) for the complete successful + HTTP response body, including surrounding whitespace. The limit is fixed, + internal, and application-neutral. Exactly the limit is allowed; the first + byte beyond it fails as `internal/llm.ErrMalformedResponse`. Do not add a + public setting. Non-success response parsing remains outside this audit + remediation and belongs to the separate structured-generation-error + roadmap. +10. **Repair machinery:** retain and fix the internal repairer, cumulative + usage, and bounded repair state machine. The public engine must continue to + install no repairer and remain single-pass. Do not activate public repair + in this plan. + +## Stage 1: Centralize Execution-Setting And Session Invariants + +**Findings:** S05-F01, S17-F01, S14-F02. This stage also resolves the +source-specific evidence in S02-F02, S08-F01, and S11-F01. + +Add a source-neutral execution-setting validator to `internal/domain`. It must +validate temperature, maximum tokens, top-p, and timeout on a domain execution +target: temperature and top-p must be finite and within their closed ranges, +maximum tokens must be non-negative, and timeout must be non-negative and no +greater than the derived duration-safe maximum. Keep optional-pointer presence, +profile required fields, normalization, and error wrapping outside this +validator. + +Use that owner from: + +- in-memory profile validation in the root package; +- OS and `fs.FS` profile validation; +- resolved request/target validation in `internal/usecase`; and +- the built-in model client as a defensive final boundary. + +Remove the duplicated scalar comparisons from those callers. Preserve +`ErrInvalidConfig` for in-memory construction, profile-load identities for file +profiles, `ErrInvalidRequest` for runtime overrides, and the LLM package's +defensive invalid-request identity. Explicit numeric zero must retain its +presence semantics. + +Update `internal/domain.NormalizeSessionID` to reject invalid UTF-8 before +trimming or rune counting. Preserve the existing blank and 256-code-point +rules. Direct requests must still map failures to `ErrInvalidRequest`, while +session-template failures remain renderer failures. + +Add one domain-owned table for every exact setting boundary, finite neighbors, +NaN, both infinities, negative values, and the timeout representability edge. +Retain small boundary-integration cases for in-memory profiles, both file +source forms, request overrides through `Prepare` and `PrepareExecution`, and +the model-client defense. Add malformed UTF-8 session cases before, within, +and after otherwise valid content. + +Update the architecture policy and internal component overview so +`internal/domain` explicitly owns source-neutral invariants for its shared +execution values, without claiming ownership of source-specific policy. + +Run focused domain, profile, use-case, root, and LLM tests, including the +affected race-enabled request and profile cases, followed by the repository +test and vet gates. + +## Stage 2: Centralize Output-Contract Legality + +**Finding:** S17-F02, including the request-boundary symptom S11-F02. + +Add one pure `internal/domain` validator for `OutputContract`. It must enforce +the format, validation-mode, non-negative repair-attempt, and JSON-Schema path +rules fixed above. It must not load schemas or apply source/request defaults. + +Make prompt-definition normalization call the shared validator after its file- +specific normalization. Keep prompt-required fields and contextual +`ErrInvalidPromptDefinition` ownership in `internal/promptdef`. Make request +resolution default an empty replacement format to `text`, then call the same +validator and translate failure to `ErrInvalidRequest` before artifact, +rendering, validation, admission, or generation work. Keep schema loading and +compilation in `internal/validate`. + +Add a domain table covering every supported and unsupported enum, empty values, +negative and non-negative repair counts, and schema-path relationships. Retain +small prompt-source and use-case integration tables that prove correct error +categories and parity between `Prepare` and `PrepareExecution`; do not repeat +the entire domain table at those layers. + +Update the architecture and internal overview language added in Stage 1 to +include source-neutral output-contract invariants. Run focused domain, +prompt-definition, use-case, and root tests, then repository test and vet +gates. + +## Stage 3: Make JSON-Compatible Value Handling Coherent And Bounded + +**Findings:** S05-F02, S05-F03, S05-F04. + +Refactor `internal/jsonvalue` around the numeric and traversal decisions fixed +by this plan. Remove the safe-integer restriction and apply one numeric rule to +all supported representations. Preserve concrete named and unnamed scalar, +map, slice, and array types where the existing contract promises preservation; +keep nil versus empty container distinctions and `Copy` versus `CopyMap` empty- +key behavior. + +Extend the traversal state to track JSON container depth and produced-node +work. Enforce the 100-level and 100,000-node limits before allocation or +descent would cross them. Continue using active-path identity for cycle +detection; do not use alias memoization that would make distinct JSON paths +share mutable output. Errors must identify the structural path and whether the +depth or work budget was exceeded. + +Expand the focused package tables by behavior branch: signed and unsigned +integer widths, ordinary and named finite floats, `json.Number`, pointers and +interfaces, named maps/slices/arrays, nil and empty values, mixed nested trees, +arrays, mutation isolation, active cycles, alternating just-below/at/over +depth, and shared acyclic subgraphs just below and over the work budget. Tests +must derive their edges from package constants or relationships instead of +copying unexplained literals. + +Retain only representative public/backend/profile/prepared integration cases +that prove error translation and ownership. Update public GoDoc only if it +currently states the narrower safe-integer behavior; otherwise the existing +finite JSON-compatible-number contract remains canonical. Update the relevant +public value GoDoc and format/internal documentation to state that excessively +deep or large JSON-shaped values are rejected for safety; keep the exact +numeric limits owned by the internal constants rather than duplicating them +throughout consumer documentation. Run focused package and caller tests, +focused race tests, repository tests, and vet. + +## Stage 4: Consolidate Stable Public JSON And Remove Dormant Internal JSON + +**Findings:** S02-F01, S02-F05, S05-F05. + +Refactor `json.go` so each public value has one ordinary field mapping. Use +private aliases or embedded wire representations for ordinary fields and keep +only timestamp, millisecond-duration, and intentional omission exceptions +explicit. Preserve every existing JSON name and omission rule. + +Before converting `duration_ms`, reject values outside the millisecond range +that can be multiplied by `time.Millisecond` without overflow. Derive both +edges from `time.Duration` bounds. Return a contextual decode error and do not +partially update the receiver on failure. + +Add fully populated `PreparedRun` and `RunResult` contract cases. Verify all +ordinary fields, intentional omissions, zero and nonzero timing, complete +round trips, the largest safe positive and negative millisecond values, and +their first unsafe neighbors. + +Remove unused JSON tags and serialization tests from +`internal/domain.PreparedRun` after confirming production never marshals that +type. Keep credential absence protected at preparation/clone producers and +move any useful cache-control JSON assertion to the public `PreparedRun` +contract. Do not retain a parallel internal wire format. + +Run focused domain and root JSON tests, repository tests, and vet. + +## Stage 5: Harden Public Ownership And Diagnostic Contracts + +**Findings:** S01-F01, S02-F03, S02-F04, S13-F01. + +Extend the existing run-request formatting test with distinct input URI, +input-body, variable, and API-key sentinels. Require their absence from +`String`, `GoString`, `%v`, `%+v`, and `%#v` while retaining positive structural +summary assertions. + +Add one focused public-LLM-adapter ownership test. Have the injected client +mutate and retain prompt messages, cache-control pointers, nested target extra +parameters, and structured-output schema values; prove the domain/prepared +source remains unchanged and later details or execution cannot race with those +mutations. + +Add one direct all-field mapping test for `OpenAICompatibleProfile`. Populate +every field distinctly and compare the complete returned `Profile`. Keep only +the existing higher-level cases that prove normal validation and nested-value +ownership. + +Make copied `PreparedExecution` values format opaquely by using value-receiver +formatting behavior shared by non-nil pointers and values. A nil pointer may +use Go's normal `` formatting, but formatting must never panic or expose +internal types, field names, addresses, credentials, or content. Cover original +pointers, copied values, zero values, and nil pointers under string, Go-string, +and ordinary fmt verbs, and prove formatting does not claim or discard a +handle. + +Run focused root tests and the affected prepared/adapter race tests, followed +by repository tests and vet. + +## Stage 6: Correct Engine Construction Edges And Immutable Defaults + +**Findings:** S03-F01, S03-F02, S06-F01. + +Change the shared single-file option helper so trimming is used only for the +blank-input check. Perform `Stat`, path decomposition, storage, diagnostics, +and later access with the exact caller path for prompt, profile, and schema +files. Add one compact table covering existing leading- and trailing-whitespace +names through all three options. + +Strengthen engine construction tests with three discriminating cases: + +- reverse the argument order of in-memory, ordinary, fallback, and built-in + profile categories while retaining fixed category precedence; +- collide `Config.ProfileDir` with an ordinary profile option and prove the + option replaces the configuration source; and +- place a valid same-category replacement after an invalid option and prove + construction still fails at the earlier invalid option. + +Convert `internal/defaults.LLMRequestTimeoutDefault` from a variable to a +constant without changing its value or adding a setter. Do not add a test that +mutates or pins a noncontractual default; existing client deadline behavior is +the verification owner. + +Run focused engine construction, default-client construction, and race tests, +then repository tests and vet. + +## Stage 7: Contain And Preserve Prompt Content Paths + +**Finding:** S07-F01. + +Refactor prompt content resolution so both repository forms receive an +explicit source-root abstraction. Enforce the path decision fixed by this plan +before any content read. Use exact parsed path text after a separate blank +check. For OS sources, canonicalize the root and resolved target sufficiently +to reject symlink escape; for injected `fs.FS`, use its clean relative path +namespace. A parent component that remains inside the root is valid. Absolute, +escaping, and symlink-escaping targets are invalid. + +Apply the same behavioral table to an OS directory, `WithPromptFS`, and a +single-file source: ordinary sibling, nested parent still within root, parent +escape, absolute path, symlink escape where supported, and existing names with +leading or trailing whitespace. Prove rejected targets cause no outside read +and public operations preserve `ErrPromptLoad`. + +Update the framework format reference and internal source document to make the +single-file root and absolute-path rule explicit. Run focused prompt-definition +and public source tests, including race tests, then repository tests and vet. + +## Stage 8: Correct Prompt Selection, Strictness, Coverage, And Lookup Cost + +**Findings:** S07-F02, S07-F03, S07-F04, S07-F06. + +Correct both existing prompt repository paths before consolidating them in the +next stage: + +- Recover selector metadata from YAML `id` and `version`; never use a filename + stem as an identity. +- Apply normalized ID and requested-version selection before semantic + normalization or `content_file` reads. +- Associate strict YAML, semantic, and content errors only with a reliably + matching selected definition. An unidentifiable malformed file is unrelated + to point lookup; a reliably selected malformed file remains authoritative. +- Require exactly one YAML document. Comments and trailing whitespace are + allowed; a second empty or populated document and malformed trailing YAML + are `ErrInvalidYAML`. +- Continue scanning the YAML metadata required for duplicate detection, but + open content only for selected candidates. A selected content file is opened + once; unrelated and different-version bodies are never opened. + +Add paired OS and `fs.FS` regressions for same-stem/different-ID malformed +files, same-ID/different-version invalid files, selected malformed definitions, +additional YAML documents, duplicates, and counting filesystem behavior. +Add a compact normalization table for the previously uncovered missing +version, blank input name, blank message role, invalid output format, negative +repair attempts, and explicit blank default profile. Output-contract rows +should exercise the shared Stage 2 owner rather than recreate its full table. + +Run focused prompt-definition, use-case inspection, and root source tests, +focused race tests, repository tests, and vet. + +## Stage 9: Unify Prompt Repository Semantics + +**Finding:** S07-F05. + +After Stage 8 establishes correct behavior in both paths, replace their +duplicated discovery-to-selection algorithms with one source-neutral prompt +selection and normalization flow. Introduce only the small internal source +adapter needed for YAML discovery, bytes, exact content opening, display paths, +and root containment. Keep genuine OS and `fs.FS` mechanics at the adapter +edge. + +Move exact selection, version filtering, strict one-document decoding, +selected-error classification, normalization, duplicate handling, and +not-found behavior into the shared flow. Preserve point-in-time source access; +do not cache catalogs or definitions across operations. + +Turn the Stage 8 behavior matrix into a shared suite over both adapters and +retain source-specific tests only for distinct path and I/O failures. Delete +superseded duplicate helpers and tests only after the shared suite protects +their meaningful behavior. Use a counting filesystem and a before/after +benchmark over small and large prompt catalogs to confirm unrelated content is +not read and the refactor adds no second scan; do not enforce wall-clock +thresholds. + +Update the internal source document to describe the unified semantic owner. +Run focused package, integration, race, repository test, and vet gates. + +## Stage 10: Correct Profile Source Validation And Identity + +**Findings:** S08-F02, S08-F03, S08-F04, S08-F05. The non-finite scalar +symptom S08-F01 is already resolved by Stage 1. + +Make file profile normalization pass `extra_params` through +`internal/jsonvalue.CopyMap` before publishing a domain profile. Preserve +`ErrInvalidProfile` and source path context for empty keys, non-finite values, +nested invalid data, or traversal-budget failures. Do not move reserved +OpenAI-compatible field policy into the profile package. + +Use YAML metadata ID as the only selector; never infer authority from a +filename. Normalize the decoded ID once according to this plan. Require exactly +one YAML document in both metadata and strict selected decoding, so trailing +raw credentials, unknown fields, empty documents, and malformed YAML cannot be +ignored. Reliably selected malformed definitions must stop overlay fallback; +unrelated malformed files must not. + +Add shared OS and `fs.FS` tables for invalid/valid extra parameters, +same-stem/different-ID malformed files beside a valid profile, fallback +behavior, additional documents, leading/trailing/blank IDs, normalized +duplicates, and exact inspection/preparation of the normalized ID. Retain only +representative public error-translation cases. + +Update the framework format and internal source documents if needed to state +ID normalization and one-document behavior. Run focused profile, use-case, +root, and race tests, followed by repository tests and vet. + +## Stage 11: Eliminate Duplicate Profile Decoding + +**Finding:** S08-F06. + +Refactor point lookup so each file receives one metadata pass and only +canonical ID matches receive strict full decoding and normalization. Reuse +bytes already read for metadata; do not decode every unrelated full profile or +turn the repository into a cache. Preserve deterministic duplicate detection, +strict selected errors, overlay fallthrough only on not-found, and fresh +point-in-time reads on every operation. + +Add counting/parser-observation tests where stable behavior can be observed, +plus benchmarks for small and large catalogs reporting time and allocations. +Exercise valid selection, unrelated malformed files, selected malformed files, +duplicates, overlay fallthrough, and repeated lookup. Do not add brittle exact +allocation thresholds to ordinary tests. + +Run focused profile and root integration tests, benchmarks for diagnostic +comparison, race tests, repository tests, and vet. + +## Stage 12: Correct Artifact Semantics, Cancellation, And Hash Tests + +**Findings:** S09-F01, S09-F02, S09-F05. + +Treat an explicitly typed empty inline reference as a valid zero-byte artifact, +including `InlineWithURI`. Keep absence at the input map/reference boundary and +compute the same metadata and opaque equality value used for other bodies. + +For ordinary file references, inspect the target before opening and again +after opening; reject anything that is not a regular file under the decision +above. Replace unbounded `io.ReadAll` with a normal synchronous chunked read +that checks `ctx.Err()` before open, before and after each read, and before +publishing the artifact. Do not return partial artifacts, add a hidden size +limit, or launch an abandoned reader goroutine. + +Add source-parity cases for empty and nonempty inline, inline-with-URI, and file +content. Add a platform-appropriate FIFO regression proving the known FIFO is +rejected without requiring an external writer, and cancellation cases for a +pre-canceled file and a progressing regular-file read. Run them repeatedly and +under the race detector. + +Replace exact SHA-256 literals with relational assertions: nonempty and stable +for repeat reads, equal for equal inline/file bodies, unequal for changed +bodies, and propagated opaquely through preparation. Do not document or test a +specific algorithm. + +Update public GoDoc and the internal source document to describe regular-file +support and cancellation checkpoints. Run focused package, use-case, root, +race, repository test, and vet gates. + +## Stage 13: Make Rendering Cancellation-Aware And Reuse Artifact Text + +**Findings:** S09-F03, S09-F04. + +Check context before session work, before and after each template parse and +execution, before and after every message, and before returning the completed +prompt. Make the `input` helper return an error when cancellation is observed. +Do not run template execution in a detached goroutine. + +Within one `Render` call, lazily convert each named artifact body to text once +and memoize that string for the session and all messages. Build the cached +string in 64 KiB chunks with one pre-grown `strings.Builder`, checking the +context between chunks. Preserve bytes exactly, including invalid UTF-8; do not +cache across render calls or mutate artifacts. Unknown and nil inputs retain +their current errors, and a canceled conversion must not publish or cache a +partial string. + +Add deterministic tests for pre-cancellation, cancellation during the chunked +input conversion, and cancellation observed after final-message execution. +Require the context identity and no partial prompt while preserving active- +context template errors. Add benchmarks for one and repeated references across +session and messages; report allocations without hard-coded timing limits. + +Run focused renderer/use-case/root tests, benchmarks, repeated race tests, +repository tests, and vet. + +## Stage 14: Preserve Exact JSON Validation Semantics + +**Findings:** S10-F01, S10-F05. + +Create one helper for decoding exactly one JSON value with +`json.Decoder.UseNumber` and required EOF after trailing whitespace. Use it for +schema documents and JSON Schema instance values so large integers, precise +decimals, and exponents retain exact `json.Number` semantics through +compilation, prepared metadata, copying, and validation. + +For plain `ValidationJSON`, use a non-materializing complete-document syntax +check such as `json.Valid`; do not build a generic tree. Preserve the current +result distinction: malformed generated JSON is a completed failed validation, +not an operational error, and original output bytes remain unchanged. + +Add focused OS and `fs.FS` cases around `2^53`, `1e400`, precise decimals, +ordinary numbers, malformed syntax, and trailing values. Exercise schema +`const`, minimum/maximum, and `multipleOf`, and verify the public structured +schema retains exact numeric values. Add benchmarks for scalar, object, and +large-array JSON validation with allocation reporting but no wall-clock +contract. + +Update format/internal validation documentation only where it currently +implies float64-limited semantics. Run focused validator/use-case/root tests, +benchmarks, race tests, repository tests, and vet. + +## Stage 15: Escape Schema Resources And Compile Once Per Operation + +**Findings:** S10-F02, S10-F03. + +Represent schema compiler resources with `url.URL` rather than string +concatenation. Use canonical escaped file URLs for OS paths and a private +scheme URL whose path segments are escaped for `fs.FS`. Preserve separators, +decode resource paths exactly once at the loader boundary, and continue +rejecting remote and escaping references. Legal filenames containing percent, +space, `#`, `?`, or Unicode must compile, including contained relative +references. + +Unify JSON Schema preparation around `validate.PreparedValidation`: + +- the shared preparation pipeline must create one operation-local compiled + plan and derive provider-facing root schema metadata from that plan; +- `Prepare` may discard the plan after returning metadata; +- `Run` must retain and use the plan for its one operation so the schema graph + is not loaded or compiled again during validation; and +- `PrepareExecution` must retain the same plan in its frozen payload. + +Remove the document-only `SchemaDocumentLoader` capability if it has no +remaining production caller. Do not add an engine-wide or cross-operation +schema cache. Keep a clear private preparation carrier in `internal/usecase` +if needed so public `domain.PreparedRun` remains free of validator interfaces. + +Replace the existing legal-filename expected failure with valid behavior and +retain a genuine compiler-registration failure only if reachable through a +valid source. Add public parity tests for invalid keywords, malformed and +missing direct/second-level references, unsupported dialects, escapes, remote +references, and valid multi-document graphs. A counting source must show each +document read once per operation and fresh reads across separate operations. + +Update internal source, validator, and runner documentation for the unified +plan lifetime. Run focused validator/use-case/root tests, race tests, +repository tests, and vet. + +## Stage 16: Make Validation Cancellation Authoritative + +**Finding:** S10-F04. + +Apply the cancellation decision fixed above. Thread context through schema +resource loaders and all Promptkit-controlled read/decode helpers. Read opened +schema files in context-checked chunks. Check the context immediately before +and after JSON decoding, schema compilation, and schema execution; if +cancellation occurred during a synchronous dependency call, return the context +error instead of a schema or successful validation result. Do not publish a +partial plan or validation result. + +Do not place arbitrary `fs.FS` calls or JSON Schema work in goroutines merely +to race them against `ctx.Done()`. Tests must therefore distinguish: + +- prompt cancellation before work; +- cancellation between controlled read chunks; +- cancellation that becomes authoritative immediately after a synchronous + compile or validation call returns; and +- the documented limitation that Promptkit cannot preempt a dependency method + that never returns. + +Use deterministic controlled readers/contexts rather than sleeps. Assert no +goroutine growth or leaked work and preserve operational validation and public +context identities. Update validator GoDoc and internal source/runner documents +to state the synchronous cancellation boundary accurately. + +Run focused cancellation tests normally, repeatedly, and under the race +detector, followed by repository tests and vet. + +## Stage 17: Repair And Protect The Retained Internal Repair Path + +**Findings:** S12-F01, S12-F02, S12-F03. + +Retain the internal repair architecture. Extend `RepairRequest` with +`ExecutionTargetPresence` and carry the resolved presence bits unchanged into +the default repairer's `GenerateRequest`. Factor one use-case-local constructor +for common initial/repair generation fields—effective target, presence, +credential, backend identity, session, and structured output—while keeping the +initial and repair prompts intentionally separate. + +Accumulate every completed generation response's five token-usage fields into +run-level usage. The final content/raw output/artifact continues to come from +the last candidate, while usage includes initial generation and every completed +repair exactly once. A repair call that returns an error still returns no +partial public result under current error semantics. + +Replace the one-attempt-only repair coverage with a compact state-machine +table for: + +- initial success with no repair; +- ineligible basic validation despite a positive budget; +- explicit zero and inherited-zero presence across initial and repair calls; +- success before a larger budget is exhausted; +- exact exhaustion of a larger budget; and +- advancement of attempt number, maximum, prior output, diagnostics, final + status, cumulative usage, and collaborator call count. + +Retain the distinct capacity integration proving initial and repair generation +use the same backend pool and one whole-run admission lease, but simplify it if +the new focused table makes repair-state assertions redundant. + +Update the internal runner and capacity documents for presence fidelity and +cumulative usage. Do not alter `NewRunner` to install a repairer, public GoDoc +that says the engine is single-pass, or the future public repair roadmap. + +Run focused use-case, prepared, capacity, and race tests, followed by repository +tests and vet. + +## Stage 18: Preserve Transport Error Identities And Test Deterministically + +**Findings:** S14-F01, S04-F01, S16-F02. Timeout overflow S14-F02 is already +resolved through Stage 1's shared bound. + +Preserve the underlying `http.Client.Do` error in the chain while retaining +`internal/llm.ErrRequestFailed` and the public generation category. Do not add +headers, request content, endpoints, or provider bodies to error text. +Cancellation and deadline identities must survive caller cancellation, caller +deadline, generation deadline, and whole-request client timeout. + +Replace the port-9999 test with a controlled round tripper or `httptest` +endpoint that records the selected URL and returns a deliberate result. It must +make no host-dependent connection and must separately prove empty configured +base acceptance and request-endpoint precedence. + +Extend the existing external ordinary-run cancellation test to require both +`ErrLLMGenerate` and `context.Canceled`; retain lower-layer tests only for their +distinct error owners. + +Update the OpenAI-compatible integration and internal LLM documents for error +identity behavior. Run focused LLM and root tests with repetition and the race +detector, followed by repository tests and vet. + +## Stage 19: Validate And Compose Effective Provider Endpoints + +**Finding:** S14-F03. + +Add one source-neutral OpenAI-compatible base-endpoint validator in +`internal/domain`, alongside the effective execution target invariant. It must +trim surrounding configuration whitespace, require absolute HTTP or HTTPS with +a host, and reject user information, query, and fragment. Use it from backend +registration, in-memory and file profiles, resolved request overrides, and the +built-in client defense while preserving each boundary's existing config, +profile-load, invalid-request, or LLM error category. + +An empty configured base URL remains valid for a built-in client because a +resolved request endpoint may supply it later. Validate only a nonempty +configured base at construction, and always validate the final selected +endpoint before transport. Backends and endpoint-only profiles retain their +existing nonempty endpoint requirements. + +Compose the completion URL through parsed URL operations (prefer +`url.JoinPath`) so nested paths and trailing slashes reach exactly one +`/chat/completions` suffix. Never append to a raw string. + +Add endpoint tables for HTTP and HTTPS, hosts, nested paths, repeated trailing +slashes, queries, fragments, user information, relative paths, missing hosts, +unsupported schemes, and request/profile/config error mapping. Require every +invalid selected endpoint to fail before transport. + +Update the architecture/internal overview for domain endpoint invariants and +the OpenAI-compatible integration and internal LLM documents for URL behavior. +Run focused domain/backend/profile/use-case/LLM/root tests, repetition and race +tests where ownership crosses packages, followed by repository tests and vet. + +## Stage 20: Bound And Strictly Frame Successful Provider Responses + +**Findings:** S14-F04, S14-F05. + +Enforce the 16 MiB successful-response decision without first copying the +entire body. Reject an over-limit `Content-Length` immediately, but also wrap +the body in a counting/limited reader that reads at most one byte beyond the +limit so chunked or dishonest responses cannot bypass it. Exactly-limit bodies +remain valid. Always close the body; do not drain an unbounded oversized +stream. + +Decode exactly one response object. After the first decode, require only +trailing JSON whitespace and EOF. A second value, non-whitespace suffix, +truncated body, malformed JSON, or size overflow returns +`ErrMalformedResponse` with no partial response and no provider content in the +error. + +Add streaming tests just below, at, and one byte over the limit with and +without `Content-Length`, plus a continuing oversized stream. Assert bounded +bytes read, timely return, no partial result, and closure. Add trailing +whitespace success and trailing garbage/second-value failures. Retain ordinary +response mapping and redaction cases. + +Document the fixed successful-response boundary and strict one-document rule +in the integration and internal LLM documents. Explicitly leave bounded +non-success error-envelope parsing to the structured-generation-error roadmap. +Run focused transport tests normally, repeatedly, and under race, followed by +repository tests and vet. + +## Stage 21: Consolidate Transport Test Scaffolding + +**Finding:** S14-F06. + +After transport behavior is stable, introduce one small recording-provider +fixture for common request capture and successful/error response setup. +Organize focused tables around request mapping, authentication, endpoint +composition, timeout/error identity, and response framing. Keep specialized +round trippers/readers for cancellation, deadlines, byte counts, continuing +streams, and body closure. + +Retain every existing durable assertion for method, path, headers, +authentication, omission and explicit presence, reserved fields, cache +control, structured output, response mapping, usage, error redaction, and +timeout precedence. Retain all Stage 18 through 20 regressions. Delete repeated +servers, generic-map decoding, and response literals only where the fixture +makes the owning behavior clearer; do not replace wire assertions with a broad +snapshot. + +Run the LLM suite normally, with shuffle/repetition, and under the race +detector. Deliberately inspect the resulting test inventory against the audit's +transport matrix before running repository tests and vet. + +## Stage 22: Restore One Canonical Maintainer Validation Workflow + +**Finding:** S16-F01. + +Make `docs/development.md` the canonical owner of the complete local maintainer +workflow, as assigned by the documentation policy. Its validation section must +include, from the repository root: + +```sh +go test ./... +go test -race ./... +go vet ./... +go build ./... +go run ./examples/go-library/prepare +go run ./examples/go-library/run +``` + +It must also own the Go formatting, local Markdown link, `git diff --check`, +workspace/vendor/replacement, generated-output, credential, and working-tree +hygiene checks used before accepting changes. + +Change the testing policy to state the semantic requirements and link to that +canonical workflow instead of maintaining a partial competing command list. +Change the release procedure to invoke the development-guide validation as a +release prerequisite rather than presenting a separately maintained copy; +retain release-specific metadata, candidate, tag, and publication commands in +the release document. + +Run both examples offline and confirm that a missing or invalid Run example +fixture makes its command fail. Validate all changed Markdown links and ensure +current-state documentation describes only the implemented workflow. + +## Stage 23: Complete Traceability And Final Validation + +This final stage introduces no new behavior. Review the final tree against the +finding-to-stage table below and the evidence in `audit.md`. Confirm every +canonical group is implemented and every source-specific symptom retains its +required regression and error boundary. Do not mark a finding resolved merely +because a nearby refactor landed. + +Run the complete development-guide workflow, including both examples, all +formatting and link checks, and repository hygiene. Also run shuffled ordinary +tests and repeated race-enabled tests for the changed concurrency, +cancellation, prepared, validation, repair, and transport packages. Run the +accepted performance benchmarks for prompt lookup, profile lookup, rendering, +and JSON validation and record only qualitative before/after conclusions; do +not establish release timing promises. + +Inspect canonical GoDoc, formats, integration, architecture, and internal +documents against the final implementation. Confirm the public engine still +performs no output repair and the future repair entry remains future work. +Confirm the structured-generation-error feature was not implemented as part of +transport remediation. + +Leave `audit-sequence.md`, `audit.md`, and this plan in place for maintainer +review. Retire them only in a separately authorized roadmap-cleanup pass after +the remediation has been reviewed and accepted. + +## Finding-To-Stage Traceability + +| Stage | Canonical findings | Historical or source-specific records handled with the canonical owner | +| ---: | --- | --- | +| 1 | S05-F01, S17-F01, S14-F02 | S02-F02, S08-F01, S11-F01 | +| 2 | S17-F02 | S11-F02 | +| 3 | S05-F02, S05-F03, S05-F04 | None | +| 4 | S02-F01, S02-F05, S05-F05 | None | +| 5 | S01-F01, S02-F03, S02-F04, S13-F01 | None | +| 6 | S03-F01, S03-F02, S06-F01 | None | +| 7 | S07-F01 | None | +| 8 | S07-F02, S07-F03, S07-F04, S07-F06 | None | +| 9 | S07-F05 | None | +| 10 | S08-F02, S08-F03, S08-F04, S08-F05 | S08-F01 was handled in Stage 1 | +| 11 | S08-F06 | None | +| 12 | S09-F01, S09-F02, S09-F05 | None | +| 13 | S09-F03, S09-F04 | None | +| 14 | S10-F01, S10-F05 | None | +| 15 | S10-F02, S10-F03 | None | +| 16 | S10-F04 | None | +| 17 | S12-F01, S12-F02, S12-F03 | None | +| 18 | S14-F01, S04-F01, S16-F02 | S14-F02 was handled in Stage 1 | +| 19 | S14-F03 | None | +| 20 | S14-F04, S14-F05 | None | +| 21 | S14-F06 | None | +| 22 | S16-F01 | None | + +The table maps all 49 canonical remediation groups exactly once. S02-F02 is +the one superseded historical finding retained as evidence under S17-F01; +S08-F01, S11-F01, and S11-F02 retain their source-specific regression +responsibilities without being double-counted as canonical groups. + +## Open Questions + +None. The numeric contract, resource bounds, path and identity rules, +validation-cancellation limitation, repair retention, transport response +limit, and documentation ownership required to implement these stages are +fixed above.