841 lines
42 KiB
Markdown
841 lines
42 KiB
Markdown
# Audit Remediation Implementation Plan
|
|
|
|
## Purpose
|
|
|
|
This document is the decision-complete implementation plan for the accepted
|
|
findings in the [codebase audit](audit.md). It is written for a
|
|
`gpt-5.6-terra` coding agent that will implement one numbered stage per prompt,
|
|
in order.
|
|
|
|
The audit remains the evidence and rationale for each finding. This plan owns
|
|
implementation order, selected policy decisions, required code and test work,
|
|
and stage gates. It does not activate the future public output-repair feature
|
|
described in the [future feature catalog](future.md); it only corrects and
|
|
protects the retained internal repair machinery on which that later feature
|
|
may build.
|
|
|
|
## Implementation Policies
|
|
|
|
Every stage must follow the [development guide](../development.md),
|
|
[architecture policy](../policy/architecture.md),
|
|
[testing policy](../policy/testing.md), and
|
|
[documentation policy](../policy/documentation.md). Before changing a
|
|
subsystem, read the focused current-state documents identified by the
|
|
development guide and inspect the exact implementation and tests named by the
|
|
stage.
|
|
|
|
Apply these rules throughout:
|
|
|
|
- Implement exactly one stage per agent prompt. Do not combine stages or begin
|
|
a later stage early.
|
|
- Inspect the working tree before editing and preserve unrelated changes.
|
|
- Use the code knowledge graph to locate symbols, callers, and dependency
|
|
paths; confirm important conclusions against source.
|
|
- Keep the root package as the public facade and implementation under
|
|
`internal/`. Do not expose internal representations or add a public package.
|
|
- Put source-neutral invariants in their assigned internal owner while
|
|
preserving source-specific normalization, error classification, and public
|
|
translation at existing boundaries.
|
|
- Add regression protection at the narrowest stable owner in the same stage
|
|
as a behavioral fix. Retain only representative integration coverage at
|
|
higher layers.
|
|
- Do not add tests to raise coverage percentages. Do not preserve tests that
|
|
assert an incidental algorithm, private constant, dormant serialization
|
|
shape, or duplicated lower-layer truth table.
|
|
- Keep all tests deterministic, offline, race-safe, credential-free, and free
|
|
of fixed-port or mutable-service assumptions.
|
|
- Do not add engine-wide caches, generic facade abstractions, scheduler
|
|
changes, provider retry policy, or new consumer configuration unless a stage
|
|
explicitly requires it.
|
|
- Update canonical GoDoc and current-state documents in the same stage as the
|
|
behavior they describe. Do not describe a later stage as already
|
|
implemented.
|
|
- Format changed Go files. Run the stage's focused commands, then at least
|
|
`go test ./...` and `go vet ./...`. Run focused race tests wherever the
|
|
stage changes ownership, cancellation, shared state, or lifecycle behavior.
|
|
- Do not commit, push, tag, or publish unless separately instructed.
|
|
|
|
## Decisions Fixed By This Plan
|
|
|
|
The implementing agent must not reopen these choices:
|
|
|
|
1. **JSON-compatible numbers:** accept every value Go can faithfully encode as
|
|
a JSON number: every signed and unsigned integer width, finite `float32` and
|
|
`float64` values, and a `json.Number` whose text is valid JSON-number syntax.
|
|
Do not impose the current IEEE-754 safe-integer restriction. Reject NaN,
|
|
infinities, and malformed `json.Number` text. Preserve supported concrete
|
|
numeric types when copying.
|
|
2. **JSON-shaped traversal bounds:** allow at most 100 JSON container levels
|
|
and 100,000 produced JSON value nodes per `Copy` or `CopyMap` operation.
|
|
Count the root, each map/slice/array container, and every produced child
|
|
value; map keys are not separate nodes. Pointer and interface indirection
|
|
do not add JSON depth or an extra node. Repeated appearances of an acyclic
|
|
shared value count each produced occurrence. Continue rejecting active-path
|
|
cycles and return deterministic, path-aware validation errors on either
|
|
bound.
|
|
3. **Execution timeout bound:** a positive `TimeoutSeconds` must fit in
|
|
`time.Duration` after multiplication by `time.Second`. Derive the maximum
|
|
from `math.MaxInt64` and `time.Second`; do not duplicate its numeric literal
|
|
in tests or documentation.
|
|
4. **Output contracts:** the only valid formats are `text`, `markdown`, and
|
|
`json`; the only valid validation modes are `none`, `basic`, `json`, and
|
|
`json_schema`; repair attempts are non-negative; and `json_schema` requires
|
|
a nonblank schema path. A non-nil request replacement defaults an empty
|
|
format to `text` before shared validation. It does not default an empty
|
|
validation mode.
|
|
5. **Prompt content paths:** every `content_file` is an exact, relative path
|
|
resolved from its prompt file and contained by the configured prompt source
|
|
root. Directory, `fs.FS`, and single-file sources all reject absolute and
|
|
escaping paths. A single-file source's root is the containing directory of
|
|
that selected prompt file. Trimming determines only whether a value is
|
|
blank; it must not change the path opened. OS containment must account for
|
|
symlinks; containment inside an injected `fs.FS` remains expressed in that
|
|
filesystem's namespace.
|
|
6. **Profile IDs:** normalize file-backed IDs with `strings.TrimSpace` once,
|
|
just as in-memory IDs are normalized. Use the normalized value for
|
|
selection, duplicate detection, results, and diagnostics. A whitespace-only
|
|
ID is invalid, and IDs that become equal after normalization are
|
|
duplicates.
|
|
7. **Ordinary artifact files:** the built-in `File` reader supports regular
|
|
files, including symlinks whose targets are regular files. It rejects
|
|
directories, FIFOs, devices, sockets, and other non-regular targets before
|
|
consuming them. It remains unrestricted by an application root and does
|
|
not introduce an application-specific byte limit.
|
|
8. **Validation cancellation:** do not return early by abandoning goroutines
|
|
around `fs.FS` or the JSON Schema dependency. Promptkit must check
|
|
cancellation before, between, and after work it controls; read opened files
|
|
in context-checked chunks; and let a canceled context win before publishing
|
|
a result after synchronous decode, compile, or validation calls. Go's
|
|
`fs.FS` and the current JSON Schema library expose no general mechanism to
|
|
preempt a blocked `Open`, `Read`, compile, or validation method, so canonical
|
|
documentation must describe this synchronous limitation rather than claim
|
|
impossible asynchronous interruption.
|
|
9. **Successful provider-response limit:** the built-in OpenAI-compatible
|
|
client accepts at most 16 MiB (`16 << 20` bytes) for the complete successful
|
|
HTTP response body, including surrounding whitespace. The limit is fixed,
|
|
internal, and application-neutral. Exactly the limit is allowed; the first
|
|
byte beyond it fails as `internal/llm.ErrMalformedResponse`. Do not add a
|
|
public setting. Non-success response parsing remains outside this audit
|
|
remediation and belongs to the separate structured-generation-error
|
|
roadmap.
|
|
10. **Repair machinery:** retain and fix the internal repairer, cumulative
|
|
usage, and bounded repair state machine. The public engine must continue to
|
|
install no repairer and remain single-pass. Do not activate public repair
|
|
in this plan.
|
|
|
|
## Stage 1: Centralize Execution-Setting And Session Invariants
|
|
|
|
**Findings:** S05-F01, S17-F01, S14-F02. This stage also resolves the
|
|
source-specific evidence in S02-F02, S08-F01, and S11-F01.
|
|
|
|
Add a source-neutral execution-setting validator to `internal/domain`. It must
|
|
validate temperature, maximum tokens, top-p, and timeout on a domain execution
|
|
target: temperature and top-p must be finite and within their closed ranges,
|
|
maximum tokens must be non-negative, and timeout must be non-negative and no
|
|
greater than the derived duration-safe maximum. Keep optional-pointer presence,
|
|
profile required fields, normalization, and error wrapping outside this
|
|
validator.
|
|
|
|
Use that owner from:
|
|
|
|
- in-memory profile validation in the root package;
|
|
- OS and `fs.FS` profile validation;
|
|
- resolved request/target validation in `internal/usecase`; and
|
|
- the built-in model client as a defensive final boundary.
|
|
|
|
Remove the duplicated scalar comparisons from those callers. Preserve
|
|
`ErrInvalidConfig` for in-memory construction, profile-load identities for file
|
|
profiles, `ErrInvalidRequest` for runtime overrides, and the LLM package's
|
|
defensive invalid-request identity. Explicit numeric zero must retain its
|
|
presence semantics.
|
|
|
|
Update `internal/domain.NormalizeSessionID` to reject invalid UTF-8 before
|
|
trimming or rune counting. Preserve the existing blank and 256-code-point
|
|
rules. Direct requests must still map failures to `ErrInvalidRequest`, while
|
|
session-template failures remain renderer failures.
|
|
|
|
Add one domain-owned table for every exact setting boundary, finite neighbors,
|
|
NaN, both infinities, negative values, and the timeout representability edge.
|
|
Retain small boundary-integration cases for in-memory profiles, both file
|
|
source forms, request overrides through `Prepare` and `PrepareExecution`, and
|
|
the model-client defense. Add malformed UTF-8 session cases before, within,
|
|
and after otherwise valid content.
|
|
|
|
Update the architecture policy and internal component overview so
|
|
`internal/domain` explicitly owns source-neutral invariants for its shared
|
|
execution values, without claiming ownership of source-specific policy.
|
|
|
|
Run focused domain, profile, use-case, root, and LLM tests, including the
|
|
affected race-enabled request and profile cases, followed by the repository
|
|
test and vet gates.
|
|
|
|
## Stage 2: Centralize Output-Contract Legality
|
|
|
|
**Finding:** S17-F02, including the request-boundary symptom S11-F02.
|
|
|
|
Add one pure `internal/domain` validator for `OutputContract`. It must enforce
|
|
the format, validation-mode, non-negative repair-attempt, and JSON-Schema path
|
|
rules fixed above. It must not load schemas or apply source/request defaults.
|
|
|
|
Make prompt-definition normalization call the shared validator after its file-
|
|
specific normalization. Keep prompt-required fields and contextual
|
|
`ErrInvalidPromptDefinition` ownership in `internal/promptdef`. Make request
|
|
resolution default an empty replacement format to `text`, then call the same
|
|
validator and translate failure to `ErrInvalidRequest` before artifact,
|
|
rendering, validation, admission, or generation work. Keep schema loading and
|
|
compilation in `internal/validate`.
|
|
|
|
Add a domain table covering every supported and unsupported enum, empty values,
|
|
negative and non-negative repair counts, and schema-path relationships. Retain
|
|
small prompt-source and use-case integration tables that prove correct error
|
|
categories and parity between `Prepare` and `PrepareExecution`; do not repeat
|
|
the entire domain table at those layers.
|
|
|
|
Update the architecture and internal overview language added in Stage 1 to
|
|
include source-neutral output-contract invariants. Run focused domain,
|
|
prompt-definition, use-case, and root tests, then repository test and vet
|
|
gates.
|
|
|
|
## Stage 3: Make JSON-Compatible Value Handling Coherent And Bounded
|
|
|
|
**Findings:** S05-F02, S05-F03, S05-F04.
|
|
|
|
Refactor `internal/jsonvalue` around the numeric and traversal decisions fixed
|
|
by this plan. Remove the safe-integer restriction and apply one numeric rule to
|
|
all supported representations. Preserve concrete named and unnamed scalar,
|
|
map, slice, and array types where the existing contract promises preservation;
|
|
keep nil versus empty container distinctions and `Copy` versus `CopyMap` empty-
|
|
key behavior.
|
|
|
|
Extend the traversal state to track JSON container depth and produced-node
|
|
work. Enforce the 100-level and 100,000-node limits before allocation or
|
|
descent would cross them. Continue using active-path identity for cycle
|
|
detection; do not use alias memoization that would make distinct JSON paths
|
|
share mutable output. Errors must identify the structural path and whether the
|
|
depth or work budget was exceeded.
|
|
|
|
Expand the focused package tables by behavior branch: signed and unsigned
|
|
integer widths, ordinary and named finite floats, `json.Number`, pointers and
|
|
interfaces, named maps/slices/arrays, nil and empty values, mixed nested trees,
|
|
arrays, mutation isolation, active cycles, alternating just-below/at/over
|
|
depth, and shared acyclic subgraphs just below and over the work budget. Tests
|
|
must derive their edges from package constants or relationships instead of
|
|
copying unexplained literals.
|
|
|
|
Retain only representative public/backend/profile/prepared integration cases
|
|
that prove error translation and ownership. Update public GoDoc only if it
|
|
currently states the narrower safe-integer behavior; otherwise the existing
|
|
finite JSON-compatible-number contract remains canonical. Update the relevant
|
|
public value GoDoc and format/internal documentation to state that excessively
|
|
deep or large JSON-shaped values are rejected for safety; keep the exact
|
|
numeric limits owned by the internal constants rather than duplicating them
|
|
throughout consumer documentation. Run focused package and caller tests,
|
|
focused race tests, repository tests, and vet.
|
|
|
|
## Stage 4: Consolidate Stable Public JSON And Remove Dormant Internal JSON
|
|
|
|
**Findings:** S02-F01, S02-F05, S05-F05.
|
|
|
|
Refactor `json.go` so each public value has one ordinary field mapping. Use
|
|
private aliases or embedded wire representations for ordinary fields and keep
|
|
only timestamp, millisecond-duration, and intentional omission exceptions
|
|
explicit. Preserve every existing JSON name and omission rule.
|
|
|
|
Before converting `duration_ms`, reject values outside the millisecond range
|
|
that can be multiplied by `time.Millisecond` without overflow. Derive both
|
|
edges from `time.Duration` bounds. Return a contextual decode error and do not
|
|
partially update the receiver on failure.
|
|
|
|
Add fully populated `PreparedRun` and `RunResult` contract cases. Verify all
|
|
ordinary fields, intentional omissions, zero and nonzero timing, complete
|
|
round trips, the largest safe positive and negative millisecond values, and
|
|
their first unsafe neighbors.
|
|
|
|
Remove unused JSON tags and serialization tests from
|
|
`internal/domain.PreparedRun` after confirming production never marshals that
|
|
type. Keep credential absence protected at preparation/clone producers and
|
|
move any useful cache-control JSON assertion to the public `PreparedRun`
|
|
contract. Do not retain a parallel internal wire format.
|
|
|
|
Run focused domain and root JSON tests, repository tests, and vet.
|
|
|
|
## Stage 5: Harden Public Ownership And Diagnostic Contracts
|
|
|
|
**Findings:** S01-F01, S02-F03, S02-F04, S13-F01.
|
|
|
|
Extend the existing run-request formatting test with distinct input URI,
|
|
input-body, variable, and API-key sentinels. Require their absence from
|
|
`String`, `GoString`, `%v`, `%+v`, and `%#v` while retaining positive structural
|
|
summary assertions.
|
|
|
|
Add one focused public-LLM-adapter ownership test. Have the injected client
|
|
mutate and retain prompt messages, cache-control pointers, nested target extra
|
|
parameters, and structured-output schema values; prove the domain/prepared
|
|
source remains unchanged and later details or execution cannot race with those
|
|
mutations.
|
|
|
|
Add one direct all-field mapping test for `OpenAICompatibleProfile`. Populate
|
|
every field distinctly and compare the complete returned `Profile`. Keep only
|
|
the existing higher-level cases that prove normal validation and nested-value
|
|
ownership.
|
|
|
|
Make copied `PreparedExecution` values format opaquely by using value-receiver
|
|
formatting behavior shared by non-nil pointers and values. A nil pointer may
|
|
use Go's normal `<nil>` formatting, but formatting must never panic or expose
|
|
internal types, field names, addresses, credentials, or content. Cover original
|
|
pointers, copied values, zero values, and nil pointers under string, Go-string,
|
|
and ordinary fmt verbs, and prove formatting does not claim or discard a
|
|
handle.
|
|
|
|
Run focused root tests and the affected prepared/adapter race tests, followed
|
|
by repository tests and vet.
|
|
|
|
## Stage 6: Correct Engine Construction Edges And Immutable Defaults
|
|
|
|
**Findings:** S03-F01, S03-F02, S06-F01.
|
|
|
|
Change the shared single-file option helper so trimming is used only for the
|
|
blank-input check. Perform `Stat`, path decomposition, storage, diagnostics,
|
|
and later access with the exact caller path for prompt, profile, and schema
|
|
files. Add one compact table covering existing leading- and trailing-whitespace
|
|
names through all three options.
|
|
|
|
Strengthen engine construction tests with three discriminating cases:
|
|
|
|
- reverse the argument order of in-memory, ordinary, fallback, and built-in
|
|
profile categories while retaining fixed category precedence;
|
|
- collide `Config.ProfileDir` with an ordinary profile option and prove the
|
|
option replaces the configuration source; and
|
|
- place a valid same-category replacement after an invalid option and prove
|
|
construction still fails at the earlier invalid option.
|
|
|
|
Convert `internal/defaults.LLMRequestTimeoutDefault` from a variable to a
|
|
constant without changing its value or adding a setter. Do not add a test that
|
|
mutates or pins a noncontractual default; existing client deadline behavior is
|
|
the verification owner.
|
|
|
|
Run focused engine construction, default-client construction, and race tests,
|
|
then repository tests and vet.
|
|
|
|
## Stage 7: Contain And Preserve Prompt Content Paths
|
|
|
|
**Finding:** S07-F01.
|
|
|
|
Refactor prompt content resolution so both repository forms receive an
|
|
explicit source-root abstraction. Enforce the path decision fixed by this plan
|
|
before any content read. Use exact parsed path text after a separate blank
|
|
check. For OS sources, canonicalize the root and resolved target sufficiently
|
|
to reject symlink escape; for injected `fs.FS`, use its clean relative path
|
|
namespace. A parent component that remains inside the root is valid. Absolute,
|
|
escaping, and symlink-escaping targets are invalid.
|
|
|
|
Apply the same behavioral table to an OS directory, `WithPromptFS`, and a
|
|
single-file source: ordinary sibling, nested parent still within root, parent
|
|
escape, absolute path, symlink escape where supported, and existing names with
|
|
leading or trailing whitespace. Prove rejected targets cause no outside read
|
|
and public operations preserve `ErrPromptLoad`.
|
|
|
|
Update the framework format reference and internal source document to make the
|
|
single-file root and absolute-path rule explicit. Run focused prompt-definition
|
|
and public source tests, including race tests, then repository tests and vet.
|
|
|
|
## Stage 8: Correct Prompt Selection, Strictness, Coverage, And Lookup Cost
|
|
|
|
**Findings:** S07-F02, S07-F03, S07-F04, S07-F06.
|
|
|
|
Correct both existing prompt repository paths before consolidating them in the
|
|
next stage:
|
|
|
|
- Recover selector metadata from YAML `id` and `version`; never use a filename
|
|
stem as an identity.
|
|
- Apply normalized ID and requested-version selection before semantic
|
|
normalization or `content_file` reads.
|
|
- Associate strict YAML, semantic, and content errors only with a reliably
|
|
matching selected definition. An unidentifiable malformed file is unrelated
|
|
to point lookup; a reliably selected malformed file remains authoritative.
|
|
- Require exactly one YAML document. Comments and trailing whitespace are
|
|
allowed; a second empty or populated document and malformed trailing YAML
|
|
are `ErrInvalidYAML`.
|
|
- Continue scanning the YAML metadata required for duplicate detection, but
|
|
open content only for selected candidates. A selected content file is opened
|
|
once; unrelated and different-version bodies are never opened.
|
|
|
|
Add paired OS and `fs.FS` regressions for same-stem/different-ID malformed
|
|
files, same-ID/different-version invalid files, selected malformed definitions,
|
|
additional YAML documents, duplicates, and counting filesystem behavior.
|
|
Add a compact normalization table for the previously uncovered missing
|
|
version, blank input name, blank message role, invalid output format, negative
|
|
repair attempts, and explicit blank default profile. Output-contract rows
|
|
should exercise the shared Stage 2 owner rather than recreate its full table.
|
|
|
|
Run focused prompt-definition, use-case inspection, and root source tests,
|
|
focused race tests, repository tests, and vet.
|
|
|
|
## Stage 9: Unify Prompt Repository Semantics
|
|
|
|
**Finding:** S07-F05.
|
|
|
|
After Stage 8 establishes correct behavior in both paths, replace their
|
|
duplicated discovery-to-selection algorithms with one source-neutral prompt
|
|
selection and normalization flow. Introduce only the small internal source
|
|
adapter needed for YAML discovery, bytes, exact content opening, display paths,
|
|
and root containment. Keep genuine OS and `fs.FS` mechanics at the adapter
|
|
edge.
|
|
|
|
Move exact selection, version filtering, strict one-document decoding,
|
|
selected-error classification, normalization, duplicate handling, and
|
|
not-found behavior into the shared flow. Preserve point-in-time source access;
|
|
do not cache catalogs or definitions across operations.
|
|
|
|
Turn the Stage 8 behavior matrix into a shared suite over both adapters and
|
|
retain source-specific tests only for distinct path and I/O failures. Delete
|
|
superseded duplicate helpers and tests only after the shared suite protects
|
|
their meaningful behavior. Use a counting filesystem and a before/after
|
|
benchmark over small and large prompt catalogs to confirm unrelated content is
|
|
not read and the refactor adds no second scan; do not enforce wall-clock
|
|
thresholds.
|
|
|
|
Update the internal source document to describe the unified semantic owner.
|
|
Run focused package, integration, race, repository test, and vet gates.
|
|
|
|
## Stage 10: Correct Profile Source Validation And Identity
|
|
|
|
**Findings:** S08-F02, S08-F03, S08-F04, S08-F05. The non-finite scalar
|
|
symptom S08-F01 is already resolved by Stage 1.
|
|
|
|
Make file profile normalization pass `extra_params` through
|
|
`internal/jsonvalue.CopyMap` before publishing a domain profile. Preserve
|
|
`ErrInvalidProfile` and source path context for empty keys, non-finite values,
|
|
nested invalid data, or traversal-budget failures. Do not move reserved
|
|
OpenAI-compatible field policy into the profile package.
|
|
|
|
Use YAML metadata ID as the only selector; never infer authority from a
|
|
filename. Normalize the decoded ID once according to this plan. Require exactly
|
|
one YAML document in both metadata and strict selected decoding, so trailing
|
|
raw credentials, unknown fields, empty documents, and malformed YAML cannot be
|
|
ignored. Reliably selected malformed definitions must stop overlay fallback;
|
|
unrelated malformed files must not.
|
|
|
|
Add shared OS and `fs.FS` tables for invalid/valid extra parameters,
|
|
same-stem/different-ID malformed files beside a valid profile, fallback
|
|
behavior, additional documents, leading/trailing/blank IDs, normalized
|
|
duplicates, and exact inspection/preparation of the normalized ID. Retain only
|
|
representative public error-translation cases.
|
|
|
|
Update the framework format and internal source documents if needed to state
|
|
ID normalization and one-document behavior. Run focused profile, use-case,
|
|
root, and race tests, followed by repository tests and vet.
|
|
|
|
## Stage 11: Eliminate Duplicate Profile Decoding
|
|
|
|
**Finding:** S08-F06.
|
|
|
|
Refactor point lookup so each file receives one metadata pass and only
|
|
canonical ID matches receive strict full decoding and normalization. Reuse
|
|
bytes already read for metadata; do not decode every unrelated full profile or
|
|
turn the repository into a cache. Preserve deterministic duplicate detection,
|
|
strict selected errors, overlay fallthrough only on not-found, and fresh
|
|
point-in-time reads on every operation.
|
|
|
|
Add counting/parser-observation tests where stable behavior can be observed,
|
|
plus benchmarks for small and large catalogs reporting time and allocations.
|
|
Exercise valid selection, unrelated malformed files, selected malformed files,
|
|
duplicates, overlay fallthrough, and repeated lookup. Do not add brittle exact
|
|
allocation thresholds to ordinary tests.
|
|
|
|
Run focused profile and root integration tests, benchmarks for diagnostic
|
|
comparison, race tests, repository tests, and vet.
|
|
|
|
## Stage 12: Correct Artifact Semantics, Cancellation, And Hash Tests
|
|
|
|
**Findings:** S09-F01, S09-F02, S09-F05.
|
|
|
|
Treat an explicitly typed empty inline reference as a valid zero-byte artifact,
|
|
including `InlineWithURI`. Keep absence at the input map/reference boundary and
|
|
compute the same metadata and opaque equality value used for other bodies.
|
|
|
|
For ordinary file references, inspect the target before opening and again
|
|
after opening; reject anything that is not a regular file under the decision
|
|
above. Replace unbounded `io.ReadAll` with a normal synchronous chunked read
|
|
that checks `ctx.Err()` before open, before and after each read, and before
|
|
publishing the artifact. Do not return partial artifacts, add a hidden size
|
|
limit, or launch an abandoned reader goroutine.
|
|
|
|
Add source-parity cases for empty and nonempty inline, inline-with-URI, and file
|
|
content. Add a platform-appropriate FIFO regression proving the known FIFO is
|
|
rejected without requiring an external writer, and cancellation cases for a
|
|
pre-canceled file and a progressing regular-file read. Run them repeatedly and
|
|
under the race detector.
|
|
|
|
Replace exact SHA-256 literals with relational assertions: nonempty and stable
|
|
for repeat reads, equal for equal inline/file bodies, unequal for changed
|
|
bodies, and propagated opaquely through preparation. Do not document or test a
|
|
specific algorithm.
|
|
|
|
Update public GoDoc and the internal source document to describe regular-file
|
|
support and cancellation checkpoints. Run focused package, use-case, root,
|
|
race, repository test, and vet gates.
|
|
|
|
## Stage 13: Make Rendering Cancellation-Aware And Reuse Artifact Text
|
|
|
|
**Findings:** S09-F03, S09-F04.
|
|
|
|
Check context before session work, before and after each template parse and
|
|
execution, before and after every message, and before returning the completed
|
|
prompt. Make the `input` helper return an error when cancellation is observed.
|
|
Do not run template execution in a detached goroutine.
|
|
|
|
Within one `Render` call, lazily convert each named artifact body to text once
|
|
and memoize that string for the session and all messages. Build the cached
|
|
string in 64 KiB chunks with one pre-grown `strings.Builder`, checking the
|
|
context between chunks. Preserve bytes exactly, including invalid UTF-8; do not
|
|
cache across render calls or mutate artifacts. Unknown and nil inputs retain
|
|
their current errors, and a canceled conversion must not publish or cache a
|
|
partial string.
|
|
|
|
Add deterministic tests for pre-cancellation, cancellation during the chunked
|
|
input conversion, and cancellation observed after final-message execution.
|
|
Require the context identity and no partial prompt while preserving active-
|
|
context template errors. Add benchmarks for one and repeated references across
|
|
session and messages; report allocations without hard-coded timing limits.
|
|
|
|
Run focused renderer/use-case/root tests, benchmarks, repeated race tests,
|
|
repository tests, and vet.
|
|
|
|
## Stage 14: Preserve Exact JSON Validation Semantics
|
|
|
|
**Findings:** S10-F01, S10-F05.
|
|
|
|
Create one helper for decoding exactly one JSON value with
|
|
`json.Decoder.UseNumber` and required EOF after trailing whitespace. Use it for
|
|
schema documents and JSON Schema instance values so large integers, precise
|
|
decimals, and exponents retain exact `json.Number` semantics through
|
|
compilation, prepared metadata, copying, and validation.
|
|
|
|
For plain `ValidationJSON`, use a non-materializing complete-document syntax
|
|
check such as `json.Valid`; do not build a generic tree. Preserve the current
|
|
result distinction: malformed generated JSON is a completed failed validation,
|
|
not an operational error, and original output bytes remain unchanged.
|
|
|
|
Add focused OS and `fs.FS` cases around `2^53`, `1e400`, precise decimals,
|
|
ordinary numbers, malformed syntax, and trailing values. Exercise schema
|
|
`const`, minimum/maximum, and `multipleOf`, and verify the public structured
|
|
schema retains exact numeric values. Add benchmarks for scalar, object, and
|
|
large-array JSON validation with allocation reporting but no wall-clock
|
|
contract.
|
|
|
|
Update format/internal validation documentation only where it currently
|
|
implies float64-limited semantics. Run focused validator/use-case/root tests,
|
|
benchmarks, race tests, repository tests, and vet.
|
|
|
|
## Stage 15: Escape Schema Resources And Compile Once Per Operation
|
|
|
|
**Findings:** S10-F02, S10-F03.
|
|
|
|
Represent schema compiler resources with `url.URL` rather than string
|
|
concatenation. Use canonical escaped file URLs for OS paths and a private
|
|
scheme URL whose path segments are escaped for `fs.FS`. Preserve separators,
|
|
decode resource paths exactly once at the loader boundary, and continue
|
|
rejecting remote and escaping references. Legal filenames containing percent,
|
|
space, `#`, `?`, or Unicode must compile, including contained relative
|
|
references.
|
|
|
|
Unify JSON Schema preparation around `validate.PreparedValidation`:
|
|
|
|
- the shared preparation pipeline must create one operation-local compiled
|
|
plan and derive provider-facing root schema metadata from that plan;
|
|
- `Prepare` may discard the plan after returning metadata;
|
|
- `Run` must retain and use the plan for its one operation so the schema graph
|
|
is not loaded or compiled again during validation; and
|
|
- `PrepareExecution` must retain the same plan in its frozen payload.
|
|
|
|
Remove the document-only `SchemaDocumentLoader` capability if it has no
|
|
remaining production caller. Do not add an engine-wide or cross-operation
|
|
schema cache. Keep a clear private preparation carrier in `internal/usecase`
|
|
if needed so public `domain.PreparedRun` remains free of validator interfaces.
|
|
|
|
Replace the existing legal-filename expected failure with valid behavior and
|
|
retain a genuine compiler-registration failure only if reachable through a
|
|
valid source. Add public parity tests for invalid keywords, malformed and
|
|
missing direct/second-level references, unsupported dialects, escapes, remote
|
|
references, and valid multi-document graphs. A counting source must show each
|
|
document read once per operation and fresh reads across separate operations.
|
|
|
|
Update internal source, validator, and runner documentation for the unified
|
|
plan lifetime. Run focused validator/use-case/root tests, race tests,
|
|
repository tests, and vet.
|
|
|
|
## Stage 16: Make Validation Cancellation Authoritative
|
|
|
|
**Finding:** S10-F04.
|
|
|
|
Apply the cancellation decision fixed above. Thread context through schema
|
|
resource loaders and all Promptkit-controlled read/decode helpers. Read opened
|
|
schema files in context-checked chunks. Check the context immediately before
|
|
and after JSON decoding, schema compilation, and schema execution; if
|
|
cancellation occurred during a synchronous dependency call, return the context
|
|
error instead of a schema or successful validation result. Do not publish a
|
|
partial plan or validation result.
|
|
|
|
Do not place arbitrary `fs.FS` calls or JSON Schema work in goroutines merely
|
|
to race them against `ctx.Done()`. Tests must therefore distinguish:
|
|
|
|
- prompt cancellation before work;
|
|
- cancellation between controlled read chunks;
|
|
- cancellation that becomes authoritative immediately after a synchronous
|
|
compile or validation call returns; and
|
|
- the documented limitation that Promptkit cannot preempt a dependency method
|
|
that never returns.
|
|
|
|
Use deterministic controlled readers/contexts rather than sleeps. Assert no
|
|
goroutine growth or leaked work and preserve operational validation and public
|
|
context identities. Update validator GoDoc and internal source/runner documents
|
|
to state the synchronous cancellation boundary accurately.
|
|
|
|
Run focused cancellation tests normally, repeatedly, and under the race
|
|
detector, followed by repository tests and vet.
|
|
|
|
## Stage 17: Repair And Protect The Retained Internal Repair Path
|
|
|
|
**Findings:** S12-F01, S12-F02, S12-F03.
|
|
|
|
Retain the internal repair architecture. Extend `RepairRequest` with
|
|
`ExecutionTargetPresence` and carry the resolved presence bits unchanged into
|
|
the default repairer's `GenerateRequest`. Factor one use-case-local constructor
|
|
for common initial/repair generation fields—effective target, presence,
|
|
credential, backend identity, session, and structured output—while keeping the
|
|
initial and repair prompts intentionally separate.
|
|
|
|
Accumulate every completed generation response's five token-usage fields into
|
|
run-level usage. The final content/raw output/artifact continues to come from
|
|
the last candidate, while usage includes initial generation and every completed
|
|
repair exactly once. A repair call that returns an error still returns no
|
|
partial public result under current error semantics.
|
|
|
|
Replace the one-attempt-only repair coverage with a compact state-machine
|
|
table for:
|
|
|
|
- initial success with no repair;
|
|
- ineligible basic validation despite a positive budget;
|
|
- explicit zero and inherited-zero presence across initial and repair calls;
|
|
- success before a larger budget is exhausted;
|
|
- exact exhaustion of a larger budget; and
|
|
- advancement of attempt number, maximum, prior output, diagnostics, final
|
|
status, cumulative usage, and collaborator call count.
|
|
|
|
Retain the distinct capacity integration proving initial and repair generation
|
|
use the same backend pool and one whole-run admission lease, but simplify it if
|
|
the new focused table makes repair-state assertions redundant.
|
|
|
|
Update the internal runner and capacity documents for presence fidelity and
|
|
cumulative usage. Do not alter `NewRunner` to install a repairer, public GoDoc
|
|
that says the engine is single-pass, or the future public repair roadmap.
|
|
|
|
Run focused use-case, prepared, capacity, and race tests, followed by repository
|
|
tests and vet.
|
|
|
|
## Stage 18: Preserve Transport Error Identities And Test Deterministically
|
|
|
|
**Findings:** S14-F01, S04-F01, S16-F02. Timeout overflow S14-F02 is already
|
|
resolved through Stage 1's shared bound.
|
|
|
|
Preserve the underlying `http.Client.Do` error in the chain while retaining
|
|
`internal/llm.ErrRequestFailed` and the public generation category. Do not add
|
|
headers, request content, endpoints, or provider bodies to error text.
|
|
Cancellation and deadline identities must survive caller cancellation, caller
|
|
deadline, generation deadline, and whole-request client timeout.
|
|
|
|
Replace the port-9999 test with a controlled round tripper or `httptest`
|
|
endpoint that records the selected URL and returns a deliberate result. It must
|
|
make no host-dependent connection and must separately prove empty configured
|
|
base acceptance and request-endpoint precedence.
|
|
|
|
Extend the existing external ordinary-run cancellation test to require both
|
|
`ErrLLMGenerate` and `context.Canceled`; retain lower-layer tests only for their
|
|
distinct error owners.
|
|
|
|
Update the OpenAI-compatible integration and internal LLM documents for error
|
|
identity behavior. Run focused LLM and root tests with repetition and the race
|
|
detector, followed by repository tests and vet.
|
|
|
|
## Stage 19: Validate And Compose Effective Provider Endpoints
|
|
|
|
**Finding:** S14-F03.
|
|
|
|
Add one source-neutral OpenAI-compatible base-endpoint validator in
|
|
`internal/domain`, alongside the effective execution target invariant. It must
|
|
trim surrounding configuration whitespace, require absolute HTTP or HTTPS with
|
|
a host, and reject user information, query, and fragment. Use it from backend
|
|
registration, in-memory and file profiles, resolved request overrides, and the
|
|
built-in client defense while preserving each boundary's existing config,
|
|
profile-load, invalid-request, or LLM error category.
|
|
|
|
An empty configured base URL remains valid for a built-in client because a
|
|
resolved request endpoint may supply it later. Validate only a nonempty
|
|
configured base at construction, and always validate the final selected
|
|
endpoint before transport. Backends and endpoint-only profiles retain their
|
|
existing nonempty endpoint requirements.
|
|
|
|
Compose the completion URL through parsed URL operations (prefer
|
|
`url.JoinPath`) so nested paths and trailing slashes reach exactly one
|
|
`/chat/completions` suffix. Never append to a raw string.
|
|
|
|
Add endpoint tables for HTTP and HTTPS, hosts, nested paths, repeated trailing
|
|
slashes, queries, fragments, user information, relative paths, missing hosts,
|
|
unsupported schemes, and request/profile/config error mapping. Require every
|
|
invalid selected endpoint to fail before transport.
|
|
|
|
Update the architecture/internal overview for domain endpoint invariants and
|
|
the OpenAI-compatible integration and internal LLM documents for URL behavior.
|
|
Run focused domain/backend/profile/use-case/LLM/root tests, repetition and race
|
|
tests where ownership crosses packages, followed by repository tests and vet.
|
|
|
|
## Stage 20: Bound And Strictly Frame Successful Provider Responses
|
|
|
|
**Findings:** S14-F04, S14-F05.
|
|
|
|
Enforce the 16 MiB successful-response decision without first copying the
|
|
entire body. Reject an over-limit `Content-Length` immediately, but also wrap
|
|
the body in a counting/limited reader that reads at most one byte beyond the
|
|
limit so chunked or dishonest responses cannot bypass it. Exactly-limit bodies
|
|
remain valid. Always close the body; do not drain an unbounded oversized
|
|
stream.
|
|
|
|
Decode exactly one response object. After the first decode, require only
|
|
trailing JSON whitespace and EOF. A second value, non-whitespace suffix,
|
|
truncated body, malformed JSON, or size overflow returns
|
|
`ErrMalformedResponse` with no partial response and no provider content in the
|
|
error.
|
|
|
|
Add streaming tests just below, at, and one byte over the limit with and
|
|
without `Content-Length`, plus a continuing oversized stream. Assert bounded
|
|
bytes read, timely return, no partial result, and closure. Add trailing
|
|
whitespace success and trailing garbage/second-value failures. Retain ordinary
|
|
response mapping and redaction cases.
|
|
|
|
Document the fixed successful-response boundary and strict one-document rule
|
|
in the integration and internal LLM documents. Explicitly leave bounded
|
|
non-success error-envelope parsing to the structured-generation-error roadmap.
|
|
Run focused transport tests normally, repeatedly, and under race, followed by
|
|
repository tests and vet.
|
|
|
|
## Stage 21: Consolidate Transport Test Scaffolding
|
|
|
|
**Finding:** S14-F06.
|
|
|
|
After transport behavior is stable, introduce one small recording-provider
|
|
fixture for common request capture and successful/error response setup.
|
|
Organize focused tables around request mapping, authentication, endpoint
|
|
composition, timeout/error identity, and response framing. Keep specialized
|
|
round trippers/readers for cancellation, deadlines, byte counts, continuing
|
|
streams, and body closure.
|
|
|
|
Retain every existing durable assertion for method, path, headers,
|
|
authentication, omission and explicit presence, reserved fields, cache
|
|
control, structured output, response mapping, usage, error redaction, and
|
|
timeout precedence. Retain all Stage 18 through 20 regressions. Delete repeated
|
|
servers, generic-map decoding, and response literals only where the fixture
|
|
makes the owning behavior clearer; do not replace wire assertions with a broad
|
|
snapshot.
|
|
|
|
Run the LLM suite normally, with shuffle/repetition, and under the race
|
|
detector. Deliberately inspect the resulting test inventory against the audit's
|
|
transport matrix before running repository tests and vet.
|
|
|
|
## Stage 22: Restore One Canonical Maintainer Validation Workflow
|
|
|
|
**Finding:** S16-F01.
|
|
|
|
Make `docs/development.md` the canonical owner of the complete local maintainer
|
|
workflow, as assigned by the documentation policy. Its validation section must
|
|
include, from the repository root:
|
|
|
|
```sh
|
|
go test ./...
|
|
go test -race ./...
|
|
go vet ./...
|
|
go build ./...
|
|
go run ./examples/go-library/prepare
|
|
go run ./examples/go-library/run
|
|
```
|
|
|
|
It must also own the Go formatting, local Markdown link, `git diff --check`,
|
|
workspace/vendor/replacement, generated-output, credential, and working-tree
|
|
hygiene checks used before accepting changes.
|
|
|
|
Change the testing policy to state the semantic requirements and link to that
|
|
canonical workflow instead of maintaining a partial competing command list.
|
|
Change the release procedure to invoke the development-guide validation as a
|
|
release prerequisite rather than presenting a separately maintained copy;
|
|
retain release-specific metadata, candidate, tag, and publication commands in
|
|
the release document.
|
|
|
|
Run both examples offline and confirm that a missing or invalid Run example
|
|
fixture makes its command fail. Validate all changed Markdown links and ensure
|
|
current-state documentation describes only the implemented workflow.
|
|
|
|
## Stage 23: Complete Traceability And Final Validation
|
|
|
|
This final stage introduces no new behavior. Review the final tree against the
|
|
finding-to-stage table below and the evidence in `audit.md`. Confirm every
|
|
canonical group is implemented and every source-specific symptom retains its
|
|
required regression and error boundary. Do not mark a finding resolved merely
|
|
because a nearby refactor landed.
|
|
|
|
Run the complete development-guide workflow, including both examples, all
|
|
formatting and link checks, and repository hygiene. Also run shuffled ordinary
|
|
tests and repeated race-enabled tests for the changed concurrency,
|
|
cancellation, prepared, validation, repair, and transport packages. Run the
|
|
accepted performance benchmarks for prompt lookup, profile lookup, rendering,
|
|
and JSON validation and record only qualitative before/after conclusions; do
|
|
not establish release timing promises.
|
|
|
|
Inspect canonical GoDoc, formats, integration, architecture, and internal
|
|
documents against the final implementation. Confirm the public engine still
|
|
performs no output repair and the future repair entry remains future work.
|
|
Confirm the structured-generation-error feature was not implemented as part of
|
|
transport remediation.
|
|
|
|
Leave `audit-sequence.md`, `audit.md`, and this plan in place for maintainer
|
|
review. Retire them only in a separately authorized roadmap-cleanup pass after
|
|
the remediation has been reviewed and accepted.
|
|
|
|
## Finding-To-Stage Traceability
|
|
|
|
| Stage | Canonical findings | Historical or source-specific records handled with the canonical owner |
|
|
| ---: | --- | --- |
|
|
| 1 | S05-F01, S17-F01, S14-F02 | S02-F02, S08-F01, S11-F01 |
|
|
| 2 | S17-F02 | S11-F02 |
|
|
| 3 | S05-F02, S05-F03, S05-F04 | None |
|
|
| 4 | S02-F01, S02-F05, S05-F05 | None |
|
|
| 5 | S01-F01, S02-F03, S02-F04, S13-F01 | None |
|
|
| 6 | S03-F01, S03-F02, S06-F01 | None |
|
|
| 7 | S07-F01 | None |
|
|
| 8 | S07-F02, S07-F03, S07-F04, S07-F06 | None |
|
|
| 9 | S07-F05 | None |
|
|
| 10 | S08-F02, S08-F03, S08-F04, S08-F05 | S08-F01 was handled in Stage 1 |
|
|
| 11 | S08-F06 | None |
|
|
| 12 | S09-F01, S09-F02, S09-F05 | None |
|
|
| 13 | S09-F03, S09-F04 | None |
|
|
| 14 | S10-F01, S10-F05 | None |
|
|
| 15 | S10-F02, S10-F03 | None |
|
|
| 16 | S10-F04 | None |
|
|
| 17 | S12-F01, S12-F02, S12-F03 | None |
|
|
| 18 | S14-F01, S04-F01, S16-F02 | S14-F02 was handled in Stage 1 |
|
|
| 19 | S14-F03 | None |
|
|
| 20 | S14-F04, S14-F05 | None |
|
|
| 21 | S14-F06 | None |
|
|
| 22 | S16-F01 | None |
|
|
|
|
The table maps all 49 canonical remediation groups exactly once. S02-F02 is
|
|
the one superseded historical finding retained as evidence under S17-F01;
|
|
S08-F01, S11-F01, and S11-F02 retain their source-specific regression
|
|
responsibilities without being double-counted as canonical groups.
|
|
|
|
## Open Questions
|
|
|
|
None. The numeric contract, resource bounds, path and identity rules,
|
|
validation-cancellation limitation, repair retention, transport response
|
|
limit, and documentation ownership required to implement these stages are
|
|
fixed above.
|