Plan the codebase audit remediation
This commit is contained in:
@@ -38,7 +38,33 @@ consumers.
|
||||
|
||||
## Ideas
|
||||
|
||||
No ideas currently await selection.
|
||||
### Public bounded output repair
|
||||
|
||||
After the codebase-audit remediations are complete, Promptkit should make its
|
||||
bounded output-repair capability available through the public engine. A
|
||||
consumer should be able to request a limited number of corrective generation
|
||||
attempts when JSON or JSON Schema output fails content validation, without
|
||||
having to reproduce Promptkit's generation, validation, capacity, and result-
|
||||
accounting orchestration.
|
||||
|
||||
- Repair is validation recovery, not a general provider retry, failover, or
|
||||
backoff policy. Transport failures, cancellation, and operational schema or
|
||||
validation errors must retain their ordinary error behavior.
|
||||
- Repair must stop after the first valid result or the configured attempt
|
||||
bound. Exhausting the bound should preserve the final invalid result and its
|
||||
validation diagnostics rather than inventing success.
|
||||
- Initial generation and every repair attempt must use the same resolved
|
||||
backend, effective execution settings and presence semantics, session,
|
||||
credential boundary, structured-output contract, and backend-capacity
|
||||
policy.
|
||||
- Results should report the number of repair attempts and cumulative usage for
|
||||
every model call made by the run.
|
||||
- Ordinary and prepared execution should expose coherent behavior, including
|
||||
cancellation, frozen prepared state, error identity, and capacity lifetime.
|
||||
|
||||
Select this work only after the accepted audit findings affecting shared
|
||||
execution invariants, validation, orchestration, transport, and repair
|
||||
internals have been remediated.
|
||||
|
||||
## Entry Format
|
||||
|
||||
|
||||
840
docs/roadmap/implementation.md
Normal file
840
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,840 @@
|
||||
# Audit Remediation Implementation Plan
|
||||
|
||||
## Purpose
|
||||
|
||||
This document is the decision-complete implementation plan for the accepted
|
||||
findings in the [codebase audit](audit.md). It is written for a
|
||||
`gpt-5.6-terra` coding agent that will implement one numbered stage per prompt,
|
||||
in order.
|
||||
|
||||
The audit remains the evidence and rationale for each finding. This plan owns
|
||||
implementation order, selected policy decisions, required code and test work,
|
||||
and stage gates. It does not activate the future public output-repair feature
|
||||
described in the [future feature catalog](future.md); it only corrects and
|
||||
protects the retained internal repair machinery on which that later feature
|
||||
may build.
|
||||
|
||||
## Implementation Policies
|
||||
|
||||
Every stage must follow the [development guide](../development.md),
|
||||
[architecture policy](../policy/architecture.md),
|
||||
[testing policy](../policy/testing.md), and
|
||||
[documentation policy](../policy/documentation.md). Before changing a
|
||||
subsystem, read the focused current-state documents identified by the
|
||||
development guide and inspect the exact implementation and tests named by the
|
||||
stage.
|
||||
|
||||
Apply these rules throughout:
|
||||
|
||||
- Implement exactly one stage per agent prompt. Do not combine stages or begin
|
||||
a later stage early.
|
||||
- Inspect the working tree before editing and preserve unrelated changes.
|
||||
- Use the code knowledge graph to locate symbols, callers, and dependency
|
||||
paths; confirm important conclusions against source.
|
||||
- Keep the root package as the public facade and implementation under
|
||||
`internal/`. Do not expose internal representations or add a public package.
|
||||
- Put source-neutral invariants in their assigned internal owner while
|
||||
preserving source-specific normalization, error classification, and public
|
||||
translation at existing boundaries.
|
||||
- Add regression protection at the narrowest stable owner in the same stage
|
||||
as a behavioral fix. Retain only representative integration coverage at
|
||||
higher layers.
|
||||
- Do not add tests to raise coverage percentages. Do not preserve tests that
|
||||
assert an incidental algorithm, private constant, dormant serialization
|
||||
shape, or duplicated lower-layer truth table.
|
||||
- Keep all tests deterministic, offline, race-safe, credential-free, and free
|
||||
of fixed-port or mutable-service assumptions.
|
||||
- Do not add engine-wide caches, generic facade abstractions, scheduler
|
||||
changes, provider retry policy, or new consumer configuration unless a stage
|
||||
explicitly requires it.
|
||||
- Update canonical GoDoc and current-state documents in the same stage as the
|
||||
behavior they describe. Do not describe a later stage as already
|
||||
implemented.
|
||||
- Format changed Go files. Run the stage's focused commands, then at least
|
||||
`go test ./...` and `go vet ./...`. Run focused race tests wherever the
|
||||
stage changes ownership, cancellation, shared state, or lifecycle behavior.
|
||||
- Do not commit, push, tag, or publish unless separately instructed.
|
||||
|
||||
## Decisions Fixed By This Plan
|
||||
|
||||
The implementing agent must not reopen these choices:
|
||||
|
||||
1. **JSON-compatible numbers:** accept every value Go can faithfully encode as
|
||||
a JSON number: every signed and unsigned integer width, finite `float32` and
|
||||
`float64` values, and a `json.Number` whose text is valid JSON-number syntax.
|
||||
Do not impose the current IEEE-754 safe-integer restriction. Reject NaN,
|
||||
infinities, and malformed `json.Number` text. Preserve supported concrete
|
||||
numeric types when copying.
|
||||
2. **JSON-shaped traversal bounds:** allow at most 100 JSON container levels
|
||||
and 100,000 produced JSON value nodes per `Copy` or `CopyMap` operation.
|
||||
Count the root, each map/slice/array container, and every produced child
|
||||
value; map keys are not separate nodes. Pointer and interface indirection
|
||||
do not add JSON depth or an extra node. Repeated appearances of an acyclic
|
||||
shared value count each produced occurrence. Continue rejecting active-path
|
||||
cycles and return deterministic, path-aware validation errors on either
|
||||
bound.
|
||||
3. **Execution timeout bound:** a positive `TimeoutSeconds` must fit in
|
||||
`time.Duration` after multiplication by `time.Second`. Derive the maximum
|
||||
from `math.MaxInt64` and `time.Second`; do not duplicate its numeric literal
|
||||
in tests or documentation.
|
||||
4. **Output contracts:** the only valid formats are `text`, `markdown`, and
|
||||
`json`; the only valid validation modes are `none`, `basic`, `json`, and
|
||||
`json_schema`; repair attempts are non-negative; and `json_schema` requires
|
||||
a nonblank schema path. A non-nil request replacement defaults an empty
|
||||
format to `text` before shared validation. It does not default an empty
|
||||
validation mode.
|
||||
5. **Prompt content paths:** every `content_file` is an exact, relative path
|
||||
resolved from its prompt file and contained by the configured prompt source
|
||||
root. Directory, `fs.FS`, and single-file sources all reject absolute and
|
||||
escaping paths. A single-file source's root is the containing directory of
|
||||
that selected prompt file. Trimming determines only whether a value is
|
||||
blank; it must not change the path opened. OS containment must account for
|
||||
symlinks; containment inside an injected `fs.FS` remains expressed in that
|
||||
filesystem's namespace.
|
||||
6. **Profile IDs:** normalize file-backed IDs with `strings.TrimSpace` once,
|
||||
just as in-memory IDs are normalized. Use the normalized value for
|
||||
selection, duplicate detection, results, and diagnostics. A whitespace-only
|
||||
ID is invalid, and IDs that become equal after normalization are
|
||||
duplicates.
|
||||
7. **Ordinary artifact files:** the built-in `File` reader supports regular
|
||||
files, including symlinks whose targets are regular files. It rejects
|
||||
directories, FIFOs, devices, sockets, and other non-regular targets before
|
||||
consuming them. It remains unrestricted by an application root and does
|
||||
not introduce an application-specific byte limit.
|
||||
8. **Validation cancellation:** do not return early by abandoning goroutines
|
||||
around `fs.FS` or the JSON Schema dependency. Promptkit must check
|
||||
cancellation before, between, and after work it controls; read opened files
|
||||
in context-checked chunks; and let a canceled context win before publishing
|
||||
a result after synchronous decode, compile, or validation calls. Go's
|
||||
`fs.FS` and the current JSON Schema library expose no general mechanism to
|
||||
preempt a blocked `Open`, `Read`, compile, or validation method, so canonical
|
||||
documentation must describe this synchronous limitation rather than claim
|
||||
impossible asynchronous interruption.
|
||||
9. **Successful provider-response limit:** the built-in OpenAI-compatible
|
||||
client accepts at most 16 MiB (`16 << 20` bytes) for the complete successful
|
||||
HTTP response body, including surrounding whitespace. The limit is fixed,
|
||||
internal, and application-neutral. Exactly the limit is allowed; the first
|
||||
byte beyond it fails as `internal/llm.ErrMalformedResponse`. Do not add a
|
||||
public setting. Non-success response parsing remains outside this audit
|
||||
remediation and belongs to the separate structured-generation-error
|
||||
roadmap.
|
||||
10. **Repair machinery:** retain and fix the internal repairer, cumulative
|
||||
usage, and bounded repair state machine. The public engine must continue to
|
||||
install no repairer and remain single-pass. Do not activate public repair
|
||||
in this plan.
|
||||
|
||||
## Stage 1: Centralize Execution-Setting And Session Invariants
|
||||
|
||||
**Findings:** S05-F01, S17-F01, S14-F02. This stage also resolves the
|
||||
source-specific evidence in S02-F02, S08-F01, and S11-F01.
|
||||
|
||||
Add a source-neutral execution-setting validator to `internal/domain`. It must
|
||||
validate temperature, maximum tokens, top-p, and timeout on a domain execution
|
||||
target: temperature and top-p must be finite and within their closed ranges,
|
||||
maximum tokens must be non-negative, and timeout must be non-negative and no
|
||||
greater than the derived duration-safe maximum. Keep optional-pointer presence,
|
||||
profile required fields, normalization, and error wrapping outside this
|
||||
validator.
|
||||
|
||||
Use that owner from:
|
||||
|
||||
- in-memory profile validation in the root package;
|
||||
- OS and `fs.FS` profile validation;
|
||||
- resolved request/target validation in `internal/usecase`; and
|
||||
- the built-in model client as a defensive final boundary.
|
||||
|
||||
Remove the duplicated scalar comparisons from those callers. Preserve
|
||||
`ErrInvalidConfig` for in-memory construction, profile-load identities for file
|
||||
profiles, `ErrInvalidRequest` for runtime overrides, and the LLM package's
|
||||
defensive invalid-request identity. Explicit numeric zero must retain its
|
||||
presence semantics.
|
||||
|
||||
Update `internal/domain.NormalizeSessionID` to reject invalid UTF-8 before
|
||||
trimming or rune counting. Preserve the existing blank and 256-code-point
|
||||
rules. Direct requests must still map failures to `ErrInvalidRequest`, while
|
||||
session-template failures remain renderer failures.
|
||||
|
||||
Add one domain-owned table for every exact setting boundary, finite neighbors,
|
||||
NaN, both infinities, negative values, and the timeout representability edge.
|
||||
Retain small boundary-integration cases for in-memory profiles, both file
|
||||
source forms, request overrides through `Prepare` and `PrepareExecution`, and
|
||||
the model-client defense. Add malformed UTF-8 session cases before, within,
|
||||
and after otherwise valid content.
|
||||
|
||||
Update the architecture policy and internal component overview so
|
||||
`internal/domain` explicitly owns source-neutral invariants for its shared
|
||||
execution values, without claiming ownership of source-specific policy.
|
||||
|
||||
Run focused domain, profile, use-case, root, and LLM tests, including the
|
||||
affected race-enabled request and profile cases, followed by the repository
|
||||
test and vet gates.
|
||||
|
||||
## Stage 2: Centralize Output-Contract Legality
|
||||
|
||||
**Finding:** S17-F02, including the request-boundary symptom S11-F02.
|
||||
|
||||
Add one pure `internal/domain` validator for `OutputContract`. It must enforce
|
||||
the format, validation-mode, non-negative repair-attempt, and JSON-Schema path
|
||||
rules fixed above. It must not load schemas or apply source/request defaults.
|
||||
|
||||
Make prompt-definition normalization call the shared validator after its file-
|
||||
specific normalization. Keep prompt-required fields and contextual
|
||||
`ErrInvalidPromptDefinition` ownership in `internal/promptdef`. Make request
|
||||
resolution default an empty replacement format to `text`, then call the same
|
||||
validator and translate failure to `ErrInvalidRequest` before artifact,
|
||||
rendering, validation, admission, or generation work. Keep schema loading and
|
||||
compilation in `internal/validate`.
|
||||
|
||||
Add a domain table covering every supported and unsupported enum, empty values,
|
||||
negative and non-negative repair counts, and schema-path relationships. Retain
|
||||
small prompt-source and use-case integration tables that prove correct error
|
||||
categories and parity between `Prepare` and `PrepareExecution`; do not repeat
|
||||
the entire domain table at those layers.
|
||||
|
||||
Update the architecture and internal overview language added in Stage 1 to
|
||||
include source-neutral output-contract invariants. Run focused domain,
|
||||
prompt-definition, use-case, and root tests, then repository test and vet
|
||||
gates.
|
||||
|
||||
## Stage 3: Make JSON-Compatible Value Handling Coherent And Bounded
|
||||
|
||||
**Findings:** S05-F02, S05-F03, S05-F04.
|
||||
|
||||
Refactor `internal/jsonvalue` around the numeric and traversal decisions fixed
|
||||
by this plan. Remove the safe-integer restriction and apply one numeric rule to
|
||||
all supported representations. Preserve concrete named and unnamed scalar,
|
||||
map, slice, and array types where the existing contract promises preservation;
|
||||
keep nil versus empty container distinctions and `Copy` versus `CopyMap` empty-
|
||||
key behavior.
|
||||
|
||||
Extend the traversal state to track JSON container depth and produced-node
|
||||
work. Enforce the 100-level and 100,000-node limits before allocation or
|
||||
descent would cross them. Continue using active-path identity for cycle
|
||||
detection; do not use alias memoization that would make distinct JSON paths
|
||||
share mutable output. Errors must identify the structural path and whether the
|
||||
depth or work budget was exceeded.
|
||||
|
||||
Expand the focused package tables by behavior branch: signed and unsigned
|
||||
integer widths, ordinary and named finite floats, `json.Number`, pointers and
|
||||
interfaces, named maps/slices/arrays, nil and empty values, mixed nested trees,
|
||||
arrays, mutation isolation, active cycles, alternating just-below/at/over
|
||||
depth, and shared acyclic subgraphs just below and over the work budget. Tests
|
||||
must derive their edges from package constants or relationships instead of
|
||||
copying unexplained literals.
|
||||
|
||||
Retain only representative public/backend/profile/prepared integration cases
|
||||
that prove error translation and ownership. Update public GoDoc only if it
|
||||
currently states the narrower safe-integer behavior; otherwise the existing
|
||||
finite JSON-compatible-number contract remains canonical. Update the relevant
|
||||
public value GoDoc and format/internal documentation to state that excessively
|
||||
deep or large JSON-shaped values are rejected for safety; keep the exact
|
||||
numeric limits owned by the internal constants rather than duplicating them
|
||||
throughout consumer documentation. Run focused package and caller tests,
|
||||
focused race tests, repository tests, and vet.
|
||||
|
||||
## Stage 4: Consolidate Stable Public JSON And Remove Dormant Internal JSON
|
||||
|
||||
**Findings:** S02-F01, S02-F05, S05-F05.
|
||||
|
||||
Refactor `json.go` so each public value has one ordinary field mapping. Use
|
||||
private aliases or embedded wire representations for ordinary fields and keep
|
||||
only timestamp, millisecond-duration, and intentional omission exceptions
|
||||
explicit. Preserve every existing JSON name and omission rule.
|
||||
|
||||
Before converting `duration_ms`, reject values outside the millisecond range
|
||||
that can be multiplied by `time.Millisecond` without overflow. Derive both
|
||||
edges from `time.Duration` bounds. Return a contextual decode error and do not
|
||||
partially update the receiver on failure.
|
||||
|
||||
Add fully populated `PreparedRun` and `RunResult` contract cases. Verify all
|
||||
ordinary fields, intentional omissions, zero and nonzero timing, complete
|
||||
round trips, the largest safe positive and negative millisecond values, and
|
||||
their first unsafe neighbors.
|
||||
|
||||
Remove unused JSON tags and serialization tests from
|
||||
`internal/domain.PreparedRun` after confirming production never marshals that
|
||||
type. Keep credential absence protected at preparation/clone producers and
|
||||
move any useful cache-control JSON assertion to the public `PreparedRun`
|
||||
contract. Do not retain a parallel internal wire format.
|
||||
|
||||
Run focused domain and root JSON tests, repository tests, and vet.
|
||||
|
||||
## Stage 5: Harden Public Ownership And Diagnostic Contracts
|
||||
|
||||
**Findings:** S01-F01, S02-F03, S02-F04, S13-F01.
|
||||
|
||||
Extend the existing run-request formatting test with distinct input URI,
|
||||
input-body, variable, and API-key sentinels. Require their absence from
|
||||
`String`, `GoString`, `%v`, `%+v`, and `%#v` while retaining positive structural
|
||||
summary assertions.
|
||||
|
||||
Add one focused public-LLM-adapter ownership test. Have the injected client
|
||||
mutate and retain prompt messages, cache-control pointers, nested target extra
|
||||
parameters, and structured-output schema values; prove the domain/prepared
|
||||
source remains unchanged and later details or execution cannot race with those
|
||||
mutations.
|
||||
|
||||
Add one direct all-field mapping test for `OpenAICompatibleProfile`. Populate
|
||||
every field distinctly and compare the complete returned `Profile`. Keep only
|
||||
the existing higher-level cases that prove normal validation and nested-value
|
||||
ownership.
|
||||
|
||||
Make copied `PreparedExecution` values format opaquely by using value-receiver
|
||||
formatting behavior shared by non-nil pointers and values. A nil pointer may
|
||||
use Go's normal `<nil>` formatting, but formatting must never panic or expose
|
||||
internal types, field names, addresses, credentials, or content. Cover original
|
||||
pointers, copied values, zero values, and nil pointers under string, Go-string,
|
||||
and ordinary fmt verbs, and prove formatting does not claim or discard a
|
||||
handle.
|
||||
|
||||
Run focused root tests and the affected prepared/adapter race tests, followed
|
||||
by repository tests and vet.
|
||||
|
||||
## Stage 6: Correct Engine Construction Edges And Immutable Defaults
|
||||
|
||||
**Findings:** S03-F01, S03-F02, S06-F01.
|
||||
|
||||
Change the shared single-file option helper so trimming is used only for the
|
||||
blank-input check. Perform `Stat`, path decomposition, storage, diagnostics,
|
||||
and later access with the exact caller path for prompt, profile, and schema
|
||||
files. Add one compact table covering existing leading- and trailing-whitespace
|
||||
names through all three options.
|
||||
|
||||
Strengthen engine construction tests with three discriminating cases:
|
||||
|
||||
- reverse the argument order of in-memory, ordinary, fallback, and built-in
|
||||
profile categories while retaining fixed category precedence;
|
||||
- collide `Config.ProfileDir` with an ordinary profile option and prove the
|
||||
option replaces the configuration source; and
|
||||
- place a valid same-category replacement after an invalid option and prove
|
||||
construction still fails at the earlier invalid option.
|
||||
|
||||
Convert `internal/defaults.LLMRequestTimeoutDefault` from a variable to a
|
||||
constant without changing its value or adding a setter. Do not add a test that
|
||||
mutates or pins a noncontractual default; existing client deadline behavior is
|
||||
the verification owner.
|
||||
|
||||
Run focused engine construction, default-client construction, and race tests,
|
||||
then repository tests and vet.
|
||||
|
||||
## Stage 7: Contain And Preserve Prompt Content Paths
|
||||
|
||||
**Finding:** S07-F01.
|
||||
|
||||
Refactor prompt content resolution so both repository forms receive an
|
||||
explicit source-root abstraction. Enforce the path decision fixed by this plan
|
||||
before any content read. Use exact parsed path text after a separate blank
|
||||
check. For OS sources, canonicalize the root and resolved target sufficiently
|
||||
to reject symlink escape; for injected `fs.FS`, use its clean relative path
|
||||
namespace. A parent component that remains inside the root is valid. Absolute,
|
||||
escaping, and symlink-escaping targets are invalid.
|
||||
|
||||
Apply the same behavioral table to an OS directory, `WithPromptFS`, and a
|
||||
single-file source: ordinary sibling, nested parent still within root, parent
|
||||
escape, absolute path, symlink escape where supported, and existing names with
|
||||
leading or trailing whitespace. Prove rejected targets cause no outside read
|
||||
and public operations preserve `ErrPromptLoad`.
|
||||
|
||||
Update the framework format reference and internal source document to make the
|
||||
single-file root and absolute-path rule explicit. Run focused prompt-definition
|
||||
and public source tests, including race tests, then repository tests and vet.
|
||||
|
||||
## Stage 8: Correct Prompt Selection, Strictness, Coverage, And Lookup Cost
|
||||
|
||||
**Findings:** S07-F02, S07-F03, S07-F04, S07-F06.
|
||||
|
||||
Correct both existing prompt repository paths before consolidating them in the
|
||||
next stage:
|
||||
|
||||
- Recover selector metadata from YAML `id` and `version`; never use a filename
|
||||
stem as an identity.
|
||||
- Apply normalized ID and requested-version selection before semantic
|
||||
normalization or `content_file` reads.
|
||||
- Associate strict YAML, semantic, and content errors only with a reliably
|
||||
matching selected definition. An unidentifiable malformed file is unrelated
|
||||
to point lookup; a reliably selected malformed file remains authoritative.
|
||||
- Require exactly one YAML document. Comments and trailing whitespace are
|
||||
allowed; a second empty or populated document and malformed trailing YAML
|
||||
are `ErrInvalidYAML`.
|
||||
- Continue scanning the YAML metadata required for duplicate detection, but
|
||||
open content only for selected candidates. A selected content file is opened
|
||||
once; unrelated and different-version bodies are never opened.
|
||||
|
||||
Add paired OS and `fs.FS` regressions for same-stem/different-ID malformed
|
||||
files, same-ID/different-version invalid files, selected malformed definitions,
|
||||
additional YAML documents, duplicates, and counting filesystem behavior.
|
||||
Add a compact normalization table for the previously uncovered missing
|
||||
version, blank input name, blank message role, invalid output format, negative
|
||||
repair attempts, and explicit blank default profile. Output-contract rows
|
||||
should exercise the shared Stage 2 owner rather than recreate its full table.
|
||||
|
||||
Run focused prompt-definition, use-case inspection, and root source tests,
|
||||
focused race tests, repository tests, and vet.
|
||||
|
||||
## Stage 9: Unify Prompt Repository Semantics
|
||||
|
||||
**Finding:** S07-F05.
|
||||
|
||||
After Stage 8 establishes correct behavior in both paths, replace their
|
||||
duplicated discovery-to-selection algorithms with one source-neutral prompt
|
||||
selection and normalization flow. Introduce only the small internal source
|
||||
adapter needed for YAML discovery, bytes, exact content opening, display paths,
|
||||
and root containment. Keep genuine OS and `fs.FS` mechanics at the adapter
|
||||
edge.
|
||||
|
||||
Move exact selection, version filtering, strict one-document decoding,
|
||||
selected-error classification, normalization, duplicate handling, and
|
||||
not-found behavior into the shared flow. Preserve point-in-time source access;
|
||||
do not cache catalogs or definitions across operations.
|
||||
|
||||
Turn the Stage 8 behavior matrix into a shared suite over both adapters and
|
||||
retain source-specific tests only for distinct path and I/O failures. Delete
|
||||
superseded duplicate helpers and tests only after the shared suite protects
|
||||
their meaningful behavior. Use a counting filesystem and a before/after
|
||||
benchmark over small and large prompt catalogs to confirm unrelated content is
|
||||
not read and the refactor adds no second scan; do not enforce wall-clock
|
||||
thresholds.
|
||||
|
||||
Update the internal source document to describe the unified semantic owner.
|
||||
Run focused package, integration, race, repository test, and vet gates.
|
||||
|
||||
## Stage 10: Correct Profile Source Validation And Identity
|
||||
|
||||
**Findings:** S08-F02, S08-F03, S08-F04, S08-F05. The non-finite scalar
|
||||
symptom S08-F01 is already resolved by Stage 1.
|
||||
|
||||
Make file profile normalization pass `extra_params` through
|
||||
`internal/jsonvalue.CopyMap` before publishing a domain profile. Preserve
|
||||
`ErrInvalidProfile` and source path context for empty keys, non-finite values,
|
||||
nested invalid data, or traversal-budget failures. Do not move reserved
|
||||
OpenAI-compatible field policy into the profile package.
|
||||
|
||||
Use YAML metadata ID as the only selector; never infer authority from a
|
||||
filename. Normalize the decoded ID once according to this plan. Require exactly
|
||||
one YAML document in both metadata and strict selected decoding, so trailing
|
||||
raw credentials, unknown fields, empty documents, and malformed YAML cannot be
|
||||
ignored. Reliably selected malformed definitions must stop overlay fallback;
|
||||
unrelated malformed files must not.
|
||||
|
||||
Add shared OS and `fs.FS` tables for invalid/valid extra parameters,
|
||||
same-stem/different-ID malformed files beside a valid profile, fallback
|
||||
behavior, additional documents, leading/trailing/blank IDs, normalized
|
||||
duplicates, and exact inspection/preparation of the normalized ID. Retain only
|
||||
representative public error-translation cases.
|
||||
|
||||
Update the framework format and internal source documents if needed to state
|
||||
ID normalization and one-document behavior. Run focused profile, use-case,
|
||||
root, and race tests, followed by repository tests and vet.
|
||||
|
||||
## Stage 11: Eliminate Duplicate Profile Decoding
|
||||
|
||||
**Finding:** S08-F06.
|
||||
|
||||
Refactor point lookup so each file receives one metadata pass and only
|
||||
canonical ID matches receive strict full decoding and normalization. Reuse
|
||||
bytes already read for metadata; do not decode every unrelated full profile or
|
||||
turn the repository into a cache. Preserve deterministic duplicate detection,
|
||||
strict selected errors, overlay fallthrough only on not-found, and fresh
|
||||
point-in-time reads on every operation.
|
||||
|
||||
Add counting/parser-observation tests where stable behavior can be observed,
|
||||
plus benchmarks for small and large catalogs reporting time and allocations.
|
||||
Exercise valid selection, unrelated malformed files, selected malformed files,
|
||||
duplicates, overlay fallthrough, and repeated lookup. Do not add brittle exact
|
||||
allocation thresholds to ordinary tests.
|
||||
|
||||
Run focused profile and root integration tests, benchmarks for diagnostic
|
||||
comparison, race tests, repository tests, and vet.
|
||||
|
||||
## Stage 12: Correct Artifact Semantics, Cancellation, And Hash Tests
|
||||
|
||||
**Findings:** S09-F01, S09-F02, S09-F05.
|
||||
|
||||
Treat an explicitly typed empty inline reference as a valid zero-byte artifact,
|
||||
including `InlineWithURI`. Keep absence at the input map/reference boundary and
|
||||
compute the same metadata and opaque equality value used for other bodies.
|
||||
|
||||
For ordinary file references, inspect the target before opening and again
|
||||
after opening; reject anything that is not a regular file under the decision
|
||||
above. Replace unbounded `io.ReadAll` with a normal synchronous chunked read
|
||||
that checks `ctx.Err()` before open, before and after each read, and before
|
||||
publishing the artifact. Do not return partial artifacts, add a hidden size
|
||||
limit, or launch an abandoned reader goroutine.
|
||||
|
||||
Add source-parity cases for empty and nonempty inline, inline-with-URI, and file
|
||||
content. Add a platform-appropriate FIFO regression proving the known FIFO is
|
||||
rejected without requiring an external writer, and cancellation cases for a
|
||||
pre-canceled file and a progressing regular-file read. Run them repeatedly and
|
||||
under the race detector.
|
||||
|
||||
Replace exact SHA-256 literals with relational assertions: nonempty and stable
|
||||
for repeat reads, equal for equal inline/file bodies, unequal for changed
|
||||
bodies, and propagated opaquely through preparation. Do not document or test a
|
||||
specific algorithm.
|
||||
|
||||
Update public GoDoc and the internal source document to describe regular-file
|
||||
support and cancellation checkpoints. Run focused package, use-case, root,
|
||||
race, repository test, and vet gates.
|
||||
|
||||
## Stage 13: Make Rendering Cancellation-Aware And Reuse Artifact Text
|
||||
|
||||
**Findings:** S09-F03, S09-F04.
|
||||
|
||||
Check context before session work, before and after each template parse and
|
||||
execution, before and after every message, and before returning the completed
|
||||
prompt. Make the `input` helper return an error when cancellation is observed.
|
||||
Do not run template execution in a detached goroutine.
|
||||
|
||||
Within one `Render` call, lazily convert each named artifact body to text once
|
||||
and memoize that string for the session and all messages. Build the cached
|
||||
string in 64 KiB chunks with one pre-grown `strings.Builder`, checking the
|
||||
context between chunks. Preserve bytes exactly, including invalid UTF-8; do not
|
||||
cache across render calls or mutate artifacts. Unknown and nil inputs retain
|
||||
their current errors, and a canceled conversion must not publish or cache a
|
||||
partial string.
|
||||
|
||||
Add deterministic tests for pre-cancellation, cancellation during the chunked
|
||||
input conversion, and cancellation observed after final-message execution.
|
||||
Require the context identity and no partial prompt while preserving active-
|
||||
context template errors. Add benchmarks for one and repeated references across
|
||||
session and messages; report allocations without hard-coded timing limits.
|
||||
|
||||
Run focused renderer/use-case/root tests, benchmarks, repeated race tests,
|
||||
repository tests, and vet.
|
||||
|
||||
## Stage 14: Preserve Exact JSON Validation Semantics
|
||||
|
||||
**Findings:** S10-F01, S10-F05.
|
||||
|
||||
Create one helper for decoding exactly one JSON value with
|
||||
`json.Decoder.UseNumber` and required EOF after trailing whitespace. Use it for
|
||||
schema documents and JSON Schema instance values so large integers, precise
|
||||
decimals, and exponents retain exact `json.Number` semantics through
|
||||
compilation, prepared metadata, copying, and validation.
|
||||
|
||||
For plain `ValidationJSON`, use a non-materializing complete-document syntax
|
||||
check such as `json.Valid`; do not build a generic tree. Preserve the current
|
||||
result distinction: malformed generated JSON is a completed failed validation,
|
||||
not an operational error, and original output bytes remain unchanged.
|
||||
|
||||
Add focused OS and `fs.FS` cases around `2^53`, `1e400`, precise decimals,
|
||||
ordinary numbers, malformed syntax, and trailing values. Exercise schema
|
||||
`const`, minimum/maximum, and `multipleOf`, and verify the public structured
|
||||
schema retains exact numeric values. Add benchmarks for scalar, object, and
|
||||
large-array JSON validation with allocation reporting but no wall-clock
|
||||
contract.
|
||||
|
||||
Update format/internal validation documentation only where it currently
|
||||
implies float64-limited semantics. Run focused validator/use-case/root tests,
|
||||
benchmarks, race tests, repository tests, and vet.
|
||||
|
||||
## Stage 15: Escape Schema Resources And Compile Once Per Operation
|
||||
|
||||
**Findings:** S10-F02, S10-F03.
|
||||
|
||||
Represent schema compiler resources with `url.URL` rather than string
|
||||
concatenation. Use canonical escaped file URLs for OS paths and a private
|
||||
scheme URL whose path segments are escaped for `fs.FS`. Preserve separators,
|
||||
decode resource paths exactly once at the loader boundary, and continue
|
||||
rejecting remote and escaping references. Legal filenames containing percent,
|
||||
space, `#`, `?`, or Unicode must compile, including contained relative
|
||||
references.
|
||||
|
||||
Unify JSON Schema preparation around `validate.PreparedValidation`:
|
||||
|
||||
- the shared preparation pipeline must create one operation-local compiled
|
||||
plan and derive provider-facing root schema metadata from that plan;
|
||||
- `Prepare` may discard the plan after returning metadata;
|
||||
- `Run` must retain and use the plan for its one operation so the schema graph
|
||||
is not loaded or compiled again during validation; and
|
||||
- `PrepareExecution` must retain the same plan in its frozen payload.
|
||||
|
||||
Remove the document-only `SchemaDocumentLoader` capability if it has no
|
||||
remaining production caller. Do not add an engine-wide or cross-operation
|
||||
schema cache. Keep a clear private preparation carrier in `internal/usecase`
|
||||
if needed so public `domain.PreparedRun` remains free of validator interfaces.
|
||||
|
||||
Replace the existing legal-filename expected failure with valid behavior and
|
||||
retain a genuine compiler-registration failure only if reachable through a
|
||||
valid source. Add public parity tests for invalid keywords, malformed and
|
||||
missing direct/second-level references, unsupported dialects, escapes, remote
|
||||
references, and valid multi-document graphs. A counting source must show each
|
||||
document read once per operation and fresh reads across separate operations.
|
||||
|
||||
Update internal source, validator, and runner documentation for the unified
|
||||
plan lifetime. Run focused validator/use-case/root tests, race tests,
|
||||
repository tests, and vet.
|
||||
|
||||
## Stage 16: Make Validation Cancellation Authoritative
|
||||
|
||||
**Finding:** S10-F04.
|
||||
|
||||
Apply the cancellation decision fixed above. Thread context through schema
|
||||
resource loaders and all Promptkit-controlled read/decode helpers. Read opened
|
||||
schema files in context-checked chunks. Check the context immediately before
|
||||
and after JSON decoding, schema compilation, and schema execution; if
|
||||
cancellation occurred during a synchronous dependency call, return the context
|
||||
error instead of a schema or successful validation result. Do not publish a
|
||||
partial plan or validation result.
|
||||
|
||||
Do not place arbitrary `fs.FS` calls or JSON Schema work in goroutines merely
|
||||
to race them against `ctx.Done()`. Tests must therefore distinguish:
|
||||
|
||||
- prompt cancellation before work;
|
||||
- cancellation between controlled read chunks;
|
||||
- cancellation that becomes authoritative immediately after a synchronous
|
||||
compile or validation call returns; and
|
||||
- the documented limitation that Promptkit cannot preempt a dependency method
|
||||
that never returns.
|
||||
|
||||
Use deterministic controlled readers/contexts rather than sleeps. Assert no
|
||||
goroutine growth or leaked work and preserve operational validation and public
|
||||
context identities. Update validator GoDoc and internal source/runner documents
|
||||
to state the synchronous cancellation boundary accurately.
|
||||
|
||||
Run focused cancellation tests normally, repeatedly, and under the race
|
||||
detector, followed by repository tests and vet.
|
||||
|
||||
## Stage 17: Repair And Protect The Retained Internal Repair Path
|
||||
|
||||
**Findings:** S12-F01, S12-F02, S12-F03.
|
||||
|
||||
Retain the internal repair architecture. Extend `RepairRequest` with
|
||||
`ExecutionTargetPresence` and carry the resolved presence bits unchanged into
|
||||
the default repairer's `GenerateRequest`. Factor one use-case-local constructor
|
||||
for common initial/repair generation fields—effective target, presence,
|
||||
credential, backend identity, session, and structured output—while keeping the
|
||||
initial and repair prompts intentionally separate.
|
||||
|
||||
Accumulate every completed generation response's five token-usage fields into
|
||||
run-level usage. The final content/raw output/artifact continues to come from
|
||||
the last candidate, while usage includes initial generation and every completed
|
||||
repair exactly once. A repair call that returns an error still returns no
|
||||
partial public result under current error semantics.
|
||||
|
||||
Replace the one-attempt-only repair coverage with a compact state-machine
|
||||
table for:
|
||||
|
||||
- initial success with no repair;
|
||||
- ineligible basic validation despite a positive budget;
|
||||
- explicit zero and inherited-zero presence across initial and repair calls;
|
||||
- success before a larger budget is exhausted;
|
||||
- exact exhaustion of a larger budget; and
|
||||
- advancement of attempt number, maximum, prior output, diagnostics, final
|
||||
status, cumulative usage, and collaborator call count.
|
||||
|
||||
Retain the distinct capacity integration proving initial and repair generation
|
||||
use the same backend pool and one whole-run admission lease, but simplify it if
|
||||
the new focused table makes repair-state assertions redundant.
|
||||
|
||||
Update the internal runner and capacity documents for presence fidelity and
|
||||
cumulative usage. Do not alter `NewRunner` to install a repairer, public GoDoc
|
||||
that says the engine is single-pass, or the future public repair roadmap.
|
||||
|
||||
Run focused use-case, prepared, capacity, and race tests, followed by repository
|
||||
tests and vet.
|
||||
|
||||
## Stage 18: Preserve Transport Error Identities And Test Deterministically
|
||||
|
||||
**Findings:** S14-F01, S04-F01, S16-F02. Timeout overflow S14-F02 is already
|
||||
resolved through Stage 1's shared bound.
|
||||
|
||||
Preserve the underlying `http.Client.Do` error in the chain while retaining
|
||||
`internal/llm.ErrRequestFailed` and the public generation category. Do not add
|
||||
headers, request content, endpoints, or provider bodies to error text.
|
||||
Cancellation and deadline identities must survive caller cancellation, caller
|
||||
deadline, generation deadline, and whole-request client timeout.
|
||||
|
||||
Replace the port-9999 test with a controlled round tripper or `httptest`
|
||||
endpoint that records the selected URL and returns a deliberate result. It must
|
||||
make no host-dependent connection and must separately prove empty configured
|
||||
base acceptance and request-endpoint precedence.
|
||||
|
||||
Extend the existing external ordinary-run cancellation test to require both
|
||||
`ErrLLMGenerate` and `context.Canceled`; retain lower-layer tests only for their
|
||||
distinct error owners.
|
||||
|
||||
Update the OpenAI-compatible integration and internal LLM documents for error
|
||||
identity behavior. Run focused LLM and root tests with repetition and the race
|
||||
detector, followed by repository tests and vet.
|
||||
|
||||
## Stage 19: Validate And Compose Effective Provider Endpoints
|
||||
|
||||
**Finding:** S14-F03.
|
||||
|
||||
Add one source-neutral OpenAI-compatible base-endpoint validator in
|
||||
`internal/domain`, alongside the effective execution target invariant. It must
|
||||
trim surrounding configuration whitespace, require absolute HTTP or HTTPS with
|
||||
a host, and reject user information, query, and fragment. Use it from backend
|
||||
registration, in-memory and file profiles, resolved request overrides, and the
|
||||
built-in client defense while preserving each boundary's existing config,
|
||||
profile-load, invalid-request, or LLM error category.
|
||||
|
||||
An empty configured base URL remains valid for a built-in client because a
|
||||
resolved request endpoint may supply it later. Validate only a nonempty
|
||||
configured base at construction, and always validate the final selected
|
||||
endpoint before transport. Backends and endpoint-only profiles retain their
|
||||
existing nonempty endpoint requirements.
|
||||
|
||||
Compose the completion URL through parsed URL operations (prefer
|
||||
`url.JoinPath`) so nested paths and trailing slashes reach exactly one
|
||||
`/chat/completions` suffix. Never append to a raw string.
|
||||
|
||||
Add endpoint tables for HTTP and HTTPS, hosts, nested paths, repeated trailing
|
||||
slashes, queries, fragments, user information, relative paths, missing hosts,
|
||||
unsupported schemes, and request/profile/config error mapping. Require every
|
||||
invalid selected endpoint to fail before transport.
|
||||
|
||||
Update the architecture/internal overview for domain endpoint invariants and
|
||||
the OpenAI-compatible integration and internal LLM documents for URL behavior.
|
||||
Run focused domain/backend/profile/use-case/LLM/root tests, repetition and race
|
||||
tests where ownership crosses packages, followed by repository tests and vet.
|
||||
|
||||
## Stage 20: Bound And Strictly Frame Successful Provider Responses
|
||||
|
||||
**Findings:** S14-F04, S14-F05.
|
||||
|
||||
Enforce the 16 MiB successful-response decision without first copying the
|
||||
entire body. Reject an over-limit `Content-Length` immediately, but also wrap
|
||||
the body in a counting/limited reader that reads at most one byte beyond the
|
||||
limit so chunked or dishonest responses cannot bypass it. Exactly-limit bodies
|
||||
remain valid. Always close the body; do not drain an unbounded oversized
|
||||
stream.
|
||||
|
||||
Decode exactly one response object. After the first decode, require only
|
||||
trailing JSON whitespace and EOF. A second value, non-whitespace suffix,
|
||||
truncated body, malformed JSON, or size overflow returns
|
||||
`ErrMalformedResponse` with no partial response and no provider content in the
|
||||
error.
|
||||
|
||||
Add streaming tests just below, at, and one byte over the limit with and
|
||||
without `Content-Length`, plus a continuing oversized stream. Assert bounded
|
||||
bytes read, timely return, no partial result, and closure. Add trailing
|
||||
whitespace success and trailing garbage/second-value failures. Retain ordinary
|
||||
response mapping and redaction cases.
|
||||
|
||||
Document the fixed successful-response boundary and strict one-document rule
|
||||
in the integration and internal LLM documents. Explicitly leave bounded
|
||||
non-success error-envelope parsing to the structured-generation-error roadmap.
|
||||
Run focused transport tests normally, repeatedly, and under race, followed by
|
||||
repository tests and vet.
|
||||
|
||||
## Stage 21: Consolidate Transport Test Scaffolding
|
||||
|
||||
**Finding:** S14-F06.
|
||||
|
||||
After transport behavior is stable, introduce one small recording-provider
|
||||
fixture for common request capture and successful/error response setup.
|
||||
Organize focused tables around request mapping, authentication, endpoint
|
||||
composition, timeout/error identity, and response framing. Keep specialized
|
||||
round trippers/readers for cancellation, deadlines, byte counts, continuing
|
||||
streams, and body closure.
|
||||
|
||||
Retain every existing durable assertion for method, path, headers,
|
||||
authentication, omission and explicit presence, reserved fields, cache
|
||||
control, structured output, response mapping, usage, error redaction, and
|
||||
timeout precedence. Retain all Stage 18 through 20 regressions. Delete repeated
|
||||
servers, generic-map decoding, and response literals only where the fixture
|
||||
makes the owning behavior clearer; do not replace wire assertions with a broad
|
||||
snapshot.
|
||||
|
||||
Run the LLM suite normally, with shuffle/repetition, and under the race
|
||||
detector. Deliberately inspect the resulting test inventory against the audit's
|
||||
transport matrix before running repository tests and vet.
|
||||
|
||||
## Stage 22: Restore One Canonical Maintainer Validation Workflow
|
||||
|
||||
**Finding:** S16-F01.
|
||||
|
||||
Make `docs/development.md` the canonical owner of the complete local maintainer
|
||||
workflow, as assigned by the documentation policy. Its validation section must
|
||||
include, from the repository root:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go vet ./...
|
||||
go build ./...
|
||||
go run ./examples/go-library/prepare
|
||||
go run ./examples/go-library/run
|
||||
```
|
||||
|
||||
It must also own the Go formatting, local Markdown link, `git diff --check`,
|
||||
workspace/vendor/replacement, generated-output, credential, and working-tree
|
||||
hygiene checks used before accepting changes.
|
||||
|
||||
Change the testing policy to state the semantic requirements and link to that
|
||||
canonical workflow instead of maintaining a partial competing command list.
|
||||
Change the release procedure to invoke the development-guide validation as a
|
||||
release prerequisite rather than presenting a separately maintained copy;
|
||||
retain release-specific metadata, candidate, tag, and publication commands in
|
||||
the release document.
|
||||
|
||||
Run both examples offline and confirm that a missing or invalid Run example
|
||||
fixture makes its command fail. Validate all changed Markdown links and ensure
|
||||
current-state documentation describes only the implemented workflow.
|
||||
|
||||
## Stage 23: Complete Traceability And Final Validation
|
||||
|
||||
This final stage introduces no new behavior. Review the final tree against the
|
||||
finding-to-stage table below and the evidence in `audit.md`. Confirm every
|
||||
canonical group is implemented and every source-specific symptom retains its
|
||||
required regression and error boundary. Do not mark a finding resolved merely
|
||||
because a nearby refactor landed.
|
||||
|
||||
Run the complete development-guide workflow, including both examples, all
|
||||
formatting and link checks, and repository hygiene. Also run shuffled ordinary
|
||||
tests and repeated race-enabled tests for the changed concurrency,
|
||||
cancellation, prepared, validation, repair, and transport packages. Run the
|
||||
accepted performance benchmarks for prompt lookup, profile lookup, rendering,
|
||||
and JSON validation and record only qualitative before/after conclusions; do
|
||||
not establish release timing promises.
|
||||
|
||||
Inspect canonical GoDoc, formats, integration, architecture, and internal
|
||||
documents against the final implementation. Confirm the public engine still
|
||||
performs no output repair and the future repair entry remains future work.
|
||||
Confirm the structured-generation-error feature was not implemented as part of
|
||||
transport remediation.
|
||||
|
||||
Leave `audit-sequence.md`, `audit.md`, and this plan in place for maintainer
|
||||
review. Retire them only in a separately authorized roadmap-cleanup pass after
|
||||
the remediation has been reviewed and accepted.
|
||||
|
||||
## Finding-To-Stage Traceability
|
||||
|
||||
| Stage | Canonical findings | Historical or source-specific records handled with the canonical owner |
|
||||
| ---: | --- | --- |
|
||||
| 1 | S05-F01, S17-F01, S14-F02 | S02-F02, S08-F01, S11-F01 |
|
||||
| 2 | S17-F02 | S11-F02 |
|
||||
| 3 | S05-F02, S05-F03, S05-F04 | None |
|
||||
| 4 | S02-F01, S02-F05, S05-F05 | None |
|
||||
| 5 | S01-F01, S02-F03, S02-F04, S13-F01 | None |
|
||||
| 6 | S03-F01, S03-F02, S06-F01 | None |
|
||||
| 7 | S07-F01 | None |
|
||||
| 8 | S07-F02, S07-F03, S07-F04, S07-F06 | None |
|
||||
| 9 | S07-F05 | None |
|
||||
| 10 | S08-F02, S08-F03, S08-F04, S08-F05 | S08-F01 was handled in Stage 1 |
|
||||
| 11 | S08-F06 | None |
|
||||
| 12 | S09-F01, S09-F02, S09-F05 | None |
|
||||
| 13 | S09-F03, S09-F04 | None |
|
||||
| 14 | S10-F01, S10-F05 | None |
|
||||
| 15 | S10-F02, S10-F03 | None |
|
||||
| 16 | S10-F04 | None |
|
||||
| 17 | S12-F01, S12-F02, S12-F03 | None |
|
||||
| 18 | S14-F01, S04-F01, S16-F02 | S14-F02 was handled in Stage 1 |
|
||||
| 19 | S14-F03 | None |
|
||||
| 20 | S14-F04, S14-F05 | None |
|
||||
| 21 | S14-F06 | None |
|
||||
| 22 | S16-F01 | None |
|
||||
|
||||
The table maps all 49 canonical remediation groups exactly once. S02-F02 is
|
||||
the one superseded historical finding retained as evidence under S17-F01;
|
||||
S08-F01, S11-F01, and S11-F02 retain their source-specific regression
|
||||
responsibilities without being double-counted as canonical groups.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The numeric contract, resource bounds, path and identity rules,
|
||||
validation-cancellation limitation, repair retention, transport response
|
||||
limit, and documentation ownership required to implement these stages are
|
||||
fixed above.
|
||||
Reference in New Issue
Block a user