eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record request preparation baseline

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-09 00:25:39 +00:00
parent 3f157150e4
commit c63ca5352f
1 changed files with 43 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

44
docs/roadmap/implementation.md
View File

@@ -2,7 +2,7 @@
Implement the remaining local work in the [performance roadmap](performance.md)
in order. Each sequential stage is scoped for one Luna coding-agent prompt.
Stages 1–4 are complete; Stages 5–14 remain pending. Update status and evidence
Stages 1–5 are complete; Stages 6–14 remain pending. Update status and evidence
as each work unit lands.
The completed inspection plan is preserved at
`481e2d4:docs/roadmap/implementation.md` in Git history.
@@ -216,6 +216,12 @@ Existing progress-cancellation and nonregular-file tests remain authoritative.
## Stage 5 — Establish Encoding Benchmarks And Error-Boundary Coverage
**Status: Complete (2026-10-09).** `PrepareGeneration` now has an offline
benchmark baseline for no extras, small extras, a 1 MiB string extra, nested
JSON extras, and structured output. Tests protect nested invalid-extra context
and request-error classification when the schema cannot be encoded. No
production behavior changed.
**Outcome:** A repeatable request-preparation baseline and focused protection
for the classifications that Stage 6 must preserve. No production change.
@@ -744,6 +750,42 @@ through EOF. Focused verification passed: `go test -mod=readonly
./internal/artifact`; the existing cancellation and nonregular-file checks
remain in the passing suite.
### Request-Preparation Baseline
The benchmark and boundary-test fixture is committed at `3f15715` (`Add
request preparation baseline and boundary tests`). The benchmark measures
`PrepareGeneration` without dispatch, with client construction, request
fixtures, and the structured schema created before timing. It uses a synthetic
endpoint and no credential or provider calls. Environment: Linux/amd64, Intel
Core i5-7260U @ 2.20 GHz, benchmark parallelism 4, Go 1.26.8, ConfigKit
`v0.5.0`, JSON Schema `v6.0.2`, and backend modules `v3.20260930.0`.
Command (five 500 ms samples per case, without the race detector):
```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \
go test -mod=readonly -run '^$' -bench '^BenchmarkPrepareGeneration$' \
-benchmem -benchtime=500ms -count=5 ./internal/llm
```
Times list all five `ns/op` samples in command order, followed by median and
range. Allocation figures are `B/op; allocs/op`.
| Preparation case | `ns/op` samples (median; range) | Allocations |
| --- | --- | --- |
| No extras | 2771, 2873, 2872, 2907, 2885 (2873; 2771–2907) | 1632; 23 |
| Small extras | 4216, 4255, 4440, 4090, 4245 (4245; 4090–4440) | 1928; 32 |
| 1 MiB string extra | 3108655, 3046197, 3165484, 3115831, 3074278 (3108655; 3046197–3165484) | 2147898–2213942; 26–27 |
| Nested JSON extras | 5933, 5803, 5675, 5490, 5611 (5675; 5490–5933) | 2304; 40 |
| Structured output | 6421, 6018, 6479, 6153, 6712 (6421; 6018–6712) | 2632–2633; 47 |
The baseline covers payload construction and complete JSON encoding for both
ordinary and representative structured requests. The 1 MiB case's higher
allocation volume is expected to include the large string and encoded body;
these measurements are the comparison reference for the following request
encoding change, not a provider-latency estimate. Focused verification passed:
`go test -mod=readonly ./internal/llm`.
## Open Questions
No consumer-policy decisions remain open. File reservation and encoding error