eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record requirement inspection benchmarks

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-09 00:31:52 +00:00
parent 04d5d695e0
commit ef012ea1e5
1 changed files with 53 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/roadmap/implementation.md
View File

@@ -286,6 +286,10 @@ exercise one frozen body per generation.
## Stage 7 — Establish Inspection Benchmarks
**Status: Complete (2026-10-09).** Captured configuration lookup and public
requirement-inspection benchmarks are committed. Repeated benchmark-only
baseline measurements are recorded below; no production behavior changed.
**Outcome:** Owned lookup and assembled requirement-inspection baselines, with
no production change. This separates fixture work from the selection refactor.
@@ -542,6 +546,55 @@ rather than rerunning merely to create the final commit.
## Execution Evidence
### Inspection Baseline
The benchmark-only fixture is committed at `04d5d695e0201a97c1070e1ff49ef4f0c9a7f5ba`
(`Add requirement inspection benchmarks`). The configuration benchmarks build
one captured snapshot per case before timing and measure the existing prompt
snapshot and owned-profile lookups. The public benchmark builds one real Engine
and both requests before timing, then calls only `InspectRequirements` with
the caller variable supplied. None of the paths reads artifacts or schemas or
contacts a provider. The small fixture has two provider-option entries; the
larger fixture has 64 nested provider maps, each with mutable label/route slices
and nested limit/route maps. Prompt lookup allocation is independent of profile
extra size because it does not copy the selected profile. The profile lookup
and assembled requirement inspection expose the mutable-map copy cost being
measured. Environment: Linux/amd64, Intel Core i5-7260U @ 2.20 GHz, benchmark
parallelism 4, Go 1.26.8, ConfigKit `v0.5.0`, and JSON Schema `v6.0.2`.
Command (five 500 ms samples per case, without the race detector):
```sh
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \
go test -mod=readonly -run '^$' \
-bench 'BenchmarkInspectionSelection|BenchmarkRequirementInspection' \
-benchmem -benchtime=500ms -count=5 ./internal/configuration .
```
Times list all five `ns/op` samples in command order, followed by median and
range. Allocation figures are `B/op; allocs/op`.
| Configuration lookup | `ns/op` samples (median; range) | Allocations |
| --- | --- | --- |
| Prompt, small extras | 356.8, 354.2, 370.1, 352.8, 364.5 (356.8; 352.8–370.1) | 400; 4 |
| Prompt, nested extras | 423.2, 478.8, 456.2, 436.6, 460.4 (456.2; 423.2–478.8) | 400; 4 |
| Profile, small extras | 18785, 21077, 18952, 19701, 18682 (18952; 18682–21077) | 9538; 211 |
| Profile, nested extras | 624104, 635513, 600281, 600667, 620228 (620228; 600281–635513) | 261024–261028; 5979 |
| Public requirement inspection | `ns/op` samples (median; range) | Allocations |
| --- | --- | --- |
| Small extras | 23022, 25193, 24687, 24974, 23157 (24687; 23022–25193) | 11922; 240 |
| Nested extras | 635726, 642297, 591992, 605001, 590312 (605001; 590312–642297) | 263370–263372; 6008 |
The prompt snapshot lookup remained at 400 B and four allocations in both
fixtures, while owned profile lookup rose from about 9.5 KB/211 allocations to
about 261 KB/5979 allocations. Public requirement inspection rose from about
11.9 KB/240 allocations to about 263 KB/6008 allocations. These results
establish a baseline in which unrelated nested provider extras dominate the
large profile-selection and requirement-inspection cases; they are local
allocation measurements, not end-to-end provider-latency claims. Fixture
identity and prompt/requirement inventories were checked outside timing.
### Validation And Compiled-Rendering Baseline
The benchmark-only baseline is commit