From ef012ea1e5d88a44148b71c454da00a2e06bb4e6 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 9 Oct 2026 00:31:52 +0000 Subject: [PATCH] Record requirement inspection benchmarks --- docs/roadmap/implementation.md | 53 ++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index a398d0b..eb83b1c 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -286,6 +286,10 @@ exercise one frozen body per generation. ## Stage 7 — Establish Inspection Benchmarks +**Status: Complete (2026-10-09).** Captured configuration lookup and public +requirement-inspection benchmarks are committed. Repeated benchmark-only +baseline measurements are recorded below; no production behavior changed. + **Outcome:** Owned lookup and assembled requirement-inspection baselines, with no production change. This separates fixture work from the selection refactor. @@ -542,6 +546,55 @@ rather than rerunning merely to create the final commit. ## Execution Evidence +### Inspection Baseline + +The benchmark-only fixture is committed at `04d5d695e0201a97c1070e1ff49ef4f0c9a7f5ba` +(`Add requirement inspection benchmarks`). The configuration benchmarks build +one captured snapshot per case before timing and measure the existing prompt +snapshot and owned-profile lookups. The public benchmark builds one real Engine +and both requests before timing, then calls only `InspectRequirements` with +the caller variable supplied. None of the paths reads artifacts or schemas or +contacts a provider. The small fixture has two provider-option entries; the +larger fixture has 64 nested provider maps, each with mutable label/route slices +and nested limit/route maps. Prompt lookup allocation is independent of profile +extra size because it does not copy the selected profile. The profile lookup +and assembled requirement inspection expose the mutable-map copy cost being +measured. Environment: Linux/amd64, Intel Core i5-7260U @ 2.20 GHz, benchmark +parallelism 4, Go 1.26.8, ConfigKit `v0.5.0`, and JSON Schema `v6.0.2`. + +Command (five 500 ms samples per case, without the race detector): + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GONOPROXY=none GOSUMDB=off \ + go test -mod=readonly -run '^$' \ + -bench 'BenchmarkInspectionSelection|BenchmarkRequirementInspection' \ + -benchmem -benchtime=500ms -count=5 ./internal/configuration . +``` + +Times list all five `ns/op` samples in command order, followed by median and +range. Allocation figures are `B/op; allocs/op`. + +| Configuration lookup | `ns/op` samples (median; range) | Allocations | +| --- | --- | --- | +| Prompt, small extras | 356.8, 354.2, 370.1, 352.8, 364.5 (356.8; 352.8–370.1) | 400; 4 | +| Prompt, nested extras | 423.2, 478.8, 456.2, 436.6, 460.4 (456.2; 423.2–478.8) | 400; 4 | +| Profile, small extras | 18785, 21077, 18952, 19701, 18682 (18952; 18682–21077) | 9538; 211 | +| Profile, nested extras | 624104, 635513, 600281, 600667, 620228 (620228; 600281–635513) | 261024–261028; 5979 | + +| Public requirement inspection | `ns/op` samples (median; range) | Allocations | +| --- | --- | --- | +| Small extras | 23022, 25193, 24687, 24974, 23157 (24687; 23022–25193) | 11922; 240 | +| Nested extras | 635726, 642297, 591992, 605001, 590312 (605001; 590312–642297) | 263370–263372; 6008 | + +The prompt snapshot lookup remained at 400 B and four allocations in both +fixtures, while owned profile lookup rose from about 9.5 KB/211 allocations to +about 261 KB/5979 allocations. Public requirement inspection rose from about +11.9 KB/240 allocations to about 263 KB/6008 allocations. These results +establish a baseline in which unrelated nested provider extras dominate the +large profile-selection and requirement-inspection cases; they are local +allocation measurements, not end-to-end provider-latency claims. Fixture +identity and prompt/requirement inventories were checked outside timing. + ### Validation And Compiled-Rendering Baseline The benchmark-only baseline is commit