Add report and scope benchmarks

This commit is contained in:
2026-10-09 01:23:35 +00:00
parent aed4c5921c
commit a431540e19
2 changed files with 405 additions and 3 deletions

View File

@@ -1,6 +1,6 @@
# Performance implementation plan
Status: implementation in progress; Stage 1 is complete, Stages 2–11 have not started.
Status: implementation in progress; Stages 1–2 are complete, Stages 3–11 have not started.
Date: 2026-10-09.
@@ -228,7 +228,7 @@ shows why later comparisons should report run variation.
## Stage 2: establish report and scope measurements
**Status:** not started. **Prerequisite:** Stage 1.
**Status:** complete. **Prerequisite:** Stage 1.
**Files:** new `performance_reports_test.go` and evidence here. Production code
stays unchanged in this stage.
@@ -258,7 +258,57 @@ owner of actual library loading semantics.
**Completion:** PF-01, PF-04, and PF-06 have comparable baseline measurements.
**Evidence:** pending.
**Evidence:** Fixtures were measured from source revision `aed4c59` with
`performance_reports_test.go` in the working tree (committed with this evidence).
Environment matched the first baseline: Go 1.26.8, linux/amd64, Linux
6.17.0-41-generic, Intel Core i5-7260U CPU @ 2.20 GHz. Root package compilation
passed with `GOWORK=off GOTOOLCHAIN=local GOPROXY=off GONOPROXY=none GOSUMDB=off go test -mod=readonly -run '^$' .`. The five-sample benchmark command passed:
`GOWORK=off GOTOOLCHAIN=local GOPROXY=off GONOPROXY=none GOSUMDB=off go test -mod=readonly -run '^$' -bench '^BenchmarkPerformance(PromptReports|YAMLFraming|ConfigScopes|ConfigRegistration)$' -benchmem -benchtime=200ms -count=5 .`.
Raw output is in `/tmp/commandkit-performance-reports-baseline.txt`.
Fixtures: PromptKit callbacks returned prebuilt inspection records directly;
prompt reports used 4 and 128 input records, while profile `ExtraParams` used
2 groups with 3 nested values each or 24 groups with 8 nested values each and a
nested sequence. Each was measured as text and YAML through `App.Execute`.
Native YAML fixtures contain unique short mapping keys and are 4,096 bytes with
an ordinary comment preamble and 1,048,591 bytes with a `%YAML` directive
preamble. Framing was measured directly; aggregate `config show --scope all`
measured two documents of each selected fixture size/style, and single-scope
display used the 4 KiB ordinary document. Scope operations used 2 or 32 scopes
with the requested show/validate capabilities and cheap in-memory acquisition
callbacks. Suite registration used 4 or 64 distinct selectors and a fresh app.
The scope counts also covered discovery, named validation, and aggregate
validation. Setup verified success and report shape outside timed loops; native
fixtures, records, suites, and apps were prepared outside repeated loops.
Representative five-sample ranges (ns/op; B/op; allocs/op): prompt, 4 inputs,
text 41,287–52,115; 26,160–26,161; 294, and YAML 88,991–104,671;
74,730–74,733; 557. Prompt, 128 inputs, text 169,155–198,281;
145,524–145,531; 1,171, and YAML 1,418,692–1,490,181; 1,393,508–1,393,568;
6,039–6,040. Profile, small extra map, text 74,109–89,337; 70,914–70,918;
462, and YAML 161,470–368,813; 162,688–162,717; 755. Profile, large extra map,
text 1,277,396–2,583,950; 836,141–836,296; 3,027–3,028, and YAML
1,756,571–2,377,499; 1,586,728–1,586,849; 5,878–5,879. Larger profile timings
varied, while allocation counts and bytes were stable.
Direct framing measured 4 KiB ordinary at 5,883–6,842 ns/op, 11,056 B/op,
3 allocs/op, and 1 MiB directive at 1,208,678–1,502,417 ns/op,
2,539,520–2,539,523 B/op, 2 allocs/op. Aggregate show measured 4 KiB documents
at 44,395–48,161 ns/op, 55,537–55,538 B/op, 211 allocs/op, and 1 MiB documents
at 4,818,907–6,009,877 ns/op, 8,695,182–8,695,197 B/op, 215–216 allocs/op.
Single-scope 4 KiB display measured 20,229–21,791 ns/op, 17,935–17,936 B/op,
183 allocs/op. With 2 scopes, discovery measured 23,840–30,007 ns/op,
21,161–21,162 B/op, 223 allocs/op; named validation 24,841–26,340 ns/op,
21,817 B/op, 228 allocs/op; aggregate validation 25,231–31,126 ns/op,
22,361–22,362 B/op, 237 allocs/op. With 32 scopes, discovery measured
31,439–37,167 ns/op, 28,141–28,142 B/op, 257 allocs/op; named validation
29,123–74,374 ns/op, 22,775–22,778 B/op, 228 allocs/op; aggregate validation
45,548–63,471 ns/op, 29,781–29,782 B/op, 269 allocs/op. Registration with 4
selectors measured 12,573–14,789 ns/op, 8,911 B/op, 104 allocs/op; with 64 it
measured 91,675–105,548 ns/op, 82,817–82,818 B/op, 485 allocs/op. Large-report
cost includes full output framing/delivery; no segmented-delivery comparison
was selected. These measurements establish baselines and show timing variation;
they are not fixed thresholds or speedup claims.
## Stage 3: render only the selected PromptKit format