ConfigKit/docs/roadmap/startup-benchmark.md

Startup construction baseline and first optimization proposal

Status: Stages 1–3 complete; the first Stage 4 experiment is retained. Further optimization and complete consumer-workload verification remain outstanding.

Measured on 2026-10-09 under the startup roadmap. The consumer request owns the submitted workload and measurements. This document records the reproduced baselines, maintained benchmark, profiles, and proposed first change. It introduces no performance contract or release gate.

Environment and revisions

• PromptKit checkout: eb830bb61346672f0f6c5bce6119b76ccc6d61ac for both downstream runs. The adapter and benchmark are unchanged from its earlier review revision, de6443f3aef4a66169e323e9f9a5f41594fbface.
• Submitted ConfigKit dependency: v0.5.0, source commit dc09959cca4e105f49ec316c6d4ba585bd2340bd.
• Current ConfigKit baseline: f1722274c02c08ff28556dd66f29e0da07a9f2c3. The benchmark added with this report changes no production implementation.
• Go: /snap/bin/go, go1.26.8, linux/amd64.
• CPU: Intel Core i5-7260U at 2.20 GHz; two physical cores, four logical CPUs.
• GOMAXPROCS=4, GOTOOLCHAIN=local, GOWORK=off, GOPROXY=off, and readonly module mode. GOGC and GOMEMLIMIT were unset (runtime defaults).
• Dependencies were already cached. No consumer repository's committed dependencies were changed, and no adoption was published. Separate temporary clones excluded consumer worktree edits; the current-code comparison used a temporary module replacement.
• Logs, test binaries, and profiles are outside both repositories under /tmp/configkit-startup.L9vnUo. These are disposable evidence, not repository dependencies; the commands below regenerate them.

The processor and toolchain match the submitted environment. The machine was shared, with no exclusive CPU allocation or frequency control. An initial cold downstream pass overlapped discovery work and ranged from 90.95 to 256.19 ms; it was excluded from the warmed comparison below. Subsequent local runs also showed substantial timing variation. All final-run observations, including outliers, are recorded below. Allocation counts were much steadier than elapsed time. These measurements do not establish statistical timing significance.

Stage 1: Downstream reproduction

PromptKit's BenchmarkConfigurationConstruction/independent prepares 128 profiles and 128 prompt versions outside timing, including the captured configkit.Mapping source. Each profile supplies an endpoint and model. Each prompt has version key 1, one user message, a profile reference, and text output with validation disabled. There are 256 definitions, one tier, and no files, imports, inheritance, credentials, or inference calls.

Each timed iteration invokes the real adapter's load function. Within LoadLayered, it captures optional-field presence with Resolution.InputInfo, normalizes profile values, performs domain validation, and supplies a path-based secret classifier. The supplied endpoints are already canonical, so this fixture records no normalization changes. After ConfigKit returns, the adapter materializes prompts, compiles templates, computes content hashes, and builds execution profiles. Those operations are included in the downstream timing.

The comparison used an isolated PromptKit checkout at the revision above. First it retained the declared v0.5.0 dependency. Then its temporary go.mod received a replacement pointing at an isolated ConfigKit checkout at the current baseline. Both used:

GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
  go test -mod=readonly -run '^$' \
  -bench '^BenchmarkConfigurationConstruction/independent$' \
  -benchmem -benchtime=500ms -count=5 ./internal/configuration

To reproduce the temporary wiring, clone with --no-hardlinks --local, check out the recorded revisions, and run this only in the disposable PromptKit copy:

go mod edit -replace=gitea.maximumdirect.net/eric/ConfigKit=/absolute/path/to/configkit-copy

ConfigKit	Run	ms/construction	B/construction	allocations/construction
v0.5.0	1	74.339	50,155,869	538,590
v0.5.0	2	69.340	50,159,666	538,541
v0.5.0	3	71.453	50,131,114	538,542
v0.5.0	4	70.164	50,113,244	538,541
v0.5.0	5	72.495	50,165,238	538,540
current	1	73.567	50,147,688	538,593
current	2	70.959	50,152,860	538,540
current	3	67.509	50,167,353	538,541
current	4	71.491	50,181,043	538,543
current	5	68.786	50,131,684	538,541

The per-metric medians are 71.453 ms / 50,155,869 B / 538,541 allocations for v0.5.0 and 70.959 ms / 50,152,860 B / 538,541 allocations for current code. The submitted timing median was approximately 67.740 ms; its approximately 50.16 MB and 538,542 allocations are reproduced. The modest warmed timing difference and the cold-run outlier are consistent with machine/runtime variation; they do not establish a code regression. The production loading code is identical between these ConfigKit revisions: their Go-file differences are release/development tests. No startup improvement is claimed.

Stage 2: Maintained public-API benchmark

startup_bench_test.go adds BenchmarkStartupConstruction in the external configkit_test package. It measures LoadLayered through the supported root facade at 32, 128, and 512 profile/prompt pairs (64, 256, and 1,024 definitions).

The input has two root collections. Profiles contain endpoint and model scalars; prompts contain a numeric mapping key 1, a profile reference, a message sequence with role/content, and an output mapping with format/validation mode. The source tree has 13 * pairs + 3 nodes (419, 1,667, and 6,659 respectively), before typed projection adds omitted-field metadata. Typed profiles have twelve optional fields; typed prompts retain optional metadata, inputs, messages, and output fields. This records configuration shape as well as definition count.

Fixture generation, Mapping capture, and one successful lifecycle/metadata check are outside timing. Timed work includes typed decoding, each callback, resolution auditing, validation, final publication, and retention of the latest result. The resolver captures all twelve profile field states and parses their canonical endpoints. Validation visits sorted profile IDs. A simple path-based secret predicate is installed, although the fixture contains no credentials. The preflight checks callback execution, collection sizes, explicit versus omitted profile fields, and metadata for the message reached through mapping key 1 and sequence index 0.

This is representative rather than an exact copy of the consumer adapter: it omits the unused backend collection and message cache-control type, simplifies domain validation/classification, and does not compile templates, compute hashes, or build runtime registries. Its lower allocation total must not be treated as an exact subtraction of PromptKit overhead. It adds no imports of PromptKit and runs offline without credentials. Its measurements are not pass/fail thresholds.

Run from ConfigKit:

GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
  go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction$' \
  -benchmem -benchtime=500ms -count=5

Pairs	Run	ms/construction	B/construction	allocations/construction
32	1	18.327	11,999,640	129,136
32	2	18.418	11,999,423	129,136
32	3	50.366	11,999,534	129,137
32	4	55.028	11,999,548	129,137
32	5	25.824	11,999,468	129,136
128	1	67.610	47,408,046	513,722
128	2	65.090	47,451,718	513,724
128	3	65.682	47,584,876	513,727
128	4	62.574	47,324,919	513,722
128	5	123.565	47,273,050	513,721
512	1	276.663	188,313,648	2,051,961
512	2	284.979	189,649,296	2,052,000
512	3	497.075	189,234,128	2,051,985
512	4	257.534	189,084,032	2,051,981
512	5	265.167	188,679,712	2,051,973

Per-metric medians:

Pairs	ms	B	Allocations
32	25.824	11,999,534	129,136
128	65.682	47,408,046	513,722
512	276.663	189,084,032	2,051,981

Increasing the catalog fourfold multiplies allocation counts by approximately 3.98 and 3.99, and allocated bytes by approximately 3.95 and 3.99. At these fixed-depth shapes, allocations scale approximately with catalog size; the measurements do not demonstrate a superlinear defect or establish a general complexity guarantee. Timing is too variable for a precise scaling claim. The original exploratory fixture lacked the classifier and had lower allocation counts; only the final fixture's results above are the maintained baseline.

Complementary existing benchmarks

The following existing benchmarks were run sequentially, with GOMAXPROCS=4, local/offline toolchain settings, readonly modules, -benchtime=200ms, and -count=3:

GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
  go test ./internal/engine -mod=readonly -run '^$' \
  -bench '^(BenchmarkComposeWideMappings|BenchmarkUnderlayBroadCatalog|BenchmarkUnderlayRenderedHistory|BenchmarkInheritanceInspectedLoad)$' \
  -benchmem -benchtime=200ms -count=3

These per-metric medians provide comparison points for later changes. They have different timing boundaries and must not be compared as equivalent loads.

Benchmark / size	ms	B	Allocations
ComposeWideMappings / 128	1.815	848,512	9,658
ComposeWideMappings / 256	3.531	1,703,590	19,652
ComposeWideMappings / 512	9.532	3,626,298	39,643
UnderlayBroadCatalog / 128	3.299	1,265,032	15,904
UnderlayBroadCatalog / 256	3.436	2,581,000	33,473
UnderlayBroadCatalog / 512	8.472	5,268,937	68,581
InheritanceInspectedLoad / 16	2.571	876,236	11,050
InheritanceInspectedLoad / 32	4.066	1,545,216	19,943
InheritanceInspectedLoad / 64	8.358	2,906,067	37,779
UnderlayRenderedHistory / 32	0.336	504,546	3,294
UnderlayRenderedHistory / 64	0.661	1,006,053	6,529
UnderlayRenderedHistory / 128	1.709	2,031,853	12,997

Composition measures document combination; broad underlays measure an engine load; inspected inheritance includes parsing, loading, and an explanation query; rendered history measures explanation rendering after an untimed load. The existing inspection benchmark supplies coverage without adding another suite.

Stage 3: Profiles and implementation trace

CPU and sampled allocation profiles were collected in separate two-second runs at 128 pairs. The current-code downstream run measured 71.877 ms, 50,125,624 B, and 538,552 allocations. The final local fixture's profiled run measured 66.153 ms, 47,342,971 B, and 513,721 allocations. Profiled timings are not used in the ordinary timing comparison.

To regenerate local profiles outside the checkout:

profile_dir=$(mktemp -d)
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
  go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction/128$' \
  -benchtime=2s -cpuprofile="$profile_dir/startup.cpu" \
  -memprofile="$profile_dir/startup.mem" -o "$profile_dir/startup.test"
go tool pprof -top -nodecount=12 -alloc_space "$profile_dir/startup.test" "$profile_dir/startup.mem"
go tool pprof -top -nodecount=12 "$profile_dir/startup.test" "$profile_dir/startup.cpu"
go tool pprof -list='engine.Path.String' -alloc_space "$profile_dir/startup.test" "$profile_dir/startup.mem"

For downstream profiles, use the stage 1 benchmark selector and package, the same two-second settings, and separate output filenames in a temporary directory.

Allocation site (flat sampled bytes)	Local	Downstream
clonePathSegments	31.00%	30.18%
strings.Builder.WriteString	10.61%	11.02%
collectOrigins	5.04%	4.86%
collectFieldInfo	4.16%	5.17%
ParsePath	4.11%	4.37%
effectivePathIndex.func1	3.69%	3.13%
strings.Builder.WriteByte	3.30%	3.48%

Path.String accounts cumulatively for 13.90% of sampled allocated bytes and 23.03% of CPU samples locally, versus 14.51% and 21.80% downstream. The local line profile attributes 48.50 MB to writing separators and 156.01 MB to writing tokens across the profiled run, exposing repeated builder growth. These are sampled totals across many operations, not bytes per construction. Flat and cumulative shares overlap and must not be added together. GC/scanning, allocation, and map operations also occupy substantial CPU samples.

The source trace identifies these responsibilities:

• Path construction: Key and Index copy segments to preserve immutable parents and contextual mapping/sequence intent. Keys builds through repeated Key calls. Path.String starts an ungrown builder and formats every segment on every invocation.
• Metadata collection: collectFieldInfo formats each visited path and detaches origin records. Decode's collectOrigins likewise traverses and formats paths.
• Path indexes: effectivePathIndex walks and retains contextual paths; finalizeMetadataPaths repeats walks for effective and historical trees. These indexes serve different query roles.
• Sensitivity classification: classifySecretPaths indexes several trees and repeatedly parses textual pointers through addSecretPath. It also evaluates the caller's classifier.
• Publication: final typed detachment, metadata finalization, and sensitivity classification happen before return. Detachment protects against equal borrowed objects supplied by callbacks.
• Root conversion: callback adapters delegate to live engine contexts; result publication wraps the detached configuration and inspection. Paths delegate to engine paths. The dominant sampled work is inside the engine, not a second root-owned tree conversion.

The profiles support reducing redundant allocation and formatting. They do not justify dropping ownership copies, historical trees, indexes, or sensitivity processing. The consumer callback work is part of the measured lifecycle; post-return materialization remains downstream work.

Proposed first optimization

Try pre-sizing the local strings.Builder in engine Path.String using the encoded pointer length before writing tokens. Keep the existing segment kinds, canonical escaping, index formatting, root behavior, and independent returned strings. A straightforward length calculation plus Grow should reduce buffer growth without changing the Path representation or introducing shared state. Avoid a cache or buffer pool for this first experiment. This is a hypothesis, not an implemented improvement or a promise of one allocation per path.

Although segment cloning is the largest allocation site, changing it requires more ownership analysis across branches and captured records. Buffer sizing is a smaller first change supported by both profiles, with a clear measurement signal: fewer allocated bytes/allocations at all catalog sizes and less sampled builder-growth work. CPU benefit must be measured because length calculation also adds work. Escaped tokens and larger indexes require particular care; preserve ordinary handling of invalid paths rather than adding validation here.

Use existing TestPathRoundTripsSemanticKeys, TestPathPreservesSegmentIntentAndImmutability, and TestParsePathRejectsMalformedPointers, plus schema/metadata tests for numeric keys versus sequence indexes. Verify root-facade integration, history, explanation, and secret tests through the ordinary development suite. Add a focused case only for a consequential boundary not covered already. Keep inspection/rendering measurements alongside construction measurements; run race validation if a later change affects ownership or concurrency. No runtime ownership or concurrency behavior changes are proposed for buffer sizing.

Then compare repeated matching before/after runs at all three sizes and the downstream fixture; reprofile to confirm the attributed reduction. Keep the change only if repeatable benefit justifies its complexity. If it does not, return to profiling and assess pointer reparsing or repeated traversal rather than assuming the next optimization in advance.

Initial validation and checkpoint

The maintained fixture's preflight and measured runs succeeded at every size. ./scripts/check.sh passed formatting and whitespace checks, the offline suite (including local documentation links and facade boundaries), full vet, and package builds. At that checkpoint, Stages 4–6 remained unstarted and no production implementation had changed. The evidence supported one focused optimization experiment; wider representation changes needed their own measured justification.

Focused experiment: Pre-size Path.String

Status: Retained on 2026-10-09.

Change and comparison setup

The fresh unoptimized baseline is 91ae4fa7eff1571e7c05de01f9e922d75df7453d, including the maintained benchmark above. An isolated checkout preserved that baseline while the working candidate changed only engine Path.String and extended the existing path test. Toolchain, CPU, runtime settings, fixture sizes, and timing boundaries are as recorded above. Fresh measurements used the same benchmark commands: five 500-ms observations for construction/PromptKit, three 200-ms observations for complementary cases, and separate two-second profiles. All measurement families ran sequentially. Logs and binaries are outside the repositories at /tmp/configkit-path-sizing.JpbMpD.

The sizing pass counts separators, token bytes, and the extra bytes required for ~ and / escaping. A local 20-byte scratch array with strconv.AppendInt measures index text without allocating another string. strings.Builder.Grow reserves the calculated capacity. The original escaping, index-writing, root, and invalid-path behavior remain intact. There is no cache, pool, shared state, new validation, or change to segment ownership. The sizing pass adds work; benefit is measured rather than assumed.

The PromptKit checkout remained at eb830bb61346672f0f6c5bce6119b76ccc6d61ac. Its temporary replacement first selected the isolated baseline and then the ConfigKit working candidate. No consumer repository or committed dependency changed.

Fresh construction results

Each cell below is the median of that metric across five observations. Timing improvements describe this run, without claiming statistical significance or machine-independent speedups.

Workload	Before ms	After ms	Before B	After B	Before allocations	After allocations
Local / 32 pairs	18.251	17.495	11,999,426	11,036,879	129,136	80,465
Local / 128 pairs	68.096	60.859	47,320,688	43,490,008	513,720	319,131
Local / 512 pairs	262.823	252.779	188,764,416	173,640,328	2,051,979	1,273,697
PromptKit / independent	73.386	65.899	50,127,948	46,171,529	538,541	338,171

Allocated bytes fell by 8.02%, 8.10%, and 8.01% locally; allocation counts fell by 37.69%, 37.88%, and 37.93%. PromptKit showed 7.89% fewer allocated bytes and 37.21% fewer allocations. These reductions recur across observations and catalog sizes. The downstream total continues to include its adapter and post-return processing; it is not an isolated ConfigKit cost.

Timing ranges (ms), keeping the first PromptKit outlier:

Workload	Before range	After range
Local / 32 pairs	17.468–18.518	16.719–19.131
Local / 128 pairs	66.211–79.363	58.048–63.724
Local / 512 pairs	256.783–287.161	241.361–272.743
PromptKit / independent	68.308–134.999	63.664–74.520

The local 128-pair median and downstream median were approximately 10% lower; the other local medians were 4% lower. Timing remains sensitive to shared-host load, so the allocation reduction supplies the stronger evidence for retention.

Complementary workloads and possible timing regressions

The original comparison's per-metric medians follow. These fixtures retain the different boundaries explained above.

Benchmark / size	Before ms	After ms	Before B	After B	Before allocations	After allocations
ComposeWideMappings / 128	0.883	0.806	848,507	797,562	9,658	5,962
ComposeWideMappings / 256	1.781	1.708	1,703,582	1,595,293	19,652	11,860
ComposeWideMappings / 512	4.460	3.963	3,626,263	3,403,282	39,643	23,659
UnderlayBroadCatalog / 128	1.718	1.520	1,265,027	1,229,413	15,904	12,628
UnderlayBroadCatalog / 256	3.707	3.231	2,580,999	2,459,362	33,473	24,821
UnderlayBroadCatalog / 512	7.094	6.676	5,268,929	4,975,269	68,581	49,177
InheritanceInspectedLoad / 16	1.124	1.296	876,204	835,688	11,050	8,060
InheritanceInspectedLoad / 32	2.248	2.180	1,545,178	1,466,920	19,943	14,145
InheritanceInspectedLoad / 64	4.030	3.784	2,906,020	2,752,258	37,778	26,360
UnderlayRenderedHistory / 32	0.289	0.305	504,547	503,459	3,294	3,221
UnderlayRenderedHistory / 64	0.624	0.735	1,006,052	1,003,942	6,529	6,392
UnderlayRenderedHistory / 128	1.879	1.283	2,031,854	2,027,694	12,997	12,732

Allocations improved in every case. Rendering benefits are smaller because its load is outside timing. The first batch's 16-entry inheritance and smaller rendering times were higher after the change, so they received longer follow-up runs in reverse order (after, then before).

Inheritance / 16 used -benchtime=500ms -count=5: before times were 2.537, 1.153, 1.148, 1.155, and 1.153 ms; after times were 1.308, 1.088, 1.086, 1.112, and 1.082 ms. Its median changed from 1.153 to 1.088 ms, reversing the initial apparent regression, with the same allocation reduction.

Rendering used -benchtime=500ms -count=3. Its timing medians were:

History size	Before ms	After ms
32	0.290	0.302
64	1.538	0.610
128	7.213	1.329

The before / 64 observations spanned 0.643–3.004 ms and before / 128 spanned 2.211–7.909 ms, exposing substantial host variation. Rendering / 32 remained about 4% slower in this second batch, but the ranges overlapped (before 0.288–0.317 ms; after 0.299–0.307 ms). A small extra cost from the sizing pass on short inspection paths cannot be ruled out. There is insufficient evidence to claim every operation is faster; the stronger construction allocation benefit and limited implementation complexity justify this tradeoff. These measurements concern cumulative allocation; retained heap was not measured separately. The change introduces no cache or altered data lifetime.

Reprofiled attribution and retention decision

Fresh profiles used the local 128-pair fixture. The before profiled run measured 72.765 ms, 47,297,038 B, and 513,721 allocations; after measured 64.367 ms, 43,532,356 B, and 319,132 allocations. These remain separate from ordinary timing comparisons.

Path.String accounted for 14.38% of sampled allocation bytes before and 7.43% after. The line profile previously attributed 62.00 MB to separator writes and 183.01 MB to token writes across the run; afterward it attributed 123.00 MB to the initial Grow, with no sampled allocation at those writes. The profile runs completed different operation counts (36 and 39), so raw MB totals must not be treated as per-load differences. Direct benchmark measurements above establish the per-load reduction. Path.String's cumulative CPU share changed from 21.15% to 18.96%; that share includes the added sizing work.

Retain the change: a short local sizing pass removes approximately 195,000 allocations and 3.83 MB from a representative 128-pair construction, with matching benefits in the consumer and complementary cases. The existing segment ownership and public behavior stay intact. The extra scan and uncertain small-rendering timing cost are recorded rather than converted into a new performance promise.

Next candidate and remaining work

clonePathSegments remains the largest flat allocation site: 31.29% before and 34.99% after. Its larger share reflects other allocations disappearing; the underlying copies were not changed. ParsePath remains at 4.92% after, while effectivePathIndex.func1 accounts for 3.84%. GC and map operations remain prominent. Pointer formatting still accounts for nearly 19% of sampled CPU, including strings.Count work, so further formatting changes would need to justify their own extra complexity.

The recommended next experiment is to investigate combining the two walks of the final tree in finalizeMetadataPaths: it first calls effectivePathIndex and then walks the same final tree again for historical paths. One traversal could populate both indexes and format each final path once, while retaining the separate records, contextual segment kinds, ownership copies, and all walks of earlier/application/inheritance trees. This is a concrete redundancy to measure before considering broader path representation or ownership changes. Its expected benefit is not established yet. A generic Keys constructor rewrite is smaller, but only accounted for 0.78% of sampled allocated bytes in the after profile, so it is a lower-priority candidate.

The path/metadata/provenance/query/explanation/secret tests passed with the candidate. The existing path test now additionally checks decimal formatting at three-digit and signed integer limits, including negative invalid indexes. ./scripts/check.sh passed formatting, whitespace, the offline suite (including documentation links and facade boundaries), vet, and builds. This experiment does not complete Stage 5: PromptKit's file-backed, shared-base, and layered fixtures plus its applicable configuration tests still belong to the final consumer verification. No consumer upgrade or release is published.