925 lines
50 KiB
Markdown
925 lines
50 KiB
Markdown
# Startup construction measurements and focused experiments
|
||
|
||
Status: Complete. Both focused optimizations are retained; final consumer
|
||
verification and handoff are recorded below. Acceptance rests on allocation
|
||
reductions and behavioral checks; busy-host timing remains inconclusive.
|
||
|
||
Measured on 2026-10-09 under the [startup roadmap](startup.md). The
|
||
[consumer request](promptkit-performance-request.md) owns the submitted workload
|
||
and measurements. This document records the reproduced baselines, maintained
|
||
benchmark, profiles, and focused experiment results. It introduces no performance
|
||
contract or release gate.
|
||
|
||
## Environment and revisions
|
||
|
||
- PromptKit checkout: `eb830bb61346672f0f6c5bce6119b76ccc6d61ac` for both
|
||
downstream runs. The adapter and benchmark are unchanged from its earlier
|
||
review revision, `de6443f3aef4a66169e323e9f9a5f41594fbface`.
|
||
- Submitted ConfigKit dependency: v0.5.0, source commit
|
||
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`.
|
||
- Current ConfigKit baseline: `f1722274c02c08ff28556dd66f29e0da07a9f2c3`.
|
||
The benchmark added with this report changes no production implementation.
|
||
- Go: `/snap/bin/go`, `go1.26.8`, `linux/amd64`.
|
||
- CPU: Intel Core i5-7260U at 2.20 GHz; two physical cores, four logical CPUs.
|
||
- `GOMAXPROCS=4`, `GOTOOLCHAIN=local`, `GOWORK=off`, `GOPROXY=off`, and
|
||
readonly module mode. `GOGC` and `GOMEMLIMIT` were unset (runtime defaults).
|
||
- Dependencies were already cached. No consumer repository's committed
|
||
dependencies were changed, and no adoption was published. Separate temporary
|
||
clones excluded consumer worktree edits; the current-code comparison used a
|
||
temporary module replacement.
|
||
- Logs, test binaries, and profiles are outside both repositories under
|
||
`/tmp/configkit-startup.L9vnUo`. These are disposable evidence, not repository
|
||
dependencies; the commands below regenerate them.
|
||
|
||
The processor and toolchain match the submitted environment. The machine was
|
||
shared, with no exclusive CPU allocation or frequency control. An initial cold
|
||
downstream pass overlapped discovery work and ranged from 90.95 to 256.19 ms;
|
||
it was excluded from the warmed comparison below. Subsequent local runs also
|
||
showed substantial timing variation. All final-run observations, including
|
||
outliers, are recorded below. Allocation counts were much steadier than elapsed
|
||
time. These measurements do not establish statistical timing significance.
|
||
|
||
## Stage 1: Downstream reproduction
|
||
|
||
PromptKit's `BenchmarkConfigurationConstruction/independent` prepares 128
|
||
profiles and 128 prompt versions outside timing, including the captured
|
||
`configkit.Mapping` source. Each profile supplies an endpoint and model. Each
|
||
prompt has version key `1`, one user message, a profile reference, and text output
|
||
with validation disabled. There are 256 definitions, one tier, and no files,
|
||
imports, inheritance, credentials, or inference calls.
|
||
|
||
Each timed iteration invokes the real adapter's `load` function. Within
|
||
`LoadLayered`, it captures optional-field presence with `Resolution.InputInfo`,
|
||
normalizes profile values, performs domain validation, and supplies a path-based
|
||
secret classifier. The supplied endpoints are already canonical, so this
|
||
fixture records no normalization changes. After ConfigKit returns, the adapter
|
||
materializes prompts, compiles templates, computes content hashes, and builds
|
||
execution profiles. Those operations are included in the downstream timing.
|
||
|
||
The comparison used an isolated PromptKit checkout at the revision above. First
|
||
it retained the declared v0.5.0 dependency. Then its temporary `go.mod` received
|
||
a replacement pointing at an isolated ConfigKit checkout at the current
|
||
baseline. Both used:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' \
|
||
-bench '^BenchmarkConfigurationConstruction/independent$' \
|
||
-benchmem -benchtime=500ms -count=5 ./internal/configuration
|
||
```
|
||
|
||
To reproduce the temporary wiring, clone with `--no-hardlinks --local`, check
|
||
out the recorded revisions, and run this only in the disposable PromptKit copy:
|
||
|
||
```sh
|
||
go mod edit -replace=gitea.maximumdirect.net/eric/ConfigKit=/absolute/path/to/configkit-copy
|
||
```
|
||
|
||
| ConfigKit | Run | ms/construction | B/construction | allocations/construction |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| v0.5.0 | 1 | 74.339 | 50,155,869 | 538,590 |
|
||
| v0.5.0 | 2 | 69.340 | 50,159,666 | 538,541 |
|
||
| v0.5.0 | 3 | 71.453 | 50,131,114 | 538,542 |
|
||
| v0.5.0 | 4 | 70.164 | 50,113,244 | 538,541 |
|
||
| v0.5.0 | 5 | 72.495 | 50,165,238 | 538,540 |
|
||
| current | 1 | 73.567 | 50,147,688 | 538,593 |
|
||
| current | 2 | 70.959 | 50,152,860 | 538,540 |
|
||
| current | 3 | 67.509 | 50,167,353 | 538,541 |
|
||
| current | 4 | 71.491 | 50,181,043 | 538,543 |
|
||
| current | 5 | 68.786 | 50,131,684 | 538,541 |
|
||
|
||
The per-metric medians are 71.453 ms / 50,155,869 B / 538,541 allocations for
|
||
v0.5.0 and 70.959 ms / 50,152,860 B / 538,541 allocations for current code.
|
||
The submitted timing median was approximately 67.740 ms; its approximately
|
||
50.16 MB and 538,542 allocations are reproduced. The modest warmed timing
|
||
difference and the cold-run outlier are consistent with machine/runtime
|
||
variation; they do not establish a code regression. The production loading
|
||
code is identical between these ConfigKit revisions: their Go-file differences
|
||
are release/development tests. No startup improvement is claimed.
|
||
|
||
## Stage 2: Maintained public-API benchmark
|
||
|
||
[startup_bench_test.go](../../startup_bench_test.go) adds
|
||
`BenchmarkStartupConstruction` in the external `configkit_test` package. It
|
||
measures `LoadLayered` through the supported root facade at 32, 128, and 512
|
||
profile/prompt pairs (64, 256, and 1,024 definitions).
|
||
|
||
The input has two root collections. Profiles contain endpoint and model scalars;
|
||
prompts contain a numeric mapping key `1`, a profile reference, a message
|
||
sequence with role/content, and an output mapping with format/validation mode.
|
||
The source tree has `13 * pairs + 3` nodes (419, 1,667, and 6,659 respectively),
|
||
before typed projection adds omitted-field metadata. Typed profiles have twelve
|
||
optional fields; typed prompts retain optional metadata, inputs, messages, and
|
||
output fields. This records configuration shape as well as definition count.
|
||
|
||
Fixture generation, `Mapping` capture, and one successful lifecycle/metadata
|
||
check are outside timing. Timed work includes typed decoding, each callback,
|
||
resolution auditing, validation, final publication, and retention of the latest
|
||
result. The resolver captures all twelve profile field states and parses their
|
||
canonical endpoints. Validation visits sorted profile IDs. A simple path-based
|
||
secret predicate is installed, although the fixture contains no credentials.
|
||
The preflight checks callback execution, collection sizes, explicit versus
|
||
omitted profile fields, and metadata for the message reached through mapping
|
||
key `1` and sequence index 0.
|
||
|
||
This is representative rather than an exact copy of the consumer adapter: it
|
||
omits the unused backend collection and message cache-control type, simplifies
|
||
domain validation/classification, and does not compile templates, compute
|
||
hashes, or build runtime registries. Its lower allocation total must not be
|
||
treated as an exact subtraction of PromptKit overhead. It adds no imports of
|
||
PromptKit and runs offline without credentials. Its measurements are not
|
||
pass/fail thresholds.
|
||
|
||
Run from ConfigKit:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction$' \
|
||
-benchmem -benchtime=500ms -count=5
|
||
```
|
||
|
||
| Pairs | Run | ms/construction | B/construction | allocations/construction |
|
||
| ---: | ---: | ---: | ---: | ---: |
|
||
| 32 | 1 | 18.327 | 11,999,640 | 129,136 |
|
||
| 32 | 2 | 18.418 | 11,999,423 | 129,136 |
|
||
| 32 | 3 | 50.366 | 11,999,534 | 129,137 |
|
||
| 32 | 4 | 55.028 | 11,999,548 | 129,137 |
|
||
| 32 | 5 | 25.824 | 11,999,468 | 129,136 |
|
||
| 128 | 1 | 67.610 | 47,408,046 | 513,722 |
|
||
| 128 | 2 | 65.090 | 47,451,718 | 513,724 |
|
||
| 128 | 3 | 65.682 | 47,584,876 | 513,727 |
|
||
| 128 | 4 | 62.574 | 47,324,919 | 513,722 |
|
||
| 128 | 5 | 123.565 | 47,273,050 | 513,721 |
|
||
| 512 | 1 | 276.663 | 188,313,648 | 2,051,961 |
|
||
| 512 | 2 | 284.979 | 189,649,296 | 2,052,000 |
|
||
| 512 | 3 | 497.075 | 189,234,128 | 2,051,985 |
|
||
| 512 | 4 | 257.534 | 189,084,032 | 2,051,981 |
|
||
| 512 | 5 | 265.167 | 188,679,712 | 2,051,973 |
|
||
|
||
Per-metric medians:
|
||
|
||
| Pairs | ms | B | Allocations |
|
||
| ---: | ---: | ---: | ---: |
|
||
| 32 | 25.824 | 11,999,534 | 129,136 |
|
||
| 128 | 65.682 | 47,408,046 | 513,722 |
|
||
| 512 | 276.663 | 189,084,032 | 2,051,981 |
|
||
|
||
Increasing the catalog fourfold multiplies allocation counts by approximately
|
||
3.98 and 3.99, and allocated bytes by approximately 3.95 and 3.99. At these
|
||
fixed-depth shapes, allocations scale approximately with catalog size; the
|
||
measurements do not demonstrate a superlinear defect or establish a general
|
||
complexity guarantee. Timing is too variable for a precise scaling claim.
|
||
The original exploratory fixture lacked the classifier and had lower allocation
|
||
counts; only the final fixture's results above are the maintained baseline.
|
||
|
||
### Complementary existing benchmarks
|
||
|
||
The following existing benchmarks were run sequentially, with
|
||
`GOMAXPROCS=4`, local/offline toolchain settings, readonly modules,
|
||
`-benchtime=200ms`, and `-count=3`:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test ./internal/engine -mod=readonly -run '^$' \
|
||
-bench '^(BenchmarkComposeWideMappings|BenchmarkUnderlayBroadCatalog|BenchmarkUnderlayRenderedHistory|BenchmarkInheritanceInspectedLoad)$' \
|
||
-benchmem -benchtime=200ms -count=3
|
||
```
|
||
|
||
These per-metric medians provide comparison points for later changes. They have
|
||
different timing boundaries and must not be compared as equivalent loads.
|
||
|
||
| Benchmark / size | ms | B | Allocations |
|
||
| --- | ---: | ---: | ---: |
|
||
| ComposeWideMappings / 128 | 1.815 | 848,512 | 9,658 |
|
||
| ComposeWideMappings / 256 | 3.531 | 1,703,590 | 19,652 |
|
||
| ComposeWideMappings / 512 | 9.532 | 3,626,298 | 39,643 |
|
||
| UnderlayBroadCatalog / 128 | 3.299 | 1,265,032 | 15,904 |
|
||
| UnderlayBroadCatalog / 256 | 3.436 | 2,581,000 | 33,473 |
|
||
| UnderlayBroadCatalog / 512 | 8.472 | 5,268,937 | 68,581 |
|
||
| InheritanceInspectedLoad / 16 | 2.571 | 876,236 | 11,050 |
|
||
| InheritanceInspectedLoad / 32 | 4.066 | 1,545,216 | 19,943 |
|
||
| InheritanceInspectedLoad / 64 | 8.358 | 2,906,067 | 37,779 |
|
||
| UnderlayRenderedHistory / 32 | 0.336 | 504,546 | 3,294 |
|
||
| UnderlayRenderedHistory / 64 | 0.661 | 1,006,053 | 6,529 |
|
||
| UnderlayRenderedHistory / 128 | 1.709 | 2,031,853 | 12,997 |
|
||
|
||
Composition measures document combination; broad underlays measure an engine
|
||
load; inspected inheritance includes parsing, loading, and an explanation query;
|
||
rendered history measures explanation rendering after an untimed load. The
|
||
existing inspection benchmark supplies coverage without adding another suite.
|
||
|
||
## Stage 3: Profiles and implementation trace
|
||
|
||
CPU and sampled allocation profiles were collected in separate two-second runs
|
||
at 128 pairs. The current-code downstream run measured 71.877 ms, 50,125,624 B,
|
||
and 538,552 allocations. The final local fixture's profiled run measured
|
||
66.153 ms, 47,342,971 B, and 513,721 allocations. Profiled timings are not used
|
||
in the ordinary timing comparison.
|
||
|
||
To regenerate local profiles outside the checkout:
|
||
|
||
```sh
|
||
profile_dir=$(mktemp -d)
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction/128$' \
|
||
-benchtime=2s -cpuprofile="$profile_dir/startup.cpu" \
|
||
-memprofile="$profile_dir/startup.mem" -o "$profile_dir/startup.test"
|
||
go tool pprof -top -nodecount=12 -alloc_space "$profile_dir/startup.test" "$profile_dir/startup.mem"
|
||
go tool pprof -top -nodecount=12 "$profile_dir/startup.test" "$profile_dir/startup.cpu"
|
||
go tool pprof -list='engine.Path.String' -alloc_space "$profile_dir/startup.test" "$profile_dir/startup.mem"
|
||
```
|
||
|
||
For downstream profiles, use the stage 1 benchmark selector and package, the
|
||
same two-second settings, and separate output filenames in a temporary directory.
|
||
|
||
| Allocation site (flat sampled bytes) | Local | Downstream |
|
||
| --- | ---: | ---: |
|
||
| `clonePathSegments` | 31.00% | 30.18% |
|
||
| `strings.Builder.WriteString` | 10.61% | 11.02% |
|
||
| `collectOrigins` | 5.04% | 4.86% |
|
||
| `collectFieldInfo` | 4.16% | 5.17% |
|
||
| `ParsePath` | 4.11% | 4.37% |
|
||
| `effectivePathIndex.func1` | 3.69% | 3.13% |
|
||
| `strings.Builder.WriteByte` | 3.30% | 3.48% |
|
||
|
||
`Path.String` accounts cumulatively for 13.90% of sampled allocated bytes and
|
||
23.03% of CPU samples locally, versus 14.51% and 21.80% downstream. The local
|
||
line profile attributes 48.50 MB to writing separators and 156.01 MB to writing
|
||
tokens across the profiled run, exposing repeated builder growth. These are
|
||
sampled totals across many operations, not bytes per construction. Flat and
|
||
cumulative shares overlap and must not be added together. GC/scanning,
|
||
allocation, and map operations also occupy substantial CPU samples.
|
||
|
||
The source trace identifies these responsibilities:
|
||
|
||
- [Path construction](../../internal/engine/path.go): `Key` and `Index` copy
|
||
segments to preserve immutable parents and contextual mapping/sequence intent.
|
||
`Keys` builds through repeated `Key` calls. `Path.String` starts an ungrown
|
||
builder and formats every segment on every invocation.
|
||
- [Metadata collection](../../internal/engine/load_metadata.go):
|
||
`collectFieldInfo` formats each visited path and detaches origin records.
|
||
Decode's `collectOrigins` likewise traverses and formats paths.
|
||
- [Path indexes](../../internal/engine/metadata.go): `effectivePathIndex`
|
||
walks and retains contextual paths; `finalizeMetadataPaths` repeats walks for
|
||
effective and historical trees. These indexes serve different query roles.
|
||
- [Sensitivity classification](../../internal/engine/secret.go):
|
||
`classifySecretPaths` indexes several trees and repeatedly parses textual
|
||
pointers through `addSecretPath`. It also evaluates the caller's classifier.
|
||
- [Publication](../../internal/engine/load_result.go): final typed detachment,
|
||
metadata finalization, and sensitivity classification happen before return.
|
||
Detachment protects against equal borrowed objects supplied by callbacks.
|
||
- [Root conversion](../../lifecycle.go): callback adapters delegate to live
|
||
engine contexts; result publication wraps the detached configuration and
|
||
inspection. Paths delegate to engine paths. The dominant sampled work is
|
||
inside the engine, not a second root-owned tree conversion.
|
||
|
||
The profiles support reducing redundant allocation and formatting. They do not
|
||
justify dropping ownership copies, historical trees, indexes, or sensitivity
|
||
processing. The consumer callback work is part of the measured lifecycle;
|
||
post-return materialization remains downstream work.
|
||
|
||
### Proposed first optimization
|
||
|
||
Try pre-sizing the local `strings.Builder` in engine `Path.String` using the
|
||
encoded pointer length before writing tokens. Keep the existing segment kinds,
|
||
canonical escaping, index formatting, root behavior, and independent returned
|
||
strings. A straightforward length calculation plus `Grow` should reduce buffer
|
||
growth without changing the `Path` representation or introducing shared state.
|
||
Avoid a cache or buffer pool for this first experiment. This is a hypothesis,
|
||
not an implemented improvement or a promise of one allocation per path.
|
||
|
||
Although segment cloning is the largest allocation site, changing it requires
|
||
more ownership analysis across branches and captured records. Buffer sizing is
|
||
a smaller first change supported by both profiles, with a clear measurement
|
||
signal: fewer allocated bytes/allocations at all catalog sizes and less sampled
|
||
builder-growth work. CPU benefit must be measured because length calculation
|
||
also adds work. Escaped tokens and larger indexes require particular care;
|
||
preserve ordinary handling of invalid paths rather than adding validation here.
|
||
|
||
Use existing `TestPathRoundTripsSemanticKeys`,
|
||
`TestPathPreservesSegmentIntentAndImmutability`, and
|
||
`TestParsePathRejectsMalformedPointers`, plus schema/metadata tests for numeric
|
||
keys versus sequence indexes. Verify root-facade integration, history,
|
||
explanation, and secret tests through the ordinary development suite. Add a
|
||
focused case only for a consequential boundary not covered already. Keep
|
||
inspection/rendering measurements alongside construction measurements; run
|
||
race validation if a later change affects ownership or concurrency. No runtime
|
||
ownership or concurrency behavior changes are proposed for buffer sizing.
|
||
|
||
Then compare repeated matching before/after runs at all three sizes and the
|
||
downstream fixture; reprofile to confirm the attributed reduction. Keep the
|
||
change only if repeatable benefit justifies its complexity. If it does not,
|
||
return to profiling and assess pointer reparsing or repeated traversal rather
|
||
than assuming the next optimization in advance.
|
||
|
||
## Initial validation and checkpoint
|
||
|
||
The maintained fixture's preflight and measured runs succeeded at every size.
|
||
`./scripts/check.sh` passed formatting and whitespace checks, the offline suite
|
||
(including local documentation links and facade boundaries), full vet, and
|
||
package builds. At that checkpoint, Stages 4–6 remained unstarted and no
|
||
production implementation had changed. The evidence supported one focused
|
||
optimization experiment; wider representation changes needed their own measured
|
||
justification.
|
||
|
||
## Focused experiment: Pre-size `Path.String`
|
||
|
||
Status: Retained on 2026-10-09.
|
||
|
||
### Change and comparison setup
|
||
|
||
The fresh unoptimized baseline is `91ae4fa7eff1571e7c05de01f9e922d75df7453d`,
|
||
including the maintained benchmark above. An isolated checkout preserved that
|
||
baseline while the working candidate changed only engine `Path.String` and
|
||
extended the existing path test. Toolchain, CPU, runtime settings, fixture sizes,
|
||
and timing boundaries are as recorded above. Fresh measurements used the same
|
||
benchmark commands: five 500-ms observations for construction/PromptKit, three
|
||
200-ms observations for complementary cases, and separate two-second profiles.
|
||
All measurement families ran sequentially. Logs and binaries are outside the
|
||
repositories at `/tmp/configkit-path-sizing.JpbMpD`.
|
||
|
||
The sizing pass counts separators, token bytes, and the extra bytes required for
|
||
`~` and `/` escaping. A local 20-byte scratch array with `strconv.AppendInt`
|
||
measures index text without allocating another string. `strings.Builder.Grow`
|
||
reserves the calculated capacity. The original escaping, index-writing, root,
|
||
and invalid-path behavior remain intact. There is no cache, pool, shared state,
|
||
new validation, or change to segment ownership. The sizing pass adds work;
|
||
benefit is measured rather than assumed.
|
||
|
||
The PromptKit checkout remained at
|
||
`eb830bb61346672f0f6c5bce6119b76ccc6d61ac`. Its temporary replacement first
|
||
selected the isolated baseline and then the ConfigKit working candidate. No
|
||
consumer repository or committed dependency changed.
|
||
|
||
### Fresh construction results
|
||
|
||
Each cell below is the median of that metric across five observations. Timing
|
||
improvements describe this run, without claiming statistical significance or
|
||
machine-independent speedups.
|
||
|
||
| Workload | Before ms | After ms | Before B | After B | Before allocations | After allocations |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| Local / 32 pairs | 18.251 | 17.495 | 11,999,426 | 11,036,879 | 129,136 | 80,465 |
|
||
| Local / 128 pairs | 68.096 | 60.859 | 47,320,688 | 43,490,008 | 513,720 | 319,131 |
|
||
| Local / 512 pairs | 262.823 | 252.779 | 188,764,416 | 173,640,328 | 2,051,979 | 1,273,697 |
|
||
| PromptKit / independent | 73.386 | 65.899 | 50,127,948 | 46,171,529 | 538,541 | 338,171 |
|
||
|
||
Allocated bytes fell by 8.02%, 8.10%, and 8.01% locally; allocation counts fell
|
||
by 37.69%, 37.88%, and 37.93%. PromptKit showed 7.89% fewer allocated bytes and
|
||
37.21% fewer allocations. These reductions recur across observations and catalog
|
||
sizes. The downstream total continues to include its adapter and post-return
|
||
processing; it is not an isolated ConfigKit cost.
|
||
|
||
Timing ranges (ms), keeping the first PromptKit outlier:
|
||
|
||
| Workload | Before range | After range |
|
||
| --- | ---: | ---: |
|
||
| Local / 32 pairs | 17.468–18.518 | 16.719–19.131 |
|
||
| Local / 128 pairs | 66.211–79.363 | 58.048–63.724 |
|
||
| Local / 512 pairs | 256.783–287.161 | 241.361–272.743 |
|
||
| PromptKit / independent | 68.308–134.999 | 63.664–74.520 |
|
||
|
||
The local 128-pair median and downstream median were approximately 10% lower;
|
||
the other local medians were 4% lower. Timing remains sensitive to shared-host
|
||
load, so the allocation reduction supplies the stronger evidence for retention.
|
||
|
||
### Complementary workloads and possible timing regressions
|
||
|
||
The original comparison's per-metric medians follow. These fixtures retain the
|
||
different boundaries explained above.
|
||
|
||
| Benchmark / size | Before ms | After ms | Before B | After B | Before allocations | After allocations |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| ComposeWideMappings / 128 | 0.883 | 0.806 | 848,507 | 797,562 | 9,658 | 5,962 |
|
||
| ComposeWideMappings / 256 | 1.781 | 1.708 | 1,703,582 | 1,595,293 | 19,652 | 11,860 |
|
||
| ComposeWideMappings / 512 | 4.460 | 3.963 | 3,626,263 | 3,403,282 | 39,643 | 23,659 |
|
||
| UnderlayBroadCatalog / 128 | 1.718 | 1.520 | 1,265,027 | 1,229,413 | 15,904 | 12,628 |
|
||
| UnderlayBroadCatalog / 256 | 3.707 | 3.231 | 2,580,999 | 2,459,362 | 33,473 | 24,821 |
|
||
| UnderlayBroadCatalog / 512 | 7.094 | 6.676 | 5,268,929 | 4,975,269 | 68,581 | 49,177 |
|
||
| InheritanceInspectedLoad / 16 | 1.124 | 1.296 | 876,204 | 835,688 | 11,050 | 8,060 |
|
||
| InheritanceInspectedLoad / 32 | 2.248 | 2.180 | 1,545,178 | 1,466,920 | 19,943 | 14,145 |
|
||
| InheritanceInspectedLoad / 64 | 4.030 | 3.784 | 2,906,020 | 2,752,258 | 37,778 | 26,360 |
|
||
| UnderlayRenderedHistory / 32 | 0.289 | 0.305 | 504,547 | 503,459 | 3,294 | 3,221 |
|
||
| UnderlayRenderedHistory / 64 | 0.624 | 0.735 | 1,006,052 | 1,003,942 | 6,529 | 6,392 |
|
||
| UnderlayRenderedHistory / 128 | 1.879 | 1.283 | 2,031,854 | 2,027,694 | 12,997 | 12,732 |
|
||
|
||
Allocations improved in every case. Rendering benefits are smaller because its
|
||
load is outside timing. The first batch's 16-entry inheritance and smaller
|
||
rendering times were higher after the change, so they received longer follow-up
|
||
runs in reverse order (after, then before).
|
||
|
||
Inheritance / 16 used `-benchtime=500ms -count=5`: before times were
|
||
2.537, 1.153, 1.148, 1.155, and 1.153 ms; after times were 1.308, 1.088, 1.086,
|
||
1.112, and 1.082 ms. Its median changed from 1.153 to 1.088 ms, reversing the
|
||
initial apparent regression, with the same allocation reduction.
|
||
|
||
Rendering used `-benchtime=500ms -count=3`. Its timing medians were:
|
||
|
||
| History size | Before ms | After ms |
|
||
| ---: | ---: | ---: |
|
||
| 32 | 0.290 | 0.302 |
|
||
| 64 | 1.538 | 0.610 |
|
||
| 128 | 7.213 | 1.329 |
|
||
|
||
The before / 64 observations spanned 0.643–3.004 ms and before / 128 spanned
|
||
2.211–7.909 ms, exposing substantial host variation. Rendering / 32 remained
|
||
about 4% slower in this second batch, but the ranges overlapped (before
|
||
0.288–0.317 ms; after 0.299–0.307 ms). A small extra cost from the sizing pass
|
||
on short inspection paths cannot be ruled out. There is insufficient evidence
|
||
to claim every operation is faster; the stronger construction allocation benefit
|
||
and limited implementation complexity justify this tradeoff. These measurements
|
||
concern cumulative allocation; retained heap was not measured separately. The
|
||
change introduces no cache or altered data lifetime.
|
||
|
||
### Reprofiled attribution and retention decision
|
||
|
||
Fresh profiles used the local 128-pair fixture. The before profiled run measured
|
||
72.765 ms, 47,297,038 B, and 513,721 allocations; after measured 64.367 ms,
|
||
43,532,356 B, and 319,132 allocations. These remain separate from ordinary
|
||
timing comparisons.
|
||
|
||
`Path.String` accounted for 14.38% of sampled allocation bytes before and 7.43%
|
||
after. The line profile previously attributed 62.00 MB to separator writes and
|
||
183.01 MB to token writes across the run; afterward it attributed 123.00 MB to
|
||
the initial `Grow`, with no sampled allocation at those writes. The profile
|
||
runs completed different operation counts (36 and 39), so raw MB totals must
|
||
not be treated as per-load differences. Direct benchmark measurements above
|
||
establish the per-load reduction. `Path.String`'s cumulative CPU share changed
|
||
from 21.15% to 18.96%; that share includes the added sizing work.
|
||
|
||
Retain the change: a short local sizing pass removes approximately 195,000
|
||
allocations and 3.83 MB from a representative 128-pair construction, with
|
||
matching benefits in the consumer and complementary cases. The existing
|
||
segment ownership and public behavior stay intact. The extra scan and uncertain
|
||
small-rendering timing cost are recorded rather than converted into a new
|
||
performance promise.
|
||
|
||
### Next candidate and remaining work
|
||
|
||
`clonePathSegments` remains the largest flat allocation site: 31.29% before and
|
||
34.99% after. Its larger share reflects other allocations disappearing; the
|
||
underlying copies were not changed. `ParsePath` remains at 4.92% after, while
|
||
`effectivePathIndex.func1` accounts for 3.84%. GC and map operations remain
|
||
prominent. Pointer formatting still accounts for nearly 19% of sampled CPU,
|
||
including `strings.Count` work, so further formatting changes would need to
|
||
justify their own extra complexity.
|
||
|
||
At completion of the first experiment, the recommended next experiment was to
|
||
investigate combining the two walks of the final tree in `finalizeMetadataPaths`:
|
||
it first called `effectivePathIndex` and then walked the same final tree again
|
||
for historical paths. One traversal could populate both indexes and format each
|
||
final path once, while retaining
|
||
the separate records, contextual segment kinds, ownership copies, and all walks
|
||
of earlier/application/inheritance trees. This is a concrete redundancy to
|
||
measure before considering broader path representation or ownership changes.
|
||
Its benefit had not yet been established; the experiment below now records it.
|
||
A generic `Keys` constructor rewrite is smaller, but only accounted for 0.78% of
|
||
sampled allocated bytes in the after profile, so it is a lower-priority candidate.
|
||
|
||
The path/metadata/provenance/query/explanation/secret tests passed with the
|
||
candidate. The existing path test now additionally checks decimal formatting
|
||
at three-digit and signed integer limits, including negative invalid indexes.
|
||
`./scripts/check.sh` passed formatting, whitespace, the offline suite (including
|
||
documentation links and facade boundaries), vet, and builds. This experiment
|
||
does not complete Stage 5: PromptKit's file-backed, shared-base,
|
||
and layered fixtures plus its applicable configuration tests still belong to
|
||
the final consumer verification. No consumer upgrade or release is published.
|
||
|
||
## Focused experiment: combine final metadata traversal
|
||
|
||
Status: Complete; retain the change. Measured on 2026-10-09. This is the second
|
||
Stage 4 experiment, not completion of the final consumer verification.
|
||
|
||
### Change and ownership boundaries
|
||
|
||
`finalizeMetadataPaths` formerly built the effective index with
|
||
`effectivePathIndex(final)`, then walked the same final tree again to seed the
|
||
historical index. It now allocates the two maps separately and walks the final
|
||
tree once. Each visited path is formatted once for insertion into those maps.
|
||
The effective index stores its own `path.clone()`; the existing historical
|
||
insertion/deduplication logic stores a separate clone. The maps may share an
|
||
immutable pointer string, not mutable path-segment storage.
|
||
|
||
The historical helper accepts the already-formatted pointer so the final
|
||
callback can reuse it. Earlier trees, override applications, and inheritance
|
||
controls/transfers retain their existing walks, order, contextual path kinds,
|
||
and deduplication. `effectivePathIndex` remains unchanged for resolution input.
|
||
Nil final input still skips traversal while leaving both maps initialized.
|
||
There is no public API, configuration-semantic, dependency, cache, pooling,
|
||
release-gate, or result-lifetime change.
|
||
|
||
### Revisions and reproduction
|
||
|
||
- Before: `673faea097fb9a178b5743e71cbdcbecd215568f`, including the retained
|
||
`Path.String` optimization. These are fresh measurements, not comparisons
|
||
that bundle the two optimizations together.
|
||
- After: the candidate accompanying this section. Its only production change is
|
||
in `internal/engine/metadata.go`, Git blob
|
||
`e22804d659efb203a3829382a21e4e8d059b6c5f`; the accompanying tests and this
|
||
report are not part of timed construction.
|
||
- PromptKit: `eb830bb61346672f0f6c5bce6119b76ccc6d61ac`, unchanged for both
|
||
candidates. Its independent fixture retains the timing boundaries described
|
||
in Stage 1, including adapter processing after ConfigKit returns.
|
||
- Toolchain, hardware, runtime settings, fixture sizes, and setup boundaries
|
||
match the environment and maintained benchmark above. All measurements ran
|
||
sequentially; profiles and correctness checks did not overlap timed runs.
|
||
Shared-host scheduling and CPU frequency were not controlled.
|
||
- Raw benchmark logs, profiles, and binaries are outside both repositories at
|
||
`/tmp/configkit-final-walk.uDFTyU`. A disposable ConfigKit clone holds the
|
||
before revision; a disposable PromptKit clone alternately uses a module
|
||
replacement pointing to that clone or the working candidate. No original
|
||
consumer files or committed dependencies changed.
|
||
|
||
Run from each ConfigKit candidate with matching settings:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction$' \
|
||
-benchmem -benchtime=500ms -count=5
|
||
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly ./internal/engine -run '^$' \
|
||
-bench '^(BenchmarkComposeWideMappings|BenchmarkUnderlayBroadCatalog|BenchmarkUnderlayRenderedHistory|BenchmarkInheritanceInspectedLoad)$' \
|
||
-benchmem -benchtime=200ms -count=3
|
||
```
|
||
|
||
For each candidate, rewire only the disposable PromptKit copy with the
|
||
`go mod edit -replace` command in Stage 1, then run its same independent
|
||
benchmark with `-benchtime=500ms -count=5`. Initial batches ran before, then
|
||
after. Longer repeats reversed that order: after, then before. Construction
|
||
and consumer repeats used `-benchtime=1s -count=5`; complementary repeats
|
||
used `-benchtime=500ms -count=5`.
|
||
|
||
### Fresh construction results
|
||
|
||
Every cell is the per-metric median of five observations. No outliers were
|
||
removed. These timing differences are descriptive, not statistical evidence
|
||
of a speedup or regression.
|
||
|
||
| Workload | Before ms | After ms | Before B | After B | Before allocations | After allocations |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| Local / 32 pairs | 17.363 | 17.175 | 11,036,876 | 10,887,031 | 80,465 | 78,605 |
|
||
| Local / 128 pairs | 61.612 | 62.541 | 43,513,114 | 42,912,759 | 319,130 | 311,702 |
|
||
| Local / 512 pairs | 251.336 | 256.408 | 173,804,200 | 170,796,144 | 1,273,712 | 1,244,003 |
|
||
| PromptKit / independent | 66.861 | 87.909 | 46,174,781 | 45,525,968 | 338,168 | 330,483 |
|
||
|
||
Local allocation counts fell by 2.31%, 2.33%, and 2.33%, respectively; allocated
|
||
bytes fell by 1.36%, 1.38%, and 1.73%. PromptKit showed 2.27% fewer allocations
|
||
and 1.41% fewer allocated bytes. Its total still includes consumer adapter
|
||
work, rather than measuring only ConfigKit.
|
||
|
||
Initial timing ranges (ms):
|
||
|
||
| Workload | Before range | After range |
|
||
| --- | ---: | ---: |
|
||
| Local / 32 pairs | 16.547–19.567 | 16.740–20.823 |
|
||
| Local / 128 pairs | 58.494–69.982 | 58.401–68.902 |
|
||
| Local / 512 pairs | 243.146–279.740 | 249.116–439.932 |
|
||
| PromptKit / independent | 64.033–126.770 | 84.062–125.970 |
|
||
|
||
The initial PromptKit median was 31% higher after the change; the larger local
|
||
medians were also slightly higher. Longer reverse-order repeats gave:
|
||
|
||
| Workload | Before ms | After ms | Before range | After range |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| Local / 32 pairs | 18.397 | 17.487 | 18.323–19.124 | 17.322–18.628 |
|
||
| Local / 128 pairs | 62.551 | 61.602 | 58.959–66.066 | 58.348–63.903 |
|
||
| Local / 512 pairs | 258.394 | 251.583 | 239.896–263.039 | 247.995–304.867 |
|
||
| PromptKit / independent | 68.283 | 68.134 | 63.908–71.703 | 64.020–73.127 |
|
||
|
||
The consumer's initial slowdown did not recur. All four repeat ranges overlap.
|
||
Allocation reductions recur: local count medians remain 2.31–2.33% lower and
|
||
byte medians 1.36–1.43% lower; PromptKit remains 2.27% and 1.38% lower,
|
||
respectively. At 128 pairs, the longer run changed from 43,510,872 B and
|
||
319,132 allocations to 42,888,256 B and 311,702 allocations.
|
||
|
||
### Complementary workloads and timing investigation
|
||
|
||
Initial complementary results are per-metric medians of three observations:
|
||
|
||
| Benchmark / size | Before ms | After ms | Before B | After B | Before allocations | After allocations |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| ComposeWideMappings / 128 | 0.824 | 1.166 | 797,564 | 797,564 | 5,962 | 5,962 |
|
||
| ComposeWideMappings / 256 | 1.676 | 1.755 | 1,595,291 | 1,595,293 | 11,860 | 11,860 |
|
||
| ComposeWideMappings / 512 | 4.235 | 4.293 | 3,403,281 | 3,403,290 | 23,659 | 23,659 |
|
||
| InheritanceInspectedLoad / 16 | 1.127 | 1.211 | 835,681 | 832,237 | 8,060 | 7,990 |
|
||
| InheritanceInspectedLoad / 32 | 2.035 | 1.960 | 1,466,907 | 1,460,263 | 14,145 | 14,011 |
|
||
| InheritanceInspectedLoad / 64 | 4.093 | 4.150 | 2,752,251 | 2,739,211 | 26,360 | 26,098 |
|
||
| UnderlayBroadCatalog / 128 | 1.618 | 1.613 | 1,229,411 | 1,218,907 | 12,628 | 12,370 |
|
||
| UnderlayBroadCatalog / 256 | 3.296 | 3.479 | 2,459,362 | 2,437,599 | 24,821 | 24,307 |
|
||
| UnderlayBroadCatalog / 512 | 6.703 | 8.523 | 4,975,266 | 4,930,978 | 49,177 | 48,151 |
|
||
| UnderlayRenderedHistory / 32 | 0.290 | 0.659 | 503,458 | 503,461 | 3,221 | 3,221 |
|
||
| UnderlayRenderedHistory / 64 | 2.085 | 1.204 | 1,003,942 | 1,003,947 | 6,392 | 6,392 |
|
||
| UnderlayRenderedHistory / 128 | 3.331 | 2.723 | 2,027,690 | 2,027,695 | 12,732 | 12,732 |
|
||
|
||
Inheritance loading and broad-underlay loading allocate less in every case.
|
||
Pure composition does not run metadata finalization and its allocation counts
|
||
are unchanged. History rendering also has unchanged allocation counts, as
|
||
expected: loading is outside its timed section. Small byte differences in
|
||
those unaffected measurements do not establish an allocation improvement.
|
||
|
||
The longer reverse-order complementary batches produced these medians and
|
||
ranges (ms), retaining all five observations:
|
||
|
||
| Benchmark / size | Before ms | After ms | Before range | After range |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| ComposeWideMappings / 128 | 0.915 | 0.858 | 0.883–1.512 | 0.839–0.922 |
|
||
| ComposeWideMappings / 256 | 1.762 | 1.733 | 1.732–1.857 | 1.613–1.962 |
|
||
| ComposeWideMappings / 512 | 4.853 | 4.443 | 4.531–5.321 | 4.197–6.346 |
|
||
| InheritanceInspectedLoad / 16 | 1.200 | 1.822 | 1.133–1.494 | 1.652–1.951 |
|
||
| InheritanceInspectedLoad / 32 | 2.141 | 3.177 | 2.130–2.208 | 3.055–3.634 |
|
||
| InheritanceInspectedLoad / 64 | 4.169 | 5.066 | 4.054–4.460 | 3.883–8.171 |
|
||
| UnderlayBroadCatalog / 128 | 1.736 | 1.852 | 1.595–1.831 | 1.718–2.510 |
|
||
| UnderlayBroadCatalog / 256 | 3.712 | 3.743 | 3.413–3.843 | 3.378–7.093 |
|
||
| UnderlayBroadCatalog / 512 | 7.479 | 12.938 | 7.140–8.168 | 11.075–13.368 |
|
||
| UnderlayRenderedHistory / 32 | 0.326 | 0.563 | 0.311–1.048 | 0.530–0.583 |
|
||
| UnderlayRenderedHistory / 64 | 0.719 | 1.083 | 0.651–3.182 | 0.931–1.145 |
|
||
| UnderlayRenderedHistory / 128 | 1.435 | 2.567 | 1.361–5.152 | 2.289–2.887 |
|
||
|
||
Allocation-count medians in these repeats match the initial complementary
|
||
table. Timing remains mixed: several loading and rendering cases were much
|
||
slower in the after batch, while the unchanged pure-composition cases were
|
||
faster. To investigate the substantial apparent regressions, two additional
|
||
alternating rounds ran after/before, then after/before, with
|
||
`-benchtime=1s -count=3` and this benchmark selector:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly ./internal/engine -run '^$' \
|
||
-bench '^(BenchmarkInheritanceInspectedLoad|BenchmarkUnderlayBroadCatalog|BenchmarkUnderlayRenderedHistory)$/^(16|32|512)$' \
|
||
-benchmem -benchtime=1s -count=3
|
||
```
|
||
|
||
| Round / benchmark / size | Before ms | After ms | Before range | After range |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| 1 / InheritanceInspectedLoad / 16 | 1.126 | 1.166 | 1.116–1.151 | 1.139–1.187 |
|
||
| 1 / InheritanceInspectedLoad / 32 | 2.064 | 2.070 | 2.023–2.133 | 2.053–2.148 |
|
||
| 1 / UnderlayBroadCatalog / 512 | 6.949 | 6.822 | 6.904–7.461 | 6.785–7.204 |
|
||
| 1 / UnderlayRenderedHistory / 32 | 0.312 | 0.311 | 0.304–0.317 | 0.306–0.314 |
|
||
| 2 / InheritanceInspectedLoad / 16 | 1.196 | 1.174 | 1.135–1.270 | 1.125–1.218 |
|
||
| 2 / InheritanceInspectedLoad / 32 | 2.196 | 2.047 | 2.108–2.224 | 2.023–2.084 |
|
||
| 2 / UnderlayBroadCatalog / 512 | 7.514 | 7.286 | 7.291–7.645 | 6.767–7.319 |
|
||
| 2 / UnderlayRenderedHistory / 32 | 0.332 | 0.324 | 0.327–0.345 | 0.323–0.342 |
|
||
|
||
The large regressions did not recur. Inheritance / 16 was about 4% slower in
|
||
the first alternating round and about 2% faster in the second; the ranges
|
||
overlap in both. The other targeted cases were similar or faster after the
|
||
change, with the same allocation counts as the initial table. This supports
|
||
host/run variation as the explanation for the large batch differences, but
|
||
does not establish that every operation is faster or exclude small timing
|
||
costs. The allocation reduction is the more repeatable result.
|
||
|
||
### Profile attribution and retention decision
|
||
|
||
Separate profiles used the local 128-pair fixture and
|
||
`-benchtime=2s`, with explicit output paths outside the repositories:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction/128$' \
|
||
-benchmem -benchtime=2s -cpuprofile=/absolute/temp/cpu.pprof \
|
||
-memprofile=/absolute/temp/mem.pprof -o /absolute/temp/configkit.test
|
||
|
||
go tool pprof -list='engine.finalizeMetadataPaths' -alloc_space /absolute/temp/mem.pprof
|
||
go tool pprof -list='engine.finalizeMetadataPaths$' /absolute/temp/cpu.pprof
|
||
go tool pprof -top -nodecount=12 -alloc_space /absolute/temp/mem.pprof
|
||
```
|
||
|
||
The before profiled run completed 34 timed operations at 66.339 ms,
|
||
43,468,634 B, and 319,128 allocations per operation. After completed 37 at
|
||
62.460 ms, 42,872,869 B, and 311,703 allocations. These are separate from
|
||
ordinary timing comparisons; raw profile totals include different operation
|
||
counts and must not be compared as per-load costs.
|
||
|
||
The finalizer's cumulative share of sampled allocation bytes changed from
|
||
12.04% to 9.75%. Its line profile now attributes final-index construction to
|
||
one walk, rather than the former two call sites. The remaining
|
||
`effectivePathIndex.func1` belongs to resolution input; its flat allocation
|
||
share changed from 3.67% to 2.31%. `clonePathSegments` remains the largest flat
|
||
site (34.64% before, 33.49% after); required ownership copies remain in place.
|
||
The finalizer's sampled cumulative CPU share was 8.01% before and 7.87% after,
|
||
while `Path.String` changed from 17.21% to 19.10%. These short profiles support
|
||
allocation attribution, not a general CPU-speedup claim.
|
||
|
||
Retain the change. The longer 128-pair run removes approximately 7,430
|
||
allocations and 623 KB per construction; the reduction is reproduced at all
|
||
local sizes and in the consumer. Combining the initial walks adds little
|
||
implementation complexity while retaining both indexes, independent path
|
||
storage, and historical shapes. The possible small timing costs and noisy
|
||
batch results are recorded, not turned into a performance promise. Retained
|
||
heap was not measured separately; no persistent cache or new result lifetime
|
||
is introduced.
|
||
|
||
### Validation and remaining work
|
||
|
||
The two new metadata tests passed against the original production code before
|
||
the refactor and then against the candidate. They verify independently owned
|
||
effective/historical segments in both mutation directions, and retention of
|
||
both numeric-key and sequence-index historical shapes despite repeated tree
|
||
contributions. Existing tests continue to own removed descendants, root and
|
||
escaped paths, detached queries, resolution snapshots, independent loads,
|
||
inheritance, safe explanation, and sensitivity.
|
||
|
||
The focused engine tests for paths, metadata, provenance, queries, resolution,
|
||
inheritance, explanation, sensitivity, secrets, and independence passed.
|
||
`go test -race ./internal/engine` passed. `./scripts/check.sh` passed
|
||
formatting, whitespace, the offline default suite, vet, and builds.
|
||
Documentation changes passed the local-link and whitespace checks.
|
||
|
||
At completion of this focused experiment, Stage 4 had two retained changes and
|
||
Stage 5's complete consumer verification remained outstanding. The final pass
|
||
below completes those checks. No consumer dependency upgrade or release is
|
||
published.
|
||
|
||
## Final consumer verification and handoff
|
||
|
||
Status: Complete; retain both optimizations and conclude this performance
|
||
effort. Measured on 2026-10-09. The maintainer confirmed that the host was very
|
||
busy and selected allocations as the better comparison metric. This pass
|
||
therefore establishes allocation improvements and behavioral compatibility,
|
||
not an elapsed-time or CPU-speedup claim.
|
||
|
||
### Comparison and reproduction
|
||
|
||
- Before: `91ae4fa7eff1571e7c05de01f9e922d75df7453d`, the maintained benchmark
|
||
commit before either optimization. Its production code matches the original
|
||
current-code baseline `f1722274c02c08ff28556dd66f29e0da07a9f2c3`.
|
||
- After: `c546124029645b0635a9babe2d97a74ac77f7f8f`, containing
|
||
`673faea`'s path-buffer sizing and `c546124`'s combined final metadata walk.
|
||
These remain the only production changes relative to the before revision.
|
||
- PromptKit: `eb830bb61346672f0f6c5bce6119b76ccc6d61ac` for both comparisons,
|
||
with its benchmark, adapter, tests, and other dependencies unchanged.
|
||
- Separate disposable ConfigKit clones hold the exact before/after commits;
|
||
only a disposable PromptKit clone's module replacement selects between them.
|
||
Original consumer files and committed dependencies remain unchanged.
|
||
- Toolchain, hardware, runtime settings, and offline/readonly module settings
|
||
match the recorded environment above. Timed runs were sequential and did not
|
||
overlap our correctness checks or profile analysis. Other host work was not
|
||
controlled.
|
||
- All logs and disposable checkouts are outside the repositories at
|
||
`/tmp/configkit-startup-final.jO52hW`. This pass adds no production code,
|
||
dependencies, benchmark fixtures, or release gates.
|
||
|
||
The primary tables use per-metric medians of five longer observations, run
|
||
after, then before. Initial batches ran before, then after with
|
||
`-benchtime=500ms -count=5`; their allocation reductions were reproduced in
|
||
the longer batches. No outliers were removed from the completed paired batches.
|
||
|
||
Run from each ConfigKit copy:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction$' \
|
||
-benchmem -benchtime=1s -count=5
|
||
```
|
||
|
||
In the disposable PromptKit copy, select the relevant ConfigKit copy using
|
||
the Stage 1 `go mod edit -replace` command, then run:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \
|
||
go test -mod=readonly -run '^$' -bench '^BenchmarkConfigurationConstruction$' \
|
||
-benchmem -benchtime=1s -count=5 ./internal/configuration
|
||
```
|
||
|
||
### Combined local construction results
|
||
|
||
The maintained fixture and timing boundaries are unchanged. These are total
|
||
allocated bytes and allocation counts per construction, not retained heap.
|
||
|
||
| Catalog | Before B | After B | Byte reduction | Before allocations | After allocations | Allocation reduction |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| 32 pairs | 11,999,452 | 10,887,026 | 9.27% | 129,136 | 78,605 | 39.13% |
|
||
| 128 pairs | 47,264,285 | 42,917,578 | 9.20% | 513,719 | 311,704 | 39.32% |
|
||
| 512 pairs | 189,216,372 | 171,145,796 | 9.55% | 2,051,983 | 1,244,004 | 39.38% |
|
||
|
||
The reductions hold across all three selected sizes. This is scaling evidence
|
||
for these fixtures, not an asymptotic guarantee. At 128 pairs, the combined
|
||
changes remove approximately 202,000 allocations and 4.35 MB per construction.
|
||
|
||
### Broader PromptKit workloads
|
||
|
||
All fixture preparation, reusable source capture, and fixture files are outside
|
||
timing. Every construction still includes the real adapter's ConfigKit load,
|
||
callbacks, validation, and post-return prompt materialization, compilation,
|
||
hashing, and execution-target capture.
|
||
|
||
- `independent`: 128 profiles and 128 prompt versions; 256 input/effective
|
||
entries in one tier.
|
||
- `file_backed`: 64 profiles and 64 prompt versions; 128 input/effective
|
||
entries in one tier. Message bodies are read through a captured
|
||
`fstest.MapFS` namespace, not a physical-disk benchmark. Native-file behavior
|
||
is separately covered by the configuration tests.
|
||
- `shared_base`: one base profile, 128 sibling-derived profiles, and one prompt
|
||
version; 130 input/effective entries in one tier. Each child uses `extends`
|
||
and overrides temperature while inheriting endpoint, model, and nested
|
||
provider parameters.
|
||
- `layered`: three underlays and main input; 513 input entries become 129
|
||
effective entries across four tiers. Profile fields come from different
|
||
precedence tiers; one prompt selects the final profile.
|
||
|
||
| Workload | Before B | After B | Byte reduction | Before allocations | After allocations | Allocation reduction |
|
||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||
| independent | 50,158,875 | 45,512,786 | 9.26% | 538,542 | 330,484 | 38.63% |
|
||
| file_backed | 27,045,487 | 24,715,761 | 8.61% | 280,679 | 175,854 | 37.35% |
|
||
| shared_base | 30,860,505 | 28,240,215 | 8.49% | 383,695 | 243,357 | 36.58% |
|
||
| layered | 28,415,777 | 25,542,623 | 10.11% | 342,923 | 208,686 | 39.14% |
|
||
|
||
The broader consumer cases reproduce reductions through the existing public
|
||
API. These totals include PromptKit work and must not be presented as isolated
|
||
ConfigKit costs. No consumer source or dependency upgrade was needed.
|
||
|
||
### Complementary allocation evidence
|
||
|
||
The original complementary comparison used
|
||
`-benchtime=200ms -count=3` with the Stage 4 selector for composition,
|
||
broad underlays, inheritance loading, and rendered history. Per-metric medians:
|
||
|
||
| Benchmark / size | Before B | After B | Before allocations | After allocations |
|
||
| --- | ---: | ---: | ---: | ---: |
|
||
| ComposeWideMappings / 128 | 848,509 | 797,562 | 9,658 | 5,962 |
|
||
| ComposeWideMappings / 256 | 1,703,587 | 1,595,291 | 19,652 | 11,860 |
|
||
| ComposeWideMappings / 512 | 3,626,286 | 3,403,288 | 39,643 | 23,659 |
|
||
| InheritanceInspectedLoad / 16 | 876,217 | 832,232 | 11,050 | 7,990 |
|
||
| InheritanceInspectedLoad / 32 | 1,545,193 | 1,460,268 | 19,943 | 14,011 |
|
||
| InheritanceInspectedLoad / 64 | 2,906,050 | 2,739,206 | 37,778 | 26,098 |
|
||
| UnderlayBroadCatalog / 128 | 1,265,034 | 1,218,918 | 15,904 | 12,370 |
|
||
| UnderlayBroadCatalog / 256 | 2,581,008 | 2,437,650 | 33,473 | 24,307 |
|
||
| UnderlayBroadCatalog / 512 | 5,269,009 | 4,931,073 | 68,581 | 48,151 |
|
||
| UnderlayRenderedHistory / 32 | 504,551 | 503,463 | 3,294 | 3,221 |
|
||
| UnderlayRenderedHistory / 64 | 1,006,071 | 1,003,952 | 6,529 | 6,392 |
|
||
| UnderlayRenderedHistory / 128 | 2,031,859 | 2,027,696 | 12,997 | 12,732 |
|
||
|
||
Every case allocates less. Composition exercises the retained path-formatting
|
||
change but not final metadata indexing. History rendering loads outside timing;
|
||
its small allocation reduction comes from path formatting, not the fused load
|
||
traversal.
|
||
|
||
Apparent timing regressions prompted five-observation reverse-order repeats of
|
||
broad-underlay / 128 and / 256 and rendered-history / 32, / 64, and / 128 using
|
||
`-benchtime=500ms -count=5` and selector
|
||
`^(BenchmarkUnderlayBroadCatalog|BenchmarkUnderlayRenderedHistory)$/^(32|64|128|256)$`.
|
||
Their allocation-count medians exactly match the corresponding initial rows,
|
||
and allocated-byte reductions recur. No persistent cache or result-lifetime
|
||
change was introduced; retained heap was not measured separately.
|
||
|
||
### Timing limitation
|
||
|
||
Timing varied greatly without source changes. For example, the final
|
||
candidate's local / 32 median was 45.098 ms in the initial batch and 17.362 ms
|
||
in the longer batch. Both showed approximately 78,605 allocations. The initial
|
||
baseline's / 512 observations ranged from 574.500 to 2,027.077 ms, whereas its
|
||
longer repeat ranged from 261.986 to 269.806 ms.
|
||
|
||
Complementary timing repeats also produced large apparent regressions. Before
|
||
the maintainer's busy-host notice, one further candidate-only diagnostic
|
||
completed: broad-underlay / 128 returned from a 7.556 ms median to 1.589 ms,
|
||
and rendered-history / 32 from 2.386 ms to 0.306 ms, with unchanged source and
|
||
allocation counts. The subsequent baseline diagnostic was intentionally
|
||
interrupted, and further timing comparisons stopped. Its partial output is not
|
||
included in matched results; the interrupted diagnostic is not a test failure.
|
||
|
||
Raw logs retain elapsed-time observations, but this pass establishes neither
|
||
a general speedup nor a reproducible timing regression attributable to the
|
||
changes. Optional quiet-host timing work can refine that assessment. It is not
|
||
a remaining requirement for this allocation-focused outcome or a release gate.
|
||
|
||
### Behavioral verification and final decision
|
||
|
||
Both ConfigKit selections passed PromptKit's complete configuration-package
|
||
tests and 15 selected public contract tests:
|
||
|
||
```sh
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off \
|
||
go test -mod=readonly -count=1 ./internal/configuration
|
||
|
||
GOTOOLCHAIN=local GOWORK=off GOPROXY=off \
|
||
go test -mod=readonly -count=1 \
|
||
-run '^Test(ConfigurationInspection|PublicConstruction|ProfileInheritance|EngineCapturesPromptContent|PreparationStillReads)' .
|
||
```
|
||
|
||
Those checks exercise final effective values, explicit zero/null/omission,
|
||
normalization history, strict conflicts and tier precedence, imported and
|
||
native file content, source-context ownership and closing, public inheritance,
|
||
required credentials, error identities, captured content, and redacted
|
||
inspection without provider calls. Expectations and fixtures were unchanged.
|
||
|
||
ConfigKit's applicable semantic, ownership, race, facade, and everyday-check
|
||
results from Stage 4 apply to the identical production candidate. This pass
|
||
reuses those successful checks; its documentation updates passed local-link
|
||
and whitespace validation. Component responsibilities and documented
|
||
internal flow remain accurate, so no architecture, ADR, public-contract, or
|
||
internal-document change is needed.
|
||
|
||
Retain both changes. They remove redundant buffer growth and final-tree
|
||
traversal while preserving separately owned metadata indexes and the existing
|
||
configuration model. Allocation reductions reproduce locally and across the
|
||
recorded consumer's independent, content-file, inheritance, and layered
|
||
workloads, with existing behavioral tests passing. No further optimization is
|
||
selected for this effort.
|
||
|
||
Stages 4–6 are complete. No required consumer verification remains for this
|
||
scope; timing and retained-heap limits are explicit. Release preparation,
|
||
roadmap retirement, publication, and any consumer dependency update remain
|
||
separate work.
|