From 48619de024d794cf41e90b171b7fcbe609dcd437 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 9 Oct 2026 01:19:13 +0000 Subject: [PATCH] Verify startup loading improvements --- docs/roadmap/startup-benchmark.md | 205 +++++++++++++++++++++++++++++- docs/roadmap/startup.md | 21 ++- 2 files changed, 214 insertions(+), 12 deletions(-) diff --git a/docs/roadmap/startup-benchmark.md b/docs/roadmap/startup-benchmark.md index f8dc7fc..3b36b6e 100644 --- a/docs/roadmap/startup-benchmark.md +++ b/docs/roadmap/startup-benchmark.md @@ -1,7 +1,8 @@ # Startup construction measurements and focused experiments -Status: Stages 1–3 complete; Stage 4 is in progress with two retained experiments. -Complete consumer-workload verification remains outstanding. +Status: Complete. Both focused optimizations are retained; final consumer +verification and handoff are recorded below. Acceptance rests on allocation +reductions and behavioral checks; busy-host timing remains inconclusive. Measured on 2026-10-09 under the [startup roadmap](startup.md). The [consumer request](promptkit-performance-request.md) owns the submitted workload @@ -725,7 +726,199 @@ inheritance, explanation, sensitivity, secrets, and independence passed. formatting, whitespace, the offline default suite, vet, and builds. Documentation changes passed the local-link and whitespace checks. -This focused experiment is complete. Stage 4 has two retained changes, while -Stage 5's complete consumer verification remains outstanding: PromptKit's -file-backed, shared-base, and layered fixtures and its applicable configuration -tests. No consumer dependency upgrade or release is published. +At completion of this focused experiment, Stage 4 had two retained changes and +Stage 5's complete consumer verification remained outstanding. The final pass +below completes those checks. No consumer dependency upgrade or release is +published. + +## Final consumer verification and handoff + +Status: Complete; retain both optimizations and conclude this performance +effort. Measured on 2026-10-09. The maintainer confirmed that the host was very +busy and selected allocations as the better comparison metric. This pass +therefore establishes allocation improvements and behavioral compatibility, +not an elapsed-time or CPU-speedup claim. + +### Comparison and reproduction + +- Before: `91ae4fa7eff1571e7c05de01f9e922d75df7453d`, the maintained benchmark + commit before either optimization. Its production code matches the original + current-code baseline `f1722274c02c08ff28556dd66f29e0da07a9f2c3`. +- After: `c546124029645b0635a9babe2d97a74ac77f7f8f`, containing + `673faea`'s path-buffer sizing and `c546124`'s combined final metadata walk. + These remain the only production changes relative to the before revision. +- PromptKit: `eb830bb61346672f0f6c5bce6119b76ccc6d61ac` for both comparisons, + with its benchmark, adapter, tests, and other dependencies unchanged. +- Separate disposable ConfigKit clones hold the exact before/after commits; + only a disposable PromptKit clone's module replacement selects between them. + Original consumer files and committed dependencies remain unchanged. +- Toolchain, hardware, runtime settings, and offline/readonly module settings + match the recorded environment above. Timed runs were sequential and did not + overlap our correctness checks or profile analysis. Other host work was not + controlled. +- All logs and disposable checkouts are outside the repositories at + `/tmp/configkit-startup-final.jO52hW`. This pass adds no production code, + dependencies, benchmark fixtures, or release gates. + +The primary tables use per-metric medians of five longer observations, run +after, then before. Initial batches ran before, then after with +`-benchtime=500ms -count=5`; their allocation reductions were reproduced in +the longer batches. No outliers were removed from the completed paired batches. + +Run from each ConfigKit copy: + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \ + go test -mod=readonly -run '^$' -bench '^BenchmarkStartupConstruction$' \ + -benchmem -benchtime=1s -count=5 +``` + +In the disposable PromptKit copy, select the relevant ConfigKit copy using +the Stage 1 `go mod edit -replace` command, then run: + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off GOMAXPROCS=4 \ + go test -mod=readonly -run '^$' -bench '^BenchmarkConfigurationConstruction$' \ + -benchmem -benchtime=1s -count=5 ./internal/configuration +``` + +### Combined local construction results + +The maintained fixture and timing boundaries are unchanged. These are total +allocated bytes and allocation counts per construction, not retained heap. + +| Catalog | Before B | After B | Byte reduction | Before allocations | After allocations | Allocation reduction | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| 32 pairs | 11,999,452 | 10,887,026 | 9.27% | 129,136 | 78,605 | 39.13% | +| 128 pairs | 47,264,285 | 42,917,578 | 9.20% | 513,719 | 311,704 | 39.32% | +| 512 pairs | 189,216,372 | 171,145,796 | 9.55% | 2,051,983 | 1,244,004 | 39.38% | + +The reductions hold across all three selected sizes. This is scaling evidence +for these fixtures, not an asymptotic guarantee. At 128 pairs, the combined +changes remove approximately 202,000 allocations and 4.35 MB per construction. + +### Broader PromptKit workloads + +All fixture preparation, reusable source capture, and fixture files are outside +timing. Every construction still includes the real adapter's ConfigKit load, +callbacks, validation, and post-return prompt materialization, compilation, +hashing, and execution-target capture. + +- `independent`: 128 profiles and 128 prompt versions; 256 input/effective + entries in one tier. +- `file_backed`: 64 profiles and 64 prompt versions; 128 input/effective + entries in one tier. Message bodies are read through a captured + `fstest.MapFS` namespace, not a physical-disk benchmark. Native-file behavior + is separately covered by the configuration tests. +- `shared_base`: one base profile, 128 sibling-derived profiles, and one prompt + version; 130 input/effective entries in one tier. Each child uses `extends` + and overrides temperature while inheriting endpoint, model, and nested + provider parameters. +- `layered`: three underlays and main input; 513 input entries become 129 + effective entries across four tiers. Profile fields come from different + precedence tiers; one prompt selects the final profile. + +| Workload | Before B | After B | Byte reduction | Before allocations | After allocations | Allocation reduction | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| independent | 50,158,875 | 45,512,786 | 9.26% | 538,542 | 330,484 | 38.63% | +| file_backed | 27,045,487 | 24,715,761 | 8.61% | 280,679 | 175,854 | 37.35% | +| shared_base | 30,860,505 | 28,240,215 | 8.49% | 383,695 | 243,357 | 36.58% | +| layered | 28,415,777 | 25,542,623 | 10.11% | 342,923 | 208,686 | 39.14% | + +The broader consumer cases reproduce reductions through the existing public +API. These totals include PromptKit work and must not be presented as isolated +ConfigKit costs. No consumer source or dependency upgrade was needed. + +### Complementary allocation evidence + +The original complementary comparison used +`-benchtime=200ms -count=3` with the Stage 4 selector for composition, +broad underlays, inheritance loading, and rendered history. Per-metric medians: + +| Benchmark / size | Before B | After B | Before allocations | After allocations | +| --- | ---: | ---: | ---: | ---: | +| ComposeWideMappings / 128 | 848,509 | 797,562 | 9,658 | 5,962 | +| ComposeWideMappings / 256 | 1,703,587 | 1,595,291 | 19,652 | 11,860 | +| ComposeWideMappings / 512 | 3,626,286 | 3,403,288 | 39,643 | 23,659 | +| InheritanceInspectedLoad / 16 | 876,217 | 832,232 | 11,050 | 7,990 | +| InheritanceInspectedLoad / 32 | 1,545,193 | 1,460,268 | 19,943 | 14,011 | +| InheritanceInspectedLoad / 64 | 2,906,050 | 2,739,206 | 37,778 | 26,098 | +| UnderlayBroadCatalog / 128 | 1,265,034 | 1,218,918 | 15,904 | 12,370 | +| UnderlayBroadCatalog / 256 | 2,581,008 | 2,437,650 | 33,473 | 24,307 | +| UnderlayBroadCatalog / 512 | 5,269,009 | 4,931,073 | 68,581 | 48,151 | +| UnderlayRenderedHistory / 32 | 504,551 | 503,463 | 3,294 | 3,221 | +| UnderlayRenderedHistory / 64 | 1,006,071 | 1,003,952 | 6,529 | 6,392 | +| UnderlayRenderedHistory / 128 | 2,031,859 | 2,027,696 | 12,997 | 12,732 | + +Every case allocates less. Composition exercises the retained path-formatting +change but not final metadata indexing. History rendering loads outside timing; +its small allocation reduction comes from path formatting, not the fused load +traversal. + +Apparent timing regressions prompted five-observation reverse-order repeats of +broad-underlay / 128 and / 256 and rendered-history / 32, / 64, and / 128 using +`-benchtime=500ms -count=5` and selector +`^(BenchmarkUnderlayBroadCatalog|BenchmarkUnderlayRenderedHistory)$/^(32|64|128|256)$`. +Their allocation-count medians exactly match the corresponding initial rows, +and allocated-byte reductions recur. No persistent cache or result-lifetime +change was introduced; retained heap was not measured separately. + +### Timing limitation + +Timing varied greatly without source changes. For example, the final +candidate's local / 32 median was 45.098 ms in the initial batch and 17.362 ms +in the longer batch. Both showed approximately 78,605 allocations. The initial +baseline's / 512 observations ranged from 574.500 to 2,027.077 ms, whereas its +longer repeat ranged from 261.986 to 269.806 ms. + +Complementary timing repeats also produced large apparent regressions. Before +the maintainer's busy-host notice, one further candidate-only diagnostic +completed: broad-underlay / 128 returned from a 7.556 ms median to 1.589 ms, +and rendered-history / 32 from 2.386 ms to 0.306 ms, with unchanged source and +allocation counts. The subsequent baseline diagnostic was intentionally +interrupted, and further timing comparisons stopped. Its partial output is not +included in matched results; the interrupted diagnostic is not a test failure. + +Raw logs retain elapsed-time observations, but this pass establishes neither +a general speedup nor a reproducible timing regression attributable to the +changes. Optional quiet-host timing work can refine that assessment. It is not +a remaining requirement for this allocation-focused outcome or a release gate. + +### Behavioral verification and final decision + +Both ConfigKit selections passed PromptKit's complete configuration-package +tests and 15 selected public contract tests: + +```sh +GOTOOLCHAIN=local GOWORK=off GOPROXY=off \ + go test -mod=readonly -count=1 ./internal/configuration + +GOTOOLCHAIN=local GOWORK=off GOPROXY=off \ + go test -mod=readonly -count=1 \ + -run '^Test(ConfigurationInspection|PublicConstruction|ProfileInheritance|EngineCapturesPromptContent|PreparationStillReads)' . +``` + +Those checks exercise final effective values, explicit zero/null/omission, +normalization history, strict conflicts and tier precedence, imported and +native file content, source-context ownership and closing, public inheritance, +required credentials, error identities, captured content, and redacted +inspection without provider calls. Expectations and fixtures were unchanged. + +ConfigKit's applicable semantic, ownership, race, facade, and everyday-check +results from Stage 4 apply to the identical production candidate. This pass +reuses those successful checks; its documentation updates passed local-link +and whitespace validation. Component responsibilities and documented +internal flow remain accurate, so no architecture, ADR, public-contract, or +internal-document change is needed. + +Retain both changes. They remove redundant buffer growth and final-tree +traversal while preserving separately owned metadata indexes and the existing +configuration model. Allocation reductions reproduce locally and across the +recorded consumer's independent, content-file, inheritance, and layered +workloads, with existing behavioral tests passing. No further optimization is +selected for this effort. + +Stages 4–6 are complete. No required consumer verification remains for this +scope; timing and retained-heap limits are explicit. Release preparation, +roadmap retirement, publication, and any consumer dependency update remain +separate work. diff --git a/docs/roadmap/startup.md b/docs/roadmap/startup.md index 8905f34..2a420a1 100644 --- a/docs/roadmap/startup.md +++ b/docs/roadmap/startup.md @@ -1,7 +1,7 @@ # Startup Loading Performance Roadmap -Status: Stages 1–3 complete; Stage 4 is in progress with two optimizations -retained. Findings and reproduction commands are in +Status: Complete. Two optimizations are retained, with final consumer +verification and handoff recorded. Findings and reproduction commands are in [startup-benchmark.md](startup-benchmark.md). Audience: ConfigKit maintainers implementing and reviewing this work. @@ -122,11 +122,11 @@ finding and recommend whether further investigation is justified. ### 4. Implement and measure focused changes -Status: In progress. `Path.String` buffer sizing and combined final metadata +Status: Complete. `Path.String` buffer sizing and combined final metadata traversal are retained after fresh construction, consumer, inspection, and profile comparisons. See the [path-formatting experiment](startup-benchmark.md#focused-experiment-pre-size-pathstring) and [metadata-traversal experiment](startup-benchmark.md#focused-experiment-combine-final-metadata-traversal). -Complete consumer-workload verification remains outstanding. +No further optimization is selected for this effort. 1. Implement the selected internal change while preserving the supported API and established behavior. Audit any reduced copying or reused data for @@ -152,7 +152,12 @@ justify their complexity; a predetermined optimization count is unnecessary. ### 5. Verify the complete result against consumer workloads -Status: Not started. +Status: Complete. The combined candidate was compared with the pre-optimization +baseline across all local sizes, complementary benchmarks, and all four recorded +PromptKit workloads. Consumer configuration and public contract tests passed. +Acceptance rests on reproducible allocation reductions; the maintainer confirmed +that busy-host timing is not a reliable comparison here. See the +[final verification](startup-benchmark.md#final-consumer-verification-and-handoff). 1. Compare the final candidate with the unoptimized current-code baseline using the local fixture and relevant existing benchmarks. Confirm that gains hold @@ -175,7 +180,11 @@ benefit has been verified. ### 6. Record the outcome and hand off -Status: Not started. +Status: Complete. Retained changes, revisions, allocation comparisons, +reproduction commands, behavioral checks, and timing/retained-heap limitations +are recorded in the [final handoff](startup-benchmark.md#final-consumer-verification-and-handoff). +No required verification remains for this scope. Release preparation and +publication remain separate work. Summarize the retained changes, before/after results, revisions, reproduction commands, validation, tradeoffs, and unresolved questions in this roadmap.