eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add an ordered roadmap for startup loading performance

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-08 23:58:35 +00:00
parent d53021ea2c
commit f1722274c0
1 changed files with 182 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

182
docs/roadmap/startup.md Normal file
View File

@@ -0,0 +1,182 @@
# Startup Loading Performance Roadmap
Status: Approach accepted; execution has not started.
Audience: ConfigKit maintainers implementing and reviewing this work.
## Objective and scope
Reduce avoidable CPU work and temporary allocations during configuration
construction through the existing public loading API. The
[PromptKit request](promptkit-performance-request.md) owns the downstream
workload, original measurements, profiling observations, and reproduction
instructions. Its measurements include PromptKit adapter work; they are not
standalone ConfigKit costs or evidence of retained-memory growth.
The initial scope is measurement and internal optimization. Preserve the
[configuration contract](../config.md), the public facade, and the ownership
boundaries described by the [architecture policy](../policy/architecture.md).
In particular, improvements must preserve presence and explicit null, source
contexts, provenance and history, recorded resolution, safe inspection, secret
handling, and independence of callbacks and published results. Those documents
remain the owners of the exact requirements.
Choose implementations from measured evidence. Path copying, pointer
formatting/parsing, and repeated metadata/index construction are initial
investigation candidates, not prescribed solutions. This roadmap does not select
a cache, alternate metadata mode, new public API, fixed speedup, allocation budget,
complexity guarantee, or additional release gate. Any need to expand scope must
be assessed against the consumer evidence and existing policies before adoption.
## Execution order
Execute the stages below in order. Update each stage's status and record its
evidence here as work proceeds. Within stage 4, complete and measure one focused
change before selecting the next. If measurements invalidate a hypothesis, return
to stage 3 rather than carrying an unsupported design forward. Follow the
[development guide](../development.md) and its task-specific policy reading guide
throughout execution.
### 1. Establish the downstream and current-code baselines
Status: Not started.
1. Inspect PromptKit's recorded benchmark and configuration adapter at the
revision identified in its request. Identify which work is timed, including
source construction, callbacks, and processing after ConfigKit returns.
2. Reproduce the independent fixture with the request's ConfigKit dependency,
then with the current ConfigKit revision, using the same PromptKit revision,
environment, and benchmark settings. Use isolated checkouts or temporary
module wiring without changing committed consumer dependencies.
3. Record exact revisions, toolchain, platform, processor, process parallelism,
fixture size, setup/timing boundaries, commands, and repeated results. Report
elapsed time, allocated bytes, and allocations per construction separately.
Keep generated profiles and binaries outside the repositories.
4. Explain material differences from the submitted measurements before using
them to attribute costs. If the recorded consumer checkout or dependencies
are unavailable, document the missing evidence and continue with the local
benchmark stages; downstream verification remains outstanding.
Completion evidence: A reproducible baseline distinguishing the submitted
version from current code, or an explicit record of unavailable downstream
evidence. Do not attribute all adapter costs to ConfigKit.
### 2. Add a representative ConfigKit construction benchmark
Status: Not started.
1. Add a maintained benchmark through the root public API, using a synthetic
configuration shape representative of PromptKit's profiles and versioned
prompts. Exercise mapping input, typed decoding, resolution, validation, and
publication. Match the relevant callback work observed in stage 1 without
importing PromptKit or reproducing its runtime machinery.
2. Include the reported fixture size and smaller and larger catalogs to measure
scaling. Record both definition counts and the configuration shape; definition
counts alone do not describe all fields and metadata paths.
3. Make fixture setup and timed source-construction boundaries explicit. Keep
unrelated fixture generation outside timing and align comparisons on the
work they actually include. Confirm the fixture loads successfully and
exercises the intended behavior before collecting results.
4. Reuse the existing assembly, underlay, inheritance, and inspection benchmarks
where they supply complementary evidence. Add related cases only when they
cover a material gap, such as the callback workload or the public facade.
Completion evidence: Repeatable local construction measurements at several
sizes, with commands and fixture definitions recorded. The benchmark remains
offline and independent of credentials. Follow the
[testing policy](../policy/testing.md); timings and allocation counts are
measurements, not machine-specific pass/fail assertions.
### 3. Profile and choose the first optimization
Status: Not started.
1. Collect CPU and allocation profiles for the representative local fixture.
Compare them with the downstream profile where available, distinguishing
ConfigKit loading costs from adapter work. Keep profiled runs separate from
ordinary timing comparisons.
2. Trace the dominant costs through the current implementation. Begin with path
construction and cloning, repeated pointer formatting/parsing, and repeated
metadata traversal or index construction. Inspect both the engine and root
conversion boundary where profiles point to them.
3. Distinguish required ownership separation from redundant internal work.
Assess simple local changes before introducing shared state or a broader
representation change. Preserve contextual mapping-key versus sequence-index
semantics when considering path reuse.
4. Select the smallest useful change supported by the profile. Record the
hypothesis, affected boundaries, expected measurement signal, and existing
behavioral tests that protect those boundaries. Assess scaling from the
measurements rather than inferring it from one fixture size.
Completion evidence: A concrete first optimization with a measured motivation
and proportionate validation plan. If no worthwhile change emerges, record that
finding and recommend whether further investigation is justified.
### 4. Implement and measure focused changes
Status: Not started.
1. Implement the selected internal change while preserving the supported API
and established behavior. Audit any reduced copying or reused data for
aliases across traversal branches, callbacks, queries, and independent loads.
2. Run the relevant existing behavioral tests. Add targeted coverage only for
a consequential risk not already protected, including contextual path
semantics, history, ownership, or secret redaction where affected. Do not
rewrite behavioral expectations merely to accommodate an internal refactor.
3. Compare repeated before/after measurements under matching conditions at all
selected sizes. Reprofile when needed to explain the result. Measure affected
inspection operations as well as loading; if reuse or caching changes data
lifetime, examine retained memory separately from cumulative allocations.
4. Keep changes with repeatable benefit and acceptable complexity. Investigate
regressions, discard unsuccessful experiments, and return to stage 3 for any
next candidate. Record the evidence for each retained change.
5. Complete the applicable development checks before committing each finished
unit under the [agent policy](../policy/agents.md). Use the development guide's
race validation when ownership or concurrency changes warrant it.
Completion evidence: Reviewed, validated changes with attributable improvements
and recorded tradeoffs. Stop iterating when remaining candidates no longer
justify their complexity; a predetermined optimization count is unnecessary.
### 5. Verify the complete result against consumer workloads
Status: Not started.
1. Compare the final candidate with the unoptimized current-code baseline using
the local fixture and relevant existing benchmarks. Confirm that gains hold
across the selected sizes and inspect any material regressions.
2. Repeat PromptKit's independent fixture with the candidate under the same
conditions as stage 1. Also run its relevant file-backed, inheritance, and
layered fixtures, and applicable consumer configuration tests. Record the
exact candidate wiring and separate downstream gains from isolated ConfigKit
gains. Do not update or publish a consumer dependency as part of this work.
3. Confirm that final configuration, presence, provenance/history, inspection,
and secret behavior remain consistent through the affected paths. Complete
the everyday development check and any additional checks justified by the
changes, reusing successful results for unchanged content.
Completion evidence: Repeatable reductions for the representative construction
workload, with related-workload results, semantic validation, and any regressions
or limitations explained. If downstream verification is unavailable, identify
the exact outstanding check and report local results without claiming PromptKit
benefit has been verified.
### 6. Record the outcome and hand off
Status: Not started.
Summarize the retained changes, before/after results, revisions, reproduction
commands, validation, tradeoffs, and unresolved questions in this roadmap.
Separate completed work from outstanding consumer checks. Recommend adoption
based on the measured benefit and maintenance cost, without promising a
machine-independent threshold.
Update implemented internal documentation only where responsibilities or flow
changed. Keep benchmark evidence here and behavior in its canonical owners;
follow the [documentation policy](../policy/documentation.md). Publication and
release preparation remain separate work under the existing release procedure.
Completion evidence: A reviewable handoff that makes the improvement and its
limits clear. Mark the roadmap complete only when the applicable implementation
and verification work is finished; retire it during release preparation under
the documentation policy.