190 lines
11 KiB
Markdown
190 lines
11 KiB
Markdown
# Startup Loading Performance Roadmap
|
||
|
||
Status: Stages 1–3 complete; optimization has not started. Findings and
|
||
reproduction commands are in [startup-benchmark.md](startup-benchmark.md).
|
||
|
||
Audience: ConfigKit maintainers implementing and reviewing this work.
|
||
|
||
## Objective and scope
|
||
|
||
Reduce avoidable CPU work and temporary allocations during configuration
|
||
construction through the existing public loading API. The
|
||
[PromptKit request](promptkit-performance-request.md) owns the downstream
|
||
workload, original measurements, profiling observations, and reproduction
|
||
instructions. Its measurements include PromptKit adapter work; they are not
|
||
standalone ConfigKit costs or evidence of retained-memory growth.
|
||
|
||
The initial scope is measurement and internal optimization. Preserve the
|
||
[configuration contract](../config.md), the public facade, and the ownership
|
||
boundaries described by the [architecture policy](../policy/architecture.md).
|
||
In particular, improvements must preserve presence and explicit null, source
|
||
contexts, provenance and history, recorded resolution, safe inspection, secret
|
||
handling, and independence of callbacks and published results. Those documents
|
||
remain the owners of the exact requirements.
|
||
|
||
Choose implementations from measured evidence. Path copying, pointer
|
||
formatting/parsing, and repeated metadata/index construction are initial
|
||
investigation candidates, not prescribed solutions. This roadmap does not select
|
||
a cache, alternate metadata mode, new public API, fixed speedup, allocation budget,
|
||
complexity guarantee, or additional release gate. Any need to expand scope must
|
||
be assessed against the consumer evidence and existing policies before adoption.
|
||
|
||
## Execution order
|
||
|
||
Execute the stages below in order. Update each stage's status and record its
|
||
evidence here as work proceeds. Within stage 4, complete and measure one focused
|
||
change before selecting the next. If measurements invalidate a hypothesis, return
|
||
to stage 3 rather than carrying an unsupported design forward. Follow the
|
||
[development guide](../development.md) and its task-specific policy reading guide
|
||
throughout execution.
|
||
|
||
### 1. Establish the downstream and current-code baselines
|
||
|
||
Status: Complete. The recorded PromptKit revision was measured with v0.5.0 and
|
||
current ConfigKit; allocation costs were reproduced. See the
|
||
[downstream results](startup-benchmark.md#stage-1-downstream-reproduction).
|
||
|
||
1. Inspect PromptKit's recorded benchmark and configuration adapter at the
|
||
revision identified in its request. Identify which work is timed, including
|
||
source construction, callbacks, and processing after ConfigKit returns.
|
||
2. Reproduce the independent fixture with the request's ConfigKit dependency,
|
||
then with the current ConfigKit revision, using the same PromptKit revision,
|
||
environment, and benchmark settings. Use isolated checkouts or temporary
|
||
module wiring without changing committed consumer dependencies.
|
||
3. Record exact revisions, toolchain, platform, processor, process parallelism,
|
||
fixture size, setup/timing boundaries, commands, and repeated results. Report
|
||
elapsed time, allocated bytes, and allocations per construction separately.
|
||
Keep generated profiles and binaries outside the repositories.
|
||
4. Explain material differences from the submitted measurements before using
|
||
them to attribute costs. If the recorded consumer checkout or dependencies
|
||
are unavailable, document the missing evidence and continue with the local
|
||
benchmark stages; downstream verification remains outstanding.
|
||
|
||
Completion evidence: A reproducible baseline distinguishing the submitted
|
||
version from current code, or an explicit record of unavailable downstream
|
||
evidence. Do not attribute all adapter costs to ConfigKit.
|
||
|
||
### 2. Add a representative ConfigKit construction benchmark
|
||
|
||
Status: Complete. A maintained root-API benchmark measures 32, 128, and 512
|
||
profile/prompt pairs, with complementary existing benchmarks. See the
|
||
[local results](startup-benchmark.md#stage-2-maintained-public-api-benchmark).
|
||
|
||
1. Add a maintained benchmark through the root public API, using a synthetic
|
||
configuration shape representative of PromptKit's profiles and versioned
|
||
prompts. Exercise mapping input, typed decoding, resolution, validation, and
|
||
publication. Match the relevant callback work observed in stage 1 without
|
||
importing PromptKit or reproducing its runtime machinery.
|
||
2. Include the reported fixture size and smaller and larger catalogs to measure
|
||
scaling. Record both definition counts and the configuration shape; definition
|
||
counts alone do not describe all fields and metadata paths.
|
||
3. Make fixture setup and timed source-construction boundaries explicit. Keep
|
||
unrelated fixture generation outside timing and align comparisons on the
|
||
work they actually include. Confirm the fixture loads successfully and
|
||
exercises the intended behavior before collecting results.
|
||
4. Reuse the existing assembly, underlay, inheritance, and inspection benchmarks
|
||
where they supply complementary evidence. Add related cases only when they
|
||
cover a material gap, such as the callback workload or the public facade.
|
||
|
||
Completion evidence: Repeatable local construction measurements at several
|
||
sizes, with commands and fixture definitions recorded. The benchmark remains
|
||
offline and independent of credentials. Follow the
|
||
[testing policy](../policy/testing.md); timings and allocation counts are
|
||
measurements, not machine-specific pass/fail assertions.
|
||
|
||
### 3. Profile and choose the first optimization
|
||
|
||
Status: Complete. CPU/allocation profiles and the source trace identify
|
||
`Path.String` buffer sizing as the first experiment. See the
|
||
[profile evidence and proposal](startup-benchmark.md#stage-3-profiles-and-implementation-trace).
|
||
|
||
1. Collect CPU and allocation profiles for the representative local fixture.
|
||
Compare them with the downstream profile where available, distinguishing
|
||
ConfigKit loading costs from adapter work. Keep profiled runs separate from
|
||
ordinary timing comparisons.
|
||
2. Trace the dominant costs through the current implementation. Begin with path
|
||
construction and cloning, repeated pointer formatting/parsing, and repeated
|
||
metadata traversal or index construction. Inspect both the engine and root
|
||
conversion boundary where profiles point to them.
|
||
3. Distinguish required ownership separation from redundant internal work.
|
||
Assess simple local changes before introducing shared state or a broader
|
||
representation change. Preserve contextual mapping-key versus sequence-index
|
||
semantics when considering path reuse.
|
||
4. Select the smallest useful change supported by the profile. Record the
|
||
hypothesis, affected boundaries, expected measurement signal, and existing
|
||
behavioral tests that protect those boundaries. Assess scaling from the
|
||
measurements rather than inferring it from one fixture size.
|
||
|
||
Completion evidence: A concrete first optimization with a measured motivation
|
||
and proportionate validation plan. If no worthwhile change emerges, record that
|
||
finding and recommend whether further investigation is justified.
|
||
|
||
### 4. Implement and measure focused changes
|
||
|
||
Status: Not started.
|
||
|
||
1. Implement the selected internal change while preserving the supported API
|
||
and established behavior. Audit any reduced copying or reused data for
|
||
aliases across traversal branches, callbacks, queries, and independent loads.
|
||
2. Run the relevant existing behavioral tests. Add targeted coverage only for
|
||
a consequential risk not already protected, including contextual path
|
||
semantics, history, ownership, or secret redaction where affected. Do not
|
||
rewrite behavioral expectations merely to accommodate an internal refactor.
|
||
3. Compare repeated before/after measurements under matching conditions at all
|
||
selected sizes. Reprofile when needed to explain the result. Measure affected
|
||
inspection operations as well as loading; if reuse or caching changes data
|
||
lifetime, examine retained memory separately from cumulative allocations.
|
||
4. Keep changes with repeatable benefit and acceptable complexity. Investigate
|
||
regressions, discard unsuccessful experiments, and return to stage 3 for any
|
||
next candidate. Record the evidence for each retained change.
|
||
5. Complete the applicable development checks before committing each finished
|
||
unit under the [agent policy](../policy/agents.md). Use the development guide's
|
||
race validation when ownership or concurrency changes warrant it.
|
||
|
||
Completion evidence: Reviewed, validated changes with attributable improvements
|
||
and recorded tradeoffs. Stop iterating when remaining candidates no longer
|
||
justify their complexity; a predetermined optimization count is unnecessary.
|
||
|
||
### 5. Verify the complete result against consumer workloads
|
||
|
||
Status: Not started.
|
||
|
||
1. Compare the final candidate with the unoptimized current-code baseline using
|
||
the local fixture and relevant existing benchmarks. Confirm that gains hold
|
||
across the selected sizes and inspect any material regressions.
|
||
2. Repeat PromptKit's independent fixture with the candidate under the same
|
||
conditions as stage 1. Also run its relevant file-backed, inheritance, and
|
||
layered fixtures, and applicable consumer configuration tests. Record the
|
||
exact candidate wiring and separate downstream gains from isolated ConfigKit
|
||
gains. Do not update or publish a consumer dependency as part of this work.
|
||
3. Confirm that final configuration, presence, provenance/history, inspection,
|
||
and secret behavior remain consistent through the affected paths. Complete
|
||
the everyday development check and any additional checks justified by the
|
||
changes, reusing successful results for unchanged content.
|
||
|
||
Completion evidence: Repeatable reductions for the representative construction
|
||
workload, with related-workload results, semantic validation, and any regressions
|
||
or limitations explained. If downstream verification is unavailable, identify
|
||
the exact outstanding check and report local results without claiming PromptKit
|
||
benefit has been verified.
|
||
|
||
### 6. Record the outcome and hand off
|
||
|
||
Status: Not started.
|
||
|
||
Summarize the retained changes, before/after results, revisions, reproduction
|
||
commands, validation, tradeoffs, and unresolved questions in this roadmap.
|
||
Separate completed work from outstanding consumer checks. Recommend adoption
|
||
based on the measured benefit and maintenance cost, without promising a
|
||
machine-independent threshold.
|
||
|
||
Update implemented internal documentation only where responsibilities or flow
|
||
changed. Keep benchmark evidence here and behavior in its canonical owners;
|
||
follow the [documentation policy](../policy/documentation.md). Publication and
|
||
release preparation remain separate work under the existing release procedure.
|
||
|
||
Completion evidence: A reviewable handoff that makes the improvement and its
|
||
limits clear. Mark the roadmap complete only when the applicable implementation
|
||
and verification work is finished; retire it during release preparation under
|
||
the documentation policy.
|