204 lines
12 KiB
Markdown
204 lines
12 KiB
Markdown
# Startup Loading Performance Roadmap
|
|
|
|
Status: Complete. Two optimizations are retained, with final consumer
|
|
verification and handoff recorded. Findings and reproduction commands are in
|
|
[startup-benchmark.md](startup-benchmark.md).
|
|
|
|
Audience: ConfigKit maintainers implementing and reviewing this work.
|
|
|
|
## Objective and scope
|
|
|
|
Reduce avoidable CPU work and temporary allocations during configuration
|
|
construction through the existing public loading API. The
|
|
[PromptKit request](promptkit-performance-request.md) owns the downstream
|
|
workload, original measurements, profiling observations, and reproduction
|
|
instructions. Its measurements include PromptKit adapter work; they are not
|
|
standalone ConfigKit costs or evidence of retained-memory growth.
|
|
|
|
The initial scope is measurement and internal optimization. Preserve the
|
|
[configuration contract](../config.md), the public facade, and the ownership
|
|
boundaries described by the [architecture policy](../policy/architecture.md).
|
|
In particular, improvements must preserve presence and explicit null, source
|
|
contexts, provenance and history, recorded resolution, safe inspection, secret
|
|
handling, and independence of callbacks and published results. Those documents
|
|
remain the owners of the exact requirements.
|
|
|
|
Choose implementations from measured evidence. Path copying, pointer
|
|
formatting/parsing, and repeated metadata/index construction are initial
|
|
investigation candidates, not prescribed solutions. This roadmap does not select
|
|
a cache, alternate metadata mode, new public API, fixed speedup, allocation budget,
|
|
complexity guarantee, or additional release gate. Any need to expand scope must
|
|
be assessed against the consumer evidence and existing policies before adoption.
|
|
|
|
## Execution order
|
|
|
|
Execute the stages below in order. Update each stage's status and record its
|
|
evidence here as work proceeds. Within stage 4, complete and measure one focused
|
|
change before selecting the next. If measurements invalidate a hypothesis, return
|
|
to stage 3 rather than carrying an unsupported design forward. Follow the
|
|
[development guide](../development.md) and its task-specific policy reading guide
|
|
throughout execution.
|
|
|
|
### 1. Establish the downstream and current-code baselines
|
|
|
|
Status: Complete. The recorded PromptKit revision was measured with v0.5.0 and
|
|
current ConfigKit; allocation costs were reproduced. See the
|
|
[downstream results](startup-benchmark.md#stage-1-downstream-reproduction).
|
|
|
|
1. Inspect PromptKit's recorded benchmark and configuration adapter at the
|
|
revision identified in its request. Identify which work is timed, including
|
|
source construction, callbacks, and processing after ConfigKit returns.
|
|
2. Reproduce the independent fixture with the request's ConfigKit dependency,
|
|
then with the current ConfigKit revision, using the same PromptKit revision,
|
|
environment, and benchmark settings. Use isolated checkouts or temporary
|
|
module wiring without changing committed consumer dependencies.
|
|
3. Record exact revisions, toolchain, platform, processor, process parallelism,
|
|
fixture size, setup/timing boundaries, commands, and repeated results. Report
|
|
elapsed time, allocated bytes, and allocations per construction separately.
|
|
Keep generated profiles and binaries outside the repositories.
|
|
4. Explain material differences from the submitted measurements before using
|
|
them to attribute costs. If the recorded consumer checkout or dependencies
|
|
are unavailable, document the missing evidence and continue with the local
|
|
benchmark stages; downstream verification remains outstanding.
|
|
|
|
Completion evidence: A reproducible baseline distinguishing the submitted
|
|
version from current code, or an explicit record of unavailable downstream
|
|
evidence. Do not attribute all adapter costs to ConfigKit.
|
|
|
|
### 2. Add a representative ConfigKit construction benchmark
|
|
|
|
Status: Complete. A maintained root-API benchmark measures 32, 128, and 512
|
|
profile/prompt pairs, with complementary existing benchmarks. See the
|
|
[local results](startup-benchmark.md#stage-2-maintained-public-api-benchmark).
|
|
|
|
1. Add a maintained benchmark through the root public API, using a synthetic
|
|
configuration shape representative of PromptKit's profiles and versioned
|
|
prompts. Exercise mapping input, typed decoding, resolution, validation, and
|
|
publication. Match the relevant callback work observed in stage 1 without
|
|
importing PromptKit or reproducing its runtime machinery.
|
|
2. Include the reported fixture size and smaller and larger catalogs to measure
|
|
scaling. Record both definition counts and the configuration shape; definition
|
|
counts alone do not describe all fields and metadata paths.
|
|
3. Make fixture setup and timed source-construction boundaries explicit. Keep
|
|
unrelated fixture generation outside timing and align comparisons on the
|
|
work they actually include. Confirm the fixture loads successfully and
|
|
exercises the intended behavior before collecting results.
|
|
4. Reuse the existing assembly, underlay, inheritance, and inspection benchmarks
|
|
where they supply complementary evidence. Add related cases only when they
|
|
cover a material gap, such as the callback workload or the public facade.
|
|
|
|
Completion evidence: Repeatable local construction measurements at several
|
|
sizes, with commands and fixture definitions recorded. The benchmark remains
|
|
offline and independent of credentials. Follow the
|
|
[testing policy](../policy/testing.md); timings and allocation counts are
|
|
measurements, not machine-specific pass/fail assertions.
|
|
|
|
### 3. Profile and choose the first optimization
|
|
|
|
Status: Complete. CPU/allocation profiles and the source trace identify
|
|
`Path.String` buffer sizing as the first experiment. See the
|
|
[profile evidence and proposal](startup-benchmark.md#stage-3-profiles-and-implementation-trace).
|
|
|
|
1. Collect CPU and allocation profiles for the representative local fixture.
|
|
Compare them with the downstream profile where available, distinguishing
|
|
ConfigKit loading costs from adapter work. Keep profiled runs separate from
|
|
ordinary timing comparisons.
|
|
2. Trace the dominant costs through the current implementation. Begin with path
|
|
construction and cloning, repeated pointer formatting/parsing, and repeated
|
|
metadata traversal or index construction. Inspect both the engine and root
|
|
conversion boundary where profiles point to them.
|
|
3. Distinguish required ownership separation from redundant internal work.
|
|
Assess simple local changes before introducing shared state or a broader
|
|
representation change. Preserve contextual mapping-key versus sequence-index
|
|
semantics when considering path reuse.
|
|
4. Select the smallest useful change supported by the profile. Record the
|
|
hypothesis, affected boundaries, expected measurement signal, and existing
|
|
behavioral tests that protect those boundaries. Assess scaling from the
|
|
measurements rather than inferring it from one fixture size.
|
|
|
|
Completion evidence: A concrete first optimization with a measured motivation
|
|
and proportionate validation plan. If no worthwhile change emerges, record that
|
|
finding and recommend whether further investigation is justified.
|
|
|
|
### 4. Implement and measure focused changes
|
|
|
|
Status: Complete. `Path.String` buffer sizing and combined final metadata
|
|
traversal are retained after fresh construction, consumer, inspection, and
|
|
profile comparisons. See the [path-formatting experiment](startup-benchmark.md#focused-experiment-pre-size-pathstring)
|
|
and [metadata-traversal experiment](startup-benchmark.md#focused-experiment-combine-final-metadata-traversal).
|
|
No further optimization is selected for this effort.
|
|
|
|
1. Implement the selected internal change while preserving the supported API
|
|
and established behavior. Audit any reduced copying or reused data for
|
|
aliases across traversal branches, callbacks, queries, and independent loads.
|
|
2. Run the relevant existing behavioral tests. Add targeted coverage only for
|
|
a consequential risk not already protected, including contextual path
|
|
semantics, history, ownership, or secret redaction where affected. Do not
|
|
rewrite behavioral expectations merely to accommodate an internal refactor.
|
|
3. Compare repeated before/after measurements under matching conditions at all
|
|
selected sizes. Reprofile when needed to explain the result. Measure affected
|
|
inspection operations as well as loading; if reuse or caching changes data
|
|
lifetime, examine retained memory separately from cumulative allocations.
|
|
4. Keep changes with repeatable benefit and acceptable complexity. Investigate
|
|
regressions, discard unsuccessful experiments, and return to stage 3 for any
|
|
next candidate. Record the evidence for each retained change.
|
|
5. Complete the applicable development checks before committing each finished
|
|
unit under the [agent policy](../policy/agents.md). Use the development guide's
|
|
race validation when ownership or concurrency changes warrant it.
|
|
|
|
Completion evidence: Reviewed, validated changes with attributable improvements
|
|
and recorded tradeoffs. Stop iterating when remaining candidates no longer
|
|
justify their complexity; a predetermined optimization count is unnecessary.
|
|
|
|
### 5. Verify the complete result against consumer workloads
|
|
|
|
Status: Complete. The combined candidate was compared with the pre-optimization
|
|
baseline across all local sizes, complementary benchmarks, and all four recorded
|
|
PromptKit workloads. Consumer configuration and public contract tests passed.
|
|
Acceptance rests on reproducible allocation reductions; the maintainer confirmed
|
|
that busy-host timing is not a reliable comparison here. See the
|
|
[final verification](startup-benchmark.md#final-consumer-verification-and-handoff).
|
|
|
|
1. Compare the final candidate with the unoptimized current-code baseline using
|
|
the local fixture and relevant existing benchmarks. Confirm that gains hold
|
|
across the selected sizes and inspect any material regressions.
|
|
2. Repeat PromptKit's independent fixture with the candidate under the same
|
|
conditions as stage 1. Also run its relevant file-backed, inheritance, and
|
|
layered fixtures, and applicable consumer configuration tests. Record the
|
|
exact candidate wiring and separate downstream gains from isolated ConfigKit
|
|
gains. Do not update or publish a consumer dependency as part of this work.
|
|
3. Confirm that final configuration, presence, provenance/history, inspection,
|
|
and secret behavior remain consistent through the affected paths. Complete
|
|
the everyday development check and any additional checks justified by the
|
|
changes, reusing successful results for unchanged content.
|
|
|
|
Completion evidence: Repeatable reductions for the representative construction
|
|
workload, with related-workload results, semantic validation, and any regressions
|
|
or limitations explained. If downstream verification is unavailable, identify
|
|
the exact outstanding check and report local results without claiming PromptKit
|
|
benefit has been verified.
|
|
|
|
### 6. Record the outcome and hand off
|
|
|
|
Status: Complete. Retained changes, revisions, allocation comparisons,
|
|
reproduction commands, behavioral checks, and timing/retained-heap limitations
|
|
are recorded in the [final handoff](startup-benchmark.md#final-consumer-verification-and-handoff).
|
|
No required verification remains for this scope. Release preparation and
|
|
publication remain separate work.
|
|
|
|
Summarize the retained changes, before/after results, revisions, reproduction
|
|
commands, validation, tradeoffs, and unresolved questions in this roadmap.
|
|
Separate completed work from outstanding consumer checks. Recommend adoption
|
|
based on the measured benefit and maintenance cost, without promising a
|
|
machine-independent threshold.
|
|
|
|
Update implemented internal documentation only where responsibilities or flow
|
|
changed. Keep benchmark evidence here and behavior in its canonical owners;
|
|
follow the [documentation policy](../policy/documentation.md). Publication and
|
|
release preparation remain separate work under the existing release procedure.
|
|
|
|
Completion evidence: A reviewable handoff that makes the improvement and its
|
|
limits clear. Mark the roadmap complete only when the applicable implementation
|
|
and verification work is finished; retire it during release preparation under
|
|
the documentation policy.
|