ConfigKit/docs/roadmap/startup.md

Startup Loading Performance Roadmap

Status: Stages 1–3 complete; Stage 4 is in progress with the first optimization retained. Findings and reproduction commands are in startup-benchmark.md.

Audience: ConfigKit maintainers implementing and reviewing this work.

Objective and scope

Reduce avoidable CPU work and temporary allocations during configuration construction through the existing public loading API. The PromptKit request owns the downstream workload, original measurements, profiling observations, and reproduction instructions. Its measurements include PromptKit adapter work; they are not standalone ConfigKit costs or evidence of retained-memory growth.

The initial scope is measurement and internal optimization. Preserve the configuration contract, the public facade, and the ownership boundaries described by the architecture policy. In particular, improvements must preserve presence and explicit null, source contexts, provenance and history, recorded resolution, safe inspection, secret handling, and independence of callbacks and published results. Those documents remain the owners of the exact requirements.

Choose implementations from measured evidence. Path copying, pointer formatting/parsing, and repeated metadata/index construction are initial investigation candidates, not prescribed solutions. This roadmap does not select a cache, alternate metadata mode, new public API, fixed speedup, allocation budget, complexity guarantee, or additional release gate. Any need to expand scope must be assessed against the consumer evidence and existing policies before adoption.

Execution order

Execute the stages below in order. Update each stage's status and record its evidence here as work proceeds. Within stage 4, complete and measure one focused change before selecting the next. If measurements invalidate a hypothesis, return to stage 3 rather than carrying an unsupported design forward. Follow the development guide and its task-specific policy reading guide throughout execution.

1. Establish the downstream and current-code baselines

Status: Complete. The recorded PromptKit revision was measured with v0.5.0 and current ConfigKit; allocation costs were reproduced. See the downstream results.

1. Inspect PromptKit's recorded benchmark and configuration adapter at the revision identified in its request. Identify which work is timed, including source construction, callbacks, and processing after ConfigKit returns.
2. Reproduce the independent fixture with the request's ConfigKit dependency, then with the current ConfigKit revision, using the same PromptKit revision, environment, and benchmark settings. Use isolated checkouts or temporary module wiring without changing committed consumer dependencies.
3. Record exact revisions, toolchain, platform, processor, process parallelism, fixture size, setup/timing boundaries, commands, and repeated results. Report elapsed time, allocated bytes, and allocations per construction separately. Keep generated profiles and binaries outside the repositories.
4. Explain material differences from the submitted measurements before using them to attribute costs. If the recorded consumer checkout or dependencies are unavailable, document the missing evidence and continue with the local benchmark stages; downstream verification remains outstanding.

Completion evidence: A reproducible baseline distinguishing the submitted version from current code, or an explicit record of unavailable downstream evidence. Do not attribute all adapter costs to ConfigKit.

2. Add a representative ConfigKit construction benchmark

Status: Complete. A maintained root-API benchmark measures 32, 128, and 512 profile/prompt pairs, with complementary existing benchmarks. See the local results.

1. Add a maintained benchmark through the root public API, using a synthetic configuration shape representative of PromptKit's profiles and versioned prompts. Exercise mapping input, typed decoding, resolution, validation, and publication. Match the relevant callback work observed in stage 1 without importing PromptKit or reproducing its runtime machinery.
2. Include the reported fixture size and smaller and larger catalogs to measure scaling. Record both definition counts and the configuration shape; definition counts alone do not describe all fields and metadata paths.
3. Make fixture setup and timed source-construction boundaries explicit. Keep unrelated fixture generation outside timing and align comparisons on the work they actually include. Confirm the fixture loads successfully and exercises the intended behavior before collecting results.
4. Reuse the existing assembly, underlay, inheritance, and inspection benchmarks where they supply complementary evidence. Add related cases only when they cover a material gap, such as the callback workload or the public facade.

Completion evidence: Repeatable local construction measurements at several sizes, with commands and fixture definitions recorded. The benchmark remains offline and independent of credentials. Follow the testing policy; timings and allocation counts are measurements, not machine-specific pass/fail assertions.

3. Profile and choose the first optimization

Status: Complete. CPU/allocation profiles and the source trace identify Path.String buffer sizing as the first experiment. See the profile evidence and proposal.

1. Collect CPU and allocation profiles for the representative local fixture. Compare them with the downstream profile where available, distinguishing ConfigKit loading costs from adapter work. Keep profiled runs separate from ordinary timing comparisons.
2. Trace the dominant costs through the current implementation. Begin with path construction and cloning, repeated pointer formatting/parsing, and repeated metadata traversal or index construction. Inspect both the engine and root conversion boundary where profiles point to them.
3. Distinguish required ownership separation from redundant internal work. Assess simple local changes before introducing shared state or a broader representation change. Preserve contextual mapping-key versus sequence-index semantics when considering path reuse.
4. Select the smallest useful change supported by the profile. Record the hypothesis, affected boundaries, expected measurement signal, and existing behavioral tests that protect those boundaries. Assess scaling from the measurements rather than inferring it from one fixture size.

Completion evidence: A concrete first optimization with a measured motivation and proportionate validation plan. If no worthwhile change emerges, record that finding and recommend whether further investigation is justified.

4. Implement and measure focused changes

Status: In progress. Path.String buffer sizing is retained after fresh construction, consumer, inspection, and profile comparisons. See the focused experiment. The next candidate is duplicate final-tree traversal during metadata finalization.

1. Implement the selected internal change while preserving the supported API and established behavior. Audit any reduced copying or reused data for aliases across traversal branches, callbacks, queries, and independent loads.
2. Run the relevant existing behavioral tests. Add targeted coverage only for a consequential risk not already protected, including contextual path semantics, history, ownership, or secret redaction where affected. Do not rewrite behavioral expectations merely to accommodate an internal refactor.
3. Compare repeated before/after measurements under matching conditions at all selected sizes. Reprofile when needed to explain the result. Measure affected inspection operations as well as loading; if reuse or caching changes data lifetime, examine retained memory separately from cumulative allocations.
4. Keep changes with repeatable benefit and acceptable complexity. Investigate regressions, discard unsuccessful experiments, and return to stage 3 for any next candidate. Record the evidence for each retained change.
5. Complete the applicable development checks before committing each finished unit under the agent policy. Use the development guide's race validation when ownership or concurrency changes warrant it.

Completion evidence: Reviewed, validated changes with attributable improvements and recorded tradeoffs. Stop iterating when remaining candidates no longer justify their complexity; a predetermined optimization count is unnecessary.

5. Verify the complete result against consumer workloads

Status: Not started.

1. Compare the final candidate with the unoptimized current-code baseline using the local fixture and relevant existing benchmarks. Confirm that gains hold across the selected sizes and inspect any material regressions.
2. Repeat PromptKit's independent fixture with the candidate under the same conditions as stage 1. Also run its relevant file-backed, inheritance, and layered fixtures, and applicable consumer configuration tests. Record the exact candidate wiring and separate downstream gains from isolated ConfigKit gains. Do not update or publish a consumer dependency as part of this work.
3. Confirm that final configuration, presence, provenance/history, inspection, and secret behavior remain consistent through the affected paths. Complete the everyday development check and any additional checks justified by the changes, reusing successful results for unchanged content.

Completion evidence: Repeatable reductions for the representative construction workload, with related-workload results, semantic validation, and any regressions or limitations explained. If downstream verification is unavailable, identify the exact outstanding check and report local results without claiming PromptKit benefit has been verified.

6. Record the outcome and hand off

Status: Not started.

Summarize the retained changes, before/after results, revisions, reproduction commands, validation, tradeoffs, and unresolved questions in this roadmap. Separate completed work from outstanding consumer checks. Recommend adoption based on the measured benefit and maintenance cost, without promising a machine-independent threshold.

Update implemented internal documentation only where responsibilities or flow changed. Keep benchmark evidence here and behavior in its canonical owners; follow the documentation policy. Publication and release preparation remain separate work under the existing release procedure.

Completion evidence: A reviewable handoff that makes the improvement and its limits clear. Mark the roadmap complete only when the applicable implementation and verification work is finished; retire it during release preparation under the documentation policy.