diff --git a/docs/roadmap/startup.md b/docs/roadmap/startup.md new file mode 100644 index 0000000..bd21fee --- /dev/null +++ b/docs/roadmap/startup.md @@ -0,0 +1,182 @@ +# Startup Loading Performance Roadmap + +Status: Approach accepted; execution has not started. + +Audience: ConfigKit maintainers implementing and reviewing this work. + +## Objective and scope + +Reduce avoidable CPU work and temporary allocations during configuration +construction through the existing public loading API. The +[PromptKit request](promptkit-performance-request.md) owns the downstream +workload, original measurements, profiling observations, and reproduction +instructions. Its measurements include PromptKit adapter work; they are not +standalone ConfigKit costs or evidence of retained-memory growth. + +The initial scope is measurement and internal optimization. Preserve the +[configuration contract](../config.md), the public facade, and the ownership +boundaries described by the [architecture policy](../policy/architecture.md). +In particular, improvements must preserve presence and explicit null, source +contexts, provenance and history, recorded resolution, safe inspection, secret +handling, and independence of callbacks and published results. Those documents +remain the owners of the exact requirements. + +Choose implementations from measured evidence. Path copying, pointer +formatting/parsing, and repeated metadata/index construction are initial +investigation candidates, not prescribed solutions. This roadmap does not select +a cache, alternate metadata mode, new public API, fixed speedup, allocation budget, +complexity guarantee, or additional release gate. Any need to expand scope must +be assessed against the consumer evidence and existing policies before adoption. + +## Execution order + +Execute the stages below in order. Update each stage's status and record its +evidence here as work proceeds. Within stage 4, complete and measure one focused +change before selecting the next. If measurements invalidate a hypothesis, return +to stage 3 rather than carrying an unsupported design forward. Follow the +[development guide](../development.md) and its task-specific policy reading guide +throughout execution. + +### 1. Establish the downstream and current-code baselines + +Status: Not started. + +1. Inspect PromptKit's recorded benchmark and configuration adapter at the + revision identified in its request. Identify which work is timed, including + source construction, callbacks, and processing after ConfigKit returns. +2. Reproduce the independent fixture with the request's ConfigKit dependency, + then with the current ConfigKit revision, using the same PromptKit revision, + environment, and benchmark settings. Use isolated checkouts or temporary + module wiring without changing committed consumer dependencies. +3. Record exact revisions, toolchain, platform, processor, process parallelism, + fixture size, setup/timing boundaries, commands, and repeated results. Report + elapsed time, allocated bytes, and allocations per construction separately. + Keep generated profiles and binaries outside the repositories. +4. Explain material differences from the submitted measurements before using + them to attribute costs. If the recorded consumer checkout or dependencies + are unavailable, document the missing evidence and continue with the local + benchmark stages; downstream verification remains outstanding. + +Completion evidence: A reproducible baseline distinguishing the submitted +version from current code, or an explicit record of unavailable downstream +evidence. Do not attribute all adapter costs to ConfigKit. + +### 2. Add a representative ConfigKit construction benchmark + +Status: Not started. + +1. Add a maintained benchmark through the root public API, using a synthetic + configuration shape representative of PromptKit's profiles and versioned + prompts. Exercise mapping input, typed decoding, resolution, validation, and + publication. Match the relevant callback work observed in stage 1 without + importing PromptKit or reproducing its runtime machinery. +2. Include the reported fixture size and smaller and larger catalogs to measure + scaling. Record both definition counts and the configuration shape; definition + counts alone do not describe all fields and metadata paths. +3. Make fixture setup and timed source-construction boundaries explicit. Keep + unrelated fixture generation outside timing and align comparisons on the + work they actually include. Confirm the fixture loads successfully and + exercises the intended behavior before collecting results. +4. Reuse the existing assembly, underlay, inheritance, and inspection benchmarks + where they supply complementary evidence. Add related cases only when they + cover a material gap, such as the callback workload or the public facade. + +Completion evidence: Repeatable local construction measurements at several +sizes, with commands and fixture definitions recorded. The benchmark remains +offline and independent of credentials. Follow the +[testing policy](../policy/testing.md); timings and allocation counts are +measurements, not machine-specific pass/fail assertions. + +### 3. Profile and choose the first optimization + +Status: Not started. + +1. Collect CPU and allocation profiles for the representative local fixture. + Compare them with the downstream profile where available, distinguishing + ConfigKit loading costs from adapter work. Keep profiled runs separate from + ordinary timing comparisons. +2. Trace the dominant costs through the current implementation. Begin with path + construction and cloning, repeated pointer formatting/parsing, and repeated + metadata traversal or index construction. Inspect both the engine and root + conversion boundary where profiles point to them. +3. Distinguish required ownership separation from redundant internal work. + Assess simple local changes before introducing shared state or a broader + representation change. Preserve contextual mapping-key versus sequence-index + semantics when considering path reuse. +4. Select the smallest useful change supported by the profile. Record the + hypothesis, affected boundaries, expected measurement signal, and existing + behavioral tests that protect those boundaries. Assess scaling from the + measurements rather than inferring it from one fixture size. + +Completion evidence: A concrete first optimization with a measured motivation +and proportionate validation plan. If no worthwhile change emerges, record that +finding and recommend whether further investigation is justified. + +### 4. Implement and measure focused changes + +Status: Not started. + +1. Implement the selected internal change while preserving the supported API + and established behavior. Audit any reduced copying or reused data for + aliases across traversal branches, callbacks, queries, and independent loads. +2. Run the relevant existing behavioral tests. Add targeted coverage only for + a consequential risk not already protected, including contextual path + semantics, history, ownership, or secret redaction where affected. Do not + rewrite behavioral expectations merely to accommodate an internal refactor. +3. Compare repeated before/after measurements under matching conditions at all + selected sizes. Reprofile when needed to explain the result. Measure affected + inspection operations as well as loading; if reuse or caching changes data + lifetime, examine retained memory separately from cumulative allocations. +4. Keep changes with repeatable benefit and acceptable complexity. Investigate + regressions, discard unsuccessful experiments, and return to stage 3 for any + next candidate. Record the evidence for each retained change. +5. Complete the applicable development checks before committing each finished + unit under the [agent policy](../policy/agents.md). Use the development guide's + race validation when ownership or concurrency changes warrant it. + +Completion evidence: Reviewed, validated changes with attributable improvements +and recorded tradeoffs. Stop iterating when remaining candidates no longer +justify their complexity; a predetermined optimization count is unnecessary. + +### 5. Verify the complete result against consumer workloads + +Status: Not started. + +1. Compare the final candidate with the unoptimized current-code baseline using + the local fixture and relevant existing benchmarks. Confirm that gains hold + across the selected sizes and inspect any material regressions. +2. Repeat PromptKit's independent fixture with the candidate under the same + conditions as stage 1. Also run its relevant file-backed, inheritance, and + layered fixtures, and applicable consumer configuration tests. Record the + exact candidate wiring and separate downstream gains from isolated ConfigKit + gains. Do not update or publish a consumer dependency as part of this work. +3. Confirm that final configuration, presence, provenance/history, inspection, + and secret behavior remain consistent through the affected paths. Complete + the everyday development check and any additional checks justified by the + changes, reusing successful results for unchanged content. + +Completion evidence: Repeatable reductions for the representative construction +workload, with related-workload results, semantic validation, and any regressions +or limitations explained. If downstream verification is unavailable, identify +the exact outstanding check and report local results without claiming PromptKit +benefit has been verified. + +### 6. Record the outcome and hand off + +Status: Not started. + +Summarize the retained changes, before/after results, revisions, reproduction +commands, validation, tradeoffs, and unresolved questions in this roadmap. +Separate completed work from outstanding consumer checks. Recommend adoption +based on the measured benefit and maintenance cost, without promising a +machine-independent threshold. + +Update implemented internal documentation only where responsibilities or flow +changed. Keep benchmark evidence here and behavior in its canonical owners; +follow the [documentation policy](../policy/documentation.md). Publication and +release preparation remain separate work under the existing release procedure. + +Completion evidence: A reviewable handoff that makes the improvement and its +limits clear. Mark the roadmap complete only when the applicable implementation +and verification work is finished; retire it during release preparation under +the documentation policy.