ConfigKit/docs/roadmap/implementation.md

Implement the performance audit revisions

Status: Planned. Audience: a Luna coding agent implementing one stage at a time, in numerical order. The performance audit roadmap owns the target and rationale. This plan replaces the completed ReleaseKit plan, which remains available in Git history at 0c77dee.

Planning baseline: 0c77deed05417edb32f42c7bf24df2fdc2581a25. The production baseline is unchanged from the audit's 48619de revision. Keep the earlier Path.String sizing and combined final-metadata traversal improvements. This sprint changes private algorithms, not public API, configuration semantics, dependencies, Go minimum, or release gates.

Working instructions

• Read development.md, the applicable agent, architecture, testing, and documentation policies, and owning contract/test files before each stage. Use the graph tools for code discovery as instructed by AGENTS.md.
• Preserve user changes. Implement only the current stage and its regressions; update its status/evidence here and commit the validated unit with a short plain-English message. Do not publish, push, or modify consumer dependencies.
• Operation indexes are private phase-local data. Inspection indexes are fully built before publication and only read afterward. Returned records remain detached. Keep effective and historical indexes separate with independently owned mutable path storage.
• Preserve the existing configuration contract: presence, null/empty distinctions, composition, diagnostics, chronology, sensitivity, callback lifetime, secret isolation, and handle closure. Do not add rejection rules, limits, stricter callbacks, concurrency promises, or complexity guarantees.
• Run each stage's focused checks and ./scripts/check.sh before handing off production changes. Run go test -race ./internal/engine after retained-index or ownership changes, and the full race suite in Stage 18. Reuse checks for unchanged content. No new CI wait, release gate, or performance threshold.
• Prefer algorithm reasoning for this sprint. Benchmark runs are not prerequisites. Optional measurements use the same prepared local Go, offline dependencies, sequential runs, and artifacts outside repositories. Favor B/op and allocs/op on the busy host; do not infer CPU improvements from noisy timings.
• Extend existing behavioral tests for missing risks, not traversal counts. Small test-only baseline algorithms may support bounded differential coverage. Do not keep alternate production engines or switches. If the selected algorithm cannot preserve behavior, record the discrepancy and seek review rather than strengthening the contract or silently dropping work.

Resolved implementation choices

1. Containment and path deduplication use canonical token spelling, as today. Metadata/document queries additionally enforce contextual key/index shape. These predicates remain distinct.
2. Use a small private engine prefix trie with canonical escaped pointer tokens and integer IDs into owner-held slices. Root is zero tokens; / is one empty token. Scan separator boundaries without decoding/rejoining tokens. Escapes contain no slash. The trie does not validate document shape.
3. The helper returns IDs, not values/provenance. Owners select deepest ancestor, earliest inserted ancestor/intersection, or all relevant IDs, explicitly restoring owner order. No general path framework or public hook is introduced.
4. Sensitivity uses a worklist: ordinary dependencies activate on intersection, while directed inheritance links activate on a classified ancestor/equal path of their dependency. Preserve both rules.
5. Inspection retains full replay and on-demand safe rendering. Prepare immutable coalesced resolution descriptors and inheritance lookup data once, with no per-field tree or rendered-explanation cache.
6. Defer branch-local replay, public metadata-copy redesign, global schema caching, and complex contributor-trace membership indexing. Their ownership/memory/ equality risks exceed the selected scope. Existing clones and linked traces stay in place. No new dependency, pool, or package-global mutable cache is needed.

Stage 1: Confirm the baseline and semantic evidence — Complete

Scope: orientation and test-only evidence, with no optimization yet.

1. Inspect status/revisions. Read the audit and configuration contract sections for metadata, sensitivity, explanation, resolution, and overrides, plus their implementation/test owners.
2. Run ./scripts/check.sh. Record revision, local go version, result, and environmental limitations in an evidence subsection here. Do not repeat or revise the completed startup timing campaign.
3. Identify existing coverage for numeric shapes, removed paths, escaped keys, inheritance, callback detachment, and independent loads. Add test-only reference helpers if useful for later containment, coverage/coalescing, overlap, and sensitivity equivalence tests. Keep fixtures bounded and deterministic; do not copy the loader. Helpers may instead accompany their first consuming stage.

Done: baseline checks recorded and semantic test owners identified. Existing contracts/tests are authoritative; reference algorithms add equivalence evidence, not new public guarantees. A documentation-only stage needs documentation checks.

Evidence

• Current revision: 52793301ba80eb49f91956009e60cd812f92e94c; planning baseline: 0c77deed05417edb32f42c7bf24df2fdc2581a25. The working tree was clean, and production Go files match the audit baseline 48619de024d794cf41e90b171b7fcbe609dcd437.
• GOTOOLCHAIN=local go version: go1.26.8 linux/amd64. GOTOOLCHAIN=local ./scripts/check.sh: passed (go test ./..., go vet ./..., and go build ./..., in addition to formatting and whitespace checks). No environmental limitations encountered; no benchmark campaign was repeated.
• Existing semantic coverage includes numeric/key-index and escaped path shapes (path_test.go, provenance_test.go, override_test.go), removed/history paths (metadata_test.go, resolution_test.go, inheritance_history_test.go), nested and sequence-contained inheritance (inheritance_test.go, inheritance_walk_test.go), detached callback/metadata records (callback_ownership_test.go, resolution_test.go, metadata_test.go), and independent loads (provenance_test.go). Differential reference helpers are deferred to the first stage that consumes each algorithm so they can remain small and bounded.

Stage 2: Make presence-only queries lightweight — Complete

Roadmap candidate 1. Owner: internal/engine/metadata.go.

1. Make Metadata.IsExplicit validate effective path/shape and read the stored boolean, without calling Lookup, cloning origins, or projecting inheritance. A shared private effective-field lookup may serve both methods; Lookup continues returning complete detached records.
2. Format each query pointer once per lookup. Preserve root, zero metadata, invalid/unknown/removed paths, parsed numeric tokens, and explicit key/index distinctions. Leave Resolution.IsExplicit entry-state semantics unchanged.
3. Extend metadata/presence tests only where agreement with Lookup lacks coverage: direct/inherited input, defaults, and explicit false/zero/null.

Validate: go test ./internal/engine -run 'Metadata|Presence|Explicit|Inherited', relevant root facade tests, and the everyday check. No allocation assertion.

Stage 3: Detect inheritance without constructing paths — Complete

Roadmap candidate 9. Owner: internal/engine/inheritance_load.go.

1. Replace detection's full path-building walk with a nil-safe boolean visitor: inspect mapping keys for extends, recurse into values/sequence children, and return immediately on a match.
2. Leave validation, expansion order, provenance, and budgets unchanged. Detection is not validation and must still find nested invalid syntax.
3. Reuse inheritance tests; add missing sequence-contained or invalid nested cases without assertions about how many nodes were visited.

Validate: go test ./internal/engine -run 'Inheritance|Inherited' and the everyday check. Inputs without inheritance still need a linear scan.

Stage 4: Compare path tokens without whole-pointer formatting — Complete

Roadmap candidate 2. Owners: engine path.go, resolution.go, metadata.go.

1. Add private token-spelling equality. Key/parsed segments compare literal bytes; two indexes compare integers. For index versus key/parsed, render the index into stack scratch space with strconv.AppendInt and compare bytes. Do not parse literal tokens as integers: 01, +1, and 1 differ.
2. Implement pathContains with existing validity/depth checks and token-prefix equality. Root behavior remains unchanged. Escaping is injective, so raw token spelling comparison preserves canonical pointer equivalence.
3. Replace textual comparison in matchesPathShape with token equality, then retain its existing explicit/contextual shape checks and candidate-side handling. Apply the same spelling predicate to local textual path deduplication.
4. Differentially compare with the String-based baseline over roots, invalid paths, empty/slash/tilde/numeric keys, parsed tokens, nested paths, and integer extrema. Reuse existing effective/historical query-shape coverage.

Validate: go test ./internal/engine -run 'Path|Metadata|Resolution|Override|Inheritance' and the everyday check. Public Path.String stays unchanged.

Stage 5: Format sorting keys once per item — Complete

Roadmap candidate 2. Owners: engine resolution, overlay, override application, document comparison, and diagnostics.

1. Decorate path-bearing items with precomputed canonical pointers, sort with existing comparisons/tie-breakers, then project back. Cover orderedResolutionRecords, displacedOrigins, affectedOverrideOrigins, and the final delta sort in compareConfigurationValues.
2. In canonicalDiagnostics, precompute paths without changing comparator precedence, primary/related location ordering, or standalone comparator use. Do not add permanent pointer fields to every record for sorting.
3. Gather recursive displaced definitions into one owned output and sort at the outer owner, avoiding repeated sorting of the same nested subtree. Preserve final lexical order and duplicate behavior.
4. Reuse ordering tests; add missing tie/escaped-key cases. Do not promise stable order for items whose full comparison is equal.

Validate: go test ./internal/engine -run 'Resolution|Override|Overlay|Diagnostic|CompareDocument|Provenance' and the everyday check. Expected benefit is reduced formatting, not measured CPU.

Stage 6: Add a private canonical-pointer prefix index — Complete

Scope: one helper plus tests, no caller migration yet.

1. Add internal/engine/path_prefix.go. Nodes have child maps keyed by escaped tokens, terminal integer payload IDs, and the minimum inserted ID within the subtree. IDs are nonnegative owner ordinals; represent absence explicitly.
2. Support insertion, ancestor payload visits (root/exact included), descendant payload visits (exact included), and earliest intersecting ID using ancestor IDs and the queried subtree minimum. Preserve multiple IDs at one terminal. No deletion, wildcard search, provider abstraction, or serialization API.
3. Tokenize canonical pointers by separator scanning without split slices. Preserve root versus empty keys and /a versus /ab boundaries. Owners supply canonical pointers from valid paths; no new input validation contract. Retain IDs only, not nodes, credential bytes, or mutable path segments.
4. Differentially check results against simple prefix scans for ancestors, descendants, duplicates, insertion minima, escaping, and root/empty tokens.

Validate: focused helper tests and the everyday check. Storage follows indexed token material/payloads; tests do not fix representation or allocation counts.

Stage 7: Index resolution coverage and record reads — Complete

Roadmap candidate 3, first integration. Owner: internal/engine/resolution.go.

1. Index already ordered records after the callback. Select deepest covering ancestor, earliest owner ID for equal depth, matching the current scan. Use this for delta coverage in runResolution; retain all validity/helper checks and diagnostic order.
2. Add a small snapshot coverage adapter with the same rules for Stage 8. IDs refer to owner-held slices and cannot outlive or survive their reordering.
3. Build operation-local read-only mapping lookups for completed before/after trees, reusing mappingIndex mechanics without mutating trees. Add a private indexed document lookup variant with the same validity, shape, bounds, and absence checks; retain the plain helper for callers without reuse.
4. Cover parent/exact/more-specific records, equal-depth private fixtures, added/removed values, unrecorded changes, and helper-target mismatch.

Validate: go test ./internal/engine -run 'Resolution|Resolve|Callback|Path' and the everyday check. Snapshot ownership/contents remain unchanged.

Stage 8: Replace per-delta whole-metadata scans — Complete

Roadmap candidate 3, second integration. Owner: engine load_metadata.go.

1. In applyResolutionMetadata, build one final pointer/node index and one snapshot coverage index. Reuse final membership for descendant history and field refresh; do not rebuild indexes per delta/field.
2. Index delta paths by original delta ID. Enumerate the union of historical and final pointers once, obtain covering delta IDs, and bucket affected pointers by delta. Parse paths only when a retained semantic path is actually needed. Sort buckets lexically and retain original delta processing order.
3. Apply exact snapshot events first, descendant history second, effective origin updates third, then final field refresh, as today. Exclude the covering record's own pointer from descendant history and retain the original Previous origins. Handle overlapping delta IDs in reference fixtures without assuming every possible input is disjoint. Normal structural comparison emits disjoint deltas.
4. For fields, visit pointers once and process matching delta IDs in original order. For new fields, use snapshot coverage; preserve null/Explicit state, default resolution-origin fallback, and removed-path history.
5. Extend existing tests for many disjoint changes, replacement, removed/new descendants, and numeric container-shape changes. A bounded test-only reference metadata update can protect phase/order equivalence.

Validate: go test ./internal/engine -run 'Resolution|Metadata|Provenance|History|Explain', engine race tests, and the everyday check. Complexity is output-sensitive to affected events, not every delta times every field.

Stage 9: Collect sensitivity directly from trees — Complete

Roadmap candidate 5. Owner: engine secret.go. Keep fixed-point propagation unchanged until Stage 11 so collection is independently reviewable.

1. Replace temporary pointer/node indexes used only to enumerate classification paths/flags with direct visitors into local path and sensitivity accumulators. Remove the redundant initial final-tree scan; final still participates in the late projection phase before schema/predicate classification.
2. Preserve phase precedence: applications; history/dependencies; inheritance; projections then final. Tree/pointer contributions currently install parsed segments, while history/contributor contributions can install typed paths. Preserve those representations and subsequent overwrites.
3. Skip reparsing only if the current stored path already has that parsed representation. Otherwise perform the existing parsed replacement. Do not simply substitute explicit walker key/index paths or retain the first shape.
4. Retain all historical/removed paths, controls, dependency-only paths, default/ dynamic tagged projections, and cloned predicate arguments.
5. Add only missing numeric shape-transition, historical predicate, dynamic tag, and predicate-detachment cases in existing sensitivity/inheritance test owners.

Validate: go test ./internal/engine -run 'Secret|Sensitivity|Inherited|Effective|Explain', engine race tests, and the everyday check. No new predicate order/count guarantee.

Stage 10: Build and test indexed sensitivity triggers — Complete

Roadmap candidate 6, isolated algorithm. Add private internal/engine/secret_propagation.go and focused tests.

1. Ordinary edges have dependency/target pointers and activate on classified ancestor or descendant intersection. Inheritance links become two directed edges, each activated only by a classified ancestor/equal dependency path.
2. Index dependency pointers with payload IDs. A new classified pointer visits ordinary triggers at ancestors and descendants; inheritance triggers only at descendants. Avoid double-visiting exact terminals. Track fired IDs and enqueue a target only when it becomes newly sensitive.
3. Seed from the initial sensitive set. Preserve the ordinary dependency filter: the old secretIntersectsPath sees only classified pointers present in the collected paths map. Inheritance's pathIsSecret does not require membership. Relations use textual spelling, not document query-shape validation.
4. Differentially compare final sets to the old fixed-point algorithm for forward/reverse chains, cycles, duplicate edges, classified ancestors/ descendants, classified pointers absent from paths, root, escapes/numeric spellings, and mixed ordinary/inheritance graphs. Use no real credentials.

Validate: focused propagation tests and the everyday check. Match/output-sized work may still be large for broad dependencies; no blanket linear guarantee.

Stage 11: Integrate worklist sensitivity into publication — Complete

Roadmap candidate 6, integration. Owner: engine secret.go.

1. Feed collected paths, seeds, dependencies, and inheritance links into Stage 10's closure. Remove the production fixed-point loop/unused scan helper after migration; retain the useful reference only in tests.
2. Verify full-load effective rendering and explanations for mixed derived/ inherited sensitivity, replaced/removed paths, tagged containers, consumed controls, and nested copies. All credentials remain outside trees/indexes.
3. Verify independent loads and repeated queries retain no mutable propagation state. Queues and trigger indexes are phase-local and discarded on return.

Validate: go test ./internal/engine -run 'Secret|Sensitivity|Inheritance|History|Explain|Independent', relevant root facade secret/inheritance tests, engine race tests, and the everyday check. Record collection and propagation outcomes separately.

Stage 12: Index CLI overlaps and validation reads — Complete

Roadmap candidate 4. Owner: engine override.go.

1. Retain input order and the duplicate seen map. Replace the prior-accepted overlap scan with earliest intersecting trie lookup using original override indices. Duplicate diagnostics continue preceding overlap checks.
2. Insert exactly where pathAccepted[index] = true occurs today: after path, overlap, version, and directive checks, before value conversion/budgets. A later bad value still influences subsequent overlap diagnostics. Do not index paths rejected earlier or omit those accepted before value failure.
3. Reuse an operation-local mapping index for read-only base traversal during path validation. Preserve missing mapping ancestors, preparation-before-apply, CLI source order/identity, rules, and limits.
4. Compare with baseline for disjoint/duplicate/ancestor/descendant paths, multiple conflicts, escapes, rejected paths, and later value failures. Protect diagnostic codes, paths, and first related-source selection.

Validate: go test ./internal/engine -run 'Override|CLI|Provenance|Diagnostic' and the everyday check. No policy changes.

Stage 13: Coalesce resolution replay once with indexed ancestors — Complete

Roadmap candidate 7. Owners: engine history.go, load.go, load_result.go.

1. Replace group scans with ancestor lookup selecting the earliest inserted matching group, not the deepest. Insert only new groups; merged child paths are not independently indexed as groups.
2. Retain original before/after trees and normalization precedence. Use ordered membership sets for reason/dependency unions, retaining first occurrence and canonical pointer equivalence.
3. Prepare immutable coalesced descriptors once before publication. Explain reads them without mutating Changes or borrowed snapshots. Retain original snapshots for their other metadata/history-budget uses; node accounting is unchanged. No renderer or public records are cached.
4. Differentially check independent/parent/child records, multiple ancestors, root, equal writes, union order, normalization, and repeated detached results.

Validate: go test ./internal/engine -run 'Resolution|History|Explain|Ownership', engine race tests, and the everyday check. Full replay semantics stay intact.

Stage 14: Index relevant inheritance history operations — Complete

Roadmap candidate 7. Owners: engine metadata, metadata cloning, inheritance history.

1. During final metadata preparation, index inheritance operation IDs by control path or transfer destination. Collect intersecting IDs for History, deduplicate, restore original operation order, and run existing projection logic.
2. Preserve stored-history splitting before resolution and Previous-origin construction. Keep fields/history/path clones independent. A fully frozen ID-only index may be shared in cloneMetadata, like immutable evidence; no query builds or updates it. Zero/private unprepared metadata must still produce correct results (use the existing scan if no prepared index exists).
3. Leave global appendInheritanceExplanation replay intact: non-intersecting operations can affect working state and must not be skipped with this filter.
4. Compare indexed history to the scan for multi-scope controls, removed paths, ancestor queries, and query mutations/captured metadata independence.

Validate: go test ./internal/engine -run 'Inheritance|Inherited|History|Metadata|Independent', engine race tests, and the everyday check.

Stage 15: Reuse frozen inspection mapping lookups — Complete

Roadmap candidate 7. Owners: publication, inspection, document/snapshot/history helpers.

1. Reuse Stage 7's mapping lookup builder for immutable decoded-input, final, and application trees queried repeatedly by inspection. Populate all retained maps before return, visiting shared node identities once. Retain lookup positions and borrowed immutable nodes, not extra trees or Path copies.
2. Thread read-only lookups into applicable document/snapshot and application origin/relevance reads. Preserve validity, shape, bounds, and absence. Read populated maps directly, not lazy entries methods that write on first query.
3. Keep plain document lookup on mutable replay trees in this stage. Their existing query-local merge indexes are unchanged; do not reuse a frozen map after cloning, deletion, or entry-position changes. This avoids adding an invalidation protocol just to optimize the selected immutable reads.
4. Reuse shape/history/render tests; add only missing repeated/concurrent read-only inspection ownership cases. Do not add a public concurrency promise. Record extra retained-index storage honestly; this is a lookup tradeoff, not a free allocation reduction. Baseline/full-tree replay remains unchanged.

Validate: go test ./internal/engine -run 'Path|History|Explain|Effective|Independent|Parallel', engine race tests, and the everyday check.

Stage 16: Index materialized inheritance-step membership

Roadmap candidate 8. Owner: engine inheritance_origin.go.

1. Keep output order and compact linked traces. Use a comparable key with three canonical From/To/Reference strings and one encoded ordered Locations string. Encode the location count and each SourceID byte length, then its bytes and delimited decimal line/column integers. This is collision-free, including empty IDs, delimiter characters, and signed coordinates. Arbitrary unescaped delimiter concatenation is incorrect; no hashing dependency is needed.
2. Seed membership from existing Origin.Inheritance but retain its existing duplicates/order. Only newly proposed steps are deduped, as today.
3. Format candidate pointers once; clone retained paths/locations only for newly accepted steps. Keep contributor traversal and transfer direction intact.
4. Compare chains, shared contributors, different location/order steps, existing duplicates, escaped/numeric paths, and returned-record mutation to baseline.

Validate: go test ./internal/engine -run 'Inheritance|Inherited|Metadata|Provenance|History' and the everyday check. Complex contributor-trace equality stays unchanged.

Stage 17: Index simple resolution and origin membership

Roadmap candidate 8. Owners: engine resolution.go and load_metadata.go.

1. Maintain lazily initialized first-occurrence reason/dependency sets on resolution records. Dependencies use canonical pointer equality; retained slices and callback validation/order/nil behavior remain unchanged.
2. Use exact Location membership in originContainedBy, retaining Layer/ Operation and contributor-empty checks. Use SourceID membership in originRepresentedByChange, which has a different source-coverage predicate. Do not conflate these equality rules.
3. Leave complex contributor scans and sameInheritanceTrace intact. Their concrete segment/location equality cannot be replaced by pointer identity or canonical spelling. Sets are phase-local, not exported record fields.
4. Reuse existing provenance tests; cover duplicate unions, equal source IDs with differing locations, and location coverage with contributor mismatch only where protection is missing.

Validate: go test ./internal/engine -run 'Resolution|Provenance|Overlay|Override|Inheritance|History' and the everyday check. Small lists and outputs preserve existing semantics.

Stage 18: Review the completed result and record outcomes

1. Review the diff from baseline. Map all nine candidates to selected implemented changes and expressly deferred portions. Remove dead helpers/imports/state; keep useful small equivalence tests, no alternate production engines.
2. Run ./scripts/check.sh and go test -race ./.... Verify public/import boundaries, examples, independent loads, shapes, presence, history, redaction, and complete-on-return behavior through their existing tests.
3. Record algorithmic before/after work, lifetime/storage tradeoffs, revisions, actual commands/results, and unmeasured claims in an evidence subsection here. Optional probes may reuse existing startup/assembly/underlay/inheritance benchmarks and small record/override/dependency/query fixtures. Measurements are not prerequisites; timing claims await a representative host.
4. Update stage and roadmap status accurately. Update focused internal docs only where they describe changed implemented mechanics. Keep public contracts and policy language unchanged. Do not retire other historical records or roadmaps as incidental cleanup.
5. Review links/whitespace/staging, commit the completed unit, and report outcomes, checks, remaining deferred experiments, and workspace status. No release or downstream bump is included.

Done: selected changes preserve behavior, remain proportionate/maintainable, pass validation, and distinguish algorithmic expectations from measured results. No public performance guarantee is created.

Open Questions

None required for the selected scope. Algorithm, ownership, relation, ordering, and scope choices above are resolved. Expressly deferred experiments require separate design/evidence and do not block this sequence.