ConfigKit/docs/roadmap/efficiency.md

Algorithmic Efficiency Audit

Reviewed revision: 5098b56655bd383e22a57d6cf09d99db3e92090f. Audit date: 2026-10-25.

Scope And Assessment

This source-level audit reviewed the public facade, acquisition and discovery, strict composition, precedence assembly, collection policy, inheritance, schema and typed-value operations, callbacks, secrets, inspection, and comparison. Knowledge-graph searches and caller traces guided source inspection; the bounds below follow from the actual loops and handoffs, not timing measurements.

The implementation already uses indexed mapping merges, tuple-indexed sequence matching, sparse item projections, scoped import adjacency, inheritance policy localization, and worklist sensitivity propagation. The findings are remaining batch-processing costs, not a recommendation to replace those mechanisms.

The review excludes expressly selected algorithms, including deterministic sorting, stable Kahn scheduling, exact numeric identity normalization, and the pairwise registration-overlap check expressly selected in the completed sequence plan at d0a605a:docs/roadmap/implementation.md. It also leaves the deferred performance experiments deferred. Required detached copies and genuinely expanding configuration/history output are not treated as avoidable overhead.

Complexities describe the identified internal work, not an end-to-end runtime bound or a new consumer guarantee. Examples use bounded-depth paths and constant-sized records unless noted; hash-index alternatives use expected lookup costs, with string processing and actual copied payload accounted for separately. Existing resource limits bound realizable inputs but do not remove these growth patterns. No new limits, public caches, dependencies, or release gates are proposed.

No additional functional defect was established during this pass. The four open findings in the separate functional correctness audit remain applicable and are not duplicated here.

1. Policy Retirement Scans Every Collection For Every Replacement

Status: open. Priority: medium.

Relevant code. collectionPolicyScope.retireCollectionPoliciesAtOrBelow in collection_policy_state.go, lines 163–174, scans the complete policy table. The replacement hook in mergeOverlayMappingsWithCollectionPolicy (overlay.go, lines 182–185) calls it for each ordinary replaced value. applyOverlaySequenceEvidence also calls it for whole-collection replacement.

Scaling case. A lower tier has P independently declared collection policies and Q ordinary scalar settings elsewhere. A higher tier changes those Q scalar settings without touching a collection. Each scalar replacement scans all P policies even though none is affected: O(P × Q) policy visits. Taking P = Q = N gives quadratic work for linear-sized inputs, effective values, and policy state. This occurs within one precedence merge; it does not depend on many tiers, registration-overlap checking, or expanding inheritance output.

Potential refactoring. Prepare an operation-local semantic path index over the policy table and use descendant queries for retirement, maintaining it as policies change. Alternatively, batch retirement against the old policy state where its timing can be preserved. For the disjoint example, preparation plus queries can be O(P + Q); generally charge additional work to affected entries and path lengths. Preserve explicit-mode persistence, record ownership, and the point at which old identity is retired. A delayed blanket retirement must not clear newly established higher-tier policy. Existing inheritance policy indexing provides a related mechanism to evaluate, not a requirement to unify its interface.

2. Small Tiers Recopy Unrelated Accumulated Collection Policy

Status: open. Priority: medium.

Relevant code. resolveStrictCollectionPolicies in compose.go, lines 330–334, first obtains effectiveLower through lowerCollectionPoliciesForClaims, then clones the result. The latter helper (collection_retirement.go, lines 65–71) already deep-clones and scans the entire lower scope. cloneCollectionPolicyScope / cloneCollectionPolicy in collection_policy_state.go also copy record-order and nested record-policy state.

assembleLayeredInput in load_request.go supplies the accumulated lower policy when composing each underlay group and when applying that group to the accumulator. The overlay resolver repeats this whole-table preparation and additionally checks all retained policy destinations against the incoming tree (overlay.go, lines 145–150).

Scaling case. An initial underlay establishes P constant-sized collection policies. Each of T later underlays contributes only one unrelated scalar, with no declarations or collection changes. Despite having no relevant collection work, each tier copies/scans all P policies: O(T × P) work and cumulative allocation. P = T = N yields quadratic bookkeeping for O(P + T) input and final policy/value sizes. These are transient copies of unchanged private policy, not T separately published snapshots required by the public ownership contract.

Potential refactoring. Separate read-only lower-policy lookup from the owned policy changes produced by the current tier. Resolve only policies intersecting incoming claims/declarations, and apply those changes to the owned assembly state. A path index or localized view can avoid scanning unrelated entries; retain copies where state actually crosses a mutable ownership boundary. At minimum, avoid the immediate duplicate deep copy, but removing only that copy does not change the O(T × P) growth. Aim for work proportional to initial state, incoming material, and affected policy rather than the complete accumulated table per tier. Preserve pending declarations, replacement eligibility, and independent nested record-policy ownership.

This is distinct from finding 1: one tier can avoid repeated retirement scans and still recopy the full table at every subsequent tier. It is also distinct from the deferred metadata-copy/public-record handoff experiment.

3. Tier-History Projection Repeatedly Searches Wide Mappings

Status: open. Priority: medium.

Relevant code. projectTierHistoryApplications in sequence_history_projection.go, lines 17–30, calls unindexed documentAtPath for each mapped sequence in a source application. That helper delegates to documentAtPathWithLookup with no lookup; document_walk.go, lines 51–58, then linearly searches each mapping on the path.

Scaling case. A source has N sibling collections under collections, each with a constant number of records. A single wildcard registration can identify all of them. When this tier upserts/appends over lower collections, the source application contains N sequence contributions requiring projection. Locating each /collections/<key> searches the same N-entry mapping, costing 1 + 2 + ... + N entry comparisons: O(N²), despite linear-sized source material and projection output. This does not require N application registrations.

Potential refactoring. Lazily build one read-only documentLookup for each original application value with active mapped contributions and reuse it, or collect the relevant sequence nodes in one traversal. With V input nodes, S contributions, maximum depth D, and I projected items, the lookup/projection portion can be O(V + S × D + I), rather than repeated wide scans. Preserve authored coordinates, noncontiguous item projections, replacement evidence, and independent output storage. This concerns load-time coordinate projection, not the deferred branch-local explanation-replay experiment.

4. Compound Identity Restoration Rescans Both Records Per Field

Status: open. Priority: medium.

Relevant code. preserveInheritanceIdentityFields in inheritance_scope.go, lines 143–161, searches the child's mapping for each identity field, then searches the merged mapping again. combineInheritanceTemplate invokes it after merging the parent payload with the child in inheritance_combine.go, line 104.

Scaling case. For K identity fields, C child entries, and M merged entries, the searches cost O(K × (C + M)), plus the required identity-value clones. A single parent/child relationship with K = C = M = N direct scalar identity fields performs quadratic comparisons for linear-sized records and identity metadata. There is no declared small bound on compound-identity width. The shared identity validator and parent-identity exclusion already use mapping or membership indexes; this restoration step does not.

Potential refactoring. Index merged entry positions once and walk child entries against an identity-field set, or use equivalent operation-local mapping indexes. This can be O(K + C + M), plus required copies. Preserve child-owned identity values and locations, existing mapping order, and detached nodes; do not change parent lookup or identity equality. If the step proves redundant after the established merge, removal needs a separate equivalence review rather than an assumption that identity preservation is unnecessary.

5. Secret Failure Reporting Repeats Unindexed Location Searches

Status: open. Priority: low; affects batches of failures.

Relevant code. resolveSecretValue reports each failed reference through secretDiagnostic in secret_reference.go, lines 53–78 and 112–117. That helper calls unindexed nearestDocumentLocation; the mapping branch of nearestDocumentLocationWithLookup in validation.go, lines 144–155, falls back to a linear search when no lookup is supplied.

Scaling case. An N-entry mapping of secret references whose environment values are all absent produces N diagnostics. Each location lookup starts at the root and scans the mapping for its key, yielding O(N²) comparisons for O(N) references and diagnostics. Reusing the same environment name still reproduces the lookup pattern even though environment lookup itself is cached. Deterministic traversal and diagnostic sorting add their ordinary O(N log N) costs; they do not require the quadratic location work.

Potential refactoring. Share a lazily initialized read-only documentLookup for the duration of secret resolution, using nearestDocumentLocationWithLookup for failures, or carry matching document/location state through traversal. For V input nodes and F failed references at depth D, attribution can be O(V + F × D), before existing diagnostic sorting. Keep the success path free of an eager index if it is unnecessary. Preserve all diagnostics, nearest-known ancestor fallback, source coordinates, and safe messages; failing fast instead would change behavior rather than optimize it.

6. Dynamic Values Repeat A Type-Graph Eligibility Walk

Status: open. Priority: low; most relevant to repeated dynamic container types.

Relevant code. valueOperation.dynamicDescriptor in value_operation.go, lines 36–48, uses the schema descriptor cache but still calls descriptorContainsSecret with a fresh visited map on every value. That helper (lines 51–68) traverses the reachable descriptor graph. Dynamic branches in projection and cloning invoke it for every non-nil interface value, before handling the concrete value.

Scaling case. Let N dynamic values have the same concrete pointer type whose descriptor graph has size S (nodes plus member/element edges). Each call rechecks the same graph, costing O(N × S), even after descriptor construction is cached. For example, typed defaults can supply N interface entries holding nil pointers to one wide, supported, secret-free struct type. Actual values and projected output contain only N nulls, while the struct has S fields. Taking N = S makes the repeated eligibility walk quadratic in the combined type/value description, without requiring quadratic value expansion.

Potential refactoring. Remember successfully approved dynamic descriptors by concrete type within valueOperation, after complete construction and the secret-semantics check. The repeated-type case becomes O(S + N) eligibility work, with one entry per approved concrete type. Do not treat a traversal-local visited marker as a completed negative result for a cyclic descriptor graph. Do not cache partial descriptors or path-bearing failures; coordinate with finding 2 of the correctness audit, which identifies failed construction cache poisoning. Preserve rejection of dynamic secret-reference types, including nil pointers to them. This is operation-local reuse, not the deferred cross-load schema cache or a redesign of typed cloning.

Validation And Recommended Order

Existing focused tests passed on Go 1.26.10, Linux amd64:

go test ./internal/engine -run 'Collection|Sequence|Inheritance|Secret|Dynamic' -count=1
go test . -run '^TestLocalDocumentationLinks$' -count=1
git diff --check

No benchmarks, timing comparisons, or new tests were needed to establish the loop bounds. Passing tests do not establish efficient growth. Production code and existing correctness findings are unchanged by this audit.

Consider findings 1 and 2 together when selecting a policy-state refactoring, while keeping their acceptance cases distinct. Findings 3 and 4 are narrower independent changes; findings 5 and 6 are lower-priority follow-ups. These are opportunities for consideration, not an implementation sequence or new release blockers. Any later refactoring should preserve the existing contracts and use focused behavioral coverage without adding performance thresholds.