ConfigKit/docs/roadmap/implementation.md

# Implement the performance audit revisions

Status: Planned. Audience: a Luna coding agent implementing one stage at a time,
in numerical order. The [performance audit roadmap](performance-audit.md) owns
the target and rationale. This plan replaces the completed ReleaseKit plan,
which remains available in Git history at `0c77dee`.

Planning baseline: `0c77deed05417edb32f42c7bf24df2fdc2581a25`.
The production baseline is unchanged from the audit's `48619de` revision.
Keep the earlier Path.String sizing and combined final-metadata traversal
improvements. This sprint changes private algorithms, not public API,
configuration semantics, dependencies, Go minimum, or release gates.

## Working instructions

- Read [development.md](../development.md), the applicable agent, architecture,
  testing, and documentation policies, and owning contract/test files before
  each stage. Use the graph tools for code discovery as instructed by AGENTS.md.
- Preserve user changes. Implement only the current stage and its regressions;
  update its status/evidence here and commit the validated unit with a short
  plain-English message. Do not publish, push, or modify consumer dependencies.
- Operation indexes are private phase-local data. Inspection indexes are fully
  built before publication and only read afterward. Returned records remain
  detached. Keep effective and historical indexes separate with independently
  owned mutable path storage.
- Preserve the existing [configuration contract](../config.md): presence,
  null/empty distinctions, composition, diagnostics, chronology, sensitivity,
  callback lifetime, secret isolation, and handle closure. Do not add rejection
  rules, limits, stricter callbacks, concurrency promises, or complexity guarantees.
- Run each stage's focused checks and `./scripts/check.sh` before handing off
  production changes. Run `go test -race ./internal/engine` after retained-index
  or ownership changes, and the full race suite in Stage 18. Reuse checks for
  unchanged content. No new CI wait, release gate, or performance threshold.
- Prefer algorithm reasoning for this sprint. Benchmark runs are not prerequisites.
  Optional measurements use the same prepared local Go, offline dependencies,
  sequential runs, and artifacts outside repositories. Favor B/op and allocs/op
  on the busy host; do not infer CPU improvements from noisy timings.
- Extend existing behavioral tests for missing risks, not traversal counts.
  Small test-only baseline algorithms may support bounded differential coverage.
  Do not keep alternate production engines or switches. If the selected algorithm
  cannot preserve behavior, record the discrepancy and seek review rather than
  strengthening the contract or silently dropping work.

## Resolved implementation choices

1. Containment and path deduplication use canonical token spelling, as today.
   Metadata/document queries additionally enforce contextual key/index shape.
   These predicates remain distinct.
2. Use a small private engine prefix trie with canonical **escaped** pointer
   tokens and integer IDs into owner-held slices. Root is zero tokens; `/`
   is one empty token. Scan separator boundaries without decoding/rejoining
   tokens. Escapes contain no slash. The trie does not validate document shape.
3. The helper returns IDs, not values/provenance. Owners select deepest ancestor,
   earliest inserted ancestor/intersection, or all relevant IDs, explicitly
   restoring owner order. No general path framework or public hook is introduced.
4. Sensitivity uses a worklist: ordinary dependencies activate on intersection,
   while directed inheritance links activate on a classified ancestor/equal
   path of their dependency. Preserve both rules.
5. Inspection retains full replay and on-demand safe rendering. Prepare immutable
   coalesced resolution descriptors and inheritance lookup data once, with no
   per-field tree or rendered-explanation cache.
6. Defer branch-local replay, public metadata-copy redesign, global schema caching,
   and complex contributor-trace membership indexing. Their ownership/memory/
   equality risks exceed the selected scope. Existing clones and linked traces
   stay in place. No new dependency, pool, or package-global mutable cache is needed.

## Stage 1: Confirm the baseline and semantic evidence — Complete

Scope: orientation and test-only evidence, with no optimization yet.

1. Inspect status/revisions. Read the audit and configuration contract sections
   for metadata, sensitivity, explanation, resolution, and overrides, plus their
   implementation/test owners.
2. Run `./scripts/check.sh`. Record revision, local `go version`, result, and
   environmental limitations in an evidence subsection here. Do not repeat or
   revise the completed startup timing campaign.
3. Identify existing coverage for numeric shapes, removed paths, escaped keys,
   inheritance, callback detachment, and independent loads. Add test-only reference
   helpers if useful for later containment, coverage/coalescing, overlap, and
   sensitivity equivalence tests. Keep fixtures bounded and deterministic; do not
   copy the loader. Helpers may instead accompany their first consuming stage.

Done: baseline checks recorded and semantic test owners identified. Existing
contracts/tests are authoritative; reference algorithms add equivalence evidence,
not new public guarantees. A documentation-only stage needs documentation checks.

### Evidence

- Current revision: `52793301ba80eb49f91956009e60cd812f92e94c`; planning baseline:
  `0c77deed05417edb32f42c7bf24df2fdc2581a25`. The working tree was clean, and
  production Go files match the audit baseline `48619de024d794cf41e90b171b7fcbe609dcd437`.
- `GOTOOLCHAIN=local go version`: `go1.26.8 linux/amd64`.
  `GOTOOLCHAIN=local ./scripts/check.sh`: passed (`go test ./...`, `go vet ./...`,
  and `go build ./...`, in addition to formatting and whitespace checks). No
  environmental limitations encountered; no benchmark campaign was repeated.
- Existing semantic coverage includes numeric/key-index and escaped path shapes
  (`path_test.go`, `provenance_test.go`, `override_test.go`), removed/history
  paths (`metadata_test.go`, `resolution_test.go`, `inheritance_history_test.go`),
  nested and sequence-contained inheritance (`inheritance_test.go`,
  `inheritance_walk_test.go`), detached callback/metadata records
  (`callback_ownership_test.go`, `resolution_test.go`, `metadata_test.go`), and
  independent loads (`provenance_test.go`). Differential reference helpers are
  deferred to the first stage that consumes each algorithm so they can remain
  small and bounded.

## Stage 2: Make presence-only queries lightweight — Complete

Roadmap candidate 1. Owner: `internal/engine/metadata.go`.

1. Make `Metadata.IsExplicit` validate effective path/shape and read the stored
   boolean, without calling `Lookup`, cloning origins, or projecting inheritance.
   A shared private effective-field lookup may serve both methods; `Lookup`
   continues returning complete detached records.
2. Format each query pointer once per lookup. Preserve root, zero metadata,
   invalid/unknown/removed paths, parsed numeric tokens, and explicit key/index
   distinctions. Leave `Resolution.IsExplicit` entry-state semantics unchanged.
3. Extend metadata/presence tests only where agreement with `Lookup` lacks
   coverage: direct/inherited input, defaults, and explicit false/zero/null.

Validate: `go test ./internal/engine -run 'Metadata|Presence|Explicit|Inherited'`,
relevant root facade tests, and the everyday check. No allocation assertion.

## Stage 3: Detect inheritance without constructing paths — Complete

Roadmap candidate 9. Owner: `internal/engine/inheritance_load.go`.

1. Replace detection's full path-building walk with a nil-safe boolean visitor:
   inspect mapping keys for `extends`, recurse into values/sequence children,
   and return immediately on a match.
2. Leave validation, expansion order, provenance, and budgets unchanged.
   Detection is not validation and must still find nested invalid syntax.
3. Reuse inheritance tests; add missing sequence-contained or invalid nested
   cases without assertions about how many nodes were visited.

Validate: `go test ./internal/engine -run 'Inheritance|Inherited'` and the
everyday check. Inputs without inheritance still need a linear scan.

## Stage 4: Compare path tokens without whole-pointer formatting — Complete

Roadmap candidate 2. Owners: engine `path.go`, `resolution.go`, `metadata.go`.

1. Add private token-spelling equality. Key/parsed segments compare literal
   bytes; two indexes compare integers. For index versus key/parsed, render the
   index into stack scratch space with `strconv.AppendInt` and compare bytes.
   Do not parse literal tokens as integers: `01`, `+1`, and `1` differ.
2. Implement `pathContains` with existing validity/depth checks and token-prefix
   equality. Root behavior remains unchanged. Escaping is injective, so raw
   token spelling comparison preserves canonical pointer equivalence.
3. Replace textual comparison in `matchesPathShape` with token equality, then
   retain its existing explicit/contextual shape checks and candidate-side
   handling. Apply the same spelling predicate to local textual path deduplication.
4. Differentially compare with the String-based baseline over roots, invalid
   paths, empty/slash/tilde/numeric keys, parsed tokens, nested paths, and integer
   extrema. Reuse existing effective/historical query-shape coverage.

Validate: `go test ./internal/engine -run 'Path|Metadata|Resolution|Override|Inheritance'`
and the everyday check. Public `Path.String` stays unchanged.

## Stage 5: Format sorting keys once per item — Complete

Roadmap candidate 2. Owners: engine resolution, overlay, override application,
document comparison, and diagnostics.

1. Decorate path-bearing items with precomputed canonical pointers, sort with
   existing comparisons/tie-breakers, then project back. Cover
   `orderedResolutionRecords`, `displacedOrigins`, `affectedOverrideOrigins`,
   and the final delta sort in `compareConfigurationValues`.
2. In `canonicalDiagnostics`, precompute paths without changing comparator
   precedence, primary/related location ordering, or standalone comparator use.
   Do not add permanent pointer fields to every record for sorting.
3. Gather recursive displaced definitions into one owned output and sort at
   the outer owner, avoiding repeated sorting of the same nested subtree.
   Preserve final lexical order and duplicate behavior.
4. Reuse ordering tests; add missing tie/escaped-key cases. Do not promise stable
   order for items whose full comparison is equal.

Validate: `go test ./internal/engine -run 'Resolution|Override|Overlay|Diagnostic|CompareDocument|Provenance'`
and the everyday check. Expected benefit is reduced formatting, not measured CPU.

## Stage 6: Add a private canonical-pointer prefix index — Complete

Scope: one helper plus tests, no caller migration yet.

1. Add `internal/engine/path_prefix.go`. Nodes have child maps keyed by escaped
   tokens, terminal integer payload IDs, and the minimum inserted ID within the
   subtree. IDs are nonnegative owner ordinals; represent absence explicitly.
2. Support insertion, ancestor payload visits (root/exact included), descendant
   payload visits (exact included), and earliest intersecting ID using ancestor
   IDs and the queried subtree minimum. Preserve multiple IDs at one terminal.
   No deletion, wildcard search, provider abstraction, or serialization API.
3. Tokenize canonical pointers by separator scanning without split slices.
   Preserve root versus empty keys and `/a` versus `/ab` boundaries. Owners
   supply canonical pointers from valid paths; no new input validation contract.
   Retain IDs only, not nodes, credential bytes, or mutable path segments.
4. Differentially check results against simple prefix scans for ancestors,
   descendants, duplicates, insertion minima, escaping, and root/empty tokens.

Validate: focused helper tests and the everyday check. Storage follows indexed
token material/payloads; tests do not fix representation or allocation counts.

## Stage 7: Index resolution coverage and record reads — Complete

Roadmap candidate 3, first integration. Owner: `internal/engine/resolution.go`.

1. Index already ordered records after the callback. Select deepest covering
   ancestor, earliest owner ID for equal depth, matching the current scan.
   Use this for delta coverage in `runResolution`; retain all validity/helper
   checks and diagnostic order.
2. Add a small snapshot coverage adapter with the same rules for Stage 8.
   IDs refer to owner-held slices and cannot outlive or survive their reordering.
3. Build operation-local read-only mapping lookups for completed before/after
   trees, reusing `mappingIndex` mechanics without mutating trees. Add a private
   indexed document lookup variant with the same validity, shape, bounds, and
   absence checks; retain the plain helper for callers without reuse.
4. Cover parent/exact/more-specific records, equal-depth private fixtures,
   added/removed values, unrecorded changes, and helper-target mismatch.

Validate: `go test ./internal/engine -run 'Resolution|Resolve|Callback|Path'`
and the everyday check. Snapshot ownership/contents remain unchanged.

## Stage 8: Replace per-delta whole-metadata scans — Complete

Roadmap candidate 3, second integration. Owner: engine `load_metadata.go`.

1. In `applyResolutionMetadata`, build one final pointer/node index and one
   snapshot coverage index. Reuse final membership for descendant history and
   field refresh; do not rebuild indexes per delta/field.
2. Index delta paths by original delta ID. Enumerate the union of historical
   and final pointers once, obtain covering delta IDs, and bucket affected pointers
   by delta. Parse paths only when a retained semantic path is actually needed.
   Sort buckets lexically and retain original delta processing order.
3. Apply exact snapshot events first, descendant history second, effective origin
   updates third, then final field refresh, as today. Exclude the covering record's
   own pointer from descendant history and retain the original Previous origins.
   Handle overlapping delta IDs in reference fixtures without assuming every
   possible input is disjoint. Normal structural comparison emits disjoint deltas.
4. For fields, visit pointers once and process matching delta IDs in original
   order. For new fields, use snapshot coverage; preserve null/Explicit state,
   default resolution-origin fallback, and removed-path history.
5. Extend existing tests for many disjoint changes, replacement, removed/new
   descendants, and numeric container-shape changes. A bounded test-only
   reference metadata update can protect phase/order equivalence.

Validate: `go test ./internal/engine -run 'Resolution|Metadata|Provenance|History|Explain'`,
engine race tests, and the everyday check. Complexity is output-sensitive to
affected events, not every delta times every field.

## Stage 9: Collect sensitivity directly from trees — Complete

Roadmap candidate 5. Owner: engine `secret.go`. Keep fixed-point propagation
unchanged until Stage 11 so collection is independently reviewable.

1. Replace temporary pointer/node indexes used only to enumerate classification
   paths/flags with direct visitors into local path and sensitivity accumulators.
   Remove the redundant initial final-tree scan; final still participates in
   the late projection phase before schema/predicate classification.
2. Preserve phase precedence: applications; history/dependencies; inheritance;
   projections then final. Tree/pointer contributions currently install parsed
   segments, while history/contributor contributions can install typed paths.
   Preserve those representations and subsequent overwrites.
3. Skip reparsing only if the current stored path already has that parsed
   representation. Otherwise perform the existing parsed replacement. Do not
   simply substitute explicit walker key/index paths or retain the first shape.
4. Retain all historical/removed paths, controls, dependency-only paths, default/
   dynamic tagged projections, and cloned predicate arguments.
5. Add only missing numeric shape-transition, historical predicate, dynamic tag,
   and predicate-detachment cases in existing sensitivity/inheritance test owners.

Validate: `go test ./internal/engine -run 'Secret|Sensitivity|Inherited|Effective|Explain'`,
engine race tests, and the everyday check. No new predicate order/count guarantee.

## Stage 10: Build and test indexed sensitivity triggers — Complete

Roadmap candidate 6, isolated algorithm. Add private
`internal/engine/secret_propagation.go` and focused tests.

1. Ordinary edges have dependency/target pointers and activate on classified
   ancestor or descendant intersection. Inheritance links become two directed
   edges, each activated only by a classified ancestor/equal dependency path.
2. Index dependency pointers with payload IDs. A new classified pointer visits
   ordinary triggers at ancestors and descendants; inheritance triggers only
   at descendants. Avoid double-visiting exact terminals. Track fired IDs and
   enqueue a target only when it becomes newly sensitive.
3. Seed from the initial sensitive set. Preserve the ordinary dependency filter:
   the old `secretIntersectsPath` sees only classified pointers present in the
   collected paths map. Inheritance's `pathIsSecret` does not require membership.
   Relations use textual spelling, not document query-shape validation.
4. Differentially compare final sets to the old fixed-point algorithm for
   forward/reverse chains, cycles, duplicate edges, classified ancestors/
   descendants, classified pointers absent from paths, root, escapes/numeric
   spellings, and mixed ordinary/inheritance graphs. Use no real credentials.

Validate: focused propagation tests and the everyday check. Match/output-sized
work may still be large for broad dependencies; no blanket linear guarantee.

## Stage 11: Integrate worklist sensitivity into publication — Complete

Roadmap candidate 6, integration. Owner: engine `secret.go`.

1. Feed collected paths, seeds, dependencies, and inheritance links into Stage
   10's closure. Remove the production fixed-point loop/unused scan helper after
   migration; retain the useful reference only in tests.
2. Verify full-load effective rendering and explanations for mixed derived/
   inherited sensitivity, replaced/removed paths, tagged containers, consumed
   controls, and nested copies. All credentials remain outside trees/indexes.
3. Verify independent loads and repeated queries retain no mutable propagation
   state. Queues and trigger indexes are phase-local and discarded on return.

Validate: `go test ./internal/engine -run 'Secret|Sensitivity|Inheritance|History|Explain|Independent'`,
relevant root facade secret/inheritance tests, engine race tests, and the everyday
check. Record collection and propagation outcomes separately.

## Stage 12: Index CLI overlaps and validation reads — Complete

Roadmap candidate 4. Owner: engine `override.go`.

1. Retain input order and the duplicate seen map. Replace the prior-accepted
   overlap scan with earliest intersecting trie lookup using original override
   indices. Duplicate diagnostics continue preceding overlap checks.
2. Insert exactly where `pathAccepted[index] = true` occurs today: after path,
   overlap, version, and directive checks, before value conversion/budgets.
   A later bad value still influences subsequent overlap diagnostics. Do not
   index paths rejected earlier or omit those accepted before value failure.
3. Reuse an operation-local mapping index for read-only base traversal during
   path validation. Preserve missing mapping ancestors, preparation-before-apply,
   CLI source order/identity, rules, and limits.
4. Compare with baseline for disjoint/duplicate/ancestor/descendant paths,
   multiple conflicts, escapes, rejected paths, and later value failures.
   Protect diagnostic codes, paths, and first related-source selection.

Validate: `go test ./internal/engine -run 'Override|CLI|Provenance|Diagnostic'`
and the everyday check. No policy changes.

## Stage 13: Coalesce resolution replay once with indexed ancestors — Complete

Roadmap candidate 7. Owners: engine `history.go`, `load.go`, `load_result.go`.

1. Replace group scans with ancestor lookup selecting the earliest inserted
   matching group, not the deepest. Insert only new groups; merged child paths
   are not independently indexed as groups.
2. Retain original before/after trees and normalization precedence. Use ordered
   membership sets for reason/dependency unions, retaining first occurrence and
   canonical pointer equivalence.
3. Prepare immutable coalesced descriptors once before publication. Explain
   reads them without mutating Changes or borrowed snapshots. Retain original
   snapshots for their other metadata/history-budget uses; node accounting is
   unchanged. No renderer or public records are cached.
4. Differentially check independent/parent/child records, multiple ancestors,
   root, equal writes, union order, normalization, and repeated detached results.

Validate: `go test ./internal/engine -run 'Resolution|History|Explain|Ownership'`,
engine race tests, and the everyday check. Full replay semantics stay intact.

## Stage 14: Index relevant inheritance history operations — Complete

Roadmap candidate 7. Owners: engine metadata, metadata cloning, inheritance history.

1. During final metadata preparation, index inheritance operation IDs by control
   path or transfer destination. Collect intersecting IDs for History, deduplicate,
   restore original operation order, and run existing projection logic.
2. Preserve stored-history splitting before resolution and Previous-origin
   construction. Keep fields/history/path clones independent. A fully frozen
   ID-only index may be shared in `cloneMetadata`, like immutable evidence;
   no query builds or updates it. Zero/private unprepared metadata must still
   produce correct results (use the existing scan if no prepared index exists).
3. Leave global `appendInheritanceExplanation` replay intact: non-intersecting
   operations can affect working state and must not be skipped with this filter.
4. Compare indexed history to the scan for multi-scope controls, removed paths,
   ancestor queries, and query mutations/captured metadata independence.

Validate: `go test ./internal/engine -run 'Inheritance|Inherited|History|Metadata|Independent'`,
engine race tests, and the everyday check.

## Stage 15: Reuse frozen inspection mapping lookups — Complete

Roadmap candidate 7. Owners: publication, inspection, document/snapshot/history helpers.

1. Reuse Stage 7's mapping lookup builder for immutable decoded-input, final,
   and application trees queried repeatedly by inspection. Populate all retained
   maps before return, visiting shared node identities once. Retain lookup positions
   and borrowed immutable nodes, not extra trees or Path copies.
2. Thread read-only lookups into applicable document/snapshot and application
   origin/relevance reads. Preserve validity, shape, bounds, and absence. Read
   populated maps directly, not lazy entries methods that write on first query.
3. Keep plain document lookup on mutable replay trees in this stage. Their
   existing query-local merge indexes are unchanged; do not reuse a frozen map
   after cloning, deletion, or entry-position changes. This avoids adding an
   invalidation protocol just to optimize the selected immutable reads.
4. Reuse shape/history/render tests; add only missing repeated/concurrent
   read-only inspection ownership cases. Do not add a public concurrency promise.
   Record extra retained-index storage honestly; this is a lookup tradeoff,
   not a free allocation reduction. Baseline/full-tree replay remains unchanged.

Validate: `go test ./internal/engine -run 'Path|History|Explain|Effective|Independent|Parallel'`,
engine race tests, and the everyday check.

## Stage 16: Index materialized inheritance-step membership

Roadmap candidate 8. Owner: engine `inheritance_origin.go`.

1. Keep output order and compact linked traces. Use a comparable key with three
   canonical From/To/Reference strings and one encoded ordered Locations string.
   Encode the location count and each SourceID byte length, then its bytes and
   delimited decimal line/column integers. This is collision-free, including
   empty IDs, delimiter characters, and signed coordinates. Arbitrary unescaped
   delimiter concatenation is incorrect; no hashing dependency is needed.
2. Seed membership from existing Origin.Inheritance but retain its existing
   duplicates/order. Only newly proposed steps are deduped, as today.
3. Format candidate pointers once; clone retained paths/locations only for newly
   accepted steps. Keep contributor traversal and transfer direction intact.
4. Compare chains, shared contributors, different location/order steps, existing
   duplicates, escaped/numeric paths, and returned-record mutation to baseline.

Validate: `go test ./internal/engine -run 'Inheritance|Inherited|Metadata|Provenance|History'`
and the everyday check. Complex contributor-trace equality stays unchanged.

## Stage 17: Index simple resolution and origin membership

Roadmap candidate 8. Owners: engine `resolution.go` and `load_metadata.go`.

1. Maintain lazily initialized first-occurrence reason/dependency sets on
   resolution records. Dependencies use canonical pointer equality; retained
   slices and callback validation/order/nil behavior remain unchanged.
2. Use exact Location membership in `originContainedBy`, retaining Layer/
   Operation and contributor-empty checks. Use SourceID membership in
   `originRepresentedByChange`, which has a different source-coverage predicate.
   Do not conflate these equality rules.
3. Leave complex contributor scans and `sameInheritanceTrace` intact. Their
   concrete segment/location equality cannot be replaced by pointer identity
   or canonical spelling. Sets are phase-local, not exported record fields.
4. Reuse existing provenance tests; cover duplicate unions, equal source IDs
   with differing locations, and location coverage with contributor mismatch
   only where protection is missing.

Validate: `go test ./internal/engine -run 'Resolution|Provenance|Overlay|Override|Inheritance|History'`
and the everyday check. Small lists and outputs preserve existing semantics.

## Stage 18: Review the completed result and record outcomes

1. Review the diff from baseline. Map all nine candidates to selected implemented
   changes and expressly deferred portions. Remove dead helpers/imports/state;
   keep useful small equivalence tests, no alternate production engines.
2. Run `./scripts/check.sh` and `go test -race ./...`. Verify public/import
   boundaries, examples, independent loads, shapes, presence, history, redaction,
   and complete-on-return behavior through their existing tests.
3. Record algorithmic before/after work, lifetime/storage tradeoffs, revisions,
   actual commands/results, and unmeasured claims in an evidence subsection here.
   Optional probes may reuse existing startup/assembly/underlay/inheritance
   benchmarks and small record/override/dependency/query fixtures. Measurements
   are not prerequisites; timing claims await a representative host.
4. Update stage and roadmap status accurately. Update focused internal docs
   only where they describe changed implemented mechanics. Keep public contracts
   and policy language unchanged. Do not retire other historical records or
   roadmaps as incidental cleanup.
5. Review links/whitespace/staging, commit the completed unit, and report outcomes,
   checks, remaining deferred experiments, and workspace status. No release or
   downstream bump is included.

Done: selected changes preserve behavior, remain proportionate/maintainable,
pass validation, and distinguish algorithmic expectations from measured results.
No public performance guarantee is created.

## Open Questions

None required for the selected scope. Algorithm, ownership, relation, ordering,
and scope choices above are resolved. Expressly deferred experiments require
separate design/evidence and do not block this sequence.