diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 4a5b3a1..5948443 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,208 +1,448 @@ -# Implement ReleaseKit adoption +# Implement the performance audit revisions -Status: Complete. All seven stages below have been implemented and verified. -The [ReleaseKit adoption roadmap](releasekit.md) owns the target state and -policy choices; this plan records the execution checklist. Use ReleaseKit v0.2.0 -at tagged commit -`e030794969fb7787e647a90bbc4eb81bd0610f07` for the initial migration. -Do not publish a real ConfigKit version as part of implementation. Keep the -current release procedure authoritative until the replacement is working. -Follow the task-specific reading guide in [development.md](../development.md) -and avoid adding gates, compatibility promises, or restrictions beyond the -roadmap and adopted policies. Stages 3–5 are one documentation-and-machinery -cutover; do not commit or hand off their intermediate state as a release-ready -candidate. +Status: Planned. Audience: a Luna coding agent implementing one stage at a time, +in numerical order. The [performance audit roadmap](performance-audit.md) owns +the target and rationale. This plan replaces the completed ReleaseKit plan, +which remains available in Git history at `0c77dee`. -## Stage 1: Select Go and declare the release tool — Complete +Planning baseline: `0c77deed05417edb32f42c7bf24df2fdc2581a25`. +The production baseline is unchanged from the audit's `48619de` revision. +Keep the earlier Path.String sizing and combined final-metadata traversal +improvements. This sprint changes private algorithms, not public API, +configuration semantics, dependencies, Go minimum, or release gates. -1. Identify the latest locally available patch in the newest stable Go series - available in the development environment. Make it the `go` on `PATH` and - verify it with `GOTOOLCHAIN=local go version`. Keep ConfigKit's current - `go.mod` `go` minimum until the release-workflow cutover in Stage 4: the - existing release checker still selects Go 1.25 and cannot validate a module - requiring Go 1.26. Do not substitute an automatically downloaded toolchain - during later validation. -2. Using that same Go executable, add - `gitea.maximumdirect.net/eric/ReleaseKit/cmd/releasekit@v0.2.0` as a - `go.mod` tool dependency (the Go `tool` directive and its module - requirement). Update `go.sum`, run `go mod tidy`, and inspect the module - diff. Preserve ConfigKit's runtime dependency boundary: production library - packages must not import ReleaseKit. Do not introduce a second tool-version - pin or an `@version` installation path in the release procedure. -3. Prepare the module cache and build the selected CLI package with that Go - executable into a temporary path outside the worktree. Confirm that the - resolved ReleaseKit module is v0.2.0 and that the built executable runs. - Review actual dependency and license treatment under the - [OSS policy](../policy/oss.md); a tool used only for development and release - work does not by itself require an attribution entry. +## Working instructions -Completion evidence: `go.mod` retains its current Go minimum and selects the -pinned ReleaseKit tool, `go.sum` is consistent, `go mod tidy` leaves no -unintended changes, the module-selected executable builds with the selected -local Go, and the ordinary offline Go suite still passes after dependencies -are prepared. If the new tool dependency itself requires a higher minimum, -handle that compatibility change together with the cutover rather than -leaving the current release checker unusable. +- Read [development.md](../development.md), the applicable agent, architecture, + testing, and documentation policies, and owning contract/test files before + each stage. Use the graph tools for code discovery as instructed by AGENTS.md. +- Preserve user changes. Implement only the current stage and its regressions; + update its status/evidence here and commit the validated unit with a short + plain-English message. Do not publish, push, or modify consumer dependencies. +- Operation indexes are private phase-local data. Inspection indexes are fully + built before publication and only read afterward. Returned records remain + detached. Keep effective and historical indexes separate with independently + owned mutable path storage. +- Preserve the existing [configuration contract](../config.md): presence, + null/empty distinctions, composition, diagnostics, chronology, sensitivity, + callback lifetime, secret isolation, and handle closure. Do not add rejection + rules, limits, stricter callbacks, concurrency promises, or complexity guarantees. +- Run each stage's focused checks and `./scripts/check.sh` before handing off + production changes. Run `go test -race ./internal/engine` after retained-index + or ownership changes, and the full race suite in Stage 18. Reuse checks for + unchanged content. No new CI wait, release gate, or performance threshold. +- Prefer algorithm reasoning for this sprint. Benchmark runs are not prerequisites. + Optional measurements use the same prepared local Go, offline dependencies, + sequential runs, and artifacts outside repositories. Favor B/op and allocs/op + on the busy host; do not infer CPU improvements from noisy timings. +- Extend existing behavioral tests for missing risks, not traversal counts. + Small test-only baseline algorithms may support bounded differential coverage. + Do not keep alternate production engines or switches. If the selected algorithm + cannot preserve behavior, record the discrepancy and seek review rather than + strengthening the contract or silently dropping work. -## Stage 2: Add ConfigKit's ReleaseKit profile — Complete +## Resolved implementation choices -1. Add `releasekit.json` at the repository root with `product` set to - `ConfigKit`, `remote` set to `origin`, and `expected_push_url` set to - `https://gitea.maximumdirect.net/eric/ConfigKit.git`. Select - `validation.go.race: true`, `validation.go.full_vet: true`, and - `validation.go.compile_targets` in this order: `linux/amd64`, `linux/arm64`. - Omit `build`, `delivery`, `checks`, and the other optional validation - selections. This is a source-only release; macOS is best effort. -2. Add one small, offline ConfigKit-owned test of the committed profile's - selected identity, endpoint, and checks. Keep it limited to configuration - wiring and the agreed gate set; do not replicate ReleaseKit's publication, - recovery, or Go-check implementation tests. Stage 6 will exercise the - profile with the actual tool against an isolated remote. +1. Containment and path deduplication use canonical token spelling, as today. + Metadata/document queries additionally enforce contextual key/index shape. + These predicates remain distinct. +2. Use a small private engine prefix trie with canonical **escaped** pointer + tokens and integer IDs into owner-held slices. Root is zero tokens; `/` + is one empty token. Scan separator boundaries without decoding/rejoining + tokens. Escapes contain no slash. The trie does not validate document shape. +3. The helper returns IDs, not values/provenance. Owners select deepest ancestor, + earliest inserted ancestor/intersection, or all relevant IDs, explicitly + restoring owner order. No general path framework or public hook is introduced. +4. Sensitivity uses a worklist: ordinary dependencies activate on intersection, + while directed inheritance links activate on a classified ancestor/equal + path of their dependency. Preserve both rules. +5. Inspection retains full replay and on-demand safe rendering. Prepare immutable + coalesced resolution descriptors and inheritance lookup data once, with no + per-field tree or rendered-explanation cache. +6. Defer branch-local replay, public metadata-copy redesign, global schema caching, + and complex contributor-trace membership indexing. Their ownership/memory/ + equality risks exceed the selected scope. Existing clones and linked traces + stay in place. No new dependency, pool, or package-global mutable cache is needed. -Completion evidence: the profile is valid JSON, the focused test and ordinary -Go suite pass, and review shows only the selected gates and no binary or hosted -delivery configuration. +## Stage 1: Confirm the baseline and semantic evidence -## Stage 3: Align release policy with the adopted workflow — Complete +Scope: orientation and test-only evidence, with no optimization yet. -1. Revise [the release policy](../policy/release.md) so candidate preparation - explicitly publishes the committed candidate to authoritative `main` - before the one ReleaseKit `publish` invocation validates it and publishes - the annotated tag. Keep the checked-in note and tag as the only deliverables, - successful intended-tag publication as completion, and CI, macOS, hosted - delivery, and consumer retrieval outside release gates. -2. Replace references to the local release scripts and fixed Go 1.25 release - baseline with the selected local Go executable, the current `go.mod` - minimum, the module-selected ReleaseKit CLI, its one race suite with - test-time vet disabled, full vet, and Linux amd64/arm64 package compilation. - State that preparation records the initial stable release of the newest - locally available Go series in `go.mod`, retaining a higher patch minimum - only when required by a dependency or specific correctness fix. Use the - latest locally available compatible patch in that series for ReleaseKit's - build and validation, and review any minimum-version increase under - existing compatibility rules. Keep dependency acquisition and tool build - before offline candidate validation; do not add a separate successful test - or CI run as a publication prerequisite. -3. Align the guard descriptions with ReleaseKit v0.2.0: use its release-note - H1 and exact effective push-URL checks, rather than claiming an automated - exact-first-line or configured fetch-URL check. Retain the required - `# ConfigKit ` first-line format for notes, the source/tag - immutability rules, and the existing recovery principles. Do not require - per-test skip listings; test failures still block publication. +1. Inspect status/revisions. Read the audit and configuration contract sections + for metadata, sensitivity, explanation, resolution, and overrides, plus their + implementation/test owners. +2. Run `./scripts/check.sh`. Record revision, local `go version`, result, and + environmental limitations in an evidence subsection here. Do not repeat or + revise the completed startup timing campaign. +3. Identify existing coverage for numeric shapes, removed paths, escaped keys, + inheritance, callback detachment, and independent loads. Add test-only reference + helpers if useful for later containment, coverage/coalescing, overlap, and + sensitivity equivalence tests. Keep fixtures bounded and deterministic; do not + copy the loader. Helpers may instead accompany their first consuming stage. -Completion evidence: policy and roadmap agree on order, gates, compatibility, -and completion; no policy passage still describes the retired script or its -stronger automated guards as current behavior. +Done: baseline checks recorded and semantic test owners identified. Existing +contracts/tests are authoritative; reference algorithms add equivalence evidence, +not new public guarantees. A documentation-only stage needs documentation checks. -## Stage 4: Replace the maintainer release runbook — Complete +## Stage 2: Make presence-only queries lightweight -1. Revise [the runbook](../release.md) into copyable steps for the adopted - process. During this cutover, move `go.mod` from its current minimum to the - first stable release of the newest Go series available in the development - environment (for example, `go 1.26.0` when using Go 1.26.8), unless a - dependency or specific correctness fix needs a higher patch minimum. Have - the runbook derive the series from `GOTOOLCHAIN=local go env GOVERSION` and - use `go mod edit` to record its `.0` release, without lowering an existing - required higher patch minimum. Keep that series' latest locally available - compatible patch first on `PATH` for both the tool build and validation. - Explain the minimum-Go compatibility impact and version choice, including - the pre-v1.0.0 minor-release rule. Prepare - dependencies, then build the module-selected `cmd/releasekit` with that - executable into temporary storage outside the repository. Use the native - executable for publication so its exit status and diagnostic state are - preserved. Its child Go commands must find the same selected `go` on `PATH`; - the old `CONFIGKIT_BASELINE_TOOLCHAIN` selector cannot choose them. -2. Keep ordinary candidate preparation: check access to the authoritative - remote, choose an unused stable version, review compatibility and content, - write `docs/releases/.md` with the exact ConfigKit first line and - license footer, retire completed roadmaps, and commit the entire candidate. - Do not turn those reviews into additional approval gates. Push only that - committed candidate to authoritative `main` using the literal authoritative - URL and `HEAD:refs/heads/main` refspec, with implicit tag following - disabled. ReleaseKit requires the remote `main` to equal the candidate - before it validates. A failure before this push leaves publication - unstarted; a failed validation after it leaves an untagged `main` candidate - that must be corrected by a new commit and another validation. -3. Document one `releasekit publish vMAJOR.MINOR.PATCH` invocation using the - built, module-selected executable. Describe its selected race suite, full - vet, Linux package builds, release-note and effective-push-URL guards, - exact tag publication, and confirmed-tag completion. Keep `check` optional - for diagnosis, not a repeated required pass. Update the partial/unknown - push recovery directions to ReleaseKit's reported candidate, endpoint, - tag-object ID, and bounded authoritative-state reconciliation. Preserve - optional post-publication consumer verification and no-CI-wait behavior. - Remove old script commands, Go 1.25 pin examples, and promises of verbose - skipped-test output. +Roadmap candidate 1. Owner: `internal/engine/metadata.go`. -Completion evidence: a maintainer can follow the runbook from a clean checkout -through confirmed tag publication without a ConfigKit release wrapper or a -second required Go/test run; all commands, version placeholders, and documented -failure states match ReleaseKit v0.2.0. Do not execute its real publication -command in this stage. +1. Make `Metadata.IsExplicit` validate effective path/shape and read the stored + boolean, without calling `Lookup`, cloning origins, or projecting inheritance. + A shared private effective-field lookup may serve both methods; `Lookup` + continues returning complete detached records. +2. Format each query pointer once per lookup. Preserve root, zero metadata, + invalid/unknown/removed paths, parsed numeric tokens, and explicit key/index + distinctions. Leave `Resolution.IsExplicit` entry-state semantics unchanged. +3. Extend metadata/presence tests only where agreement with `Lookup` lacks + coverage: direct/inherited input, defaults, and explicit false/zero/null. -## Stage 5: Retire duplicated machinery and update developer guidance — Complete +Validate: `go test ./internal/engine -run 'Metadata|Presence|Explicit|Inherited'`, +relevant root facade tests, and the everyday check. No allocation assertion. -1. Remove `scripts/release.sh`, `scripts/release-check.sh`, and - `release_publish_test.go` after Stages 1–4 supply the replacement. Remove - `TestCheckScriptsReleaseGates` and its now-unused helpers from - `release_check_test.go`. Preserve - `TestDevelopmentCheckHandlesFilenamesAndStopsOnFailure` and - `scripts/check.sh`; rename the test file if helpful so its name reflects - its remaining owner. Retain the small Stage 2 profile-wiring test. Do not - transplant the old publication guard/recovery matrix into ConfigKit tests. -2. Update [the development guide](../development.md) to point to the new - release runbook, remove retired-script references and the fixed release - baseline, and adjust its task-to-test map. Keep the everyday development - check and release validation distinct. Review other live references to the - removed scripts and update only those made stale by this migration; - historical release notes remain unchanged. +## Stage 3: Detect inheritance without constructing paths -Completion evidence: no live release instruction invokes a removed script; -the default suite and `./scripts/check.sh` pass with the selected Go and -prepared dependencies; the retained tests protect development checks and -profile wiring without duplicating ReleaseKit's generic behavior. +Roadmap candidate 9. Owner: `internal/engine/inheritance_load.go`. -## Stage 6: Exercise the migration without publishing ConfigKit — Complete +1. Replace detection's full path-building walk with a nil-safe boolean visitor: + inspect mapping keys for `extends`, recurse into values/sequence children, + and return immediately on a match. +2. Leave validation, expansion order, provenance, and budgets unchanged. + Detection is not validation and must still find nested invalid syntax. +3. Reuse inheritance tests; add missing sequence-contained or invalid nested + cases without assertions about how many nodes were visited. -1. Build ReleaseKit from the candidate's `go.mod` selection with the latest - selected local Go patch for that minimum's series. Use that same executable - for ReleaseKit's validation, without requiring its patch number to equal - the `go.mod` minimum. In disposable Git fixtures, use - ConfigKit's selected profile with only `expected_push_url` changed to a - temporary bare remote, and provide a matching committed fixture release - note. Never push a ConfigKit tag or source commit to its real remote. -2. Push a committed fixture candidate to temporary `main` and run the built - `publish` command with a deliberately failing fixture test. Confirm - validation fails and creates no release tag, while `main` remains at that - candidate. Correct and commit the fixture, push the new candidate to - temporary `main`, and run `publish` again. Confirm one race-enabled standard - suite, one full vet pass, and one CGO-disabled package compile for each - selected Linux target in that successful invocation; confirm the remote - annotated tag identifies the corrected candidate and contains the checked-in - note. Record child Go invocations with a fixture-only forwarding executable - on `PATH`, or use equivalent direct evidence, rather than inferring the - selected command count merely from success. +Validate: `go test ./internal/engine -run 'Inheritance|Inherited'` and the +everyday check. Inputs without inheritance still need a linear scan. -Completion evidence: the isolated failure and success paths demonstrate the -selected profile, module-selected tool, command order, gates, and tag-only -publication. No ConfigKit ref is pushed to its real remote. +## Stage 4: Compare path tokens without whole-pointer formatting -## Stage 7: Check endpoint failure and finish the cutover review — Complete +Roadmap candidate 2. Owners: engine `path.go`, `resolution.go`, `metadata.go`. -1. In a fresh disposable fixture/version without an existing tag, check that - a wrong effective push URL is rejected before validation or tag - publication. Do not reuse the version already published in Stage 6, which - would make the existing-tag guard obscure the endpoint result. -2. Compare the runbook's recovery directions with ReleaseKit's reported state - for a failed or uncertain tag push. Repair any mismatch in the runbook; - do not duplicate ReleaseKit's recovery machinery in ConfigKit. This is a - bounded migration exercise, not a new per-release gate or a requirement to - wait for CI. -3. Re-run the affected repository checks after any corrections. Review the - final policy, runbook, development guide, and roadmap together for a - coherent process; confirm that only intended files changed and that no - real ConfigKit release or push occurred. +1. Add private token-spelling equality. Key/parsed segments compare literal + bytes; two indexes compare integers. For index versus key/parsed, render the + index into stack scratch space with `strconv.AppendInt` and compare bytes. + Do not parse literal tokens as integers: `01`, `+1`, and `1` differ. +2. Implement `pathContains` with existing validity/depth checks and token-prefix + equality. Root behavior remains unchanged. Escaping is injective, so raw + token spelling comparison preserves canonical pointer equivalence. +3. Replace textual comparison in `matchesPathShape` with token equality, then + retain its existing explicit/contextual shape checks and candidate-side + handling. Apply the same spelling predicate to local textual path deduplication. +4. Differentially compare with the String-based baseline over roots, invalid + paths, empty/slash/tilde/numeric keys, parsed tokens, nested paths, and integer + extrema. Reuse existing effective/historical query-shape coverage. -Completion evidence: the wrong-endpoint fixture fails for the intended reason, -recovery guidance matches ReleaseKit v0.2.0, tests and documentation checks -pass, and the repository is ready for a separately authorized release. +Validate: `go test ./internal/engine -run 'Path|Metadata|Resolution|Override|Inheritance'` +and the everyday check. Public `Path.String` stays unchanged. + +## Stage 5: Format sorting keys once per item + +Roadmap candidate 2. Owners: engine resolution, overlay, override application, +document comparison, and diagnostics. + +1. Decorate path-bearing items with precomputed canonical pointers, sort with + existing comparisons/tie-breakers, then project back. Cover + `orderedResolutionRecords`, `displacedOrigins`, `affectedOverrideOrigins`, + and the final delta sort in `compareConfigurationValues`. +2. In `canonicalDiagnostics`, precompute paths without changing comparator + precedence, primary/related location ordering, or standalone comparator use. + Do not add permanent pointer fields to every record for sorting. +3. Gather recursive displaced definitions into one owned output and sort at + the outer owner, avoiding repeated sorting of the same nested subtree. + Preserve final lexical order and duplicate behavior. +4. Reuse ordering tests; add missing tie/escaped-key cases. Do not promise stable + order for items whose full comparison is equal. + +Validate: `go test ./internal/engine -run 'Resolution|Override|Overlay|Diagnostic|CompareDocument|Provenance'` +and the everyday check. Expected benefit is reduced formatting, not measured CPU. + +## Stage 6: Add a private canonical-pointer prefix index + +Scope: one helper plus tests, no caller migration yet. + +1. Add `internal/engine/path_prefix.go`. Nodes have child maps keyed by escaped + tokens, terminal integer payload IDs, and the minimum inserted ID within the + subtree. IDs are nonnegative owner ordinals; represent absence explicitly. +2. Support insertion, ancestor payload visits (root/exact included), descendant + payload visits (exact included), and earliest intersecting ID using ancestor + IDs and the queried subtree minimum. Preserve multiple IDs at one terminal. + No deletion, wildcard search, provider abstraction, or serialization API. +3. Tokenize canonical pointers by separator scanning without split slices. + Preserve root versus empty keys and `/a` versus `/ab` boundaries. Owners + supply canonical pointers from valid paths; no new input validation contract. + Retain IDs only, not nodes, credential bytes, or mutable path segments. +4. Differentially check results against simple prefix scans for ancestors, + descendants, duplicates, insertion minima, escaping, and root/empty tokens. + +Validate: focused helper tests and the everyday check. Storage follows indexed +token material/payloads; tests do not fix representation or allocation counts. + +## Stage 7: Index resolution coverage and record reads + +Roadmap candidate 3, first integration. Owner: `internal/engine/resolution.go`. + +1. Index already ordered records after the callback. Select deepest covering + ancestor, earliest owner ID for equal depth, matching the current scan. + Use this for delta coverage in `runResolution`; retain all validity/helper + checks and diagnostic order. +2. Add a small snapshot coverage adapter with the same rules for Stage 8. + IDs refer to owner-held slices and cannot outlive or survive their reordering. +3. Build operation-local read-only mapping lookups for completed before/after + trees, reusing `mappingIndex` mechanics without mutating trees. Add a private + indexed document lookup variant with the same validity, shape, bounds, and + absence checks; retain the plain helper for callers without reuse. +4. Cover parent/exact/more-specific records, equal-depth private fixtures, + added/removed values, unrecorded changes, and helper-target mismatch. + +Validate: `go test ./internal/engine -run 'Resolution|Resolve|Callback|Path'` +and the everyday check. Snapshot ownership/contents remain unchanged. + +## Stage 8: Replace per-delta whole-metadata scans + +Roadmap candidate 3, second integration. Owner: engine `load_metadata.go`. + +1. In `applyResolutionMetadata`, build one final pointer/node index and one + snapshot coverage index. Reuse final membership for descendant history and + field refresh; do not rebuild indexes per delta/field. +2. Index delta paths by original delta ID. Enumerate the union of historical + and final pointers once, obtain covering delta IDs, and bucket affected pointers + by delta. Parse paths only when a retained semantic path is actually needed. + Sort buckets lexically and retain original delta processing order. +3. Apply exact snapshot events first, descendant history second, effective origin + updates third, then final field refresh, as today. Exclude the covering record's + own pointer from descendant history and retain the original Previous origins. + Handle overlapping delta IDs in reference fixtures without assuming every + possible input is disjoint. Normal structural comparison emits disjoint deltas. +4. For fields, visit pointers once and process matching delta IDs in original + order. For new fields, use snapshot coverage; preserve null/Explicit state, + default resolution-origin fallback, and removed-path history. +5. Extend existing tests for many disjoint changes, replacement, removed/new + descendants, and numeric container-shape changes. A bounded test-only + reference metadata update can protect phase/order equivalence. + +Validate: `go test ./internal/engine -run 'Resolution|Metadata|Provenance|History|Explain'`, +engine race tests, and the everyday check. Complexity is output-sensitive to +affected events, not every delta times every field. + +## Stage 9: Collect sensitivity directly from trees + +Roadmap candidate 5. Owner: engine `secret.go`. Keep fixed-point propagation +unchanged until Stage 11 so collection is independently reviewable. + +1. Replace temporary pointer/node indexes used only to enumerate classification + paths/flags with direct visitors into local path and sensitivity accumulators. + Remove the redundant initial final-tree scan; final still participates in + the late projection phase before schema/predicate classification. +2. Preserve phase precedence: applications; history/dependencies; inheritance; + projections then final. Tree/pointer contributions currently install parsed + segments, while history/contributor contributions can install typed paths. + Preserve those representations and subsequent overwrites. +3. Skip reparsing only if the current stored path already has that parsed + representation. Otherwise perform the existing parsed replacement. Do not + simply substitute explicit walker key/index paths or retain the first shape. +4. Retain all historical/removed paths, controls, dependency-only paths, default/ + dynamic tagged projections, and cloned predicate arguments. +5. Add only missing numeric shape-transition, historical predicate, dynamic tag, + and predicate-detachment cases in existing sensitivity/inheritance test owners. + +Validate: `go test ./internal/engine -run 'Secret|Sensitivity|Inherited|Effective|Explain'`, +engine race tests, and the everyday check. No new predicate order/count guarantee. + +## Stage 10: Build and test indexed sensitivity triggers + +Roadmap candidate 6, isolated algorithm. Add private +`internal/engine/secret_propagation.go` and focused tests. + +1. Ordinary edges have dependency/target pointers and activate on classified + ancestor or descendant intersection. Inheritance links become two directed + edges, each activated only by a classified ancestor/equal dependency path. +2. Index dependency pointers with payload IDs. A new classified pointer visits + ordinary triggers at ancestors and descendants; inheritance triggers only + at descendants. Avoid double-visiting exact terminals. Track fired IDs and + enqueue a target only when it becomes newly sensitive. +3. Seed from the initial sensitive set. Preserve the ordinary dependency filter: + the old `secretIntersectsPath` sees only classified pointers present in the + collected paths map. Inheritance's `pathIsSecret` does not require membership. + Relations use textual spelling, not document query-shape validation. +4. Differentially compare final sets to the old fixed-point algorithm for + forward/reverse chains, cycles, duplicate edges, classified ancestors/ + descendants, classified pointers absent from paths, root, escapes/numeric + spellings, and mixed ordinary/inheritance graphs. Use no real credentials. + +Validate: focused propagation tests and the everyday check. Match/output-sized +work may still be large for broad dependencies; no blanket linear guarantee. + +## Stage 11: Integrate worklist sensitivity into publication + +Roadmap candidate 6, integration. Owner: engine `secret.go`. + +1. Feed collected paths, seeds, dependencies, and inheritance links into Stage + 10's closure. Remove the production fixed-point loop/unused scan helper after + migration; retain the useful reference only in tests. +2. Verify full-load effective rendering and explanations for mixed derived/ + inherited sensitivity, replaced/removed paths, tagged containers, consumed + controls, and nested copies. All credentials remain outside trees/indexes. +3. Verify independent loads and repeated queries retain no mutable propagation + state. Queues and trigger indexes are phase-local and discarded on return. + +Validate: `go test ./internal/engine -run 'Secret|Sensitivity|Inheritance|History|Explain|Independent'`, +relevant root facade secret/inheritance tests, engine race tests, and the everyday +check. Record collection and propagation outcomes separately. + +## Stage 12: Index CLI overlaps and validation reads + +Roadmap candidate 4. Owner: engine `override.go`. + +1. Retain input order and the duplicate seen map. Replace the prior-accepted + overlap scan with earliest intersecting trie lookup using original override + indices. Duplicate diagnostics continue preceding overlap checks. +2. Insert exactly where `pathAccepted[index] = true` occurs today: after path, + overlap, version, and directive checks, before value conversion/budgets. + A later bad value still influences subsequent overlap diagnostics. Do not + index paths rejected earlier or omit those accepted before value failure. +3. Reuse an operation-local mapping index for read-only base traversal during + path validation. Preserve missing mapping ancestors, preparation-before-apply, + CLI source order/identity, rules, and limits. +4. Compare with baseline for disjoint/duplicate/ancestor/descendant paths, + multiple conflicts, escapes, rejected paths, and later value failures. + Protect diagnostic codes, paths, and first related-source selection. + +Validate: `go test ./internal/engine -run 'Override|CLI|Provenance|Diagnostic'` +and the everyday check. No policy changes. + +## Stage 13: Coalesce resolution replay once with indexed ancestors + +Roadmap candidate 7. Owners: engine `history.go`, `load.go`, `load_result.go`. + +1. Replace group scans with ancestor lookup selecting the earliest inserted + matching group, not the deepest. Insert only new groups; merged child paths + are not independently indexed as groups. +2. Retain original before/after trees and normalization precedence. Use ordered + membership sets for reason/dependency unions, retaining first occurrence and + canonical pointer equivalence. +3. Prepare immutable coalesced descriptors once before publication. Explain + reads them without mutating Changes or borrowed snapshots. Retain original + snapshots for their other metadata/history-budget uses; node accounting is + unchanged. No renderer or public records are cached. +4. Differentially check independent/parent/child records, multiple ancestors, + root, equal writes, union order, normalization, and repeated detached results. + +Validate: `go test ./internal/engine -run 'Resolution|History|Explain|Ownership'`, +engine race tests, and the everyday check. Full replay semantics stay intact. + +## Stage 14: Index relevant inheritance history operations + +Roadmap candidate 7. Owners: engine metadata, metadata cloning, inheritance history. + +1. During final metadata preparation, index inheritance operation IDs by control + path or transfer destination. Collect intersecting IDs for History, deduplicate, + restore original operation order, and run existing projection logic. +2. Preserve stored-history splitting before resolution and Previous-origin + construction. Keep fields/history/path clones independent. A fully frozen + ID-only index may be shared in `cloneMetadata`, like immutable evidence; + no query builds or updates it. Zero/private unprepared metadata must still + produce correct results (use the existing scan if no prepared index exists). +3. Leave global `appendInheritanceExplanation` replay intact: non-intersecting + operations can affect working state and must not be skipped with this filter. +4. Compare indexed history to the scan for multi-scope controls, removed paths, + ancestor queries, and query mutations/captured metadata independence. + +Validate: `go test ./internal/engine -run 'Inheritance|Inherited|History|Metadata|Independent'`, +engine race tests, and the everyday check. + +## Stage 15: Reuse frozen inspection mapping lookups + +Roadmap candidate 7. Owners: publication, inspection, document/snapshot/history helpers. + +1. Reuse Stage 7's mapping lookup builder for immutable decoded-input, final, + and application trees queried repeatedly by inspection. Populate all retained + maps before return, visiting shared node identities once. Retain lookup positions + and borrowed immutable nodes, not extra trees or Path copies. +2. Thread read-only lookups into applicable document/snapshot and application + origin/relevance reads. Preserve validity, shape, bounds, and absence. Read + populated maps directly, not lazy entries methods that write on first query. +3. Keep plain document lookup on mutable replay trees in this stage. Their + existing query-local merge indexes are unchanged; do not reuse a frozen map + after cloning, deletion, or entry-position changes. This avoids adding an + invalidation protocol just to optimize the selected immutable reads. +4. Reuse shape/history/render tests; add only missing repeated/concurrent + read-only inspection ownership cases. Do not add a public concurrency promise. + Record extra retained-index storage honestly; this is a lookup tradeoff, + not a free allocation reduction. Baseline/full-tree replay remains unchanged. + +Validate: `go test ./internal/engine -run 'Path|History|Explain|Effective|Independent|Parallel'`, +engine race tests, and the everyday check. + +## Stage 16: Index materialized inheritance-step membership + +Roadmap candidate 8. Owner: engine `inheritance_origin.go`. + +1. Keep output order and compact linked traces. Use a comparable key with three + canonical From/To/Reference strings and one encoded ordered Locations string. + Encode the location count and each SourceID byte length, then its bytes and + delimited decimal line/column integers. This is collision-free, including + empty IDs, delimiter characters, and signed coordinates. Arbitrary unescaped + delimiter concatenation is incorrect; no hashing dependency is needed. +2. Seed membership from existing Origin.Inheritance but retain its existing + duplicates/order. Only newly proposed steps are deduped, as today. +3. Format candidate pointers once; clone retained paths/locations only for newly + accepted steps. Keep contributor traversal and transfer direction intact. +4. Compare chains, shared contributors, different location/order steps, existing + duplicates, escaped/numeric paths, and returned-record mutation to baseline. + +Validate: `go test ./internal/engine -run 'Inheritance|Inherited|Metadata|Provenance|History'` +and the everyday check. Complex contributor-trace equality stays unchanged. + +## Stage 17: Index simple resolution and origin membership + +Roadmap candidate 8. Owners: engine `resolution.go` and `load_metadata.go`. + +1. Maintain lazily initialized first-occurrence reason/dependency sets on + resolution records. Dependencies use canonical pointer equality; retained + slices and callback validation/order/nil behavior remain unchanged. +2. Use exact Location membership in `originContainedBy`, retaining Layer/ + Operation and contributor-empty checks. Use SourceID membership in + `originRepresentedByChange`, which has a different source-coverage predicate. + Do not conflate these equality rules. +3. Leave complex contributor scans and `sameInheritanceTrace` intact. Their + concrete segment/location equality cannot be replaced by pointer identity + or canonical spelling. Sets are phase-local, not exported record fields. +4. Reuse existing provenance tests; cover duplicate unions, equal source IDs + with differing locations, and location coverage with contributor mismatch + only where protection is missing. + +Validate: `go test ./internal/engine -run 'Resolution|Provenance|Overlay|Override|Inheritance|History'` +and the everyday check. Small lists and outputs preserve existing semantics. + +## Stage 18: Review the completed result and record outcomes + +1. Review the diff from baseline. Map all nine candidates to selected implemented + changes and expressly deferred portions. Remove dead helpers/imports/state; + keep useful small equivalence tests, no alternate production engines. +2. Run `./scripts/check.sh` and `go test -race ./...`. Verify public/import + boundaries, examples, independent loads, shapes, presence, history, redaction, + and complete-on-return behavior through their existing tests. +3. Record algorithmic before/after work, lifetime/storage tradeoffs, revisions, + actual commands/results, and unmeasured claims in an evidence subsection here. + Optional probes may reuse existing startup/assembly/underlay/inheritance + benchmarks and small record/override/dependency/query fixtures. Measurements + are not prerequisites; timing claims await a representative host. +4. Update stage and roadmap status accurately. Update focused internal docs + only where they describe changed implemented mechanics. Keep public contracts + and policy language unchanged. Do not retire other historical records or + roadmaps as incidental cleanup. +5. Review links/whitespace/staging, commit the completed unit, and report outcomes, + checks, remaining deferred experiments, and workspace status. No release or + downstream bump is included. + +Done: selected changes preserve behavior, remain proportionate/maintainable, +pass validation, and distinguish algorithmic expectations from measured results. +No public performance guarantee is created. + +## Open Questions + +None required for the selected scope. Algorithm, ownership, relation, ordering, +and scope choices above are resolved. Expressly deferred experiments require +separate design/evidence and do not block this sequence. diff --git a/docs/roadmap/performance-audit.md b/docs/roadmap/performance-audit.md index 45761b5..c4d9a8e 100644 --- a/docs/roadmap/performance-audit.md +++ b/docs/roadmap/performance-audit.md @@ -28,6 +28,26 @@ themselves grow substantially. An output-sensitive algorithm cannot avoid producing the output required by the existing contract. Expected map lookup costs and path processing costs also qualify any proposed linear bound. +## Target state and selected scope + +The implementation target is the same public configuration model with less +avoidable auxiliary work: presence-only queries do not materialize provenance; +path comparisons and sorts avoid repeated formatting; resolution coverage, +override overlaps, and sensitivity propagation use local path indexes; +classification collects paths directly; and frozen inspection reuses prepared +resolution/inheritance lookup data. Inheritance detection does not construct +semantic paths merely to detect a keyword. Ordered provenance collections use +membership indexes where their equality rules are straightforward. + +The [implementation plan](implementation.md) selects bounded revisions from +all nine candidates below. Whole-tree explanation replay remains in place; +branch-local replay, changing the public metadata-copy handoff, global schema +caching, and indexing complex contributor trace equality remain later +experiments. These have larger ownership, memory, or equivalence risks than the +selected changes. No new consumer-facing complexity bound or performance +threshold is part of the target. The candidate descriptions retain the broader +audit context; the plan owns the selected implementation details and sequence. + ## Candidates ### 1. Avoid materializing provenance for presence-only queries @@ -67,9 +87,9 @@ Many nested scans below repeatedly invoke it. Sorting comparators also repeatedl format paths, making formatting/allocation work proportional to comparisons rather than items. `Lookup` formats the same query for multiple lookups. -First reuse an already formatted pointer within a function or decorate sortable -items with a pointer once. Consider an allocation-free containment comparison -over token spellings as a separate experiment. Complexity remains proportional +Reusing an already formatted pointer within a function and decorating sortable +items with a pointer once avoid repeated formatting. Allocation-free containment +comparison over token spellings is also in the selected scope. Cost remains proportional to path length, but repeated string allocations can disappear. Containment currently compares canonical pointer spellings, while metadata @@ -203,11 +223,11 @@ linearly scans each mapping on its path; querying every key in a wide mapping can therefore add quadratic lookup work. These costs compound across queries, even if each query asks about a small unrelated leaf. -First consider preparing immutable coalesced replay descriptors once when an -inspection is constructed, and indexing relevant inheritance operations and -frozen-tree lookups. Coalescing can use an ancestor index rather than a group -scan. A later, separate experiment could replay only the queried branch and its -necessary dependencies instead of cloning whole trees. +The selected target prepares immutable coalesced replay descriptors once when an +inspection is constructed and indexes relevant inheritance history operations +and frozen-tree lookups. Coalescing uses ancestor lookup rather than a group +scan. Replaying only the queried branch and its necessary dependencies remains +a deferred experiment; whole-tree replay remains in the selected target. Preserve the first covering coalesced group, ordered reason/dependency unions, normalization precedence, chronology, intermediate absence/replacement, typed @@ -274,30 +294,18 @@ to change scope rules, traversal ordering during expansion, or resource budgets. | Public facade and detached records | Root-owned wrappers and explicit conversions preserve the public/private boundary. Some records are copied in the engine and again during conversion; eliminating a copy needs a clear internal ownership handoff, not public aliases or borrowed mutable records. `Result.Metadata()` also clones the complete metadata before queries; consumers repeatedly obtaining it pay repeated full-copy costs. Treat redesign of that private handoff as a separate, lower-priority retained-memory/ownership experiment. | | Effective YAML, diagnostics, examples, and release/development tooling | Deterministic rendering and diagnostics require ordering and output-sized work; path comparator allocation is covered above. Safe snapshots and YAML adaptation allocate separate trees. Maintained consumer examples and check tooling are not production library hot paths; no gate or tooling changes proposed. ReleaseKit internals are outside this repository audit. | -## Recommended follow-up +## Acceptance and boundaries -Start with candidate 1 or 9 as a small, reviewable optimization, then candidate -2's local formatting reuse and candidate 5's direct classification traversal. -These avoid work without demanding new architectural machinery. Next address -candidate 3 for many recorded changes and candidate 6 for large dependency -graphs. Candidates 4, 7, and 8 are useful follow-ups for consumers with many -overrides, inspection queries, or inherited contributions. +The selected revisions preserve the existing [configuration contract](../config.md) +and [architecture policy](../policy/architecture.md): complete load results, +effective versus historical path shapes, provenance, ownership, presence, +secret isolation, and synchronous completion remain unchanged. New caps, +input restrictions, composition rules, or performance promises are not part of +this work. The plan owns implementation sequencing, validation, and evidence. -This is a prioritized backlog, not a staged implementation plan or a requirement -to implement every candidate. Before each production change, confirm the proposed -algorithm against the existing [configuration contract](../config.md), relevant -tests, and [architecture policy](../policy/architecture.md). Preserve complete -load results, effective versus historical path shapes, provenance, ownership, -presence, secret isolation, and synchronous completion. Do not add caps, reject -previously valid inputs, change composition rules, or create performance promises -to simplify an optimization. - -For a selected change, add only missing behavioral regressions, run the relevant -package tests and everyday [development check](../development.md), and use race -checks where ownership/shared state changes warrant them. Follow-up measurements -can reuse existing startup, composition, underlay, inheritance, and history -benchmarks, with targeted fixtures for record count, disjoint overrides, -dependency chains, and repeated queries where coverage is missing. Prefer -allocations and bytes on the busy host; defer CPU improvement claims until -timing is representative. Benchmarks help decide whether the implementation's -complexity is worthwhile, but are not new release gates or test thresholds. +Success is removal of the identified redundant work and broad pairwise scans +without disproportionate machinery or semantic regressions. Output-sized work, +deterministic ordering, and required ownership copies remain legitimate costs. +Benchmarks may inform the tradeoff, especially allocations on the busy host, +but are not new release gates or test thresholds. Unmeasured improvements +remain algorithmic expectations rather than quantified runtime claims.