eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan the performance improvements

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-09 03:13:22 +00:00
parent 0c77deed05
commit 52793301ba
2 changed files with 467 additions and 219 deletions
Download Patch File Download Diff File Expand all files Collapse all files

612
docs/roadmap/implementation.md
View File

@@ -1,208 +1,448 @@
# Implement ReleaseKit adoption
# Implement the performance audit revisions
Status: Complete. All seven stages below have been implemented and verified.
The [ReleaseKit adoption roadmap](releasekit.md) owns the target state and
policy choices; this plan records the execution checklist. Use ReleaseKit v0.2.0
at tagged commit
`e030794969fb7787e647a90bbc4eb81bd0610f07` for the initial migration.
Do not publish a real ConfigKit version as part of implementation. Keep the
current release procedure authoritative until the replacement is working.
Follow the task-specific reading guide in [development.md](../development.md)
and avoid adding gates, compatibility promises, or restrictions beyond the
roadmap and adopted policies. Stages 3–5 are one documentation-and-machinery
cutover; do not commit or hand off their intermediate state as a release-ready
candidate.
Status: Planned. Audience: a Luna coding agent implementing one stage at a time,
in numerical order. The [performance audit roadmap](performance-audit.md) owns
the target and rationale. This plan replaces the completed ReleaseKit plan,
which remains available in Git history at `0c77dee`.
## Stage 1: Select Go and declare the release tool — Complete
Planning baseline: `0c77deed05417edb32f42c7bf24df2fdc2581a25`.
The production baseline is unchanged from the audit's `48619de` revision.
Keep the earlier Path.String sizing and combined final-metadata traversal
improvements. This sprint changes private algorithms, not public API,
configuration semantics, dependencies, Go minimum, or release gates.
1. Identify the latest locally available patch in the newest stable Go series
available in the development environment. Make it the `go` on `PATH` and
verify it with `GOTOOLCHAIN=local go version`. Keep ConfigKit's current
`go.mod` `go` minimum until the release-workflow cutover in Stage 4: the
existing release checker still selects Go 1.25 and cannot validate a module
requiring Go 1.26. Do not substitute an automatically downloaded toolchain
during later validation.
2. Using that same Go executable, add
`gitea.maximumdirect.net/eric/ReleaseKit/cmd/releasekit@v0.2.0` as a
`go.mod` tool dependency (the Go `tool` directive and its module
requirement). Update `go.sum`, run `go mod tidy`, and inspect the module
diff. Preserve ConfigKit's runtime dependency boundary: production library
packages must not import ReleaseKit. Do not introduce a second tool-version
pin or an `@version` installation path in the release procedure.
3. Prepare the module cache and build the selected CLI package with that Go
executable into a temporary path outside the worktree. Confirm that the
resolved ReleaseKit module is v0.2.0 and that the built executable runs.
Review actual dependency and license treatment under the
[OSS policy](../policy/oss.md); a tool used only for development and release
work does not by itself require an attribution entry.
## Working instructions
Completion evidence: `go.mod` retains its current Go minimum and selects the
pinned ReleaseKit tool, `go.sum` is consistent, `go mod tidy` leaves no
unintended changes, the module-selected executable builds with the selected
local Go, and the ordinary offline Go suite still passes after dependencies
are prepared. If the new tool dependency itself requires a higher minimum,
handle that compatibility change together with the cutover rather than
leaving the current release checker unusable.
- Read [development.md](../development.md), the applicable agent, architecture,
testing, and documentation policies, and owning contract/test files before
each stage. Use the graph tools for code discovery as instructed by AGENTS.md.
- Preserve user changes. Implement only the current stage and its regressions;
update its status/evidence here and commit the validated unit with a short
plain-English message. Do not publish, push, or modify consumer dependencies.
- Operation indexes are private phase-local data. Inspection indexes are fully
built before publication and only read afterward. Returned records remain
detached. Keep effective and historical indexes separate with independently
owned mutable path storage.
- Preserve the existing [configuration contract](../config.md): presence,
null/empty distinctions, composition, diagnostics, chronology, sensitivity,
callback lifetime, secret isolation, and handle closure. Do not add rejection
rules, limits, stricter callbacks, concurrency promises, or complexity guarantees.
- Run each stage's focused checks and `./scripts/check.sh` before handing off
production changes. Run `go test -race ./internal/engine` after retained-index
or ownership changes, and the full race suite in Stage 18. Reuse checks for
unchanged content. No new CI wait, release gate, or performance threshold.
- Prefer algorithm reasoning for this sprint. Benchmark runs are not prerequisites.
Optional measurements use the same prepared local Go, offline dependencies,
sequential runs, and artifacts outside repositories. Favor B/op and allocs/op
on the busy host; do not infer CPU improvements from noisy timings.
- Extend existing behavioral tests for missing risks, not traversal counts.
Small test-only baseline algorithms may support bounded differential coverage.
Do not keep alternate production engines or switches. If the selected algorithm
cannot preserve behavior, record the discrepancy and seek review rather than
strengthening the contract or silently dropping work.
## Stage 2: Add ConfigKit's ReleaseKit profile — Complete
## Resolved implementation choices
1. Add `releasekit.json` at the repository root with `product` set to
`ConfigKit`, `remote` set to `origin`, and `expected_push_url` set to
`https://gitea.maximumdirect.net/eric/ConfigKit.git`. Select
`validation.go.race: true`, `validation.go.full_vet: true`, and
`validation.go.compile_targets` in this order: `linux/amd64`, `linux/arm64`.
Omit `build`, `delivery`, `checks`, and the other optional validation
selections. This is a source-only release; macOS is best effort.
2. Add one small, offline ConfigKit-owned test of the committed profile's
selected identity, endpoint, and checks. Keep it limited to configuration
wiring and the agreed gate set; do not replicate ReleaseKit's publication,
recovery, or Go-check implementation tests. Stage 6 will exercise the
profile with the actual tool against an isolated remote.
1. Containment and path deduplication use canonical token spelling, as today.
Metadata/document queries additionally enforce contextual key/index shape.
These predicates remain distinct.
2. Use a small private engine prefix trie with canonical **escaped** pointer
tokens and integer IDs into owner-held slices. Root is zero tokens; `/`
is one empty token. Scan separator boundaries without decoding/rejoining
tokens. Escapes contain no slash. The trie does not validate document shape.
3. The helper returns IDs, not values/provenance. Owners select deepest ancestor,
earliest inserted ancestor/intersection, or all relevant IDs, explicitly
restoring owner order. No general path framework or public hook is introduced.
4. Sensitivity uses a worklist: ordinary dependencies activate on intersection,
while directed inheritance links activate on a classified ancestor/equal
path of their dependency. Preserve both rules.
5. Inspection retains full replay and on-demand safe rendering. Prepare immutable
coalesced resolution descriptors and inheritance lookup data once, with no
per-field tree or rendered-explanation cache.
6. Defer branch-local replay, public metadata-copy redesign, global schema caching,
and complex contributor-trace membership indexing. Their ownership/memory/
equality risks exceed the selected scope. Existing clones and linked traces
stay in place. No new dependency, pool, or package-global mutable cache is needed.
Completion evidence: the profile is valid JSON, the focused test and ordinary
Go suite pass, and review shows only the selected gates and no binary or hosted
delivery configuration.
## Stage 1: Confirm the baseline and semantic evidence
## Stage 3: Align release policy with the adopted workflow — Complete
Scope: orientation and test-only evidence, with no optimization yet.
1. Revise [the release policy](../policy/release.md) so candidate preparation
explicitly publishes the committed candidate to authoritative `main`
before the one ReleaseKit `publish` invocation validates it and publishes
the annotated tag. Keep the checked-in note and tag as the only deliverables,
successful intended-tag publication as completion, and CI, macOS, hosted
delivery, and consumer retrieval outside release gates.
2. Replace references to the local release scripts and fixed Go 1.25 release
baseline with the selected local Go executable, the current `go.mod`
minimum, the module-selected ReleaseKit CLI, its one race suite with
test-time vet disabled, full vet, and Linux amd64/arm64 package compilation.
State that preparation records the initial stable release of the newest
locally available Go series in `go.mod`, retaining a higher patch minimum
only when required by a dependency or specific correctness fix. Use the
latest locally available compatible patch in that series for ReleaseKit's
build and validation, and review any minimum-version increase under
existing compatibility rules. Keep dependency acquisition and tool build
before offline candidate validation; do not add a separate successful test
or CI run as a publication prerequisite.
3. Align the guard descriptions with ReleaseKit v0.2.0: use its release-note
H1 and exact effective push-URL checks, rather than claiming an automated
exact-first-line or configured fetch-URL check. Retain the required
`# ConfigKit <version>` first-line format for notes, the source/tag
immutability rules, and the existing recovery principles. Do not require
per-test skip listings; test failures still block publication.
1. Inspect status/revisions. Read the audit and configuration contract sections
for metadata, sensitivity, explanation, resolution, and overrides, plus their
implementation/test owners.
2. Run `./scripts/check.sh`. Record revision, local `go version`, result, and
environmental limitations in an evidence subsection here. Do not repeat or
revise the completed startup timing campaign.
3. Identify existing coverage for numeric shapes, removed paths, escaped keys,
inheritance, callback detachment, and independent loads. Add test-only reference
helpers if useful for later containment, coverage/coalescing, overlap, and
sensitivity equivalence tests. Keep fixtures bounded and deterministic; do not
copy the loader. Helpers may instead accompany their first consuming stage.
Completion evidence: policy and roadmap agree on order, gates, compatibility,
and completion; no policy passage still describes the retired script or its
stronger automated guards as current behavior.
Done: baseline checks recorded and semantic test owners identified. Existing
contracts/tests are authoritative; reference algorithms add equivalence evidence,
not new public guarantees. A documentation-only stage needs documentation checks.
## Stage 4: Replace the maintainer release runbook — Complete
## Stage 2: Make presence-only queries lightweight
1. Revise [the runbook](../release.md) into copyable steps for the adopted
process. During this cutover, move `go.mod` from its current minimum to the
first stable release of the newest Go series available in the development
environment (for example, `go 1.26.0` when using Go 1.26.8), unless a
dependency or specific correctness fix needs a higher patch minimum. Have
the runbook derive the series from `GOTOOLCHAIN=local go env GOVERSION` and
use `go mod edit` to record its `.0` release, without lowering an existing
required higher patch minimum. Keep that series' latest locally available
compatible patch first on `PATH` for both the tool build and validation.
Explain the minimum-Go compatibility impact and version choice, including
the pre-v1.0.0 minor-release rule. Prepare
dependencies, then build the module-selected `cmd/releasekit` with that
executable into temporary storage outside the repository. Use the native
executable for publication so its exit status and diagnostic state are
preserved. Its child Go commands must find the same selected `go` on `PATH`;
the old `CONFIGKIT_BASELINE_TOOLCHAIN` selector cannot choose them.
2. Keep ordinary candidate preparation: check access to the authoritative
remote, choose an unused stable version, review compatibility and content,
write `docs/releases/<version>.md` with the exact ConfigKit first line and
license footer, retire completed roadmaps, and commit the entire candidate.
Do not turn those reviews into additional approval gates. Push only that
committed candidate to authoritative `main` using the literal authoritative
URL and `HEAD:refs/heads/main` refspec, with implicit tag following
disabled. ReleaseKit requires the remote `main` to equal the candidate
before it validates. A failure before this push leaves publication
unstarted; a failed validation after it leaves an untagged `main` candidate
that must be corrected by a new commit and another validation.
3. Document one `releasekit publish vMAJOR.MINOR.PATCH` invocation using the
built, module-selected executable. Describe its selected race suite, full
vet, Linux package builds, release-note and effective-push-URL guards,
exact tag publication, and confirmed-tag completion. Keep `check` optional
for diagnosis, not a repeated required pass. Update the partial/unknown
push recovery directions to ReleaseKit's reported candidate, endpoint,
tag-object ID, and bounded authoritative-state reconciliation. Preserve
optional post-publication consumer verification and no-CI-wait behavior.
Remove old script commands, Go 1.25 pin examples, and promises of verbose
skipped-test output.
Roadmap candidate 1. Owner: `internal/engine/metadata.go`.
Completion evidence: a maintainer can follow the runbook from a clean checkout
through confirmed tag publication without a ConfigKit release wrapper or a
second required Go/test run; all commands, version placeholders, and documented
failure states match ReleaseKit v0.2.0. Do not execute its real publication
command in this stage.
1. Make `Metadata.IsExplicit` validate effective path/shape and read the stored
boolean, without calling `Lookup`, cloning origins, or projecting inheritance.
A shared private effective-field lookup may serve both methods; `Lookup`
continues returning complete detached records.
2. Format each query pointer once per lookup. Preserve root, zero metadata,
invalid/unknown/removed paths, parsed numeric tokens, and explicit key/index
distinctions. Leave `Resolution.IsExplicit` entry-state semantics unchanged.
3. Extend metadata/presence tests only where agreement with `Lookup` lacks
coverage: direct/inherited input, defaults, and explicit false/zero/null.
## Stage 5: Retire duplicated machinery and update developer guidance — Complete
Validate: `go test ./internal/engine -run 'Metadata|Presence|Explicit|Inherited'`,
relevant root facade tests, and the everyday check. No allocation assertion.
1. Remove `scripts/release.sh`, `scripts/release-check.sh`, and
`release_publish_test.go` after Stages 1–4 supply the replacement. Remove
`TestCheckScriptsReleaseGates` and its now-unused helpers from
`release_check_test.go`. Preserve
`TestDevelopmentCheckHandlesFilenamesAndStopsOnFailure` and
`scripts/check.sh`; rename the test file if helpful so its name reflects
its remaining owner. Retain the small Stage 2 profile-wiring test. Do not
transplant the old publication guard/recovery matrix into ConfigKit tests.
2. Update [the development guide](../development.md) to point to the new
release runbook, remove retired-script references and the fixed release
baseline, and adjust its task-to-test map. Keep the everyday development
check and release validation distinct. Review other live references to the
removed scripts and update only those made stale by this migration;
historical release notes remain unchanged.
## Stage 3: Detect inheritance without constructing paths
Completion evidence: no live release instruction invokes a removed script;
the default suite and `./scripts/check.sh` pass with the selected Go and
prepared dependencies; the retained tests protect development checks and
profile wiring without duplicating ReleaseKit's generic behavior.
Roadmap candidate 9. Owner: `internal/engine/inheritance_load.go`.
## Stage 6: Exercise the migration without publishing ConfigKit — Complete
1. Replace detection's full path-building walk with a nil-safe boolean visitor:
inspect mapping keys for `extends`, recurse into values/sequence children,
and return immediately on a match.
2. Leave validation, expansion order, provenance, and budgets unchanged.
Detection is not validation and must still find nested invalid syntax.
3. Reuse inheritance tests; add missing sequence-contained or invalid nested
cases without assertions about how many nodes were visited.
1. Build ReleaseKit from the candidate's `go.mod` selection with the latest
selected local Go patch for that minimum's series. Use that same executable
for ReleaseKit's validation, without requiring its patch number to equal
the `go.mod` minimum. In disposable Git fixtures, use
ConfigKit's selected profile with only `expected_push_url` changed to a
temporary bare remote, and provide a matching committed fixture release
note. Never push a ConfigKit tag or source commit to its real remote.
2. Push a committed fixture candidate to temporary `main` and run the built
`publish` command with a deliberately failing fixture test. Confirm
validation fails and creates no release tag, while `main` remains at that
candidate. Correct and commit the fixture, push the new candidate to
temporary `main`, and run `publish` again. Confirm one race-enabled standard
suite, one full vet pass, and one CGO-disabled package compile for each
selected Linux target in that successful invocation; confirm the remote
annotated tag identifies the corrected candidate and contains the checked-in
note. Record child Go invocations with a fixture-only forwarding executable
on `PATH`, or use equivalent direct evidence, rather than inferring the
selected command count merely from success.
Validate: `go test ./internal/engine -run 'Inheritance|Inherited'` and the
everyday check. Inputs without inheritance still need a linear scan.
Completion evidence: the isolated failure and success paths demonstrate the
selected profile, module-selected tool, command order, gates, and tag-only
publication. No ConfigKit ref is pushed to its real remote.
## Stage 4: Compare path tokens without whole-pointer formatting
## Stage 7: Check endpoint failure and finish the cutover review — Complete
Roadmap candidate 2. Owners: engine `path.go`, `resolution.go`, `metadata.go`.
1. In a fresh disposable fixture/version without an existing tag, check that
a wrong effective push URL is rejected before validation or tag
publication. Do not reuse the version already published in Stage 6, which
would make the existing-tag guard obscure the endpoint result.
2. Compare the runbook's recovery directions with ReleaseKit's reported state
for a failed or uncertain tag push. Repair any mismatch in the runbook;
do not duplicate ReleaseKit's recovery machinery in ConfigKit. This is a
bounded migration exercise, not a new per-release gate or a requirement to
wait for CI.
3. Re-run the affected repository checks after any corrections. Review the
final policy, runbook, development guide, and roadmap together for a
coherent process; confirm that only intended files changed and that no
real ConfigKit release or push occurred.
1. Add private token-spelling equality. Key/parsed segments compare literal
bytes; two indexes compare integers. For index versus key/parsed, render the
index into stack scratch space with `strconv.AppendInt` and compare bytes.
Do not parse literal tokens as integers: `01`, `+1`, and `1` differ.
2. Implement `pathContains` with existing validity/depth checks and token-prefix
equality. Root behavior remains unchanged. Escaping is injective, so raw
token spelling comparison preserves canonical pointer equivalence.
3. Replace textual comparison in `matchesPathShape` with token equality, then
retain its existing explicit/contextual shape checks and candidate-side
handling. Apply the same spelling predicate to local textual path deduplication.
4. Differentially compare with the String-based baseline over roots, invalid
paths, empty/slash/tilde/numeric keys, parsed tokens, nested paths, and integer
extrema. Reuse existing effective/historical query-shape coverage.
Completion evidence: the wrong-endpoint fixture fails for the intended reason,
recovery guidance matches ReleaseKit v0.2.0, tests and documentation checks
pass, and the repository is ready for a separately authorized release.
Validate: `go test ./internal/engine -run 'Path|Metadata|Resolution|Override|Inheritance'`
and the everyday check. Public `Path.String` stays unchanged.
## Stage 5: Format sorting keys once per item
Roadmap candidate 2. Owners: engine resolution, overlay, override application,
document comparison, and diagnostics.
1. Decorate path-bearing items with precomputed canonical pointers, sort with
existing comparisons/tie-breakers, then project back. Cover
`orderedResolutionRecords`, `displacedOrigins`, `affectedOverrideOrigins`,
and the final delta sort in `compareConfigurationValues`.
2. In `canonicalDiagnostics`, precompute paths without changing comparator
precedence, primary/related location ordering, or standalone comparator use.
Do not add permanent pointer fields to every record for sorting.
3. Gather recursive displaced definitions into one owned output and sort at
the outer owner, avoiding repeated sorting of the same nested subtree.
Preserve final lexical order and duplicate behavior.
4. Reuse ordering tests; add missing tie/escaped-key cases. Do not promise stable
order for items whose full comparison is equal.
Validate: `go test ./internal/engine -run 'Resolution|Override|Overlay|Diagnostic|CompareDocument|Provenance'`
and the everyday check. Expected benefit is reduced formatting, not measured CPU.
## Stage 6: Add a private canonical-pointer prefix index
Scope: one helper plus tests, no caller migration yet.
1. Add `internal/engine/path_prefix.go`. Nodes have child maps keyed by escaped
tokens, terminal integer payload IDs, and the minimum inserted ID within the
subtree. IDs are nonnegative owner ordinals; represent absence explicitly.
2. Support insertion, ancestor payload visits (root/exact included), descendant
payload visits (exact included), and earliest intersecting ID using ancestor
IDs and the queried subtree minimum. Preserve multiple IDs at one terminal.
No deletion, wildcard search, provider abstraction, or serialization API.
3. Tokenize canonical pointers by separator scanning without split slices.
Preserve root versus empty keys and `/a` versus `/ab` boundaries. Owners
supply canonical pointers from valid paths; no new input validation contract.
Retain IDs only, not nodes, credential bytes, or mutable path segments.
4. Differentially check results against simple prefix scans for ancestors,
descendants, duplicates, insertion minima, escaping, and root/empty tokens.
Validate: focused helper tests and the everyday check. Storage follows indexed
token material/payloads; tests do not fix representation or allocation counts.
## Stage 7: Index resolution coverage and record reads
Roadmap candidate 3, first integration. Owner: `internal/engine/resolution.go`.
1. Index already ordered records after the callback. Select deepest covering
ancestor, earliest owner ID for equal depth, matching the current scan.
Use this for delta coverage in `runResolution`; retain all validity/helper
checks and diagnostic order.
2. Add a small snapshot coverage adapter with the same rules for Stage 8.
IDs refer to owner-held slices and cannot outlive or survive their reordering.
3. Build operation-local read-only mapping lookups for completed before/after
trees, reusing `mappingIndex` mechanics without mutating trees. Add a private
indexed document lookup variant with the same validity, shape, bounds, and
absence checks; retain the plain helper for callers without reuse.
4. Cover parent/exact/more-specific records, equal-depth private fixtures,
added/removed values, unrecorded changes, and helper-target mismatch.
Validate: `go test ./internal/engine -run 'Resolution|Resolve|Callback|Path'`
and the everyday check. Snapshot ownership/contents remain unchanged.
## Stage 8: Replace per-delta whole-metadata scans
Roadmap candidate 3, second integration. Owner: engine `load_metadata.go`.
1. In `applyResolutionMetadata`, build one final pointer/node index and one
snapshot coverage index. Reuse final membership for descendant history and
field refresh; do not rebuild indexes per delta/field.
2. Index delta paths by original delta ID. Enumerate the union of historical
and final pointers once, obtain covering delta IDs, and bucket affected pointers
by delta. Parse paths only when a retained semantic path is actually needed.
Sort buckets lexically and retain original delta processing order.
3. Apply exact snapshot events first, descendant history second, effective origin
updates third, then final field refresh, as today. Exclude the covering record's
own pointer from descendant history and retain the original Previous origins.
Handle overlapping delta IDs in reference fixtures without assuming every
possible input is disjoint. Normal structural comparison emits disjoint deltas.
4. For fields, visit pointers once and process matching delta IDs in original
order. For new fields, use snapshot coverage; preserve null/Explicit state,
default resolution-origin fallback, and removed-path history.
5. Extend existing tests for many disjoint changes, replacement, removed/new
descendants, and numeric container-shape changes. A bounded test-only
reference metadata update can protect phase/order equivalence.
Validate: `go test ./internal/engine -run 'Resolution|Metadata|Provenance|History|Explain'`,
engine race tests, and the everyday check. Complexity is output-sensitive to
affected events, not every delta times every field.
## Stage 9: Collect sensitivity directly from trees
Roadmap candidate 5. Owner: engine `secret.go`. Keep fixed-point propagation
unchanged until Stage 11 so collection is independently reviewable.
1. Replace temporary pointer/node indexes used only to enumerate classification
paths/flags with direct visitors into local path and sensitivity accumulators.
Remove the redundant initial final-tree scan; final still participates in
the late projection phase before schema/predicate classification.
2. Preserve phase precedence: applications; history/dependencies; inheritance;
projections then final. Tree/pointer contributions currently install parsed
segments, while history/contributor contributions can install typed paths.
Preserve those representations and subsequent overwrites.
3. Skip reparsing only if the current stored path already has that parsed
representation. Otherwise perform the existing parsed replacement. Do not
simply substitute explicit walker key/index paths or retain the first shape.
4. Retain all historical/removed paths, controls, dependency-only paths, default/
dynamic tagged projections, and cloned predicate arguments.
5. Add only missing numeric shape-transition, historical predicate, dynamic tag,
and predicate-detachment cases in existing sensitivity/inheritance test owners.
Validate: `go test ./internal/engine -run 'Secret|Sensitivity|Inherited|Effective|Explain'`,
engine race tests, and the everyday check. No new predicate order/count guarantee.
## Stage 10: Build and test indexed sensitivity triggers
Roadmap candidate 6, isolated algorithm. Add private
`internal/engine/secret_propagation.go` and focused tests.
1. Ordinary edges have dependency/target pointers and activate on classified
ancestor or descendant intersection. Inheritance links become two directed
edges, each activated only by a classified ancestor/equal dependency path.
2. Index dependency pointers with payload IDs. A new classified pointer visits
ordinary triggers at ancestors and descendants; inheritance triggers only
at descendants. Avoid double-visiting exact terminals. Track fired IDs and
enqueue a target only when it becomes newly sensitive.
3. Seed from the initial sensitive set. Preserve the ordinary dependency filter:
the old `secretIntersectsPath` sees only classified pointers present in the
collected paths map. Inheritance's `pathIsSecret` does not require membership.
Relations use textual spelling, not document query-shape validation.
4. Differentially compare final sets to the old fixed-point algorithm for
forward/reverse chains, cycles, duplicate edges, classified ancestors/
descendants, classified pointers absent from paths, root, escapes/numeric
spellings, and mixed ordinary/inheritance graphs. Use no real credentials.
Validate: focused propagation tests and the everyday check. Match/output-sized
work may still be large for broad dependencies; no blanket linear guarantee.
## Stage 11: Integrate worklist sensitivity into publication
Roadmap candidate 6, integration. Owner: engine `secret.go`.
1. Feed collected paths, seeds, dependencies, and inheritance links into Stage
10's closure. Remove the production fixed-point loop/unused scan helper after
migration; retain the useful reference only in tests.
2. Verify full-load effective rendering and explanations for mixed derived/
inherited sensitivity, replaced/removed paths, tagged containers, consumed
controls, and nested copies. All credentials remain outside trees/indexes.
3. Verify independent loads and repeated queries retain no mutable propagation
state. Queues and trigger indexes are phase-local and discarded on return.
Validate: `go test ./internal/engine -run 'Secret|Sensitivity|Inheritance|History|Explain|Independent'`,
relevant root facade secret/inheritance tests, engine race tests, and the everyday
check. Record collection and propagation outcomes separately.
## Stage 12: Index CLI overlaps and validation reads
Roadmap candidate 4. Owner: engine `override.go`.
1. Retain input order and the duplicate seen map. Replace the prior-accepted
overlap scan with earliest intersecting trie lookup using original override
indices. Duplicate diagnostics continue preceding overlap checks.
2. Insert exactly where `pathAccepted[index] = true` occurs today: after path,
overlap, version, and directive checks, before value conversion/budgets.
A later bad value still influences subsequent overlap diagnostics. Do not
index paths rejected earlier or omit those accepted before value failure.
3. Reuse an operation-local mapping index for read-only base traversal during
path validation. Preserve missing mapping ancestors, preparation-before-apply,
CLI source order/identity, rules, and limits.
4. Compare with baseline for disjoint/duplicate/ancestor/descendant paths,
multiple conflicts, escapes, rejected paths, and later value failures.
Protect diagnostic codes, paths, and first related-source selection.
Validate: `go test ./internal/engine -run 'Override|CLI|Provenance|Diagnostic'`
and the everyday check. No policy changes.
## Stage 13: Coalesce resolution replay once with indexed ancestors
Roadmap candidate 7. Owners: engine `history.go`, `load.go`, `load_result.go`.
1. Replace group scans with ancestor lookup selecting the earliest inserted
matching group, not the deepest. Insert only new groups; merged child paths
are not independently indexed as groups.
2. Retain original before/after trees and normalization precedence. Use ordered
membership sets for reason/dependency unions, retaining first occurrence and
canonical pointer equivalence.
3. Prepare immutable coalesced descriptors once before publication. Explain
reads them without mutating Changes or borrowed snapshots. Retain original
snapshots for their other metadata/history-budget uses; node accounting is
unchanged. No renderer or public records are cached.
4. Differentially check independent/parent/child records, multiple ancestors,
root, equal writes, union order, normalization, and repeated detached results.
Validate: `go test ./internal/engine -run 'Resolution|History|Explain|Ownership'`,
engine race tests, and the everyday check. Full replay semantics stay intact.
## Stage 14: Index relevant inheritance history operations
Roadmap candidate 7. Owners: engine metadata, metadata cloning, inheritance history.
1. During final metadata preparation, index inheritance operation IDs by control
path or transfer destination. Collect intersecting IDs for History, deduplicate,
restore original operation order, and run existing projection logic.
2. Preserve stored-history splitting before resolution and Previous-origin
construction. Keep fields/history/path clones independent. A fully frozen
ID-only index may be shared in `cloneMetadata`, like immutable evidence;
no query builds or updates it. Zero/private unprepared metadata must still
produce correct results (use the existing scan if no prepared index exists).
3. Leave global `appendInheritanceExplanation` replay intact: non-intersecting
operations can affect working state and must not be skipped with this filter.
4. Compare indexed history to the scan for multi-scope controls, removed paths,
ancestor queries, and query mutations/captured metadata independence.
Validate: `go test ./internal/engine -run 'Inheritance|Inherited|History|Metadata|Independent'`,
engine race tests, and the everyday check.
## Stage 15: Reuse frozen inspection mapping lookups
Roadmap candidate 7. Owners: publication, inspection, document/snapshot/history helpers.
1. Reuse Stage 7's mapping lookup builder for immutable decoded-input, final,
and application trees queried repeatedly by inspection. Populate all retained
maps before return, visiting shared node identities once. Retain lookup positions
and borrowed immutable nodes, not extra trees or Path copies.
2. Thread read-only lookups into applicable document/snapshot and application
origin/relevance reads. Preserve validity, shape, bounds, and absence. Read
populated maps directly, not lazy entries methods that write on first query.
3. Keep plain document lookup on mutable replay trees in this stage. Their
existing query-local merge indexes are unchanged; do not reuse a frozen map
after cloning, deletion, or entry-position changes. This avoids adding an
invalidation protocol just to optimize the selected immutable reads.
4. Reuse shape/history/render tests; add only missing repeated/concurrent
read-only inspection ownership cases. Do not add a public concurrency promise.
Record extra retained-index storage honestly; this is a lookup tradeoff,
not a free allocation reduction. Baseline/full-tree replay remains unchanged.
Validate: `go test ./internal/engine -run 'Path|History|Explain|Effective|Independent|Parallel'`,
engine race tests, and the everyday check.
## Stage 16: Index materialized inheritance-step membership
Roadmap candidate 8. Owner: engine `inheritance_origin.go`.
1. Keep output order and compact linked traces. Use a comparable key with three
canonical From/To/Reference strings and one encoded ordered Locations string.
Encode the location count and each SourceID byte length, then its bytes and
delimited decimal line/column integers. This is collision-free, including
empty IDs, delimiter characters, and signed coordinates. Arbitrary unescaped
delimiter concatenation is incorrect; no hashing dependency is needed.
2. Seed membership from existing Origin.Inheritance but retain its existing
duplicates/order. Only newly proposed steps are deduped, as today.
3. Format candidate pointers once; clone retained paths/locations only for newly
accepted steps. Keep contributor traversal and transfer direction intact.
4. Compare chains, shared contributors, different location/order steps, existing
duplicates, escaped/numeric paths, and returned-record mutation to baseline.
Validate: `go test ./internal/engine -run 'Inheritance|Inherited|Metadata|Provenance|History'`
and the everyday check. Complex contributor-trace equality stays unchanged.
## Stage 17: Index simple resolution and origin membership
Roadmap candidate 8. Owners: engine `resolution.go` and `load_metadata.go`.
1. Maintain lazily initialized first-occurrence reason/dependency sets on
resolution records. Dependencies use canonical pointer equality; retained
slices and callback validation/order/nil behavior remain unchanged.
2. Use exact Location membership in `originContainedBy`, retaining Layer/
Operation and contributor-empty checks. Use SourceID membership in
`originRepresentedByChange`, which has a different source-coverage predicate.
Do not conflate these equality rules.
3. Leave complex contributor scans and `sameInheritanceTrace` intact. Their
concrete segment/location equality cannot be replaced by pointer identity
or canonical spelling. Sets are phase-local, not exported record fields.
4. Reuse existing provenance tests; cover duplicate unions, equal source IDs
with differing locations, and location coverage with contributor mismatch
only where protection is missing.
Validate: `go test ./internal/engine -run 'Resolution|Provenance|Overlay|Override|Inheritance|History'`
and the everyday check. Small lists and outputs preserve existing semantics.
## Stage 18: Review the completed result and record outcomes
1. Review the diff from baseline. Map all nine candidates to selected implemented
changes and expressly deferred portions. Remove dead helpers/imports/state;
keep useful small equivalence tests, no alternate production engines.
2. Run `./scripts/check.sh` and `go test -race ./...`. Verify public/import
boundaries, examples, independent loads, shapes, presence, history, redaction,
and complete-on-return behavior through their existing tests.
3. Record algorithmic before/after work, lifetime/storage tradeoffs, revisions,
actual commands/results, and unmeasured claims in an evidence subsection here.
Optional probes may reuse existing startup/assembly/underlay/inheritance
benchmarks and small record/override/dependency/query fixtures. Measurements
are not prerequisites; timing claims await a representative host.
4. Update stage and roadmap status accurately. Update focused internal docs
only where they describe changed implemented mechanics. Keep public contracts
and policy language unchanged. Do not retire other historical records or
roadmaps as incidental cleanup.
5. Review links/whitespace/staging, commit the completed unit, and report outcomes,
checks, remaining deferred experiments, and workspace status. No release or
downstream bump is included.
Done: selected changes preserve behavior, remain proportionate/maintainable,
pass validation, and distinguish algorithmic expectations from measured results.
No public performance guarantee is created.
## Open Questions
None required for the selected scope. Algorithm, ownership, relation, ordering,
and scope choices above are resolved. Expressly deferred experiments require
separate design/evidence and do not block this sequence.

74
docs/roadmap/performance-audit.md
View File

@@ -28,6 +28,26 @@ themselves grow substantially. An output-sensitive algorithm cannot avoid
producing the output required by the existing contract. Expected map lookup
costs and path processing costs also qualify any proposed linear bound.
## Target state and selected scope
The implementation target is the same public configuration model with less
avoidable auxiliary work: presence-only queries do not materialize provenance;
path comparisons and sorts avoid repeated formatting; resolution coverage,
override overlaps, and sensitivity propagation use local path indexes;
classification collects paths directly; and frozen inspection reuses prepared
resolution/inheritance lookup data. Inheritance detection does not construct
semantic paths merely to detect a keyword. Ordered provenance collections use
membership indexes where their equality rules are straightforward.
The [implementation plan](implementation.md) selects bounded revisions from
all nine candidates below. Whole-tree explanation replay remains in place;
branch-local replay, changing the public metadata-copy handoff, global schema
caching, and indexing complex contributor trace equality remain later
experiments. These have larger ownership, memory, or equivalence risks than the
selected changes. No new consumer-facing complexity bound or performance
threshold is part of the target. The candidate descriptions retain the broader
audit context; the plan owns the selected implementation details and sequence.
## Candidates
### 1. Avoid materializing provenance for presence-only queries
@@ -67,9 +87,9 @@ Many nested scans below repeatedly invoke it. Sorting comparators also repeatedl
format paths, making formatting/allocation work proportional to comparisons
rather than items. `Lookup` formats the same query for multiple lookups.
First reuse an already formatted pointer within a function or decorate sortable
items with a pointer once. Consider an allocation-free containment comparison
over token spellings as a separate experiment. Complexity remains proportional
Reusing an already formatted pointer within a function and decorating sortable
items with a pointer once avoid repeated formatting. Allocation-free containment
comparison over token spellings is also in the selected scope. Cost remains proportional
to path length, but repeated string allocations can disappear.
Containment currently compares canonical pointer spellings, while metadata
@@ -203,11 +223,11 @@ linearly scans each mapping on its path; querying every key in a wide mapping
can therefore add quadratic lookup work. These costs compound across queries,
even if each query asks about a small unrelated leaf.
First consider preparing immutable coalesced replay descriptors once when an
inspection is constructed, and indexing relevant inheritance operations and
frozen-tree lookups. Coalescing can use an ancestor index rather than a group
scan. A later, separate experiment could replay only the queried branch and its
necessary dependencies instead of cloning whole trees.
The selected target prepares immutable coalesced replay descriptors once when an
inspection is constructed and indexes relevant inheritance history operations
and frozen-tree lookups. Coalescing uses ancestor lookup rather than a group
scan. Replaying only the queried branch and its necessary dependencies remains
a deferred experiment; whole-tree replay remains in the selected target.
Preserve the first covering coalesced group, ordered reason/dependency unions,
normalization precedence, chronology, intermediate absence/replacement, typed
@@ -274,30 +294,18 @@ to change scope rules, traversal ordering during expansion, or resource budgets.
| Public facade and detached records | Root-owned wrappers and explicit conversions preserve the public/private boundary. Some records are copied in the engine and again during conversion; eliminating a copy needs a clear internal ownership handoff, not public aliases or borrowed mutable records. `Result.Metadata()` also clones the complete metadata before queries; consumers repeatedly obtaining it pay repeated full-copy costs. Treat redesign of that private handoff as a separate, lower-priority retained-memory/ownership experiment. |
| Effective YAML, diagnostics, examples, and release/development tooling | Deterministic rendering and diagnostics require ordering and output-sized work; path comparator allocation is covered above. Safe snapshots and YAML adaptation allocate separate trees. Maintained consumer examples and check tooling are not production library hot paths; no gate or tooling changes proposed. ReleaseKit internals are outside this repository audit. |
## Recommended follow-up
## Acceptance and boundaries
Start with candidate 1 or 9 as a small, reviewable optimization, then candidate
2's local formatting reuse and candidate 5's direct classification traversal.
These avoid work without demanding new architectural machinery. Next address
candidate 3 for many recorded changes and candidate 6 for large dependency
graphs. Candidates 4, 7, and 8 are useful follow-ups for consumers with many
overrides, inspection queries, or inherited contributions.
The selected revisions preserve the existing [configuration contract](../config.md)
and [architecture policy](../policy/architecture.md): complete load results,
effective versus historical path shapes, provenance, ownership, presence,
secret isolation, and synchronous completion remain unchanged. New caps,
input restrictions, composition rules, or performance promises are not part of
this work. The plan owns implementation sequencing, validation, and evidence.
This is a prioritized backlog, not a staged implementation plan or a requirement
to implement every candidate. Before each production change, confirm the proposed
algorithm against the existing [configuration contract](../config.md), relevant
tests, and [architecture policy](../policy/architecture.md). Preserve complete
load results, effective versus historical path shapes, provenance, ownership,
presence, secret isolation, and synchronous completion. Do not add caps, reject
previously valid inputs, change composition rules, or create performance promises
to simplify an optimization.
For a selected change, add only missing behavioral regressions, run the relevant
package tests and everyday [development check](../development.md), and use race
checks where ownership/shared state changes warrant them. Follow-up measurements
can reuse existing startup, composition, underlay, inheritance, and history
benchmarks, with targeted fixtures for record count, disjoint overrides,
dependency chains, and repeated queries where coverage is missing. Prefer
allocations and bytes on the busy host; defer CPU improvement claims until
timing is representative. Benchmarks help decide whether the implementation's
complexity is worthwhile, but are not new release gates or test thresholds.
Success is removal of the identified redundant work and broad pairwise scans
without disproportionate machinery or semantic regressions. Output-sized work,
deterministic ordering, and required ownership copies remain legitimate costs.
Benchmarks may inform the tradeoff, especially allocations on the busy host,
but are not new release gates or test thresholds. Unmeasured improvements
remain algorithmic expectations rather than quantified runtime claims.