1026 lines
61 KiB
Markdown
1026 lines
61 KiB
Markdown
# Audit Remediation Implementation Plan
|
||
|
||
Status: in progress; completion is recorded per stage below.
|
||
Prepared: 2026-10-25.
|
||
Planning baseline: `4a5316b7fc2da57dcced33190cb6499c9a1beb5c`.
|
||
Audience: a Luna coding agent implementing the stages sequentially.
|
||
|
||
## Scope And Execution
|
||
|
||
Implement the 29 selected entries in the [roadmap](roadmap.md). The
|
||
[audit](audit.md#settled-scope-decisions) owns the evidence, accepted scope, and
|
||
detailed acceptance cases. This plan owns implementation order, private design
|
||
choices, validation, and execution progress. Preserve the finding IDs when
|
||
recording completion. The production baseline is unchanged by the intervening
|
||
audit and planning commits; nevertheless, inspect actual callers before deleting
|
||
code. Follow `AGENTS.md`, [agent policy](../policy/agents.md), and the
|
||
[development guide](../development.md).
|
||
|
||
This is one end-to-end sprint. Intermediate stages may temporarily fail to
|
||
compile or have pending tests. Do not create compatibility shims, duplicate
|
||
implementations, or extra public interfaces just to make each stage independently
|
||
green. Run the focused checks when their affected code is buildable; otherwise
|
||
record exactly what is pending and run it once the consuming stage lands. Never
|
||
describe an unfinished stage or deferred check as complete. Commit focused,
|
||
nonempty units under repository policy, with plain-English messages and accurate
|
||
validation notes. Final completion requires the integrated end state and checks,
|
||
not just completion markers.
|
||
|
||
For each stage, record its status, affected finding IDs, and actual validation
|
||
here. Include already known commit identifiers in later stage updates; report the
|
||
final commit in the handoff rather than making a commit self-referential. Keep
|
||
execution status in this plan rather than maintaining
|
||
another progress inventory in `audit.md` or `roadmap.md`. A finding split across
|
||
stages closes only after all of its stages and acceptance cases are satisfied.
|
||
|
||
### Boundaries
|
||
|
||
- Preserve supported public types, signatures, root ownership, configuration
|
||
syntax, tier semantics, collection operations, source/version conventions,
|
||
callback ordering, diagnostics, confidentiality, and detached ownership.
|
||
- Repair the nine reported defects and their regressions. Safe rejection of
|
||
invalid values, acceptance of supported finite values, and enforcement of the
|
||
existing quotas are remediation, not new restrictions.
|
||
- Keep panic propagation, borrowed-resource ownership, regular-file close policy,
|
||
partial bootstrap/writer behavior, and existing optional-absence rules.
|
||
- Keep raw recorded resolution snapshots, active history, and required copies.
|
||
Remove only the proven unused copies and evidence named below.
|
||
- C03 glob readability extraction and F07 wider facade transport simplification
|
||
remain deferred in [future work](future.md). Do not redesign the clone visitor,
|
||
explanation replay, public metadata handoff, or cross-load schema caching.
|
||
- Add no dependencies, public options, global caches, traversal caps, background
|
||
activity, release gates, or performance guarantees. Do not publish a release,
|
||
update consumer dependencies, or edit accepted ADR decisions during this sprint.
|
||
|
||
### Validation Conventions
|
||
|
||
Commands below run from the repository root with the installed compatible Go
|
||
toolchain and already available dependencies. Use `GOTOOLCHAIN=local`,
|
||
`GOPROXY=off`, and `GOTELEMETRY=off` for offline checks. Keep disposable diagnostics,
|
||
coverage files, and binaries outside the repository. Do not change `go.mod` or
|
||
`go.sum` to obtain audit tooling.
|
||
|
||
Focused `-run` expressions identify behavior owners, not required names for new
|
||
tests. Include the new regression explicitly in the focused invocation if its
|
||
name falls outside the expression. Extend existing test owners; do not duplicate
|
||
the internal matrix through every facade. Tests for termination use a bounded
|
||
child process/watchdog where a regression could crash or hang the test process;
|
||
that watchdog is not a performance threshold. Reuse the subprocess pattern in
|
||
`internal/engine/source_fifo_test.go` where appropriate.
|
||
|
||
Use small controlled budgets and bounded filesystem operation observation for
|
||
resource/scaling cases. No benchmark campaign, exact syscall choreography,
|
||
coverage percentage, or timing ratio is an acceptance criterion. Final development,
|
||
race, consumer, and documentation validation is specified in Stage 33.
|
||
|
||
## Stage 1 — Terminate Pointer-Only Descriptor Traversal
|
||
|
||
Finding: **A01**; also the `schemaPath` portion of **F06**.
|
||
Owners: `internal/engine/schema.go`, `decode.go`, `secret.go`, and their tests.
|
||
|
||
Status: **complete** (A01; schema-path portion of F06). Validation passed:
|
||
`GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Schema|Decode|Resolution|Secret|PointerOnly' -count=1`, the specified focused check, and `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`.
|
||
|
||
1. Introduce one private pointer-descriptor terminal lookup that detects a
|
||
repeated descriptor, using a traversal-local visited set. Return failure for
|
||
a pointer-only loop; do not reject that type during schema construction.
|
||
2. Use it where decoding must unwrap a non-null pointer chain, in
|
||
`acceptsMapping`, and before schema/secret path traversal consumes the next
|
||
segment. A non-null value for an uninhabitable pointer-only chain produces
|
||
the existing `CodeDecodeType` classification at the input path/location.
|
||
Keep null handling before this check so supported nil pointers still decode.
|
||
For decoding a finite chain, peel the base/descriptors iteratively, decode
|
||
the terminal value once, and rebuild pointer wrappers in reverse. Preserve
|
||
existing typed-default behavior; do not rescan the remaining chain at every
|
||
recursive pointer call and introduce quadratic chain traversal.
|
||
3. Remove unread `schemaPath.field` and `schemaPath.dynamic` and the terminal
|
||
unwrapping used only to compute them. A path ending at a nullable pointer
|
||
field may still resolve to its descriptor; reject attempts to descend through
|
||
its pointer-only loop. Update consumers/tests accordingly.
|
||
4. Review other pointer-unwrapping helpers. Apply the shared check to analogous
|
||
descriptor-only loops; retain finite value traversal and existing inline-field
|
||
restrictions rather than adding a blanket recursive-type rejection.
|
||
|
||
Acceptance: reproduce `type Loop *Loop` non-null decoding and helper-path
|
||
termination in bounded subprocesses; verify null/omitted values, direct nullable
|
||
field operations, and normal recursive structs with acyclic values. Check safe
|
||
codes/paths rather than raw values or exact new wording.
|
||
Focused check: `go test ./internal/engine -run 'Schema|Decode|Resolution|Secret'`.
|
||
|
||
## Stage 2 — Roll Back Failed Dynamic Descriptor Builds
|
||
|
||
Finding: **A02**.
|
||
Owners: `internal/engine/schema.go`, `value_operation.go`, projection/resolution tests.
|
||
|
||
Status: **complete** (A02). Previous completed unit: `2736ddf`. Validation
|
||
passed: `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Dynamic|Projection|Clone|Resolution' -count=1` and `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`.
|
||
|
||
1. Give `schemaBuilder` a journal of types newly inserted during a build.
|
||
Record insertion only after a cache miss; cached, completed descriptors are
|
||
borrowed and not mutated. Recursive new descriptors may refer to one another
|
||
through their provisional entries.
|
||
2. In `dynamicDescriptor`, treat construction plus dynamic-secret eligibility
|
||
as one operation. On failure, delete every journaled cache entry, including
|
||
ancestor/container placeholders, before returning the existing path-specific
|
||
projection issue. Deleting an entry already removed by a local failure is safe.
|
||
3. On success, retain the completed additions in the existing load-local cache.
|
||
Do not clone the entire cache on each request, cache failures, or clear valid
|
||
preexisting entries. Initial schema construction can use the same journaling
|
||
mechanics without introducing another persistent cache.
|
||
|
||
Acceptance: the two `Normalize` calls from A02—typed nil then non-nil pointer to
|
||
an unsupported dynamic struct—report safe failures without a panic. Cover nested
|
||
and mutually recursive provisional entries, subsequent valid operations, prior
|
||
valid cache entries, and repeated failures at different paths in one load.
|
||
Focused check: `go test ./internal/engine -run 'Dynamic|Projection|Clone|Resolution'`.
|
||
|
||
## Stage 3 — Distinguish Finite Slice Views From Cycles
|
||
|
||
Finding: **A03**.
|
||
Owners: `internal/engine/value_operation.go`, `value.go`, `clone.go`, and tests.
|
||
|
||
Status: **complete** (A03). Prior stage commits: `2736ddf`, `deb9aa5`.
|
||
Validation passed: `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Mapping|Value|Projection|Clone|Override|Cycle|FiniteSlice|SliceCycles' -count=1` and `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`.
|
||
|
||
Add visible slice length to `valueVisit` identity in `visitFor`. Keep type, kind,
|
||
and starting pointer, existing empty/nil handling, and active-path removal on
|
||
unwinding. Capacity is not part of the visible traversed graph. Maps and pointers
|
||
retain their current identity rules. Follow all shared callers rather than
|
||
patching only one projection entry point.
|
||
|
||
Acceptance: `shared := []any{"leaf", nil}; shared[1] = shared[:1]` succeeds through
|
||
mapping input, typed defaults/projection, cloning, and CLI override construction.
|
||
Equivalent detached values agree. Actual self/mutual cycles still fail safely;
|
||
mutation of an existing returned descendant must not affect the original.
|
||
Focused check: `go test ./internal/engine -run 'Mapping|Value|Projection|Clone|Override|Cycle'`.
|
||
|
||
## Stage 4 — Check Container Lower Bounds Before Bulk Allocation
|
||
|
||
Finding: **A04**.
|
||
Owners: `internal/engine/projection.go`, `value_operation.go`, `projection_test.go`.
|
||
|
||
Status: **complete** (A04). Prior stage commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`. Validation passed: `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Projection|Inline|Resolution' -count=1` and `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`.
|
||
|
||
1. After charging a container node and before its bulk reservation, compare its
|
||
minimum remaining cost with the current node allowance: at least one node per
|
||
sequence element and one key plus one value node per map entry. Use comparisons
|
||
or division against remaining allowance, not overflowing size multiplication.
|
||
2. Reject an impossible container with `CodeSchemaProjectionLimit` before
|
||
`make`/map-key materialization. Preserve negative/unlimited internal allowances,
|
||
per-child charging, depth checks, key sorting, and inline-container accounting.
|
||
3. Do not generalize this into a new container-length or heap-size limit. Review
|
||
related reservations for the same known-lower-bound error without redesigning
|
||
typed cloning or making unrelated allocation promises.
|
||
|
||
Acceptance: bounded tests cover exact/insufficient remaining allowances for maps,
|
||
slices, arrays, and flattened inline containers. Use a child process for the
|
||
maximum-length zero-sized slice case; prove safe rejection without a huge backing
|
||
allocation. Check defaults and resolution paths using the shared projector.
|
||
Focused check: `go test ./internal/engine -run 'Projection|Inline|Resolution'`.
|
||
|
||
## Stage 5 — Close Callback Contexts On Every Exit
|
||
|
||
Finding: **B01**.
|
||
Owners: `internal/engine/resolution.go`, `validation.go`, callback tests.
|
||
|
||
Status: **complete** (B01). Prior stage commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`. Validation passed: `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Callback|Resolution|Validation|Version' -count=1` and `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`.
|
||
|
||
Wrap each application callback invocation in a small function with deferred
|
||
context deactivation. The defer must run when that callback returns or panics,
|
||
not at the end of subsequent projection/auditing work. Apply this independently
|
||
to resolution, final validation, and the shared version checker (`reportVersion`).
|
||
Retain the existing error/warning handling and propagate the original panic
|
||
unchanged; do not recover, synchronize, or broaden concurrent callback support.
|
||
|
||
Acceptance: external recovery retains each context and confirms its operations
|
||
now produce the established closed-context panic. Use a distinct panic sentinel
|
||
to verify original propagation. Retain normal-return closure and lifecycle tests.
|
||
Focused check: `go test ./internal/engine -run 'Callback|Resolution|Validation|Version'`.
|
||
|
||
## Stage 6 — Close The Owned Import Root During Panic Unwinding
|
||
|
||
Finding: **B02**.
|
||
Owners: `internal/engine/import_acquisition.go`, `filesystem_access.go`, import tests.
|
||
|
||
Status: **complete** (B02). Prior stage commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`. Validation passed: `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Import|SourceVersion|Filesystem' -count=1` and `GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`.
|
||
|
||
Install deferred fallback closure at the entry of `acquireImportedWithAccess`,
|
||
before operations that may invoke application code. Use a local closed flag or
|
||
one local close-once function so existing explicit close sites preserve their
|
||
normal timing, return diagnostics, and primary-error precedence. Mark closure
|
||
attempted even if closing returns an error; do not re-close an owned handle or
|
||
close the borrowed native filesystem. The deferred fallback must not replace a
|
||
propagating callback panic with a cleanup failure.
|
||
|
||
Acceptance: capture a real owned `os.Root` when invoking this acquisition boundary
|
||
with temporary host files, make an imported version checker panic, recover, and
|
||
verify that operations on that same root report closed state. Also verify child
|
||
file closure and normal/error exits. This observes the actual host root without
|
||
requiring a production FD counter or treating native file counts as equivalent.
|
||
Document genuine platform prerequisites; do not create a human validation gate.
|
||
Focused check: `go test ./internal/engine -run 'Import|SourceVersion|Filesystem'`.
|
||
|
||
## Stage 7 — Preserve Directory Failure Phase And Per-Handle Evidence
|
||
|
||
Finding: **C01**, including its integrated test improvements.
|
||
Owners: `internal/engine/filesystem_access.go`, `import_selection.go`, `glob.go`,
|
||
`native_filesystem_test.go`, and `resource_context_test.go`.
|
||
|
||
Status: **complete** (C01). Prior completed commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`, `877c497`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Directory|Glob|Filesystem|Resource' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
1. Return private directory failure information distinguishing unsuccessful
|
||
initial open from enumeration/close failures. Preserve error unwrapping for
|
||
classification; do not expose backing error text.
|
||
2. Apply optional/missing behavior only to `ErrNotExist` from the initial open.
|
||
Batched `ReadDir`, no-progress, visit-stop closure, and final close failures
|
||
after opening are failures. Keep read failure ahead of an accompanying close
|
||
failure, and preserve existing selector diagnostics and entry accounting.
|
||
3. The fallback closes the initial file before using `fs.ReadDir` on the native
|
||
filesystem. Since initial opening succeeded, any fallback error, including
|
||
`ErrNotExist`, is a post-open failure. Preserve standard `ReadDirFS` dispatch
|
||
instead of imposing a new native-filesystem interface requirement.
|
||
4. Update both directory and glob selectors and all callers. Enhance the shared
|
||
fake to track individual opened-handle closure, including the fallback, rather
|
||
than only aggregate counts. Do not change regular-file close-error policy.
|
||
|
||
Acceptance: directory plus glob, batched plus fallback paths, data/entries with
|
||
errors, read/close `ErrNotExist`, real missing-on-open controls, and a successful
|
||
second directory that must not hide failure. Assert nil result, structured
|
||
code/source, per-handle closure, and no canary in messages or rendered diagnostics.
|
||
Use a compact table/shared fixture, not duplicated matrices in every package.
|
||
Focused check: `go test ./internal/engine -run 'Directory|Glob|Filesystem|Resource'`.
|
||
|
||
## Stage 8 — Memoize Completed Native Link Dependencies
|
||
|
||
Finding: native part of **C02**.
|
||
Owners: `internal/engine/filesystem_links.go`, `filesystem_links_test.go`,
|
||
`directory_links_test.go`, and glob link tests.
|
||
|
||
Status: **complete** (native C02). Prior completed commits: `2736ddf`,
|
||
`deb9aa5`, `ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`.
|
||
Validation passed: `env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Link|Directories|Glob' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
1. Extract the existing portable frame evaluator into one private resolution
|
||
operation with `active` and `completed` maps. A small set of operation-local
|
||
functions supplies `Lstat`, link reading, and validated target normalization.
|
||
Keep this inside the engine; it is not a general filesystem/provider interface.
|
||
2. Wire native resolution first. Keep native prefix/root containment and target
|
||
admission rules in its binding. Key maps by the complete normalized link path;
|
||
the binding fixes the namespace for this one invocation.
|
||
3. An active back-edge remains a cycle. On a completed hit, restore the cached
|
||
canonical target and process the caller's remaining segments. Record completion
|
||
only when the link's own target frame finishes, not when first discovering it
|
||
or after resolving the entire outer pathname. Cache successful targets only.
|
||
|
||
Acceptance: finite repeated links, nested/non-dot roots, self/mutual cycles,
|
||
outside-root targets, and follow-disabled behavior. Extend real-link fixtures
|
||
with a counting native wrapper for the A/B-doubling graph from C02. Use a generous
|
||
linear operation ceiling on small graph sizes, with a bounded abort to keep the
|
||
old failure safe; do not assert an exact call trace or elapsed time. Exercise both
|
||
directory and glob selection through this binding.
|
||
Focused check: `go test ./internal/engine -run 'Link|Directories|Glob'`.
|
||
|
||
## Stage 9 — Use The Same Dependency Evaluation For Host Links
|
||
|
||
Finding: remaining host part of **C02**.
|
||
Owners: `internal/engine/filesystem_links.go` and the same link/discovery tests.
|
||
|
||
Status: **complete** (C02). Prior completed commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`, `7bf2f0c`.
|
||
Validation passed: `env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Link|Directory|Glob|Filesystem|Discovery' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
Bind rooted host `Lstat`/`Readlink` operations to Stage 8's evaluator and retire the
|
||
duplicate host frame loop. Retain portable component evaluation, rooted access,
|
||
host-relative targets, and contained absolute targets translated with `filepath`.
|
||
Preserve the final-entry follow policy and handling of links in parent components.
|
||
Keep the completed map invocation-local; do not attach a cross-call cache to
|
||
`filesystemAccess` or change behavior for concurrently changing filesystems.
|
||
|
||
Acceptance: wrap real host-root operation functions in the private binding to
|
||
count bounded work for the same finite graph; this needs no production observer
|
||
hook. Verify contained absolute links, escaping targets, parent links, true cycles,
|
||
finite aliases, directory/glob integration, and independent namespaces. Run all
|
||
native cases again after the shared evaluator is fully wired. Remove stale
|
||
comments describing the old duplicated implementation.
|
||
Focused check: `go test ./internal/engine -run 'Link|Directory|Glob|Filesystem|Discovery'`.
|
||
|
||
## Stage 10 — Remove The Two Confirmed Unused History Copies
|
||
|
||
Findings: **D02, D03**.
|
||
Owners: `internal/engine/override_decode.go`, `override_context.go`, `load.go`,
|
||
`load_result.go`, and corresponding tests.
|
||
|
||
Status: **complete** (D02, D03). Prior completed commits: `2736ddf`,
|
||
`deb9aa5`, `ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`,
|
||
`7bf2f0c`, `0f9ecf1`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Override|History|Explain|Secret|Independent' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
Remove `decodedOverrideResult.preCLI`, its clone, and assertions only about that
|
||
unused output. Keep the pre-CLI context actually built from `before.final`.
|
||
Remove `Inspection.structuralInput`, the final publication clone made solely for
|
||
it, and that clone's immutable-lookup inclusion. Preserve earlier structural-input
|
||
uses for metadata and sensitivity; preserve `Inspection.resolution` and all active
|
||
replay trees. Do not combine indexes, share mutable paths, or optimize other copies.
|
||
|
||
Acceptance: CLI, inherited/removed paths, default reconciliation, historical
|
||
sensitivity, and independent-load/query ownership remain correct. Use existing
|
||
tests rather than adding tests of the absence of private fields. D01 is not fixed
|
||
by these deletions; its remaining root set is addressed next.
|
||
Focused check: `go test ./internal/engine -run 'Override|History|Explain|Secret|Independent'`.
|
||
|
||
## Stage 11 — Remove Dead Combination Evidence And Orphan Helpers
|
||
|
||
Findings: **D04, F05**.
|
||
Owners: `internal/engine/sequence_combine.go`, `inheritance*.go`, `compose.go`,
|
||
`history.go`, `secret.go`, `destination.go`, and associated tests.
|
||
|
||
Status: **complete** (D04, F05). Prior completed commits: `2736ddf`,
|
||
`deb9aa5`, `ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`,
|
||
`7bf2f0c`, `0f9ecf1`, `119b42c`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Sequence|Collection|Inheritance|History|Secret|Destination' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
Remove unread `discardedLower` and `inheritanceReplacement.path`, `previous`, and
|
||
`effective`, plus calculations solely producing them. Keep replacement `before`
|
||
and `after` trees for accounting, projections, active origins/history, and append's
|
||
use of `sequenceIndexes`. Keep overlay's uses of `displacedOrigins`.
|
||
|
||
Reconfirm and remove the six uncalled definitions listed in audit F05:
|
||
`mergeDisjointMappings`, `mergeHistoryDocument`, `inheritanceDirectiveLocation`,
|
||
`addSecretEvidencePath`, `PlacementModeFromFacade`, and `PlacementModeForFacade`.
|
||
Preserve their active counterparts and implicit/interface invocation elsewhere.
|
||
Update only private assertions about the removed evidence and stale comments/imports.
|
||
|
||
Acceptance: sequence replace/append/upsert, inheritance history, sensitivity,
|
||
and retained replacement accounting still work. Inspect all remaining references;
|
||
absence of graph callers alone is not the deletion criterion.
|
||
Focused check: `go test ./internal/engine -run 'Sequence|Collection|Inheritance|History|Secret|Destination'`.
|
||
|
||
## Stage 12 — Prepare The Actual Inspection Document State Once
|
||
|
||
Finding: preparation portion of **D01**.
|
||
Owners: `internal/engine/load.go`, `load_result.go`, `history.go`, `inheritance_load.go`.
|
||
|
||
Status: **complete** (preparation portion of D01). Prior completed commits:
|
||
`2736ddf`, `deb9aa5`, `ae3408a`, `ea27f17`, `08a8c25`, `877c497`,
|
||
`8aba43d`, `7bf2f0c`, `0f9ecf1`, `119b42c`, `247a12b`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Load|History|Explain|Inheritance|Validation|Secret' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
1. Extract preparation of the existing `Inspection` document/history fields from
|
||
publication. Produce the actual owned pre-CLI copy, inherited-operation copies,
|
||
application lists, raw resolution snapshots, and coalesced replay that the result
|
||
will retain. Reuse the existing `Inspection` owner, not another universal DTO.
|
||
2. Prepare this state at the current history-check boundary: after resolution and
|
||
before secret environment lookup and final validation. Metadata's separately
|
||
cloned inheritance evidence is part of this state too. Keep required copies;
|
||
this stage moves their ownership, not their semantics.
|
||
3. Pass that prepared object onward privately. Publication fills metadata paths,
|
||
sensitivity, source evidence, contexts, diagnostics, and validated configuration
|
||
using the same document objects; it must not create fresh document/history
|
||
copies after the final quota check. Build the immutable lookup after admission.
|
||
4. Keep application callback/predicate order unchanged. In particular, do not
|
||
move secret classification/predicate calls ahead of validation just to prepare
|
||
the document roots. Earlier structural trees may remain temporary inputs to
|
||
finalization without becoming published roots.
|
||
|
||
Acceptance: existing lifecycle and explanation behavior stays intact. Document
|
||
the concrete retained document fields for Stage 13. This preparatory stage alone
|
||
does not close D01; avoid temporary public state or compatibility APIs if the
|
||
handoff is not yet buildable.
|
||
|
||
Prepared retained roots for Stage 13 admission are `Inspection.baseline`,
|
||
`input`, `final`, `preCLI`, source/CLI application values, raw and coalesced
|
||
resolution before/after values, and inspection inheritance parent/effective
|
||
values. `Metadata.inheritance` holds its separate cloned parent/effective values;
|
||
the immutable document lookup is built only during publication.
|
||
Focused check when buildable: `go test ./internal/engine -run 'Load|History|Explain|Inheritance|Validation|Secret'`.
|
||
|
||
## Stage 13 — Enforce The Quota On The Prepared Published Graph
|
||
|
||
Finding: completes **D01**.
|
||
Owners: `internal/engine/history.go`, `load.go`, `load_result.go`, history/lifecycle tests.
|
||
|
||
Status: **complete** (D01). Prior completed commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`, `7bf2f0c`,
|
||
`0f9ecf1`, `119b42c`, `247a12b`, `4fcff5f`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'History|Explain|Inheritance|Override|Resolution|Load' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
1. Use one explicit root visitor on the prepared inspection, feeding a
|
||
pointer-aware `retainedNodeCounter` fresh for this publication, not the earlier
|
||
transient expansion counter. Include baseline, input, final, pre-CLI,
|
||
source/CLI application values, every raw and coalesced resolution before/after
|
||
tree, inspection inheritance parents/effective trees, and metadata inheritance
|
||
parents/effective trees. Count each distinct node once; different copies count
|
||
separately. Lookup keys may reference only trees covered by that retained set.
|
||
Keep the admission counter temporary; do not publish its visited-node map.
|
||
2. Replace the mismatched final checkpoint with this admission before secrets
|
||
and validation. Keep existing early inheritance guards; do not cumulatively
|
||
charge unrelated transient graphs against the final published-root check.
|
||
Share that publication counter with preparation to charge actual retained
|
||
copies as they become owned, stopping before further copies after overflow.
|
||
Do not charge an unretained source merely because it is about to be cloned.
|
||
Use source-size prechecks where straightforward, without a
|
||
new budget-aware generic clone framework or universal allocation guarantee.
|
||
3. Keep production `historyNodeLimit` unchanged. Pass a private history-limit
|
||
argument from the two engine entry paths to `executeLoadRequest` so engine
|
||
tests can exercise the real lifecycle with a small allowance. Do not add a
|
||
public option, global mutable limit, or environment switch.
|
||
4. Overflow returns nil result and the existing safe `CodeExplainLimitExceeded`.
|
||
Publication consumes the admitted prepared state with no unaccounted document
|
||
copies. Reuse the accounting visitor rather than maintaining two root lists
|
||
for separate ordinary/grouped implementations.
|
||
|
||
Acceptance: use the real execution/publication path for a simple load and a small
|
||
layered CLI+inheritance+resolution case. Independently enumerate document fields
|
||
from the resulting inspection in test code, rather than using the production
|
||
collector as its own oracle. Exact count succeeds; one less fails before final
|
||
validation/environment lookup. Verify shared-node count-once behavior and distinct
|
||
copies. Keep required raw snapshot retention and detached inspection tests.
|
||
Focused check: `go test ./internal/engine -run 'History|Explain|Inheritance|Override|Resolution|Load'`.
|
||
|
||
## Stage 14 — Cache Completed Dynamic Eligibility Within An Operation
|
||
|
||
Finding: **A05**; requires Stage 2.
|
||
Owners: `internal/engine/value_operation.go`, projection/clone tests.
|
||
|
||
Status: **complete** (A05). Prior completed commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`, `7bf2f0c`,
|
||
`0f9ecf1`, `119b42c`, `247a12b`, `4fcff5f`, `12d350a`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -run 'Dynamic|Projection|Clone|Resolution' -count=1`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine -count=1`,
|
||
and `git diff --check`.
|
||
|
||
Add an operation-local set of successfully approved concrete types/descriptors.
|
||
After a completed descriptor build and a full successful secret-eligibility walk,
|
||
record approval. Subsequent values of that type in the same `valueOperation` reuse
|
||
it. Do not treat cycle-walker visits as approvals, retain partial results, cache
|
||
path-bearing failures, or transfer this set across loads/operations.
|
||
|
||
Acceptance: repeated typed nil pointers to a wide supported type avoid repeated
|
||
eligibility walks; true unsupported/secret-bearing types still fail, including
|
||
recursive descriptor graphs. Verify failed construction remains recoverable and
|
||
diagnostics identify the current path. Source/algorithm review must show O(S + N)
|
||
eligibility work for N repetitions of one graph of size S; no timing threshold or
|
||
new production instrumentation is necessary.
|
||
Focused check: `go test ./internal/engine -run 'Dynamic|Projection|Clone|Resolution'`.
|
||
|
||
## Stage 15 — Establish Public Historical-Path And Record Ownership Coverage
|
||
|
||
Findings: **F09**, replacement path coverage for **F12**, prerequisites for **F04**.
|
||
Owners: root external-package facade tests and existing source-snapshot tests.
|
||
|
||
Status: **complete** (F09; replacement path coverage for F12; public snapshot
|
||
ownership review for F04). Prior completed commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`, `7bf2f0c`,
|
||
`0f9ecf1`, `119b42c`, `247a12b`, `4fcff5f`, `12d350a`, `08f2a8f`.
|
||
Validation passed: `env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test . -run 'Historical|Metadata|Facade|Source|WrappedPaths' -count=1`
|
||
and `git diff --check`. Existing public source-version and import-selection
|
||
snapshot tests cover repeated-query detachment of location/version pointers and
|
||
nested selections; no duplicate ownership matrix was added.
|
||
|
||
Add compact public tests using only root APIs. Cover effective and removed
|
||
historical paths, sorted unique pointer spelling, zero metadata, escaped keys,
|
||
and a numeric-key/sequence shape change. Feed returned paths into `Lookup` and
|
||
`History` to verify contextual shape behavior. Replace an element of a returned
|
||
slice and query again; do not expose private segment storage to test detachment.
|
||
|
||
Before deleting clone scaffolding, verify meaningful mutable returned source
|
||
records through repeated public queries: location/version pointers and nested
|
||
selection slices stay detached, with existing nil/empty distinctions preserved.
|
||
Reuse existing cases where they already protect the risk rather than duplicating
|
||
the engine matrix. Remove `TestWrappedPathsRetainEngineSegmentIntent` only after
|
||
its useful public behavior is covered.
|
||
|
||
Acceptance: a broken root path conversion or shallow returned record would fail
|
||
these tests even while engine tests pass. Engine path-intent tests remain.
|
||
Focused check: `go test . -run 'Historical|Metadata|Facade|Source|WrappedPaths'`.
|
||
|
||
## Stage 16 — Retire Disconnected Facade Clones And Redundant Scaffolding
|
||
|
||
Findings: **F04**, remaining **F12**; requires Stage 15.
|
||
Owners: root `records.go`, `records_test.go`, `facade_internal_test.go`,
|
||
`facade_compatibility_test.go`, `facade_candidate_test.go`, `facade_equivalence_test.go`.
|
||
|
||
Status: **complete** (F04; remaining F12). Prior completed commits: `2736ddf`,
|
||
`deb9aa5`, `ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`, `7bf2f0c`,
|
||
`0f9ecf1`, `119b42c`, `247a12b`, `4fcff5f`, `12d350a`, `08f2a8f`, `826062a`.
|
||
Validation passed: `env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test .`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./examples/consumers`,
|
||
and `git diff --check`.
|
||
|
||
Remove the complete private clone family named in audit F04 after reconfirming
|
||
that only the tests enter it. Preserve engine clones, active `Path.clone`, explicit
|
||
root conversions, and nil/empty conversion checks. Remove the ineffective
|
||
whole-Destination assignment assertion and private clone-only assertions, retaining
|
||
conversion/public query ownership protection from Stage 15.
|
||
|
||
Remove the duplicate six-type ownership loop and trivial assignment equality
|
||
assertion, keeping the comprehensive root type owner and compile-time interfaces.
|
||
Route source-ownership tests through public `Load` and root error typing instead
|
||
of `source.toEngine()` plus direct engine loads. Consolidate reader cases with the
|
||
existing equivalence owner while preserving byte mutation and copied-reader cases.
|
||
|
||
Acceptance: public construction, errors, callback lifetime, source identity,
|
||
disclosure, and returned ownership remain represented. Do not delete static public
|
||
or dependency-boundary tests or infer that engine coverage replaces facade tests.
|
||
Focused check: `go test .` and `go test ./examples/consumers`.
|
||
|
||
## Stage 17 — Remove The Remaining Unread Private Members
|
||
|
||
Finding: completes **F06**, following its schema-path subset in Stage 1.
|
||
Owners: the exact records/files listed in [audit F06](audit.md#f06-trim-unread-private-result-and-transport-members).
|
||
|
||
Status: **complete** (F06). Prior completed commits: `2736ddf`, `deb9aa5`,
|
||
`ae3408a`, `ea27f17`, `08a8c25`, `877c497`, `8aba43d`, `7bf2f0c`,
|
||
`0f9ecf1`, `119b42c`, `247a12b`, `4fcff5f`, `12d350a`, `08f2a8f`,
|
||
`826062a`, `e4a5ecb`. Validation passed:
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./internal/engine`,
|
||
`env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test .`, and
|
||
`git diff --check`.
|
||
|
||
Remove `collectionPolicyResolution`'s four named unread fields; `inputPresence.null`;
|
||
directive extraction's two presence flags; `importSelection.outcome`;
|
||
`importDeclarationSpec.excludeSet`; `selectedOverlayRequest.filesystem`;
|
||
`sequenceDeclarationSpec.destinationLocation`; source-version evidence's `absence`
|
||
and `checked` and ancestor `kind`; and `postDecodeOverrideContext.sources`.
|
||
Remove computations only feeding those slots and adjust their private assertions.
|
||
|
||
Keep declaration normalization, effective policy flags, candidate/admission rules,
|
||
public empty-selection outcomes, actual version callbacks, ancestor sensitivity
|
||
paths, public null metadata, location reporting, and the local source list used
|
||
to construct resolution input. Check references/build constraints again; do not
|
||
extend this deletion list to identity tokens, map-key fields, or interface methods.
|
||
|
||
Acceptance: use public behavior for tests that formerly checked inactive annotations.
|
||
No active validation or distinction is removed with its redundant transport flag.
|
||
Focused check: `go test ./internal/engine` and `go test .`.
|
||
|
||
## Stage 18 — Retire The Parallel Import Directive Parser
|
||
|
||
Finding: **F03**.
|
||
Owners: `internal/engine/import_spec.go`, `import_spec_test.go`, directive/selector tests.
|
||
|
||
Change `parseImportSpecsYAML` to enter `extractRootDirectives` with the appropriate
|
||
source context and return its imports/diagnostics. Keep parsing source fixtures
|
||
through the real YAML boundary. Remove `parseImportSpecificationDirective`, its
|
||
parallel mapping/field validation, and support used solely by it. Do not remove
|
||
`parseImportDeclarations` or the imports-only context still used after layered
|
||
outer-directive extraction.
|
||
|
||
Acceptance: malformed fields/types/selectors and exact diagnostic paths continue
|
||
to be checked by the actual production parser; valid import fixtures retain their
|
||
meaning. Inspect the assertions before accepting wording changes from replacing
|
||
the obsolete parser. Keep classifications/sites, not incidental wording, as the
|
||
contractual expectations.
|
||
Focused check: `go test ./internal/engine -run 'Import|Directive|Directory|Glob|SourceVersion'`.
|
||
|
||
## Stage 19 — Strengthen Ownership Fuzzing And Observable Import Ordering
|
||
|
||
Findings: **F10, F11**.
|
||
Owners: `internal/engine/composition_fuzz_test.go`, `import_selection_test.go`.
|
||
|
||
In `FuzzCompositionAndOverlay`, mutate an existing retained descendant or in-length
|
||
element before comparing both original inputs with their saved values. Handle
|
||
empty containers without assuming append affects shared storage. Keep existing
|
||
bounded inputs, immutability checks, and commutativity properties; do not add
|
||
another fuzz target or ownership matrix.
|
||
|
||
Replace the exact two-stats-per-file assertion in
|
||
`TestSelectNamedImportsChecksNormalizedPathsLexically` with ordered candidate paths
|
||
and meaningful first-failure location. Consolidate with existing lexical-selection
|
||
tests if this subsumes the standalone case. Keep non-opening of excluded contents
|
||
and genuine first-error behavior where those are the tested requirements.
|
||
|
||
Acceptance: a shallow existing-descendant alias would fail the fuzz assertion,
|
||
while a harmless change in stat implementation would not fail ordering coverage.
|
||
Focused check: `go test ./internal/engine -run 'FuzzCompositionAndOverlay|SelectNamedImports'`.
|
||
Run the bounded composition campaign in Stage 33 after the later assembly changes
|
||
have landed, rather than treating an earlier run as evidence for those changes.
|
||
|
||
## Stage 20 — Use Only Deferred Acquisition In Production And Fixtures
|
||
|
||
Finding: **F01**.
|
||
Owners: `internal/engine/acquisition.go`, `import_acquisition.go`, `imports_test.go`,
|
||
and callers identified by request-construction tracing.
|
||
|
||
Remove `deferAssembly` and the eager branch. Retain pending-item placement
|
||
reservation and return acquired scopes/import graph/accounting consistently.
|
||
Migrate test helpers needing final values to invoke the real composition boundary
|
||
explicitly after acquisition. Keep acquisition-only tests on the acquired scope
|
||
instead of silently composing it. Ordinary and grouped production requests use
|
||
the same handoff without a replacement mode flag.
|
||
|
||
Acceptance: imports, item-local scopes, nested destinations, dependency ordering,
|
||
version protections, optional absence, resource closure, and byte/node accounting
|
||
still run through the active protocol. Keep tests' intended boundaries; do not
|
||
delete meaningful composition cases when changing fixture setup.
|
||
Focused check: `go test ./internal/engine -run 'Import|Destination|SourceVersion|LoadGrouped|Underlay'`.
|
||
|
||
## Stage 21 — Isolate Parsed-Input Directive And Version Processing
|
||
|
||
Finding: first part of **F02**; requires Stage 20.
|
||
Owners: `internal/engine/acquisition.go`, existing directive/source-version records.
|
||
|
||
Extract the directive/version-processing block from `acquireInput` into a cohesive
|
||
operation returning the existing `sourceVersionProcessingResult` and diagnostics.
|
||
Pass the parsed source, existing `inputAcquisitionRequest`, and load-local version
|
||
state explicitly; read role, destination/placement, context, and protection from
|
||
that request rather than introducing a long flattened parameter list. Compute
|
||
source ID once. Keep updating
|
||
`mainProcessing` with a clearly owned completed result, not pointers into incidental
|
||
locals scattered among branches.
|
||
|
||
Retain distinct source and layered processing: ordinary sources process their
|
||
root directives with their source policy; layered loading extracts its outer
|
||
directives before enabled version handling and restores those declarations to
|
||
the processing result. Default/enabled mode timing and warning accumulation must
|
||
not become one inferred common policy. Keep the imports-only processing context
|
||
where it remains necessary. Do not add a new context hierarchy or public request.
|
||
|
||
Acceptance: source-version tests for main, imports, underlays, mounted/item
|
||
destinations, selected overlays, and NoSource preserve callback counts, order,
|
||
warnings, safe diagnostics, and retained declaration locations.
|
||
Focused check: `go test ./internal/engine -run 'SourceVersion|Version|Directive|Destination'`.
|
||
|
||
## Stage 22 — Isolate Standalone Placement And Acquisition Completion
|
||
|
||
Finding: second part of **F02**.
|
||
Owners: `internal/engine/acquisition.go`, `load_input.go`, destination/context helpers.
|
||
|
||
Extract standalone contribution placement/registration and passive-context
|
||
completion into explicit operations on the existing assembled-input/acquisition
|
||
owners. Keep discovery selection and parsing visible in `acquireInput`, then make
|
||
its imported-versus-standalone dispatch and completion readable. Each helper
|
||
returns its result/diagnostic; avoid closures mutating a collection of surrounding
|
||
locals. Keep live handles confined to their acquisition owner.
|
||
|
||
Preserve local versus placed documents, unresolved item destinations, wrapper-node
|
||
reservation, source identity, NoSource/discovery behavior, and the ordinary overlay
|
||
case that defers filesystem completion. Review aliasing of source slices and
|
||
version evidence when assembling the return value.
|
||
|
||
Acceptance: existing discovery, standalone mapping/reader/native/host input,
|
||
destination accounting, item placement, and passive resource-context tests still
|
||
exercise the same observable behavior. Do not require exact private helper calls.
|
||
Focused check: `go test ./internal/engine -run 'Discovery|Destination|Source|Resource|Filesystem'`.
|
||
|
||
## Stage 23 — Clarify Ordinary Context And Structural-Expansion Handoffs
|
||
|
||
Finding: third part of **F02**.
|
||
Owners: `internal/engine/load.go`, `load_input.go`, inheritance/version helpers.
|
||
|
||
Extract the substantial ordinary-load filesystem/resource-context fallback into
|
||
an operation that returns the completed `assembledInput` plus diagnostics. Keep
|
||
existing resolved filesystem facts when available rather than reopening handles
|
||
to recover the same information; retain required validation when facts are absent.
|
||
Isolate structural inheritance expansion/version-path
|
||
projection as one further handoff using the existing composed/assembled records.
|
||
|
||
Keep the coordinator's visible order: schema/source-version timing, acquisition,
|
||
ordinary/grouped assembly and selected overlay, structural expansion, decoding,
|
||
CLI, resolution, Stage 13 history admission, secrets, identity validation, final
|
||
validation, and publication. Do not move callback execution into an extraction
|
||
that changes when errors suppress later phases.
|
||
|
||
Acceptance: ordinary imports and non-import overlays, absent main input with
|
||
explicit context, resource paths, inheritance version protection, callback order,
|
||
and fail-before-later-callback cases remain intact. Re-run panic cleanup and
|
||
retained-history integration cases after moving the handoffs.
|
||
Focused check: `go test ./internal/engine -run 'Load|Overlay|Resource|Inheritance|Version|History'`.
|
||
|
||
## Stage 24 — Share Layered Tier Application And Evidence Projection
|
||
|
||
Finding: completes **F02**.
|
||
Owners: `internal/engine/load_request.go`, `load_input.go`, tier projection helpers.
|
||
|
||
Extract the repeated completed-tier application operation: collect that tier's
|
||
declarations, initialize or apply through the assembly owner, then project only
|
||
the incoming sources/version evidence using the returned sequence operations.
|
||
Use explicit source/evidence ranges from the caller; keep main-version projection
|
||
and its protection rules recognizable rather than hiding them in a generic loop.
|
||
|
||
Keep acquisition order separate from precedence application, distinct tier-local
|
||
and load-wide accounting, absence of empty groups, main-only default version
|
||
checking, and selected-overlay rules. This helper must accept the existing policy
|
||
scope contract so Stages 27–28 can later change that private contract once, at
|
||
named composition/application boundaries.
|
||
|
||
Acceptance: grouped source ordering, underlay precedence, strict-within-tier
|
||
conflicts, mounted source/version coordinates, ordinary/grouped differences,
|
||
metadata/history/diff, and warnings remain stable. Update the implementation-flow
|
||
description only after the handoffs actually land, without restating YAML policy.
|
||
Focused check: `go test ./internal/engine -run 'Grouped|Layered|Underlay|SourceVersion|Sequence|Diff'`.
|
||
|
||
## Stage 25 — Separate Collection Preparation, Application, And Finalization
|
||
|
||
Finding: preparation/ownership part of **E03**.
|
||
Owners: `internal/engine/overlay.go`, `compose.go`, `collection_policy*.go`.
|
||
|
||
Refactor `mergeOverlayMappingsWithCollectionPolicy` into three named responsibilities:
|
||
|
||
- Prepare incoming claims/declaration groups and per-destination resolutions
|
||
against read-only lower state, with no value/origin mutation.
|
||
- Apply recursive value combination through one small overlay operation owner
|
||
holding the existing result/index plus its accumulated diagnostics, sequence
|
||
history, and destinations receiving values. Use its named callback methods
|
||
rather than many callbacks capturing separate mutable maps/slices. Put the
|
||
root-origin merge in this application phase as well.
|
||
- Finalize declaration-only policy against the completed value and emit sorted
|
||
sequence evidence. Build any retained-value lookup after mutation, not before.
|
||
|
||
Keep these as engine-private operations, not generic strategy interfaces. Preserve
|
||
current data semantics here; Stages 26–28 replace the full-state work. A temporary
|
||
large clone retained by this extraction is not completion of E01/E02.
|
||
|
||
Acceptance: policy declarations/conflicts, pending keys, mode-only changes,
|
||
replace/append/upsert, nested record ownership, origin/history ordering, and failed
|
||
combination diagnostics remain protected. Newly separated inputs/results make
|
||
lower versus incoming versus retained ownership explicit.
|
||
Focused check: `go test ./internal/engine -run 'Collection|Sequence|Overlay|Inheritance|History'`.
|
||
|
||
## Stage 26 — Index Assembly Policy Retirement By Semantic Path
|
||
|
||
Finding: **E01**.
|
||
Owners: `internal/engine/collection_policy_state.go`, `compose.go`, `overlay.go`,
|
||
and a small private policy-index file if that clarifies ownership.
|
||
|
||
1. Add a mutable semantic-segment retirement trie alongside the assembly's policy
|
||
table. Index destinations with state cleared by `clearCollectionIdentity`;
|
||
mode-only entries stay in the table for exact lookup but need no retirement
|
||
terminal. Use `pathSegment` keys so numeric mapping keys and sequence indices
|
||
stay distinct. Build once when the assembly adopts its initial owned scope,
|
||
not for every replacement or tier.
|
||
2. Route assembly policy writes/retirement through its owner so table and index
|
||
remain synchronized. Keep immutable/local scopes outside this owner independent;
|
||
do not add index rebuilding to their every lookup or snapshot.
|
||
3. Descendant retirement walks only the indexed subtree for the replaced prefix.
|
||
Clear old YAML identity and record-owned state, preserving explicit modes and
|
||
application constraints. Retire before installing policy introduced by the
|
||
incoming replacement; declarations established afterward must survive. Remove
|
||
cleared terminals and prune empty branches. Replacing an ancestor again must
|
||
not rescan already cleared mode-only policies; later writes reindex new state.
|
||
4. Reuse the semantic-index conventions of `inheritancePolicyIndex`, but keep
|
||
this mutable assembly owner distinct from that borrowed record-local index.
|
||
Do not change the pointer-spelling index used for history/dependencies.
|
||
|
||
Acceptance: retain `collection_retirement_test.go` cases for all replacement kinds,
|
||
ancestor removal with new pending identity, explicit modes, and registrations.
|
||
Add escaped/numeric semantic-path cases and a bounded disjoint P-policy/Q-scalar
|
||
case, plus repeated retirement above retained mode-only entries. Review the
|
||
complete caller path: one index construction plus prefix/affected-subtree work
|
||
must replace Q scans of all P entries. No exact syscall/timing test
|
||
or public complexity guarantee is needed.
|
||
Focused check: `go test ./internal/engine -run 'Collection|Retirement|Sequence|Inheritance'`.
|
||
|
||
## Stage 27 — Borrow Lower Policy And Produce Sparse Strict-Tier State
|
||
|
||
Finding: first part of **E02**; coordinates **E03**.
|
||
Owners: `internal/engine/collection_retirement.go`, `compose.go`, `import_scope.go`,
|
||
and nested/acquired scope composition callers.
|
||
|
||
1. Replace the full lower-scope copy used for strict resolution with a small
|
||
borrowed read view: the existing lower table plus current replacement masks.
|
||
Exact lookup returns an immutable policy view; if old identity is masked at
|
||
that destination, clear it in the returned header, not in the borrowed table.
|
||
Keep explicit modes. Reuse `collectionReplacementPaths`' inclusive/exclusive
|
||
prefix semantics; do not accumulate a chain of views across tiers.
|
||
2. Compute replacement eligibility from incoming claims/declarations and exact
|
||
lower/application lookups. Do not enumerate the full lower table just to clear
|
||
entries in a transient copy. Clone an affected policy only when constructing
|
||
an independently owned result or changing record-owned state.
|
||
3. `resolveStrictCollectionPolicies` produces identity information and an owned
|
||
sparse policy scope for destinations touched by incoming values/declarations.
|
||
That scope is not another complete copy of accumulated lower policy. Preserve
|
||
record ownership when a declaration has no new value.
|
||
4. Thread this contract through acquired/item-local strict composition. Keep
|
||
complete independently owned snapshots where local record scopes/history need
|
||
them; sparse incoming state must not accidentally borrow mutable record maps.
|
||
Update private tests that assumed an incoming tier included unrelated lower
|
||
entries to check the complete assembled result instead.
|
||
|
||
Acceptance: strict duplicate rejection still sees applicable lower identity;
|
||
higher declarations cannot repair invalid lower composition. Pending declarations,
|
||
replacement masks, mode persistence, and nested item-local scopes remain correct.
|
||
Inspect the scalar-only-tier path: work depends on incoming claims/declarations,
|
||
not P unrelated retained policies. Wiring to assembly is finished in Stage 28;
|
||
do not retain an eager full-scope adapter merely to make this stage compile.
|
||
Focused check when buildable: `go test ./internal/engine -run 'Compose|Collection|Item|Underlay|Sequence'`.
|
||
|
||
## Stage 28 — Apply Sparse Incoming Policy Without Full-Table Preparation
|
||
|
||
Finding: completes **E02**; uses Stage 26 and Stage 27.
|
||
Owners: `internal/engine/overlay.go`, `load_request.go`, `import_scope.go`,
|
||
`inheritance_prepare.go`, and collection retirement callers.
|
||
|
||
1. In overlay preparation, discover relevant destinations by walking incoming
|
||
claims and declaration paths, checking the lower table at those exact paths.
|
||
Replace the loop over every retained policy and the whole-overlay lookup built
|
||
solely to support that loop. Include non-sequence claims at exact retained
|
||
policy destinations so invalid append/upsert inputs still fail as before.
|
||
2. Use the borrowed lower view for operation resolution; retain only incoming
|
||
owned resolutions and sparse policy changes. Apply those changes at the named
|
||
assembly owner using Stage 26's synchronized updates. Unchanged lower entries
|
||
remain in that owner, not copied into every incoming tier.
|
||
3. Retirement caused by replaced ancestors uses the live index at the existing
|
||
mutation boundary. Reconcile declaration-only updates after value application.
|
||
Preserve current-tier policy established after retirement and validate a
|
||
retained sequence when a declaration alone requires it.
|
||
4. Update tier application and any inheritance callers of the old lower-policy
|
||
preparation helper. For inheritance, retain independent parent/child local
|
||
scopes where ownership requires them; share the replacement-mask/read-view
|
||
logic rather than copying the load-wide policy store into a tier.
|
||
|
||
Acceptance: with P retained policies and T disjoint scalar-only tiers, no path may
|
||
clone/scan all P policies per tier or rebuild their index. Inspect all preparation,
|
||
strict composition, application, and finalization callers for hidden full-table
|
||
fallbacks. Add a small multi-tier case preserving untouched rules and nested
|
||
record owners, plus retained-key-only/mode-only and ancestor-replacement cases.
|
||
Required value/history copies and affected record payload remain legitimate costs.
|
||
Focused check: `go test ./internal/engine -run 'Collection|Retirement|Sequence|Underlay|Layered|Inheritance|History'`.
|
||
|
||
## Stage 29 — Consolidate Sequence-History Translation And Finalize Policy Owners
|
||
|
||
Finding: completes **E03**.
|
||
Owners: `internal/engine/overlay.go`, `inheritance_combine.go`,
|
||
`inheritance_collection.go`, and sequence-history helpers.
|
||
|
||
Extract one pure sequence-history descriptor constructor taking destination,
|
||
operation, incoming value, and `sequenceCombinationResult`. It owns replace-versus-
|
||
mapped translation and detached projection/replacement storage. Use it in both
|
||
overlay and inheritance instead of independently rebuilding that descriptor.
|
||
|
||
Keep overlay and inheritance record-policy rebuilding as separate named operations:
|
||
overlay retains incoming/lower record ownership, while inheritance uses current
|
||
versus inherited scope conversions with different mode lifetimes. Share existing
|
||
projection indexes but do not add a callback/strategy framework or common mutable
|
||
origin owner to unify those distinct operations. This resolves the roadmap's
|
||
record-selection-sharing evaluation in favor of explicit ownership-specific loops.
|
||
|
||
Acceptance: value combination still has one `combineSequence` owner; preparation,
|
||
application, and declaration-only finalization have clear inputs/results without
|
||
reintroducing full-table work. Verify sequence history and sparse nested projection,
|
||
record replacement/removal, modes, origin locations, and sensitivity for both
|
||
precedence and inheritance.
|
||
Focused check: `go test ./internal/engine -run 'Sequence|Collection|Inheritance|History|Explain|Sensitivity'`.
|
||
|
||
## Stage 30 — Index Original Tier-History Values Lazily
|
||
|
||
Finding: **E04**.
|
||
Owners: `internal/engine/sequence_history_projection.go`, projection/history tests.
|
||
|
||
In `projectTierHistoryApplications`, lazily construct one `documentLookup` for an
|
||
original application value only when an applicable mapped sequence needs its
|
||
container. Reuse it for that application's sequence contributions through
|
||
`documentAtPathWithLookup`. Keep source values immutable and authored coordinates
|
||
separate from assigned output positions. Rebuild projected sidecar indexes after
|
||
edits as before; do not retain the temporary lookup or cache it across loads.
|
||
|
||
Acceptance: wildcard-selected sibling collections, sparse/noncontiguous mappings,
|
||
nested sequence projections, and upsert replacement evidence remain correct and
|
||
detached. Source review shows O(V + S × D + I) lookup/projection work for one
|
||
application, excluding existing sorting/copies, rather than scanning a V-wide
|
||
mapping for each of S contributions. Use existing wide/sparse fixtures, not timing
|
||
thresholds or another explanation-replay implementation.
|
||
Focused check: `go test ./internal/engine -run 'Tier|Projection|Sequence|History'`.
|
||
|
||
## Stage 31 — Index Compound Identity Restoration
|
||
|
||
Finding: **E05**.
|
||
Owners: `internal/engine/inheritance_scope.go` and identified-inheritance tests.
|
||
|
||
Build an identity-field membership set and a mapping-position index for the merged
|
||
record. Walk child entries once; for identity members, replace the corresponding
|
||
merged entry with a detached copy preserving the child's key/value locations.
|
||
Keep merged mapping order and exact identity comparison/parent selection. Do not
|
||
remove the restoration step or reinterpret upsert as recursive body merging.
|
||
|
||
Acceptance: compound identity, many nonidentity siblings, escaped/literal keys,
|
||
child locations, mapping order, missing-key validation, and detached retained
|
||
templates remain correct. Source review establishes O(K + C + M) lookup work plus
|
||
required copied values for K identity fields and C/M mapping entries.
|
||
Focused check: `go test ./internal/engine -run 'Identified|Identity|Inheritance'`.
|
||
|
||
## Stage 32 — Index Secret Failure Locations Only When Needed
|
||
|
||
Finding: **F08**.
|
||
Owners: `internal/engine/secret_reference.go`, `validation.go` lookup helper,
|
||
`secret_reference_test.go`.
|
||
|
||
Let the existing per-resolution secret operation own a nil `documentLookup` until
|
||
its first diagnostic needs attribution. On that failure, build it from the fixed
|
||
reference-input tree and reuse `nearestDocumentLocationWithLookup` for subsequent
|
||
failures. Thread that operation-local state through the existing walk/diagnostic
|
||
helper rather than rebuilding an index per reference or introducing global state.
|
||
|
||
Acceptance: many missing/empty/invalid references in a wide mapping retain every
|
||
safe diagnostic and exact/ancestor fallback location; repeated environment names
|
||
still use the existing lookup cache. Include escaped/numeric and defaults-only
|
||
paths. The successful path builds no failure-only document index. Inspect for
|
||
O(V + F × D) location work, apart from sorting, rather than repeated full scans;
|
||
keep the environment cache and credential bytes out of diagnostics/history.
|
||
Focused check: `go test ./internal/engine -run 'Secret|Location|Validation'`.
|
||
|
||
## Stage 33 — Integrate, Document, And Verify The Complete Sprint
|
||
|
||
Findings: final acceptance for **all selected IDs**.
|
||
|
||
1. Review the traceability table below against actual changes and tests. Finish
|
||
every check deferred by an intermediate unbuildable stage. Reconfirm no
|
||
selected work was silently replaced by a smaller cleanup or deferred profiling.
|
||
2. Review boundary interactions: transactional descriptors plus eligibility
|
||
caching; slice views plus detachment; callback panic plus root closure; both
|
||
directory-reading implementations and link bindings; complete published-history
|
||
admission after copy cleanup; policy deltas/retirement plus tier history,
|
||
inheritance, source-version projection, and secret locations.
|
||
3. Reuse representative public/consumer cases to verify agreement among final
|
||
values, presence/null, origins, historical queries, explanation, redaction,
|
||
diff, safe errors, and independent loads. Add only a missing integration case
|
||
with a concrete risk; do not duplicate every engine test at the root.
|
||
4. Update `docs/internal/overview.md` and `docs/internal/imports.md` for the actual
|
||
resulting responsibilities. Correct stale implementation-stage comments in
|
||
changed owners. Update canonical declaration/configuration wording only where
|
||
needed to accurately describe a repaired existing behavior; do not recast
|
||
private algorithms or tests as new public promises. No new ADR/policy is needed.
|
||
5. Perform the checks below, inspect the diff for unintended public/API, dependency,
|
||
limit, or scope changes, and record completion/validation per stage. Keep the
|
||
selected planning records until normal release preparation retires them.
|
||
|
||
Final checks, from the repository root:
|
||
|
||
```sh
|
||
env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off ./scripts/check.sh
|
||
env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test -race ./...
|
||
env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./... -shuffle=on -count=3
|
||
env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off \
|
||
go test ./internal/engine -run '^$' -fuzz '^FuzzCompositionAndOverlay$' -fuzztime=30s -parallel=2
|
||
env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off go test ./examples/consumers -count=1
|
||
env GOTOOLCHAIN=local GOPROXY=off GOTELEMETRY=off \
|
||
go test . -run 'TestLocalDocumentationLinks|TestDependencyImportBoundary|TestPublicFacadeBoundary' -count=1
|
||
git diff --check
|
||
```
|
||
|
||
The consumer/boundary reruns are focused confirmation of the changed cross-package
|
||
surface, not new release prerequisites. Reuse unchanged successful checks when
|
||
appropriate under the testing policy. Record actual platform skips and limitations;
|
||
no express macOS validation, CI wait, or release publication is part of completion.
|
||
All Go and documentation changes must be committed in focused nonempty units,
|
||
with a clean workspace and a concise summary of fixed behavior and any remaining
|
||
out-of-scope work. Do not create empty commits for validation-only stages.
|
||
|
||
## Finding-To-Stage Traceability
|
||
|
||
| Finding | Owning stages |
|
||
| --- | --- |
|
||
| A01 | 1 |
|
||
| A02 | 2 |
|
||
| A03 | 3 |
|
||
| A04 | 4 |
|
||
| A05 | 14 |
|
||
| B01 | 5 |
|
||
| B02 | 6 |
|
||
| C01 | 7 |
|
||
| C02 | 8–9 |
|
||
| D01 | 12–13 |
|
||
| D02 | 10 |
|
||
| D03 | 10 |
|
||
| D04 | 11 |
|
||
| E01 | 26 |
|
||
| E02 | 27–28 |
|
||
| E03 | 25, 29, coordinated with 26–28 |
|
||
| E04 | 30 |
|
||
| E05 | 31 |
|
||
| F01 | 20 |
|
||
| F02 | 21–24 |
|
||
| F03 | 18 |
|
||
| F04 | 16, with replacement coverage in 15 |
|
||
| F05 | 11 |
|
||
| F06 | 1, 17 |
|
||
| F08 | 32 |
|
||
| F09 | 15 |
|
||
| F10 | 19 |
|
||
| F11 | 19 |
|
||
| F12 | 15–16 |
|
||
|
||
Stage 33 verifies all 29 entries. C03 and F07 remain deferred, not missing from
|
||
this table. No user/product question remains open; the implementation choices
|
||
necessary for this scope are selected above. Helper names/file splits not fixed
|
||
by an existing declaration can follow local idiom without changing those choices.
|