710 lines
45 KiB
Markdown
710 lines
45 KiB
Markdown
# Consolidation Implementation Plan
|
||
|
||
Status: in progress. Stages 1 through 8 are complete; follow Stage 9 through Stage 20 in order.
|
||
The target executor is a Luna coding agent working through one end-to-end sprint.
|
||
|
||
## Authority, Baseline, And Execution
|
||
|
||
The [consolidation roadmap](consolidation.md) owns the target state, rationale,
|
||
and settled scope. Start from `8a99e6107001c69117a41dc37bc6ea09d2910af9` plus the
|
||
commit introducing this plan. Production code is unchanged from the audited
|
||
`5959537d1adc2516e7f7e8881df72b707ec48c3f`; intervening commits consolidated the
|
||
planning documents and settled the public-API and clone decisions.
|
||
|
||
Read [development.md](../development.md), [agents.md](../policy/agents.md),
|
||
[architecture.md](../policy/architecture.md), [testing.md](../policy/testing.md),
|
||
and [documentation.md](../policy/documentation.md) as applicable. Use the code
|
||
graph for discovery and confirm source at the implementation baseline. Current
|
||
public comments and [config.md](../config.md) define behavior; this plan does not
|
||
add public obligations.
|
||
|
||
Implement only the selected consolidation. Retain the public `Load`,
|
||
`LoadLayered`, options, inputs, callbacks, source/search descriptions, inspection,
|
||
and sequence-registration APIs. Keep the root-to-engine-to-document/YAML
|
||
dependency direction. Clone traversal review/refactoring remains deferred under
|
||
[future.md](future.md#4-clone-traversal-review). Changes to shared descriptors must
|
||
preserve existing cloning behavior. This sprint does not publish a release,
|
||
update dependencies/toolchain requirements, change configuration semantics, or
|
||
introduce complexity/performance gates.
|
||
|
||
Stages are bounded work assignments, not independently shippable commits.
|
||
Intermediate stages may leave callers, tests, or compilation unfinished until a
|
||
later stage. Proceed directly through those dependencies; do not build temporary
|
||
compatibility layers solely to keep each stage compiling. Update affected tests
|
||
with their callers, and run the focused checks below when that portion builds.
|
||
Record any deferred verification accurately. The completed sprint must compile,
|
||
pass its checks, and leave no superseded plumbing. Commit completed, reviewable
|
||
units under the agent policy; stage boundaries do not require separate commits.
|
||
|
||
Preserve existing meaningful validation/processing order, diagnostics, ownership,
|
||
and limits. Private helper order and representation are not new contracts. Do
|
||
not add rejection rules, error recovery, caches, new merge strategies, another
|
||
pipeline framework, or tests for incidental helper choreography. A correctness
|
||
issue discovered during the refactor needs identifiable regression evidence and
|
||
an explicit explanation, rather than an unexplained expectation change.
|
||
|
||
## Chosen Private Design
|
||
|
||
Use the existing `loadRequest[T]` as the normalized complete-load model and
|
||
`loadInput` for each selected input. Keep generic application definitions and
|
||
the final lifecycle in `executeLoadRequest[T]`; acquisition is non-generic.
|
||
The following records are ordinary engine-private data, not extension points.
|
||
Use these names to make migrations concrete; their fields describe the selected
|
||
ownership model, not a requirement to duplicate data already available there.
|
||
|
||
| Record | Fields and ownership |
|
||
| --- | --- |
|
||
| `acquisitionPolicy` | Read-only operation snapshot of `importDefaults`, `sequencePolicy`, and `importLimits`. Snapshot caller-owned defaults at entry, preserving nil versus empty. |
|
||
| `acquisitionState` | Borrowed pointers to the load's `importAccounting` and `sourceVersionState`. One instance per load; version state no longer transports import defaults. |
|
||
| `tierAcquisitionState` | The tier's visited-resource map and shared host namespace token. Underlay groups each own a fresh instance. Existing group graph aggregation and namespace anchors retain their current owners. |
|
||
| `inputAcquisitionRequest` | `input loadInput`, source-version role, root-directive context, applicable protection rule, deferred-assembly choice, and `ordinaryOverlay string`. The last field carries only the ordinary main input's overlay-dependent validation/context timing; it is empty for underlays and layered main. |
|
||
| `importAcquisitionRequest` | The input request, parsed main source, and already computed source-version/directive processing result. It preserves the distinction between source-local processing and application-path protection. |
|
||
| `resolvedFilesystemContext` | `native *nativeNamespace` (nil means host), `hostToken *namespaceToken` where supplied, and `rootPath`, `basePath`, `mainPath` strings. Native facts use slash-relative paths; host facts use the existing canonical host paths. The record contains no open root/file or acquisition access object. |
|
||
|
||
The main private entry is
|
||
`acquireInput(request inputAcquisitionRequest, policy acquisitionPolicy, state *acquisitionState, tier *tierAcquisitionState) (assembledInput, []Diagnostic)`.
|
||
Import acquisition uses the corresponding named request with the same borrowed
|
||
policy/state/tier owners. An access-based worker receives a resolved filesystem
|
||
description and an explicitly owned `*filesystemAccess`, rather than more
|
||
positional paths and independent optional policy parameters. Retain
|
||
`assembledInput`, `acquisitionScope`, and `acquiredContribution` as the structural
|
||
handoffs; add an optional resolved-context field to `assembledInput` for reuse.
|
||
|
||
`importTraversal` borrows the policy and state references. Active routes, stack,
|
||
current item scope/prefix/fallback, and the acquired inclusion graph remain
|
||
traversal-local. Nested item traversal shares the active graph and tier repetition
|
||
state; it does not create another tier. Reusing a file in a later tier remains
|
||
valid. Do not collapse namespace identity, lexical candidate spelling, and stable
|
||
scheduling keys into one key.
|
||
|
||
Context facts are obtained at the existing points where those facts are needed.
|
||
There is no global eager context-validation pass or persistent live-handle owner.
|
||
Directory checks, canonicalization, handle opening/closing, and passive view
|
||
creation remain distinguishable. Reusing a description does not authorize
|
||
earlier checks that bypass an existing earlier error or implicit loading activity
|
||
after publication. Consumer-initiated passive filesystem operations retain their
|
||
existing contract.
|
||
|
||
For inheritance, retain both scope adapters and one executor. Use private
|
||
prepared-child/prepared-parent records for combination inputs, then an
|
||
`inheritanceTemplateResult` containing the resolved value, origins, local policy,
|
||
policy templates, and the controls/transfers/replacements/operations produced by
|
||
that template. Sequence position evidence lives in the existing operation
|
||
records. Merge callbacks mutate that local result, not the enclosing executor's
|
||
accumulator. The executor appends the result and applies existing accounting.
|
||
|
||
For reflection, keep `valueOperation` and `schemaBuilder` as their existing state
|
||
owners. Extract substantial value-kind and inline-field operations as methods;
|
||
do not introduce a generic reflection visitor or merge the two schema views.
|
||
|
||
## Validation Groups
|
||
|
||
Use the installed compatible Go toolchain and already acquired dependencies as
|
||
described in the development guide. Commands run from the repository root with
|
||
`GOTOOLCHAIN=local` and `GOPROXY=off`. These are development checks for this
|
||
refactor, not additions to the release procedure.
|
||
|
||
| Group | Commands and purpose |
|
||
| --- | --- |
|
||
| F: facade cleanup | `go test . -run 'SourceInfo|SourceVersionCheck|ImportSelection|Facade|PublicFacadeBoundary|DependencyImportBoundary'` checks useful forward evidence and the existing public boundary. |
|
||
| A: acquisition integration | `go test ./internal/engine -run 'Load|Grouped|Source|Search|Discovery|Import|Glob|Filesystem|Native|Destination|Overlay|Resource|Version|Sequence|Collection'`; `go test . -run 'Facade|EquivalentSourceForms|Search|SourceContext|ImportDefaults|SourceVersion'`; `go test ./examples/consumers`. Covers normalization, ordinary/layered acquisition, selection, contexts, scopes, and consumer workflows. |
|
||
| I: inheritance | `go test ./internal/engine -run 'Inheritance|Inherited|Sequence|Collection|Version|History|Explain|Sensitivity|Secret'`; `go test . -run 'Inheritance|Inherited|SequenceIdentity|FinalSequence'`. Covers shared adapters and their evidence. |
|
||
| R: reflection | `go test ./internal/engine -run 'Schema|Decode|Projection|Clone|Inline|CompareDocument|Callback|Ownership|Independent|Secret'`. Covers descriptors, projection, shared cloning behavior, and ownership. |
|
||
| Final | `./scripts/check.sh` followed by `go test -race ./...`. The first owns formatting, whitespace, ordinary tests/examples/doc links, vet, and build; race validation is appropriate because state and ownership handoffs change throughout this sprint. |
|
||
|
||
Run F after Stage 2, A after Stages 9 and 13, I after Stage 16, and R after
|
||
Stage 18 when buildable. If earlier migrations still prevent compilation, repair
|
||
them at the next integration point and run the deferred group there. Run the
|
||
final checks once on the finished code/documentation; repeat only for subsequent
|
||
changes affecting their evidence. No benchmark campaign, timing threshold,
|
||
traversal-count test, extended fuzz campaign, CI wait, or manual platform check is
|
||
required by this plan. Add narrow behavior tests only for consequential gaps
|
||
identified below that existing tests do not already cover.
|
||
|
||
## Stage 1 — Establish The Starting Map (Complete)
|
||
|
||
**Work:**
|
||
|
||
1. Confirm the revision/worktree and preserve unrelated work. Read the roadmap's
|
||
settled decisions and the selected owners above.
|
||
2. Map the current callers of `selectOptions`, `assembleBaseInput...`, both
|
||
`loadImported...` ladders, `contextsForAcquiredSources`, the selected-overlay
|
||
loaders, and `applyInheritanceScopeTemplates`. Inspect test callers as well
|
||
as production callers; some old entry variants exist mainly for tests.
|
||
3. Identify the existing behavioral owners for repeated resources within/across
|
||
tiers, enabled/disabled version checking, selected/absent searches, ordinary
|
||
versus independent layered overlays, item-local helper scopes, inherited
|
||
identity/history, inline fields, and detached records.
|
||
4. Keep brief working notes or implementation progress in this plan. Reuse
|
||
existing validation evidence where applicable; establish an implicated
|
||
baseline only when needed to distinguish a pre-existing failure. Do not
|
||
generate another broad audit or blanket characterization-test suite.
|
||
|
||
**Finish:** a concrete caller/coverage map and any actual coverage gaps. No
|
||
production change or empty commit is needed for orientation alone.
|
||
|
||
Progress notes (2026-10-25): the worktree was clean at `8ab1568`, immediately
|
||
after the planning baseline `8a99e61`; the audited production baseline
|
||
`5959537` is present. `selectOptions` is called by the ordinary and grouped
|
||
load entry points. The `assembleBaseInput...` wrappers converge on
|
||
`assembleBaseInputTrackedForRole`, reached from ordinary loading, layered main
|
||
loading, and underlay-group acquisition. The tracked import wrappers converge
|
||
on `loadImportedDocumentsTrackedWithProcessing`; production callers are input
|
||
acquisition and underlay-group acquisition, while older convenience variants
|
||
also have direct test callers. `contextsForAcquiredSources` is shared by ordinary
|
||
loading, layered assembly, input acquisition, and overlay application. Selected
|
||
overlay loading is reached from ordinary overlay application and layered input
|
||
assembly. Both inheritance adapters, for ordinary and identified sequences,
|
||
feed `applyInheritanceScopeTemplates`.
|
||
|
||
Existing behavioral owners cover repeated/cyclic resources and cross-tier reuse
|
||
(`internal/engine/imports_test.go`, `facade_equivalence_test.go`); default and
|
||
enabled source-version modes (`internal/engine/source_version_test.go`,
|
||
`facade_source_version_mode_test.go`); selected/absent discovery
|
||
(`internal/engine/discovery_load_test.go`, `facade_discovery_test.go`); ordinary
|
||
and layered overlays (`internal/engine/selected_overlay_test.go`,
|
||
`facade_layered_test.go`); item-local helper scopes (`internal/engine/imports_test.go`,
|
||
`facade_import_lifecycle_test.go`); inheritance identity/history
|
||
(`internal/engine/inheritance_identity_history_test.go`,
|
||
`internal/engine/inheritance_history_test.go`); inline fields
|
||
(`internal/engine/schema_test.go`, `internal/engine/projection_test.go`); and
|
||
detached results and callback ownership (`internal/engine/metadata_test.go`,
|
||
`internal/engine/callback_ownership_test.go`, `facade_diff_test.go`). No specific
|
||
coverage gap surfaced in this map; later focused checks remain assigned in the
|
||
validation groups above.
|
||
|
||
## Stage 2 — Remove Unused Reverse Bridges And Share Real Facade Conversion (Complete)
|
||
|
||
**Owners:** root `conversion.go`, `records_test.go`, `lifecycle.go`, `layered.go`,
|
||
and engine `records.go`.
|
||
|
||
1. Reconfirm callers, then delete `toEngineSourceInfo`, `toEngineOrigin`,
|
||
`toEngineSourceVersionCheck`, and `toEngineImportSelections`, plus
|
||
`SourceInfoWithPlacement`, `SourceInfoWithImportSelections`,
|
||
`importSelectorFromString`, and `importSelectionOutcomeFromString` when their
|
||
only remaining purpose is the deleted chain.
|
||
2. Rewrite the three reverse-round-trip record tests using genuine engine output
|
||
and forward conversion/public inspection. Retain placement, version-evidence,
|
||
selection-evidence, nil/empty, and detached-copy protection. Engine test
|
||
fixtures can use existing loading APIs; do not add production fixture setters.
|
||
3. Extract one root `toEngineOverrides` helper and use it from both option
|
||
converters. Reuse `toEngineInput`, filesystem, path, definition, and default
|
||
conversion where already shared. Preserve slice ownership and nil/empty
|
||
behavior; do not make conversion interpret/validate input values.
|
||
4. Retain reverse diagnostic conversions used by `WriteDiagnostics`, forward
|
||
source/origin inspection conversion, and all public named types.
|
||
|
||
**Finish:** the unnecessary cross-package inverse interface is gone and real
|
||
conversion is shared without a new transport layer. Run F.
|
||
|
||
Progress notes (2026-10-25): removed the unused reverse source-record bridge
|
||
and engine setters, replaced synthetic round-trip fixtures with engine loads,
|
||
and shared override conversion across ordinary and layered options. Validation
|
||
group F passed with `GOTOOLCHAIN=local GOPROXY=off go test . -run
|
||
'SourceInfo|SourceVersionCheck|ImportSelection|Facade|PublicFacadeBoundary|DependencyImportBoundary' -count=1`.
|
||
|
||
## Stage 3 — Normalize Preflight Without Fabricated Options (Complete)
|
||
|
||
**Owners:** engine `options.go`, ordinary entry in `load.go`, grouped entry in
|
||
`load_request.go`, and their option tests.
|
||
|
||
1. Extract non-generic input normalization/validation over `loadInput`, returning
|
||
the normalized input plus diagnostics. Pass the ordinary overlay name only
|
||
where its context requirements apply. Keep discovery-description validation,
|
||
placement checks, normalized destinations, and filesystem option checks at
|
||
their existing pre-acquisition boundary.
|
||
2. Separate load-wide import-default validation, version-rule/mode validation,
|
||
and sequence-registration compilation from per-input checks. Share their
|
||
mechanisms between entries without adding a second interpretation of options.
|
||
3. Keep the public/engine entry arity handling and each entry's existing error
|
||
classification. Ordinary loading still translates its one selected options
|
||
value; layered loading validates main, all underlay inputs, and its independent
|
||
overlay context in the existing meaningful order. Preserve canonical diagnostic
|
||
accumulation and early returns, including invalid options before reads.
|
||
4. Replace the grouped entry's constructed `Options[T]` values with direct
|
||
normalization of `loadInput`; set the effective protection rule from the
|
||
normalized main placement. Leave typed application behavior in the definition.
|
||
5. Remove `importDefaultsValidated` and preflight-only forwarding machinery when
|
||
callers migrate. Update tests through the entry/helper that owns the behavior,
|
||
rather than keeping `selectOptions` solely as a test compatibility surface.
|
||
|
||
**Finish:** both entries produce validated `loadRequest[T]` values without using
|
||
the ordinary options type to validate grouped inputs. Acquisition may still use
|
||
old options until Stage 6; do not broaden this stage into that migration.
|
||
|
||
Progress notes (2026-10-25): added shared normalization over `loadInput`, moved
|
||
load-wide defaults, source-version, and sequence-registration validation into
|
||
shared settings preparation, and removed `selectOptions` plus its validation
|
||
flag from grouped preflight. `GOTOOLCHAIN=local GOPROXY=off go test ./...` passed.
|
||
|
||
## Stage 4 — Express Input Acquisition With Non-Generic Requests (Complete)
|
||
|
||
**Owners:** new engine `acquisition.go` and existing `load_input.go`.
|
||
|
||
1. Define `acquisitionPolicy`, `acquisitionState`, `tierAcquisitionState`,
|
||
`inputAcquisitionRequest`, and `importAcquisitionRequest` as selected above.
|
||
Reuse existing enums, policy records, and accounting; introduce no new public
|
||
controls or duplicate budget counter.
|
||
2. Replace the generic `assembleBaseInputTrackedForRole` implementation with
|
||
`acquireInput`. Translate its references to source, imports, filesystem,
|
||
placement, role, directive context, version rule, policy, and deferred
|
||
assembly into explicit request/policy/state fields.
|
||
3. Keep its phases visible: search selection; parse and failed-source evidence;
|
||
disabled-directive checks; source-local version/directive processing; import
|
||
acquisition or single-input placement; accounting/repetition registration;
|
||
context handoff. Small helpers may isolate version/directive preparation.
|
||
4. Preserve discovery resolution once per input consumption and the acquired
|
||
concrete source on both success and failure. Keep mapping diagnostic
|
||
destination projection and the ordinary-overlay context deferral unchanged.
|
||
5. Retain `assembledInput`'s existing structural/evidence fields. Do not merge
|
||
parsed/local payloads with placed documents or eagerly compose deferred input.
|
||
|
||
**Finish:** the shared input-acquisition implementation no longer depends on
|
||
`Options[T]` or an application type. Call-site and import-worker migration is
|
||
completed by Stages 5–6; temporary non-compilation is acceptable.
|
||
|
||
Progress notes (2026-10-25): introduced the non-generic acquisition request and
|
||
policy/state/tier records, moved the input-acquisition implementation into
|
||
`acquireInput`, and added the resolved-context handoff field. The focused engine
|
||
test command currently cannot compile because the old assembly callers in
|
||
`load_input.go` and `load_request.go` are scheduled to migrate in Stage 6; test
|
||
verification is deferred until that integration.
|
||
|
||
## Stage 5 — Consolidate Import Requests And Default Ownership (Complete)
|
||
|
||
**Owners:** engine import entry points, `importTraversal`, `source_version.go`,
|
||
and tests that directly invoke the old import variants.
|
||
|
||
1. Replace the `loadImportedDocumentsTracked...` ladder with
|
||
`acquireImportedInput` using `importAcquisitionRequest` and the shared
|
||
policy/state/tier owners. Replace the access-based ladder with one worker
|
||
receiving the same request plus explicit access/resolved facts. Keep only
|
||
genuinely distinct convenience entries, not one wrapper per optional field.
|
||
2. Move `importDefaults` out of `sourceVersionState`. Resolve declarations from
|
||
`acquisitionPolicy.importDefaults`, including nested item scopes. Import
|
||
defaults must work independently of whether a version rule/state is active.
|
||
Preserve enabled version processing and its separate local/effective rules.
|
||
3. Make traversal references to aggregate accounting, version evidence, tier
|
||
repetition, and immutable policies point to their existing operation owners.
|
||
Keep active-route tracking, scope switching, and inclusion edges local to
|
||
the traversal. Do not allocate a fresh visited map inside a nested item scope.
|
||
4. Preserve nil version-state behavior used by lower-level operations and the
|
||
existing default processing branch. Replace optional positional/variadic
|
||
policy passing with explicit inputs for meaningful operation distinctions.
|
||
Private test entry spellings do not need compatibility wrappers; preserve
|
||
their relevant behavioral cases through the resulting operation.
|
||
5. Migrate direct import tests to named inputs. Small test-only constructors may
|
||
supply default state/policy values when this reduces repetitive fixtures.
|
||
Keep their behavior assertions intact.
|
||
|
||
**Finish:** the import request surface has named cohesive inputs; defaults no
|
||
longer travel through version processing; no forwarding ladder remains merely
|
||
to supply another nil/default positional parameter.
|
||
|
||
Progress notes (2026-10-25): replaced the import entry ladders with named
|
||
requests and one access worker, moved defaults to acquisition policy, and
|
||
retargeted direct import tests. The focused engine test command cannot compile
|
||
yet because the remaining ordinary and grouped assembly callers are scheduled
|
||
for Stage 6; verification is deferred until that integration. `git diff --check`
|
||
passed.
|
||
|
||
## Stage 6 — Wire Ordinary And Grouped Lifecycle To Acquisition (Complete)
|
||
|
||
**Owners:** engine `load.go`, `load_request.go`, `load_input.go`, related tests.
|
||
|
||
1. Construct the load's acquisition policy and mutable state once in
|
||
`executeLoadRequest`. Keep the definition and schema/callback lifecycle there.
|
||
2. Call `acquireInput` for main using its ordinary/layered directive context,
|
||
main version role, current protection rule, and deferred assembly. Preserve
|
||
the existing schema-build and version-check timing, including the different
|
||
default/enabled source-version paths.
|
||
3. Have `acquireUnderlayGroupScopes` use the same load policy/accounting/version
|
||
state and one fresh tier state per group. Remove `Options[struct{}]`
|
||
construction. Preserve explicit namespace anchors and graph reanchoring.
|
||
4. Main acquisition still precedes underlays. Strict assembly still follows
|
||
the existing dependency-ready scopes and lower-tier identity information.
|
||
Keep group-local repetition separate from load-wide limits, and retain the
|
||
existing selected-overlay duplicate-source checks.
|
||
5. Replace the ordinary `applyInputLayers` options argument with the precise
|
||
main/overlay/protection/policy inputs it uses. A named overlay request will
|
||
complete that migration in Stage 9. Remove remaining production
|
||
`assembleBaseInput...` variants and the acquisition-only `sequencePolicy`,
|
||
`deferAssembly`, and `directiveContext` fields from `Options`. Retain the
|
||
placement, destination, and import-default fields used by facade translation.
|
||
6. Retain real lower-level test/benchmark capabilities with direct request
|
||
construction or test-only adapters. Do not delete meaningful source/limit
|
||
cases to make removed helpers disappear.
|
||
|
||
**Finish:** no executor or underlay caller reconstructs broad/generic options
|
||
for acquisition. The shared path is wired end to end; context and import-flow
|
||
extraction continue in later stages.
|
||
|
||
Progress notes (2026-10-25): `executeLoadRequest` now creates one acquisition
|
||
policy/state and acquires the main input through `acquireInput`; underlay groups
|
||
reuse those load owners with a fresh tier state. Removed the generic base-input
|
||
assembly variants and the acquisition-only fields from `Options`, and adapted
|
||
direct tests to construct named requests. A search-source regression was caught
|
||
and fixed during verification. The focused engine selection, facade selection,
|
||
consumer example tests, and `GOTOOLCHAIN=local GOPROXY=off go test ./...` passed.
|
||
|
||
## Stage 7 — Separate Resolved Filesystem Facts From Access (Complete)
|
||
|
||
**Owners:** new engine `filesystem_context.go`, `filesystem.go`,
|
||
`filesystem_access.go`.
|
||
|
||
1. Add `resolvedFilesystemContext` with the fields selected above. Reuse native
|
||
namespace/token identity; do not compare arbitrary `fs.FS` values or equate
|
||
equal labels with equal namespaces.
|
||
2. Extract host and native fact derivation from `importFilesystem` and
|
||
`nativeFilesystemAccess`. Prefer a selected `SourceInfo.Path` when it already
|
||
supplies canonical source identity. Preserve source-derived file bases,
|
||
explicit in-memory roots/bases, native slash paths, and the root-only absent
|
||
host default.
|
||
3. Keep separate functions for the directory checks and opening access that
|
||
currently occur at each operation. A returned description does not mean
|
||
every possible check has run. Do not add validation-state flags or an
|
||
all-purpose eager resolver with stronger acceptance rules.
|
||
4. Adapt the import worker to consume resolved facts and explicitly own its
|
||
access until the existing acquisition handoff closes it. Keep close-error
|
||
reporting and primary-error precedence unchanged. Native access borrows its
|
||
caller filesystem and must not close it.
|
||
5. Keep any old derivation adapters only while genuine callers migrate in
|
||
Stages 8–9; remove them at the end of that migration.
|
||
|
||
**Finish:** common context facts have one private representation while validation
|
||
timing and live resource ownership remain explicit.
|
||
|
||
Progress notes (2026-10-25): added shared host/native filesystem descriptions,
|
||
separated directory validation and access creation, and migrated import
|
||
acquisition to pass resolved facts alongside its owned access. The prior
|
||
derivation adapters remain for the validator and overlay migration. Focused
|
||
acquisition tests, `GOTOOLCHAIN=local GOPROXY=off go test ./...`, and
|
||
`git diff --check` passed.
|
||
|
||
## Stage 8 — Carry Context Facts Through Input And Passive Views (Complete)
|
||
|
||
**Owners:** engine `load_input.go`, `source_context.go`, `load_request.go`, and
|
||
the ordinary context backfill in `load.go`.
|
||
|
||
1. Add the optional resolved-context field to `assembledInput`. Populate it
|
||
from import acquisition or the existing non-import context validation point.
|
||
When context work is deferred for an ordinary overlay, leave it deferred.
|
||
2. Replace the source-kind derivation branches of `contextsForAcquiredSources`
|
||
with `contextsForResolvedSources`: consume resolved facts and source records,
|
||
construct the existing rooted host/native passive view, and compute each
|
||
source's relative base using the appropriate path rules.
|
||
3. Share relative-base/containment calculation where applicable; retain each
|
||
acquired source's own path and source-ID association. Do not substitute the
|
||
first source record for the selected main file or attach all values to main.
|
||
4. Migrate normal input acquisition, underlay contexts, and ordinary context
|
||
backfill. Resolve only still-missing facts at the same lifecycle point as
|
||
before. Retained descriptions/views never capture `filesystemAccess` or
|
||
`*os.Root`; the existing passive `ResourceContext` contract stays unchanged.
|
||
5. Retain failure evidence, root-only absent host context, discovered native
|
||
outcomes, and deferred search-base behavior. Update focused context tests
|
||
only where private function signatures changed.
|
||
|
||
**Finish:** ordinary/underlay acquisition and passive views use the common facts;
|
||
selected-overlay callers are the remaining migration in Stage 9.
|
||
|
||
Progress notes (2026-10-25): input acquisition now retains resolved filesystem
|
||
facts and builds passive contexts from those facts for ordinary and underlay
|
||
sources. Migrated the ordinary backfill and reordered-source context test. The
|
||
selected-overlay adapters remain for Stage 9. Focused context/acquisition tests,
|
||
`GOTOOLCHAIN=local GOPROXY=off go test ./...`, and `git diff --check` passed.
|
||
|
||
## Stage 9 — Migrate Selected Overlays Without Merging Their Semantics
|
||
|
||
**Owners:** engine `selected_overlay.go`, ordinary `applyInputLayers`, and
|
||
`assembleLayeredInput`.
|
||
|
||
1. Define a private `selectedOverlayRequest` containing the overlay name,
|
||
resolved filesystem description, protection rule, sequence policy, and
|
||
existing limits/version state. Derive context at the caller's existing
|
||
lifecycle point. Pass assembly and source evidence explicitly to the worker.
|
||
2. Route ordinary overlays through the main source's context; route layered
|
||
overlays through their independent `OverlayFilesystem`. Use named operation
|
||
inputs instead of fabricating a `sourceBytes`/empty parsed-main pair merely
|
||
to select a context branch. Preserve the actual absent-host root behavior.
|
||
3. Return or retain the resolved description used by overlay acquisition and
|
||
build overlay passive context from it. Reuse main facts only when that
|
||
ordinary operation previously used the same root. Context facts do not share
|
||
a live handle between the main import traversal and the later overlay read.
|
||
4. Keep the production collection-aware overlay path. Migrate meaningful
|
||
lower-level tests to that path with explicit policy/limits, retaining their
|
||
budget, version, and failure assertions. Delete the test-only legacy branch
|
||
selected by missing variadic policy and its exclusive adapters, rather than
|
||
treating optional argument count as a contract. The baseline production entry
|
||
points all supply the policy argument.
|
||
5. Keep overlay source IDs, duplicate detection, version protection, directory
|
||
validation, read/close errors, aggregate accounting, and post-read application
|
||
order. Remove superseded context derivation and overlay-forwarding adapters.
|
||
|
||
**Finish:** all acquisition/context callers share facts and named inputs while
|
||
ordinary and independent layered overlay semantics remain distinct. Run A when
|
||
buildable; carry any genuine remaining migration dependencies to the next
|
||
integration opportunity instead of adding scaffolding or stopping the sprint.
|
||
|
||
## Stage 10 — Give Import Responsibilities Focused Files
|
||
|
||
**Owners:** engine `imports.go` and new `import_acquisition.go`,
|
||
`import_scope.go`, `import_projection.go`.
|
||
|
||
1. Put the request entry/access worker and traversal admission/orchestration in
|
||
`import_acquisition.go`. Keep `imports.go` as the small common owner for
|
||
limits, accounting/route/traversal data, and shared import diagnostics.
|
||
2. Move `composeAcquiredInputWithPolicies`, `composeAcquiredScopeWithPolicy`,
|
||
pending-item composition, placement reservation, and their item-scope
|
||
assembly helpers into `import_scope.go`.
|
||
3. Move nested-origin/policy/evidence projection, acquired source destinations,
|
||
version-evidence projection, sequence-prefix helpers, and diagnostic-path
|
||
projection into `import_projection.go`. Scheduling and graph identity remain
|
||
with their existing graph/scheduler owners.
|
||
4. Make the acquisition-to-closed-resources-to-assembly handoff visible. Keep
|
||
evidence projection based on existing composition applications/position maps;
|
||
file movement does not authorize another merge, full-tree replay, or scan of
|
||
every unrelated policy for each record.
|
||
|
||
**Finish:** maintainers can locate acquisition, local strict assembly, and evidence
|
||
work separately. Keep this stage structural; parse/admission extraction follows.
|
||
|
||
## Stage 11 — Extract Import Declaration Parsing Operations
|
||
|
||
**Owner:** engine `import_spec.go`.
|
||
|
||
1. Retain the outer mapping check, exactly-one-selector check, source-order field
|
||
walk, applicability checks, and final selector parsing in
|
||
`parseImportDeclarationSpec`.
|
||
2. Extract boolean assignment, wildcard-set parsing/validation (shared by
|
||
`pattern` and `exclude`), placement parsing, and selector-value parsing into
|
||
ordinary private helpers. Reuse existing `parseStringSet`, authored-member
|
||
sites, and destination/depth parsers.
|
||
3. Preserve unspecified pointers and set flags, especially nil versus empty
|
||
pattern lists. Keep error paths/locations tied to the authored selector or
|
||
list member. Do not reorder consequential checks or replace the selector
|
||
matrix with a configurable parser registry.
|
||
|
||
**Finish:** field decoding is readable without changing declaration syntax,
|
||
selector rules, default resolution, or diagnostics. Keep behavior tests as the
|
||
owner; no helper-by-helper assertion inventory is needed.
|
||
|
||
## Stage 12 — Extract Directory Entry Admission And Descent
|
||
|
||
**Owner:** engine `import_selection.go`, using existing directory/link helpers.
|
||
|
||
1. Keep root normalization/deduplication, root ordering, the explicit traversal
|
||
stack, candidate union, sorting, exclusion, and required-empty decision visible
|
||
in `selectDirectoryImportCandidates`.
|
||
2. Extract coherent entry inspection and descent preparation. Return a small
|
||
result identifying skip, admitted file, or directory frame to enter, plus a
|
||
diagnostic when appropriate. Keep stack/active-directory mutation with the
|
||
traversal so its lifetime is visible; do not invent a general visitor API.
|
||
3. Preserve hidden/follow/depth behavior, distinct lexical candidates versus
|
||
canonical access identity, resource-entry accounting, root-error handling,
|
||
and active-path cycle detection. A later non-active alias remains eligible;
|
||
exclusions still do not suppress discovery errors.
|
||
4. Keep globs and first-match main-file discovery on their existing mechanisms.
|
||
Share a helper only where its existing admission semantics actually coincide.
|
||
|
||
**Finish:** substantial entry/descent policy is separated from stack orchestration
|
||
without changing the selection algorithm or adding repeated root walks.
|
||
|
||
## Stage 13 — Extract Imported-Source Admission From Recursion
|
||
|
||
**Owner:** engine `import_acquisition.go` after Stage 10.
|
||
|
||
1. Keep `importTraversal.visitPath` as the visible coordinator of containment,
|
||
canonical identity, cycle/repetition checks, read/parse, directive/version
|
||
processing, placement, and recursive declaration traversal.
|
||
2. Extract source admission/read-and-process results into a private
|
||
`admittedImport` carrying canonical resource/node identity, source record,
|
||
processed payload/declarations, and processing evidence. Return an explicit
|
||
optional-missing outcome; do not represent a read failure as absence.
|
||
3. Preserve when source records are appended, rolled back for optional missing
|
||
input, and retained for failures; preserve source IDs, accounting increments,
|
||
version-callback ordering, and inclusion-edge creation on repetition/cycles.
|
||
4. Pass the admitted record to mapping/item placement helpers rather than
|
||
repeating its payload, declarations, and processing evidence as parallel
|
||
arguments to `visitItemSource`.
|
||
5. Factor route push/pop and item-scope save/restore only where it makes their
|
||
shared lifetime clearer. Keep active graph state across item scopes and
|
||
reserve placement nodes once. Item helpers complete before outer placement;
|
||
they are strict local peers, not a precedence layer.
|
||
|
||
**Finish:** admission decisions and recursion have clear handoffs, with one
|
||
read/process path and the original scheduling/evidence algorithms. Run A again
|
||
because Stages 10–13 changed the import implementation after the first checkpoint.
|
||
|
||
## Stage 14 — Extract Inheritance Child And Parent Preparation
|
||
|
||
**Owners:** engine `inheritance_schedule.go`, new `inheritance_prepare.go`, and
|
||
existing scope/policy helpers.
|
||
|
||
1. Extract preparation of the child's directive-free payload, direct origins,
|
||
local policy templates, and optional control/control-origin evidence.
|
||
2. Extract parent preparation: clone/project policies, lower them against the
|
||
child's claims, exclude parent identity from the inherited payload, and
|
||
prepare the separate parent history snapshot where currently required.
|
||
Keep effective policy templates and transfer construction available to the
|
||
combination result rather than reconstructing them afterward.
|
||
3. Use focused prepared-child/prepared-parent records of these related values.
|
||
Borrow authored/resolved parent inputs read-only; retain existing clone points
|
||
for mutable payloads and history. Keep the current preliminary lower-bound
|
||
budget decisions at the executor's existing point before materialization.
|
||
4. Keep mapping and identified-sequence adapters, the stable Kahn scheduler, and
|
||
the parentless/cancelled path semantics intact. Do not flatten nested directives
|
||
before their destination-local evaluation.
|
||
|
||
**Finish:** the executor no longer embeds child/control and parent/policy
|
||
preparation. Actual recursive combination/evidence moves in Stage 15.
|
||
|
||
## Stage 15 — Return One Inheritance Combination Result
|
||
|
||
**Owners:** new engine `inheritance_combine.go`, `inheritance_collection.go`,
|
||
and the shared executor.
|
||
|
||
1. Add `inheritanceTemplateResult` as described above and extract the existing
|
||
parent/child combination into a helper receiving prepared inputs and the
|
||
application policy. Parentless templates return the same result shape.
|
||
2. Keep `mergeRecursiveMappings` and `combineSequence` as the combination
|
||
mechanisms. Their callbacks update local origins/policy and local replacement
|
||
evidence. Have `applyInheritanceSequenceOrigins` return its replacement slice
|
||
instead of taking the whole enclosing scope result; append that slice to the
|
||
local template result while retaining its updates to the supplied origin maps.
|
||
3. Derive sequence history, transferred origins, retired/replaced policies,
|
||
identity preservation, and the transfer operation from that same combination.
|
||
Preserve `historyParent` and all policy/identity fields on the existing
|
||
`inheritanceOperation`; do not rerun a merge or project values to recover them.
|
||
4. Keep sequence validation, diagnostic canonicalization, location assignment,
|
||
replacement order, and no-parent cloning behavior at their established
|
||
semantic points. Keep child identity absent from inherited-parent history but
|
||
present with its own origin in the effective child.
|
||
|
||
**Finish:** merge callbacks no longer mutate the executor accumulator. The
|
||
resolved template and all evidence travel together for Stage 16 integration.
|
||
|
||
## Stage 16 — Finish The Inheritance Executor And Accounting
|
||
|
||
**Owner:** engine `inheritance_schedule.go` and focused finalization helpers.
|
||
|
||
1. Reduce the executor to preparation, budget precheck, combination, accumulation,
|
||
and final rebuild. Append each template's control before its transfer/history
|
||
in the same order as before, and update the indexed resolved value/origin/
|
||
policy/template state once per template.
|
||
2. Extract retained-history charging for one template's returned operations and
|
||
replacements, reusing the same `retainedNodeCounter` across the scope as
|
||
before. Preserve pointer-based shared-node accounting, lower-bound checks,
|
||
scope-slot overhead, final expansion check, limits, and diagnostics.
|
||
3. Extract final rebuilding and scope policy/origin projection where it clarifies
|
||
the coordinator. Reuse `newInheritancePolicyIndex` once for scope localization;
|
||
do not return to scanning all policy entries for every slot.
|
||
4. Confirm both adapters and nested walking consume the same output. Retain
|
||
source-version movement, sensitivity, independent snapshots, assembled order,
|
||
cancellation, and policy retirement on actual replacement.
|
||
|
||
**Finish:** one readable executor coordinates explicit value/policy/evidence
|
||
handoffs, with no parallel inheritance implementation. Run I.
|
||
|
||
## Stage 17 — Extract Substantial Projection Handlers
|
||
|
||
**Owners:** engine `projection.go`, optionally new `projection_collection.go`.
|
||
|
||
1. Keep `projectConfiguration`'s schema/type guard and one `valueOperation`.
|
||
Keep `project`'s depth check, pointer/dynamic dispatch, and scalar conversion
|
||
choices visible; delegate structs, maps, and sequences to named methods.
|
||
2. Move struct member/inline flattening, collision checks, mapping-key charging,
|
||
and flattened-field/inline-map secret classification into `projectStruct`.
|
||
Preserve declaration order and the current decrement for a flattened
|
||
container that is not retained in the output.
|
||
3. Move deterministic sorted-map traversal into `projectMap`; keep its cycle
|
||
entry active until all recursive children complete. Move slice/array
|
||
construction into `projectSequence`, preserving nil-slice handling and the
|
||
slice-only cycle guard.
|
||
4. All recursive children call the same operation's dispatcher. Avoid duplicate
|
||
node charging/depth increments or early cycle release introduced by extraction.
|
||
Preserve float/text/duration/secret scalar handling and error paths.
|
||
|
||
**Finish:** dispatch and substantial handlers are separately readable, using the
|
||
original operation state. Do not refactor `clone`, `decodeValue`, or
|
||
`valueBuilder.build` as an additional cleanup project.
|
||
|
||
## Stage 18 — Separate Schema Field Discovery And Assembly
|
||
|
||
**Owner:** engine `schema.go`, optionally new `schema_fields.go`.
|
||
|
||
1. Keep `describeFields` as declaration-order enumeration, exported/ignored/tag
|
||
decisions, and dispatch to ordinary or inline handling.
|
||
2. Extract inline pointer-chain inspection and inline-struct/map assembly,
|
||
retaining recursion/building-state checks and the existing codec-versus-
|
||
structural distinction. Reuse the one builder/cache.
|
||
3. Make ordinary registration and inline flattening visibly maintain both
|
||
`members` and flattened `fields`/`fieldByName`/`inlineMap`. Preserve index
|
||
paths, nested secret propagation, ambiguity paths, and the single-inline-map
|
||
rule. Share registration helpers only for genuinely identical updates.
|
||
4. Preserve descriptor reuse and the physical-member view used by typed cloning.
|
||
Do not unify these views or add alternate field/tag policy. Existing schema,
|
||
projection, decoding, and cloning expectations remain the behavior baseline.
|
||
|
||
**Finish:** field discovery and descriptor assembly have explicit responsibilities
|
||
without a new registration framework. Run R for both Stages 17–18.
|
||
|
||
## Stage 19 — Update Consumer And Internal Navigation
|
||
|
||
**Owners:** `docs/consumers/README.md`, `docs/internal/overview.md`,
|
||
`docs/internal/imports.md`, and affected links/test-owner entries in
|
||
`docs/development.md`.
|
||
|
||
1. Add a concise consumer choice guide: ordinary `Load` for one selected source,
|
||
explicit discovery as a source usable by either entry, and `LoadLayered` for
|
||
underlay groups. Distinguish input-owned placement/import/filesystem choices
|
||
from load-wide definition/default/version/override choices and independent
|
||
layered overlay context. Link current API/config owners for exact rules.
|
||
2. Link the existing simple README example, bundled-file/layered examples,
|
||
import-focused prompt catalog, and discovered sequence catalog. Keep the
|
||
README's simple path simple; do not duplicate complete examples or rewrite
|
||
consumer domain rules to showcase new internal types.
|
||
3. Update implementation ownership and flow for normalized requests, acquisition
|
||
policy/state/tier lifetimes, resolved context versus live handles, import file
|
||
owners, and explicit inheritance/projection/schema handoffs. Fix moved-symbol
|
||
and behavioral-test navigation and stale comments affected by the refactor.
|
||
4. Preserve public contract definitions and accepted ADR decisions. Keep deferred
|
||
clone work in `future.md`. No policy change or new ADR is needed for the
|
||
selected private refactor.
|
||
|
||
**Finish:** current documentation describes implemented structure, with the
|
||
roadmap/plan retaining future work and progress until release cleanup.
|
||
|
||
## Stage 20 — Verify And Close The Combined Sprint
|
||
|
||
1. Search for obsolete import/input forwarding ladders, synthetic `Options`
|
||
acquisition construction, `sourceVersionState.importDefaults`, discarded
|
||
context derivation branches, unused inverse bridges, and temporary scaffolds.
|
||
Remove leftovers along with exclusive test scaffolding. Preserve distinct
|
||
operations and real diagnostic conversions; do not delete private code solely
|
||
because it resembles another helper.
|
||
2. Review the combined call flow: one normalization model, non-generic acquisition,
|
||
load versus tier versus traversal state, library-owned acquisition handles
|
||
closed before composition/publication, one combination outcome for inheritance
|
||
evidence, and one state owner per reflective operation. Review for added whole-tree
|
||
traversals, policy scans, or ownership copies without requiring measurements.
|
||
3. Account for every selected finding using the table below. Confirm any new
|
||
handoff coverage addresses a concrete gap. Existing correctness/algorithmic
|
||
regressions, public-boundary tests, and meaningful assertions stay effective.
|
||
4. Run the Final validation group, resolve failures, inspect the staged diff and
|
||
workspace, and commit the completed remaining work with plain-English
|
||
messages. Report changes, checks, limitations, and any unrelated work retained.
|
||
Do not tag, push, publish, or update consumer repositories in this sprint.
|
||
|
||
| Audit finding | Completion evidence |
|
||
| --- | --- |
|
||
| 1: imports | Stages 3–6 and 10–13 remove options reconstruction/default coupling and forwarding ladders; acquisition/scope/evidence owners and admission operations are clear. A and final checks pass. |
|
||
| 2: inheritance | Stages 14–16 return explicit template results, keep one executor, and preserve accounting and value/history/identity agreement. I and final checks pass. |
|
||
| 3: projection | Stage 17 extracts substantial handlers sharing operation state. R and final checks pass. Clone review remains deferred, not implicitly completed. |
|
||
| 4: schema | Stage 18 clarifies ordinary/inline assembly while preserving both descriptor views. R and final checks pass. |
|
||
| 5: contexts | Stages 7–9 share resolved facts across input/import/overlay/passive contexts without retaining handles or changing validation timing. A and final checks pass. |
|
||
| 6: facade | Stage 2 removes unused inverse conversions/adapters and shares real conversion. F and final boundary/consumer checks pass. |
|
||
|
||
**Finish:** the selected roadmap target is implemented, validated, documented,
|
||
and reviewable as one complete sprint. No unresolved product/scope decisions are
|
||
needed to begin these stages.
|