Plan sibling inheritance implementation

docs/roadmap/implementation.md

@@ -0,0 +1,288 @@
+# Sibling Inheritance Implementation Plan
+
+## Scope and working rules
+
+Implement the agreed [named-inheritance roadmap](named-inheritance.md), starting
+from `01e38b5`. The roadmap owns feature policy; this document owns execution and
+the concrete implementation choices below. Follow stages in order. Each stage is
+a focused task for a gpt-5.6-terra coding agent and should end with relevant
+passing checks and a plain-English commit. Do not publish or tag a release during
+this sprint.
+
+Read `docs/development.md`, `docs/policy/agents.md`, and task-relevant policies.
+Preserve unrelated work. The root owns public types; the private engine owns
+processing. Reuse the parser-independent document representation and existing
+helpers. Do not add dependencies, configurable keywords, registration APIs,
+merge strategies, background work, or consumer-specific semantics.
+
+Stages 1–6 develop private machinery without activating inheritance in public
+loading. Stage 7 connects the complete pipeline. Intermediate commits must not
+publish resolved values with incomplete provenance or redaction. Private names
+and file splits below are guidance, not new public contracts.
+
+## Decisions carried into implementation
+
+- `extends` is reserved input syntax. A string names an exact sibling mapping;
+  null cancels a relationship; omission retains lower contributions. Other types,
+  missing/non-mapping parents, self-reference, and cycles are errors. Empty strings
+  are exact names. Root and direct sequence-element mappings cannot use string
+  references; null needs no lookup.
+- Strict composition still rejects overlapping non-mapping contributions,
+  including duplicate `extends` definitions. Complete all source-tier and CLI
+  precedence before expanding the effective input. Overwritten earlier references
+  are historical evidence, not active relationships to validate.
+- Expand enclosing parent chains before resolving nested relationships in each
+  destination. Reusable parent templates retain unresolved nested instructions.
+  Sort mapping keys lexicographically, follow parent dependencies as needed, and
+  visit sequences by index. Sorting never determines value precedence.
+- Parent-to-child combination uses recursive overlay, including explicit
+  zero/false/empty/null replacement and whole-sequence replacement. Consume the
+  current mapping's own relationship during enclosing expansion; nested
+  relationships travel with nested payload until their scope is resolved.
+- Inherited values count as explicit input for presence/defaulting queries.
+  Provenance distinguishes direct input from inherited input.
+- Version checking stays at its existing main-input point. Expansion follows
+  source assembly and precedes defaults/decoding, recorded resolution, secret
+  lookup, validation, and publication. Invalid effective relationships fail even
+  in entries the application does not select for execution.
+- Preserve source records and passive resource contexts. Inheritance is neither
+  a new acquired source nor another precedence layer.
+
+### Concrete metadata representation
+
+Extend engine and root `Origin` with `Inheritance []InheritanceStep`. Define
+root-owned and engine counterpart records with `From Path`, `To Path`,
+`Reference Path`, and `Locations []Location`. A step describes a payload-path
+transfer. `Reference` is the destination's controlling `extends` path; `Locations`
+identifies that directive's contributing source coordinates. Paths are concrete
+semantic paths, not names requiring another lookup. Records contain no values.
+
+Keep `Origin.Layer` and `Origin.Locations` for actual value contributors. Order
+steps by derivation; composite origins retain contributing branches rather than
+inventing a single chain. Project concrete descendant paths when queried. Keep
+original contributor paths privately alongside locations, so synthetic sources
+do not depend on file coordinates to identify original semantic paths. Direct
+replacement keeps its own effective origin; displaced inheritance stays in history.
+
+Add inspection-only `ChangeInheritance = "inheritance"`, like `ChangeDecoding`;
+do not accept it through `Resolution.Record`. Inheritance events identify
+parent/control dependencies through existing `Change`, `ValueChange`, and
+`Explanation` surfaces. No separate public query family is needed. Effective
+inherited origins retain source layers and transfer evidence; expansion history
+events use the new operation kind.
+
+### Budgets and diagnostics
+
+Keep acquired-input accounting unchanged. Use `defaultProjectionLimits()` for
+expanded tree depth/nodes, counted by logical materialized nodes rather than
+unique pointers; report `CodeSchemaProjectionLimit`. Check during materialization
+before uncontrolled allocation and typed decoding. Do not charge copies as
+acquired input or add a chain-length cap.
+
+Extend `historyNodesWithin` to retained inheritance templates and snapshots,
+counting shared immutable document nodes once under the existing `historyNodeLimit`;
+report `CodeExplainLimitExceeded`. Compact relationship records refer to shared
+evidence instead of copying a full chain/tree per field. Do not redefine node
+limits as byte limits or introduce a new public limit. Check retained growth as
+it is accumulated and release temporary templates. Use private small test budgets.
+
+Add mirrored engine/root codes: `CodeInheritanceInvalid` (`inheritance.invalid`)
+for invalid type/position or non-mapping parent; `CodeInheritanceMissing`
+(`inheritance.missing`); `CodeInheritanceCycle` (`inheritance.cycle`); and
+`CodeInheritanceVersion` (`inheritance.version`). Use directive paths/locations
+as primary evidence and related definition locations for dependencies/cycles.
+Messages must not interpolate raw reference strings or payload. Existing syntax,
+composition, decode, and budget codes retain their meanings.
+
+## Stage 1 — Prepare shared recursive combination
+
+Inspect `internal/engine/overlay.go`, `compose.go`, `mapping_index.go`,
+`history.go`, and `internal/document/document.go`. `applyOverlay` already shares
+`mergeOverlayMappings` across underlays, main-over-underlay assembly, and selected
+overlays. Strict composition uses `mergeDisjointMappings` after claim validation;
+history replay has a separate value-only traversal.
+
+Extract or adapt the smallest private recursive-overlay primitive needed for
+arbitrary mapping roots, preserving ordered entries and indexed lookup. Keep
+source-tier bookkeeping in its adapter. Provide path/replacement information for
+inheritance's later adapter; do not model inheritance as a fake layer. Reuse
+`document.Clone` and `mappingIndex`. Keep strict composition distinct. Consolidate
+value-only replay only if doing so preserves its behavior without extra machinery;
+do not build a public or general-purpose merge framework.
+
+Validation: existing compose/overlay/override and underlay tests, plus a compact
+table covering recursion, atomic sequence/scalar/null replacement, empty mappings,
+and detached ownership. Existing behavioral expectations should not change.
+Run `./scripts/check.sh` before committing.
+
+## Stage 2 — Expand one sibling scope
+
+Add a private inheritance resolver, callable only from engine tests initially.
+Index effective sibling definitions. Resolve single-parent dependencies with
+visiting/resolved states and memoized enclosing-chain templates. Use an explicit
+work stack so graph chain length is not tied to the Go call stack. Overlay parent
+payload with child payload using Stage 1. Consume the current mapping's own
+`extends`, leaving nested instructions intact. Handle null before lookup.
+
+Implement internal diagnostic codes and deterministic traversal. Preserve
+locations and collect transfer/control evidence for Stage 4. Keep original input
+unchanged and templates private; a child must not mutate its parent. Failed
+expansion returns diagnostics and no usable expanded result.
+
+Validation: forward/backward references, chains, shared parents, child replacement,
+null cancellation, exact unusual/empty names, missing/non-mapping parents, invalid
+types, self/long cycles, and input independence. Assert results and diagnostic
+fields, not stack layout or exact prose. Run focused tests and the everyday checker.
+
+## Stage 3 — Resolve nested destination scopes and bound expansion
+
+Extend to the complete effective tree. Distinguish enclosing-chain templates from
+fully resolved destination subtrees. After materializing an enclosing mapping,
+resolve its nested sibling scopes independently. Visit sequence-contained mappings;
+reject string references on root/direct sequence elements without ancestor lookup.
+
+Scope memoization to each effective destination namespace. Copying a template must
+not copy its already-resolved nested children. Preserve inherited nested control
+evidence and support cancellation through a child's nested null directive. Whole-
+container replacement removes those active relationships at that destination;
+the parent's own surviving relationships still require validation.
+
+Apply the selected expanded-tree budgets, counting logical output from shared
+templates and releasing transient state. No global caches or unrelated-scope lookup.
+
+Validation: the roadmap's nested `primary`/`secondary` example; three-generation
+enclosing chains; nested reference replacement/cancellation; destination-only
+cycles; scopes inside lists; root/list-element errors; and parent names sorting
+before or after children. Exercise small node/depth budgets at and beyond their
+boundaries. Run the everyday checker.
+
+## Stage 4 — Preserve derivation, presence, and resource origins
+
+Implement the selected metadata representation privately in the engine. Carry
+original semantic contributor paths, locations/layers, transfer edges, and control
+evidence through copies and recursive combination. Preserve composite origins and
+sequence-descendant origins, including atomic sequence contributions. Do not infer
+ownership from the last matching layer: multiple sources can share a layer.
+
+Return an expanded tree, origins, and inheritance evidence for the decoded metadata
+pipeline. Presence comes from expanded input. Direct replacement removes effective
+inheritance at replaced paths but retains displaced evidence. Extend cloning,
+comparison/deduplication, and private carriers; share immutable internal evidence
+while detaching query outputs.
+
+Validation: inherited zero/false/null presence, chains and nested transfers,
+composites, same-layer multiple sources, and replacement evidence. Demonstrate
+that an inherited relative resource points to its original source context rather
+than the child's directory. Inheritance does not open referenced resources.
+Run the everyday checker.
+
+## Stage 5 — Integrate historical inspection
+
+Extend `history.go`, `load_metadata.go`, `explain.go`, and explanation rendering.
+Retain original source applications; represent structural expansion separately,
+between source replay and canonical decoding, followed by recorded application
+resolution. Do not attribute inherited values or consumed directives to decoding.
+
+Destination history must show parent contribution and child replacement when both
+exist, even if expansion leaves the effective child value unchanged. Include
+derivation dependencies, preserve source history at original paths, and make
+chains navigable through concrete paths. Do not fabricate repeated chronological
+source loading at every child. Explain consumed directives as historical control
+input absent from effective metadata/configuration. Preserve cancellation and
+superseded directive evidence.
+
+Extend historical path enumeration, ancestor replacement, origin comparison, and
+retained-node accounting. Keep typed reconciliation separate. Share scope evidence
+and project relevant changes instead of storing a full snapshot per leaf.
+
+Validation: inherited values, equal child replacements, displaced inherited
+subtrees, parent source changes, consumed/canceled directives, destination-specific
+nested results, and subsequent application-derived replacement. Check agreement
+among final values, `History`, `Explain`, and historical paths. Exercise small
+history budgets. Run the everyday checker.
+
+## Stage 6 — Preserve sensitivity through inheritance
+
+Connect transfer dependencies to existing secret classification and historical
+redaction. Classify both source and destination paths under schema and
+`Definition.Secret` rules; propagate sensitivity through chains/nested copies.
+Include displaced evidence and control history. Preserve existing conservative
+historical redaction without declaring unrelated siblings secret.
+
+Copy unresolved secret references only; lookup remains at the existing lifecycle
+point. Credentials never enter expansion trees/history. Independent replacements
+retain their own effective provenance while previous secrets remain redacted.
+Queries and renderers must agree.
+
+Validation: source-only and destination-only classification, shared parents,
+transitive inheritance, replacement/removal, and lookup after expansion. Use
+disposable canaries across affected inspection/failure surfaces. Avoid real
+credentials and parallel environment mutation. Run focused secret/history tests
+and the everyday checker.
+
+## Stage 7 — Activate the common lifecycle and facade metadata
+
+Insert expansion in `executeLoadRequest` after ordinary `applyInputLayers` or
+layered `assembleLayeredInput`, before `decodeLoadInput` invokes defaults/decoding.
+Include plain single-source loads, creating the minimal composition carrier needed
+to preserve original history. Update `load_result.go` and inspection carriers with
+Stages 4–6 evidence. Preserve no-input behavior and use a no-directive fast path
+where practical. Do not repeat acquisition, version callbacks, or user resolution.
+
+Keep the version path main-owned. Detect inherited contributions, replacement, or
+directive removal affecting it, including protected ancestor replacement; report
+`CodeInheritanceVersion`. Do not reject an entire enclosing mapping merely because
+it contains an unchanged version field. A version rule targeting an `extends`
+field cannot silently lose its checked declaration when the directive is consumed.
+Keep the checker at its original point and retain accepted version representations.
+
+Mirror records/codes in root `records.go`, conversions, clones, and API comments.
+Ensure root `Origin.Inheritance` and `InheritanceStep` expose only root types and
+detached slices/paths. Keep `ChangeInheritance` inspection-only. Update root/engine
+`IsExplicit` comments, including resolver-entry queries. Preserve `SourceContext`.
+
+Do not treat `extends` as the acquisition-time `configkit` directive. Imports,
+selected overlays, Mapping input, and CLI overrides carry it through combination
+until expansion. Keep existing `configkit` validation and CLI path restrictions.
+Typed defaults and values added by application resolution do not initiate another
+structural pass; inheritance processes assembled source input only.
+
+Validation: public ordinary/layered loads; imports and strict conflicts; multiple
+underlays; selected overlay and CLI changes to parent/child/reference; resolver
+defaulting using inherited presence; version ownership and callback timing;
+typed unknown-field behavior after consumption; no later hooks on expansion
+failure; partial bases with normal validation; and no-directive regressions.
+Include a filesystem resource-origin example and mapping/bytes equivalence case,
+not a full duplication of every mechanism test. Run `./scripts/check.sh` and
+`go test -race ./...` for ownership changes.
+
+## Stage 8 — Validate scaling and complete documentation
+
+Add focused benchmarks beside assembly/underlay benchmarks: long chains, wide
+shared-parent scopes, nested destination expansion, and replacements. Evaluate
+allocations/work against input, materialized output, and retained history,
+including sorting costs. Fix repeated ancestor work or whole-tree copying without
+changing semantics. No unconditional O(n) promise or timing release gate.
+
+Add a maintained consumer example showing one-load inheritance, explicit false,
+nested destination behavior, and source/derivation inspection. Update
+`docs/config.md` as behavior owner, root comments as API owners, README/consumer
+guidance as summaries, and internal overview/development test map as navigation.
+Update architecture/deferred-scope descriptions for implemented inheritance.
+Explain reserved-key migration, null cancellation, no registration, sibling lookup,
+source-first ordering, presence versus provenance, and unchanged version/lifecycle
+policy. No release-policy or platform-gate changes are part of this feature.
+
+Write the next available ADR (currently 0009) recording ConfigKit's chosen
+architecture and rationale. Keep downstream-specific discussion out; rejected
+choices belong only in Alternatives Considered. Do not edit accepted ADR decision
+bodies or silently accept earlier drafts. Mark the feature roadmap's implementation
+status accurately and link the implemented contract. Release notes/tagging remain
+separate release-preparation work.
+
+Run the everyday checker, reuse race evidence if ownership code is unchanged, and
+run benchmarks for evidence without pass/fail thresholds. Review the final diff
+against the roadmap, source equivalence, confidentiality, and public boundary.
+Confirm no independent merge rule, registration API, extra release gate, or
+background/resource lifecycle was introduced. Commit and report validation and
+any remaining limitations.

docs/roadmap/named-inheritance.md

@@ -1,7 +1,7 @@
 # Built-In Sibling Configuration Inheritance
 
 Status: agreed feature design; not implemented. This document defines the intended
-end state. A separate implementation plan will define execution steps.
+end state. The [implementation plan](implementation.md) defines execution steps.
 
 ## Purpose and priorities
 
@@ -275,9 +275,9 @@ output than the source declarations. Use representative chains, shared-base tree
 wide collections, and replacement cases to evaluate growth. Performance evidence
 is not a new release timing gate.
 
-Existing acquired-input accounting remains in force. The eventual implementation
+Existing acquired-input accounting remains in force. Expanded output and retained
-plan must define how expanded output and retained derivation fit existing decoded
+derivation respect existing decoded tree/history limits; the implementation plan
-tree/history limits, including where expansion is checked. Avoid silently charging
+defines accounting and check placement. Avoid silently charging
 one inherited value as a newly acquired source or inventing arbitrary entry/chain
 caps. Any necessary new public limit requires an explicit design decision.
 
@@ -302,7 +302,7 @@ A new ADR should record the selected inheritance architecture.
 Architecture/deferred-scope documentation must recognize
 the new structural responsibility; public API, configuration, consumer, and
 internal documentation should describe implemented behavior when it lands.
-Implementation sequencing belongs in a separate future plan. Reserving `extends`
+Implementation sequencing belongs in the implementation plan. Reserving `extends`
 changes behavior for applications that currently use it as ordinary data. Document
 that compatibility change and the required migration rather than describing this
 as a purely additive opt-in feature. Configurations without `extends` retain their