Revise inheritance roadmap around built-in sibling references

docs/roadmap/deferred.md

@@ -28,7 +28,7 @@ Applications may build their own validate, effective-value, source, bootstrap, a
 explanation commands around the library. Broadening these boundaries requires
 concrete consumer evidence, compatibility review, and an explicit design change.
 
-Opt-in inheritance between named configuration entries is proposed in the
+Built-in sibling inheritance using `extends` is proposed in the
 [named-inheritance roadmap](named-inheritance.md), based on the
 [downstream request](feature-request-promptkit-inheritance.md). It is not yet
 implemented and is distinct from inheritance between selected overlay files.

docs/roadmap/named-inheritance.md

@@ -1,4 +1,4 @@
-# Opt-In Named Configuration Inheritance
+# Built-In Sibling Configuration Inheritance
 
 Status: proposed feature design for agreement; not implemented. This document
 defines the intended end state and the remaining architectural decisions. A
@@ -6,9 +6,10 @@ separate implementation plan will define execution steps after design agreement.
 
 ## Purpose and priorities
 
-Allow an application to designate a collection of named configuration entries
-whose members can inherit settings from one parent in that collection. ConfigKit
-resolves the relationships during loading and returns final typed configuration
+Allow a configuration mapping to inherit settings from one sibling mapping by
+using the built-in `extends` keyword. No collection registration, configurable
+reference-field name, or inheritance field in application Go types is needed.
+ConfigKit resolves the relationships during loading and returns final typed configuration
 with inspectable provenance. Applications retain ownership of field meanings,
 domain validation, and execution.
 
@@ -50,35 +51,47 @@ complete-on-return design in [draft ADR 0008](../adr/0008-complete-on-return-loa
 
 ## Target configuration model
 
-### Explicit collection declarations
+### Built-in keyword and sibling lookup
 
-Applications opt in particular string-keyed collections through their reusable
-configuration definition, consistently for ordinary and layered loading. Each
-mapping key is the entry's identity. References use exact names within the
-declared collection, independent of filesystem namespaces or declaration order.
-One entry can name one parent; chains and shared parents are supported.
+A mapping containing `extends: foo` inherits from the mapping named `foo` in
+its immediately enclosing mapping. Lookup uses an exact sibling key, independent
+of declaration order. It does not interpret names as paths, search enclosing
+scopes, or search unrelated branches. One mapping can name one parent; chains
+and shared parents are supported.
 
-The declaration identifies structural metadata rather than enabling callbacks
-that choose arbitrary lookup or merge strategies. Ordinary fields remain ordinary
-data outside declared collections. The declaration syntax and treatment of the
-parent-reference field are addressed in Open Decisions below.
+```yaml
+baz:
+  foo:
+    width: 10
+    length: 20
+  bar:
+    extends: foo
+    width: 20
+```
 
-Mappings with no parent remain independent entries. Explicit null or scalar
-entries retain their ordinary data meaning and are subject to normal final
-schema validation; an entry used as a parent must resolve to a mapping so its
-settings can combine with the child. Missing collections contribute no entries.
-A present collection must have the mapping shape required by its declaration.
+The resulting `bar` has `width: 20` and `length: 20`. ConfigKit consumes the
+`extends` field as structural metadata, retaining its provenance for inspection
+rather than passing it to the application's typed payload.
 
-Identity comes from the collection key, not a copied field in the parent payload.
-The child keeps its key. Application fields such as `name` or `id` have ordinary
-combination semantics unless consumed as explicitly declared structural metadata.
+`extends` becomes reserved configuration syntax throughout input mappings,
+consistently for ordinary and layered loading and every supported source kind.
+It is not a configurable keyword or an application-selected collection feature.
+Mappings without a relationship retain their own payload, which can still contain
+nested inheritance. Explicit null or scalar siblings retain ordinary data meaning
+and final schema validation; a referenced parent must be a mapping.
+
+Identity comes from the sibling key, not a copied field in the parent payload.
+The child keeps its key. Application fields such as `name` or `id` follow ordinary
+combination semantics. Reference-value details and positions without a sibling
+namespace are addressed in Open Decisions below.
 
 ### Source precedence first, inheritance second
 
 First assemble the effective input under the existing source rules: strict
 composition within tiers and recursive overlay across underlays, main input,
 selected overlay, and explicit CLI overrides. Strict conflicts remain errors.
-Then resolve inheritance against the resulting effective collection.
+Then resolve inheritance against the resulting effective tree. Inheritance is
+the last structural composition operation, not the last loading lifecycle phase.
 
 Every parent reference sees that parent's effective definition, including source
 overrides. For each entry, resolve the parent's chain, then recursively overlay
@@ -103,17 +116,61 @@ Parent-to-child combination uses the existing recursive mapping overlay:
 - Explicit null replaces the inherited value, subject to final schema acceptance.
 - Cross-kind replacement follows the same overlay rules and final schema checks.
 
-The parent's inheritance-control metadata is not inherited as a payload setting.
-The child supplies its own relationship. Inherited payload values can themselves
-contain ordinary strings or fields with similar names without acquiring reference
-semantics outside the declared position.
+The parent's own `extends` relationship is consumed when its parent chain is
+expanded; it is not copied as the child's own relationship. Nested `extends`
+fields remain structural instructions and travel with the nested payload until
+that destination scope is resolved, as described below.
 
 These rules intentionally differ from the request's suggested null-as-unspecified
 and whole-parameter-map replacement. Consumers adopting the feature adopt
 ConfigKit's shared rules. Multiple inheritance, sequence merging, per-field
-strategies, deletion operators, cross-collection references, and named-overlay
+strategies, deletion operators, cross-scope references, and named-overlay
 file inheritance are outside this feature's initial scope.
 
+### Enclosing-first, dependency-aware resolution
+
+Resolve inheritance from the root outward. Within each mapping scope, parent
+dependencies determine resolution order: expand a sibling's parent chain before
+combining that sibling's own payload. Then resolve nested relationships within
+the resulting destination mappings. Nested relationships inherited from a parent
+are evaluated in the destination context, after enclosing combination.
+
+```yaml
+presets:
+  base:
+    settings:
+      primary:
+        width: 10
+      secondary:
+        extends: primary
+  custom:
+    extends: base
+    settings:
+      primary:
+        width: 20
+```
+
+Here `base.settings.secondary.width` becomes `10`, and
+`custom.settings.secondary.width` becomes `20`. The nested relationship is
+resolved independently in each resulting `settings` mapping. Copying the parent's
+already fully resolved nested settings would produce the wrong result for `custom`.
+
+The engine therefore distinguishes expansion of a mapping's own parent chain
+from resolution of its nested payload. It retains unresolved nested instructions
+while combining enclosing mappings. Memoized work must respect that distinction
+and the destination namespace; a fully resolved parent subtree is not a reusable
+template for every child.
+
+Visit mapping keys in lexicographic order for deterministic traversal and
+diagnostics, following dependencies as needed. Dependencies and nesting determine
+the result; alphabetical order never determines which values win. A parent may
+sort before or after its child without changing the meaning. Breadth-first versus
+recursive traversal is an implementation choice subject to these semantics.
+
+Nested mappings inside sequences are also visited, while sequence combination
+continues to replace whole sequences. The treatment of a relationship on a
+sequence element itself is addressed below.
+
 ## Loading and validation
 
 The proposed lifecycle is: existing acquisition/composition and main-owned version
@@ -130,11 +187,11 @@ existing contract.
 
 Preserve the existing version owner's declaration and checking point. Inheritance
 must not supply, replace, or remove the protected version path or bypass ancestor
-protections. Identify attempted protected-path changes explicitly during design
-of declarations rather than moving the checker to a different lifecycle point.
+protections. Account for attempted protected-path changes in the expansion
+design rather than moving the checker to a different lifecycle point.
 
-All declared relationships resolve during the load, including entries the
-application might not select for execution. Missing parents, non-mapping parents,
+All relationships in the effective configuration resolve during the load, including
+entries the application might not select for execution. Missing parents, non-mapping parents,
 invalid reference values, self-reference, and longer cycles fail the load with
 safe structured diagnostics and relevant source locations. Traverse deterministically;
 diagnostics should identify the relationship and chain rather than expose payload
@@ -180,11 +237,12 @@ credential values belong in inheritance trees, metadata, or diagnostics.
 
 ## Efficiency and ownership
 
-Model each collection as entries and single-parent edges. Use indexed name lookup,
+Model each sibling scope as entries and single-parent edges. Use indexed name lookup,
 cycle detection, and memoized parent resolution to avoid repeated ancestor work.
 Materialize independent public results while allowing carefully owned immutable
 private structures to share work. Do not introduce global caches or reload sources
-for individual entries.
+for individual entries. Cache enclosing expansion separately from destination-local
+nested resolution, so shared work does not change reference meaning.
 
 Assess work against input size, relationship edges, materialized effective values,
 and retained provenance/history. Many children can legitimately produce much more
@@ -200,14 +258,16 @@ caps. Any necessary new public limit requires an explicit design decision.
 
 ## Work needed from the baseline
 
-The feature requires root-owned opt-in declarations, a private relationship graph
-and expansion phase, and lifecycle integration before typed decoding. Presence,
+The feature requires built-in recognition of `extends`, a private dependency-aware
+expansion phase with destination-local nested resolution, and lifecycle integration
+before typed decoding. No collection-registration API is needed. Presence,
 derivation history, diagnostics, sensitivity propagation, and source-relative
 resource origins must extend together. The public parser-independent facade
 remains the only consumer boundary.
 
 Representative examples and behavioral tests should establish source/inheritance
-ordering, chains and shared parents, failure diagnostics, partial bases, explicit
+ordering, chains and shared parents, enclosing-first nested resolution, independence
+from parent/child name sort order, failure diagnostics, partial bases, explicit
 values versus omission, unchanged sequence/null behavior, origin/resource
 preservation, result independence, and safe secret inspection. Both loading entry
 points and supported source representations should share those semantics.
@@ -216,14 +276,16 @@ A new ADR should record the selected inheritance architecture after the open
 decisions are resolved. Architecture/deferred-scope documentation must recognize
 the new structural responsibility; public API, configuration, consumer, and
 internal documentation should describe implemented behavior when it lands.
-Implementation sequencing belongs in a separate future plan. Existing callers
-that do not opt in retain their behavior; an additive minor release is the target,
-with release preparation and publication handled separately.
+Implementation sequencing belongs in a separate future plan. Reserving `extends`
+changes behavior for applications that currently use it as ordinary data. Document
+that compatibility change and the required migration rather than describing this
+as a purely additive opt-in feature. Configurations without `extends` retain their
+existing behavior. Release preparation and publication are separate work.
 
 ## Completion criteria
 
-A consumer can declare a named collection, load mixed-source configuration once,
-and receive resolved independent entries with one set of combination rules.
+A consumer can use `extends` between sibling mappings, load mixed-source
+configuration once, and receive resolved independent entries with one set of combination rules.
 It can inspect whether each setting was direct, inherited, defaulted, or derived,
 follow inherited resource origins, and validate final domain meaning. Missing or
 cyclic relationships fail before a usable result is returned. The consumer needs
@@ -231,23 +293,26 @@ no separate graph resolver, generic merge pass, or provenance reconstruction.
 
 ## Open Decisions
 
-### 1. Declaring collections and parent references
+### 1. Reference values and positions without a sibling namespace
 
-**Recommendation:** a root-owned declaration in `Definition` identifies a
-collection path and one direct parent-reference field name within each entry.
-Consume that field as structural metadata during expansion and retain its source
-location for inspection. An omitted reference means no parent; explicit null
-removes a lower-tier relationship. A string names an exact key in the effective
-collection, including an empty string if such an entry exists. Other shapes are
-reference errors. Outside the opted-in position, the field remains ordinary data.
-This integrates naturally with existing source precedence and avoids a globally
-reserved `extends` field or duplicated entry identifiers.
+**Recommendation:** an omitted `extends` means no relationship; explicit null
+cancels a relationship supplied by a lower-precedence source or inherited nested
+payload. A string names an exact sibling key, including an empty string if that
+key exists. Other value shapes are reference errors. This follows existing
+presence and replacement rules without introducing a deletion operator.
 
-**Alternative:** declare a separate name-to-parent table at a designated input
-path. Payload fields never overlap control metadata and application structs need
-no inheritance field, but settings and relationships become separated and must
-be kept consistent. It also introduces a second keyed collection into diagnostics
-and source composition for each inheritance relationship set.
+A string-valued `extends` on the document root or directly on a sequence-element
+mapping has no enclosing mapping of named siblings. Report a location-bearing
+reference error rather than searching elsewhere. Mappings nested within those
+structures can still use their own sibling namespaces normally. This follows
+the selected lookup rule, rather than imposing a general ban on nested inheritance.
+
+**Alternative:** require every present `extends` to be a string. This makes the
+keyword's value type simpler, but leaves no direct way to cancel an existing
+relationship while retaining the mapping's other fields. For root/sequence-element
+references, an ancestor-scope fallback could supply a namespace, but would introduce
+a positional exception to the agreed immediate-sibling rule; adopting it would
+require revisiting that decision rather than treating it as an implementation detail.
 
 ### 2. Meaning of inherited presence
 
@@ -263,19 +328,3 @@ a new effective-input-presence query/defaulting rule. This preserves a narrow
 direct-declaration meaning of explicitness, but requires two presence concepts
 through the resolver lifecycle and risks defaults overwriting inherited settings
 unless callers adopt the new query consistently.
-
-### 3. Nested or overlapping collection declarations
-
-**Recommendation:** initially support multiple independent, non-overlapping
-declared collection paths. Report overlapping declarations as an options error.
-An outer inheritance expansion can copy or replace an inner collection, so
-overlap otherwise needs a selected rule for whether inner inheritance resolves
-before copying or after arrival at each destination. This restriction makes that
-unresolved interaction explicit rather than allowing declaration order to choose
-semantics. It does not restrict ordinary nested payload mappings.
-
-**Alternative:** define destination-based outer-to-inner expansion, resolving each
-inner collection after the enclosing entry is materialized. This supports nested
-presets, but creates additional namespace, dependency, and provenance semantics
-and can multiply graph instances. It is viable if a concrete consumer need
-justifies that additional model; it should not emerge accidentally from traversal.