Focus the source roadmap on recursive underlays

docs/roadmap/sources-and-layering.md

@@ -1,4 +1,4 @@
-# Filesystem Sources and Explicit Layered Loading
+# Filesystem Sources and Source-Backed Underlays
 
 Status: proposed target for design agreement; not implemented. Significant API
 and semantic choices are collected under Open Decisions. A later implementation
@@ -6,12 +6,13 @@ plan will own staged execution, detailed signatures, and acceptance commands.
 
 ## Purpose and priorities
 
-Implement the two prerequisites in the
+Address the filesystem and layered-input needs in the
 [PromptKit request](feature-request-promptkit.md) as one coherent extension:
-native `fs.FS` sources and public, explicitly ordered configuration layers,
-including programmatic input. Neither capability should require temporary files,
-a generated root YAML document, or serialization through YAML merely to transfer
-already available values.
+native `fs.FS` sources and explicit source-backed underlays, including
+programmatic input. ConfigKit's recursive combination model takes precedence
+over PromptKit's requested whole-definition replacement. Neither capability
+should require temporary files, a generated root YAML document, or serialization
+through YAML merely to transfer already available values.
 
 Design priorities, in order:
 
@@ -125,17 +126,31 @@ configuration or remain valid after the caller closes their filesystem.
 
 ### Explicit operations, not discovery order
 
-Expose a public loading entry point for an explicit ordered set of sources and
-operations, using the same typed definition/result lifecycle as ordinary `Load`.
+Extend public loading with explicitly supplied lower-precedence configuration
+sources, using the same typed definition/result lifecycle as ordinary `Load`.
 Support native filesystem, host-file, byte, reader, and programmatic inputs.
-The order is supplied by the caller; ConfigKit does not discover priorities or
-know about PromptKit's catalog tiers.
+The central workflow is bundled defaults overridden by application configuration,
+not an unrestricted sequence of user-selectable merge algorithms. Underlays can
+form explicitly ordered default tiers; ConfigKit does not discover priorities or
+know about PromptKit's catalog tiers. Source kind does not confer priority:
+programmatic data can supply the appropriate supported input role without being
+misrepresented as CLI flags.
+
+The precedence model, from lowest to highest, is:
+
+**Typed application defaults < source underlays < strictly composed main
+configuration < selected overlay < explicit CLI overrides.**
+
+Typed defaults remain the existing Go-value baseline. Source underlays add
+configuration-backed defaults with imports, filesystem context, and source
+provenance; they are not a replacement for the typed-defaults facility.
 
 The selected operation determines how values combine, not their source format
 or loading entry point. Equivalent file, embedded, byte/reader, and programmatic
-input uses the same presence and combination rules. Explicitly choosing compose,
-overlay, or replacement expresses different intentions; merely changing the input
-medium must not switch between them. This applies to both ordinary `Load` and
+input uses the same presence and combination rules. Strictly composing independent
+contributions and intentionally overlaying precedence tiers express different
+intentions; merely changing the input medium must not switch between them.
+This applies to both ordinary `Load` and
 the new layered-loading interface wherever they offer the same operation.
 The existing opt-in version-path ownership rule remains an explicit metadata
 constraint, independent of whether its owning source is a file or programmatic
@@ -147,15 +162,27 @@ Keep strict composition and intentional overriding visibly distinct:
 | --- | --- |
 | Compose independent contributions | Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations. |
 | Overlay settings | Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace. |
-| Replace a definition | Replace the selected input subtree as a unit, without carrying over omitted lower-input children. The inclusion and public expression of this operation are Decision 1. |
 
-Each independently supplied root may bring its explicitly enabled imports and
-source context. Its import graph is strictly composed before intentional
-precedence is applied. An overlay must not hide malformed YAML or a composition
-conflict in an earlier contribution. There is no automatic matching by an `id`
-field, implicit sequence merge, universal profile registry, or merge callback.
+Underlay is a precedence role, not another combination algorithm. Assemble the
+main document and all its imports by strict composition before applying that
+result over the underlays using recursive overlay. A conflicting main/import
+definition still fails, even if it would overwrite an underlay or be overwritten
+by the selected overlay. Never flatten main and imported documents into a
+last-wins list.
 
-Preserve version checking after base composition and before intentional overlays
+An underlay source may bring its explicitly enabled imports and source context.
+Independent contributions within an underlay tier compose strictly; intentional
+precedence between explicitly ordered tiers uses recursive overlay. Discovery
+order does not establish tiers or authorize overriding. Syntax, source, and
+composition failures cannot be hidden by higher-priority input.
+
+There is no whole-definition replacement mode, per-layer recursive/atomic switch,
+collection-specific identity matching, implicit sequence merge, profile registry,
+or merge callback in this feature. Existing atomic scalar, sequence, null, and
+cross-kind replacement under overlay remains unchanged.
+
+Preserve the main configuration's version check after its strict composition and
+schema construction, before precedence assembly with underlays, selected overlay,
 and CLI contributions. Assemble the effective input before typed defaults/decoding,
 application resolution, secret resolution, semantic validation, and final
 snapshotting. Each lifecycle phase runs once per load, not once per layer.
@@ -163,45 +190,50 @@ Higher layers may replace values that would not fit the final typed schema;
 effective typed validation is not validation of each intermediate snapshot.
 Input syntax, source access, and composition remain validated at acquisition.
 
-The standard precedence relationship is typed defaults below assembled input,
-then the existing selected-overlay convenience and explicit CLI overrides where
-those are used, followed by recorded application resolution. Named-overlay lookup
-has an unambiguous filesystem context, separate from version ownership: ordinary
+Recorded application resolution follows input assembly and decoding. Named-overlay
+lookup has an unambiguous filesystem context, separate from version ownership: ordinary
 loading retains its source/root convention; a multi-context load explicitly
 identifies the lookup context when that convenience is used. Other sources retain
 their own import and resource contexts. Callers do not manufacture a primary
 document merely to combine layers, and no overlay lookup context is needed when
 no named overlay is selected.
 
-Replacing lower input does not mean resetting or deleting typed application
-defaults. Existing defaults and null rules still apply during final decoding.
-The distinction must be visible in examples, especially empty maps and partially
-specified definitions. No new reset/delete operator is implied by replacement.
-For example, replacing input `{timeout: 10, retries: 2}` with `{timeout: 20}`
-removes the lower input's `retries: 2`; a typed default of `retries: 3` can still
-supply the final value. The metadata then attributes `retries` to defaults, not
-to the replaced input. An empty replacement entry likewise removes lower-input
-children without suppressing applicable typed defaults.
+Higher-priority mappings retain omitted lower-input children. Empty mappings do
+not clear those children; sequences, including empty sequences, replace as a
+unit, and null remains explicit subject to the final schema. Typed defaults
+continue to fill fields absent from the assembled input under existing rules.
+No reset/delete operator is added.
+
+For example, a bundled profile containing `{model: example-model, timeout: 30}`
+combined with application input `{model: another-model}` retains `timeout: 30`
+from the bundle. Its model origin points to the application and its timeout
+origin points to the underlay. PromptKit must validate that final profile rather
+than assume the application entry discarded all bundled settings. This knowingly
+differs from its requested identity-based whole-definition replacement. Consumer
+examples should adopt the shared semantics, not implement a hidden replacement
+pass after loading. A replacement operation could be considered separately if
+future concrete needs justify it, but is not an open choice in this roadmap.
 
 ### Preserve existing schema-version semantics
 
 An application opts into version checking by configuring a version path. A key
 spelled `version` has no automatic special status. When checking is configured,
 one designated input owns that declaration; fragments cannot contribute it and
-overlays or CLI overrides cannot alter it, even to an equal value. New layering
-operations must respect that ownership rather than bypass it through subtree
-replacement. Preserve the current version-path and ancestor protections.
+underlays, overlays, or CLI overrides cannot supply or alter it, even to an equal
+value. Preserve the current version-path and ancestor protections when integrating
+the new input role.
 
 Keep the checker at its existing lifecycle point: after base composition and
-schema construction, before intentional overlays, CLI overrides, defaults, and
-typed decoding. Preserve supported scalar forms, absent/null distinction,
+schema construction, before underlay precedence assembly, selected overlays,
+CLI overrides, defaults, and typed decoding. Preserve supported scalar forms,
+absent/null distinction,
 diagnostics, failure propagation, and version-path integer-to-string decoding.
 An absent declaration remains absent for the checker; an invalid version type
 fails without invoking it. Later input cannot mask a version-check failure.
 
 Filesystem sources and layered loading do not require a different version
 policy. The layered API needs a way to identify the version-owning input when a
-`VersionRule` is configured; Decision 4 concerns that integration only. Without
+`VersionRule` is configured; Decision 3 concerns that integration only. Without
 version checking, no version owner or synthetic primary document is required.
 Named-overlay lookup context is a separate concern, not a reason to impose
 version ownership on unversioned loads.
@@ -218,7 +250,7 @@ Programmatic input has a non-secret source label and uses explicit structure to
 represent presence. Omitted mapping keys are absent; present zero, false, empty
 string, empty collection, and nil/null values remain explicit. Use ConfigKit's
 existing canonical scalar and collection conventions, not YAML round trips or
-Go zero-value inference. Decision 2 selects the initial input representation.
+Go zero-value inference. Decision 1 selects the initial input representation.
 
 Do not label generic programmatic sources as CLI input or misuse synthetic flag
 names to fit the existing override API. Preserve parser-independent conversion,
@@ -282,8 +314,9 @@ The target implementation:
   persistent-tree framework merely to meet an asymptotic target.
 
 Use representative benchmarks and allocation measurements for broad keyed
-catalogs, many small layers, repeated replacement, imported filesystem defaults,
-and retained history. Compare growth as inputs scale, including small common
+catalogs, many small underlay tiers, repeated scalar/sequence replacement,
+imported filesystem defaults, and retained history. Compare growth as inputs
+scale, including small common
 loads. Favor changes addressing demonstrated scaling problems; preserve stable
 diagnostic order and ownership rather than trading them for a superficial O(n)
 claim. No timing threshold or additional release gate is requested.
@@ -295,9 +328,9 @@ These are responsibility areas, not implementation stages:
 - Extend the root-owned source/context facade and private acquisition boundary
   for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling
   without exporting parser or engine types.
-- Add explicit layered loading and programmatic input, sharing one engine path
+- Add source-backed underlays and programmatic input, sharing one engine path
   with ordinary loading rather than building a parallel configuration system.
-- Resolve the fixed operation and ownership decisions below, then extend source
+- Resolve the representation and ownership decisions below, then extend source
   records, histories, diagnostics, and rendering consistently across input kinds.
 - Integrate version-owner selection into layered loading while preserving the
   existing version lifecycle and protections. Keep filesystem-context selection
@@ -313,11 +346,14 @@ These are responsibility areas, not implementation stages:
 - Compare equivalent logical inputs across supported source kinds and loading
   interfaces. Assert consistent values, presence, operation outcomes, and origin
   meaning while allowing genuine location/label differences. Cover defaulted and
-  derived values, replacement followed by typed defaults, and preservation of
+  derived values, recursively retained underlay fields, unchanged atomic-value
+  replacement, typed defaults, strict main/import conflicts, and preservation of
   version ownership/check timing across both loading interfaces. Verify that
   unversioned layered loads need no synthetic version-owning root.
 - Provide maintained simple and PromptKit-shaped examples: embedded defaults,
   operator configuration from another context, and labeled programmatic input.
+  Show explicitly that a same-name profile recursively combines, rather than
+  replacing its whole lower definition.
   Demonstrate exact operations and resource origins without provider calls or
   recreating PromptKit itself. The consumer supplies sources, explicit operations,
   types, and domain rules; it does not implement generic merging, presence
@@ -346,30 +382,13 @@ defaults, and recorded domain resolution, and metadata distinguishes explicit
 zero/false from absence even when the typed values are equal. Benchmarks
 demonstrate that adding inputs does not entail avoidable full-collection copying
 or quadratic key lookup. Public contracts and examples clearly distinguish
-composition, overlay, any selected replacement operation, and downstream domain
+strict within-tier composition, recursive cross-tier overlay, and downstream domain
 resolution. Equivalent-input tests and the maintained consumer example establish
 semantic consistency and reduced integration code, not merely API availability.
 
 ## Open Decisions
 
-### 1. Whole-definition replacement
-
-**Recommendation:** include a small fixed replacement operation alongside compose
-and recursive overlay. For PromptKit-shaped catalogs, explicitly identify a
-string-keyed collection path and replace each supplied entry as a whole; retain
-unmentioned entries. Do not infer identities from object fields. An empty supplied
-collection contributes no entries; an explicitly supplied empty entry replaces
-that entry's lower input. Use the same rule for every consumer, not arbitrary
-per-field policies. This meets the concrete identity-precedence need without
-silently redefining ordinary overlay. Final typed defaults still apply.
-
-**Alternative:** ship only existing composition/overlay semantics and leave
-whole-definition selection downstream. This is smaller and closest to today's
-surface, but may force consumers to parse/select definitions and reconstruct
-origins outside ConfigKit, undermining the request's purpose. Confirm with the
-combined consumer example before claiming this narrower scope meets the need.
-
-### 2. Initial programmatic representation
+### 1. Initial programmatic representation
 
 **Recommendation:** labeled string-keyed mappings with the scalar/list/map value
 set already supported by explicit overrides. Missing keys express omission and
@@ -383,7 +402,7 @@ bindings, masks, nulls, and typed zero values and a larger public contract. Do n
 infer omission from zero values. Add typed construction only if real examples
 show that the simple mapping representation imposes substantial repetitive work.
 
-### 3. How consumers obtain filesystem resource context
+### 2. How consumers obtain filesystem resource context
 
 **Recommendation:** a separate source-context query exposing standard filesystem
 access and a source-relative base location through root-owned API types, while
@@ -398,19 +417,19 @@ serializable, but every consumer must maintain and correctly reconcile another
 registry. This is viable if limiting retained filesystem capabilities proves
 more important than the additional integration machinery.
 
-### 4. Selecting the version-owning input for layered loads
+### 3. Selecting the version-owning input for layered loads
 
-**Recommendation:** let callers explicitly designate one of the supplied base
-inputs as the version owner when configuring a `VersionRule`. It need not be a
-host file or a newly manufactured root document. Other contributions retain
-fragment/override roles with respect to that path. Explicit designation avoids
-making version ownership depend on incidental list order and preserves the
-existing selected-source model without changing version acceptance or timing.
-No designation is needed when version checking is not configured.
+**Recommendation:** the designated main source owns the version when configuring
+a `VersionRule`, extending the existing selected-source role rather than letting
+an underlay win ownership by list position. It need not be a host file or a newly
+manufactured root document. Its imports remain fragments and underlays cannot
+declare the application's version. No special version-owner designation is
+needed when checking is not configured, including loads using only underlays.
 
-**Alternative:** make the first base input the version owner when checking is
-configured. This reduces API ceremony and naturally extends a main-source-plus-
-fragments model, but couples ownership to ordering and can make reordering
-otherwise independent contributions consequential. Either choice must preserve
-the existing version contract; this decision does not reopen effective-version
+**Alternative:** allow explicit designation of a base input as the version owner
+separately from its main/underlay role. This supports bundles carrying the
+application version, but adds another ownership choice and complicates the
+otherwise clear distinction between defaults and application configuration.
+Either choice must preserve the existing version contract; this decision does
+not reopen effective-version
 or per-document version validation.