From e1ce24aabb6979d06b84718dc28c9220c4fe8bca Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 29 Sep 2026 16:06:31 +0000 Subject: [PATCH] Focus the source roadmap on recursive underlays --- docs/roadmap/sources-and-layering.md | 175 +++++++++++++++------------ 1 file changed, 97 insertions(+), 78 deletions(-) diff --git a/docs/roadmap/sources-and-layering.md b/docs/roadmap/sources-and-layering.md index 77934cf..4865c9d 100644 --- a/docs/roadmap/sources-and-layering.md +++ b/docs/roadmap/sources-and-layering.md @@ -1,4 +1,4 @@ -# Filesystem Sources and Explicit Layered Loading +# Filesystem Sources and Source-Backed Underlays Status: proposed target for design agreement; not implemented. Significant API and semantic choices are collected under Open Decisions. A later implementation @@ -6,12 +6,13 @@ plan will own staged execution, detailed signatures, and acceptance commands. ## Purpose and priorities -Implement the two prerequisites in the +Address the filesystem and layered-input needs in the [PromptKit request](feature-request-promptkit.md) as one coherent extension: -native `fs.FS` sources and public, explicitly ordered configuration layers, -including programmatic input. Neither capability should require temporary files, -a generated root YAML document, or serialization through YAML merely to transfer -already available values. +native `fs.FS` sources and explicit source-backed underlays, including +programmatic input. ConfigKit's recursive combination model takes precedence +over PromptKit's requested whole-definition replacement. Neither capability +should require temporary files, a generated root YAML document, or serialization +through YAML merely to transfer already available values. Design priorities, in order: @@ -125,17 +126,31 @@ configuration or remain valid after the caller closes their filesystem. ### Explicit operations, not discovery order -Expose a public loading entry point for an explicit ordered set of sources and -operations, using the same typed definition/result lifecycle as ordinary `Load`. +Extend public loading with explicitly supplied lower-precedence configuration +sources, using the same typed definition/result lifecycle as ordinary `Load`. Support native filesystem, host-file, byte, reader, and programmatic inputs. -The order is supplied by the caller; ConfigKit does not discover priorities or -know about PromptKit's catalog tiers. +The central workflow is bundled defaults overridden by application configuration, +not an unrestricted sequence of user-selectable merge algorithms. Underlays can +form explicitly ordered default tiers; ConfigKit does not discover priorities or +know about PromptKit's catalog tiers. Source kind does not confer priority: +programmatic data can supply the appropriate supported input role without being +misrepresented as CLI flags. + +The precedence model, from lowest to highest, is: + +**Typed application defaults < source underlays < strictly composed main +configuration < selected overlay < explicit CLI overrides.** + +Typed defaults remain the existing Go-value baseline. Source underlays add +configuration-backed defaults with imports, filesystem context, and source +provenance; they are not a replacement for the typed-defaults facility. The selected operation determines how values combine, not their source format or loading entry point. Equivalent file, embedded, byte/reader, and programmatic -input uses the same presence and combination rules. Explicitly choosing compose, -overlay, or replacement expresses different intentions; merely changing the input -medium must not switch between them. This applies to both ordinary `Load` and +input uses the same presence and combination rules. Strictly composing independent +contributions and intentionally overlaying precedence tiers express different +intentions; merely changing the input medium must not switch between them. +This applies to both ordinary `Load` and the new layered-loading interface wherever they offer the same operation. The existing opt-in version-path ownership rule remains an explicit metadata constraint, independent of whether its owning source is a file or programmatic @@ -147,15 +162,27 @@ Keep strict composition and intentional overriding visibly distinct: | --- | --- | | Compose independent contributions | Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations. | | Overlay settings | Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace. | -| Replace a definition | Replace the selected input subtree as a unit, without carrying over omitted lower-input children. The inclusion and public expression of this operation are Decision 1. | -Each independently supplied root may bring its explicitly enabled imports and -source context. Its import graph is strictly composed before intentional -precedence is applied. An overlay must not hide malformed YAML or a composition -conflict in an earlier contribution. There is no automatic matching by an `id` -field, implicit sequence merge, universal profile registry, or merge callback. +Underlay is a precedence role, not another combination algorithm. Assemble the +main document and all its imports by strict composition before applying that +result over the underlays using recursive overlay. A conflicting main/import +definition still fails, even if it would overwrite an underlay or be overwritten +by the selected overlay. Never flatten main and imported documents into a +last-wins list. -Preserve version checking after base composition and before intentional overlays +An underlay source may bring its explicitly enabled imports and source context. +Independent contributions within an underlay tier compose strictly; intentional +precedence between explicitly ordered tiers uses recursive overlay. Discovery +order does not establish tiers or authorize overriding. Syntax, source, and +composition failures cannot be hidden by higher-priority input. + +There is no whole-definition replacement mode, per-layer recursive/atomic switch, +collection-specific identity matching, implicit sequence merge, profile registry, +or merge callback in this feature. Existing atomic scalar, sequence, null, and +cross-kind replacement under overlay remains unchanged. + +Preserve the main configuration's version check after its strict composition and +schema construction, before precedence assembly with underlays, selected overlay, and CLI contributions. Assemble the effective input before typed defaults/decoding, application resolution, secret resolution, semantic validation, and final snapshotting. Each lifecycle phase runs once per load, not once per layer. @@ -163,45 +190,50 @@ Higher layers may replace values that would not fit the final typed schema; effective typed validation is not validation of each intermediate snapshot. Input syntax, source access, and composition remain validated at acquisition. -The standard precedence relationship is typed defaults below assembled input, -then the existing selected-overlay convenience and explicit CLI overrides where -those are used, followed by recorded application resolution. Named-overlay lookup -has an unambiguous filesystem context, separate from version ownership: ordinary +Recorded application resolution follows input assembly and decoding. Named-overlay +lookup has an unambiguous filesystem context, separate from version ownership: ordinary loading retains its source/root convention; a multi-context load explicitly identifies the lookup context when that convenience is used. Other sources retain their own import and resource contexts. Callers do not manufacture a primary document merely to combine layers, and no overlay lookup context is needed when no named overlay is selected. -Replacing lower input does not mean resetting or deleting typed application -defaults. Existing defaults and null rules still apply during final decoding. -The distinction must be visible in examples, especially empty maps and partially -specified definitions. No new reset/delete operator is implied by replacement. -For example, replacing input `{timeout: 10, retries: 2}` with `{timeout: 20}` -removes the lower input's `retries: 2`; a typed default of `retries: 3` can still -supply the final value. The metadata then attributes `retries` to defaults, not -to the replaced input. An empty replacement entry likewise removes lower-input -children without suppressing applicable typed defaults. +Higher-priority mappings retain omitted lower-input children. Empty mappings do +not clear those children; sequences, including empty sequences, replace as a +unit, and null remains explicit subject to the final schema. Typed defaults +continue to fill fields absent from the assembled input under existing rules. +No reset/delete operator is added. + +For example, a bundled profile containing `{model: example-model, timeout: 30}` +combined with application input `{model: another-model}` retains `timeout: 30` +from the bundle. Its model origin points to the application and its timeout +origin points to the underlay. PromptKit must validate that final profile rather +than assume the application entry discarded all bundled settings. This knowingly +differs from its requested identity-based whole-definition replacement. Consumer +examples should adopt the shared semantics, not implement a hidden replacement +pass after loading. A replacement operation could be considered separately if +future concrete needs justify it, but is not an open choice in this roadmap. ### Preserve existing schema-version semantics An application opts into version checking by configuring a version path. A key spelled `version` has no automatic special status. When checking is configured, one designated input owns that declaration; fragments cannot contribute it and -overlays or CLI overrides cannot alter it, even to an equal value. New layering -operations must respect that ownership rather than bypass it through subtree -replacement. Preserve the current version-path and ancestor protections. +underlays, overlays, or CLI overrides cannot supply or alter it, even to an equal +value. Preserve the current version-path and ancestor protections when integrating +the new input role. Keep the checker at its existing lifecycle point: after base composition and -schema construction, before intentional overlays, CLI overrides, defaults, and -typed decoding. Preserve supported scalar forms, absent/null distinction, +schema construction, before underlay precedence assembly, selected overlays, +CLI overrides, defaults, and typed decoding. Preserve supported scalar forms, +absent/null distinction, diagnostics, failure propagation, and version-path integer-to-string decoding. An absent declaration remains absent for the checker; an invalid version type fails without invoking it. Later input cannot mask a version-check failure. Filesystem sources and layered loading do not require a different version policy. The layered API needs a way to identify the version-owning input when a -`VersionRule` is configured; Decision 4 concerns that integration only. Without +`VersionRule` is configured; Decision 3 concerns that integration only. Without version checking, no version owner or synthetic primary document is required. Named-overlay lookup context is a separate concern, not a reason to impose version ownership on unversioned loads. @@ -218,7 +250,7 @@ Programmatic input has a non-secret source label and uses explicit structure to represent presence. Omitted mapping keys are absent; present zero, false, empty string, empty collection, and nil/null values remain explicit. Use ConfigKit's existing canonical scalar and collection conventions, not YAML round trips or -Go zero-value inference. Decision 2 selects the initial input representation. +Go zero-value inference. Decision 1 selects the initial input representation. Do not label generic programmatic sources as CLI input or misuse synthetic flag names to fit the existing override API. Preserve parser-independent conversion, @@ -282,8 +314,9 @@ The target implementation: persistent-tree framework merely to meet an asymptotic target. Use representative benchmarks and allocation measurements for broad keyed -catalogs, many small layers, repeated replacement, imported filesystem defaults, -and retained history. Compare growth as inputs scale, including small common +catalogs, many small underlay tiers, repeated scalar/sequence replacement, +imported filesystem defaults, and retained history. Compare growth as inputs +scale, including small common loads. Favor changes addressing demonstrated scaling problems; preserve stable diagnostic order and ownership rather than trading them for a superficial O(n) claim. No timing threshold or additional release gate is requested. @@ -295,9 +328,9 @@ These are responsibility areas, not implementation stages: - Extend the root-owned source/context facade and private acquisition boundary for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling without exporting parser or engine types. -- Add explicit layered loading and programmatic input, sharing one engine path +- Add source-backed underlays and programmatic input, sharing one engine path with ordinary loading rather than building a parallel configuration system. -- Resolve the fixed operation and ownership decisions below, then extend source +- Resolve the representation and ownership decisions below, then extend source records, histories, diagnostics, and rendering consistently across input kinds. - Integrate version-owner selection into layered loading while preserving the existing version lifecycle and protections. Keep filesystem-context selection @@ -313,11 +346,14 @@ These are responsibility areas, not implementation stages: - Compare equivalent logical inputs across supported source kinds and loading interfaces. Assert consistent values, presence, operation outcomes, and origin meaning while allowing genuine location/label differences. Cover defaulted and - derived values, replacement followed by typed defaults, and preservation of + derived values, recursively retained underlay fields, unchanged atomic-value + replacement, typed defaults, strict main/import conflicts, and preservation of version ownership/check timing across both loading interfaces. Verify that unversioned layered loads need no synthetic version-owning root. - Provide maintained simple and PromptKit-shaped examples: embedded defaults, operator configuration from another context, and labeled programmatic input. + Show explicitly that a same-name profile recursively combines, rather than + replacing its whole lower definition. Demonstrate exact operations and resource origins without provider calls or recreating PromptKit itself. The consumer supplies sources, explicit operations, types, and domain rules; it does not implement generic merging, presence @@ -346,30 +382,13 @@ defaults, and recorded domain resolution, and metadata distinguishes explicit zero/false from absence even when the typed values are equal. Benchmarks demonstrate that adding inputs does not entail avoidable full-collection copying or quadratic key lookup. Public contracts and examples clearly distinguish -composition, overlay, any selected replacement operation, and downstream domain +strict within-tier composition, recursive cross-tier overlay, and downstream domain resolution. Equivalent-input tests and the maintained consumer example establish semantic consistency and reduced integration code, not merely API availability. ## Open Decisions -### 1. Whole-definition replacement - -**Recommendation:** include a small fixed replacement operation alongside compose -and recursive overlay. For PromptKit-shaped catalogs, explicitly identify a -string-keyed collection path and replace each supplied entry as a whole; retain -unmentioned entries. Do not infer identities from object fields. An empty supplied -collection contributes no entries; an explicitly supplied empty entry replaces -that entry's lower input. Use the same rule for every consumer, not arbitrary -per-field policies. This meets the concrete identity-precedence need without -silently redefining ordinary overlay. Final typed defaults still apply. - -**Alternative:** ship only existing composition/overlay semantics and leave -whole-definition selection downstream. This is smaller and closest to today's -surface, but may force consumers to parse/select definitions and reconstruct -origins outside ConfigKit, undermining the request's purpose. Confirm with the -combined consumer example before claiming this narrower scope meets the need. - -### 2. Initial programmatic representation +### 1. Initial programmatic representation **Recommendation:** labeled string-keyed mappings with the scalar/list/map value set already supported by explicit overrides. Missing keys express omission and @@ -383,7 +402,7 @@ bindings, masks, nulls, and typed zero values and a larger public contract. Do n infer omission from zero values. Add typed construction only if real examples show that the simple mapping representation imposes substantial repetitive work. -### 3. How consumers obtain filesystem resource context +### 2. How consumers obtain filesystem resource context **Recommendation:** a separate source-context query exposing standard filesystem access and a source-relative base location through root-owned API types, while @@ -398,19 +417,19 @@ serializable, but every consumer must maintain and correctly reconcile another registry. This is viable if limiting retained filesystem capabilities proves more important than the additional integration machinery. -### 4. Selecting the version-owning input for layered loads +### 3. Selecting the version-owning input for layered loads -**Recommendation:** let callers explicitly designate one of the supplied base -inputs as the version owner when configuring a `VersionRule`. It need not be a -host file or a newly manufactured root document. Other contributions retain -fragment/override roles with respect to that path. Explicit designation avoids -making version ownership depend on incidental list order and preserves the -existing selected-source model without changing version acceptance or timing. -No designation is needed when version checking is not configured. +**Recommendation:** the designated main source owns the version when configuring +a `VersionRule`, extending the existing selected-source role rather than letting +an underlay win ownership by list position. It need not be a host file or a newly +manufactured root document. Its imports remain fragments and underlays cannot +declare the application's version. No special version-owner designation is +needed when checking is not configured, including loads using only underlays. -**Alternative:** make the first base input the version owner when checking is -configured. This reduces API ceremony and naturally extends a main-source-plus- -fragments model, but couples ownership to ordering and can make reordering -otherwise independent contributions consequential. Either choice must preserve -the existing version contract; this decision does not reopen effective-version +**Alternative:** allow explicit designation of a base input as the version owner +separately from its main/underlay role. This supports bundles carrying the +application version, but adds another ownership choice and complicates the +otherwise clear distinction between defaults and application configuration. +Either choice must preserve the existing version contract; this decision does +not reopen effective-version or per-document version validation.