Plan filesystem sources and underlays

docs/roadmap/sources-and-layering.md

@@ -1,8 +1,8 @@
 # Filesystem Sources and Source-Backed Underlays
 
-Status: proposed target for design agreement; not implemented. Significant API
-and semantic choices are collected under Open Decisions. A later implementation
-plan will own staged execution, detailed signatures, and acceptance commands.
+Status: agreed target for implementation; not implemented. This document owns
+the desired behavior and design rationale. The [implementation plan](implementation.md)
+owns API signatures, sequential work, and validation commands.
 
 ## Purpose and priorities
 
@@ -74,6 +74,15 @@ Redesigning schema-version policy is outside this roadmap's scope.
 
 ## Target source model
 
+The extension preserves the
+[synchronous, complete-on-return lifecycle](../config.md#completion-and-resource-lifetime).
+Underlay assembly and the final configuration/metadata snapshot complete within
+the load call. Resource-context retention is passive; resource access and any
+subsequent loading require explicit application calls. The proposed architectural
+rationale is recorded in [ADR 0008](../adr/0008-complete-on-return-loading.md).
+The handle-ownership requirements below apply to the new source/context APIs as
+they are implemented.
+
 ### Native filesystem input
 
 A caller can supply a filesystem, an explicit file location within it, and a
@@ -122,6 +131,29 @@ and interpreting message text, rendering, and handling resource errors. Context
 retention must not imply that external resources are snapshotted with the loaded
 configuration or remain valid after the caller closes their filesystem.
 
+The context contract covers caller-supplied filesystems and host-file sources.
+ConfigKit closes every library-owned filesystem handle before loading returns,
+on success or failure, including input files and directory/root handles. Returned
+configuration, provenance, and resource-context records retain no such handles
+and require no filesystem cleanup by the consumer. Configuration and provenance
+remain usable independently of subsequent filesystem access.
+
+Caller-supplied filesystems and readers remain caller-owned. A supplied filesystem
+may itself depend on handles whose lifetime the caller manages; retaining its
+context does not extend that lifetime or transfer ownership to ConfigKit.
+
+Host-file resource contexts retain the information needed to open resources later,
+preserving the host backend's established containment semantics. Later resource
+access acquires its own handles: helpers close handles they use internally, and
+callers close any resource handles explicitly returned to them. Host contexts
+use a passive path-backed filesystem adapter that opens a fresh contained root
+for each resource open and releases it with the returned file. Resource
+opens observe the backing filesystem at access time and can fail independently
+of an earlier successful configuration load. Sources without filesystem context
+report its absence explicitly; composite origins expose their contributing
+contexts rather than choosing a directory. These rules keep configuration
+snapshot ownership separate from resource-access lifetime.
+
 ## Target layering model
 
 ### Explicit operations, not discovery order
@@ -145,6 +177,17 @@ Typed defaults remain the existing Go-value baseline. Source underlays add
 configuration-backed defaults with imports, filesystem context, and source
 provenance; they are not a replacement for the typed-defaults facility.
 
+An underlay tier is an explicitly supplied group of source contributions. Each
+group is strictly composed, including enabled imports, before it participates
+in precedence assembly. Consumers supply zero or more tiers in lowest-to-highest
+precedence order; later tiers overlay earlier tiers recursively. A source's
+position within a group does not authorize replacement. The main input is
+optional: an underlay-only load uses the same lifecycle, and a load with neither
+main input nor underlays retains defaults-only behavior with supported explicit
+overrides. Selecting a named overlay still requires its explicit lookup context.
+The public representation exposes grouping and ordering directly. Empty groups
+contribute nothing; explicit defaults-only sources also contribute no document.
+
 The selected operation determines how values combine, not their source format
 or loading entry point. Equivalent file, embedded, byte/reader, and programmatic
 input uses the same presence and combination rules. Strictly composing independent
@@ -152,6 +195,11 @@ contributions and intentionally overlaying precedence tiers express different
 intentions; merely changing the input medium must not switch between them.
 This applies to both ordinary `Load` and
 the new layered-loading interface wherever they offer the same operation.
+Consistency applies to combination, presence, and origin meaning. Source-specific
+acquisition and lifetime rules remain explicit: a reader is single-consumption,
+bytes are snapshotted, and filesystem input can observe backing-store changes.
+CLI records retain their existing path restrictions. Source equivalence does not
+promise identical reuse, locations, resource access, or diagnostic labels.
 The existing opt-in version-path ownership rule remains an explicit metadata
 constraint, independent of whether its owning source is a file or programmatic
 input; it is not a source-format-dependent merge rule.
@@ -232,9 +280,12 @@ An absent declaration remains absent for the checker; an invalid version type
 fails without invoking it. Later input cannot mask a version-check failure.
 
 Filesystem sources and layered loading do not require a different version
-policy. The layered API needs a way to identify the version-owning input when a
-`VersionRule` is configured; Decision 3 concerns that integration only. Without
-version checking, no version owner or synthetic primary document is required.
+policy. The designated main input owns the version when a `VersionRule` is
+configured, extending the existing selected-source role. It may use any supported
+source kind. With no main input, the checker receives an absent declaration,
+consistent with existing `NoSource` behavior; underlays cannot supply that
+declaration. Without version checking, no version owner or synthetic primary
+document is required.
 Named-overlay lookup context is a separate concern, not a reason to impose
 version ownership on unversioned loads.
 
@@ -250,7 +301,12 @@ Programmatic input has a non-secret source label and uses explicit structure to
 represent presence. Omitted mapping keys are absent; present zero, false, empty
 string, empty collection, and nil/null values remain explicit. Use ConfigKit's
 existing canonical scalar and collection conventions, not YAML round trips or
-Go zero-value inference. Decision 1 selects the initial input representation.
+Go zero-value inference. The initial representation is a labeled string-keyed
+mapping using the scalar/list/map value set supported by explicit overrides.
+This reuses established conventions and keeps presence directly expressible.
+The source snapshots the mapping at construction, like byte sources. A nil root
+mapping represents null and fails the existing mapping-root requirement; an empty
+non-nil mapping is a valid empty document.
 
 Do not label generic programmatic sources as CLI input or misuse synthetic flag
 names to fit the existing override API. Preserve parser-independent conversion,
@@ -323,15 +379,16 @@ claim. No timing threshold or additional release gate is requested.
 
 ## Work needed from the baseline
 
-These are responsibility areas, not implementation stages:
+The target requires changes in the following responsibility areas; the linked
+implementation plan owns their execution order and concrete validation:
 
 - Extend the root-owned source/context facade and private acquisition boundary
   for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling
   without exporting parser or engine types.
 - Add source-backed underlays and programmatic input, sharing one engine path
   with ordinary loading rather than building a parallel configuration system.
-- Resolve the representation and ownership decisions below, then extend source
-  records, histories, diagnostics, and rendering consistently across input kinds.
+- Extend source records, histories, diagnostics, and rendering consistently
+  across input kinds.
 - Integrate version-owner selection into layered loading while preserving the
   existing version lifecycle and protections. Keep filesystem-context selection
   independent of version checking.
@@ -363,11 +420,8 @@ These are responsibility areas, not implementation stages:
   as behavior lands. Record significant boundary changes in an ADR without
   rewriting accepted ADR history. Keep named inheritance explicitly separate.
 
-Plan an additive minor release for the new public capability, preserving current
-version semantics. If another design choice requires changing documented behavior,
-identify that change expressly and provide migration guidance rather than calling
-it an internal optimization.
-No release, implementation change, or current-policy revision occurs in this pass.
+The capability is additive and intended for a minor release, preserving current
+version semantics. Release preparation and publication remain separate tasks.
 
 ## Completion criteria
 
@@ -386,50 +440,17 @@ strict within-tier composition, recursive cross-tier overlay, and downstream dom
 resolution. Equivalent-input tests and the maintained consumer example establish
 semantic consistency and reduced integration code, not merely API availability.
 
-## Open Decisions
+## Selected resource-context interface
 
-### 1. Initial programmatic representation
+A separate source-ID-based query exposes standard filesystem access and a
+source-relative base location through root-owned API types. Ordinary source
+records remain descriptive. The query reports absent context explicitly and
+supports imported sources as well as initial inputs. Consumers follow each
+contributing origin independently for composite values. This avoids a mandatory
+downstream filesystem registry and preserves provenance through recursive
+combination.
 
-**Recommendation:** labeled string-keyed mappings with the scalar/list/map value
-set already supported by explicit overrides. Missing keys express omission and
-nil expresses null. Reuse private conversion machinery, not CLI-specific public
-records. This is predictable, directly expressible in Go, and consistent with
-current dynamic configuration semantics.
-
-**Alternative:** typed Go values with a separate explicit-presence description.
-This can reduce downstream mapping code, but introduces reconciliation of field
-bindings, masks, nulls, and typed zero values and a larger public contract. Do not
-infer omission from zero values. Add typed construction only if real examples
-show that the simple mapping representation imposes substantial repetitive work.
-
-### 2. How consumers obtain filesystem resource context
-
-**Recommendation:** a separate source-context query exposing standard filesystem
-access and a source-relative base location through root-owned API types, while
-ordinary source records remain descriptive. The returned filesystem is a borrowed
-capability, not a cloned or immutable object. This avoids a mandatory downstream
-registry and supports origins from imports as well as initial sources. Make
-lifetime and nil/no-context behavior explicit before implementation.
-
-**Alternative:** expose opaque context IDs and relative paths; callers retain a
-registry mapping IDs to their filesystems. Metadata stays capability-free and
-serializable, but every consumer must maintain and correctly reconcile another
-registry. This is viable if limiting retained filesystem capabilities proves
-more important than the additional integration machinery.
-
-### 3. Selecting the version-owning input for layered loads
-
-**Recommendation:** the designated main source owns the version when configuring
-a `VersionRule`, extending the existing selected-source role rather than letting
-an underlay win ownership by list position. It need not be a host file or a newly
-manufactured root document. Its imports remain fragments and underlays cannot
-declare the application's version. No special version-owner designation is
-needed when checking is not configured, including loads using only underlays.
-
-**Alternative:** allow explicit designation of a base input as the version owner
-separately from its main/underlay role. This supports bundles carrying the
-application version, but adds another ownership choice and complicates the
-otherwise clear distinction between defaults and application configuration.
-Either choice must preserve the existing version contract; this decision does
-not reopen effective-version
-or per-document version validation.
+Caller-supplied filesystems are borrowed capabilities; host contexts are passive
+path-backed adapters. Neither is a cloned or immutable filesystem. Queries perform
+no I/O, and later resource opens follow the ownership rules above. Configuration
+results retain no library-owned open handles.