Plan filesystem sources and underlays

docs/adr/0007-source-backed-underlays.md

@@ -0,0 +1,118 @@
+# ADR 0007: Add Source-Backed Underlays with Consistent Recursive Precedence
+
+## Status
+
+Proposed draft for review. The
+[filesystem sources and underlays roadmap](../roadmap/sources-and-layering.md)
+owns implementation status, detailed target behavior, and remaining API choices.
+
+This extends [ADR 0001](0001-separate-composition-and-overlay.md) and preserves
+its distinction between strict composition and intentional overlay.
+
+## Date
+
+2026-09-29
+
+## Context
+
+Configuration-backed defaults provide lower-priority settings that applications
+can override. These sources carry imports, locations, and relative-resource
+contexts. Integrating them into ConfigKit's loading lifecycle gives their values
+the same presence tracking and inspectable provenance as other configuration
+inputs.
+
+The [architecture goals](../policy/architecture.md#success-criteria) prioritize
+one coherent, predictable configuration model with consistent rules across
+source representations and loading interfaces. Consumers should load once,
+receive a complete validated configuration, and inspect its final values' origins.
+Reducing integration code is important subject to those principles.
+
+## Decision
+
+### Add underlays to the existing precedence model
+
+Introduce explicitly supplied source-backed underlays between typed defaults
+and main configuration. The proposed precedence, from lowest to highest, is:
+
+**Typed application defaults < source underlays < strictly composed main
+configuration < selected overlay < explicit CLI overrides.**
+
+Typed defaults remain the Go-value baseline. Underlays supply lower-priority
+configuration with source provenance and, where applicable, imports and resource
+context. This ordering describes value precedence; typed defaults fill fields
+absent from the assembled input under the existing defaulting rules.
+
+Preserve strict composition of the main document and its imports before applying
+the resulting configuration over underlays. Independent contributions within an
+underlay tier also compose strictly. Explicitly ordered tiers express intentional
+precedence. Strict composition reports conflicting contributions within each tier.
+
+Across precedence tiers, reuse the existing recursive mapping-overlay semantics.
+Omitted children retain lower-priority input; existing scalar, sequence, null,
+and cross-kind replacement semantics remain unchanged. Reusing these operations
+keeps the combination rules consistent across the loading model.
+
+### Keep source representation separate from composition semantics
+
+Native `fs.FS` and programmatic inputs join the shared loading model.
+Equivalent inputs using the same
+operation have the same presence and precedence semantics regardless of medium.
+Programmatic values enter directly as labeled configuration sources.
+
+Retain the contributing source's identity and resource context through layering.
+A retained underlay value keeps its originating source's resource context.
+Consumers remain responsible for opening and interpreting their domain resources
+under the supplied filesystem's lifetime and access semantics. Exact
+context-access and input APIs remain roadmap decisions.
+
+### Deliver one final configuration and its inspection metadata
+
+Assemble effective input, then run the typed loading lifecycle once for the load.
+Preserve explicit zero,
+false, empty, and null input versus omission through configuration and metadata
+together.
+
+Extend existing provenance and history through underlays, defaults, and recorded
+application resolution. The returned result includes the completed combination,
+defaulting, and provenance work. Domain resolution and validation
+remain application-owned within the recorded lifecycle, consistent with
+[ADR 0003](0003-preserve-presence-and-resolution-history.md).
+
+Preserve the existing opt-in schema-version ownership, path protections, and
+check timing before precedence
+assembly and typed decoding. The roadmap owns how layered input identifies that
+owner when version checking is configured.
+
+Favor near-linear processing in input, retained history, and output size through
+incremental assembly of load-owned state. This is an engineering objective
+subordinate to clarity and semantic consistency. Preserve the root-owned facade and
+private implementation boundary established by [ADR 0006](0006-public-facade.md).
+
+## Alternatives considered
+
+- **Leave bundled defaults to consumers.** Avoids extending loading, but makes
+  consumers reproduce merging, presence tracking, and provenance reconciliation.
+  It does not meet the goal of obtaining a complete configuration in one load.
+- **Offer recursive or atomic combination per layer.** Makes both behaviors
+  explicit and could serve more consumers, but expands the semantic surface and
+  its interactions with presence and provenance without a demonstrated need that
+  outweighs a single consistent rule. A future concrete requirement may justify
+  reconsideration; this feature does not add the option.
+- **Flatten all documents into last-wins layers.** Simplifies ordering but turns
+  accidental main/import conflicts into overrides, weakening the intentional
+  distinction established by ADR 0001.
+
+## Consequences
+
+Consumers gain source-backed defaults within the shared loading lifecycle.
+Strict contributions and intentional precedence remain distinct, with consistent
+combination rules across source media. Presence and provenance remain
+part of the same loading lifecycle as the final values.
+
+Higher-priority mappings retain omitted lower-priority children, so application
+validation assesses the final combined configuration.
+
+Implementation must integrate source context and underlay history into the shared
+engine with attention to assembly and history-retention costs. Current behavior
+documentation will change when the feature lands. The roadmap retains ownership
+of the remaining API decisions.

docs/adr/0008-complete-on-return-loading.md

@@ -0,0 +1,83 @@
+# ADR 0008: Synchronous, Complete-on-Return Configuration Loading
+
+## Status
+
+Proposed draft for review. The
+[configuration contract](../config.md#completion-and-resource-lifetime) describes
+the implemented loading lifecycle. The
+[source-backed underlays roadmap](../roadmap/sources-and-layering.md) owns its
+application to the planned source and resource-context APIs.
+
+## Date
+
+2026-09-29
+
+## Context
+
+Applications need a clear point at which configuration is ready for use.
+Completing configuration processing and inspection metadata together gives
+applications a coherent result whose values and origins agree. A bounded loading
+lifecycle also makes resource ownership and startup failures straightforward to
+reason about.
+
+ConfigKit's [architecture goals](../policy/architecture.md#success-criteria)
+emphasize complete configuration, inspectable provenance, predictable behavior,
+and minimal consumer bookkeeping. Applications own their execution lifecycle and
+the timing of configuration acquisition.
+
+## Decision
+
+Make loading a synchronous, complete-on-return operation. Within each load,
+ConfigKit completes input assembly, defaulting, typed decoding, configured
+application resolution, secret lookup, validation, and snapshotting before
+returning the final configuration and metadata. Reported failures return
+diagnostics and a nil result.
+
+Release all library-owned loading handles before returning on success or failure.
+The resulting configuration and metadata are independently usable, caller-owned
+data with a retained inspection snapshot. Caller-supplied readers and filesystems
+remain under caller ownership and lifetime control.
+
+After the call completes, ConfigKit remains inactive until explicitly invoked
+again. Applications normally load once at or near startup and may initiate
+additional loads when needed. Each load follows the same lifecycle and source
+reuse contracts. Inspection, rendering, and separately requested operations run
+in response to explicit application calls.
+
+For the planned resource-context capability, retain source context as passive
+information or a borrowed caller-supplied filesystem. Subsequent resource access
+acquires its own handles with explicit ownership. This preserves a clear boundary
+between configuration completion and application resource use.
+
+These choices give consumers one completion boundary for configuration and
+provenance, predictable resource lifetimes, and control over when ConfigKit runs.
+Application callbacks remain responsible for their own effects and resources.
+
+## Alternatives considered
+
+- **Background watching or refresh.** Can automate responses to source changes,
+  but adds scheduling, shutdown, publication, and resource-lifetime concerns to
+  the library. Explicit calls keep those responsibilities with the application.
+- **Lazy completion after return.** Can defer work until particular values are
+  used, but moves configuration failures into application execution and makes
+  the readiness of values and metadata harder to establish.
+- **Transactional atomic loading.** Could imply simultaneous observation of all
+  external inputs and rollback of callback effects. Filesystems, environment
+  variables, and application callbacks do not share a transaction mechanism.
+  Synchronous completion provides the required usable-result boundary without
+  promising those external guarantees.
+- **Retain loading handles in results.** Can preserve open filesystem access,
+  but gives configuration results an ongoing resource lifecycle and cleanup
+  obligation. Passive source context supports later explicit resource access
+  while keeping loading resources scoped to the load call.
+
+## Consequences
+
+Applications can treat successful return as the point at which the final
+configuration and its metadata are ready for use. Startup loading remains simple,
+and additional loads have an explicit application-owned trigger.
+
+Configuration inspection operates on retained data. Resource access observes the
+backing resource when explicitly requested and has its own failure and ownership
+boundary. Implementations of new source kinds and layering features preserve the
+same completion boundary, including cleanup on reported load failures.