ConfigKit/docs/roadmap/sources-and-layering.md

# Filesystem Sources and Source-Backed Underlays

Status: agreed target for implementation; not implemented. This document owns
the desired behavior and design rationale. The [implementation plan](implementation.md)
owns API signatures, sequential work, and validation commands.

## Purpose and priorities

Address the filesystem and layered-input needs in the
[PromptKit request](feature-request-promptkit.md) as one coherent extension:
native `fs.FS` sources and explicit source-backed underlays, including
programmatic input. ConfigKit's recursive combination model takes precedence
over PromptKit's requested whole-definition replacement. Neither capability
should require temporary files, a generated root YAML document, or serialization
through YAML merely to transfer already available values.

Design priorities, in order:

1. Clear, standardized behavior consistent with ConfigKit's existing conventions.
   PromptKit may adapt its formats and APIs; its historical conventions are not
   new ConfigKit compatibility obligations.
2. Efficient loading and retained state, aiming for linear or near-linear work
   in actual input and output size. Prefer clarity and maintainability where a
   more elaborate optimization would compromise the first priority.
3. A small public surface with cohesive private implementation, usable by simple
   applications without requiring them to learn the layered-loading machinery.

Consumers load once and receive a complete configuration and its inspection
metadata. They should not need a second generic merge/defaulting pass, a separate
presence tracker, or reconstructed provenance. Domain-specific resolution remains
consumer-owned, but fits into the recorded loading lifecycle before the result
is validated and returned. Subject to consistent semantics and correct presence
and provenance, minimizing consumer integration code is an explicit design goal.

The [named-inheritance request](feature-request-promptkit-inheritance.md) remains
separate and optional. This roadmap neither implements it nor adds speculative
infrastructure for it.

## Current baseline

Planning baseline: `ec26591b275bd74f89120bdea69c5f6459dfcafb`, following v0.4.2.
Current behavior is owned by the [configuration contract](../config.md), not by
these proposals.

- `Load[T]` takes one `File`, `Bytes`, `Reader`, or `NoSource` input. Imports
  strictly compose documents; one named overlay uses a root-relative directory
  convention; explicit CLI overrides follow it.
- Filesystem context uses host directories and an `os.Root`-based implementation.
  Byte and reader labels do not imply filesystem paths. There is no native
  `fs.FS` source or public batch of heterogeneous layers.
- Composition rejects non-mapping overlaps, including equal values. Overlay
  recursively merges mappings and replaces other values. Empty mappings do not
  clear existing children; null is an explicit value, not omission.
- An explicitly configured schema-version path currently belongs to the selected
  source. Its checker runs before overlay/CLI input, which cannot alter that path;
  imported fragments cannot contribute it. This roadmap preserves those ownership
  and timing rules.
- CLI overrides already convert supported programmatic scalars, lists, and
  string-keyed maps directly to the private document representation. They have
  CLI-specific paths, labels, restrictions, and origins, not a general source API.
- Source records and resolution queries expose acquired-source identity and
  provenance, but do not provide an `fs.FS` resource context.
- Root-owned public wrappers delegate to `internal/engine`; parser-independent
  trees belong to `internal/document`, and YAML parsing stays in `internal/yaml`.
- Current mapping assembly uses repeated linear key searches. Overlay application
  clones the accumulated document and provenance; subtree-origin removal scans
  the whole origin index. Straightforward repetition across many layers would
  amplify these costs.

Existing source/lifecycle contracts, ownership, secret handling, and diagnostics
remain the compatibility baseline. This work is not permission to silently
reinterpret current `Load`, imports, selected overlays, or CLI overrides.
Redesigning schema-version policy is outside this roadmap's scope.

## Target source model

The extension preserves the
[synchronous, complete-on-return lifecycle](../config.md#completion-and-resource-lifetime).
Underlay assembly and the final configuration/metadata snapshot complete within
the load call. Resource-context retention is passive; resource access and any
subsequent loading require explicit application calls. The proposed architectural
rationale is recorded in [ADR 0008](../adr/0008-complete-on-return-loading.md).
The handle-ownership requirements below apply to the new source/context APIs as
they are implemented.

### Native filesystem input

A caller can supply a filesystem, an explicit file location within it, and a
meaningful diagnostic label/context. `embed.FS` is an ordinary supported example,
not a separate parsing mode. Use standard slash-separated `io/fs` path conventions;
do not apply host drive-letter or separator rules to virtual paths.

The source retains the filesystem context needed to acquire its imports, glob
matches, and selected overlay. Relative imports resolve from the importing
document; named overlay lookup remains relative to that configuration context's
root. References must never fall back to the host filesystem simply because a
virtual path is missing. Imports/globs retain their explicit opt-in, deterministic
ordering, composition semantics, and actionable errors.

Filesystem namespace identity and a normalized path identify a resource; labels
are presentation, not uniqueness keys. Identical names in different filesystems
must not collide. Use a reusable load-local context identity, not comparison or
hashing of arbitrary `fs.FS` interface values. Repeated-import and cycle checks
operate within the relevant input composition graph; selecting explicit layers
is distinct from accidentally importing a file twice.

Keep the host-file backend's established canonicalization and containment behavior.
For generic `fs.FS`, describe namespace-relative resolution and the supplied
filesystem's actual behavior. `fs.FS` does not universally provide symlink
canonicalization or a security sandbox; do not invent such guarantees or reject
all implementations that cannot provide host-specific operations. The caller
chooses the filesystem, its acquisition behavior, and its lifetime. ConfigKit
closes handles it opens, not a caller-owned filesystem as a whole.

### Origins and downstream resources

Locations remain meaningful for host files, filesystem-relative files, labeled
bytes, and programmatic values. Preserve deterministic source IDs and traversal
order while distinguishing source identity, input kind, and precedence role.
Do not invent host filenames or line/column coordinates for synthetic inputs.

A consumer can follow an effective field's origin to its contributing source and
obtain the context needed to resolve a relative resource. For a field inherited
through layering, that is the contributing source's context, not simply the last
layer's directory. Composite origins remain composite; ConfigKit must not choose
one contributing directory arbitrarily.

Resource-context access is explicit and separate from serializable diagnostic
records. Querying metadata itself does not read resources. PromptKit owns opening
and interpreting message text, rendering, and handling resource errors. Context
retention must not imply that external resources are snapshotted with the loaded
configuration or remain valid after the caller closes their filesystem.

The context contract covers caller-supplied filesystems and host-file sources.
ConfigKit closes every library-owned filesystem handle before loading returns,
on success or failure, including input files and directory/root handles. Returned
configuration, provenance, and resource-context records retain no such handles
and require no filesystem cleanup by the consumer. Configuration and provenance
remain usable independently of subsequent filesystem access.

Caller-supplied filesystems and readers remain caller-owned. A supplied filesystem
may itself depend on handles whose lifetime the caller manages; retaining its
context does not extend that lifetime or transfer ownership to ConfigKit.

Host-file resource contexts retain the information needed to open resources later,
preserving the host backend's established containment semantics. Later resource
access acquires its own handles: helpers close handles they use internally, and
callers close any resource handles explicitly returned to them. Host contexts
use a passive path-backed filesystem adapter that opens a fresh contained root
for each resource open and releases it with the returned file. Resource
opens observe the backing filesystem at access time and can fail independently
of an earlier successful configuration load. Sources without filesystem context
report its absence explicitly; composite origins expose their contributing
contexts rather than choosing a directory. These rules keep configuration
snapshot ownership separate from resource-access lifetime.

## Target layering model

### Explicit operations, not discovery order

Extend public loading with explicitly supplied lower-precedence configuration
sources, using the same typed definition/result lifecycle as ordinary `Load`.
Support native filesystem, host-file, byte, reader, and programmatic inputs.
The central workflow is bundled defaults overridden by application configuration,
not an unrestricted sequence of user-selectable merge algorithms. Underlays can
form explicitly ordered default tiers; ConfigKit does not discover priorities or
know about PromptKit's catalog tiers. Source kind does not confer priority:
programmatic data can supply the appropriate supported input role without being
misrepresented as CLI flags.

The precedence model, from lowest to highest, is:

**Typed application defaults < source underlays < strictly composed main
configuration < selected overlay < explicit CLI overrides.**

Typed defaults remain the existing Go-value baseline. Source underlays add
configuration-backed defaults with imports, filesystem context, and source
provenance; they are not a replacement for the typed-defaults facility.

An underlay tier is an explicitly supplied group of source contributions. Each
group is strictly composed, including enabled imports, before it participates
in precedence assembly. Consumers supply zero or more tiers in lowest-to-highest
precedence order; later tiers overlay earlier tiers recursively. A source's
position within a group does not authorize replacement. The main input is
optional: an underlay-only load uses the same lifecycle, and a load with neither
main input nor underlays retains defaults-only behavior with supported explicit
overrides. Selecting a named overlay still requires its explicit lookup context.
The public representation exposes grouping and ordering directly. Empty groups
contribute nothing; explicit defaults-only sources also contribute no document.

The selected operation determines how values combine, not their source format
or loading entry point. Equivalent file, embedded, byte/reader, and programmatic
input uses the same presence and combination rules. Strictly composing independent
contributions and intentionally overlaying precedence tiers express different
intentions; merely changing the input medium must not switch between them.
This applies to both ordinary `Load` and
the new layered-loading interface wherever they offer the same operation.
Consistency applies to combination, presence, and origin meaning. Source-specific
acquisition and lifetime rules remain explicit: a reader is single-consumption,
bytes are snapshotted, and filesystem input can observe backing-store changes.
CLI records retain their existing path restrictions. Source equivalence does not
promise identical reuse, locations, resource access, or diagnostic labels.
The existing opt-in version-path ownership rule remains an explicit metadata
constraint, independent of whether its owning source is a file or programmatic
input; it is not a source-format-dependent merge rule.

Keep strict composition and intentional overriding visibly distinct:

| Operation | Desired semantics |
| --- | --- |
| Compose independent contributions | Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations. |
| Overlay settings | Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace. |

Underlay is a precedence role, not another combination algorithm. Assemble the
main document and all its imports by strict composition before applying that
result over the underlays using recursive overlay. A conflicting main/import
definition still fails, even if it would overwrite an underlay or be overwritten
by the selected overlay. Never flatten main and imported documents into a
last-wins list.

An underlay source may bring its explicitly enabled imports and source context.
Independent contributions within an underlay tier compose strictly; intentional
precedence between explicitly ordered tiers uses recursive overlay. Discovery
order does not establish tiers or authorize overriding. Syntax, source, and
composition failures cannot be hidden by higher-priority input.

There is no whole-definition replacement mode, per-layer recursive/atomic switch,
collection-specific identity matching, implicit sequence merge, profile registry,
or merge callback in this feature. Existing atomic scalar, sequence, null, and
cross-kind replacement under overlay remains unchanged.

Preserve the main configuration's version check after its strict composition and
schema construction, before precedence assembly with underlays, selected overlay,
and CLI contributions. Assemble the effective input before typed defaults/decoding,
application resolution, secret resolution, semantic validation, and final
snapshotting. Each lifecycle phase runs once per load, not once per layer.
Higher layers may replace values that would not fit the final typed schema;
effective typed validation is not validation of each intermediate snapshot.
Input syntax, source access, and composition remain validated at acquisition.

Recorded application resolution follows input assembly and decoding. Named-overlay
lookup has an unambiguous filesystem context, separate from version ownership: ordinary
loading retains its source/root convention; a multi-context load explicitly
identifies the lookup context when that convenience is used. Other sources retain
their own import and resource contexts. Callers do not manufacture a primary
document merely to combine layers, and no overlay lookup context is needed when
no named overlay is selected.

Higher-priority mappings retain omitted lower-input children. Empty mappings do
not clear those children; sequences, including empty sequences, replace as a
unit, and null remains explicit subject to the final schema. Typed defaults
continue to fill fields absent from the assembled input under existing rules.
No reset/delete operator is added.

For example, a bundled profile containing `{model: example-model, timeout: 30}`
combined with application input `{model: another-model}` retains `timeout: 30`
from the bundle. Its model origin points to the application and its timeout
origin points to the underlay. PromptKit must validate that final profile rather
than assume the application entry discarded all bundled settings. This knowingly
differs from its requested identity-based whole-definition replacement. Consumer
examples should adopt the shared semantics, not implement a hidden replacement
pass after loading. A replacement operation could be considered separately if
future concrete needs justify it, but is not an open choice in this roadmap.

### Preserve existing schema-version semantics

An application opts into version checking by configuring a version path. A key
spelled `version` has no automatic special status. When checking is configured,
one designated input owns that declaration; fragments cannot contribute it and
underlays, overlays, or CLI overrides cannot supply or alter it, even to an equal
value. Preserve the current version-path and ancestor protections when integrating
the new input role.

Keep the checker at its existing lifecycle point: after base composition and
schema construction, before underlay precedence assembly, selected overlays,
CLI overrides, defaults, and typed decoding. Preserve supported scalar forms,
absent/null distinction,
diagnostics, failure propagation, and version-path integer-to-string decoding.
An absent declaration remains absent for the checker; an invalid version type
fails without invoking it. Later input cannot mask a version-check failure.

Filesystem sources and layered loading do not require a different version
policy. The designated main input owns the version when a `VersionRule` is
configured, extending the existing selected-source role. It may use any supported
source kind. With no main input, the checker receives an absent declaration,
consistent with existing `NoSource` behavior; underlays cannot supply that
declaration. Without version checking, no version owner or synthetic primary
document is required.
Named-overlay lookup context is a separate concern, not a reason to impose
version ownership on unversioned loads.

Consumers continue to define version meaning and acceptance. Effective-version
last-wins validation, independent per-source version checks, schema selection,
and migration are not added by this feature. A future concrete need can motivate
a separate version-policy proposal; preserving the current contract here does
not settle that broader question permanently.

### Programmatic presence

Programmatic input has a non-secret source label and uses explicit structure to
represent presence. Omitted mapping keys are absent; present zero, false, empty
string, empty collection, and nil/null values remain explicit. Use ConfigKit's
existing canonical scalar and collection conventions, not YAML round trips or
Go zero-value inference. The initial representation is a labeled string-keyed
mapping using the scalar/list/map value set supported by explicit overrides.
This reuses established conventions and keeps presence directly expressible.
The source snapshots the mapping at construction, like byte sources. A nil root
mapping represents null and fails the existing mapping-root requirement; an empty
non-nil mapping is a valid empty document.

Do not label generic programmatic sources as CLI input or misuse synthetic flag
names to fit the existing override API. Preserve parser-independent conversion,
safe errors, deterministic map ordering, and existing secret-reference semantics;
this is not a new route for embedding resolved credentials in document trees.

Caller-owned mutable input is detached at a clearly documented ingestion
boundary. Results and inspection remain independently owned under existing
contracts. This does not promise language-enforced immutability of Go values or
an immutable filesystem.

A final Go value alone need not encode presence: an omitted boolean and an
explicit `false` can both decode to `false`. The returned configuration together
with its presence/provenance metadata preserves that distinction through
defaulting, layering, and recorded resolution. Null, empty values, zero, and
absence must not be collapsed by source conversion or merging.

### Provenance and operational boundaries

Origins, presence, history, explanation, and effective rendering agree with the
final typed result. Overridden and replaced definitions remain explainable without
retaining stale effective descendants. Replacements identify their scope and
displaced sources; equal-value explicit overrides are still meaningful history.
Include typed defaults and application-derived values, not only source-supplied
values, in final-value provenance. Retain redaction and callback-lifetime behavior.

Apply existing resource-accounting principles across the whole load rather than
resetting allowances for every layer. Audit how bytes, programmatic nodes,
accepted source occurrences, and glob work are counted without accidental double
charging. Do not add arbitrary new budgets or infer that efficiency work waives
existing ones; any needed adjustment must be explicit in the later design.

## Efficiency target

Measure cost against total acquired bytes/nodes, reference and path data, final
materialized values, and retained provenance/history. Layer count alone is not
a useful size measure. Large history or output can legitimately dominate cost.

Aim for O(N + H + R) processing and storage where N is total input/visited-path
work, H is retained history/provenance, and R is produced result data. Account
separately for deterministic sorting, filesystem traversal, schema processing,
and caller callbacks. This is an engineering target, not an unconditional public
complexity guarantee or a fixed performance gate.

The target implementation:

- Acquires/parses each accepted source occurrence once and directly converts
  programmatic input. It does not decode or serialize accumulated configurations
  between layers. Respect single-consumption readers and mutable file sources;
  no implicit cross-load cache is needed.
- Uses indexed lookup alongside ordered mapping storage, retaining deterministic
  iteration and diagnostics without repeatedly scanning growing mappings.
- Builds one load-owned working assembly instead of cloning the accumulated
  configuration and history for every added layer. Detach at meaningful ownership
  boundaries; safely share immutable private history where appropriate.
- Updates affected subtrees and their provenance rather than scanning all fields
  for every replacement. Avoid repeatedly copying ever-growing ancestor-origin
  lists or rechecking the same conflict group for every member.
- Reuses one schema/lifecycle evaluation per load and freezes the final result
  once. It does not require global caches, copy-on-write public APIs, or a general
  persistent-tree framework merely to meet an asymptotic target.

Use representative benchmarks and allocation measurements for broad keyed
catalogs, many small underlay tiers, repeated scalar/sequence replacement,
imported filesystem defaults, and retained history. Compare growth as inputs
scale, including small common
loads. Favor changes addressing demonstrated scaling problems; preserve stable
diagnostic order and ownership rather than trading them for a superficial O(n)
claim. No timing threshold or additional release gate is requested.

## Work needed from the baseline

The target requires changes in the following responsibility areas; the linked
implementation plan owns their execution order and concrete validation:

- Extend the root-owned source/context facade and private acquisition boundary
  for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling
  without exporting parser or engine types.
- Add source-backed underlays and programmatic input, sharing one engine path
  with ordinary loading rather than building a parallel configuration system.
- Extend source records, histories, diagnostics, and rendering consistently
  across input kinds.
- Integrate version-owner selection into layered loading while preserving the
  existing version lifecycle and protections. Keep filesystem-context selection
  independent of version checking.
- Refactor mapping assembly and provenance updates where current algorithms
  would scale poorly under layered workloads. Keep optimization scope tied to
  the new workflow rather than rewriting unrelated library subsystems.
- Add focused behavioral coverage for mixed sources, relative resources,
  namespace collisions, import failures, composition conflicts, explicit empty
  and null values, caller/result independence, and source-correct redaction and
  explanation. Add comparative benchmarks without making performance a brittle
  pass/fail contract.
- Compare equivalent logical inputs across supported source kinds and loading
  interfaces. Assert consistent values, presence, operation outcomes, and origin
  meaning while allowing genuine location/label differences. Cover defaulted and
  derived values, recursively retained underlay fields, unchanged atomic-value
  replacement, typed defaults, strict main/import conflicts, and preservation of
  version ownership/check timing across both loading interfaces. Verify that
  unversioned layered loads need no synthetic version-owning root.
- Provide maintained simple and PromptKit-shaped examples: embedded defaults,
  operator configuration from another context, and labeled programmatic input.
  Show explicitly that a same-name profile recursively combines, rather than
  replacing its whole lower definition.
  Demonstrate exact operations and resource origins without provider calls or
  recreating PromptKit itself. The consumer supplies sources, explicit operations,
  types, and domain rules; it does not implement generic merging, presence
  tracking, or origin reconstruction after loading.
- Update Go declaration documentation, the configuration contract, consumer
  guidance, component overview, and relevant architecture/deferred-scope records
  as behavior lands. Record significant boundary changes in an ADR without
  rewriting accepted ADR history. Keep named inheritance explicitly separate.

The capability is additive and intended for a minor release, preserving current
version semantics. Release preparation and publication remain separate tasks.

## Completion criteria

A consumer can load embedded configuration with relative imports, combine it
with independently located operator configuration and explicit programmatic
values, and obtain one typed result with accurate presence, sources, history,
and resource context. No temporary configuration files or serialization round
trip or subsequent generic configuration processing is needed. Simple `Load`
usage retains its current behavior, including version ownership and validation
timing. Layered loading uses the same version policy. Final-value provenance covers input,
defaults, and recorded domain resolution, and metadata distinguishes explicit
zero/false from absence even when the typed values are equal. Benchmarks
demonstrate that adding inputs does not entail avoidable full-collection copying
or quadratic key lookup. Public contracts and examples clearly distinguish
strict within-tier composition, recursive cross-tier overlay, and downstream domain
resolution. Equivalent-input tests and the maintained consumer example establish
semantic consistency and reduced integration code, not merely API availability.

## Selected resource-context interface

A separate source-ID-based query exposes standard filesystem access and a
source-relative base location through root-owned API types. Ordinary source
records remain descriptive. The query reports absent context explicitly and
supports imported sources as well as initial inputs. Consumers follow each
contributing origin independently for composite values. This avoids a mandatory
downstream filesystem registry and preserves provenance through recursive
combination.

Caller-supplied filesystems are borrowed capabilities; host contexts are passive
path-backed adapters. Neither is a cloned or immutable filesystem. Queries perform
no I/O, and later resource opens follow the ownership rules above. Configuration
results retain no library-owned open handles.