ConfigKit/docs/roadmap/sources-and-layering.md

Filesystem Sources and Explicit Layered Loading

Status: proposed target for design agreement; not implemented. Significant API and semantic choices are collected under Open Decisions. A later implementation plan will own staged execution, detailed signatures, and acceptance commands.

Purpose and priorities

Implement the two prerequisites in the PromptKit request as one coherent extension: native fs.FS sources and public, explicitly ordered configuration layers, including programmatic input. Neither capability should require temporary files, a generated root YAML document, or serialization through YAML merely to transfer already available values.

Design priorities, in order:

1. Clear, standardized behavior consistent with ConfigKit's existing conventions. PromptKit may adapt its formats and APIs; its historical conventions are not new ConfigKit compatibility obligations.
2. Efficient loading and retained state, aiming for linear or near-linear work in actual input and output size. Prefer clarity and maintainability where a more elaborate optimization would compromise the first priority.
3. A small public surface with cohesive private implementation, usable by simple applications without requiring them to learn the layered-loading machinery.

The named-inheritance request remains separate and optional. This roadmap neither implements it nor adds speculative infrastructure for it.

Current baseline

Planning baseline: ec26591b275bd74f89120bdea69c5f6459dfcafb, following v0.4.2. Current behavior is owned by the configuration contract, not by these proposals.

• Load[T] takes one File, Bytes, Reader, or NoSource input. Imports strictly compose documents; one named overlay uses a root-relative directory convention; explicit CLI overrides follow it.
• Filesystem context uses host directories and an os.Root-based implementation. Byte and reader labels do not imply filesystem paths. There is no native fs.FS source or public batch of heterogeneous layers.
• Composition rejects non-mapping overlaps, including equal values. Overlay recursively merges mappings and replaces other values. Empty mappings do not clear existing children; null is an explicit value, not omission.
• CLI overrides already convert supported programmatic scalars, lists, and string-keyed maps directly to the private document representation. They have CLI-specific paths, labels, restrictions, and origins, not a general source API.
• Source records and resolution queries expose acquired-source identity and provenance, but do not provide an fs.FS resource context.
• Root-owned public wrappers delegate to internal/engine; parser-independent trees belong to internal/document, and YAML parsing stays in internal/yaml.
• Current mapping assembly uses repeated linear key searches. Overlay application clones the accumulated document and provenance; subtree-origin removal scans the whole origin index. Straightforward repetition across many layers would amplify these costs.

Existing source/lifecycle contracts, ownership, secret handling, and diagnostics remain the compatibility baseline. This work is not permission to silently reinterpret current Load, imports, selected overlays, or CLI overrides.

Target source model

Native filesystem input

A caller can supply a filesystem, an explicit file location within it, and a meaningful diagnostic label/context. embed.FS is an ordinary supported example, not a separate parsing mode. Use standard slash-separated io/fs path conventions; do not apply host drive-letter or separator rules to virtual paths.

The source retains the filesystem context needed to acquire its imports, glob matches, and selected overlay. Relative imports resolve from the importing document; named overlay lookup remains relative to that configuration context's root. References must never fall back to the host filesystem simply because a virtual path is missing. Imports/globs retain their explicit opt-in, deterministic ordering, composition semantics, and actionable errors.

Filesystem namespace identity and a normalized path identify a resource; labels are presentation, not uniqueness keys. Identical names in different filesystems must not collide. Use a reusable load-local context identity, not comparison or hashing of arbitrary fs.FS interface values. Repeated-import and cycle checks operate within the relevant input composition graph; selecting explicit layers is distinct from accidentally importing a file twice.

Keep the host-file backend's established canonicalization and containment behavior. For generic fs.FS, describe namespace-relative resolution and the supplied filesystem's actual behavior. fs.FS does not universally provide symlink canonicalization or a security sandbox; do not invent such guarantees or reject all implementations that cannot provide host-specific operations. The caller chooses the filesystem, its acquisition behavior, and its lifetime. ConfigKit closes handles it opens, not a caller-owned filesystem as a whole.

Origins and downstream resources

Locations remain meaningful for host files, filesystem-relative files, labeled bytes, and programmatic values. Preserve deterministic source IDs and traversal order while distinguishing source identity, input kind, and precedence role. Do not invent host filenames or line/column coordinates for synthetic inputs.

A consumer can follow an effective field's origin to its contributing source and obtain the context needed to resolve a relative resource. For a field inherited through layering, that is the contributing source's context, not simply the last layer's directory. Composite origins remain composite; ConfigKit must not choose one contributing directory arbitrarily.

Resource-context access is explicit and separate from serializable diagnostic records. Querying metadata itself does not read resources. PromptKit owns opening and interpreting message text, rendering, and handling resource errors. Context retention must not imply that external resources are snapshotted with the loaded configuration or remain valid after the caller closes their filesystem.

Target layering model

Explicit operations, not discovery order

Expose a public loading entry point for an explicit ordered set of sources and operations, using the same typed definition/result lifecycle as ordinary Load. Support native filesystem, host-file, byte, reader, and programmatic inputs. The order is supplied by the caller; ConfigKit does not discover priorities or know about PromptKit's catalog tiers.

Keep strict composition and intentional overriding visibly distinct:

Operation	Desired semantics
Compose independent contributions	Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations.
Overlay settings	Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace.
Replace a definition	Replace the selected input subtree as a unit, without carrying over omitted lower-input children. The inclusion and public expression of this operation are Decision 1.

Each independently supplied root may bring its explicitly enabled imports and source context. Its import graph is strictly composed before intentional precedence is applied. An overlay must not hide malformed YAML or a composition conflict in an earlier contribution. There is no automatic matching by an id field, implicit sequence merge, universal profile registry, or merge callback.

Assemble the effective input before applying the typed defaults/decoding, application resolution, secret resolution, semantic validation, and final snapshot lifecycle. Those latter operations run once per load, not once per layer. Higher layers may replace values that would not fit the final typed schema; effective typed validation is not validation of each intermediate snapshot. Input syntax, source access, and composition remain validated at acquisition.

The standard precedence relationship is typed defaults below assembled input, then the existing selected-overlay convenience and explicit CLI overrides where those are used, followed by recorded application resolution. The new API must make the selected overlay's owning filesystem context unambiguous; Decision 4 addresses the primary-root role alongside schema-version ownership.

Replacing lower input does not mean resetting or deleting typed application defaults. Existing defaults and null rules still apply during final decoding. The distinction must be visible in examples, especially empty maps and partially specified definitions. No new reset/delete operator is implied by replacement.

Programmatic presence

Programmatic input has a non-secret source label and uses explicit structure to represent presence. Omitted mapping keys are absent; present zero, false, empty string, empty collection, and nil/null values remain explicit. Use ConfigKit's existing canonical scalar and collection conventions, not YAML round trips or Go zero-value inference. Decision 2 selects the initial input representation.

Do not label generic programmatic sources as CLI input or misuse synthetic flag names to fit the existing override API. Preserve parser-independent conversion, safe errors, deterministic map ordering, and existing secret-reference semantics; this is not a new route for embedding resolved credentials in document trees.

Caller-owned mutable input is detached at a clearly documented ingestion boundary. Results and inspection remain independently owned under existing contracts. This does not promise language-enforced immutability of Go values or an immutable filesystem.

Provenance and operational boundaries

Origins, presence, history, explanation, and effective rendering agree with the final typed result. Overridden and replaced definitions remain explainable without retaining stale effective descendants. Replacements identify their scope and displaced sources; equal-value explicit overrides are still meaningful history. Retain redaction and callback-lifetime behavior.

Apply existing resource-accounting principles across the whole load rather than resetting allowances for every layer. Audit how bytes, programmatic nodes, accepted source occurrences, and glob work are counted without accidental double charging. Do not add arbitrary new budgets or infer that efficiency work waives existing ones; any needed adjustment must be explicit in the later design.

Efficiency target

Measure cost against total acquired bytes/nodes, reference and path data, final materialized values, and retained provenance/history. Layer count alone is not a useful size measure. Large history or output can legitimately dominate cost.

Aim for O(N + H + R) processing and storage where N is total input/visited-path work, H is retained history/provenance, and R is produced result data. Account separately for deterministic sorting, filesystem traversal, schema processing, and caller callbacks. This is an engineering target, not an unconditional public complexity guarantee or a fixed performance gate.

The target implementation:

• Acquires/parses each accepted source occurrence once and directly converts programmatic input. It does not decode or serialize accumulated configurations between layers. Respect single-consumption readers and mutable file sources; no implicit cross-load cache is needed.
• Uses indexed lookup alongside ordered mapping storage, retaining deterministic iteration and diagnostics without repeatedly scanning growing mappings.
• Builds one load-owned working assembly instead of cloning the accumulated configuration and history for every added layer. Detach at meaningful ownership boundaries; safely share immutable private history where appropriate.
• Updates affected subtrees and their provenance rather than scanning all fields for every replacement. Avoid repeatedly copying ever-growing ancestor-origin lists or rechecking the same conflict group for every member.
• Reuses one schema/lifecycle evaluation per load and freezes the final result once. It does not require global caches, copy-on-write public APIs, or a general persistent-tree framework merely to meet an asymptotic target.

Use representative benchmarks and allocation measurements for broad keyed catalogs, many small layers, repeated replacement, imported filesystem defaults, and retained history. Compare growth as inputs scale, including small common loads. Favor changes addressing demonstrated scaling problems; preserve stable diagnostic order and ownership rather than trading them for a superficial O(n) claim. No timing threshold or additional release gate is requested.

Work needed from the baseline

These are responsibility areas, not implementation stages:

• Extend the root-owned source/context facade and private acquisition boundary for fs.FS; adapt import, glob, overlay, identity, and resource-context handling without exporting parser or engine types.
• Add explicit layered loading and programmatic input, sharing one engine path with ordinary loading rather than building a parallel configuration system.
• Resolve the fixed operation and ownership decisions below, then extend source records, histories, diagnostics, and rendering consistently across input kinds.
• Refactor mapping assembly and provenance updates where current algorithms would scale poorly under layered workloads. Keep optimization scope tied to the new workflow rather than rewriting unrelated library subsystems.
• Add focused behavioral coverage for mixed sources, relative resources, namespace collisions, import failures, composition conflicts, explicit empty and null values, caller/result independence, and source-correct redaction and explanation. Add comparative benchmarks without making performance a brittle pass/fail contract.
• Provide maintained simple and PromptKit-shaped examples: embedded defaults, operator configuration from another context, and labeled programmatic input. Demonstrate exact operations and resource origins without provider calls or recreating PromptKit itself.
• Update Go declaration documentation, the configuration contract, consumer guidance, component overview, and relevant architecture/deferred-scope records as behavior lands. Record significant boundary changes in an ADR without rewriting accepted ADR history. Keep named inheritance explicitly separate.

Plan an additive minor release for the new public capability. If any design choice requires changing an existing documented behavior, identify it expressly and provide migration guidance; do not describe it as an internal optimization. No release, implementation change, or current-policy revision occurs in this pass.

Completion criteria

A consumer can load embedded configuration with relative imports, combine it with independently located operator configuration and explicit programmatic values, and obtain one typed result with accurate presence, sources, history, and resource context. No temporary configuration files or serialization round trip is needed. Simple Load usage retains its current behavior. Benchmarks demonstrate that adding inputs does not entail avoidable full-collection copying or quadratic key lookup. Public contracts and examples clearly distinguish composition, overlay, any selected replacement operation, and downstream domain resolution.

Open Decisions

1. Whole-definition replacement

Recommendation: include a small fixed replacement operation alongside compose and recursive overlay. For PromptKit-shaped catalogs, explicitly identify a string-keyed collection path and replace each supplied entry as a whole; retain unmentioned entries. Do not infer identities from object fields. An empty supplied collection contributes no entries; an explicitly supplied empty entry replaces that entry's lower input. Use the same rule for every consumer, not arbitrary per-field policies. This meets the concrete identity-precedence need without silently redefining ordinary overlay. Final typed defaults still apply.

Alternative: ship only existing composition/overlay semantics and leave whole-definition selection downstream. This is smaller and closest to today's surface, but may force consumers to parse/select definitions and reconstruct origins outside ConfigKit, undermining the request's purpose. Confirm with the combined consumer example before claiming this narrower scope meets the need.

2. Initial programmatic representation

Recommendation: labeled string-keyed mappings with the scalar/list/map value set already supported by explicit overrides. Missing keys express omission and nil expresses null. Reuse private conversion machinery, not CLI-specific public records. This is predictable, directly expressible in Go, and consistent with current dynamic configuration semantics.

Alternative: typed Go values with a separate explicit-presence description. This can reduce downstream mapping code, but introduces reconciliation of field bindings, masks, nulls, and typed zero values and a larger public contract. Do not infer omission from zero values. Add typed construction only if real examples show that the simple mapping representation imposes substantial repetitive work.

3. How consumers obtain filesystem resource context

Recommendation: a separate source-context query exposing standard filesystem access and a source-relative base location through root-owned API types, while ordinary source records remain descriptive. The returned filesystem is a borrowed capability, not a cloned or immutable object. This avoids a mandatory downstream registry and supports origins from imports as well as initial sources. Make lifetime and nil/no-context behavior explicit before implementation.

Alternative: expose opaque context IDs and relative paths; callers retain a registry mapping IDs to their filesystems. Metadata stays capability-free and serializable, but every consumer must maintain and correctly reconcile another registry. This is viable if limiting retained filesystem capabilities proves more important than the additional integration machinery.

4. Primary-root ownership in multi-source loads

Recommendation: retain an explicit primary source/root role for application schema version and the optional named-overlay convention. Additional inputs are fragments or explicit precedence contributions, not independent version owners. This best preserves current version ownership and prevents precedence from hiding a version failure. An in-memory primary mapping can carry the version without requiring a generated file. Other source contexts still govern their own imports and resource references.

Alternative: permit independently versioned roots and validate each before assembly, with a separately selected context for named-overlay lookup. This fits standalone catalogs more naturally, but needs new rules for combining accepted versions, callback counts, and the effective version's provenance. It should not silently invoke the existing version callback once per layer or let the last version value win. Resolve this choice with a realistic mixed-catalog example before producing the staged implementation plan.