Document filesystem sources and layered loading roadmap

docs/roadmap/sources-and-layering.md

@@ -0,0 +1,332 @@
+# Filesystem Sources and Explicit Layered Loading
+
+Status: proposed target for design agreement; not implemented. Significant API
+and semantic choices are collected under Open Decisions. A later implementation
+plan will own staged execution, detailed signatures, and acceptance commands.
+
+## Purpose and priorities
+
+Implement the two prerequisites in the
+[PromptKit request](feature-request-promptkit.md) as one coherent extension:
+native `fs.FS` sources and public, explicitly ordered configuration layers,
+including programmatic input. Neither capability should require temporary files,
+a generated root YAML document, or serialization through YAML merely to transfer
+already available values.
+
+Design priorities, in order:
+
+1. Clear, standardized behavior consistent with ConfigKit's existing conventions.
+   PromptKit may adapt its formats and APIs; its historical conventions are not
+   new ConfigKit compatibility obligations.
+2. Efficient loading and retained state, aiming for linear or near-linear work
+   in actual input and output size. Prefer clarity and maintainability where a
+   more elaborate optimization would compromise the first priority.
+3. A small public surface with cohesive private implementation, usable by simple
+   applications without requiring them to learn the layered-loading machinery.
+
+The [named-inheritance request](feature-request-promptkit-inheritance.md) remains
+separate and optional. This roadmap neither implements it nor adds speculative
+infrastructure for it.
+
+## Current baseline
+
+Planning baseline: `ec26591b275bd74f89120bdea69c5f6459dfcafb`, following v0.4.2.
+Current behavior is owned by the [configuration contract](../config.md), not by
+these proposals.
+
+- `Load[T]` takes one `File`, `Bytes`, `Reader`, or `NoSource` input. Imports
+  strictly compose documents; one named overlay uses a root-relative directory
+  convention; explicit CLI overrides follow it.
+- Filesystem context uses host directories and an `os.Root`-based implementation.
+  Byte and reader labels do not imply filesystem paths. There is no native
+  `fs.FS` source or public batch of heterogeneous layers.
+- Composition rejects non-mapping overlaps, including equal values. Overlay
+  recursively merges mappings and replaces other values. Empty mappings do not
+  clear existing children; null is an explicit value, not omission.
+- CLI overrides already convert supported programmatic scalars, lists, and
+  string-keyed maps directly to the private document representation. They have
+  CLI-specific paths, labels, restrictions, and origins, not a general source API.
+- Source records and resolution queries expose acquired-source identity and
+  provenance, but do not provide an `fs.FS` resource context.
+- Root-owned public wrappers delegate to `internal/engine`; parser-independent
+  trees belong to `internal/document`, and YAML parsing stays in `internal/yaml`.
+- Current mapping assembly uses repeated linear key searches. Overlay application
+  clones the accumulated document and provenance; subtree-origin removal scans
+  the whole origin index. Straightforward repetition across many layers would
+  amplify these costs.
+
+Existing source/lifecycle contracts, ownership, secret handling, and diagnostics
+remain the compatibility baseline. This work is not permission to silently
+reinterpret current `Load`, imports, selected overlays, or CLI overrides.
+
+## Target source model
+
+### Native filesystem input
+
+A caller can supply a filesystem, an explicit file location within it, and a
+meaningful diagnostic label/context. `embed.FS` is an ordinary supported example,
+not a separate parsing mode. Use standard slash-separated `io/fs` path conventions;
+do not apply host drive-letter or separator rules to virtual paths.
+
+The source retains the filesystem context needed to acquire its imports, glob
+matches, and selected overlay. Relative imports resolve from the importing
+document; named overlay lookup remains relative to that configuration context's
+root. References must never fall back to the host filesystem simply because a
+virtual path is missing. Imports/globs retain their explicit opt-in, deterministic
+ordering, composition semantics, and actionable errors.
+
+Filesystem namespace identity and a normalized path identify a resource; labels
+are presentation, not uniqueness keys. Identical names in different filesystems
+must not collide. Use a reusable load-local context identity, not comparison or
+hashing of arbitrary `fs.FS` interface values. Repeated-import and cycle checks
+operate within the relevant input composition graph; selecting explicit layers
+is distinct from accidentally importing a file twice.
+
+Keep the host-file backend's established canonicalization and containment behavior.
+For generic `fs.FS`, describe namespace-relative resolution and the supplied
+filesystem's actual behavior. `fs.FS` does not universally provide symlink
+canonicalization or a security sandbox; do not invent such guarantees or reject
+all implementations that cannot provide host-specific operations. The caller
+chooses the filesystem, its acquisition behavior, and its lifetime. ConfigKit
+closes handles it opens, not a caller-owned filesystem as a whole.
+
+### Origins and downstream resources
+
+Locations remain meaningful for host files, filesystem-relative files, labeled
+bytes, and programmatic values. Preserve deterministic source IDs and traversal
+order while distinguishing source identity, input kind, and precedence role.
+Do not invent host filenames or line/column coordinates for synthetic inputs.
+
+A consumer can follow an effective field's origin to its contributing source and
+obtain the context needed to resolve a relative resource. For a field inherited
+through layering, that is the contributing source's context, not simply the last
+layer's directory. Composite origins remain composite; ConfigKit must not choose
+one contributing directory arbitrarily.
+
+Resource-context access is explicit and separate from serializable diagnostic
+records. Querying metadata itself does not read resources. PromptKit owns opening
+and interpreting message text, rendering, and handling resource errors. Context
+retention must not imply that external resources are snapshotted with the loaded
+configuration or remain valid after the caller closes their filesystem.
+
+## Target layering model
+
+### Explicit operations, not discovery order
+
+Expose a public loading entry point for an explicit ordered set of sources and
+operations, using the same typed definition/result lifecycle as ordinary `Load`.
+Support native filesystem, host-file, byte, reader, and programmatic inputs.
+The order is supplied by the caller; ConfigKit does not discover priorities or
+know about PromptKit's catalog tiers.
+
+Keep strict composition and intentional overriding visibly distinct:
+
+| Operation | Desired semantics |
+| --- | --- |
+| Compose independent contributions | Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations. |
+| Overlay settings | Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace. |
+| Replace a definition | Replace the selected input subtree as a unit, without carrying over omitted lower-input children. The inclusion and public expression of this operation are Decision 1. |
+
+Each independently supplied root may bring its explicitly enabled imports and
+source context. Its import graph is strictly composed before intentional
+precedence is applied. An overlay must not hide malformed YAML or a composition
+conflict in an earlier contribution. There is no automatic matching by an `id`
+field, implicit sequence merge, universal profile registry, or merge callback.
+
+Assemble the effective input before applying the typed defaults/decoding,
+application resolution, secret resolution, semantic validation, and final
+snapshot lifecycle. Those latter operations run once per load, not once per
+layer. Higher layers may replace values that would not fit the final typed schema;
+effective typed validation is not validation of each intermediate snapshot.
+Input syntax, source access, and composition remain validated at acquisition.
+
+The standard precedence relationship is typed defaults below assembled input,
+then the existing selected-overlay convenience and explicit CLI overrides where
+those are used, followed by recorded application resolution. The new API must
+make the selected overlay's owning filesystem context unambiguous; Decision 4
+addresses the primary-root role alongside schema-version ownership.
+
+Replacing lower input does not mean resetting or deleting typed application
+defaults. Existing defaults and null rules still apply during final decoding.
+The distinction must be visible in examples, especially empty maps and partially
+specified definitions. No new reset/delete operator is implied by replacement.
+
+### Programmatic presence
+
+Programmatic input has a non-secret source label and uses explicit structure to
+represent presence. Omitted mapping keys are absent; present zero, false, empty
+string, empty collection, and nil/null values remain explicit. Use ConfigKit's
+existing canonical scalar and collection conventions, not YAML round trips or
+Go zero-value inference. Decision 2 selects the initial input representation.
+
+Do not label generic programmatic sources as CLI input or misuse synthetic flag
+names to fit the existing override API. Preserve parser-independent conversion,
+safe errors, deterministic map ordering, and existing secret-reference semantics;
+this is not a new route for embedding resolved credentials in document trees.
+
+Caller-owned mutable input is detached at a clearly documented ingestion
+boundary. Results and inspection remain independently owned under existing
+contracts. This does not promise language-enforced immutability of Go values or
+an immutable filesystem.
+
+### Provenance and operational boundaries
+
+Origins, presence, history, explanation, and effective rendering agree with the
+final typed result. Overridden and replaced definitions remain explainable without
+retaining stale effective descendants. Replacements identify their scope and
+displaced sources; equal-value explicit overrides are still meaningful history.
+Retain redaction and callback-lifetime behavior.
+
+Apply existing resource-accounting principles across the whole load rather than
+resetting allowances for every layer. Audit how bytes, programmatic nodes,
+accepted source occurrences, and glob work are counted without accidental double
+charging. Do not add arbitrary new budgets or infer that efficiency work waives
+existing ones; any needed adjustment must be explicit in the later design.
+
+## Efficiency target
+
+Measure cost against total acquired bytes/nodes, reference and path data, final
+materialized values, and retained provenance/history. Layer count alone is not
+a useful size measure. Large history or output can legitimately dominate cost.
+
+Aim for O(N + H + R) processing and storage where N is total input/visited-path
+work, H is retained history/provenance, and R is produced result data. Account
+separately for deterministic sorting, filesystem traversal, schema processing,
+and caller callbacks. This is an engineering target, not an unconditional public
+complexity guarantee or a fixed performance gate.
+
+The target implementation:
+
+- Acquires/parses each accepted source occurrence once and directly converts
+  programmatic input. It does not decode or serialize accumulated configurations
+  between layers. Respect single-consumption readers and mutable file sources;
+  no implicit cross-load cache is needed.
+- Uses indexed lookup alongside ordered mapping storage, retaining deterministic
+  iteration and diagnostics without repeatedly scanning growing mappings.
+- Builds one load-owned working assembly instead of cloning the accumulated
+  configuration and history for every added layer. Detach at meaningful ownership
+  boundaries; safely share immutable private history where appropriate.
+- Updates affected subtrees and their provenance rather than scanning all fields
+  for every replacement. Avoid repeatedly copying ever-growing ancestor-origin
+  lists or rechecking the same conflict group for every member.
+- Reuses one schema/lifecycle evaluation per load and freezes the final result
+  once. It does not require global caches, copy-on-write public APIs, or a general
+  persistent-tree framework merely to meet an asymptotic target.
+
+Use representative benchmarks and allocation measurements for broad keyed
+catalogs, many small layers, repeated replacement, imported filesystem defaults,
+and retained history. Compare growth as inputs scale, including small common
+loads. Favor changes addressing demonstrated scaling problems; preserve stable
+diagnostic order and ownership rather than trading them for a superficial O(n)
+claim. No timing threshold or additional release gate is requested.
+
+## Work needed from the baseline
+
+These are responsibility areas, not implementation stages:
+
+- Extend the root-owned source/context facade and private acquisition boundary
+  for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling
+  without exporting parser or engine types.
+- Add explicit layered loading and programmatic input, sharing one engine path
+  with ordinary loading rather than building a parallel configuration system.
+- Resolve the fixed operation and ownership decisions below, then extend source
+  records, histories, diagnostics, and rendering consistently across input kinds.
+- Refactor mapping assembly and provenance updates where current algorithms
+  would scale poorly under layered workloads. Keep optimization scope tied to
+  the new workflow rather than rewriting unrelated library subsystems.
+- Add focused behavioral coverage for mixed sources, relative resources,
+  namespace collisions, import failures, composition conflicts, explicit empty
+  and null values, caller/result independence, and source-correct redaction and
+  explanation. Add comparative benchmarks without making performance a brittle
+  pass/fail contract.
+- Provide maintained simple and PromptKit-shaped examples: embedded defaults,
+  operator configuration from another context, and labeled programmatic input.
+  Demonstrate exact operations and resource origins without provider calls or
+  recreating PromptKit itself.
+- Update Go declaration documentation, the configuration contract, consumer
+  guidance, component overview, and relevant architecture/deferred-scope records
+  as behavior lands. Record significant boundary changes in an ADR without
+  rewriting accepted ADR history. Keep named inheritance explicitly separate.
+
+Plan an additive minor release for the new public capability. If any design choice
+requires changing an existing documented behavior, identify it expressly and
+provide migration guidance; do not describe it as an internal optimization.
+No release, implementation change, or current-policy revision occurs in this pass.
+
+## Completion criteria
+
+A consumer can load embedded configuration with relative imports, combine it
+with independently located operator configuration and explicit programmatic
+values, and obtain one typed result with accurate presence, sources, history,
+and resource context. No temporary configuration files or serialization round
+trip is needed. Simple `Load` usage retains its current behavior. Benchmarks
+demonstrate that adding inputs does not entail avoidable full-collection copying
+or quadratic key lookup. Public contracts and examples clearly distinguish
+composition, overlay, any selected replacement operation, and downstream domain
+resolution.
+
+## Open Decisions
+
+### 1. Whole-definition replacement
+
+**Recommendation:** include a small fixed replacement operation alongside compose
+and recursive overlay. For PromptKit-shaped catalogs, explicitly identify a
+string-keyed collection path and replace each supplied entry as a whole; retain
+unmentioned entries. Do not infer identities from object fields. An empty supplied
+collection contributes no entries; an explicitly supplied empty entry replaces
+that entry's lower input. Use the same rule for every consumer, not arbitrary
+per-field policies. This meets the concrete identity-precedence need without
+silently redefining ordinary overlay. Final typed defaults still apply.
+
+**Alternative:** ship only existing composition/overlay semantics and leave
+whole-definition selection downstream. This is smaller and closest to today's
+surface, but may force consumers to parse/select definitions and reconstruct
+origins outside ConfigKit, undermining the request's purpose. Confirm with the
+combined consumer example before claiming this narrower scope meets the need.
+
+### 2. Initial programmatic representation
+
+**Recommendation:** labeled string-keyed mappings with the scalar/list/map value
+set already supported by explicit overrides. Missing keys express omission and
+nil expresses null. Reuse private conversion machinery, not CLI-specific public
+records. This is predictable, directly expressible in Go, and consistent with
+current dynamic configuration semantics.
+
+**Alternative:** typed Go values with a separate explicit-presence description.
+This can reduce downstream mapping code, but introduces reconciliation of field
+bindings, masks, nulls, and typed zero values and a larger public contract. Do not
+infer omission from zero values. Add typed construction only if real examples
+show that the simple mapping representation imposes substantial repetitive work.
+
+### 3. How consumers obtain filesystem resource context
+
+**Recommendation:** a separate source-context query exposing standard filesystem
+access and a source-relative base location through root-owned API types, while
+ordinary source records remain descriptive. The returned filesystem is a borrowed
+capability, not a cloned or immutable object. This avoids a mandatory downstream
+registry and supports origins from imports as well as initial sources. Make
+lifetime and nil/no-context behavior explicit before implementation.
+
+**Alternative:** expose opaque context IDs and relative paths; callers retain a
+registry mapping IDs to their filesystems. Metadata stays capability-free and
+serializable, but every consumer must maintain and correctly reconcile another
+registry. This is viable if limiting retained filesystem capabilities proves
+more important than the additional integration machinery.
+
+### 4. Primary-root ownership in multi-source loads
+
+**Recommendation:** retain an explicit primary source/root role for application
+schema version and the optional named-overlay convention. Additional inputs are
+fragments or explicit precedence contributions, not independent version owners.
+This best preserves current version ownership and prevents precedence from hiding
+a version failure. An in-memory primary mapping can carry the version without
+requiring a generated file. Other source contexts still govern their own imports
+and resource references.
+
+**Alternative:** permit independently versioned roots and validate each before
+assembly, with a separately selected context for named-overlay lookup. This fits
+standalone catalogs more naturally, but needs new rules for combining accepted
+versions, callback counts, and the effective version's provenance. It should not
+silently invoke the existing version callback once per layer or let the last
+version value win. Resolve this choice with a realistic mixed-catalog example
+before producing the staged implementation plan.