diff --git a/docs/roadmap/sources-and-layering.md b/docs/roadmap/sources-and-layering.md new file mode 100644 index 0000000..6fb57c8 --- /dev/null +++ b/docs/roadmap/sources-and-layering.md @@ -0,0 +1,332 @@ +# Filesystem Sources and Explicit Layered Loading + +Status: proposed target for design agreement; not implemented. Significant API +and semantic choices are collected under Open Decisions. A later implementation +plan will own staged execution, detailed signatures, and acceptance commands. + +## Purpose and priorities + +Implement the two prerequisites in the +[PromptKit request](feature-request-promptkit.md) as one coherent extension: +native `fs.FS` sources and public, explicitly ordered configuration layers, +including programmatic input. Neither capability should require temporary files, +a generated root YAML document, or serialization through YAML merely to transfer +already available values. + +Design priorities, in order: + +1. Clear, standardized behavior consistent with ConfigKit's existing conventions. + PromptKit may adapt its formats and APIs; its historical conventions are not + new ConfigKit compatibility obligations. +2. Efficient loading and retained state, aiming for linear or near-linear work + in actual input and output size. Prefer clarity and maintainability where a + more elaborate optimization would compromise the first priority. +3. A small public surface with cohesive private implementation, usable by simple + applications without requiring them to learn the layered-loading machinery. + +The [named-inheritance request](feature-request-promptkit-inheritance.md) remains +separate and optional. This roadmap neither implements it nor adds speculative +infrastructure for it. + +## Current baseline + +Planning baseline: `ec26591b275bd74f89120bdea69c5f6459dfcafb`, following v0.4.2. +Current behavior is owned by the [configuration contract](../config.md), not by +these proposals. + +- `Load[T]` takes one `File`, `Bytes`, `Reader`, or `NoSource` input. Imports + strictly compose documents; one named overlay uses a root-relative directory + convention; explicit CLI overrides follow it. +- Filesystem context uses host directories and an `os.Root`-based implementation. + Byte and reader labels do not imply filesystem paths. There is no native + `fs.FS` source or public batch of heterogeneous layers. +- Composition rejects non-mapping overlaps, including equal values. Overlay + recursively merges mappings and replaces other values. Empty mappings do not + clear existing children; null is an explicit value, not omission. +- CLI overrides already convert supported programmatic scalars, lists, and + string-keyed maps directly to the private document representation. They have + CLI-specific paths, labels, restrictions, and origins, not a general source API. +- Source records and resolution queries expose acquired-source identity and + provenance, but do not provide an `fs.FS` resource context. +- Root-owned public wrappers delegate to `internal/engine`; parser-independent + trees belong to `internal/document`, and YAML parsing stays in `internal/yaml`. +- Current mapping assembly uses repeated linear key searches. Overlay application + clones the accumulated document and provenance; subtree-origin removal scans + the whole origin index. Straightforward repetition across many layers would + amplify these costs. + +Existing source/lifecycle contracts, ownership, secret handling, and diagnostics +remain the compatibility baseline. This work is not permission to silently +reinterpret current `Load`, imports, selected overlays, or CLI overrides. + +## Target source model + +### Native filesystem input + +A caller can supply a filesystem, an explicit file location within it, and a +meaningful diagnostic label/context. `embed.FS` is an ordinary supported example, +not a separate parsing mode. Use standard slash-separated `io/fs` path conventions; +do not apply host drive-letter or separator rules to virtual paths. + +The source retains the filesystem context needed to acquire its imports, glob +matches, and selected overlay. Relative imports resolve from the importing +document; named overlay lookup remains relative to that configuration context's +root. References must never fall back to the host filesystem simply because a +virtual path is missing. Imports/globs retain their explicit opt-in, deterministic +ordering, composition semantics, and actionable errors. + +Filesystem namespace identity and a normalized path identify a resource; labels +are presentation, not uniqueness keys. Identical names in different filesystems +must not collide. Use a reusable load-local context identity, not comparison or +hashing of arbitrary `fs.FS` interface values. Repeated-import and cycle checks +operate within the relevant input composition graph; selecting explicit layers +is distinct from accidentally importing a file twice. + +Keep the host-file backend's established canonicalization and containment behavior. +For generic `fs.FS`, describe namespace-relative resolution and the supplied +filesystem's actual behavior. `fs.FS` does not universally provide symlink +canonicalization or a security sandbox; do not invent such guarantees or reject +all implementations that cannot provide host-specific operations. The caller +chooses the filesystem, its acquisition behavior, and its lifetime. ConfigKit +closes handles it opens, not a caller-owned filesystem as a whole. + +### Origins and downstream resources + +Locations remain meaningful for host files, filesystem-relative files, labeled +bytes, and programmatic values. Preserve deterministic source IDs and traversal +order while distinguishing source identity, input kind, and precedence role. +Do not invent host filenames or line/column coordinates for synthetic inputs. + +A consumer can follow an effective field's origin to its contributing source and +obtain the context needed to resolve a relative resource. For a field inherited +through layering, that is the contributing source's context, not simply the last +layer's directory. Composite origins remain composite; ConfigKit must not choose +one contributing directory arbitrarily. + +Resource-context access is explicit and separate from serializable diagnostic +records. Querying metadata itself does not read resources. PromptKit owns opening +and interpreting message text, rendering, and handling resource errors. Context +retention must not imply that external resources are snapshotted with the loaded +configuration or remain valid after the caller closes their filesystem. + +## Target layering model + +### Explicit operations, not discovery order + +Expose a public loading entry point for an explicit ordered set of sources and +operations, using the same typed definition/result lifecycle as ordinary `Load`. +Support native filesystem, host-file, byte, reader, and programmatic inputs. +The order is supplied by the caller; ConfigKit does not discover priorities or +know about PromptKit's catalog tiers. + +Keep strict composition and intentional overriding visibly distinct: + +| Operation | Desired semantics | +| --- | --- | +| Compose independent contributions | Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations. | +| Overlay settings | Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace. | +| Replace a definition | Replace the selected input subtree as a unit, without carrying over omitted lower-input children. The inclusion and public expression of this operation are Decision 1. | + +Each independently supplied root may bring its explicitly enabled imports and +source context. Its import graph is strictly composed before intentional +precedence is applied. An overlay must not hide malformed YAML or a composition +conflict in an earlier contribution. There is no automatic matching by an `id` +field, implicit sequence merge, universal profile registry, or merge callback. + +Assemble the effective input before applying the typed defaults/decoding, +application resolution, secret resolution, semantic validation, and final +snapshot lifecycle. Those latter operations run once per load, not once per +layer. Higher layers may replace values that would not fit the final typed schema; +effective typed validation is not validation of each intermediate snapshot. +Input syntax, source access, and composition remain validated at acquisition. + +The standard precedence relationship is typed defaults below assembled input, +then the existing selected-overlay convenience and explicit CLI overrides where +those are used, followed by recorded application resolution. The new API must +make the selected overlay's owning filesystem context unambiguous; Decision 4 +addresses the primary-root role alongside schema-version ownership. + +Replacing lower input does not mean resetting or deleting typed application +defaults. Existing defaults and null rules still apply during final decoding. +The distinction must be visible in examples, especially empty maps and partially +specified definitions. No new reset/delete operator is implied by replacement. + +### Programmatic presence + +Programmatic input has a non-secret source label and uses explicit structure to +represent presence. Omitted mapping keys are absent; present zero, false, empty +string, empty collection, and nil/null values remain explicit. Use ConfigKit's +existing canonical scalar and collection conventions, not YAML round trips or +Go zero-value inference. Decision 2 selects the initial input representation. + +Do not label generic programmatic sources as CLI input or misuse synthetic flag +names to fit the existing override API. Preserve parser-independent conversion, +safe errors, deterministic map ordering, and existing secret-reference semantics; +this is not a new route for embedding resolved credentials in document trees. + +Caller-owned mutable input is detached at a clearly documented ingestion +boundary. Results and inspection remain independently owned under existing +contracts. This does not promise language-enforced immutability of Go values or +an immutable filesystem. + +### Provenance and operational boundaries + +Origins, presence, history, explanation, and effective rendering agree with the +final typed result. Overridden and replaced definitions remain explainable without +retaining stale effective descendants. Replacements identify their scope and +displaced sources; equal-value explicit overrides are still meaningful history. +Retain redaction and callback-lifetime behavior. + +Apply existing resource-accounting principles across the whole load rather than +resetting allowances for every layer. Audit how bytes, programmatic nodes, +accepted source occurrences, and glob work are counted without accidental double +charging. Do not add arbitrary new budgets or infer that efficiency work waives +existing ones; any needed adjustment must be explicit in the later design. + +## Efficiency target + +Measure cost against total acquired bytes/nodes, reference and path data, final +materialized values, and retained provenance/history. Layer count alone is not +a useful size measure. Large history or output can legitimately dominate cost. + +Aim for O(N + H + R) processing and storage where N is total input/visited-path +work, H is retained history/provenance, and R is produced result data. Account +separately for deterministic sorting, filesystem traversal, schema processing, +and caller callbacks. This is an engineering target, not an unconditional public +complexity guarantee or a fixed performance gate. + +The target implementation: + +- Acquires/parses each accepted source occurrence once and directly converts + programmatic input. It does not decode or serialize accumulated configurations + between layers. Respect single-consumption readers and mutable file sources; + no implicit cross-load cache is needed. +- Uses indexed lookup alongside ordered mapping storage, retaining deterministic + iteration and diagnostics without repeatedly scanning growing mappings. +- Builds one load-owned working assembly instead of cloning the accumulated + configuration and history for every added layer. Detach at meaningful ownership + boundaries; safely share immutable private history where appropriate. +- Updates affected subtrees and their provenance rather than scanning all fields + for every replacement. Avoid repeatedly copying ever-growing ancestor-origin + lists or rechecking the same conflict group for every member. +- Reuses one schema/lifecycle evaluation per load and freezes the final result + once. It does not require global caches, copy-on-write public APIs, or a general + persistent-tree framework merely to meet an asymptotic target. + +Use representative benchmarks and allocation measurements for broad keyed +catalogs, many small layers, repeated replacement, imported filesystem defaults, +and retained history. Compare growth as inputs scale, including small common +loads. Favor changes addressing demonstrated scaling problems; preserve stable +diagnostic order and ownership rather than trading them for a superficial O(n) +claim. No timing threshold or additional release gate is requested. + +## Work needed from the baseline + +These are responsibility areas, not implementation stages: + +- Extend the root-owned source/context facade and private acquisition boundary + for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling + without exporting parser or engine types. +- Add explicit layered loading and programmatic input, sharing one engine path + with ordinary loading rather than building a parallel configuration system. +- Resolve the fixed operation and ownership decisions below, then extend source + records, histories, diagnostics, and rendering consistently across input kinds. +- Refactor mapping assembly and provenance updates where current algorithms + would scale poorly under layered workloads. Keep optimization scope tied to + the new workflow rather than rewriting unrelated library subsystems. +- Add focused behavioral coverage for mixed sources, relative resources, + namespace collisions, import failures, composition conflicts, explicit empty + and null values, caller/result independence, and source-correct redaction and + explanation. Add comparative benchmarks without making performance a brittle + pass/fail contract. +- Provide maintained simple and PromptKit-shaped examples: embedded defaults, + operator configuration from another context, and labeled programmatic input. + Demonstrate exact operations and resource origins without provider calls or + recreating PromptKit itself. +- Update Go declaration documentation, the configuration contract, consumer + guidance, component overview, and relevant architecture/deferred-scope records + as behavior lands. Record significant boundary changes in an ADR without + rewriting accepted ADR history. Keep named inheritance explicitly separate. + +Plan an additive minor release for the new public capability. If any design choice +requires changing an existing documented behavior, identify it expressly and +provide migration guidance; do not describe it as an internal optimization. +No release, implementation change, or current-policy revision occurs in this pass. + +## Completion criteria + +A consumer can load embedded configuration with relative imports, combine it +with independently located operator configuration and explicit programmatic +values, and obtain one typed result with accurate presence, sources, history, +and resource context. No temporary configuration files or serialization round +trip is needed. Simple `Load` usage retains its current behavior. Benchmarks +demonstrate that adding inputs does not entail avoidable full-collection copying +or quadratic key lookup. Public contracts and examples clearly distinguish +composition, overlay, any selected replacement operation, and downstream domain +resolution. + +## Open Decisions + +### 1. Whole-definition replacement + +**Recommendation:** include a small fixed replacement operation alongside compose +and recursive overlay. For PromptKit-shaped catalogs, explicitly identify a +string-keyed collection path and replace each supplied entry as a whole; retain +unmentioned entries. Do not infer identities from object fields. An empty supplied +collection contributes no entries; an explicitly supplied empty entry replaces +that entry's lower input. Use the same rule for every consumer, not arbitrary +per-field policies. This meets the concrete identity-precedence need without +silently redefining ordinary overlay. Final typed defaults still apply. + +**Alternative:** ship only existing composition/overlay semantics and leave +whole-definition selection downstream. This is smaller and closest to today's +surface, but may force consumers to parse/select definitions and reconstruct +origins outside ConfigKit, undermining the request's purpose. Confirm with the +combined consumer example before claiming this narrower scope meets the need. + +### 2. Initial programmatic representation + +**Recommendation:** labeled string-keyed mappings with the scalar/list/map value +set already supported by explicit overrides. Missing keys express omission and +nil expresses null. Reuse private conversion machinery, not CLI-specific public +records. This is predictable, directly expressible in Go, and consistent with +current dynamic configuration semantics. + +**Alternative:** typed Go values with a separate explicit-presence description. +This can reduce downstream mapping code, but introduces reconciliation of field +bindings, masks, nulls, and typed zero values and a larger public contract. Do not +infer omission from zero values. Add typed construction only if real examples +show that the simple mapping representation imposes substantial repetitive work. + +### 3. How consumers obtain filesystem resource context + +**Recommendation:** a separate source-context query exposing standard filesystem +access and a source-relative base location through root-owned API types, while +ordinary source records remain descriptive. The returned filesystem is a borrowed +capability, not a cloned or immutable object. This avoids a mandatory downstream +registry and supports origins from imports as well as initial sources. Make +lifetime and nil/no-context behavior explicit before implementation. + +**Alternative:** expose opaque context IDs and relative paths; callers retain a +registry mapping IDs to their filesystems. Metadata stays capability-free and +serializable, but every consumer must maintain and correctly reconcile another +registry. This is viable if limiting retained filesystem capabilities proves +more important than the additional integration machinery. + +### 4. Primary-root ownership in multi-source loads + +**Recommendation:** retain an explicit primary source/root role for application +schema version and the optional named-overlay convention. Additional inputs are +fragments or explicit precedence contributions, not independent version owners. +This best preserves current version ownership and prevents precedence from hiding +a version failure. An in-memory primary mapping can carry the version without +requiring a generated file. Other source contexts still govern their own imports +and resource references. + +**Alternative:** permit independently versioned roots and validate each before +assembly, with a separately selected context for named-overlay lookup. This fits +standalone catalogs more naturally, but needs new rules for combining accepted +versions, callback counts, and the effective version's provenance. It should not +silently invoke the existing version callback once per layer or let the last +version value win. Resolve this choice with a realistic mixed-catalog example +before producing the staged implementation plan.