diff --git a/docs/adr/0007-source-backed-underlays.md b/docs/adr/0007-source-backed-underlays.md index 7bdf785..16bb745 100644 --- a/docs/adr/0007-source-backed-underlays.md +++ b/docs/adr/0007-source-backed-underlays.md @@ -2,9 +2,8 @@ ## Status -Proposed draft for review. The -[filesystem sources and underlays roadmap](../roadmap/sources-and-layering.md) -records the target and implementation progress; the +Proposed draft for review. Completed planning and remediation records are +preserved in Git history at `b604aa1` under `docs/roadmap/`; the [configuration contract](../config.md#layered-loading) owns current behavior. This extends [ADR 0001](0001-separate-composition-and-overlay.md) and preserves diff --git a/docs/development.md b/docs/development.md index b356c27..674c2d6 100644 --- a/docs/development.md +++ b/docs/development.md @@ -80,6 +80,12 @@ and current facade ownership belongs to the [component overview](internal/overvi ## Baseline validation +The completed filesystem-source and underlay roadmap, original feature request, +implementation plan, and resolved remediation record are preserved in Git history +at `b604aa1` under `docs/roadmap/`. Current behavior belongs to the +[configuration contract](config.md#layered-loading). Named-entry inheritance is +separate proposed work in its [roadmap](roadmap/named-inheritance.md). + The completed overlay rename plan and release-workflow simplification record are preserved in Git history at `73ce6e9` under `docs/roadmap/`. Current overlay behavior belongs to the [configuration contract](config.md#selected-overlays), diff --git a/docs/roadmap/deferred.md b/docs/roadmap/deferred.md index bb7b62e..2219bb7 100644 --- a/docs/roadmap/deferred.md +++ b/docs/roadmap/deferred.md @@ -16,8 +16,7 @@ The following remain deliberately excluded: - automatic `conf.d` discovery; - remote imports, arbitrary providers, and additional configuration formats; - custom merge strategies, sequence merging, and reset operators; -- named overlay inheritance, tracked separately in the - [PromptKit inheritance request](feature-request-promptkit-inheritance.md); +- named overlay inheritance; - parser-specific or custom structural-decoding extension APIs; - generic dynamic-subtree conversion and module-registry APIs; - automatic schema migrations; @@ -28,3 +27,8 @@ The following remain deliberately excluded: Applications may build their own validate, effective-value, source, bootstrap, and explanation commands around the library. Broadening these boundaries requires concrete consumer evidence, compatibility review, and an explicit design change. + +Opt-in inheritance between named configuration entries is proposed in the +[named-inheritance roadmap](named-inheritance.md), based on the +[downstream request](feature-request-promptkit-inheritance.md). It is not yet +implemented and is distinct from inheritance between selected overlay files. diff --git a/docs/roadmap/feature-request-promptkit-inheritance.md b/docs/roadmap/feature-request-promptkit-inheritance.md index 9bf937a..c643ca2 100644 --- a/docs/roadmap/feature-request-promptkit-inheritance.md +++ b/docs/roadmap/feature-request-promptkit-inheritance.md @@ -24,9 +24,12 @@ contract, if any, would be appropriate. ## Distinction From Layered Loading -The separate [filesystem and layered-loading request](feature-request-promptkit.md) -concerns acquiring configuration from different sources and combining it under -explicit composition or precedence rules. +The completed filesystem and layered-loading request is preserved in Git history +at `b604aa1:docs/roadmap/feature-request-promptkit.md`. The implemented +[layered-loading contract](../config.md#layered-loading) covers acquiring +configuration from different sources and combining it under explicit composition +or precedence rules. The proposed ConfigKit design for this remaining request +is in the [named-inheritance roadmap](named-inheritance.md). Named inheritance concerns relationships between configuration entries: a child identifies a base, which may itself identify another base. The relationship diff --git a/docs/roadmap/feature-request-promptkit.md b/docs/roadmap/feature-request-promptkit.md deleted file mode 100644 index 16d8e53..0000000 --- a/docs/roadmap/feature-request-promptkit.md +++ /dev/null @@ -1,147 +0,0 @@ -# PromptKit Feature Request: Filesystem Sources And Layered Loading - -Status: downstream request for consideration, not an approved ConfigKit design -or implementation plan. - -## Consumer Context - -PromptKit is a Go library for defining, rendering, and executing prompts against -LLM backends. Its consumers supply prompt and profile definitions through local -files, embedded application defaults, maintained backend catalogs, and -programmatic values. - -PromptKit intends to adopt ConfigKit to standardize configuration semantics and -remove custom loading machinery. Applications using both libraries should not -need to learn two different sets of generic YAML, presence, and diagnostic rules. -PromptKit is pre-1.0 and can change its definition formats and construction API -to accommodate a coherent shared configuration model, provided its substantive -prompting and execution capabilities remain available. - -PromptKit has selected the following two capabilities as prerequisites for that -refactor. They are requested together because embedded defaults and explicit -layering form one common consumer workflow. This document describes downstream -needs only; ConfigKit's public API design and internal implementation remain for -its maintainers to determine. - -## 1. Native `fs.FS` Configuration Sources - -### Need - -Allow a library or application to supply configuration from an `fs.FS`, including -`embed.FS`, without extracting files to disk or treating each file as an isolated -reader disconnected from its filesystem context. - -PromptKit packages maintained profile catalogs and supports application-bundled -fallback profiles. Consumers also embed prompt definitions and associated text -assets so deployed programs do not require a separate configuration installation. -These sources should participate in ConfigKit loading as naturally as local files. - -### Desired Consumer Outcomes - -- Load configuration from an explicitly supplied filesystem and location within - it, with no dependency on a particular host directory layout. -- Retain meaningful source identity and locations in diagnostics and origins, - even when the source has no operating-system filename. -- When configuration uses supported imports or overlays, resolve relative - configuration references in the supplied filesystem context rather than - unexpectedly looking on the host filesystem. -- Make enough source context available for a consuming library to resolve its - own relative resources. For PromptKit, those include referenced message text; - understanding or rendering that text remains PromptKit's responsibility. -- Use the same configuration semantics for embedded and ordinary file-backed - definitions, rather than maintaining a separate embedded-loading path. - -### Rationale - -Bundled defaults are useful to many Go libraries, not just PromptKit. Native -filesystem support lets the owner package those defaults while callers choose -where their configuration comes from. ConfigKit's existing reader source can -accept file bytes, but supplying bytes alone does not satisfy the filesystem- -relative workflow described here. - -## 2. Public In-Memory Layered Loading With Origins - -### Need - -Allow callers to explicitly supply configuration layers and obtain their effective -configuration with meaningful origins, without temporary files or conversion of -programmatic values to YAML and back. - -PromptKit needs to combine library-maintained defaults, application-bundled -defaults, operator-selected configuration, and programmatic definitions. These -inputs do not necessarily live in one filesystem or follow an application-owned -overlay directory convention. - -### Desired Consumer Outcomes - -- Combine explicitly supplied embedded/file/byte-backed configuration and - programmatic values, with caller-selected precedence where overriding is intended. -- Distinguish combining independent contributions from intentionally overriding - earlier values. Accidental conflicts should not become implicit precedence - merely because sources happened to be discovered in a particular order. -- Preserve the distinction between an omitted field and an explicit zero, false, - empty value, or null so downstream domain rules can make deliberate choices. - Programmatic input needs a usable way to convey this intent as well as file input. -- Explain which source supplied an effective setting and identify relevant - sources when reporting a conflict or invalid setting. Origins must remain useful - when embedded and programmatic inputs have labels rather than host file paths. -- Allow loading through the public library interface without requiring a generated - root configuration file, temporary overlay directory, or serialization round trip. - -### PromptKit-Specific Context - -PromptKit's intended source priority, highest first, is programmatic profiles, -operator profiles, application fallback profiles, PromptKit-owned profiles, and -external catalog profiles. A higher-priority definition replaces the lower -definition with the same identity. This is distinct from profile inheritance: -a selected profile may name a base profile and override selected settings. - -For inherited settings, PromptKit intends omission-based inheritance and explicit -value replacement, with a supplied provider-parameter map replacing the whole -inherited map. These examples explain why generic recursive mapping merge alone -does not cover every downstream operation. They do not request configurable -per-field merge strategies, a PromptKit-specific profile registry, or LLM rules -inside ConfigKit. The eventual public contract should make clear which generic -layering needs it serves and what domain interpretation remains with the consumer. - -### Rationale - -Many libraries provide defaults that applications then customize. A public -layered-loading capability would let those libraries share ConfigKit's presence, -validation, and origin semantics rather than each maintaining its own generic -composition and diagnostic machinery. Applications using ConfigKit directly and -indirectly through PromptKit would benefit from a consistent explanation of how -their effective configuration was obtained. - -## Combined Workflow And Efficiency Needs - -A representative consumer bundles default profiles, reads operator configuration, -adds a few programmatic settings, and constructs PromptKit once. Loading produces -an immutable set of definitions; subsequent prompt executions use that set without -reloading configuration. Errors need to be actionable at construction, before any -provider request. A new construction can capture later configuration changes. - -Because PromptKit is a shared library, loading overhead and retained memory matter -across applications and catalog sizes. The requested capabilities should support -efficient processing of many definitions and layers without requiring repeated -reads, decoding, serialization, or copying of entire accumulated collections for -each added layer. Efficiency should be assessed against total input and produced -data, including retained provenance; this is not a request for an unconditional -O(n) promise or a fixed timing threshold. - -## Scope And Follow-Up - -This request does not seek automatic discovery, hot reload, live provider checks, -prompt rendering, model selection, or credential resolution specific to LLM calls. -PromptKit retains its domain relationships and execution responsibilities. - -The follow-up design discussion should clarify supported layer inputs, how callers -express programmatic presence and precedence, filesystem-relative behavior, and -the source information consumers can inspect. PromptKit can adapt its APIs and -formats to those decisions; historical loading order is not a compatibility demand. - -The current [configuration contract](../config.md) and -[deferred scope](deferred.md) describe ConfigKit's existing behavior and exclusions. -This request proposes expanding the relevant scope, not treating the desired -features as already available. PromptKit plans to consume released ConfigKit -support before implementing the dependent refactor. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index dde424c..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,565 +0,0 @@ -# Filesystem Sources and Underlays Implementation Plan - -## Purpose and baseline - -Implement the [feature roadmap](sources-and-layering.md) in the numbered order -below. Each stage is sized as one focused implementation assignment for a -gpt-5.6-terra coding agent. The baseline is -`36172ed20e832b371506d6aceb02dc6c8f50f71b`, plus the documentation changes committed -with this plan. No feature implementation is part of this planning pass. - -Read `docs/development.md` and the policies it selects for the stage. The roadmap -owns target semantics; [ADR 0007](../adr/0007-source-backed-underlays.md) and -[ADR 0008](../adr/0008-complete-on-return-loading.md) record the proposed rationale. -Their draft status is independent of implementation progress: do not mark them -accepted automatically. The [configuration contract](../config.md) is the -compatibility baseline. Preserve the root-owned facade and private engine/tree -boundary. - -At each stage, preserve unrelated work, add focused behavioral coverage with the -change, update declaration documentation for public additions, and run the checks -specified below. Keep intermediate commits building and tested. Do not expose -placeholder public APIs that silently ignore options. Internal scaffolding may -precede the public feature. Use plain-English commits under the agent policy. - -## Settled implementation decisions - -### Public input surface - -Use these names and shapes; bodies and private fields belong to implementation: - -```go -// Root-owned opaque filesystem namespace, copying preserves namespace identity. -type Filesystem struct { /* private engine state */ } -func NewFilesystem(label string, filesystem fs.FS) Filesystem -func (f Filesystem) File(name string) Source -func Mapping(label string, values map[string]any) Source - -// Extend the existing record; nil FS selects host-filesystem context. -type FilesystemContext struct { - BaseDir string - Root string - FS *Filesystem -} - -type Input struct { - Source Source - Imports bool - Filesystem *FilesystemContext -} -type Underlay struct { Inputs []Input } -type LayeredOptions[T any] struct { - Definition Definition[T] - Underlays []Underlay - Overlay string - OverlayFilesystem *FilesystemContext - Overrides []Override -} -func LoadLayered[T any](main Input, supplied ...LayeredOptions[T]) (*Result[T], error) - -type ResourceContext struct { - FS fs.FS - BaseDir string -} -func (r *Result[T]) SourceContext(id string) (ResourceContext, bool) -func (r *Resolution) SourceContext(id string) (ResourceContext, bool) -``` - -`fs` above means the standard `io/fs` package. All named public types are defined -at the root, not aliases of engine types. Existing `Load` and `Options` stay -source-compatible apart from the additive field in `FilesystemContext`. Avoid a -parallel lifecycle implementation: adapt both entry points to one private load -request. No new dependency or minimum-Go-version change is needed. - -- Like `Load`, `LoadLayered` accepts zero or one options value. A zero `Source` - stays invalid. Represent omitted main input as `Input{Source: NoSource()}`. - `NoSource` contributes no document, source ID, or context; preserve its existing - invalid import/context combinations. An empty underlay group is a no-op. -- Each underlay contains strictly composed inputs, including enabled imports. - Groups are ordered lowest to highest. Repeated explicit inputs in different - tiers are allowed subject to each source's reuse contract. Within a tier, - repeated file identities remain import/repetition errors. Equal scalar claims - still conflict. Independent synthetic sources are distinguished by occurrence. -- `Input.Imports` opts in per input and its import graph. The main source and its - imports form the main tier. There is no programmatic high-priority escape hatch: - a mapping source can be main input or any underlay contribution; CLI overrides - retain their explicit-flag contract. -- Active named overlays in `LoadLayered` require `OverlayFilesystem` explicitly. - Their names, root-relative lookup, directive prohibition, and version protection - follow existing rules. An inactive overlay needs no context; an unused explicit - context has no effect. Ordinary `Load` retains implicit selection of its source's - filesystem context and existing `NoSource` restrictions. - -### Filesystem namespaces, paths, and acquisition - -`NewFilesystem` allocates a reusable private namespace token and retains the -caller-owned filesystem without opening it. Copies share the token; separate -constructors are separate namespaces even for the same underlying filesystem. -Never compare or hash arbitrary `fs.FS` values. Validate nonempty safe labels and -nil filesystem arguments through deferred source diagnostics, following existing -source-constructor conventions. The zero `Filesystem` yields an invalid source. - -`Filesystem.File(name)` uses a non-root `fs.ValidPath` file name relative to that -namespace. The default root is `.` and the base is the file's containing directory. -An optional filesystem context can narrow the root; file sources continue to -derive their own base directory. An explicit FS must identify the same namespace -as the file source; context overrides for native file sources supply that FS. -A host `File` cannot be rebound to a virtual namespace. -For byte, reader, and mapping inputs, an explicit context supplies their namespace -and base: with FS set, require valid slash-relative Root and BaseDir (use `.` for -the namespace root), with BaseDir inside Root. With FS nil, preserve the existing -absolute host-directory requirements. Labels never imply a context. - -Resolve import paths relative to the declaring document, normalizing joined -paths and rejecting escape from the configured root. Preserve the existing -relative-path/glob language, including contained parent references, optional -missing behavior, sorted matches, bounded traversal, and cycle diagnostics. -Virtual paths use `path`, not host `filepath` rules. Imports stay in their namespace; -generic filesystems provide their own symlink behavior. Preserve canonicalization, -regular-file checks, and `os.Root` containment for host acquisition. Use ordinary -`fs.FS` operations and `fs.ReadDirFile` where available; support the standard -`fs.ReadDir` fallback without claiming control over its internal allocation. - -Configuration reads close each file ConfigKit opens, including read/stat/parse -failures. Caller readers and caller filesystems are never closed by ConfigKit. -Namespace tokens live with source objects; deterministic load-local IDs are -assigned by acquisition, not addresses or global counters. - -### Programmatic representation and diagnostics - -`Mapping` snapshots its input at construction, matching `Bytes`. Convert directly -to the private document tree using the CLI converter's existing built-in scalar, -`[]any`, and `map[string]any` repertoire. Preserve integer precision, finite-float -validation, sorted keys, sequence order, explicit null, and cycle rejection. -Typed nil maps/slices mean null, as in existing overrides. A nil root mapping is -invalid because document roots are mappings; `{}` is a valid empty mapping. -Constructor failures are stored and surfaced on load. Detach once at construction; -do not retain caller maps or serialize them. Concurrent caller mutation during -construction is outside the supported ownership contract. - -Generic conversion returns private issues; adapt them to existing `override.*` -codes for CLI input, and `source.invalid` or `source.limit_exceeded` for mapping -input. Do not leak raw values or conversion errors. Source/path locations use -the label and semantic path, with zero line/column for synthetic input. Apply -reserved `configkit` directive rules consistently: with imports enabled, a -mapping can contain the same directive structure as YAML. Without context, -relative imports fail under existing context rules. - -### Ordering, lifecycle, budgets, and metadata - -Acquire and strictly compose main input first, construct the schema, and check -the optional version. Only the main source owns the version; main imports and -all underlay documents must satisfy fragment-style version-path protections. -With `NoSource` main input, pass an absent version to the checker. Do not run the -checker once per tier or validate the final overlaid version instead. - -Then acquire/compose underlays in listed order and combine them recursively, -apply the composed main tier, selected overlay, and CLI overrides. Execute typed -defaulting/decoding, recorded resolution, secret lookup, validation, and final -publication once. Syntax/composition/version failures cannot be masked by higher -precedence; effective typed validation sees the final assembled input only. - -Keep source-record order separate from precedence: main depth-first sources, -underlay tiers in order with each input traversed depth-first, selected overlay, -then CLI. IDs remain `source-N`. Existing ordinary loads retain their source order. -The history application order follows precedence, not acquisition order. - -Extend `Layer` with `LayerUnderlay = "underlay"`. Add to `SourceInfo` a `Kind -SourceKind` and `UnderlayIndex int` (one-based original tier position for underlay -sources, zero otherwise). Define `SourceKindFile`, `SourceKindFilesystem`, -`SourceKindBytes`, `SourceKindReader`, `SourceKindMapping`, and `SourceKindCLI` -with string values `file`, `filesystem`, `bytes`, `reader`, `mapping`, and `cli`. -Imported documents inherit tier role/index and have their actual acquisition kind. -Preserve existing host Path meaning; filesystem-source Path is namespace-relative; -synthetic Path stays empty. Labels for native files combine namespace label and -relative path for display, while identity uses the token and path. - -Origins continue to refer to source IDs through locations. Composite origins -preserve all contributors; equal-value overrides and displaced descendants stay -inspectable. Carry source context separately from serializable source records. -Keep existing inspection schema and renderer structure, adding the new layer/kind -where appropriate rather than changing existing output gratuitously. - -Use one load-wide accounting object for layered loads: existing 64 MiB aggregate -input bytes, 1,024 acquired sources, 1,000,000 input nodes, and glob-entry budget; -existing per-source 8 MiB, depth 256, active import depth 64, and retained-history -limit remain. Mapping key/string bytes use the existing CLI converter's byte -accounting. Charge each accepted occurrence once, including CLI contributions; -do not charge intermediate copies as new input. Explicit empty groups and -`NoSource` consume no source allowance. Ordinary loads retain their established -limit behavior. Use existing source/import/overlay/override limit codes according -to the acquisition that exceeds a budget. Add `CodeUnderlayVersion = -"underlay.version"` for version protection in underlay roots; imported fragments -retain `import.version`. Avoid a separate arbitrary tier-count limit. - -### Passive resource contexts - -`SourceContext` returns false for unknown IDs or sources without a context, and -performs no I/O. The result method is nil-safe; the resolution method follows -existing callback lifetime guards. Return a record copy containing a borrowed -filesystem and slash-relative source BaseDir within its configured root. -Consumers obtain source IDs from ordinary origins; there is no single-context -query that guesses among composite or derived origins. Dependencies continue to -identify the sources of application-derived values through existing metadata. - -For native inputs, use a namespace-rooted view over the caller's filesystem and -retain its lifetime semantics. For host input, retain the canonical root path, -not an open `os.Root`. A private `fs.FS` adapter validates a relative resource -name, resolves it using existing host containment semantics, opens a fresh root, -and opens the resource. Return a private `fs.File` wrapper whose Close releases -both file and root, including failed-open cleanup. Expose `fs.ReadDirFile` when -the underlying directory supports it. Resources themselves are application data: -do not apply configuration YAML parsing or configuration-byte budgets to them. -Helpers such as `fs.ReadFile` then close the returned handle normally. Errors -follow filesystem access conventions, separately from load diagnostics. - -Neither queries nor retained results keep library-owned filesystem handles open. -Files/resources can change or disappear after loading; configuration/metadata -remain usable. No background work, hidden reopening on metadata queries, -filesystem snapshot, universal sandbox, or transaction guarantee is added. - -## Stage 1: Index mapping assembly and establish scaling baselines - -**Completed 2026-09-29.** Private per-assembly mapping indexes now serve strict -composition and recursive overlay lookup while ordered entries and diagnostics -remain stable; conflict groups are checked once. Focused Compose/Overlay tests -and the everyday checker passed. On Linux amd64, the 512-entry benchmark -fixtures measured approximately 4.0 ms/op before and 3.3 ms/op after for wide -composition, and 12.9 ms/op before and 12.0 ms/op after for recursive overlay. -These are comparative evidence, not timing gates; provenance work still -dominates the latter and belongs to the next stage. - -Scope: `internal/document` and engine composition/overlay lookup, with existing -behavior only. - -1. Add parameterized benchmarks for wide disjoint composition and recursive - overlay, recording runtime and allocations at increasing sizes. Keep fixtures - within existing budgets and record the baseline in the stage handoff. -2. Replace repeated linear mapping-key lookup during bulk assembly with a private - indexed builder alongside ordered entries. Build each target index once per - assembly session and update it for appended keys. Keep tree traversal and - diagnostic order stable; avoid mutable indexes exposed through public APIs. -3. Check each strict-composition claim group once while preserving first-conflict - ordering and all relevant locations. Preserve original input ownership. -4. Extend existing composition tests only for uncovered ordering/conflict risks. - -Acceptance: existing Compose/Overlay tests pass unchanged; broad keyed input -no longer incurs a full key scan for each insertion. Run the focused composition -tests and the everyday checker. Benchmarks are evidence, not timing gates. - -## Stage 2: Prepare load-owned assembly and incremental provenance - -**Completed 2026-09-29.** Ordinary loads now apply selected overlays and CLI -overrides to one owned working document, while copy-preserving helpers remain -available for direct use. Acquired history trees stay detached; composite -contributors survive overlay application. Origin deletion visits only replaced -subtrees, mapping contributors append without copying growing lists, and history -replay shares indexed lookup across applications. Focused behavior tests, the -everyday checker, and the engine race suite passed. On Linux amd64 at 128 small -applications, owned assembly measured about 0.30 ms/op versus 6.0 ms/op for -repeated copy-preserving application; alternating subtree/scalar replacement -measured about 0.39 versus 7.3 ms/op. These benchmarks are evidence, not gates. - -Scope: `compose.go`, `overlay.go`, origin indexes, override application, and -history replay in `internal/engine`. - -1. Introduce a private load-owned accumulator with explicit ownership. Existing - copy-preserving helpers may wrap it, but sequential assembly must mutate one - owned working tree rather than clone all accumulated state per tier. -2. Retain immutable acquired inputs for history, detached from mutable working - nodes. Parameterize internal overlay application by incoming layer and origins - so a composed tier's imported contributors are preserved. -3. Remove displaced origins by visiting the affected subtree. Accumulate mapping - contributors without copying an ever-growing ancestor list on each addition; - materialize defensive public records at the existing publication boundary. -4. Preserve equal-value replacement events, descendant displacement, redaction, - and typed-decoding reconciliation. Extend benchmarks for many small applications - and repeated subtree/scalar replacement. Do not add a persistent-tree framework. - -Acceptance: existing override/history/rendering/ownership tests pass; benchmark -growth avoids accumulated full-tree copying. Run the everyday checker and -`GOPROXY=off go test -race ./internal/engine` for ownership changes. - -## Stage 3: Unify private filesystem acquisition without public changes - -**Completed 2026-09-29.** Imports, glob traversal, and selected-overlay loading -now use a private access boundary with the existing contained host backend and -a native `fs.FS` backend. Namespace tokens and occurrence identities are separate -from labels and paths; repetition tracking is composition-local and input/glob -accounting can be shared. Native tests cover slash-relative imports, sorted -globs, optional missing paths, root escape, overlays, repeated/cyclic imports, -non-comparable filesystems, and handle closure on success and representative -failures. Existing host tests, focused native tests, and the everyday checker -passed. No public constructor or context field was added. - -Scope: engine source/file, filesystem, import, glob, and selected-overlay code. - -1. Extract a small private namespace/access boundary used by imports and overlays. - Keep the host implementation's canonicalization, containment, regular-file - checks, safe diagnostics, deterministic traversal, and close behavior. -2. Separate namespace identity from source labels and resource paths. Make - repeated/cycle tracking belong to an input composition group, and accounting - capable of being shared across groups. -3. Add a native `fs.FS` backend internally. Resolve paths using slash semantics, - preserve glob grammar and optional-missing behavior, and avoid host fallback. -4. Use real host tests plus `fstest.MapFS` and a small tracking/failing filesystem - to cover imports, missing paths, directory/read failures, and handle closure. - Include a filesystem with a non-comparable dynamic value. - -Acceptance: all existing host source/import/glob/overlay tests remain green; -native backend tests establish namespace isolation and supported paths. Run the -everyday checker. No public constructors land until the next stage. - -## Stage 4: Expose native filesystem sources through the facade - -**Completed 2026-09-29.** The root API now exposes reusable native filesystem -namespaces, namespace-aware contexts, and source acquisition kinds. Ordinary -loads support native files, opt-in imports, selected overlays, and byte/reader -contexts while retaining host semantics. Public tests cover embedded fixtures, -`fstest.MapFS`, same-label namespace isolation, wrapper copies and reuse, -constructor/context diagnostics, and host/CLI source kinds. The everyday checker -passed, including public-boundary and documentation checks. - -1. Add `Filesystem`, `NewFilesystem`, `Filesystem.File`, and the additive FS - context field at root and engine boundaries. Constructors follow deferred - diagnostics; copying the wrapper preserves namespace identity. -2. Integrate native sources with ordinary `Load`, including explicitly enabled - imports and named overlays. Enforce the path/context rules settled above. -3. Add `SourceKind` and populate it for every current/native acquisition. Preserve - host paths and use namespace-relative virtual paths with accurate locations. -4. Add facade-level tests using embedded fixtures, `fstest.MapFS`, byte/reader - context, namespace-label collisions, and source reuse. Keep backend error cases - at their Stage 3 owner. Update current source/API docs to this implemented scope. - -Acceptance: native `Load` works through the public API and existing sources retain -their contracts. Run the everyday checker, including public-boundary tests. - -## Stage 5: Add snapshotted mapping sources - -**Completed 2026-09-29.** `Mapping` now snapshots supported built-in values at -construction, retaining safe deferred diagnostics for invalid roots, values, -cycles, nonfinite floats, and limits. A shared private converter preserves CLI -override behavior through a separate adapter. Mapping inputs participate in -ordinary directives, imports, version checks, decoding, and inspection with -their own source kind and synthetic locations. Public tests cover YAML -equivalence, reuse after caller mutation, null/empty/zero distinctions, exact -integers, explicit filesystem context, and safe failures. The everyday checker -and engine race suite passed. - -1. Extract reusable value conversion from `override_value.go`; keep CLI-specific - path, label, and diagnostic handling in its adapter. -2. Implement `Mapping` at root and engine boundaries with constructor-time - snapshotting and deferred safe failures. Source-ID assignment at acquisition - must not mutate a reusable stored tree shared by another load. -3. Route converted documents through the same directive processing, strict - composition, version checking, decode, and inspection path as YAML sources. -4. Test omission/zero/false/null, nil versus empty collections/root, exact integers, - invalid/cyclic values, imports with explicit context, and caller mutation after - construction. Verify mapping input does not gain CLI origins or restrictions. - -Acceptance: equivalent byte/mapping inputs have equivalent semantic results and -origin meaning, with genuine location differences allowed. Existing override -diagnostics remain stable. Run the everyday checker and engine race tests. - -## Stage 6: Implement private grouped-underlay orchestration - -**Completed 2026-09-29.** Ordinary and grouped engine loads now use one private -request lifecycle. Group-local strict composition and repetition tracking, -load-wide source/byte/node/glob accounting, main-owned version checking, -underlay layer/index metadata, and precedence-ordered assembly feed one final -decode, resolution, validation, and publication. Engine tests cover two tiers, -multiple contributions, source/history order, absent main/defaults-only loads, -main/import and group conflicts, version protection, shared limits, reusable -files across tiers, and replacement of invalid intermediate typed input. The -everyday checker and engine race suite passed. No root layered API is exposed yet. - -1. Add private request/input/group structures matching the settled facade design. - Adapt ordinary `Load` to the shared core while preserving its options and - source-order behavior. Keep its existing validation order where observable. -2. Strictly compose each group with group-scoped repetition tracking and shared - aggregate accounting. Apply the lifecycle order defined above, including the - single main-owned version check before underlay precedence assembly. -3. Add underlay layer/index metadata and version diagnostics. Keep history in - precedence order while source IDs follow acquisition order. Apply composed - main origins over underlays without relabeling main imports as overlays. -4. Reuse Stage 2 assembly and the existing decode/resolution/secret/publication - pipeline. Empty groups, omitted main, defaults-only loads, and underlay-only - loads must follow the chosen semantics. -5. Add engine coverage for two underlay tiers, multiple strict contributions, - main/import conflicts, version checks/absent main, limits across groups, and - one final decode/resolve/validate lifecycle. Test an invalid intermediate typed - value replaced by valid higher input, alongside unmaskable syntax conflicts. - -Acceptance: engine layered requests produce correct values and full provenance; -ordinary-load tests remain green. Run the everyday checker and engine race tests. - -## Stage 7: Expose layered loading and complete inspection integration - -**Completed 2026-09-29.** The root facade now offers `Input`, `Underlay`, -`LayeredOptions`, and `LoadLayered` with copied options/context records. Explicit -host or native selected-overlay contexts work independently of the main source; -repetition uses native namespace identity rather than a display path alone. -Root layer/source conversions and explanation rendering include underlays. -Public tests cover mixed host/native/mapping tiers, recursive precedence, -empty/zero/false/null and sequence replacement, source order and tier indexes, -composite and displaced origins, resolution, redaction, and zero-underlay -equivalence. The everyday checker and full race suite passed. - -1. Add `Input`, `Underlay`, `LayeredOptions`, and `LoadLayered` root wrappers and - conversions. Copy option slices/context records into private load state while - preserving reusable namespace identities and single-consumption readers. -2. Wire explicit selected-overlay context for layered loads, including host and - native namespaces. The selected file may not duplicate an acquired file in - that namespace, preserving selected-overlay repetition protection. -3. Finish root Layer/SourceInfo conversions, history/explanation rendering, - composite mapping attribution, deleted descendants, default and derived - origins, and secret redaction. New records must remain detached. -4. Add representative public tests for full precedence and same-name recursive - mapping combination; retain zero/false/empty/null distinctions and atomic - sequence replacement. Test ordinary versus zero-underlay layered equivalence - using equivalent explicit overlay context when required. - -Acceptance: callers can load native, host, and mapping tiers in one public call; -configuration, metadata, and safe rendering agree. Run the everyday checker and -`GOPROXY=off go test -race ./...`. - -## Stage 8: Add handle-free source-context queries - -**Completed 2026-09-29.** Results and active resolution callbacks now expose -passive per-source `ResourceContext` records. Native views borrow caller-owned -filesystems under their configured roots; host views retain only canonical root -paths and open/close fresh roots with each resource file. Main, imported, -selected-overlay, and explicitly contextual synthetic sources have their own -base directories; contextless and CLI sources do not. Public tests verify -origin-based resource access, no I/O on queries, later mutation/deletion, -borrowed-filesystem closure, and callback guards. Tracking tests cover resource -Open failure cleanup, returned file/directory ownership, and loading-handle -closure across success and representative failures. The everyday checker and -full race suite passed. - -1. Retain context descriptors keyed by acquired source ID, including imported - sources, in private inspection and live resolution state. Keep descriptors - separate from serializable metadata and preserve callback guards. -2. Implement the root query methods and `ResourceContext`. Native contexts expose - rooted borrowed access; host contexts use the passive adapter described above. - Include host main files even when imports are disabled, and explicit contexts - on byte/reader/mapping sources. Sources without contexts return false. -3. Verify each origin's BaseDir points to its actual source. Composite origins - require callers to follow individual locations; derived dependencies retain - their existing meaning. No arbitrary last-source context is selected. -4. Test that queries cause no filesystem operations; loaded values and metadata - survive source deletion; later resource access observes mutation/missing files; - and caller filesystem closure affects only later resource operations. -5. Verify owned files/root handles close on successful loading and representative - acquisition, parse, composition, version, and validation failures. Test resource - Open failure cleanup and returned-file Close ownership separately. Prefer - narrow tracking seams over timing-based goroutine/descriptor-count assertions. - -Acceptance: a retained underlay resource can be opened relative to its true origin -after load, with no library-owned loading handles retained. Run the everyday -checker and full race suite. - -## Stage 9: Validate cross-feature consistency and scaling - -**Completed 2026-09-29.** A compact public matrix now checks equivalent host, -native, byte, reader, and mapping inputs; tier-local repetition versus cross-tier -reuse; reader consumption; and safe diagnostics/effective/history output after -secret replacement. Engine tests verify exact occurrence accounting across -imports, shared glob/source limits, and the mixed-source CLI ceiling. Rendering -now indexes source labels once per explanation instead of scanning all records -for every location. Loading and passive context ownership were reviewed against -ADR 0008; no handles or background work are retained by queries. - -On Linux amd64 with `-benchmem` and 100 ms samples, the 512-entry assembly-only -composition measured about 3.5 ms/op (the earlier post-index baseline was about -3.3 ms), and recursive overlay about 2.2 ms/op (about 12 ms at the first-stage -handoff before subsequent provenance work). The 128-application owned assembly -remained about 0.30 ms/op versus 6.35 ms/op for copy-preserving application. -End-to-end layered measurements were about 7.15 ms/op for 512 broad catalog -entries, 2.59 ms/op for 128 small tiers, 3.27 ms/op for 128 alternating subtree -replacements, 1.86 ms/op for 64 imported native defaults, and 1.36 ms/op for -64 retained-history applications. Allocations grew roughly with these fixture -sizes. I/O, sorting, decoding, callbacks, and retained output contribute to -full-path cost; these observations are not timing gates or an unconditional -complexity guarantee. The everyday checker and full race suite passed. - -1. Consolidate a small semantic equivalence matrix across host, native, bytes, - readers, and mappings, using equivalent logical inputs and contexts. Cover - strict conflict versus intentional precedence and version-path preservation; - avoid duplicating every lower-level case at every source permutation. -2. Exercise load-wide limits across mixed sources/CLI, repeated explicit tiers, - shared namespace imports, and source IDs/history order. Protect against double - charging and reader re-consumption. Confirm secret canaries stay absent from - diagnostics and historical/effective output after replacements. -3. Run parameterized benchmarks for broad catalogs, many small tiers, subtree - replacement, imported native defaults, and retained history. Compare Stage 1 - baselines with the final path. Address demonstrated avoidable quadratic scans - or accumulated copying within this feature's scope. Account for sorting, - filesystem operations, callbacks, and output/history size explicitly. -4. Review source/context ownership and success/failure cleanup against ADR 0008. - Keep performance evidence in the stage handoff, not permanent timing gates. - -Acceptance: no semantic discrepancy remains among equivalent operations; scaling -results substantiate the engineering target without an unconditional O(n) claim. -Run the everyday checker and full race suite, reusing successful checks for an -unchanged final tree. Benchmarks run with `-benchmem` and parameterized sizes. - -## Stage 10: Complete consumer examples and architecture documentation - -**Completed 2026-09-29.** Maintained executable examples now cover a simple -embedded filesystem source and a layered catalog with imported embedded defaults, -a mapping underlay, an independent host main file, explicit false/zero values, -domain validation, and resource access through the retained origin's source ID. -The configuration contract, public declaration comments, consumer/development -guides, README, component overview, architecture/deferred text, roadmap status, -and draft ADR status links reflect implemented behavior. Named inheritance remains -separate and deferred; the ADRs remain drafts. The everyday checker and full race -suite passed. No release preparation or publication was performed. - -1. Add a maintained simple native-filesystem example and a combined example with - embedded defaults, two underlay tiers, independently located main input, - programmatic contributions, and a relative resource from a retained origin. - Keep fixtures secret-free and offline; include zero/false presence and domain - validation. The consumer should perform no generic post-load merge/defaulting - or provenance reconstruction. Reuse existing example/test conventions. -2. Update `docs/config.md`, Go declarations, `docs/consumers/`, README, - `docs/internal/overview.md`, and applicable architecture/deferred-scope text - to describe implemented support. Preserve strict-within-tier and recursive- - between-tier terminology, distinguish source kinds from precedence roles, - and document context lifetimes and the explicit source-ID query workflow. -3. Add the relevant tests/commands to the development guide's existing task map. - Review ADR links and statuses, preserving draft status pending acceptance and - accepted ADR decision bodies. Keep named inheritance separately deferred. -4. Mark implementation progress accurately in this plan and the roadmap. Do not - retire plans, choose/tag a version, prepare release gates, or publish as part - of implementation; those belong to separate release preparation. - -Acceptance: examples compile and execute, documentation describes only completed -capabilities, and simple `Load` usage remains simple. Run the everyday checker -(which includes examples, links, and boundary checks); run the full race suite -if code changed since its last successful run. Report completed stages, checks, -benchmark conclusions, and any remaining limitations honestly. - -## Validation commands - -Use the development guide's focused commands during each stage. The normal -stage-end check is `./scripts/check.sh`; it already includes tests, vet, build, -formatting, and documentation links. Ownership stages additionally use the race -commands above. Benchmarks may use: - -```sh -GOPROXY=off go test ./internal/engine -run '^$' -bench 'Compose|Overlay|Underlay' -benchmem -``` - -Name the new benchmarks to match that expression. Select representative sizes -under existing limits. No CI wait, human macOS validation, external service, -credentials, or new release gate is part of this plan. - -## Open Questions - -None required to implement this plan. The remaining roadmap recommendations -have been selected above: labeled snapshotted mappings and separate source-context -queries. Private helper names and package-local factoring remain implementation -choices within these contracts. If implementation uncovers a genuine conflict -with existing behavior, report it rather than silently adding a restriction, -weakening compatibility, or broadening a guarantee. diff --git a/docs/roadmap/named-inheritance.md b/docs/roadmap/named-inheritance.md new file mode 100644 index 0000000..1484906 --- /dev/null +++ b/docs/roadmap/named-inheritance.md @@ -0,0 +1,281 @@ +# Opt-In Named Configuration Inheritance + +Status: proposed feature design for agreement; not implemented. This document +defines the intended end state and the remaining architectural decisions. A +separate implementation plan will define execution steps after design agreement. + +## Purpose and priorities + +Allow an application to designate a collection of named configuration entries +whose members can inherit settings from one parent in that collection. ConfigKit +resolves the relationships during loading and returns final typed configuration +with inspectable provenance. Applications retain ownership of field meanings, +domain validation, and execution. + +The [downstream request](feature-request-promptkit-inheritance.md) supplies a +concrete need for reusable named defaults and chains of specialization. ConfigKit +adopts a shared structural model, using its existing recursive combination and +presence conventions. The design serves named presets, connection settings, or +similar configuration entries without interpreting their domain. + +Priorities are consistency and predictability, accurate presence/provenance, and +reduced consumer configuration infrastructure. Computational efficiency matters +within those goals: resolve shared ancestors once and account for the size of +materialized results and retained history. Avoid an unconditional O(n) promise. + +## Current baseline + +Baseline: `b604aa167b684525f0d5d806d9c4267824151ee4`, after the filesystem-source +and underlay implementation and its remediation. + +- The [configuration contract](../config.md) supports host/native filesystem, + byte, reader, and snapshotted mapping sources through a shared engine. +- Independent contributions compose strictly within a tier; explicitly ordered + tiers combine using recursive overlay. Typed defaults sit below source input; + selected overlays and explicit CLI overrides retain their documented roles. +- Named application entries can already be represented as string-keyed mappings. + ConfigKit currently treats relationships between their keys as application data. +- Recorded application resolution runs after typed decoding and preserves + dependencies and provenance. Consumers currently own relationship resolution; + there is no built-in pre-decode inheritance phase. +- Source contexts support later application-owned resource access. Loading is + synchronous and complete on return, with library-owned loading handles closed. + +This feature is a deliberate extension of structural input processing. It follows +the [architecture goals](../policy/architecture.md#success-criteria), the +composition distinction in [ADR 0001](../adr/0001-separate-composition-and-overlay.md), +the presence/provenance principles in +[ADR 0003](../adr/0003-preserve-presence-and-resolution-history.md), and the +complete-on-return design in [draft ADR 0008](../adr/0008-complete-on-return-loading.md). + +## Target configuration model + +### Explicit collection declarations + +Applications opt in particular string-keyed collections through their reusable +configuration definition, consistently for ordinary and layered loading. Each +mapping key is the entry's identity. References use exact names within the +declared collection, independent of filesystem namespaces or declaration order. +One entry can name one parent; chains and shared parents are supported. + +The declaration identifies structural metadata rather than enabling callbacks +that choose arbitrary lookup or merge strategies. Ordinary fields remain ordinary +data outside declared collections. The declaration syntax and treatment of the +parent-reference field are addressed in Open Decisions below. + +Mappings with no parent remain independent entries. Explicit null or scalar +entries retain their ordinary data meaning and are subject to normal final +schema validation; an entry used as a parent must resolve to a mapping so its +settings can combine with the child. Missing collections contribute no entries. +A present collection must have the mapping shape required by its declaration. + +Identity comes from the collection key, not a copied field in the parent payload. +The child keeps its key. Application fields such as `name` or `id` have ordinary +combination semantics unless consumed as explicitly declared structural metadata. + +### Source precedence first, inheritance second + +First assemble the effective input under the existing source rules: strict +composition within tiers and recursive overlay across underlays, main input, +selected overlay, and explicit CLI overrides. Strict conflicts remain errors. +Then resolve inheritance against the resulting effective collection. + +Every parent reference sees that parent's effective definition, including source +overrides. For each entry, resolve the parent's chain, then recursively overlay +the child's own effective payload. Each child obtains independent effective +settings; resolving a child leaves its parent and siblings unchanged. + +Source precedence and ancestry answer different questions. Source precedence +selects each entry's own settings and parent reference. Ancestry supplies settings +that entry omits. For example, if a child's own timeout is `10`, changing its +parent's timeout to `30` through higher-priority source input leaves the child's +explicit `10` intact. To change that explicit child value, override the child. +This applies equally to CLI overrides at those respective paths. + +### One combination contract + +Parent-to-child combination uses the existing recursive mapping overlay: + +- Omitted children retain inherited values. +- Present zero, false, and empty strings replace inherited values. +- Mappings combine recursively; an empty mapping retains inherited children. +- Sequences replace as a unit, including empty sequences. +- Explicit null replaces the inherited value, subject to final schema acceptance. +- Cross-kind replacement follows the same overlay rules and final schema checks. + +The parent's inheritance-control metadata is not inherited as a payload setting. +The child supplies its own relationship. Inherited payload values can themselves +contain ordinary strings or fields with similar names without acquiring reference +semantics outside the declared position. + +These rules intentionally differ from the request's suggested null-as-unspecified +and whole-parameter-map replacement. Consumers adopting the feature adopt +ConfigKit's shared rules. Multiple inheritance, sequence merging, per-field +strategies, deletion operators, cross-collection references, and named-overlay +file inheritance are outside this feature's initial scope. + +## Loading and validation + +The proposed lifecycle is: existing acquisition/composition and main-owned version +checking; finish source precedence assembly; resolve named inheritance; apply +typed defaults and decode; recorded application resolution; secret lookup; +semantic validation; publish the final configuration and metadata. + +Resolve inheritance in the private input representation while omission is still +explicit. Ordinary and layered loading use the same implementation. Typed defaults +remain the final decoder's baseline and are not converted into parent declarations +or recursively copied from parent Go values. Domain resolution receives already +expanded settings and can apply recorded normalization/defaulting under its +existing contract. + +Preserve the existing version owner's declaration and checking point. Inheritance +must not supply, replace, or remove the protected version path or bypass ancestor +protections. Identify attempted protected-path changes explicitly during design +of declarations rather than moving the checker to a different lifecycle point. + +All declared relationships resolve during the load, including entries the +application might not select for execution. Missing parents, non-mapping parents, +invalid reference values, self-reference, and longer cycles fail the load with +safe structured diagnostics and relevant source locations. Traverse deterministically; +diagnostics should identify the relationship and chain rather than expose payload +values. Synthetic inputs retain labels and semantic paths without fabricated file +coordinates. Normal syntax and strict-composition errors remain unmaskable. + +Partial base entries may omit domain-required settings. Omission is not by itself +a structural error. All returned entries still undergo ordinary typed decoding; +application validation determines which entries need to be independently executable +and which are reusable partial definitions. This adds no alternate schema, +automatic abstract-entry marker, or general exemption from validation. + +Successful loading yields the completed configuration and metadata. Inheritance +performs no background resolution and retains no filesystem handles. Applications +may load again explicitly using the existing source-reuse contracts. + +## Presence, provenance, and confidentiality + +Consumers must distinguish direct child input, inherited input, typed defaults, +and recorded application-derived values. Explicit zero/false/empty/null settings +remain distinguishable from omission. The representation of inherited presence +in the existing query surface is an open decision below; Go zero-value inference +is not an acceptable substitute. + +Each inherited value retains its originating source and source path, together +with the relationship that placed it at the child's destination path. Chains +remain explainable, including a child's explicit replacement of an inherited +value. Source-precedence history remains inspectable alongside inheritance +derivation; inheritance must not masquerade as another acquired file or source +tier. Effective metadata, resolver-entry queries, history, explanation, and +redacted rendering must agree. + +A relative resource inherited from a parent keeps the context of the source +that supplied that resource value. It does not acquire the child's directory. +Composite origins remain composite and derived dependencies remain explicit. + +Inherit secret references as unresolved configuration input; resolve typed +credentials only at the existing lifecycle point. Preserve source and destination +sensitivity through inheritance dependencies so copying a classified value cannot +make it public. A child's independent replacement has its own provenance; inherited +or displaced historical secrets remain safely explainable and redacted. No raw +credential values belong in inheritance trees, metadata, or diagnostics. + +## Efficiency and ownership + +Model each collection as entries and single-parent edges. Use indexed name lookup, +cycle detection, and memoized parent resolution to avoid repeated ancestor work. +Materialize independent public results while allowing carefully owned immutable +private structures to share work. Do not introduce global caches or reload sources +for individual entries. + +Assess work against input size, relationship edges, materialized effective values, +and retained provenance/history. Many children can legitimately produce much more +output than the source declarations. Use representative chains, shared-base trees, +wide collections, and replacement cases to evaluate growth. Performance evidence +is not a new release timing gate. + +Existing acquired-input accounting remains in force. The eventual implementation +plan must define how expanded output and retained derivation fit existing decoded +tree/history limits, including where expansion is checked. Avoid silently charging +one inherited value as a newly acquired source or inventing arbitrary entry/chain +caps. Any necessary new public limit requires an explicit design decision. + +## Work needed from the baseline + +The feature requires root-owned opt-in declarations, a private relationship graph +and expansion phase, and lifecycle integration before typed decoding. Presence, +derivation history, diagnostics, sensitivity propagation, and source-relative +resource origins must extend together. The public parser-independent facade +remains the only consumer boundary. + +Representative examples and behavioral tests should establish source/inheritance +ordering, chains and shared parents, failure diagnostics, partial bases, explicit +values versus omission, unchanged sequence/null behavior, origin/resource +preservation, result independence, and safe secret inspection. Both loading entry +points and supported source representations should share those semantics. + +A new ADR should record the selected inheritance architecture after the open +decisions are resolved. Architecture/deferred-scope documentation must recognize +the new structural responsibility; public API, configuration, consumer, and +internal documentation should describe implemented behavior when it lands. +Implementation sequencing belongs in a separate future plan. Existing callers +that do not opt in retain their behavior; an additive minor release is the target, +with release preparation and publication handled separately. + +## Completion criteria + +A consumer can declare a named collection, load mixed-source configuration once, +and receive resolved independent entries with one set of combination rules. +It can inspect whether each setting was direct, inherited, defaulted, or derived, +follow inherited resource origins, and validate final domain meaning. Missing or +cyclic relationships fail before a usable result is returned. The consumer needs +no separate graph resolver, generic merge pass, or provenance reconstruction. + +## Open Decisions + +### 1. Declaring collections and parent references + +**Recommendation:** a root-owned declaration in `Definition` identifies a +collection path and one direct parent-reference field name within each entry. +Consume that field as structural metadata during expansion and retain its source +location for inspection. An omitted reference means no parent; explicit null +removes a lower-tier relationship. A string names an exact key in the effective +collection, including an empty string if such an entry exists. Other shapes are +reference errors. Outside the opted-in position, the field remains ordinary data. +This integrates naturally with existing source precedence and avoids a globally +reserved `extends` field or duplicated entry identifiers. + +**Alternative:** declare a separate name-to-parent table at a designated input +path. Payload fields never overlap control metadata and application structs need +no inheritance field, but settings and relationships become separated and must +be kept consistent. It also introduces a second keyed collection into diagnostics +and source composition for each inheritance relationship set. + +### 2. Meaning of inherited presence + +**Recommendation:** an inherited value counts as present input for effective +`IsExplicit`/defaulting behavior, with separate derivation metadata distinguishing +it from direct child input. This treats inheritance as input expansion, keeps +typed and resolver defaults below inherited settings, and avoids consumers +reimplementing presence logic. The API must clearly explain that effective +presence differs from direct declaration at the destination path. + +**Alternative:** inherited values remain non-explicit at the child path, with +a new effective-input-presence query/defaulting rule. This preserves a narrow +direct-declaration meaning of explicitness, but requires two presence concepts +through the resolver lifecycle and risks defaults overwriting inherited settings +unless callers adopt the new query consistently. + +### 3. Nested or overlapping collection declarations + +**Recommendation:** initially support multiple independent, non-overlapping +declared collection paths. Report overlapping declarations as an options error. +An outer inheritance expansion can copy or replace an inner collection, so +overlap otherwise needs a selected rule for whether inner inheritance resolves +before copying or after arrival at each destination. This restriction makes that +unresolved interaction explicit rather than allowing declaration order to choose +semantics. It does not restrict ordinary nested payload mappings. + +**Alternative:** define destination-based outer-to-inner expansion, resolving each +inner collection after the enclosing entry is materialized. This supports nested +presets, but creates additional namespace, dependency, and provenance semantics +and can multiply graph instances. It is viable if a concrete consumer need +justifies that additional model; it should not emerge accidentally from traversal. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index e44c43c..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,141 +0,0 @@ -# Filesystem Sources and Underlays Review Findings - -Reviewed implementation: `cbf031a`, covering the implementation commits after -`27df838`. These findings concern the agreed source, precedence, and provenance -contracts in the [roadmap](sources-and-layering.md) and -[implementation plan](implementation.md). No implementation fixes are included -in this review. - -## Resolution status - -All three findings were addressed on 2026-09-29: - -- Composition now fills sequence-descendant origins separately from atomic - conflict claims, so initial underlay attribution reaches every surviving node. -- History reconciliation checks contributing origins and recognizes ancestor - replacement events without duplicating their descendant contributions. -- Native contexts use `fs.ValidPath`; host contexts use `filepath.Localize` - before the existing canonicalization and containment checks. - -`facade_underlay_origins_test.go` covers imported sequence descendants in first -and later tiers, resolver/result/explanation agreement, and later mapping -contributors. `TestResourceContextsPreserveLiteralBackslashes` covers native -and host resource reads and directory listing. The new tests reproduced all -three failures before the fixes. The findings below retain the original review -evidence and acceptance requirements. - -Post-fix validation passed: `./scripts/check.sh` and -`GOPROXY=off go test -race ./...`. Existing sequence replacement, provenance, -history, and resource-containment tests remain passing. - -## 1. Preserve underlay attribution for sequence descendants (P2) - -**Owners:** `internal/engine/load_request.go:174`, -`internal/engine/compose.go:113`, and `internal/engine/load_result.go:39`. - -The first nonempty underlay seeds the accumulator using a composed document. -Composition claims intentionally stop at sequences because sequences are atomic -for strict conflict checking. Consequently its origin index contains the sequence -itself but not sequence elements or nested fields inside them. Relabeling that -index as `LayerUnderlay` leaves those descendants with the decoder's default -`LayerBase` attribution. Later overlay insertion recursively collects origins, -so attribution also depends on whether the same input is the initial tier. - -**Reproduction:** load `NoSource()` as main and an underlay mapping containing -`items: [{name: first}]`. The load succeeds. `Metadata.Lookup` reports -`LayerUnderlay` for `/items`, but `LayerBase` for `/items/0` and `/items/0/name`, -all referring to the same underlay source ID. `Metadata.History` repeats the -incorrect base attribution for the descendants. - -**Required correction:** cover every surviving input node when assigning origins, -including descendants of atomic sequence values. Keep strict-composition claims -and sequence replacement semantics unchanged. Ensure resolver entry metadata, -returned metadata, and explanation agree with each acquired source's role. - -**Regression coverage:** a first-tier sequence containing nested mappings, -including an imported contribution, retained beneath main input; compare with -equivalent later-tier insertion. Assert source ID and underlay role for elements -and their fields as well as the sequence container. - -## 2. Preserve later same-role mapping contributions in history (P2) - -**Owner:** `internal/engine/load_metadata.go:128` (`applyOverlayMetadata`). - -The final-history reconciliation skips an effective origin when its layer and -operation match the last recorded event. Multiple underlay tiers share -`LayerUnderlay`, so those two fields do not establish that the recorded event -contains the effective source contributors. The comparison cannot distinguish -contributions from repeated tiers of the same role. - -**Reproduction:** with no main document, supply three ordered underlays: - -```yaml -# Tier 1 -x: 1 -# Tier 2 -x: {} -# Tier 3 -x: {y: 2} -``` - -The final `/x` origin correctly includes sources 2 and 3. Its metadata history -contains only source 1 and the replacement by source 2; source 3's contribution -is omitted. The returned effective provenance and chronological history disagree. - -**Required correction:** determine whether the effective provenance is already -represented using contributing origins, not merely layer/operation equality. -Preserve meaningful same-tier-role contributions without duplicating events -already represented. Keep existing replacement, equal-value, and displaced-path -history behavior. - -**Regression coverage:** scalar-to-mapping replacement followed by a recursive -mapping contribution from a later underlay; assert the final contributor appears -in history and that metadata and explanation describe consistent source origins. - -## 3. Honor standard filesystem names in resource contexts (P2) - -**Owner:** `internal/engine/resource_context.go:38` and `:77`. - -Both resource adapters reject any name containing a backslash, beyond -`fs.ValidPath` validation. A backslash is a legal literal filename character in -an `io/fs` namespace and on the Linux host filesystem. Native source acquisition -accepts such paths, but resource-context access rejects them. This introduces an -unrequested path restriction and applies host-style separator assumptions to a -generic filesystem. - -**Reproduction:** supply `fstest.MapFS` with `config.yml` and a resource named -`asset\name`. Loading `config.yml` succeeds; the resource name passes -`fs.ValidPath` and can be read directly from the supplied filesystem. -The equivalent read through the returned context fails with `fs.ErrInvalid`: - -```go -context, ok := result.SourceContext(sourceID) // ok is true -_, err := fs.ReadFile(context.FS, name) // rejects the valid name -``` - -**Required correction:** use standard slash-path validation for native contexts -and preserve literal characters accepted by the supplied filesystem. For host -contexts, use platform-appropriate conversion while preserving the existing -contained-root behavior; native namespace semantics must not inherit Windows -separator restrictions. This is not a request to broaden root traversal or -change symlink guarantees. - -**Regression coverage:** access a literal-backslash resource through both native -Open/ReadDir paths where applicable and a Linux host context; retain invalid-path -and outside-root coverage. Avoid rejecting valid native names on other platforms -merely because the corresponding host path would be invalid there. - -## Review validation - -- `./scripts/check.sh` passed (tests, vet, build, formatting, and whitespace). -- `GOPROXY=off go test -race ./...` passed. -- Broad-catalog and many-small-tier benchmarks ran with `-benchmem` and 50 ms - samples. Allocated bytes scaled approximately with tested input sizes; these - measurements are review evidence, not timing gates. -- A temporary external Go program reproduced all three findings through the - public API; no diagnostic/test code was added to the repository. - -The intended facade, shared synchronous lifecycle, strict within-tier composition, -recursive precedence, mapping snapshotting, and passive resource-context design -are present. The findings above are correctness gaps within that design, not a -recommendation to expand its scope or change its selected combination semantics. diff --git a/docs/roadmap/sources-and-layering.md b/docs/roadmap/sources-and-layering.md deleted file mode 100644 index 9944af8..0000000 --- a/docs/roadmap/sources-and-layering.md +++ /dev/null @@ -1,457 +0,0 @@ -# Filesystem Sources and Source-Backed Underlays - -Status: implemented; draft ADRs remain pending review. This document preserves -the design target and rationale, including its historical baseline. The -[configuration contract](../config.md) owns current behavior, and the -[implementation plan](implementation.md) records sequential work and validation. - -## Purpose and priorities - -Address the filesystem and layered-input needs in the -[PromptKit request](feature-request-promptkit.md) as one coherent extension: -native `fs.FS` sources and explicit source-backed underlays, including -programmatic input. ConfigKit's recursive combination model takes precedence -over PromptKit's requested whole-definition replacement. Neither capability -should require temporary files, a generated root YAML document, or serialization -through YAML merely to transfer already available values. - -Design priorities, in order: - -1. Clear, standardized behavior consistent with ConfigKit's existing conventions. - PromptKit may adapt its formats and APIs; its historical conventions are not - new ConfigKit compatibility obligations. -2. Efficient loading and retained state, aiming for linear or near-linear work - in actual input and output size. Prefer clarity and maintainability where a - more elaborate optimization would compromise the first priority. -3. A small public surface with cohesive private implementation, usable by simple - applications without requiring them to learn the layered-loading machinery. - -Consumers load once and receive a complete configuration and its inspection -metadata. They should not need a second generic merge/defaulting pass, a separate -presence tracker, or reconstructed provenance. Domain-specific resolution remains -consumer-owned, but fits into the recorded loading lifecycle before the result -is validated and returned. Subject to consistent semantics and correct presence -and provenance, minimizing consumer integration code is an explicit design goal. - -The [named-inheritance request](feature-request-promptkit-inheritance.md) remains -separate and optional. This roadmap neither implements it nor adds speculative -infrastructure for it. - -## Planning baseline (historical) - -Planning baseline: `ec26591b275bd74f89120bdea69c5f6459dfcafb`, following v0.4.2. -Current behavior is owned by the [configuration contract](../config.md), not by -these proposals. - -- `Load[T]` takes one `File`, `Bytes`, `Reader`, or `NoSource` input. Imports - strictly compose documents; one named overlay uses a root-relative directory - convention; explicit CLI overrides follow it. -- Filesystem context uses host directories and an `os.Root`-based implementation. - Byte and reader labels do not imply filesystem paths. There is no native - `fs.FS` source or public batch of heterogeneous layers. -- Composition rejects non-mapping overlaps, including equal values. Overlay - recursively merges mappings and replaces other values. Empty mappings do not - clear existing children; null is an explicit value, not omission. -- An explicitly configured schema-version path currently belongs to the selected - source. Its checker runs before overlay/CLI input, which cannot alter that path; - imported fragments cannot contribute it. This roadmap preserves those ownership - and timing rules. -- CLI overrides already convert supported programmatic scalars, lists, and - string-keyed maps directly to the private document representation. They have - CLI-specific paths, labels, restrictions, and origins, not a general source API. -- Source records and resolution queries expose acquired-source identity and - provenance, but do not provide an `fs.FS` resource context. -- Root-owned public wrappers delegate to `internal/engine`; parser-independent - trees belong to `internal/document`, and YAML parsing stays in `internal/yaml`. -- Current mapping assembly uses repeated linear key searches. Overlay application - clones the accumulated document and provenance; subtree-origin removal scans - the whole origin index. Straightforward repetition across many layers would - amplify these costs. - -Existing source/lifecycle contracts, ownership, secret handling, and diagnostics -remain the compatibility baseline. This work is not permission to silently -reinterpret current `Load`, imports, selected overlays, or CLI overrides. -Redesigning schema-version policy is outside this roadmap's scope. - -## Target source model - -The extension preserves the -[synchronous, complete-on-return lifecycle](../config.md#completion-and-resource-lifetime). -Underlay assembly and the final configuration/metadata snapshot complete within -the load call. Resource-context retention is passive; resource access and any -subsequent loading require explicit application calls. The proposed architectural -rationale is recorded in [ADR 0008](../adr/0008-complete-on-return-loading.md). -The handle-ownership requirements below apply to the new source/context APIs as -they are implemented. - -### Native filesystem input - -A caller can supply a filesystem, an explicit file location within it, and a -meaningful diagnostic label/context. `embed.FS` is an ordinary supported example, -not a separate parsing mode. Use standard slash-separated `io/fs` path conventions; -do not apply host drive-letter or separator rules to virtual paths. - -The source retains the filesystem context needed to acquire its imports, glob -matches, and selected overlay. Relative imports resolve from the importing -document; named overlay lookup remains relative to that configuration context's -root. References must never fall back to the host filesystem simply because a -virtual path is missing. Imports/globs retain their explicit opt-in, deterministic -ordering, composition semantics, and actionable errors. - -Filesystem namespace identity and a normalized path identify a resource; labels -are presentation, not uniqueness keys. Identical names in different filesystems -must not collide. Use a reusable load-local context identity, not comparison or -hashing of arbitrary `fs.FS` interface values. Repeated-import and cycle checks -operate within the relevant input composition graph; selecting explicit layers -is distinct from accidentally importing a file twice. - -Keep the host-file backend's established canonicalization and containment behavior. -For generic `fs.FS`, describe namespace-relative resolution and the supplied -filesystem's actual behavior. `fs.FS` does not universally provide symlink -canonicalization or a security sandbox; do not invent such guarantees or reject -all implementations that cannot provide host-specific operations. The caller -chooses the filesystem, its acquisition behavior, and its lifetime. ConfigKit -closes handles it opens, not a caller-owned filesystem as a whole. - -### Origins and downstream resources - -Locations remain meaningful for host files, filesystem-relative files, labeled -bytes, and programmatic values. Preserve deterministic source IDs and traversal -order while distinguishing source identity, input kind, and precedence role. -Do not invent host filenames or line/column coordinates for synthetic inputs. - -A consumer can follow an effective field's origin to its contributing source and -obtain the context needed to resolve a relative resource. For a field inherited -through layering, that is the contributing source's context, not simply the last -layer's directory. Composite origins remain composite; ConfigKit must not choose -one contributing directory arbitrarily. - -Resource-context access is explicit and separate from serializable diagnostic -records. Querying metadata itself does not read resources. PromptKit owns opening -and interpreting message text, rendering, and handling resource errors. Context -retention must not imply that external resources are snapshotted with the loaded -configuration or remain valid after the caller closes their filesystem. - -The context contract covers caller-supplied filesystems and host-file sources. -ConfigKit closes every library-owned filesystem handle before loading returns, -on success or failure, including input files and directory/root handles. Returned -configuration, provenance, and resource-context records retain no such handles -and require no filesystem cleanup by the consumer. Configuration and provenance -remain usable independently of subsequent filesystem access. - -Caller-supplied filesystems and readers remain caller-owned. A supplied filesystem -may itself depend on handles whose lifetime the caller manages; retaining its -context does not extend that lifetime or transfer ownership to ConfigKit. - -Host-file resource contexts retain the information needed to open resources later, -preserving the host backend's established containment semantics. Later resource -access acquires its own handles: helpers close handles they use internally, and -callers close any resource handles explicitly returned to them. Host contexts -use a passive path-backed filesystem adapter that opens a fresh contained root -for each resource open and releases it with the returned file. Resource -opens observe the backing filesystem at access time and can fail independently -of an earlier successful configuration load. Sources without filesystem context -report its absence explicitly; composite origins expose their contributing -contexts rather than choosing a directory. These rules keep configuration -snapshot ownership separate from resource-access lifetime. - -## Target layering model - -### Explicit operations, not discovery order - -Extend public loading with explicitly supplied lower-precedence configuration -sources, using the same typed definition/result lifecycle as ordinary `Load`. -Support native filesystem, host-file, byte, reader, and programmatic inputs. -The central workflow is bundled defaults overridden by application configuration, -not an unrestricted sequence of user-selectable merge algorithms. Underlays can -form explicitly ordered default tiers; ConfigKit does not discover priorities or -know about PromptKit's catalog tiers. Source kind does not confer priority: -programmatic data can supply the appropriate supported input role without being -misrepresented as CLI flags. - -The precedence model, from lowest to highest, is: - -**Typed application defaults < source underlays < strictly composed main -configuration < selected overlay < explicit CLI overrides.** - -Typed defaults remain the existing Go-value baseline. Source underlays add -configuration-backed defaults with imports, filesystem context, and source -provenance; they are not a replacement for the typed-defaults facility. - -An underlay tier is an explicitly supplied group of source contributions. Each -group is strictly composed, including enabled imports, before it participates -in precedence assembly. Consumers supply zero or more tiers in lowest-to-highest -precedence order; later tiers overlay earlier tiers recursively. A source's -position within a group does not authorize replacement. The main input is -optional: an underlay-only load uses the same lifecycle, and a load with neither -main input nor underlays retains defaults-only behavior with supported explicit -overrides. Selecting a named overlay still requires its explicit lookup context. -The public representation exposes grouping and ordering directly. Empty groups -contribute nothing; explicit defaults-only sources also contribute no document. - -The selected operation determines how values combine, not their source format -or loading entry point. Equivalent file, embedded, byte/reader, and programmatic -input uses the same presence and combination rules. Strictly composing independent -contributions and intentionally overlaying precedence tiers express different -intentions; merely changing the input medium must not switch between them. -This applies to both ordinary `Load` and -the new layered-loading interface wherever they offer the same operation. -Consistency applies to combination, presence, and origin meaning. Source-specific -acquisition and lifetime rules remain explicit: a reader is single-consumption, -bytes are snapshotted, and filesystem input can observe backing-store changes. -CLI records retain their existing path restrictions. Source equivalence does not -promise identical reuse, locations, resource access, or diagnostic labels. -The existing opt-in version-path ownership rule remains an explicit metadata -constraint, independent of whether its owning source is a file or programmatic -input; it is not a source-format-dependent merge rule. - -Keep strict composition and intentional overriding visibly distinct: - -| Operation | Desired semantics | -| --- | --- | -| Compose independent contributions | Existing strict composition: recurse through mappings and report non-mapping conflicts with relevant source locations. | -| Overlay settings | Existing recursive mapping overlay: omitted children retain lower input; scalar, sequence, and null values replace. | - -Underlay is a precedence role, not another combination algorithm. Assemble the -main document and all its imports by strict composition before applying that -result over the underlays using recursive overlay. A conflicting main/import -definition still fails, even if it would overwrite an underlay or be overwritten -by the selected overlay. Never flatten main and imported documents into a -last-wins list. - -An underlay source may bring its explicitly enabled imports and source context. -Independent contributions within an underlay tier compose strictly; intentional -precedence between explicitly ordered tiers uses recursive overlay. Discovery -order does not establish tiers or authorize overriding. Syntax, source, and -composition failures cannot be hidden by higher-priority input. - -There is no whole-definition replacement mode, per-layer recursive/atomic switch, -collection-specific identity matching, implicit sequence merge, profile registry, -or merge callback in this feature. Existing atomic scalar, sequence, null, and -cross-kind replacement under overlay remains unchanged. - -Preserve the main configuration's version check after its strict composition and -schema construction, before precedence assembly with underlays, selected overlay, -and CLI contributions. Assemble the effective input before typed defaults/decoding, -application resolution, secret resolution, semantic validation, and final -snapshotting. Each lifecycle phase runs once per load, not once per layer. -Higher layers may replace values that would not fit the final typed schema; -effective typed validation is not validation of each intermediate snapshot. -Input syntax, source access, and composition remain validated at acquisition. - -Recorded application resolution follows input assembly and decoding. Named-overlay -lookup has an unambiguous filesystem context, separate from version ownership: ordinary -loading retains its source/root convention; a multi-context load explicitly -identifies the lookup context when that convenience is used. Other sources retain -their own import and resource contexts. Callers do not manufacture a primary -document merely to combine layers, and no overlay lookup context is needed when -no named overlay is selected. - -Higher-priority mappings retain omitted lower-input children. Empty mappings do -not clear those children; sequences, including empty sequences, replace as a -unit, and null remains explicit subject to the final schema. Typed defaults -continue to fill fields absent from the assembled input under existing rules. -No reset/delete operator is added. - -For example, a bundled profile containing `{model: example-model, timeout: 30}` -combined with application input `{model: another-model}` retains `timeout: 30` -from the bundle. Its model origin points to the application and its timeout -origin points to the underlay. PromptKit must validate that final profile rather -than assume the application entry discarded all bundled settings. This knowingly -differs from its requested identity-based whole-definition replacement. Consumer -examples should adopt the shared semantics, not implement a hidden replacement -pass after loading. A replacement operation could be considered separately if -future concrete needs justify it, but is not an open choice in this roadmap. - -### Preserve existing schema-version semantics - -An application opts into version checking by configuring a version path. A key -spelled `version` has no automatic special status. When checking is configured, -one designated input owns that declaration; fragments cannot contribute it and -underlays, overlays, or CLI overrides cannot supply or alter it, even to an equal -value. Preserve the current version-path and ancestor protections when integrating -the new input role. - -Keep the checker at its existing lifecycle point: after base composition and -schema construction, before underlay precedence assembly, selected overlays, -CLI overrides, defaults, and typed decoding. Preserve supported scalar forms, -absent/null distinction, -diagnostics, failure propagation, and version-path integer-to-string decoding. -An absent declaration remains absent for the checker; an invalid version type -fails without invoking it. Later input cannot mask a version-check failure. - -Filesystem sources and layered loading do not require a different version -policy. The designated main input owns the version when a `VersionRule` is -configured, extending the existing selected-source role. It may use any supported -source kind. With no main input, the checker receives an absent declaration, -consistent with existing `NoSource` behavior; underlays cannot supply that -declaration. Without version checking, no version owner or synthetic primary -document is required. -Named-overlay lookup context is a separate concern, not a reason to impose -version ownership on unversioned loads. - -Consumers continue to define version meaning and acceptance. Effective-version -last-wins validation, independent per-source version checks, schema selection, -and migration are not added by this feature. A future concrete need can motivate -a separate version-policy proposal; preserving the current contract here does -not settle that broader question permanently. - -### Programmatic presence - -Programmatic input has a non-secret source label and uses explicit structure to -represent presence. Omitted mapping keys are absent; present zero, false, empty -string, empty collection, and nil/null values remain explicit. Use ConfigKit's -existing canonical scalar and collection conventions, not YAML round trips or -Go zero-value inference. The initial representation is a labeled string-keyed -mapping using the scalar/list/map value set supported by explicit overrides. -This reuses established conventions and keeps presence directly expressible. -The source snapshots the mapping at construction, like byte sources. A nil root -mapping represents null and fails the existing mapping-root requirement; an empty -non-nil mapping is a valid empty document. - -Do not label generic programmatic sources as CLI input or misuse synthetic flag -names to fit the existing override API. Preserve parser-independent conversion, -safe errors, deterministic map ordering, and existing secret-reference semantics; -this is not a new route for embedding resolved credentials in document trees. - -Caller-owned mutable input is detached at a clearly documented ingestion -boundary. Results and inspection remain independently owned under existing -contracts. This does not promise language-enforced immutability of Go values or -an immutable filesystem. - -A final Go value alone need not encode presence: an omitted boolean and an -explicit `false` can both decode to `false`. The returned configuration together -with its presence/provenance metadata preserves that distinction through -defaulting, layering, and recorded resolution. Null, empty values, zero, and -absence must not be collapsed by source conversion or merging. - -### Provenance and operational boundaries - -Origins, presence, history, explanation, and effective rendering agree with the -final typed result. Overridden and replaced definitions remain explainable without -retaining stale effective descendants. Replacements identify their scope and -displaced sources; equal-value explicit overrides are still meaningful history. -Include typed defaults and application-derived values, not only source-supplied -values, in final-value provenance. Retain redaction and callback-lifetime behavior. - -Apply existing resource-accounting principles across the whole load rather than -resetting allowances for every layer. Audit how bytes, programmatic nodes, -accepted source occurrences, and glob work are counted without accidental double -charging. Do not add arbitrary new budgets or infer that efficiency work waives -existing ones; any needed adjustment must be explicit in the later design. - -## Efficiency target - -Measure cost against total acquired bytes/nodes, reference and path data, final -materialized values, and retained provenance/history. Layer count alone is not -a useful size measure. Large history or output can legitimately dominate cost. - -Aim for O(N + H + R) processing and storage where N is total input/visited-path -work, H is retained history/provenance, and R is produced result data. Account -separately for deterministic sorting, filesystem traversal, schema processing, -and caller callbacks. This is an engineering target, not an unconditional public -complexity guarantee or a fixed performance gate. - -The target implementation: - -- Acquires/parses each accepted source occurrence once and directly converts - programmatic input. It does not decode or serialize accumulated configurations - between layers. Respect single-consumption readers and mutable file sources; - no implicit cross-load cache is needed. -- Uses indexed lookup alongside ordered mapping storage, retaining deterministic - iteration and diagnostics without repeatedly scanning growing mappings. -- Builds one load-owned working assembly instead of cloning the accumulated - configuration and history for every added layer. Detach at meaningful ownership - boundaries; safely share immutable private history where appropriate. -- Updates affected subtrees and their provenance rather than scanning all fields - for every replacement. Avoid repeatedly copying ever-growing ancestor-origin - lists or rechecking the same conflict group for every member. -- Reuses one schema/lifecycle evaluation per load and freezes the final result - once. It does not require global caches, copy-on-write public APIs, or a general - persistent-tree framework merely to meet an asymptotic target. - -Use representative benchmarks and allocation measurements for broad keyed -catalogs, many small underlay tiers, repeated scalar/sequence replacement, -imported filesystem defaults, and retained history. Compare growth as inputs -scale, including small common -loads. Favor changes addressing demonstrated scaling problems; preserve stable -diagnostic order and ownership rather than trading them for a superficial O(n) -claim. No timing threshold or additional release gate is requested. - -## Work needed from the baseline - -The target requires changes in the following responsibility areas; the linked -implementation plan owns their execution order and concrete validation: - -- Extend the root-owned source/context facade and private acquisition boundary - for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling - without exporting parser or engine types. -- Add source-backed underlays and programmatic input, sharing one engine path - with ordinary loading rather than building a parallel configuration system. -- Extend source records, histories, diagnostics, and rendering consistently - across input kinds. -- Integrate version-owner selection into layered loading while preserving the - existing version lifecycle and protections. Keep filesystem-context selection - independent of version checking. -- Refactor mapping assembly and provenance updates where current algorithms - would scale poorly under layered workloads. Keep optimization scope tied to - the new workflow rather than rewriting unrelated library subsystems. -- Add focused behavioral coverage for mixed sources, relative resources, - namespace collisions, import failures, composition conflicts, explicit empty - and null values, caller/result independence, and source-correct redaction and - explanation. Add comparative benchmarks without making performance a brittle - pass/fail contract. -- Compare equivalent logical inputs across supported source kinds and loading - interfaces. Assert consistent values, presence, operation outcomes, and origin - meaning while allowing genuine location/label differences. Cover defaulted and - derived values, recursively retained underlay fields, unchanged atomic-value - replacement, typed defaults, strict main/import conflicts, and preservation of - version ownership/check timing across both loading interfaces. Verify that - unversioned layered loads need no synthetic version-owning root. -- Provide maintained simple and PromptKit-shaped examples: embedded defaults, - operator configuration from another context, and labeled programmatic input. - Show explicitly that a same-name profile recursively combines, rather than - replacing its whole lower definition. - Demonstrate exact operations and resource origins without provider calls or - recreating PromptKit itself. The consumer supplies sources, explicit operations, - types, and domain rules; it does not implement generic merging, presence - tracking, or origin reconstruction after loading. -- Update Go declaration documentation, the configuration contract, consumer - guidance, component overview, and relevant architecture/deferred-scope records - as behavior lands. Record significant boundary changes in an ADR without - rewriting accepted ADR history. Keep named inheritance explicitly separate. - -The capability is additive and intended for a minor release, preserving current -version semantics. Release preparation and publication remain separate tasks. - -## Completion criteria - -A consumer can load embedded configuration with relative imports, combine it -with independently located operator configuration and explicit programmatic -values, and obtain one typed result with accurate presence, sources, history, -and resource context. No temporary configuration files or serialization round -trip or subsequent generic configuration processing is needed. Simple `Load` -usage retains its current behavior, including version ownership and validation -timing. Layered loading uses the same version policy. Final-value provenance covers input, -defaults, and recorded domain resolution, and metadata distinguishes explicit -zero/false from absence even when the typed values are equal. Benchmarks -demonstrate that adding inputs does not entail avoidable full-collection copying -or quadratic key lookup. Public contracts and examples clearly distinguish -strict within-tier composition, recursive cross-tier overlay, and downstream domain -resolution. Equivalent-input tests and the maintained consumer example establish -semantic consistency and reduced integration code, not merely API availability. - -## Selected resource-context interface - -A separate source-ID-based query exposes standard filesystem access and a -source-relative base location through root-owned API types. Ordinary source -records remain descriptive. The query reports absent context explicitly and -supports imported sources as well as initial inputs. Consumers follow each -contributing origin independently for composite values. This avoids a mandatory -downstream filesystem registry and preserves provenance through recursive -combination. - -Caller-supplied filesystems are borrowed capabilities; host contexts are passive -path-backed adapters. Neither is a cloned or immutable filesystem. Queries perform -no I/O, and later resource opens follow the ownership rules above. Configuration -results retain no library-owned open handles.