diff --git a/README.md b/README.md index f67d0f7..bafae96 100644 --- a/README.md +++ b/README.md @@ -41,6 +41,21 @@ load once, inspect where final values came from, and distinguish explicit values from omissions with minimal configuration bookkeeping. Applications retain their domain rules and execution responsibilities. +Loading is synchronous and complete on return. Applications normally load once +near startup and then use the final configuration and metadata; ConfigKit stays +inactive until explicitly called again. Applications may initiate additional +loads as needed. See the [loading lifecycle](docs/config.md#completion-and-resource-lifetime) +for completion and ownership details. + +Within a configuration tier, independent contributions compose strictly: shared +mappings combine, while overlapping non-mapping definitions are errors. Between +precedence tiers, mappings combine recursively, retaining omitted lower-priority +children; other values replace lower-priority values. Today, the main document +and its imports form the strictly composed base, with a selected overlay and +explicit CLI overrides above it and typed defaults below it. The +[configuration contract](docs/config.md#strict-document-composition) owns the +detailed combination rules and input-specific constraints. + The [architecture goals](docs/policy/architecture.md#success-criteria) explain these priorities and how genuine new needs can justify extending the shared model. The [deferred scope](docs/roadmap/deferred.md) records deliberate exclusions. diff --git a/docs/adr/0007-source-backed-underlays.md b/docs/adr/0007-source-backed-underlays.md new file mode 100644 index 0000000..4cd78f3 --- /dev/null +++ b/docs/adr/0007-source-backed-underlays.md @@ -0,0 +1,118 @@ +# ADR 0007: Add Source-Backed Underlays with Consistent Recursive Precedence + +## Status + +Proposed draft for review. The +[filesystem sources and underlays roadmap](../roadmap/sources-and-layering.md) +owns implementation status, detailed target behavior, and remaining API choices. + +This extends [ADR 0001](0001-separate-composition-and-overlay.md) and preserves +its distinction between strict composition and intentional overlay. + +## Date + +2026-09-29 + +## Context + +Configuration-backed defaults provide lower-priority settings that applications +can override. These sources carry imports, locations, and relative-resource +contexts. Integrating them into ConfigKit's loading lifecycle gives their values +the same presence tracking and inspectable provenance as other configuration +inputs. + +The [architecture goals](../policy/architecture.md#success-criteria) prioritize +one coherent, predictable configuration model with consistent rules across +source representations and loading interfaces. Consumers should load once, +receive a complete validated configuration, and inspect its final values' origins. +Reducing integration code is important subject to those principles. + +## Decision + +### Add underlays to the existing precedence model + +Introduce explicitly supplied source-backed underlays between typed defaults +and main configuration. The proposed precedence, from lowest to highest, is: + +**Typed application defaults < source underlays < strictly composed main +configuration < selected overlay < explicit CLI overrides.** + +Typed defaults remain the Go-value baseline. Underlays supply lower-priority +configuration with source provenance and, where applicable, imports and resource +context. This ordering describes value precedence; typed defaults fill fields +absent from the assembled input under the existing defaulting rules. + +Preserve strict composition of the main document and its imports before applying +the resulting configuration over underlays. Independent contributions within an +underlay tier also compose strictly. Explicitly ordered tiers express intentional +precedence. Strict composition reports conflicting contributions within each tier. + +Across precedence tiers, reuse the existing recursive mapping-overlay semantics. +Omitted children retain lower-priority input; existing scalar, sequence, null, +and cross-kind replacement semantics remain unchanged. Reusing these operations +keeps the combination rules consistent across the loading model. + +### Keep source representation separate from composition semantics + +Native `fs.FS` and programmatic inputs join the shared loading model. +Equivalent inputs using the same +operation have the same presence and precedence semantics regardless of medium. +Programmatic values enter directly as labeled configuration sources. + +Retain the contributing source's identity and resource context through layering. +A retained underlay value keeps its originating source's resource context. +Consumers remain responsible for opening and interpreting their domain resources +under the supplied filesystem's lifetime and access semantics. Exact +context-access and input APIs remain roadmap decisions. + +### Deliver one final configuration and its inspection metadata + +Assemble effective input, then run the typed loading lifecycle once for the load. +Preserve explicit zero, +false, empty, and null input versus omission through configuration and metadata +together. + +Extend existing provenance and history through underlays, defaults, and recorded +application resolution. The returned result includes the completed combination, +defaulting, and provenance work. Domain resolution and validation +remain application-owned within the recorded lifecycle, consistent with +[ADR 0003](0003-preserve-presence-and-resolution-history.md). + +Preserve the existing opt-in schema-version ownership, path protections, and +check timing before precedence +assembly and typed decoding. The roadmap owns how layered input identifies that +owner when version checking is configured. + +Favor near-linear processing in input, retained history, and output size through +incremental assembly of load-owned state. This is an engineering objective +subordinate to clarity and semantic consistency. Preserve the root-owned facade and +private implementation boundary established by [ADR 0006](0006-public-facade.md). + +## Alternatives considered + +- **Leave bundled defaults to consumers.** Avoids extending loading, but makes + consumers reproduce merging, presence tracking, and provenance reconciliation. + It does not meet the goal of obtaining a complete configuration in one load. +- **Offer recursive or atomic combination per layer.** Makes both behaviors + explicit and could serve more consumers, but expands the semantic surface and + its interactions with presence and provenance without a demonstrated need that + outweighs a single consistent rule. A future concrete requirement may justify + reconsideration; this feature does not add the option. +- **Flatten all documents into last-wins layers.** Simplifies ordering but turns + accidental main/import conflicts into overrides, weakening the intentional + distinction established by ADR 0001. + +## Consequences + +Consumers gain source-backed defaults within the shared loading lifecycle. +Strict contributions and intentional precedence remain distinct, with consistent +combination rules across source media. Presence and provenance remain +part of the same loading lifecycle as the final values. + +Higher-priority mappings retain omitted lower-priority children, so application +validation assesses the final combined configuration. + +Implementation must integrate source context and underlay history into the shared +engine with attention to assembly and history-retention costs. Current behavior +documentation will change when the feature lands. The roadmap retains ownership +of the remaining API decisions. diff --git a/docs/adr/0008-complete-on-return-loading.md b/docs/adr/0008-complete-on-return-loading.md new file mode 100644 index 0000000..59bfc25 --- /dev/null +++ b/docs/adr/0008-complete-on-return-loading.md @@ -0,0 +1,83 @@ +# ADR 0008: Synchronous, Complete-on-Return Configuration Loading + +## Status + +Proposed draft for review. The +[configuration contract](../config.md#completion-and-resource-lifetime) describes +the implemented loading lifecycle. The +[source-backed underlays roadmap](../roadmap/sources-and-layering.md) owns its +application to the planned source and resource-context APIs. + +## Date + +2026-09-29 + +## Context + +Applications need a clear point at which configuration is ready for use. +Completing configuration processing and inspection metadata together gives +applications a coherent result whose values and origins agree. A bounded loading +lifecycle also makes resource ownership and startup failures straightforward to +reason about. + +ConfigKit's [architecture goals](../policy/architecture.md#success-criteria) +emphasize complete configuration, inspectable provenance, predictable behavior, +and minimal consumer bookkeeping. Applications own their execution lifecycle and +the timing of configuration acquisition. + +## Decision + +Make loading a synchronous, complete-on-return operation. Within each load, +ConfigKit completes input assembly, defaulting, typed decoding, configured +application resolution, secret lookup, validation, and snapshotting before +returning the final configuration and metadata. Reported failures return +diagnostics and a nil result. + +Release all library-owned loading handles before returning on success or failure. +The resulting configuration and metadata are independently usable, caller-owned +data with a retained inspection snapshot. Caller-supplied readers and filesystems +remain under caller ownership and lifetime control. + +After the call completes, ConfigKit remains inactive until explicitly invoked +again. Applications normally load once at or near startup and may initiate +additional loads when needed. Each load follows the same lifecycle and source +reuse contracts. Inspection, rendering, and separately requested operations run +in response to explicit application calls. + +For the planned resource-context capability, retain source context as passive +information or a borrowed caller-supplied filesystem. Subsequent resource access +acquires its own handles with explicit ownership. This preserves a clear boundary +between configuration completion and application resource use. + +These choices give consumers one completion boundary for configuration and +provenance, predictable resource lifetimes, and control over when ConfigKit runs. +Application callbacks remain responsible for their own effects and resources. + +## Alternatives considered + +- **Background watching or refresh.** Can automate responses to source changes, + but adds scheduling, shutdown, publication, and resource-lifetime concerns to + the library. Explicit calls keep those responsibilities with the application. +- **Lazy completion after return.** Can defer work until particular values are + used, but moves configuration failures into application execution and makes + the readiness of values and metadata harder to establish. +- **Transactional atomic loading.** Could imply simultaneous observation of all + external inputs and rollback of callback effects. Filesystems, environment + variables, and application callbacks do not share a transaction mechanism. + Synchronous completion provides the required usable-result boundary without + promising those external guarantees. +- **Retain loading handles in results.** Can preserve open filesystem access, + but gives configuration results an ongoing resource lifecycle and cleanup + obligation. Passive source context supports later explicit resource access + while keeping loading resources scoped to the load call. + +## Consequences + +Applications can treat successful return as the point at which the final +configuration and its metadata are ready for use. Startup loading remains simple, +and additional loads have an explicit application-owned trigger. + +Configuration inspection operates on retained data. Resource access observes the +backing resource when explicitly requested and has its own failure and ownership +boundary. Implementations of new source kinds and layering features preserve the +same completion boundary, including cleanup on reported load failures. diff --git a/docs/config.md b/docs/config.md index fa85f2f..80f049e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -71,6 +71,31 @@ the returned configuration or inspection records cannot alter ConfigKit's privat snapshot. ConfigKit performs no logging. `Result.WriteEffective` exposes only the frozen redacted projection. +### Completion and resource lifetime + +`Load` is synchronous and complete on return. Success means input assembly, +defaulting, typed decoding, configured application resolution, secret lookup, +validation, and configuration/metadata snapshotting have completed. A reported +load failure returns diagnostics and a nil result, as described above. + +Before returning on success or failure, ConfigKit closes the filesystem handles +it opened for loading, including files and directory/root handles. The returned +configuration and inspection metadata require no close operation and remain +usable independently of the source files. A supplied `Reader` remains +caller-owned and is never closed by ConfigKit. + +ConfigKit performs no background work after returning: it does not watch inputs, +refresh secrets, or mutate published results. Inspection and rendering run only +when explicitly called and use the retained snapshot. Applications normally load +once near startup, but may explicitly load again using sources that satisfy their +documented reuse rules. Additional loads produce independently owned results. + +Completion is not a transaction over external state. Files and environment +variables are observed during processing rather than as one simultaneous external +snapshot. ConfigKit does not roll back application callback effects or manage work +or resources created by application callbacks. The explicit secrets-directory +bootstrap remains a separate operation invoked by the application. + ### Application schema versions `VersionRule` declares a semantic path and non-nil checker. Before defaults or typed diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 1a9f4fb..6e68179 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -108,6 +108,22 @@ must never run implicitly as part of configuration loading. Remote data can enter through application-supplied input without introducing a network provider into the library. +## Loading Lifecycle + +Configuration loading is a synchronous, complete-on-return operation. A successful +load provides the final configuration and inspection metadata after the configured +processing and validation phases complete. A failed load reports diagnostics and +returns no usable partial result. The normal application pattern is one load near +startup; applications may explicitly initiate additional loads. + +ConfigKit releases its library-owned loading handles before returning and remains +inactive until the application explicitly invokes it again. It performs no +background watching, refresh, or mutation of published results. Caller-owned +resources and application callback effects remain the caller's responsibility. +The [configuration contract](../config.md#completion-and-resource-lifetime) owns +the precise completion and ownership behavior. The proposed decision record is +[ADR 0008](../adr/0008-complete-on-return-loading.md). + ## Public API And Dependency Direction Keep one small, coherent public surface for the supported lifecycle. Simple @@ -136,9 +152,20 @@ framework on this library. ## Configuration Integrity -Composition and intentional overlay are distinct operations. Composition must -not silently become last-wins override through file ordering. The decision and -alternatives are recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md). +Within a configuration tier, independent contributions use strict composition: +shared mappings combine recursively, while overlapping non-mapping definitions +are conflicts, including equal values. Between precedence tiers, intentional +overlay combines mappings recursively, retains omitted lower-priority children, +and replaces other present values according to the existing overlay rules. +Tier precedence is explicit; traversal order within a tier establishes stable +reporting, not permission to override. The distinction and its rationale are +recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md). + +The main document and its imports currently form the strictly composed base +tier. Input-specific constraints, including CLI path rules and opt-in version +ownership, remain part of the configuration contract. Consistent combination +semantics do not require identical acquisition or lifetime behavior from every +source kind. ConfigKit must define supported YAML semantics explicitly rather than inherit parser defaults. Processing order and diagnostics must be deterministic for the diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..c9370e5 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,453 @@ +# Filesystem Sources and Underlays Implementation Plan + +## Purpose and baseline + +Implement the [feature roadmap](sources-and-layering.md) in the numbered order +below. Each stage is sized as one focused implementation assignment for a +gpt-5.6-terra coding agent. The baseline is +`36172ed20e832b371506d6aceb02dc6c8f50f71b`, plus the documentation changes committed +with this plan. No feature implementation is part of this planning pass. + +Read `docs/development.md` and the policies it selects for the stage. The roadmap +owns target semantics; [ADR 0007](../adr/0007-source-backed-underlays.md) and +[ADR 0008](../adr/0008-complete-on-return-loading.md) record the proposed rationale. +Their draft status is independent of implementation progress: do not mark them +accepted automatically. The [configuration contract](../config.md) is the +compatibility baseline. Preserve the root-owned facade and private engine/tree +boundary. + +At each stage, preserve unrelated work, add focused behavioral coverage with the +change, update declaration documentation for public additions, and run the checks +specified below. Keep intermediate commits building and tested. Do not expose +placeholder public APIs that silently ignore options. Internal scaffolding may +precede the public feature. Use plain-English commits under the agent policy. + +## Settled implementation decisions + +### Public input surface + +Use these names and shapes; bodies and private fields belong to implementation: + +```go +// Root-owned opaque filesystem namespace, copying preserves namespace identity. +type Filesystem struct { /* private engine state */ } +func NewFilesystem(label string, filesystem fs.FS) Filesystem +func (f Filesystem) File(name string) Source +func Mapping(label string, values map[string]any) Source + +// Extend the existing record; nil FS selects host-filesystem context. +type FilesystemContext struct { + BaseDir string + Root string + FS *Filesystem +} + +type Input struct { + Source Source + Imports bool + Filesystem *FilesystemContext +} +type Underlay struct { Inputs []Input } +type LayeredOptions[T any] struct { + Definition Definition[T] + Underlays []Underlay + Overlay string + OverlayFilesystem *FilesystemContext + Overrides []Override +} +func LoadLayered[T any](main Input, supplied ...LayeredOptions[T]) (*Result[T], error) + +type ResourceContext struct { + FS fs.FS + BaseDir string +} +func (r *Result[T]) SourceContext(id string) (ResourceContext, bool) +func (r *Resolution) SourceContext(id string) (ResourceContext, bool) +``` + +`fs` above means the standard `io/fs` package. All named public types are defined +at the root, not aliases of engine types. Existing `Load` and `Options` stay +source-compatible apart from the additive field in `FilesystemContext`. Avoid a +parallel lifecycle implementation: adapt both entry points to one private load +request. No new dependency or minimum-Go-version change is needed. + +- Like `Load`, `LoadLayered` accepts zero or one options value. A zero `Source` + stays invalid. Represent omitted main input as `Input{Source: NoSource()}`. + `NoSource` contributes no document, source ID, or context; preserve its existing + invalid import/context combinations. An empty underlay group is a no-op. +- Each underlay contains strictly composed inputs, including enabled imports. + Groups are ordered lowest to highest. Repeated explicit inputs in different + tiers are allowed subject to each source's reuse contract. Within a tier, + repeated file identities remain import/repetition errors. Equal scalar claims + still conflict. Independent synthetic sources are distinguished by occurrence. +- `Input.Imports` opts in per input and its import graph. The main source and its + imports form the main tier. There is no programmatic high-priority escape hatch: + a mapping source can be main input or any underlay contribution; CLI overrides + retain their explicit-flag contract. +- Active named overlays in `LoadLayered` require `OverlayFilesystem` explicitly. + Their names, root-relative lookup, directive prohibition, and version protection + follow existing rules. An inactive overlay needs no context; an unused explicit + context has no effect. Ordinary `Load` retains implicit selection of its source's + filesystem context and existing `NoSource` restrictions. + +### Filesystem namespaces, paths, and acquisition + +`NewFilesystem` allocates a reusable private namespace token and retains the +caller-owned filesystem without opening it. Copies share the token; separate +constructors are separate namespaces even for the same underlying filesystem. +Never compare or hash arbitrary `fs.FS` values. Validate nonempty safe labels and +nil filesystem arguments through deferred source diagnostics, following existing +source-constructor conventions. The zero `Filesystem` yields an invalid source. + +`Filesystem.File(name)` uses a non-root `fs.ValidPath` file name relative to that +namespace. The default root is `.` and the base is the file's containing directory. +An optional filesystem context can narrow the root; file sources continue to +derive their own base directory. An explicit FS must identify the same namespace +as the file source; context overrides for native file sources supply that FS. +A host `File` cannot be rebound to a virtual namespace. +For byte, reader, and mapping inputs, an explicit context supplies their namespace +and base: with FS set, require valid slash-relative Root and BaseDir (use `.` for +the namespace root), with BaseDir inside Root. With FS nil, preserve the existing +absolute host-directory requirements. Labels never imply a context. + +Resolve import paths relative to the declaring document, normalizing joined +paths and rejecting escape from the configured root. Preserve the existing +relative-path/glob language, including contained parent references, optional +missing behavior, sorted matches, bounded traversal, and cycle diagnostics. +Virtual paths use `path`, not host `filepath` rules. Imports stay in their namespace; +generic filesystems provide their own symlink behavior. Preserve canonicalization, +regular-file checks, and `os.Root` containment for host acquisition. Use ordinary +`fs.FS` operations and `fs.ReadDirFile` where available; support the standard +`fs.ReadDir` fallback without claiming control over its internal allocation. + +Configuration reads close each file ConfigKit opens, including read/stat/parse +failures. Caller readers and caller filesystems are never closed by ConfigKit. +Namespace tokens live with source objects; deterministic load-local IDs are +assigned by acquisition, not addresses or global counters. + +### Programmatic representation and diagnostics + +`Mapping` snapshots its input at construction, matching `Bytes`. Convert directly +to the private document tree using the CLI converter's existing built-in scalar, +`[]any`, and `map[string]any` repertoire. Preserve integer precision, finite-float +validation, sorted keys, sequence order, explicit null, and cycle rejection. +Typed nil maps/slices mean null, as in existing overrides. A nil root mapping is +invalid because document roots are mappings; `{}` is a valid empty mapping. +Constructor failures are stored and surfaced on load. Detach once at construction; +do not retain caller maps or serialize them. Concurrent caller mutation during +construction is outside the supported ownership contract. + +Generic conversion returns private issues; adapt them to existing `override.*` +codes for CLI input, and `source.invalid` or `source.limit_exceeded` for mapping +input. Do not leak raw values or conversion errors. Source/path locations use +the label and semantic path, with zero line/column for synthetic input. Apply +reserved `configkit` directive rules consistently: with imports enabled, a +mapping can contain the same directive structure as YAML. Without context, +relative imports fail under existing context rules. + +### Ordering, lifecycle, budgets, and metadata + +Acquire and strictly compose main input first, construct the schema, and check +the optional version. Only the main source owns the version; main imports and +all underlay documents must satisfy fragment-style version-path protections. +With `NoSource` main input, pass an absent version to the checker. Do not run the +checker once per tier or validate the final overlaid version instead. + +Then acquire/compose underlays in listed order and combine them recursively, +apply the composed main tier, selected overlay, and CLI overrides. Execute typed +defaulting/decoding, recorded resolution, secret lookup, validation, and final +publication once. Syntax/composition/version failures cannot be masked by higher +precedence; effective typed validation sees the final assembled input only. + +Keep source-record order separate from precedence: main depth-first sources, +underlay tiers in order with each input traversed depth-first, selected overlay, +then CLI. IDs remain `source-N`. Existing ordinary loads retain their source order. +The history application order follows precedence, not acquisition order. + +Extend `Layer` with `LayerUnderlay = "underlay"`. Add to `SourceInfo` a `Kind +SourceKind` and `UnderlayIndex int` (one-based original tier position for underlay +sources, zero otherwise). Define `SourceKindFile`, `SourceKindFilesystem`, +`SourceKindBytes`, `SourceKindReader`, `SourceKindMapping`, and `SourceKindCLI` +with string values `file`, `filesystem`, `bytes`, `reader`, `mapping`, and `cli`. +Imported documents inherit tier role/index and have their actual acquisition kind. +Preserve existing host Path meaning; filesystem-source Path is namespace-relative; +synthetic Path stays empty. Labels for native files combine namespace label and +relative path for display, while identity uses the token and path. + +Origins continue to refer to source IDs through locations. Composite origins +preserve all contributors; equal-value overrides and displaced descendants stay +inspectable. Carry source context separately from serializable source records. +Keep existing inspection schema and renderer structure, adding the new layer/kind +where appropriate rather than changing existing output gratuitously. + +Use one load-wide accounting object for layered loads: existing 64 MiB aggregate +input bytes, 1,024 acquired sources, 1,000,000 input nodes, and glob-entry budget; +existing per-source 8 MiB, depth 256, active import depth 64, and retained-history +limit remain. Mapping key/string bytes use the existing CLI converter's byte +accounting. Charge each accepted occurrence once, including CLI contributions; +do not charge intermediate copies as new input. Explicit empty groups and +`NoSource` consume no source allowance. Ordinary loads retain their established +limit behavior. Use existing source/import/overlay/override limit codes according +to the acquisition that exceeds a budget. Add `CodeUnderlayVersion = +"underlay.version"` for version protection in underlay roots; imported fragments +retain `import.version`. Avoid a separate arbitrary tier-count limit. + +### Passive resource contexts + +`SourceContext` returns false for unknown IDs or sources without a context, and +performs no I/O. The result method is nil-safe; the resolution method follows +existing callback lifetime guards. Return a record copy containing a borrowed +filesystem and slash-relative source BaseDir within its configured root. +Consumers obtain source IDs from ordinary origins; there is no single-context +query that guesses among composite or derived origins. Dependencies continue to +identify the sources of application-derived values through existing metadata. + +For native inputs, use a namespace-rooted view over the caller's filesystem and +retain its lifetime semantics. For host input, retain the canonical root path, +not an open `os.Root`. A private `fs.FS` adapter validates a relative resource +name, resolves it using existing host containment semantics, opens a fresh root, +and opens the resource. Return a private `fs.File` wrapper whose Close releases +both file and root, including failed-open cleanup. Expose `fs.ReadDirFile` when +the underlying directory supports it. Resources themselves are application data: +do not apply configuration YAML parsing or configuration-byte budgets to them. +Helpers such as `fs.ReadFile` then close the returned handle normally. Errors +follow filesystem access conventions, separately from load diagnostics. + +Neither queries nor retained results keep library-owned filesystem handles open. +Files/resources can change or disappear after loading; configuration/metadata +remain usable. No background work, hidden reopening on metadata queries, +filesystem snapshot, universal sandbox, or transaction guarantee is added. + +## Stage 1: Index mapping assembly and establish scaling baselines + +Scope: `internal/document` and engine composition/overlay lookup, with existing +behavior only. + +1. Add parameterized benchmarks for wide disjoint composition and recursive + overlay, recording runtime and allocations at increasing sizes. Keep fixtures + within existing budgets and record the baseline in the stage handoff. +2. Replace repeated linear mapping-key lookup during bulk assembly with a private + indexed builder alongside ordered entries. Build each target index once per + assembly session and update it for appended keys. Keep tree traversal and + diagnostic order stable; avoid mutable indexes exposed through public APIs. +3. Check each strict-composition claim group once while preserving first-conflict + ordering and all relevant locations. Preserve original input ownership. +4. Extend existing composition tests only for uncovered ordering/conflict risks. + +Acceptance: existing Compose/Overlay tests pass unchanged; broad keyed input +no longer incurs a full key scan for each insertion. Run the focused composition +tests and the everyday checker. Benchmarks are evidence, not timing gates. + +## Stage 2: Prepare load-owned assembly and incremental provenance + +Scope: `compose.go`, `overlay.go`, origin indexes, override application, and +history replay in `internal/engine`. + +1. Introduce a private load-owned accumulator with explicit ownership. Existing + copy-preserving helpers may wrap it, but sequential assembly must mutate one + owned working tree rather than clone all accumulated state per tier. +2. Retain immutable acquired inputs for history, detached from mutable working + nodes. Parameterize internal overlay application by incoming layer and origins + so a composed tier's imported contributors are preserved. +3. Remove displaced origins by visiting the affected subtree. Accumulate mapping + contributors without copying an ever-growing ancestor list on each addition; + materialize defensive public records at the existing publication boundary. +4. Preserve equal-value replacement events, descendant displacement, redaction, + and typed-decoding reconciliation. Extend benchmarks for many small applications + and repeated subtree/scalar replacement. Do not add a persistent-tree framework. + +Acceptance: existing override/history/rendering/ownership tests pass; benchmark +growth avoids accumulated full-tree copying. Run the everyday checker and +`GOPROXY=off go test -race ./internal/engine` for ownership changes. + +## Stage 3: Unify private filesystem acquisition without public changes + +Scope: engine source/file, filesystem, import, glob, and selected-overlay code. + +1. Extract a small private namespace/access boundary used by imports and overlays. + Keep the host implementation's canonicalization, containment, regular-file + checks, safe diagnostics, deterministic traversal, and close behavior. +2. Separate namespace identity from source labels and resource paths. Make + repeated/cycle tracking belong to an input composition group, and accounting + capable of being shared across groups. +3. Add a native `fs.FS` backend internally. Resolve paths using slash semantics, + preserve glob grammar and optional-missing behavior, and avoid host fallback. +4. Use real host tests plus `fstest.MapFS` and a small tracking/failing filesystem + to cover imports, missing paths, directory/read failures, and handle closure. + Include a filesystem with a non-comparable dynamic value. + +Acceptance: all existing host source/import/glob/overlay tests remain green; +native backend tests establish namespace isolation and supported paths. Run the +everyday checker. No public constructors land until the next stage. + +## Stage 4: Expose native filesystem sources through the facade + +1. Add `Filesystem`, `NewFilesystem`, `Filesystem.File`, and the additive FS + context field at root and engine boundaries. Constructors follow deferred + diagnostics; copying the wrapper preserves namespace identity. +2. Integrate native sources with ordinary `Load`, including explicitly enabled + imports and named overlays. Enforce the path/context rules settled above. +3. Add `SourceKind` and populate it for every current/native acquisition. Preserve + host paths and use namespace-relative virtual paths with accurate locations. +4. Add facade-level tests using embedded fixtures, `fstest.MapFS`, byte/reader + context, namespace-label collisions, and source reuse. Keep backend error cases + at their Stage 3 owner. Update current source/API docs to this implemented scope. + +Acceptance: native `Load` works through the public API and existing sources retain +their contracts. Run the everyday checker, including public-boundary tests. + +## Stage 5: Add snapshotted mapping sources + +1. Extract reusable value conversion from `override_value.go`; keep CLI-specific + path, label, and diagnostic handling in its adapter. +2. Implement `Mapping` at root and engine boundaries with constructor-time + snapshotting and deferred safe failures. Source-ID assignment at acquisition + must not mutate a reusable stored tree shared by another load. +3. Route converted documents through the same directive processing, strict + composition, version checking, decode, and inspection path as YAML sources. +4. Test omission/zero/false/null, nil versus empty collections/root, exact integers, + invalid/cyclic values, imports with explicit context, and caller mutation after + construction. Verify mapping input does not gain CLI origins or restrictions. + +Acceptance: equivalent byte/mapping inputs have equivalent semantic results and +origin meaning, with genuine location differences allowed. Existing override +diagnostics remain stable. Run the everyday checker and engine race tests. + +## Stage 6: Implement private grouped-underlay orchestration + +1. Add private request/input/group structures matching the settled facade design. + Adapt ordinary `Load` to the shared core while preserving its options and + source-order behavior. Keep its existing validation order where observable. +2. Strictly compose each group with group-scoped repetition tracking and shared + aggregate accounting. Apply the lifecycle order defined above, including the + single main-owned version check before underlay precedence assembly. +3. Add underlay layer/index metadata and version diagnostics. Keep history in + precedence order while source IDs follow acquisition order. Apply composed + main origins over underlays without relabeling main imports as overlays. +4. Reuse Stage 2 assembly and the existing decode/resolution/secret/publication + pipeline. Empty groups, omitted main, defaults-only loads, and underlay-only + loads must follow the chosen semantics. +5. Add engine coverage for two underlay tiers, multiple strict contributions, + main/import conflicts, version checks/absent main, limits across groups, and + one final decode/resolve/validate lifecycle. Test an invalid intermediate typed + value replaced by valid higher input, alongside unmaskable syntax conflicts. + +Acceptance: engine layered requests produce correct values and full provenance; +ordinary-load tests remain green. Run the everyday checker and engine race tests. + +## Stage 7: Expose layered loading and complete inspection integration + +1. Add `Input`, `Underlay`, `LayeredOptions`, and `LoadLayered` root wrappers and + conversions. Copy option slices/context records into private load state while + preserving reusable namespace identities and single-consumption readers. +2. Wire explicit selected-overlay context for layered loads, including host and + native namespaces. The selected file may not duplicate an acquired file in + that namespace, preserving selected-overlay repetition protection. +3. Finish root Layer/SourceInfo conversions, history/explanation rendering, + composite mapping attribution, deleted descendants, default and derived + origins, and secret redaction. New records must remain detached. +4. Add representative public tests for full precedence and same-name recursive + mapping combination; retain zero/false/empty/null distinctions and atomic + sequence replacement. Test ordinary versus zero-underlay layered equivalence + using equivalent explicit overlay context when required. + +Acceptance: callers can load native, host, and mapping tiers in one public call; +configuration, metadata, and safe rendering agree. Run the everyday checker and +`GOPROXY=off go test -race ./...`. + +## Stage 8: Add handle-free source-context queries + +1. Retain context descriptors keyed by acquired source ID, including imported + sources, in private inspection and live resolution state. Keep descriptors + separate from serializable metadata and preserve callback guards. +2. Implement the root query methods and `ResourceContext`. Native contexts expose + rooted borrowed access; host contexts use the passive adapter described above. + Include host main files even when imports are disabled, and explicit contexts + on byte/reader/mapping sources. Sources without contexts return false. +3. Verify each origin's BaseDir points to its actual source. Composite origins + require callers to follow individual locations; derived dependencies retain + their existing meaning. No arbitrary last-source context is selected. +4. Test that queries cause no filesystem operations; loaded values and metadata + survive source deletion; later resource access observes mutation/missing files; + and caller filesystem closure affects only later resource operations. +5. Verify owned files/root handles close on successful loading and representative + acquisition, parse, composition, version, and validation failures. Test resource + Open failure cleanup and returned-file Close ownership separately. Prefer + narrow tracking seams over timing-based goroutine/descriptor-count assertions. + +Acceptance: a retained underlay resource can be opened relative to its true origin +after load, with no library-owned loading handles retained. Run the everyday +checker and full race suite. + +## Stage 9: Validate cross-feature consistency and scaling + +1. Consolidate a small semantic equivalence matrix across host, native, bytes, + readers, and mappings, using equivalent logical inputs and contexts. Cover + strict conflict versus intentional precedence and version-path preservation; + avoid duplicating every lower-level case at every source permutation. +2. Exercise load-wide limits across mixed sources/CLI, repeated explicit tiers, + shared namespace imports, and source IDs/history order. Protect against double + charging and reader re-consumption. Confirm secret canaries stay absent from + diagnostics and historical/effective output after replacements. +3. Run parameterized benchmarks for broad catalogs, many small tiers, subtree + replacement, imported native defaults, and retained history. Compare Stage 1 + baselines with the final path. Address demonstrated avoidable quadratic scans + or accumulated copying within this feature's scope. Account for sorting, + filesystem operations, callbacks, and output/history size explicitly. +4. Review source/context ownership and success/failure cleanup against ADR 0008. + Keep performance evidence in the stage handoff, not permanent timing gates. + +Acceptance: no semantic discrepancy remains among equivalent operations; scaling +results substantiate the engineering target without an unconditional O(n) claim. +Run the everyday checker and full race suite, reusing successful checks for an +unchanged final tree. Benchmarks run with `-benchmem` and parameterized sizes. + +## Stage 10: Complete consumer examples and architecture documentation + +1. Add a maintained simple native-filesystem example and a combined example with + embedded defaults, two underlay tiers, independently located main input, + programmatic contributions, and a relative resource from a retained origin. + Keep fixtures secret-free and offline; include zero/false presence and domain + validation. The consumer should perform no generic post-load merge/defaulting + or provenance reconstruction. Reuse existing example/test conventions. +2. Update `docs/config.md`, Go declarations, `docs/consumers/`, README, + `docs/internal/overview.md`, and applicable architecture/deferred-scope text + to describe implemented support. Preserve strict-within-tier and recursive- + between-tier terminology, distinguish source kinds from precedence roles, + and document context lifetimes and the explicit source-ID query workflow. +3. Add the relevant tests/commands to the development guide's existing task map. + Review ADR links and statuses, preserving draft status pending acceptance and + accepted ADR decision bodies. Keep named inheritance separately deferred. +4. Mark implementation progress accurately in this plan and the roadmap. Do not + retire plans, choose/tag a version, prepare release gates, or publish as part + of implementation; those belong to separate release preparation. + +Acceptance: examples compile and execute, documentation describes only completed +capabilities, and simple `Load` usage remains simple. Run the everyday checker +(which includes examples, links, and boundary checks); run the full race suite +if code changed since its last successful run. Report completed stages, checks, +benchmark conclusions, and any remaining limitations honestly. + +## Validation commands + +Use the development guide's focused commands during each stage. The normal +stage-end check is `./scripts/check.sh`; it already includes tests, vet, build, +formatting, and documentation links. Ownership stages additionally use the race +commands above. Benchmarks may use: + +```sh +GOPROXY=off go test ./internal/engine -run '^$' -bench 'Compose|Overlay|Underlay' -benchmem +``` + +Name the new benchmarks to match that expression. Select representative sizes +under existing limits. No CI wait, human macOS validation, external service, +credentials, or new release gate is part of this plan. + +## Open Questions + +None required to implement this plan. The remaining roadmap recommendations +have been selected above: labeled snapshotted mappings and separate source-context +queries. Private helper names and package-local factoring remain implementation +choices within these contracts. If implementation uncovers a genuine conflict +with existing behavior, report it rather than silently adding a restriction, +weakening compatibility, or broadening a guarantee. diff --git a/docs/roadmap/sources-and-layering.md b/docs/roadmap/sources-and-layering.md index 4865c9d..12c08ce 100644 --- a/docs/roadmap/sources-and-layering.md +++ b/docs/roadmap/sources-and-layering.md @@ -1,8 +1,8 @@ # Filesystem Sources and Source-Backed Underlays -Status: proposed target for design agreement; not implemented. Significant API -and semantic choices are collected under Open Decisions. A later implementation -plan will own staged execution, detailed signatures, and acceptance commands. +Status: agreed target for implementation; not implemented. This document owns +the desired behavior and design rationale. The [implementation plan](implementation.md) +owns API signatures, sequential work, and validation commands. ## Purpose and priorities @@ -74,6 +74,15 @@ Redesigning schema-version policy is outside this roadmap's scope. ## Target source model +The extension preserves the +[synchronous, complete-on-return lifecycle](../config.md#completion-and-resource-lifetime). +Underlay assembly and the final configuration/metadata snapshot complete within +the load call. Resource-context retention is passive; resource access and any +subsequent loading require explicit application calls. The proposed architectural +rationale is recorded in [ADR 0008](../adr/0008-complete-on-return-loading.md). +The handle-ownership requirements below apply to the new source/context APIs as +they are implemented. + ### Native filesystem input A caller can supply a filesystem, an explicit file location within it, and a @@ -122,6 +131,29 @@ and interpreting message text, rendering, and handling resource errors. Context retention must not imply that external resources are snapshotted with the loaded configuration or remain valid after the caller closes their filesystem. +The context contract covers caller-supplied filesystems and host-file sources. +ConfigKit closes every library-owned filesystem handle before loading returns, +on success or failure, including input files and directory/root handles. Returned +configuration, provenance, and resource-context records retain no such handles +and require no filesystem cleanup by the consumer. Configuration and provenance +remain usable independently of subsequent filesystem access. + +Caller-supplied filesystems and readers remain caller-owned. A supplied filesystem +may itself depend on handles whose lifetime the caller manages; retaining its +context does not extend that lifetime or transfer ownership to ConfigKit. + +Host-file resource contexts retain the information needed to open resources later, +preserving the host backend's established containment semantics. Later resource +access acquires its own handles: helpers close handles they use internally, and +callers close any resource handles explicitly returned to them. Host contexts +use a passive path-backed filesystem adapter that opens a fresh contained root +for each resource open and releases it with the returned file. Resource +opens observe the backing filesystem at access time and can fail independently +of an earlier successful configuration load. Sources without filesystem context +report its absence explicitly; composite origins expose their contributing +contexts rather than choosing a directory. These rules keep configuration +snapshot ownership separate from resource-access lifetime. + ## Target layering model ### Explicit operations, not discovery order @@ -145,6 +177,17 @@ Typed defaults remain the existing Go-value baseline. Source underlays add configuration-backed defaults with imports, filesystem context, and source provenance; they are not a replacement for the typed-defaults facility. +An underlay tier is an explicitly supplied group of source contributions. Each +group is strictly composed, including enabled imports, before it participates +in precedence assembly. Consumers supply zero or more tiers in lowest-to-highest +precedence order; later tiers overlay earlier tiers recursively. A source's +position within a group does not authorize replacement. The main input is +optional: an underlay-only load uses the same lifecycle, and a load with neither +main input nor underlays retains defaults-only behavior with supported explicit +overrides. Selecting a named overlay still requires its explicit lookup context. +The public representation exposes grouping and ordering directly. Empty groups +contribute nothing; explicit defaults-only sources also contribute no document. + The selected operation determines how values combine, not their source format or loading entry point. Equivalent file, embedded, byte/reader, and programmatic input uses the same presence and combination rules. Strictly composing independent @@ -152,6 +195,11 @@ contributions and intentionally overlaying precedence tiers express different intentions; merely changing the input medium must not switch between them. This applies to both ordinary `Load` and the new layered-loading interface wherever they offer the same operation. +Consistency applies to combination, presence, and origin meaning. Source-specific +acquisition and lifetime rules remain explicit: a reader is single-consumption, +bytes are snapshotted, and filesystem input can observe backing-store changes. +CLI records retain their existing path restrictions. Source equivalence does not +promise identical reuse, locations, resource access, or diagnostic labels. The existing opt-in version-path ownership rule remains an explicit metadata constraint, independent of whether its owning source is a file or programmatic input; it is not a source-format-dependent merge rule. @@ -232,9 +280,12 @@ An absent declaration remains absent for the checker; an invalid version type fails without invoking it. Later input cannot mask a version-check failure. Filesystem sources and layered loading do not require a different version -policy. The layered API needs a way to identify the version-owning input when a -`VersionRule` is configured; Decision 3 concerns that integration only. Without -version checking, no version owner or synthetic primary document is required. +policy. The designated main input owns the version when a `VersionRule` is +configured, extending the existing selected-source role. It may use any supported +source kind. With no main input, the checker receives an absent declaration, +consistent with existing `NoSource` behavior; underlays cannot supply that +declaration. Without version checking, no version owner or synthetic primary +document is required. Named-overlay lookup context is a separate concern, not a reason to impose version ownership on unversioned loads. @@ -250,7 +301,12 @@ Programmatic input has a non-secret source label and uses explicit structure to represent presence. Omitted mapping keys are absent; present zero, false, empty string, empty collection, and nil/null values remain explicit. Use ConfigKit's existing canonical scalar and collection conventions, not YAML round trips or -Go zero-value inference. Decision 1 selects the initial input representation. +Go zero-value inference. The initial representation is a labeled string-keyed +mapping using the scalar/list/map value set supported by explicit overrides. +This reuses established conventions and keeps presence directly expressible. +The source snapshots the mapping at construction, like byte sources. A nil root +mapping represents null and fails the existing mapping-root requirement; an empty +non-nil mapping is a valid empty document. Do not label generic programmatic sources as CLI input or misuse synthetic flag names to fit the existing override API. Preserve parser-independent conversion, @@ -323,15 +379,16 @@ claim. No timing threshold or additional release gate is requested. ## Work needed from the baseline -These are responsibility areas, not implementation stages: +The target requires changes in the following responsibility areas; the linked +implementation plan owns their execution order and concrete validation: - Extend the root-owned source/context facade and private acquisition boundary for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling without exporting parser or engine types. - Add source-backed underlays and programmatic input, sharing one engine path with ordinary loading rather than building a parallel configuration system. -- Resolve the representation and ownership decisions below, then extend source - records, histories, diagnostics, and rendering consistently across input kinds. +- Extend source records, histories, diagnostics, and rendering consistently + across input kinds. - Integrate version-owner selection into layered loading while preserving the existing version lifecycle and protections. Keep filesystem-context selection independent of version checking. @@ -363,11 +420,8 @@ These are responsibility areas, not implementation stages: as behavior lands. Record significant boundary changes in an ADR without rewriting accepted ADR history. Keep named inheritance explicitly separate. -Plan an additive minor release for the new public capability, preserving current -version semantics. If another design choice requires changing documented behavior, -identify that change expressly and provide migration guidance rather than calling -it an internal optimization. -No release, implementation change, or current-policy revision occurs in this pass. +The capability is additive and intended for a minor release, preserving current +version semantics. Release preparation and publication remain separate tasks. ## Completion criteria @@ -386,50 +440,17 @@ strict within-tier composition, recursive cross-tier overlay, and downstream dom resolution. Equivalent-input tests and the maintained consumer example establish semantic consistency and reduced integration code, not merely API availability. -## Open Decisions +## Selected resource-context interface -### 1. Initial programmatic representation +A separate source-ID-based query exposes standard filesystem access and a +source-relative base location through root-owned API types. Ordinary source +records remain descriptive. The query reports absent context explicitly and +supports imported sources as well as initial inputs. Consumers follow each +contributing origin independently for composite values. This avoids a mandatory +downstream filesystem registry and preserves provenance through recursive +combination. -**Recommendation:** labeled string-keyed mappings with the scalar/list/map value -set already supported by explicit overrides. Missing keys express omission and -nil expresses null. Reuse private conversion machinery, not CLI-specific public -records. This is predictable, directly expressible in Go, and consistent with -current dynamic configuration semantics. - -**Alternative:** typed Go values with a separate explicit-presence description. -This can reduce downstream mapping code, but introduces reconciliation of field -bindings, masks, nulls, and typed zero values and a larger public contract. Do not -infer omission from zero values. Add typed construction only if real examples -show that the simple mapping representation imposes substantial repetitive work. - -### 2. How consumers obtain filesystem resource context - -**Recommendation:** a separate source-context query exposing standard filesystem -access and a source-relative base location through root-owned API types, while -ordinary source records remain descriptive. The returned filesystem is a borrowed -capability, not a cloned or immutable object. This avoids a mandatory downstream -registry and supports origins from imports as well as initial sources. Make -lifetime and nil/no-context behavior explicit before implementation. - -**Alternative:** expose opaque context IDs and relative paths; callers retain a -registry mapping IDs to their filesystems. Metadata stays capability-free and -serializable, but every consumer must maintain and correctly reconcile another -registry. This is viable if limiting retained filesystem capabilities proves -more important than the additional integration machinery. - -### 3. Selecting the version-owning input for layered loads - -**Recommendation:** the designated main source owns the version when configuring -a `VersionRule`, extending the existing selected-source role rather than letting -an underlay win ownership by list position. It need not be a host file or a newly -manufactured root document. Its imports remain fragments and underlays cannot -declare the application's version. No special version-owner designation is -needed when checking is not configured, including loads using only underlays. - -**Alternative:** allow explicit designation of a base input as the version owner -separately from its main/underlay role. This supports bundles carrying the -application version, but adds another ownership choice and complicates the -otherwise clear distinction between defaults and application configuration. -Either choice must preserve the existing version contract; this decision does -not reopen effective-version -or per-document version validation. +Caller-supplied filesystems are borrowed capabilities; host contexts are passive +path-backed adapters. Neither is a cloned or immutable filesystem. Queries perform +no I/O, and later resource opens follow the ownership rules above. Configuration +results retain no library-owned open handles.