eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan filesystem sources and underlays

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-29 16:50:11 +00:00
parent 36172ed20e
commit 27df838513
7 changed files with 805 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files

15
README.md
View File

@@ -41,6 +41,21 @@ load once, inspect where final values came from, and distinguish explicit values
from omissions with minimal configuration bookkeeping. Applications retain their
domain rules and execution responsibilities.
Loading is synchronous and complete on return. Applications normally load once
near startup and then use the final configuration and metadata; ConfigKit stays
inactive until explicitly called again. Applications may initiate additional
loads as needed. See the [loading lifecycle](docs/config.md#completion-and-resource-lifetime)
for completion and ownership details.
Within a configuration tier, independent contributions compose strictly: shared
mappings combine, while overlapping non-mapping definitions are errors. Between
precedence tiers, mappings combine recursively, retaining omitted lower-priority
children; other values replace lower-priority values. Today, the main document
and its imports form the strictly composed base, with a selected overlay and
explicit CLI overrides above it and typed defaults below it. The
[configuration contract](docs/config.md#strict-document-composition) owns the
detailed combination rules and input-specific constraints.
The [architecture goals](docs/policy/architecture.md#success-criteria) explain
these priorities and how genuine new needs can justify extending the shared
model. The [deferred scope](docs/roadmap/deferred.md) records deliberate exclusions.

118
docs/adr/0007-source-backed-underlays.md Normal file
View File

@@ -0,0 +1,118 @@
# ADR 0007: Add Source-Backed Underlays with Consistent Recursive Precedence
## Status
Proposed draft for review. The
[filesystem sources and underlays roadmap](../roadmap/sources-and-layering.md)
owns implementation status, detailed target behavior, and remaining API choices.
This extends [ADR 0001](0001-separate-composition-and-overlay.md) and preserves
its distinction between strict composition and intentional overlay.
## Date
2026-09-29
## Context
Configuration-backed defaults provide lower-priority settings that applications
can override. These sources carry imports, locations, and relative-resource
contexts. Integrating them into ConfigKit's loading lifecycle gives their values
the same presence tracking and inspectable provenance as other configuration
inputs.
The [architecture goals](../policy/architecture.md#success-criteria) prioritize
one coherent, predictable configuration model with consistent rules across
source representations and loading interfaces. Consumers should load once,
receive a complete validated configuration, and inspect its final values' origins.
Reducing integration code is important subject to those principles.
## Decision
### Add underlays to the existing precedence model
Introduce explicitly supplied source-backed underlays between typed defaults
and main configuration. The proposed precedence, from lowest to highest, is:
**Typed application defaults < source underlays < strictly composed main
configuration < selected overlay < explicit CLI overrides.**
Typed defaults remain the Go-value baseline. Underlays supply lower-priority
configuration with source provenance and, where applicable, imports and resource
context. This ordering describes value precedence; typed defaults fill fields
absent from the assembled input under the existing defaulting rules.
Preserve strict composition of the main document and its imports before applying
the resulting configuration over underlays. Independent contributions within an
underlay tier also compose strictly. Explicitly ordered tiers express intentional
precedence. Strict composition reports conflicting contributions within each tier.
Across precedence tiers, reuse the existing recursive mapping-overlay semantics.
Omitted children retain lower-priority input; existing scalar, sequence, null,
and cross-kind replacement semantics remain unchanged. Reusing these operations
keeps the combination rules consistent across the loading model.
### Keep source representation separate from composition semantics
Native `fs.FS` and programmatic inputs join the shared loading model.
Equivalent inputs using the same
operation have the same presence and precedence semantics regardless of medium.
Programmatic values enter directly as labeled configuration sources.
Retain the contributing source's identity and resource context through layering.
A retained underlay value keeps its originating source's resource context.
Consumers remain responsible for opening and interpreting their domain resources
under the supplied filesystem's lifetime and access semantics. Exact
context-access and input APIs remain roadmap decisions.
### Deliver one final configuration and its inspection metadata
Assemble effective input, then run the typed loading lifecycle once for the load.
Preserve explicit zero,
false, empty, and null input versus omission through configuration and metadata
together.
Extend existing provenance and history through underlays, defaults, and recorded
application resolution. The returned result includes the completed combination,
defaulting, and provenance work. Domain resolution and validation
remain application-owned within the recorded lifecycle, consistent with
[ADR 0003](0003-preserve-presence-and-resolution-history.md).
Preserve the existing opt-in schema-version ownership, path protections, and
check timing before precedence
assembly and typed decoding. The roadmap owns how layered input identifies that
owner when version checking is configured.
Favor near-linear processing in input, retained history, and output size through
incremental assembly of load-owned state. This is an engineering objective
subordinate to clarity and semantic consistency. Preserve the root-owned facade and
private implementation boundary established by [ADR 0006](0006-public-facade.md).
## Alternatives considered
- **Leave bundled defaults to consumers.** Avoids extending loading, but makes
consumers reproduce merging, presence tracking, and provenance reconciliation.
It does not meet the goal of obtaining a complete configuration in one load.
- **Offer recursive or atomic combination per layer.** Makes both behaviors
explicit and could serve more consumers, but expands the semantic surface and
its interactions with presence and provenance without a demonstrated need that
outweighs a single consistent rule. A future concrete requirement may justify
reconsideration; this feature does not add the option.
- **Flatten all documents into last-wins layers.** Simplifies ordering but turns
accidental main/import conflicts into overrides, weakening the intentional
distinction established by ADR 0001.
## Consequences
Consumers gain source-backed defaults within the shared loading lifecycle.
Strict contributions and intentional precedence remain distinct, with consistent
combination rules across source media. Presence and provenance remain
part of the same loading lifecycle as the final values.
Higher-priority mappings retain omitted lower-priority children, so application
validation assesses the final combined configuration.
Implementation must integrate source context and underlay history into the shared
engine with attention to assembly and history-retention costs. Current behavior
documentation will change when the feature lands. The roadmap retains ownership
of the remaining API decisions.

83
docs/adr/0008-complete-on-return-loading.md Normal file
View File

@@ -0,0 +1,83 @@
# ADR 0008: Synchronous, Complete-on-Return Configuration Loading
## Status
Proposed draft for review. The
[configuration contract](../config.md#completion-and-resource-lifetime) describes
the implemented loading lifecycle. The
[source-backed underlays roadmap](../roadmap/sources-and-layering.md) owns its
application to the planned source and resource-context APIs.
## Date
2026-09-29
## Context
Applications need a clear point at which configuration is ready for use.
Completing configuration processing and inspection metadata together gives
applications a coherent result whose values and origins agree. A bounded loading
lifecycle also makes resource ownership and startup failures straightforward to
reason about.
ConfigKit's [architecture goals](../policy/architecture.md#success-criteria)
emphasize complete configuration, inspectable provenance, predictable behavior,
and minimal consumer bookkeeping. Applications own their execution lifecycle and
the timing of configuration acquisition.
## Decision
Make loading a synchronous, complete-on-return operation. Within each load,
ConfigKit completes input assembly, defaulting, typed decoding, configured
application resolution, secret lookup, validation, and snapshotting before
returning the final configuration and metadata. Reported failures return
diagnostics and a nil result.
Release all library-owned loading handles before returning on success or failure.
The resulting configuration and metadata are independently usable, caller-owned
data with a retained inspection snapshot. Caller-supplied readers and filesystems
remain under caller ownership and lifetime control.
After the call completes, ConfigKit remains inactive until explicitly invoked
again. Applications normally load once at or near startup and may initiate
additional loads when needed. Each load follows the same lifecycle and source
reuse contracts. Inspection, rendering, and separately requested operations run
in response to explicit application calls.
For the planned resource-context capability, retain source context as passive
information or a borrowed caller-supplied filesystem. Subsequent resource access
acquires its own handles with explicit ownership. This preserves a clear boundary
between configuration completion and application resource use.
These choices give consumers one completion boundary for configuration and
provenance, predictable resource lifetimes, and control over when ConfigKit runs.
Application callbacks remain responsible for their own effects and resources.
## Alternatives considered
- **Background watching or refresh.** Can automate responses to source changes,
but adds scheduling, shutdown, publication, and resource-lifetime concerns to
the library. Explicit calls keep those responsibilities with the application.
- **Lazy completion after return.** Can defer work until particular values are
used, but moves configuration failures into application execution and makes
the readiness of values and metadata harder to establish.
- **Transactional atomic loading.** Could imply simultaneous observation of all
external inputs and rollback of callback effects. Filesystems, environment
variables, and application callbacks do not share a transaction mechanism.
Synchronous completion provides the required usable-result boundary without
promising those external guarantees.
- **Retain loading handles in results.** Can preserve open filesystem access,
but gives configuration results an ongoing resource lifecycle and cleanup
obligation. Passive source context supports later explicit resource access
while keeping loading resources scoped to the load call.
## Consequences
Applications can treat successful return as the point at which the final
configuration and its metadata are ready for use. Startup loading remains simple,
and additional loads have an explicit application-owned trigger.
Configuration inspection operates on retained data. Resource access observes the
backing resource when explicitly requested and has its own failure and ownership
boundary. Implementations of new source kinds and layering features preserve the
same completion boundary, including cleanup on reported load failures.

25
docs/config.md
View File

@@ -71,6 +71,31 @@ the returned configuration or inspection records cannot alter ConfigKit's privat
snapshot. ConfigKit performs no logging. `Result.WriteEffective` exposes only the
frozen redacted projection.
### Completion and resource lifetime
`Load` is synchronous and complete on return. Success means input assembly,
defaulting, typed decoding, configured application resolution, secret lookup,
validation, and configuration/metadata snapshotting have completed. A reported
load failure returns diagnostics and a nil result, as described above.
Before returning on success or failure, ConfigKit closes the filesystem handles
it opened for loading, including files and directory/root handles. The returned
configuration and inspection metadata require no close operation and remain
usable independently of the source files. A supplied `Reader` remains
caller-owned and is never closed by ConfigKit.
ConfigKit performs no background work after returning: it does not watch inputs,
refresh secrets, or mutate published results. Inspection and rendering run only
when explicitly called and use the retained snapshot. Applications normally load
once near startup, but may explicitly load again using sources that satisfy their
documented reuse rules. Additional loads produce independently owned results.
Completion is not a transaction over external state. Files and environment
variables are observed during processing rather than as one simultaneous external
snapshot. ConfigKit does not roll back application callback effects or manage work
or resources created by application callbacks. The explicit secrets-directory
bootstrap remains a separate operation invoked by the application.
### Application schema versions
`VersionRule` declares a semantic path and non-nil checker. Before defaults or typed

33
docs/policy/architecture.md
View File

@@ -108,6 +108,22 @@ must never run implicitly as part of configuration loading.
Remote data can enter through application-supplied input without introducing a
network provider into the library.
## Loading Lifecycle
Configuration loading is a synchronous, complete-on-return operation. A successful
load provides the final configuration and inspection metadata after the configured
processing and validation phases complete. A failed load reports diagnostics and
returns no usable partial result. The normal application pattern is one load near
startup; applications may explicitly initiate additional loads.
ConfigKit releases its library-owned loading handles before returning and remains
inactive until the application explicitly invokes it again. It performs no
background watching, refresh, or mutation of published results. Caller-owned
resources and application callback effects remain the caller's responsibility.
The [configuration contract](../config.md#completion-and-resource-lifetime) owns
the precise completion and ownership behavior. The proposed decision record is
[ADR 0008](../adr/0008-complete-on-return-loading.md).
## Public API And Dependency Direction
Keep one small, coherent public surface for the supported lifecycle. Simple
@@ -136,9 +152,20 @@ framework on this library.
## Configuration Integrity
Composition and intentional overlay are distinct operations. Composition must
not silently become last-wins override through file ordering. The decision and
alternatives are recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md).
Within a configuration tier, independent contributions use strict composition:
shared mappings combine recursively, while overlapping non-mapping definitions
are conflicts, including equal values. Between precedence tiers, intentional
overlay combines mappings recursively, retains omitted lower-priority children,
and replaces other present values according to the existing overlay rules.
Tier precedence is explicit; traversal order within a tier establishes stable
reporting, not permission to override. The distinction and its rationale are
recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md).
The main document and its imports currently form the strictly composed base
tier. Input-specific constraints, including CLI path rules and opt-in version
ownership, remain part of the configuration contract. Consistent combination
semantics do not require identical acquisition or lifetime behavior from every
source kind.
ConfigKit must define supported YAML semantics explicitly rather than inherit
parser defaults. Processing order and diagnostics must be deterministic for the

453
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,453 @@
# Filesystem Sources and Underlays Implementation Plan
## Purpose and baseline
Implement the [feature roadmap](sources-and-layering.md) in the numbered order
below. Each stage is sized as one focused implementation assignment for a
gpt-5.6-terra coding agent. The baseline is
`36172ed20e832b371506d6aceb02dc6c8f50f71b`, plus the documentation changes committed
with this plan. No feature implementation is part of this planning pass.
Read `docs/development.md` and the policies it selects for the stage. The roadmap
owns target semantics; [ADR 0007](../adr/0007-source-backed-underlays.md) and
[ADR 0008](../adr/0008-complete-on-return-loading.md) record the proposed rationale.
Their draft status is independent of implementation progress: do not mark them
accepted automatically. The [configuration contract](../config.md) is the
compatibility baseline. Preserve the root-owned facade and private engine/tree
boundary.
At each stage, preserve unrelated work, add focused behavioral coverage with the
change, update declaration documentation for public additions, and run the checks
specified below. Keep intermediate commits building and tested. Do not expose
placeholder public APIs that silently ignore options. Internal scaffolding may
precede the public feature. Use plain-English commits under the agent policy.
## Settled implementation decisions
### Public input surface
Use these names and shapes; bodies and private fields belong to implementation:
```go
// Root-owned opaque filesystem namespace, copying preserves namespace identity.
type Filesystem struct { /* private engine state */ }
func NewFilesystem(label string, filesystem fs.FS) Filesystem
func (f Filesystem) File(name string) Source
func Mapping(label string, values map[string]any) Source
// Extend the existing record; nil FS selects host-filesystem context.
type FilesystemContext struct {
BaseDir string
Root string
FS *Filesystem
}
type Input struct {
Source Source
Imports bool
Filesystem *FilesystemContext
}
type Underlay struct { Inputs []Input }
type LayeredOptions[T any] struct {
Definition Definition[T]
Underlays []Underlay
Overlay string
OverlayFilesystem *FilesystemContext
Overrides []Override
}
func LoadLayered[T any](main Input, supplied ...LayeredOptions[T]) (*Result[T], error)
type ResourceContext struct {
FS fs.FS
BaseDir string
}
func (r *Result[T]) SourceContext(id string) (ResourceContext, bool)
func (r *Resolution) SourceContext(id string) (ResourceContext, bool)
```
`fs` above means the standard `io/fs` package. All named public types are defined
at the root, not aliases of engine types. Existing `Load` and `Options` stay
source-compatible apart from the additive field in `FilesystemContext`. Avoid a
parallel lifecycle implementation: adapt both entry points to one private load
request. No new dependency or minimum-Go-version change is needed.
- Like `Load`, `LoadLayered` accepts zero or one options value. A zero `Source`
stays invalid. Represent omitted main input as `Input{Source: NoSource()}`.
`NoSource` contributes no document, source ID, or context; preserve its existing
invalid import/context combinations. An empty underlay group is a no-op.
- Each underlay contains strictly composed inputs, including enabled imports.
Groups are ordered lowest to highest. Repeated explicit inputs in different
tiers are allowed subject to each source's reuse contract. Within a tier,
repeated file identities remain import/repetition errors. Equal scalar claims
still conflict. Independent synthetic sources are distinguished by occurrence.
- `Input.Imports` opts in per input and its import graph. The main source and its
imports form the main tier. There is no programmatic high-priority escape hatch:
a mapping source can be main input or any underlay contribution; CLI overrides
retain their explicit-flag contract.
- Active named overlays in `LoadLayered` require `OverlayFilesystem` explicitly.
Their names, root-relative lookup, directive prohibition, and version protection
follow existing rules. An inactive overlay needs no context; an unused explicit
context has no effect. Ordinary `Load` retains implicit selection of its source's
filesystem context and existing `NoSource` restrictions.
### Filesystem namespaces, paths, and acquisition
`NewFilesystem` allocates a reusable private namespace token and retains the
caller-owned filesystem without opening it. Copies share the token; separate
constructors are separate namespaces even for the same underlying filesystem.
Never compare or hash arbitrary `fs.FS` values. Validate nonempty safe labels and
nil filesystem arguments through deferred source diagnostics, following existing
source-constructor conventions. The zero `Filesystem` yields an invalid source.
`Filesystem.File(name)` uses a non-root `fs.ValidPath` file name relative to that
namespace. The default root is `.` and the base is the file's containing directory.
An optional filesystem context can narrow the root; file sources continue to
derive their own base directory. An explicit FS must identify the same namespace
as the file source; context overrides for native file sources supply that FS.
A host `File` cannot be rebound to a virtual namespace.
For byte, reader, and mapping inputs, an explicit context supplies their namespace
and base: with FS set, require valid slash-relative Root and BaseDir (use `.` for
the namespace root), with BaseDir inside Root. With FS nil, preserve the existing
absolute host-directory requirements. Labels never imply a context.
Resolve import paths relative to the declaring document, normalizing joined
paths and rejecting escape from the configured root. Preserve the existing
relative-path/glob language, including contained parent references, optional
missing behavior, sorted matches, bounded traversal, and cycle diagnostics.
Virtual paths use `path`, not host `filepath` rules. Imports stay in their namespace;
generic filesystems provide their own symlink behavior. Preserve canonicalization,
regular-file checks, and `os.Root` containment for host acquisition. Use ordinary
`fs.FS` operations and `fs.ReadDirFile` where available; support the standard
`fs.ReadDir` fallback without claiming control over its internal allocation.
Configuration reads close each file ConfigKit opens, including read/stat/parse
failures. Caller readers and caller filesystems are never closed by ConfigKit.
Namespace tokens live with source objects; deterministic load-local IDs are
assigned by acquisition, not addresses or global counters.
### Programmatic representation and diagnostics
`Mapping` snapshots its input at construction, matching `Bytes`. Convert directly
to the private document tree using the CLI converter's existing built-in scalar,
`[]any`, and `map[string]any` repertoire. Preserve integer precision, finite-float
validation, sorted keys, sequence order, explicit null, and cycle rejection.
Typed nil maps/slices mean null, as in existing overrides. A nil root mapping is
invalid because document roots are mappings; `{}` is a valid empty mapping.
Constructor failures are stored and surfaced on load. Detach once at construction;
do not retain caller maps or serialize them. Concurrent caller mutation during
construction is outside the supported ownership contract.
Generic conversion returns private issues; adapt them to existing `override.*`
codes for CLI input, and `source.invalid` or `source.limit_exceeded` for mapping
input. Do not leak raw values or conversion errors. Source/path locations use
the label and semantic path, with zero line/column for synthetic input. Apply
reserved `configkit` directive rules consistently: with imports enabled, a
mapping can contain the same directive structure as YAML. Without context,
relative imports fail under existing context rules.
### Ordering, lifecycle, budgets, and metadata
Acquire and strictly compose main input first, construct the schema, and check
the optional version. Only the main source owns the version; main imports and
all underlay documents must satisfy fragment-style version-path protections.
With `NoSource` main input, pass an absent version to the checker. Do not run the
checker once per tier or validate the final overlaid version instead.
Then acquire/compose underlays in listed order and combine them recursively,
apply the composed main tier, selected overlay, and CLI overrides. Execute typed
defaulting/decoding, recorded resolution, secret lookup, validation, and final
publication once. Syntax/composition/version failures cannot be masked by higher
precedence; effective typed validation sees the final assembled input only.
Keep source-record order separate from precedence: main depth-first sources,
underlay tiers in order with each input traversed depth-first, selected overlay,
then CLI. IDs remain `source-N`. Existing ordinary loads retain their source order.
The history application order follows precedence, not acquisition order.
Extend `Layer` with `LayerUnderlay = "underlay"`. Add to `SourceInfo` a `Kind
SourceKind` and `UnderlayIndex int` (one-based original tier position for underlay
sources, zero otherwise). Define `SourceKindFile`, `SourceKindFilesystem`,
`SourceKindBytes`, `SourceKindReader`, `SourceKindMapping`, and `SourceKindCLI`
with string values `file`, `filesystem`, `bytes`, `reader`, `mapping`, and `cli`.
Imported documents inherit tier role/index and have their actual acquisition kind.
Preserve existing host Path meaning; filesystem-source Path is namespace-relative;
synthetic Path stays empty. Labels for native files combine namespace label and
relative path for display, while identity uses the token and path.
Origins continue to refer to source IDs through locations. Composite origins
preserve all contributors; equal-value overrides and displaced descendants stay
inspectable. Carry source context separately from serializable source records.
Keep existing inspection schema and renderer structure, adding the new layer/kind
where appropriate rather than changing existing output gratuitously.
Use one load-wide accounting object for layered loads: existing 64 MiB aggregate
input bytes, 1,024 acquired sources, 1,000,000 input nodes, and glob-entry budget;
existing per-source 8 MiB, depth 256, active import depth 64, and retained-history
limit remain. Mapping key/string bytes use the existing CLI converter's byte
accounting. Charge each accepted occurrence once, including CLI contributions;
do not charge intermediate copies as new input. Explicit empty groups and
`NoSource` consume no source allowance. Ordinary loads retain their established
limit behavior. Use existing source/import/overlay/override limit codes according
to the acquisition that exceeds a budget. Add `CodeUnderlayVersion =
"underlay.version"` for version protection in underlay roots; imported fragments
retain `import.version`. Avoid a separate arbitrary tier-count limit.
### Passive resource contexts
`SourceContext` returns false for unknown IDs or sources without a context, and
performs no I/O. The result method is nil-safe; the resolution method follows
existing callback lifetime guards. Return a record copy containing a borrowed
filesystem and slash-relative source BaseDir within its configured root.
Consumers obtain source IDs from ordinary origins; there is no single-context
query that guesses among composite or derived origins. Dependencies continue to
identify the sources of application-derived values through existing metadata.
For native inputs, use a namespace-rooted view over the caller's filesystem and
retain its lifetime semantics. For host input, retain the canonical root path,
not an open `os.Root`. A private `fs.FS` adapter validates a relative resource
name, resolves it using existing host containment semantics, opens a fresh root,
and opens the resource. Return a private `fs.File` wrapper whose Close releases
both file and root, including failed-open cleanup. Expose `fs.ReadDirFile` when
the underlying directory supports it. Resources themselves are application data:
do not apply configuration YAML parsing or configuration-byte budgets to them.
Helpers such as `fs.ReadFile` then close the returned handle normally. Errors
follow filesystem access conventions, separately from load diagnostics.
Neither queries nor retained results keep library-owned filesystem handles open.
Files/resources can change or disappear after loading; configuration/metadata
remain usable. No background work, hidden reopening on metadata queries,
filesystem snapshot, universal sandbox, or transaction guarantee is added.
## Stage 1: Index mapping assembly and establish scaling baselines
Scope: `internal/document` and engine composition/overlay lookup, with existing
behavior only.
1. Add parameterized benchmarks for wide disjoint composition and recursive
overlay, recording runtime and allocations at increasing sizes. Keep fixtures
within existing budgets and record the baseline in the stage handoff.
2. Replace repeated linear mapping-key lookup during bulk assembly with a private
indexed builder alongside ordered entries. Build each target index once per
assembly session and update it for appended keys. Keep tree traversal and
diagnostic order stable; avoid mutable indexes exposed through public APIs.
3. Check each strict-composition claim group once while preserving first-conflict
ordering and all relevant locations. Preserve original input ownership.
4. Extend existing composition tests only for uncovered ordering/conflict risks.
Acceptance: existing Compose/Overlay tests pass unchanged; broad keyed input
no longer incurs a full key scan for each insertion. Run the focused composition
tests and the everyday checker. Benchmarks are evidence, not timing gates.
## Stage 2: Prepare load-owned assembly and incremental provenance
Scope: `compose.go`, `overlay.go`, origin indexes, override application, and
history replay in `internal/engine`.
1. Introduce a private load-owned accumulator with explicit ownership. Existing
copy-preserving helpers may wrap it, but sequential assembly must mutate one
owned working tree rather than clone all accumulated state per tier.
2. Retain immutable acquired inputs for history, detached from mutable working
nodes. Parameterize internal overlay application by incoming layer and origins
so a composed tier's imported contributors are preserved.
3. Remove displaced origins by visiting the affected subtree. Accumulate mapping
contributors without copying an ever-growing ancestor list on each addition;
materialize defensive public records at the existing publication boundary.
4. Preserve equal-value replacement events, descendant displacement, redaction,
and typed-decoding reconciliation. Extend benchmarks for many small applications
and repeated subtree/scalar replacement. Do not add a persistent-tree framework.
Acceptance: existing override/history/rendering/ownership tests pass; benchmark
growth avoids accumulated full-tree copying. Run the everyday checker and
`GOPROXY=off go test -race ./internal/engine` for ownership changes.
## Stage 3: Unify private filesystem acquisition without public changes
Scope: engine source/file, filesystem, import, glob, and selected-overlay code.
1. Extract a small private namespace/access boundary used by imports and overlays.
Keep the host implementation's canonicalization, containment, regular-file
checks, safe diagnostics, deterministic traversal, and close behavior.
2. Separate namespace identity from source labels and resource paths. Make
repeated/cycle tracking belong to an input composition group, and accounting
capable of being shared across groups.
3. Add a native `fs.FS` backend internally. Resolve paths using slash semantics,
preserve glob grammar and optional-missing behavior, and avoid host fallback.
4. Use real host tests plus `fstest.MapFS` and a small tracking/failing filesystem
to cover imports, missing paths, directory/read failures, and handle closure.
Include a filesystem with a non-comparable dynamic value.
Acceptance: all existing host source/import/glob/overlay tests remain green;
native backend tests establish namespace isolation and supported paths. Run the
everyday checker. No public constructors land until the next stage.
## Stage 4: Expose native filesystem sources through the facade
1. Add `Filesystem`, `NewFilesystem`, `Filesystem.File`, and the additive FS
context field at root and engine boundaries. Constructors follow deferred
diagnostics; copying the wrapper preserves namespace identity.
2. Integrate native sources with ordinary `Load`, including explicitly enabled
imports and named overlays. Enforce the path/context rules settled above.
3. Add `SourceKind` and populate it for every current/native acquisition. Preserve
host paths and use namespace-relative virtual paths with accurate locations.
4. Add facade-level tests using embedded fixtures, `fstest.MapFS`, byte/reader
context, namespace-label collisions, and source reuse. Keep backend error cases
at their Stage 3 owner. Update current source/API docs to this implemented scope.
Acceptance: native `Load` works through the public API and existing sources retain
their contracts. Run the everyday checker, including public-boundary tests.
## Stage 5: Add snapshotted mapping sources
1. Extract reusable value conversion from `override_value.go`; keep CLI-specific
path, label, and diagnostic handling in its adapter.
2. Implement `Mapping` at root and engine boundaries with constructor-time
snapshotting and deferred safe failures. Source-ID assignment at acquisition
must not mutate a reusable stored tree shared by another load.
3. Route converted documents through the same directive processing, strict
composition, version checking, decode, and inspection path as YAML sources.
4. Test omission/zero/false/null, nil versus empty collections/root, exact integers,
invalid/cyclic values, imports with explicit context, and caller mutation after
construction. Verify mapping input does not gain CLI origins or restrictions.
Acceptance: equivalent byte/mapping inputs have equivalent semantic results and
origin meaning, with genuine location differences allowed. Existing override
diagnostics remain stable. Run the everyday checker and engine race tests.
## Stage 6: Implement private grouped-underlay orchestration
1. Add private request/input/group structures matching the settled facade design.
Adapt ordinary `Load` to the shared core while preserving its options and
source-order behavior. Keep its existing validation order where observable.
2. Strictly compose each group with group-scoped repetition tracking and shared
aggregate accounting. Apply the lifecycle order defined above, including the
single main-owned version check before underlay precedence assembly.
3. Add underlay layer/index metadata and version diagnostics. Keep history in
precedence order while source IDs follow acquisition order. Apply composed
main origins over underlays without relabeling main imports as overlays.
4. Reuse Stage 2 assembly and the existing decode/resolution/secret/publication
pipeline. Empty groups, omitted main, defaults-only loads, and underlay-only
loads must follow the chosen semantics.
5. Add engine coverage for two underlay tiers, multiple strict contributions,
main/import conflicts, version checks/absent main, limits across groups, and
one final decode/resolve/validate lifecycle. Test an invalid intermediate typed
value replaced by valid higher input, alongside unmaskable syntax conflicts.
Acceptance: engine layered requests produce correct values and full provenance;
ordinary-load tests remain green. Run the everyday checker and engine race tests.
## Stage 7: Expose layered loading and complete inspection integration
1. Add `Input`, `Underlay`, `LayeredOptions`, and `LoadLayered` root wrappers and
conversions. Copy option slices/context records into private load state while
preserving reusable namespace identities and single-consumption readers.
2. Wire explicit selected-overlay context for layered loads, including host and
native namespaces. The selected file may not duplicate an acquired file in
that namespace, preserving selected-overlay repetition protection.
3. Finish root Layer/SourceInfo conversions, history/explanation rendering,
composite mapping attribution, deleted descendants, default and derived
origins, and secret redaction. New records must remain detached.
4. Add representative public tests for full precedence and same-name recursive
mapping combination; retain zero/false/empty/null distinctions and atomic
sequence replacement. Test ordinary versus zero-underlay layered equivalence
using equivalent explicit overlay context when required.
Acceptance: callers can load native, host, and mapping tiers in one public call;
configuration, metadata, and safe rendering agree. Run the everyday checker and
`GOPROXY=off go test -race ./...`.
## Stage 8: Add handle-free source-context queries
1. Retain context descriptors keyed by acquired source ID, including imported
sources, in private inspection and live resolution state. Keep descriptors
separate from serializable metadata and preserve callback guards.
2. Implement the root query methods and `ResourceContext`. Native contexts expose
rooted borrowed access; host contexts use the passive adapter described above.
Include host main files even when imports are disabled, and explicit contexts
on byte/reader/mapping sources. Sources without contexts return false.
3. Verify each origin's BaseDir points to its actual source. Composite origins
require callers to follow individual locations; derived dependencies retain
their existing meaning. No arbitrary last-source context is selected.
4. Test that queries cause no filesystem operations; loaded values and metadata
survive source deletion; later resource access observes mutation/missing files;
and caller filesystem closure affects only later resource operations.
5. Verify owned files/root handles close on successful loading and representative
acquisition, parse, composition, version, and validation failures. Test resource
Open failure cleanup and returned-file Close ownership separately. Prefer
narrow tracking seams over timing-based goroutine/descriptor-count assertions.
Acceptance: a retained underlay resource can be opened relative to its true origin
after load, with no library-owned loading handles retained. Run the everyday
checker and full race suite.
## Stage 9: Validate cross-feature consistency and scaling
1. Consolidate a small semantic equivalence matrix across host, native, bytes,
readers, and mappings, using equivalent logical inputs and contexts. Cover
strict conflict versus intentional precedence and version-path preservation;
avoid duplicating every lower-level case at every source permutation.
2. Exercise load-wide limits across mixed sources/CLI, repeated explicit tiers,
shared namespace imports, and source IDs/history order. Protect against double
charging and reader re-consumption. Confirm secret canaries stay absent from
diagnostics and historical/effective output after replacements.
3. Run parameterized benchmarks for broad catalogs, many small tiers, subtree
replacement, imported native defaults, and retained history. Compare Stage 1
baselines with the final path. Address demonstrated avoidable quadratic scans
or accumulated copying within this feature's scope. Account for sorting,
filesystem operations, callbacks, and output/history size explicitly.
4. Review source/context ownership and success/failure cleanup against ADR 0008.
Keep performance evidence in the stage handoff, not permanent timing gates.
Acceptance: no semantic discrepancy remains among equivalent operations; scaling
results substantiate the engineering target without an unconditional O(n) claim.
Run the everyday checker and full race suite, reusing successful checks for an
unchanged final tree. Benchmarks run with `-benchmem` and parameterized sizes.
## Stage 10: Complete consumer examples and architecture documentation
1. Add a maintained simple native-filesystem example and a combined example with
embedded defaults, two underlay tiers, independently located main input,
programmatic contributions, and a relative resource from a retained origin.
Keep fixtures secret-free and offline; include zero/false presence and domain
validation. The consumer should perform no generic post-load merge/defaulting
or provenance reconstruction. Reuse existing example/test conventions.
2. Update `docs/config.md`, Go declarations, `docs/consumers/`, README,
`docs/internal/overview.md`, and applicable architecture/deferred-scope text
to describe implemented support. Preserve strict-within-tier and recursive-
between-tier terminology, distinguish source kinds from precedence roles,
and document context lifetimes and the explicit source-ID query workflow.
3. Add the relevant tests/commands to the development guide's existing task map.
Review ADR links and statuses, preserving draft status pending acceptance and
accepted ADR decision bodies. Keep named inheritance separately deferred.
4. Mark implementation progress accurately in this plan and the roadmap. Do not
retire plans, choose/tag a version, prepare release gates, or publish as part
of implementation; those belong to separate release preparation.
Acceptance: examples compile and execute, documentation describes only completed
capabilities, and simple `Load` usage remains simple. Run the everyday checker
(which includes examples, links, and boundary checks); run the full race suite
if code changed since its last successful run. Report completed stages, checks,
benchmark conclusions, and any remaining limitations honestly.
## Validation commands
Use the development guide's focused commands during each stage. The normal
stage-end check is `./scripts/check.sh`; it already includes tests, vet, build,
formatting, and documentation links. Ownership stages additionally use the race
commands above. Benchmarks may use:
```sh
GOPROXY=off go test ./internal/engine -run '^$' -bench 'Compose|Overlay|Underlay' -benchmem
```
Name the new benchmarks to match that expression. Select representative sizes
under existing limits. No CI wait, human macOS validation, external service,
credentials, or new release gate is part of this plan.
## Open Questions
None required to implement this plan. The remaining roadmap recommendations
have been selected above: labeled snapshotted mappings and separate source-context
queries. Private helper names and package-local factoring remain implementation
choices within these contracts. If implementation uncovers a genuine conflict
with existing behavior, report it rather than silently adding a restriction,
weakening compatibility, or broadening a guarantee.

141
docs/roadmap/sources-and-layering.md
View File

@@ -1,8 +1,8 @@
# Filesystem Sources and Source-Backed Underlays
Status: proposed target for design agreement; not implemented. Significant API
and semantic choices are collected under Open Decisions. A later implementation
plan will own staged execution, detailed signatures, and acceptance commands.
Status: agreed target for implementation; not implemented. This document owns
the desired behavior and design rationale. The [implementation plan](implementation.md)
owns API signatures, sequential work, and validation commands.
## Purpose and priorities
@@ -74,6 +74,15 @@ Redesigning schema-version policy is outside this roadmap's scope.
## Target source model
The extension preserves the
[synchronous, complete-on-return lifecycle](../config.md#completion-and-resource-lifetime).
Underlay assembly and the final configuration/metadata snapshot complete within
the load call. Resource-context retention is passive; resource access and any
subsequent loading require explicit application calls. The proposed architectural
rationale is recorded in [ADR 0008](../adr/0008-complete-on-return-loading.md).
The handle-ownership requirements below apply to the new source/context APIs as
they are implemented.
### Native filesystem input
A caller can supply a filesystem, an explicit file location within it, and a
@@ -122,6 +131,29 @@ and interpreting message text, rendering, and handling resource errors. Context
retention must not imply that external resources are snapshotted with the loaded
configuration or remain valid after the caller closes their filesystem.
The context contract covers caller-supplied filesystems and host-file sources.
ConfigKit closes every library-owned filesystem handle before loading returns,
on success or failure, including input files and directory/root handles. Returned
configuration, provenance, and resource-context records retain no such handles
and require no filesystem cleanup by the consumer. Configuration and provenance
remain usable independently of subsequent filesystem access.
Caller-supplied filesystems and readers remain caller-owned. A supplied filesystem
may itself depend on handles whose lifetime the caller manages; retaining its
context does not extend that lifetime or transfer ownership to ConfigKit.
Host-file resource contexts retain the information needed to open resources later,
preserving the host backend's established containment semantics. Later resource
access acquires its own handles: helpers close handles they use internally, and
callers close any resource handles explicitly returned to them. Host contexts
use a passive path-backed filesystem adapter that opens a fresh contained root
for each resource open and releases it with the returned file. Resource
opens observe the backing filesystem at access time and can fail independently
of an earlier successful configuration load. Sources without filesystem context
report its absence explicitly; composite origins expose their contributing
contexts rather than choosing a directory. These rules keep configuration
snapshot ownership separate from resource-access lifetime.
## Target layering model
### Explicit operations, not discovery order
@@ -145,6 +177,17 @@ Typed defaults remain the existing Go-value baseline. Source underlays add
configuration-backed defaults with imports, filesystem context, and source
provenance; they are not a replacement for the typed-defaults facility.
An underlay tier is an explicitly supplied group of source contributions. Each
group is strictly composed, including enabled imports, before it participates
in precedence assembly. Consumers supply zero or more tiers in lowest-to-highest
precedence order; later tiers overlay earlier tiers recursively. A source's
position within a group does not authorize replacement. The main input is
optional: an underlay-only load uses the same lifecycle, and a load with neither
main input nor underlays retains defaults-only behavior with supported explicit
overrides. Selecting a named overlay still requires its explicit lookup context.
The public representation exposes grouping and ordering directly. Empty groups
contribute nothing; explicit defaults-only sources also contribute no document.
The selected operation determines how values combine, not their source format
or loading entry point. Equivalent file, embedded, byte/reader, and programmatic
input uses the same presence and combination rules. Strictly composing independent
@@ -152,6 +195,11 @@ contributions and intentionally overlaying precedence tiers express different
intentions; merely changing the input medium must not switch between them.
This applies to both ordinary `Load` and
the new layered-loading interface wherever they offer the same operation.
Consistency applies to combination, presence, and origin meaning. Source-specific
acquisition and lifetime rules remain explicit: a reader is single-consumption,
bytes are snapshotted, and filesystem input can observe backing-store changes.
CLI records retain their existing path restrictions. Source equivalence does not
promise identical reuse, locations, resource access, or diagnostic labels.
The existing opt-in version-path ownership rule remains an explicit metadata
constraint, independent of whether its owning source is a file or programmatic
input; it is not a source-format-dependent merge rule.
@@ -232,9 +280,12 @@ An absent declaration remains absent for the checker; an invalid version type
fails without invoking it. Later input cannot mask a version-check failure.
Filesystem sources and layered loading do not require a different version
policy. The layered API needs a way to identify the version-owning input when a
`VersionRule` is configured; Decision 3 concerns that integration only. Without
version checking, no version owner or synthetic primary document is required.
policy. The designated main input owns the version when a `VersionRule` is
configured, extending the existing selected-source role. It may use any supported
source kind. With no main input, the checker receives an absent declaration,
consistent with existing `NoSource` behavior; underlays cannot supply that
declaration. Without version checking, no version owner or synthetic primary
document is required.
Named-overlay lookup context is a separate concern, not a reason to impose
version ownership on unversioned loads.
@@ -250,7 +301,12 @@ Programmatic input has a non-secret source label and uses explicit structure to
represent presence. Omitted mapping keys are absent; present zero, false, empty
string, empty collection, and nil/null values remain explicit. Use ConfigKit's
existing canonical scalar and collection conventions, not YAML round trips or
Go zero-value inference. Decision 1 selects the initial input representation.
Go zero-value inference. The initial representation is a labeled string-keyed
mapping using the scalar/list/map value set supported by explicit overrides.
This reuses established conventions and keeps presence directly expressible.
The source snapshots the mapping at construction, like byte sources. A nil root
mapping represents null and fails the existing mapping-root requirement; an empty
non-nil mapping is a valid empty document.
Do not label generic programmatic sources as CLI input or misuse synthetic flag
names to fit the existing override API. Preserve parser-independent conversion,
@@ -323,15 +379,16 @@ claim. No timing threshold or additional release gate is requested.
## Work needed from the baseline
These are responsibility areas, not implementation stages:
The target requires changes in the following responsibility areas; the linked
implementation plan owns their execution order and concrete validation:
- Extend the root-owned source/context facade and private acquisition boundary
for `fs.FS`; adapt import, glob, overlay, identity, and resource-context handling
without exporting parser or engine types.
- Add source-backed underlays and programmatic input, sharing one engine path
with ordinary loading rather than building a parallel configuration system.
- Resolve the representation and ownership decisions below, then extend source
records, histories, diagnostics, and rendering consistently across input kinds.
- Extend source records, histories, diagnostics, and rendering consistently
across input kinds.
- Integrate version-owner selection into layered loading while preserving the
existing version lifecycle and protections. Keep filesystem-context selection
independent of version checking.
@@ -363,11 +420,8 @@ These are responsibility areas, not implementation stages:
as behavior lands. Record significant boundary changes in an ADR without
rewriting accepted ADR history. Keep named inheritance explicitly separate.
Plan an additive minor release for the new public capability, preserving current
version semantics. If another design choice requires changing documented behavior,
identify that change expressly and provide migration guidance rather than calling
it an internal optimization.
No release, implementation change, or current-policy revision occurs in this pass.
The capability is additive and intended for a minor release, preserving current
version semantics. Release preparation and publication remain separate tasks.
## Completion criteria
@@ -386,50 +440,17 @@ strict within-tier composition, recursive cross-tier overlay, and downstream dom
resolution. Equivalent-input tests and the maintained consumer example establish
semantic consistency and reduced integration code, not merely API availability.
## Open Decisions
## Selected resource-context interface
### 1. Initial programmatic representation
A separate source-ID-based query exposes standard filesystem access and a
source-relative base location through root-owned API types. Ordinary source
records remain descriptive. The query reports absent context explicitly and
supports imported sources as well as initial inputs. Consumers follow each
contributing origin independently for composite values. This avoids a mandatory
downstream filesystem registry and preserves provenance through recursive
combination.
**Recommendation:** labeled string-keyed mappings with the scalar/list/map value
set already supported by explicit overrides. Missing keys express omission and
nil expresses null. Reuse private conversion machinery, not CLI-specific public
records. This is predictable, directly expressible in Go, and consistent with
current dynamic configuration semantics.
**Alternative:** typed Go values with a separate explicit-presence description.
This can reduce downstream mapping code, but introduces reconciliation of field
bindings, masks, nulls, and typed zero values and a larger public contract. Do not
infer omission from zero values. Add typed construction only if real examples
show that the simple mapping representation imposes substantial repetitive work.
### 2. How consumers obtain filesystem resource context
**Recommendation:** a separate source-context query exposing standard filesystem
access and a source-relative base location through root-owned API types, while
ordinary source records remain descriptive. The returned filesystem is a borrowed
capability, not a cloned or immutable object. This avoids a mandatory downstream
registry and supports origins from imports as well as initial sources. Make
lifetime and nil/no-context behavior explicit before implementation.
**Alternative:** expose opaque context IDs and relative paths; callers retain a
registry mapping IDs to their filesystems. Metadata stays capability-free and
serializable, but every consumer must maintain and correctly reconcile another
registry. This is viable if limiting retained filesystem capabilities proves
more important than the additional integration machinery.
### 3. Selecting the version-owning input for layered loads
**Recommendation:** the designated main source owns the version when configuring
a `VersionRule`, extending the existing selected-source role rather than letting
an underlay win ownership by list position. It need not be a host file or a newly
manufactured root document. Its imports remain fragments and underlays cannot
declare the application's version. No special version-owner designation is
needed when checking is not configured, including loads using only underlays.
**Alternative:** allow explicit designation of a base input as the version owner
separately from its main/underlay role. This supports bundles carrying the
application version, but adds another ownership choice and complicates the
otherwise clear distinction between defaults and application configuration.
Either choice must preserve the existing version contract; this decision does
not reopen effective-version
or per-document version validation.
Caller-supplied filesystems are borrowed capabilities; host contexts are passive
path-backed adapters. Neither is a cloned or immutable filesystem. Queries perform
no I/O, and later resource opens follow the ownership rules above. Configuration
results retain no library-owned open handles.