35 Commits

Author SHA1 Message Date
dc09959cca Prepare ConfigKit v0.5.0 release 2026-09-30 00:18:40 +00:00
83c59ea7d3 Fix nested inheritance history and budget accounting 2026-09-30 00:04:02 +00:00
b45f7a0410 Record sibling inheritance review findings 2026-09-30 00:00:09 +00:00
2ca1f80094 Document and benchmark sibling inheritance 2026-09-29 23:13:50 +00:00
df9a02539c Activate sibling inheritance in configuration loading 2026-09-29 23:04:13 +00:00
501d59f21b Preserve secret sensitivity across sibling inheritance 2026-09-29 22:56:10 +00:00
f916e7d1f9 Preserve sibling inheritance in configuration history and explanations 2026-09-29 22:52:51 +00:00
462e57df3e Preserve inherited source origins and transfer evidence 2026-09-29 22:40:49 +00:00
4207110e2f Resolve nested inheritance in destination scopes 2026-09-29 22:31:07 +00:00
fdbc306cfb Resolve sibling inheritance with memoized parent templates 2026-09-29 22:23:48 +00:00
78d4b59bbe Share recursive mapping merge across configuration assembly 2026-09-29 22:20:10 +00:00
f564fef84c Plan sibling inheritance implementation 2026-09-29 21:37:38 +00:00
01e38b5840 Record agreed inheritance reference and presence rules 2026-09-29 21:18:48 +00:00
ac1eec2e30 Revise inheritance roadmap around built-in sibling references 2026-09-29 21:10:04 +00:00
0a4b774bfe Retire completed plans and outline named inheritance 2026-09-29 20:11:42 +00:00
b604aa167b Fix underlay provenance and resource path handling 2026-09-29 20:03:39 +00:00
15f5083282 Record underlay implementation review findings 2026-09-29 19:16:18 +00:00
cbf031aedc Document and demonstrate native and layered configuration loading 2026-09-29 18:22:45 +00:00
34e18c84c1 Validate source equivalence and layered scaling 2026-09-29 18:15:10 +00:00
36362ee6af Provide passive per-source resource contexts 2026-09-29 18:08:27 +00:00
5a77728a81 Expose layered configuration loading and inspection 2026-09-29 17:58:15 +00:00
1cdf90d831 Compose grouped underlays in shared load lifecycle 2026-09-29 17:52:20 +00:00
90c6e8ed7e Add snapshotted mapping configuration sources 2026-09-29 17:43:19 +00:00
1ff3c812a7 Expose native filesystem sources through public loading API 2026-09-29 17:36:20 +00:00
6111cb23a1 Unify host and native filesystem acquisition 2026-09-29 17:29:01 +00:00
de28718941 Reuse owned assembly across configuration layers 2026-09-29 17:13:14 +00:00
8d9a3a8dfa Index mapping assembly for composition and overlays 2026-09-29 17:03:02 +00:00
27df838513 Plan filesystem sources and underlays 2026-09-29 16:50:11 +00:00
36172ed20e Clarify ConfigKit design goals and priorities 2026-09-29 16:10:16 +00:00
e1ce24aabb Focus the source roadmap on recursive underlays 2026-09-29 16:06:31 +00:00
e4bbedc695 Preserve version semantics in the layering roadmap 2026-09-29 15:52:20 +00:00
798091b276 Align layering roadmap with consistent version semantics 2026-09-29 15:45:21 +00:00
5c1aa1814c Document filesystem sources and layered loading roadmap 2026-09-29 15:25:23 +00:00
ec26591b27 Request named configuration inheritance for PromptKit 2026-09-29 15:07:21 +00:00
99338baabb Record PromptKit filesystem and layered loading requests 2026-09-29 14:54:44 +00:00
96 changed files with 7837 additions and 557 deletions

View File

@@ -5,18 +5,22 @@ An opinionated Go library for applications that use YAML configuration files.
Linux is release-validated; macOS support is best-effort. See the Linux is release-validated; macOS support is best-effort. See the
[platform support policy](docs/policy/release.md#platform-support) for details. [platform support policy](docs/policy/release.md#platform-support) for details.
ConfigKit's initial loader accepts one strict YAML ConfigKit's simple `Load` accepts one strict YAML source (or explicit
source (or explicit defaults-only input), optionally traverses explicit path and glob defaults-only input). `LoadLayered` additionally combines explicit lower-priority
imports, strictly composes their disjoint definitions, applies typed defaults, and source groups with an independently selected main input. Sources can be host
can overlay one selected fixed overlay and check an application schema version. It files, native `fs.FS` files, labeled bytes or readers, or snapshotted mappings.
returns detached metadata and safe diagnostics. Enabled path and glob imports compose strictly within their group; precedence
between groups combines mappings recursively. Both entry points return a complete,
validated configuration with detached metadata and safe diagnostics.
Applications may perform audited defaulting and normalization through a resolver, Applications may perform audited defaulting and normalization through a resolver,
then report domain errors and warnings through read-only semantic validation. Secret then report domain errors and warnings through read-only semantic validation. Secret
tags and a path predicate drive deterministic redacted effective YAML. Typed secret tags and a path predicate drive deterministic redacted effective YAML. Typed secret
references resolve environment names before validation, and explicit references resolve environment names before validation, and explicit
secrets-directory bootstrap is available as a separate startup operation. Explicit secrets-directory bootstrap is available as a separate startup operation. Explicit
CLI override records apply above overlays. Value explanation is available through CLI override records apply above overlays. Value explanation is available through
detached structured records and deterministic redacted output. detached structured records and deterministic redacted output. Built-in
`extends` relationships let a mapping inherit from a named sibling during the
same load, retaining explicit presence and source/derivation history.
```go ```go
type Config struct { type Config struct {
@@ -33,11 +37,37 @@ func ReadConfig(path string) (Config, error) {
} }
``` ```
## Design philosophy
ConfigKit favors one coherent, predictable configuration model over flexibility
for reproducing each consumer's conventions. Its goal is to let applications
load once, inspect where final values came from, and distinguish explicit values
from omissions with minimal configuration bookkeeping. Applications retain their
domain rules and execution responsibilities.
Loading is synchronous and complete on return. Applications normally load once
near startup and then use the final configuration and metadata; ConfigKit stays
inactive until explicitly called again. Applications may initiate additional
loads as needed. See the [loading lifecycle](docs/config.md#completion-and-resource-lifetime)
for completion and ownership details.
Within a configuration tier, independent contributions compose strictly: shared
mappings combine, while overlapping non-mapping definitions are errors. Between
precedence tiers, mappings combine recursively, retaining omitted lower-priority
children; other values replace lower-priority values. The main document and its
imports form one strictly composed tier. Explicit underlay groups sit below it
and above typed defaults; a selected overlay and explicit CLI overrides sit
above it. The [configuration contract](docs/config.md#strict-document-composition)
owns the detailed combination rules and input-specific constraints.
The [architecture goals](docs/policy/architecture.md#success-criteria) explain
these priorities and how genuine new needs can justify extending the shared
model. The [deferred scope](docs/roadmap/deferred.md) records deliberate exclusions.
The [configuration contract](docs/config.md) describes implemented behavior. The The [configuration contract](docs/config.md) describes implemented behavior. The
[consumer adoption guide](docs/consumers/README.md) provides maintained Narratio-, [consumer adoption guide](docs/consumers/README.md) provides maintained simple
Notarius-, WeatherReporter-, and Seriatim-shaped examples and migration guidance. native-filesystem, layered, and sibling-inheritance examples alongside the migration shapes.
The [deferred scope](docs/roadmap/deferred.md) records deliberate exclusions. See the See the [development guide](docs/development.md) for repository orientation and the
[development guide](docs/development.md) for repository orientation and the
[release procedure](docs/release.md) for maintainer-only candidate and publication [release procedure](docs/release.md) for maintainer-only candidate and publication
steps. steps.

View File

@@ -43,16 +43,36 @@ func toEngineLocations(values []Location) []engine.Location {
return result return result
} }
func fromEngineSourceInfo(value engine.SourceInfo) SourceInfo { func fromEngineSourceInfo(value engine.SourceInfo) SourceInfo {
return SourceInfo{ID: value.ID, Label: value.Label, Path: value.Path, Layer: Layer(value.Layer), ImportedFrom: fromEngineLocationPointer(value.ImportedFrom)} return SourceInfo{ID: value.ID, Label: value.Label, Path: value.Path, Layer: Layer(value.Layer), Kind: SourceKind(value.Kind), UnderlayIndex: value.UnderlayIndex, ImportedFrom: fromEngineLocationPointer(value.ImportedFrom)}
} }
func toEngineSourceInfo(value SourceInfo) engine.SourceInfo { func toEngineSourceInfo(value SourceInfo) engine.SourceInfo {
return engine.SourceInfo{ID: value.ID, Label: value.Label, Path: value.Path, Layer: engine.Layer(value.Layer), ImportedFrom: toEngineLocationPointer(value.ImportedFrom)} return engine.SourceInfo{ID: value.ID, Label: value.Label, Path: value.Path, Layer: engine.Layer(value.Layer), Kind: engine.SourceKind(value.Kind), UnderlayIndex: value.UnderlayIndex, ImportedFrom: toEngineLocationPointer(value.ImportedFrom)}
} }
func fromEngineOrigin(value engine.Origin) Origin { func fromEngineOrigin(value engine.Origin) Origin {
return Origin{Layer: Layer(value.Layer), Locations: fromEngineLocations(value.Locations), Operation: ChangeKind(value.Operation)} result := Origin{Layer: Layer(value.Layer), Locations: fromEngineLocations(value.Locations), Operation: ChangeKind(value.Operation)}
if value.Inheritance != nil {
result.Inheritance = make([]InheritanceStep, len(value.Inheritance))
for index, step := range value.Inheritance {
result.Inheritance[index] = InheritanceStep{
From: fromEnginePath(step.From), To: fromEnginePath(step.To), Reference: fromEnginePath(step.Reference),
Locations: fromEngineLocations(step.Locations),
}
}
}
return result
} }
func toEngineOrigin(value Origin) engine.Origin { func toEngineOrigin(value Origin) engine.Origin {
return engine.Origin{Layer: engine.Layer(value.Layer), Locations: toEngineLocations(value.Locations), Operation: engine.ChangeKind(value.Operation)} result := engine.Origin{Layer: engine.Layer(value.Layer), Locations: toEngineLocations(value.Locations), Operation: engine.ChangeKind(value.Operation)}
if value.Inheritance != nil {
result.Inheritance = make([]engine.InheritanceStep, len(value.Inheritance))
for index, step := range value.Inheritance {
result.Inheritance[index] = engine.InheritanceStep{
From: toEnginePath(step.From), To: toEnginePath(step.To), Reference: toEnginePath(step.Reference),
Locations: toEngineLocations(step.Locations),
}
}
}
return result
} }
func fromEngineDiagnostic(value engine.Diagnostic) Diagnostic { func fromEngineDiagnostic(value engine.Diagnostic) Diagnostic {
return Diagnostic{Code: Code(value.Code), Severity: Severity(value.Severity), Path: fromEnginePath(value.Path), Message: value.Message, Primary: fromEngineLocationPointer(value.Primary), Related: fromEngineLocations(value.Related)} return Diagnostic{Code: Code(value.Code), Severity: Severity(value.Severity), Path: fromEnginePath(value.Path), Message: value.Message, Primary: fromEngineLocationPointer(value.Primary), Related: fromEngineLocations(value.Related)}

View File

@@ -0,0 +1,118 @@
# ADR 0007: Add Source-Backed Underlays with Consistent Recursive Precedence
## Status
Proposed draft for review. Completed planning and remediation records are
preserved in Git history at `b604aa1` under `docs/roadmap/`; the
[configuration contract](../config.md#layered-loading) owns current behavior.
This extends [ADR 0001](0001-separate-composition-and-overlay.md) and preserves
its distinction between strict composition and intentional overlay.
## Date
2026-09-29
## Context
Configuration-backed defaults provide lower-priority settings that applications
can override. These sources carry imports, locations, and relative-resource
contexts. Integrating them into ConfigKit's loading lifecycle gives their values
the same presence tracking and inspectable provenance as other configuration
inputs.
The [architecture goals](../policy/architecture.md#success-criteria) prioritize
one coherent, predictable configuration model with consistent rules across
source representations and loading interfaces. Consumers should load once,
receive a complete validated configuration, and inspect its final values' origins.
Reducing integration code is important subject to those principles.
## Decision
### Add underlays to the existing precedence model
Introduce explicitly supplied source-backed underlays between typed defaults
and main configuration. The proposed precedence, from lowest to highest, is:
**Typed application defaults < source underlays < strictly composed main
configuration < selected overlay < explicit CLI overrides.**
Typed defaults remain the Go-value baseline. Underlays supply lower-priority
configuration with source provenance and, where applicable, imports and resource
context. This ordering describes value precedence; typed defaults fill fields
absent from the assembled input under the existing defaulting rules.
Preserve strict composition of the main document and its imports before applying
the resulting configuration over underlays. Independent contributions within an
underlay tier also compose strictly. Explicitly ordered tiers express intentional
precedence. Strict composition reports conflicting contributions within each tier.
Across precedence tiers, reuse the existing recursive mapping-overlay semantics.
Omitted children retain lower-priority input; existing scalar, sequence, null,
and cross-kind replacement semantics remain unchanged. Reusing these operations
keeps the combination rules consistent across the loading model.
### Keep source representation separate from composition semantics
Native `fs.FS` and programmatic inputs join the shared loading model.
Equivalent inputs using the same
operation have the same presence and precedence semantics regardless of medium.
Programmatic values enter directly as labeled configuration sources.
Retain the contributing source's identity and resource context through layering.
A retained underlay value keeps its originating source's resource context.
Consumers remain responsible for opening and interpreting their domain resources
under the supplied filesystem's lifetime and access semantics. Exact
context-access and input APIs remain roadmap decisions.
### Deliver one final configuration and its inspection metadata
Assemble effective input, then run the typed loading lifecycle once for the load.
Preserve explicit zero,
false, empty, and null input versus omission through configuration and metadata
together.
Extend existing provenance and history through underlays, defaults, and recorded
application resolution. The returned result includes the completed combination,
defaulting, and provenance work. Domain resolution and validation
remain application-owned within the recorded lifecycle, consistent with
[ADR 0003](0003-preserve-presence-and-resolution-history.md).
Preserve the existing opt-in schema-version ownership, path protections, and
check timing before precedence
assembly and typed decoding. The roadmap owns how layered input identifies that
owner when version checking is configured.
Favor near-linear processing in input, retained history, and output size through
incremental assembly of load-owned state. This is an engineering objective
subordinate to clarity and semantic consistency. Preserve the root-owned facade and
private implementation boundary established by [ADR 0006](0006-public-facade.md).
## Alternatives considered
- **Leave bundled defaults to consumers.** Avoids extending loading, but makes
consumers reproduce merging, presence tracking, and provenance reconciliation.
It does not meet the goal of obtaining a complete configuration in one load.
- **Offer recursive or atomic combination per layer.** Makes both behaviors
explicit and could serve more consumers, but expands the semantic surface and
its interactions with presence and provenance without a demonstrated need that
outweighs a single consistent rule. A future concrete requirement may justify
reconsideration; this feature does not add the option.
- **Flatten all documents into last-wins layers.** Simplifies ordering but turns
accidental main/import conflicts into overrides, weakening the intentional
distinction established by ADR 0001.
## Consequences
Consumers gain source-backed defaults within the shared loading lifecycle.
Strict contributions and intentional precedence remain distinct, with consistent
combination rules across source media. Presence and provenance remain
part of the same loading lifecycle as the final values.
Higher-priority mappings retain omitted lower-priority children, so application
validation assesses the final combined configuration.
Implementation must integrate source context and underlay history into the shared
engine with attention to assembly and history-retention costs. Current behavior
documentation will change when the feature lands. The roadmap retains ownership
of the remaining API decisions.

View File

@@ -0,0 +1,82 @@
# ADR 0008: Synchronous, Complete-on-Return Configuration Loading
## Status
Proposed draft for review. The
[configuration contract](../config.md#completion-and-resource-lifetime) describes
the implemented loading lifecycle and
[source resource contexts](../config.md#source-resource-contexts).
## Date
2026-09-29
## Context
Applications need a clear point at which configuration is ready for use.
Completing configuration processing and inspection metadata together gives
applications a coherent result whose values and origins agree. A bounded loading
lifecycle also makes resource ownership and startup failures straightforward to
reason about.
ConfigKit's [architecture goals](../policy/architecture.md#success-criteria)
emphasize complete configuration, inspectable provenance, predictable behavior,
and minimal consumer bookkeeping. Applications own their execution lifecycle and
the timing of configuration acquisition.
## Decision
Make loading a synchronous, complete-on-return operation. Within each load,
ConfigKit completes input assembly, defaulting, typed decoding, configured
application resolution, secret lookup, validation, and snapshotting before
returning the final configuration and metadata. Reported failures return
diagnostics and a nil result.
Release all library-owned loading handles before returning on success or failure.
The resulting configuration and metadata are independently usable, caller-owned
data with a retained inspection snapshot. Caller-supplied readers and filesystems
remain under caller ownership and lifetime control.
After the call completes, ConfigKit remains inactive until explicitly invoked
again. Applications normally load once at or near startup and may initiate
additional loads when needed. Each load follows the same lifecycle and source
reuse contracts. Inspection, rendering, and separately requested operations run
in response to explicit application calls.
For the planned resource-context capability, retain source context as passive
information or a borrowed caller-supplied filesystem. Subsequent resource access
acquires its own handles with explicit ownership. This preserves a clear boundary
between configuration completion and application resource use.
These choices give consumers one completion boundary for configuration and
provenance, predictable resource lifetimes, and control over when ConfigKit runs.
Application callbacks remain responsible for their own effects and resources.
## Alternatives considered
- **Background watching or refresh.** Can automate responses to source changes,
but adds scheduling, shutdown, publication, and resource-lifetime concerns to
the library. Explicit calls keep those responsibilities with the application.
- **Lazy completion after return.** Can defer work until particular values are
used, but moves configuration failures into application execution and makes
the readiness of values and metadata harder to establish.
- **Transactional atomic loading.** Could imply simultaneous observation of all
external inputs and rollback of callback effects. Filesystems, environment
variables, and application callbacks do not share a transaction mechanism.
Synchronous completion provides the required usable-result boundary without
promising those external guarantees.
- **Retain loading handles in results.** Can preserve open filesystem access,
but gives configuration results an ongoing resource lifecycle and cleanup
obligation. Passive source context supports later explicit resource access
while keeping loading resources scoped to the load call.
## Consequences
Applications can treat successful return as the point at which the final
configuration and its metadata are ready for use. Startup loading remains simple,
and additional loads have an explicit application-owned trigger.
Configuration inspection operates on retained data. Resource access observes the
backing resource when explicitly requested and has its own failure and ownership
boundary. Implementations of new source kinds and layering features preserve the
same completion boundary, including cleanup on reported load failures.

View File

@@ -0,0 +1,76 @@
# ADR 0009: Resolve Sibling Inheritance After Source Assembly
## Status
Accepted and implemented. The [configuration contract](../config.md#sibling-inheritance)
owns current behavior. The completed implementation plan is preserved in Git
history at `83c59ea`.
## Date
2026-09-29
## Context
Named configuration mappings often share settings while retaining individual
overrides. ConfigKit already distinguishes strict composition of independent
source contributions from recursive overlay between explicit precedence tiers.
Doing another untracked merge in applications would lose omission, source
context, and historical explanation information. A shared structural operation
must preserve the same presence and provenance model as the rest of loading.
Inheritance also interacts with source precedence, nested namespaces, version
ownership, and secrets. Resolving it before source assembly would validate
superseded references and copy values that a later tier should replace. Fully
resolving nested relationships inside a reusable parent would bind them to the
wrong sibling scope when copied into a child.
## Decision
Reserve `extends` as a built-in input key on mappings. A string names one exact
sibling mapping; null cancels the relationship. Complete source acquisition,
strict composition, selected overlay, and CLI precedence first. Then resolve
each effective relationship in the parser-independent document tree, before
typed defaults and decoding. A child recursively overlays its own payload on
its resolved parent using the existing mapping-combination rules. This creates
neither a source tier nor a configurable merge strategy.
Resolve enclosing sibling chains before nested relationships. Reuse memoized
parent templates with unresolved nested instructions; evaluate those instructions
at each destination. Preserve original source applications separately from
structural transfer/control evidence. Effective origins carry concrete
source-to-destination derivation, and inspection projects relevant inheritance
events between source replay and typed reconciliation. Inherited input counts as
explicit presence, while direct replacement retains its own effective origin.
The existing main-input version check remains at its original point. Reject
structural contributions or directive consumption affecting its protected path.
Classify sensitivity at corresponding source and destination paths and retain
historical redaction. Secret references remain unresolved input until the
existing lookup point. Keep all machinery behind the root-owned public facade.
## Alternatives considered
- **Application-owned inheritance after `Load`.** Leaves every consumer to
recreate ordering, presence, provenance, error reporting, and redaction after
ConfigKit has already declared a result complete.
- **Resolve each source before precedence assembly.** Makes superseded
references active and can copy a parent definition before higher-priority
input finishes changing it.
- **Resolve nested relationships in a cached parent subtree.** Shares more
ready-made output but binds nested sibling lookups to the parent's scope,
giving incorrect values at a child destination.
- **Registered collections, configurable keywords, or multiple-parent merge
modes.** Increase public setup and semantic branches without a requirement
that warrants diverging from one structural rule.
## Consequences
Consumers can load a configuration once and inspect direct, inherited,
defaulted, and application-derived values through the existing result surfaces.
`extends` is no longer an ordinary application field, so existing configurations
using that key as data must migrate. Expanding many children can materialize
substantially more output than the input contains; projection and retained-history
budgets bound that work without charging copies as newly acquired sources.
Applications still own domain validation, entry selection, and runtime resource
use.

View File

@@ -12,13 +12,21 @@ ConfigKit exposes opaque sources through these constructors:
| Constructor | Current contract | | Constructor | Current contract |
| --- | --- | | --- | --- |
| `File(path)` | Requires a nonempty path, opens a regular file for each consumption, and closes only the library-owned file handle. Missing, inaccessible, unreadable, directory, and other nonregular inputs fail. | | `File(path)` | Requires a nonempty path, opens a regular file for each consumption, and closes only the library-owned file handle. Missing, inaccessible, unreadable, directory, and other nonregular inputs fail. |
| `NewFilesystem(label, filesystem).File(name)` | Borrows a caller-owned `fs.FS` under a safe nonempty display label. `name` must be a non-root slash-relative `fs.ValidPath`. Each load reopens the selected regular file; copies of the `Filesystem` preserve namespace identity. Separate constructors create separate namespaces even for the same label or filesystem. |
| `Bytes(label, data)` | Requires a nonempty, non-secret diagnostic label and snapshots bytes during construction. Later caller mutation cannot affect the source. | | `Bytes(label, data)` | Requires a nonempty, non-secret diagnostic label and snapshots bytes during construction. Later caller mutation cannot affect the source. |
| `Reader(label, input)` | Requires a nonempty, non-secret label and non-nil reader. The reader is claimed once across all copies of the source, consumed but never closed, and cannot be reused. | | `Reader(label, input)` | Requires a nonempty, non-secret label and non-nil reader. The reader is claimed once across all copies of the source, consumed but never closed, and cannot be reused. |
| `Mapping(label, values)` | Requires a safe nonempty label and a non-nil root `map[string]any`. Snapshots supported built-in values into an owned mapping document at construction; `{}` is valid. Constructor conversion failures become safe deferred diagnostics. |
| `NoSource()` | Selects defaults-only loading, creates no source record, and marks no paths explicit. | | `NoSource()` | Selects defaults-only loading, creates no source record, and marks no paths explicit. |
The zero `Source` is invalid. Constructor argument failures are retained inside the The zero `Source` is invalid. Constructor argument failures are retained inside the
opaque source and become safe structured diagnostics when the source is consumed. opaque source and become safe structured diagnostics when the source is consumed.
File sources can observe file changes between consumptions; byte sources cannot. File and filesystem sources can observe changes between consumptions; byte and
mapping sources cannot. Mapping accepts nil, built-in string, boolean, signed or
unsigned integer, finite float, `[]any`, and `map[string]any` values recursively.
Typed nil maps and slices represent explicit null; empty collections remain empty.
Keys are sorted for stable provenance, integer decimal spelling stays exact, and
cycles or unsupported types are rejected without exposing values. Mapping input
uses the same directive and version rules as YAML, not CLI override restrictions.
Each source is limited to 8 MiB. A load using imports or an active overlay is limited Each source is limited to 8 MiB. A load using imports or an active overlay is limited
to 64 MiB across all accepted sources, 1,024 accepted sources, 64 active import to 64 MiB across all accepted sources, 1,024 accepted sources, 64 active import
@@ -26,6 +34,8 @@ edges, and 1,000,000 input nodes in aggregate. Glob expansion separately permits
1,000,000 visited directory entries, including nonmatches. Reads request at most the 1,000,000 visited directory entries, including nonmatches. Reads request at most the
remaining allowance plus one byte so excess data is rejected rather than silently remaining allowance plus one byte so excess data is rejected rather than silently
truncated. Data returned together with a read error is not accepted as partial input. truncated. Data returned together with a read error is not accepted as partial input.
For mapping sources, key and string bytes use the same accounting as programmatic
overrides, and the converted tree uses the same node and depth bounds.
## Single-source loading ## Single-source loading
@@ -44,13 +54,14 @@ The current lifecycle is:
```text ```text
validate options → parse the selected source → traverse/compose enabled imports validate options → parse the selected source → traverse/compose enabled imports
→ build schema → check the optional application version → apply a selected overlay → build schema → check the optional application version → apply a selected overlay
→ apply explicit CLI overrides → create and clone defaults → strictly decode → apply explicit CLI overrides → expand sibling inheritance
→ create and clone defaults → strictly decode
→ recorded application resolution → recorded application resolution
→ resolve typed secret references → semantic validation → classify secrets → resolve typed secret references → semantic validation → classify secrets
→ snapshot configuration and metadata → snapshot configuration and metadata
``` ```
`File`, `Bytes`, and `Reader` each contribute the first base source with deterministic `File`, `Filesystem.File`, `Bytes`, `Reader`, and `Mapping` each contribute the first base source with deterministic
ID `source-1`; enabled imports continue in depth-first order as `source-2` onward. ID `source-1`; enabled imports continue in depth-first order as `source-2` onward.
`NoSource` skips parsing, supplies an absent version to a configured version checker, `NoSource` skips parsing, supplies an absent version to a configured version checker,
contributes no source, and continues through defaults, decode, and projection. A contributes no source, and continues through defaults, decode, and projection. A
@@ -66,11 +77,166 @@ disabled.
Every source, options, schema, version, defaults, decode, resolution, or validation Every source, options, schema, version, defaults, decode, resolution, or validation
failure returns a nil result and a `*LoadError`. Successful results contain caller-owned `Config`, detached failure returns a nil result and a `*LoadError`. Successful results contain caller-owned `Config`, detached
`Metadata`, a defensive source list, and defensive nonfatal diagnostics. Mutating `Metadata`, a defensive source list, and defensive nonfatal diagnostics. Each source
record's `Kind` identifies file, filesystem, bytes, reader, mapping, or CLI acquisition
independently of its precedence `Layer`. Host file paths remain canonical;
filesystem paths are slash-relative within their namespace; synthetic inputs have
no path. Mutating
the returned configuration or inspection records cannot alter ConfigKit's private the returned configuration or inspection records cannot alter ConfigKit's private
snapshot. ConfigKit performs no logging. `Result.WriteEffective` exposes only the snapshot. ConfigKit performs no logging. `Result.WriteEffective` exposes only the
frozen redacted projection. frozen redacted projection.
### Completion and resource lifetime
`Load` is synchronous and complete on return. Success means input assembly,
defaulting, typed decoding, configured application resolution, secret lookup,
validation, and configuration/metadata snapshotting have completed. A reported
load failure returns diagnostics and a nil result, as described above.
Before returning on success or failure, ConfigKit closes the filesystem handles
it opened for loading, including files and directory/root handles. Caller-owned
filesystems are borrowed, never closed. The returned
configuration and inspection metadata require no close operation and remain
usable independently of the source files. A supplied `Reader` remains
caller-owned and is never closed by ConfigKit.
ConfigKit performs no background work after returning: it does not watch inputs,
refresh secrets, or mutate published results. Inspection and rendering run only
when explicitly called and use the retained snapshot. Applications normally load
once near startup, but may explicitly load again using sources that satisfy their
documented reuse rules. Additional loads produce independently owned results.
Completion is not a transaction over external state. Files and environment
variables are observed during processing rather than as one simultaneous external
snapshot. ConfigKit does not roll back application callback effects or manage work
or resources created by application callbacks. The explicit secrets-directory
bootstrap remains a separate operation invoked by the application.
## Layered loading
`LoadLayered[T](main, options...)` accepts zero or one `LayeredOptions[T]` value.
Its `main` is an `Input` containing a source, import opt-in, and optional
filesystem context. A zero source is invalid; `Input{Source: NoSource()}` omits
the main document. Each `Underlay` contains inputs that compose strictly as
one tier, including each input's enabled imports. Empty groups contribute
nothing. Underlay groups are listed from lowest to highest priority. Repeating
a file within one group remains an import/repetition error; the same reusable
source may be selected again in another tier.
ConfigKit acquires and composes the main input first, constructs the schema,
and checks the optional application version from the main tier only. Underlay
roots cannot claim that version path (`underlay.version`); imported fragments
retain `import.version` protection. It then acquires and composes underlay
groups in order, recursively combines the groups, and applies the main tier,
one selected overlay, and explicit CLI overrides. Scalar, sequence, and null
definitions replace lower values between tiers; mappings combine recursively.
Syntax and strict-composition errors inside a tier cannot be masked by a
higher-priority tier. Defaults, typed decoding, resolution, secret lookup,
validation, and publication run once on the final assembled input.
An active layered overlay requires `LayeredOptions.OverlayFilesystem` explicitly,
independent of the main source's namespace. Its name and root-relative lookup
follow the selected-overlay rules below; inactive overlays ignore the context.
The same path in a different filesystem namespace is not a repeated file.
Ordinary `Load` continues to infer its selected-overlay context from its source.
Source IDs follow acquisition order—main and its depth-first imports, then each
underlay tier's inputs and depth-first imports, then overlay and CLI—even though
history follows precedence order. Source `Layer` describes precedence while
`Kind` describes acquisition. Underlay records use `LayerUnderlay` and a
one-based `UnderlayIndex` giving their original tier position; all other records
have index zero. Explicit empty groups and `NoSource` consume no source ID.
Byte, source, node, and glob-entry budgets are shared across the load, while
the existing per-source and import-depth bounds still apply.
The maintained [layered consumer example](consumers/README.md#native-filesystem-and-layered-input)
shows a main host file, two underlay tiers, embedded defaults, a mapping source,
and explicit resource access without a second configuration pass.
### Sibling inheritance
Any input mapping may declare `extends: name` to inherit from the mapping named
`name` among its immediate siblings. Names are exact keys, including the empty
string; lookup does not search ancestor or unrelated scopes. The document root
and a sequence-element mapping have no sibling namespace for their own string
relationship, though mappings nested inside them can have one. An `extends`
value of null cancels a relationship without looking up a parent. Other value
types, missing or non-mapping parents, and cycles fail the load with value-free
diagnostics.
`extends` is reserved structural input across ordinary and layered loads, every
source kind, imports, selected overlays, and explicit CLI overrides. It is
consumed before typed decoding and need not appear in the application's Go
types. Applications previously using `extends` as an ordinary YAML field must
rename that key and its field. No collection registration or keyword setup is
required.
ConfigKit completes strict composition within tiers and precedence assembly
between tiers before resolving relationships. An omitted `extends` retains a
lower-priority relationship; `extends: null` cancels it while preserving the
child's own settings. Superseded references are historical evidence, not active
relationships to validate. Each child recursively overlays its own settings
over its resolved parent: explicit zero, false, empty string, sequence, and
null values replace inherited values, while mappings combine recursively.
The child retains its key. All effective relationships resolve, including
entries the application may never select for execution. Partial parent
definitions undergo normal typed decoding and application-owned validation,
not a special structural exemption.
Enclosing relationships resolve first. Nested instructions carried by a parent
are then resolved separately at each destination, so a child's nested sibling
override can change an inherited nested relationship. Typed defaults and
application-derived values do not start another inheritance pass. Inherited
input counts as explicit presence for `Metadata.IsExplicit`,
`Resolution.IsExplicit`, and absence-based resolver defaults, even for false,
zero, empty, or null input. `Origin.Inheritance` distinguishes it from a direct
declaration with concrete source, destination, and controlling directive paths.
A direct child replacement has its own effective origin; displaced ancestry
remains inspectable in history. An inherited relative resource keeps the
declaring source's passive `SourceContext`, not the child's directory.
The optional schema-version checker still runs once against the main input at
its original point. Inheritance cannot add, replace, or consume the protected
version path; an unrelated relationship inside an enclosing mapping does not
invalidate an unchanged version. Expansion uses the decoded projection
node/depth limits without charging copied values as acquired sources;
retained evidence uses the existing history-node limit. See the
[consumer example](consumers/README.md#named-sibling-inheritance) for one-load
usage.
### Source resource contexts
`Result.SourceContext(id)` and callback-scoped `Resolution.SourceContext(id)`
return a passive `ResourceContext` for a source with filesystem context, or
false for unknown and contextless IDs. The result query is nil-safe; the
resolution query follows the callback's lifetime guard. Queries perform no
filesystem access. Obtain IDs from exact origin locations rather than guessing
one context for a composite origin or an application-derived value.
`ResourceContext.FS` is rooted at the source's configured root, and `BaseDir`
is its slash-relative directory inside that root. This includes imported
sources, selected overlays, host files even without enabled imports, native
files, and byte, reader, or mapping inputs with an explicit context. CLI and
contextless synthetic inputs have no resource context. To open a relative
resource, join `BaseDir` with the application resource name using slash-path
operations and open that path through `FS`.
For a resource-valued field, query `result.Metadata().Lookup(fieldPath)`, inspect
its origin locations, choose a source ID only when the application's policy can
identify the intended contributor, then call `result.SourceContext(id)`. The
context belongs to that source even when the field survived from a lower-priority
underlay. A composite origin has no automatically selected resource base.
Native contexts borrow the caller's filesystem within the configured root.
Host contexts retain a canonical root path, not an open root handle; each
resource `Open` checks containment and opens a fresh handle that the returned
file's `Close` releases. Applications own and close the files they open.
Resources are application data, not configuration input, and do not consume
configuration byte or node budgets. The loaded configuration and metadata stay
usable if source files later change or disappear. Later resource operations
observe the backing filesystem then and may fail if files move, disappear, or
a caller-owned filesystem closes. This is not a snapshot or universal sandbox.
### Application schema versions ### Application schema versions
`VersionRule` declares a semantic path and non-nil checker. Before defaults or typed `VersionRule` declares a semantic path and non-nil checker. Before defaults or typed
@@ -108,8 +274,9 @@ successful result. Callback panics remain programming failures.
`Result.Metadata` describes every final canonical path. Queries retain explicit `Result.Metadata` describes every final canonical path. Queries retain explicit
key/index distinctions using actual effective and historical container shapes; key/index distinctions using actual effective and historical container shapes;
only parsed numeric tokens are resolved contextually. A descendant that never only parsed numeric tokens are resolved contextually. A descendant that never
existed is not historical merely because an atomic ancestor was replaced. Exact surviving input paths existed is not historical merely because an atomic ancestor was replaced. Exact
are explicit and have a base or overlay origin; omitted paths have a default origin. surviving direct or inherited input paths are explicit and retain their source
layer and derivation; omitted paths have a default origin.
An explicit parent mapping does not make omitted children explicit. Null state and An explicit parent mapping does not make omitted children explicit. Null state and
available source coordinates are retained. `History` records the Go-zero or application available source coordinates are retained. `History` records the Go-zero or application
default baseline followed by exact base or overlay definitions, including default baseline followed by exact base or overlay definitions, including
@@ -124,8 +291,9 @@ domain ownership rules remain application responsibilities. The method's Go
documentation in [metadata.go](../metadata.go) defines enumeration and path-ordering documentation in [metadata.go](../metadata.go) defines enumeration and path-ordering
details. details.
History is chronological: a path can have a synthetic default record, its exact base History is chronological: a path can have a synthetic default record, its exact
definition, an overlay replacement, and a recorded resolution operation. A descendant base definition, an overlay replacement, structural inheritance, and a recorded
resolution operation. A descendant
query uses the ancestor's path for a whole-container or whole-sequence replacement; query uses the ancestor's path for a whole-container or whole-sequence replacement;
the descendant can therefore retain history after `Lookup` stops reporting it as the descendant can therefore retain history after `Lookup` stops reporting it as
effective. Equal-valued input is still an explicit base definition, while a derived effective. Equal-valued input is still an explicit base definition, while a derived
@@ -141,6 +309,15 @@ operation. Reasons and dependency paths retain first occurrence order with dupli
removed. Source records remain ordered as selected source, depth-first imports, the removed. Source records remain ordered as selected source, depth-first imports, the
selected overlay, then synthetic CLI records in supplied order. selected overlay, then synthetic CLI records in supplied order.
Inheritance adds no acquired source or precedence layer. `Origin.Inheritance`
lists concrete transfers from contributor paths through destinations, with
each step's controlling `extends` path and location. `History` retains source
applications at their original paths and projects parent contributions, child
replacements, and consumed controls at destination paths after source assembly
and before typed decoding. `HistoricalPaths` includes consumed directives and
displaced inherited descendants. A control can have history even when `Lookup`
reports it absent. Queries return detached records.
### Secret classification and effective rendering ### Secret classification and effective rendering
Secret classification combines `configkit:"secret"` field tags with the optional Secret classification combines `configkit:"secret"` field tags with the optional
@@ -156,6 +333,11 @@ Propagation reaches a fixed point, so record/map iteration order does not affect
result and dependency cycles terminate. A dependency on a classified descendant or result and dependency cycles terminate. A dependency on a classified descendant or
on a container that contains one is treated conservatively as sensitive. Applications on a container that contains one is treated conservatively as sensitive. Applications
must classify secret-producing transformations whose dependencies are not recorded. must classify secret-producing transformations whose dependencies are not recorded.
Inherited value and control paths link source and destination classifications
through chains and nested copies; unrelated sibling fields are not classified
merely because another field is secret. Displaced historical evidence remains
redacted. Secret references are copied only as unresolved names; credential
lookup stays after decoding and application resolution.
`Result.WriteEffective` writes the frozen resolved snapshot as deterministic redacted `Result.WriteEffective` writes the frozen resolved snapshot as deterministic redacted
YAML. Mappings sort by decoded key string, sequences retain order, indentation is two YAML. Mappings sort by decoded key string, sequences retain order, indentation is two
@@ -190,7 +372,10 @@ parser values. `Present=false` with empty YAML means absence or deletion; explic
null is present and renders as `null`. The history starts with the canonical typed null is present and renders as `null`. The history starts with the canonical typed
application-default or Go-zero baseline, then includes only relevant base sources in application-default or Go-zero baseline, then includes only relevant base sources in
traversal order, the selected overlay, supplied CLI records, and audited net traversal order, the selected overlay, supplied CLI records, and audited net
resolution operations. Overlapping parent/child resolution records are rendered resolution operations. Structural inheritance events appear after source replay
and before typed reconciliation, including consumed or canceled `extends` input
and equal-valued child replacements. Overlapping parent/child resolution records
are rendered
as one net subtree event with combined reasons and dependencies, rather than as one net subtree event with combined reasons and dependencies, rather than
pretending that the recorded assignments ran in lexical order. Explicit equal pretending that the recorded assignments ran in lexical order. Explicit equal
writes remain visible. When input replay writes remain visible. When input replay
@@ -442,8 +627,8 @@ input document.
`[a-zA-Z0-9_-]+`; they are not paths. The empty name and exact lowercase name `[a-zA-Z0-9_-]+`; they are not paths. The empty name and exact lowercase name
`default` select no overlay, and `overlays/default.yml` is never loaded implicitly. `default` select no overlay, and `overlays/default.yml` is never loaded implicitly.
Every other name resolves only to `<configuration-root>/overlays/<name>.yml`, with Every other name resolves only to `<configuration-root>/overlays/<name>.yml`, with
no extension fallback, case normalization, inheritance, or overlay stack. A selected no extension fallback, case normalization, inheritance between overlay files,
missing overlay is an error. or overlay stack. A selected missing overlay is an error.
The overlay is a separately policy-validated YAML mapping. It is loaded after the The overlay is a separately policy-validated YAML mapping. It is loaded after the
base import graph is composed and its schema version is checked, but before defaults base import graph is composed and its schema version is checked, but before defaults
@@ -557,18 +742,25 @@ walk failures, matched directories, invalid documents, escapes, and repeated tar
remain errors. Final candidates are bounded by the remaining source limit before remain errors. Final candidates are bounded by the remaining source limit before
their contents are accumulated. their contents are accumulated.
The configuration root defaults to a selected file's canonical directory. A file For host files, the configuration root defaults to the selected file's canonical
source may supply `FilesystemContext.Root` to widen that root but may not supply directory. A host file may supply `FilesystemContext.Root` to widen that root but
`BaseDir`. Whenever byte and reader sources supply filesystem context, they must may not supply `BaseDir` or a native `FS`. Byte, reader, and mapping inputs with host context
provide both `BaseDir` and `Root` as absolute existing directories; their diagnostic must provide both `BaseDir` and `Root` as absolute existing directories.
labels, including URI-looking labels, never imply filesystem paths. The base and For `Filesystem.File`, the default root is `.` in its namespace and the base is
selected file must be inside the canonical root. Nonregular targets are rejected before opening to avoid blocking on FIFOs; the selected file's directory. An explicit context must use the same `Filesystem`
namespace and may set `Root` to contain the file and its base; it may not supply
`BaseDir`. Byte, reader, and mapping inputs with native context provide both `BaseDir` and
`Root` as slash-relative `fs.ValidPath` directories in `FS` (`.` means its root),
with the base inside the root. Context paths are checked even when imports and
overlays are inactive. Diagnostic labels, including URI-looking labels, never
imply filesystem paths. Nonregular targets are rejected before opening to avoid blocking on FIFOs;
post-open regular-file checks remain in place under the stable-input assumption. post-open regular-file checks remain in place under the stable-input assumption.
ConfigKit canonicalizes symlinks, ConfigKit canonicalizes symlinks,
checks containment with relative paths, and opens the canonical root-relative target checks containment with relative paths, and opens the canonical root-relative target
through a library-owned root handle. Contained relative or absolute symlink targets through a library-owned root handle. Contained relative or absolute symlink targets
are accepted; escapes are rejected. Canonical paths define identity, so hard links at are accepted; escapes are rejected. Canonical paths define identity, so hard links at
distinct paths remain distinct sources. distinct paths remain distinct sources. Native filesystems use slash-path operations
and their own symlink behavior; imports stay in their selected namespace and root.
An active-stack repeat is `import.cycle` and reports the complete cycle's import An active-stack repeat is `import.cycle` and reports the complete cycle's import
sites. A previously visited canonical path is `import.repeated` and reports both sites. A previously visited canonical path is `import.repeated` and reports both

View File

@@ -7,7 +7,7 @@ validation, schema-version policy, CLI parsing, runtime matching and conversion,
resource checks, remote acquisition, and service initialization. resource checks, remote acquisition, and service initialization.
The maintained [`examples/consumers`](../../examples/consumers) package demonstrates The maintained [`examples/consumers`](../../examples/consumers) package demonstrates
four sanitized shapes using only ConfigKit and the Go standard library: sanitized shapes using only ConfigKit and the Go standard library:
| Shape | Demonstrated fit | Application code deliberately retained | | Shape | Demonstrated fit | Application code deliberately retained |
| --- | --- | --- | | --- | --- | --- |
@@ -15,6 +15,9 @@ four sanitized shapes using only ConfigKit and the Go standard library:
| Notarius-like | Canonical object bindings, a string-backed schema version accepting string or integer YAML, strict dynamic options, and `int64` dynamic integers | Catalog selection, allowed option keys, range-checked conversion to runtime `int` | | Notarius-like | Canonical object bindings, a string-backed schema version accepting string or integer YAML, strict dynamic options, and `int64` dynamic integers | Catalog selection, allowed option keys, range-checked conversion to runtime `int` |
| WeatherReporter-like | Defaults-only loading, report maps, canonical module objects, typed option unions, explicit CLI records, and typed secret startup | Report/module compatibility, rejection of explicitly empty domain lists, CLI parsing, and startup ordering | | WeatherReporter-like | Defaults-only loading, report maps, canonical module objects, typed option unions, explicit CLI records, and typed secret startup | Report/module compatibility, rejection of explicitly empty domain lists, CLI parsing, and startup ordering |
| Seriatim-like | Independent strict ordered speaker and autocorrect rule files | First-match and ordered replacement algorithms; no central application configuration file | | Seriatim-like | Independent strict ordered speaker and autocorrect rule files | First-match and ordered replacement algorithms; no central application configuration file |
| Bundled file | Simple native `fs.FS` source through ordinary `Load` | Selection and lifetime of the application-owned embedded filesystem |
| Layered catalog | Embedded defaults with an import, a mapping underlay, an independent host main file, explicit false/zero input, and source-relative resource access | Domain validation and opening/interpreting the template resource |
| Named presets | One-load sibling inheritance, explicit false, nested destination resolution, and source/derivation inspection | Choosing and executing a preset |
The YAML under [`testdata`](../../examples/consumers/testdata) is loaded by the The YAML under [`testdata`](../../examples/consumers/testdata) is loaded by the
executable examples and ordinary tests. It contains no private application data, executable examples and ordinary tests. It contains no private application data,
@@ -48,6 +51,43 @@ both without redefining their syntax. ConfigKit overlays are configuration overl
they do not select or replace an application's LLM, model, pipeline, report, or they do not select or replace an application's LLM, model, pipeline, report, or
execution profile concepts. execution profile concepts.
## Named sibling inheritance
[`LoadPresets`](../../examples/consumers/inheritance.go) loads the maintained
[`presets.yml`](../../examples/consumers/testdata/inheritance/presets.yml) in one
call. `custom` inherits `enabled: false` from `base`, while its nested
`settings.primary` changes the inherited `settings.secondary` result at the
custom destination. The application struct has no `extends` field; ConfigKit
consumes the relationship before strict typed decoding.
The [executable example](../../examples/consumers/consumers_test.go) reads
`Metadata.IsExplicit` for the inherited false value, then follows
`Origin.Inheritance` back to the parent path while `Origin.Locations` identifies
the declaring source. Applications still choose which preset to execute and
validate domain meaning. The [configuration contract](../config.md#sibling-inheritance)
owns exact lookup, precedence, cancellation, and migration rules.
## Native filesystem and layered input
[`LoadBundled`](../../examples/consumers/layered.go) keeps the simple `Load` shape:
it selects a file in an application-owned embedded filesystem and needs no
underlay setup. [`LoadCatalog`](../../examples/consumers/layered.go) uses
`LoadLayered` with an imported embedded-defaults group, a separate
`Mapping` group, and a host main file supplied by its caller. The groups
compose strictly within themselves and recursively across precedence boundaries;
the main file's explicit `false` and `0` replace lower values without being
mistaken for omissions. Its validator owns allowed modes and resource-name policy.
[`ReadWelcome`](../../examples/consumers/layered.go) follows the effective field's
origin location to a source ID, queries `Result.SourceContext`, and opens the
resource relative to that context's `BaseDir`. It does not assume the main file
owns a retained underlay value. A composite origin needs an application choice;
ConfigKit does not guess a directory. The context query is passive, while later
resource reads observe the caller-owned filesystem or a freshly opened host root
and may fail independently of the completed configuration load. The
[configuration contract](../config.md#source-resource-contexts) owns exact
lifetime and precedence rules.
Known module catalogs should use canonical object entries with a module identifier Known module catalogs should use canonical object entries with a module identifier
and typed options. A small union struct is sufficient when the available option and typed options. A small union struct is sufficient when the available option
fields are known; validation selects which fields apply to each module. Runtime- fields are known; validation selects which fields apply to each module. Runtime-
@@ -126,6 +166,8 @@ focused behavioral owner:
| Source/import/selected-overlay pipeline rules and artifact defaults | Narratio-shaped executable example and fixture | | Source/import/selected-overlay pipeline rules and artifact defaults | Narratio-shaped executable example and fixture |
| Canonical module objects and dynamic option handling | WeatherReporter- and Notarius-shaped examples and migration tests | | Canonical module objects and dynamic option handling | WeatherReporter- and Notarius-shaped examples and migration tests |
| `LoadSecretsDir`, bootstrap reporting, typed lookup, CLI explanation | `Example_weatherStartup` | | `LoadSecretsDir`, bootstrap reporting, typed lookup, CLI explanation | `Example_weatherStartup` |
| Native filesystem, grouped underlays, programmatic mapping, retained resource context | `Example_nativeFilesystem`, `Example_layeredCatalog`, and their focused consumer tests |
| Named sibling inheritance, presence, and derivation | `Example_inheritedPresets` and `TestPresetCatalogOneLoadAndDerivation` |
| Results, metadata, diagnostics, effective output, and writer failures | Focused root-package load, provenance, rendering, and explanation tests | | Results, metadata, diagnostics, effective output, and writer failures | Focused root-package load, provenance, rendering, and explanation tests |
The package-level import-boundary test covers the consumer examples too: supported The package-level import-boundary test covers the consumer examples too: supported

View File

@@ -80,6 +80,14 @@ and current facade ownership belongs to the [component overview](internal/overvi
## Baseline validation ## Baseline validation
The completed filesystem-source and underlay roadmap, original feature request,
implementation plan, and resolved remediation record are preserved in Git history
at `b604aa1` under `docs/roadmap/`. Current behavior belongs to the
[configuration contract](config.md#layered-loading). Built-in sibling
inheritance is now part of the [configuration contract](config.md#sibling-inheritance).
The completed request, roadmap, plan, and resolved review findings are preserved
in Git history at `83c59ea` under `docs/roadmap/`.
The completed overlay rename plan and release-workflow simplification record are The completed overlay rename plan and release-workflow simplification record are
preserved in Git history at `73ce6e9` under `docs/roadmap/`. Current overlay preserved in Git history at `73ce6e9` under `docs/roadmap/`. Current overlay
behavior belongs to the [configuration contract](config.md#selected-overlays), behavior belongs to the [configuration contract](config.md#selected-overlays),
@@ -133,12 +141,18 @@ code changes. Use the component overview to find the corresponding implementatio
| --- | --- | --- | | --- | --- | --- |
| YAML syntax, diagnostics, parser limits | [Adapter tests](../internal/yaml/parse_test.go), [source translation](../internal/engine/parse_test.go) | `go test ./internal/yaml` and `go test ./internal/engine -run '^TestParseSource'` | | YAML syntax, diagnostics, parser limits | [Adapter tests](../internal/yaml/parse_test.go), [source translation](../internal/engine/parse_test.go) | `go test ./internal/yaml` and `go test ./internal/engine -run '^TestParseSource'` |
| File/reader access, imports, globs, containment | [Sources](../internal/engine/source_test.go), [FIFO](../internal/engine/source_fifo_test.go), [imports](../internal/engine/imports_test.go), [globs](../internal/engine/glob_test.go) | `go test ./internal/engine -run 'Source\|Import\|Glob\|Filesystem\|Nonregular\|RegularFile'` | | File/reader access, imports, globs, containment | [Sources](../internal/engine/source_test.go), [FIFO](../internal/engine/source_fifo_test.go), [imports](../internal/engine/imports_test.go), [globs](../internal/engine/glob_test.go) | `go test ./internal/engine -run 'Source\|Import\|Glob\|Filesystem\|Nonregular\|RegularFile'` |
| Native filesystem and mapping sources | [Native acquisition](../internal/engine/native_filesystem_test.go), [filesystem facade](../facade_filesystem_test.go), [mapping facade](../facade_mapping_test.go) | `go test ./internal/engine -run 'NativeFilesystem'` and `go test . -run 'TestFacade'` |
| Composition, selected overlays, CLI precedence | [Composition](../internal/engine/compose_test.go), [overlay](../internal/engine/overlay_test.go), [selected overlays](../internal/engine/selected_overlay_test.go), [overrides](../internal/engine/override_test.go) | `go test ./internal/engine -run 'Compose\|Overlay\|Override'` | | Composition, selected overlays, CLI precedence | [Composition](../internal/engine/compose_test.go), [overlay](../internal/engine/overlay_test.go), [selected overlays](../internal/engine/selected_overlay_test.go), [overrides](../internal/engine/override_test.go) | `go test ./internal/engine -run 'Compose\|Overlay\|Override'` |
| Sibling inheritance, provenance, history, sensitivity | [Expansion](../internal/engine/inheritance_test.go), [destination scopes](../internal/engine/inheritance_walk_test.go), [origins](../internal/engine/inheritance_origin_test.go), [history](../internal/engine/inheritance_history_test.go), [secrets](../internal/engine/inheritance_secret_test.go), [public loading](../facade_inheritance_test.go) | `go test ./internal/engine -run 'Inheritance\|Inherited'` and `go test . -run '^TestFacadeInheritance\|^TestFacadeInheritedSecrets'` |
| Layered loading and source equivalence | [Grouped lifecycle](../internal/engine/load_request_test.go), [layered facade](../facade_layered_test.go), [equivalence](../facade_equivalence_test.go) | `go test ./internal/engine -run '^TestGrouped'`, `go test . -run '^TestFacadeLayered'`, and `go test . -run '^TestEquivalentSourceForms'` |
| Schema, defaults, projection, cloning | [Schema](../internal/engine/schema_test.go), [decoding](../internal/engine/decode_test.go), [projection](../internal/engine/projection_test.go) | `go test ./internal/engine -run 'Schema\|Decode\|Projection\|Clone\|Inline\|CompareDocument'` | | Schema, defaults, projection, cloning | [Schema](../internal/engine/schema_test.go), [decoding](../internal/engine/decode_test.go), [projection](../internal/engine/projection_test.go) | `go test ./internal/engine -run 'Schema\|Decode\|Projection\|Clone\|Inline\|CompareDocument'` |
| Lifecycle, resolution, validation, callback ownership | [Loading](../internal/engine/load_test.go), [resolution](../internal/engine/resolution_test.go), [validation](../internal/engine/validation_test.go), [ownership](../internal/engine/callback_ownership_test.go), [independence](../internal/engine/independence_test.go) | `go test ./internal/engine -run 'Load\|Resolution\|Validation\|Validator\|Hook\|Callback\|Parallel'` | | Lifecycle, resolution, validation, callback ownership | [Loading](../internal/engine/load_test.go), [resolution](../internal/engine/resolution_test.go), [validation](../internal/engine/validation_test.go), [ownership](../internal/engine/callback_ownership_test.go), [independence](../internal/engine/independence_test.go) | `go test ./internal/engine -run 'Load\|Resolution\|Validation\|Validator\|Hook\|Callback\|Parallel'` |
| Paths, metadata, provenance | [Paths](../internal/engine/path_test.go), [metadata](../internal/engine/metadata_test.go), [provenance](../internal/engine/provenance_test.go) | `go test ./internal/engine -run 'Path\|Metadata\|Provenance\|Queries'` | | Paths, metadata, provenance | [Paths](../internal/engine/path_test.go), [metadata](../internal/engine/metadata_test.go), [provenance](../internal/engine/provenance_test.go) | `go test ./internal/engine -run 'Path\|Metadata\|Provenance\|Queries'` |
| Passive resource contexts | [Engine contexts](../internal/engine/resource_context_test.go), [public contexts](../facade_resource_context_test.go) | `go test ./internal/engine -run 'Resource'` and `go test . -run 'SourceContext'` |
| Secrets, bootstrap, redaction, explanation | [References](../internal/engine/secret_reference_test.go), [bootstrap](../internal/engine/secrets_test.go), [rendering](../internal/engine/render_test.go), [explanation](../internal/engine/explain_test.go) | `go test ./internal/engine -run 'Secret\|Bootstrap\|Effective\|Explain\|Explanation\|History\|Sensitivity'` | | Secrets, bootstrap, redaction, explanation | [References](../internal/engine/secret_reference_test.go), [bootstrap](../internal/engine/secrets_test.go), [rendering](../internal/engine/render_test.go), [explanation](../internal/engine/explain_test.go) | `go test ./internal/engine -run 'Secret\|Bootstrap\|Effective\|Explain\|Explanation\|History\|Sensitivity'` |
| Consumer examples and migration fixtures | [Consumer tests](../examples/consumers/consumers_test.go), [API examples](../example_test.go) | `go test ./examples/consumers` and `go test . -run '^Example'` | | Consumer examples and migration fixtures | [Consumer tests](../examples/consumers/consumers_test.go), [native and layered examples](../examples/consumers/layered_test.go), [API examples](../example_test.go) | `go test ./examples/consumers` and `go test . -run '^Example'` |
| Assembly and layered scaling evidence | [Assembly benchmarks](../internal/engine/assembly_bench_test.go), [layered benchmarks](../internal/engine/underlay_bench_test.go) | `go test ./internal/engine -run '^$' -bench '^BenchmarkUnderlay' -benchmem`; repeat with `^BenchmarkCompose` and `^BenchmarkOverlay` |
| Inheritance scaling evidence | [Expansion benchmarks](../internal/engine/inheritance_bench_test.go) | `go test ./internal/engine -run '^$' -bench '^BenchmarkInheritance' -benchmem` |
| Documentation links and dependency isolation | [Link checks](../documentation_test.go), [import boundary](../import_boundary_test.go), [public boundary](../public_boundary_test.go) | `go test . -run 'TestLocalDocumentationLinks\|TestDependencyImportBoundary\|TestPublicFacadeBoundary'` | | Documentation links and dependency isolation | [Link checks](../documentation_test.go), [import boundary](../import_boundary_test.go), [public boundary](../public_boundary_test.go) | `go test . -run 'TestLocalDocumentationLinks\|TestDependencyImportBoundary\|TestPublicFacadeBoundary'` |
| Development and release checker behavior | [Shell regression](../release_check_test.go) | `go test . -run 'TestCheckScripts\|TestDevelopmentCheck'` | | Development and release checker behavior | [Shell regression](../release_check_test.go) | `go test . -run 'TestCheckScripts\|TestDevelopmentCheck'` |
| Guarded release publication and recovery | [Publication regression](../release_publish_test.go) | `go test . -run '^TestReleasePublication'` | | Guarded release publication and recovery | [Publication regression](../release_publish_test.go) | `go test . -run '^TestReleasePublication'` |

View File

@@ -7,23 +7,28 @@ private state; public signatures and fields do not expose engine types.
Implemented component owners: Implemented component owners:
- `internal/engine` owns source acquisition, composition, schema inspection, - `internal/engine` owns host and native-filesystem source acquisition,
decoding, lifecycle coordination, callback auditing, provenance/history, grouped strict composition, recursive precedence assembly, and destination-local
secret resolution and bootstrap, and safe inspection rendering. It never sibling inheritance before typed decoding. It also owns schema
imports the root package. inspection, lifecycle coordination, callback auditing,
- Root `Source`, `Path`, `Secret`, records, lifecycle callbacks, results, and provenance/history, passive source-context retention, secret resolution and
bootstrap reports are root-defined wrappers or records. Callback wrappers pass bootstrap, and safe inspection rendering. It never imports the root package.
only live engine context during a callback; `Result.Config` is detached and its - Root `Source`, `Filesystem`, `Input`, `Underlay`, `Path`, `Secret`, records,
private inspection remains frozen independently. lifecycle callbacks, results, and bootstrap reports are root-defined wrappers
or records. Callback wrappers pass only live engine context during a callback;
`Result.Config` is detached and its private inspection remains frozen
independently.
- Root conversion files explicitly cross the path, record, callback, secret, and - Root conversion files explicitly cross the path, record, callback, secret, and
error boundaries. There are no public aliases, anonymous engine embeddings, or error boundaries. There are no public aliases, anonymous engine embeddings, or
public engine bridge methods. public engine bridge methods.
- `internal/document` owns the parser-independent ordered tree and cloning. - `internal/document` owns the parser-independent ordered tree and cloning.
`internal/yaml` is the sole upstream YAML adapter and parser/dumper boundary. `internal/yaml` is the sole upstream YAML adapter and parser/dumper boundary.
- Engine mechanism tests own source, composition, decoding, callback, provenance, - Engine mechanism tests own source, composition, inheritance, decoding,
secret, history, rendering, and bootstrap behavior. Root tests own public API callback, provenance,
compatibility, facade integration, documentation links, and dependency/public source-context, secret, history, rendering, and bootstrap behavior. Root tests
boundary checks. Consumer examples remain under `examples/consumers`. own public API compatibility, facade integration, documentation links, and
dependency/public boundary checks. Consumer examples remain under
`examples/consumers`.
Dependency direction is root to engine to document/YAML support. The root may Dependency direction is root to engine to document/YAML support. The root may
import only `internal/engine` among project internal production packages; engine import only `internal/engine` among project internal production packages; engine
@@ -36,4 +41,6 @@ semantics. The [consumer adoption guide](../consumers/README.md) owns applicatio
guidance. The [architecture policy](../policy/architecture.md) owns contributor guidance. The [architecture policy](../policy/architecture.md) owns contributor
boundaries, and [ADR 0006](../adr/0006-public-facade.md) records the facade boundaries, and [ADR 0006](../adr/0006-public-facade.md) records the facade
decision. See the [development guide](../development.md) for validation commands decision. See the [development guide](../development.md) for validation commands
and [release procedure](../release.md) for source-candidate gates. and [release procedure](../release.md) for source-candidate gates. [ADR
0009](../adr/0009-sibling-inheritance.md) records why inheritance is a built-in
post-assembly structural operation.

View File

@@ -2,7 +2,7 @@
**Base Policy Version:** 1.3 **Base Policy Version:** 1.3
**Last Updated**: 2026-09-28 **Last Updated**: 2026-09-29
**Upstream Source:** [Standard Repository Policies](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/architecture.md) **Upstream Source:** [Standard Repository Policies](https://gitea.maximumdirect.net/eric/policy/src/commit/d84a38ebab26f048cafbe59a2246700c35d29843/docs/policy/architecture.md)
@@ -34,6 +34,23 @@ Do not promote incidental implementation choices into architectural invariants.
## Success Criteria ## Success Criteria
ConfigKit prioritizes a coherent configuration model: consistency and
predictability take precedence over flexibility for accommodating individual
downstream conventions. Consumers should learn one set of rules, not different
combination or presence semantics depending on the source format or consuming
application. Distinct operations such as strict composition and intentional
overlay express distinct purposes within that model.
Consumers should be able to load once and receive a complete, validated
configuration with inspectable provenance for its final values, including
defaults and recorded application-derived values. Configuration and metadata
together must distinguish explicit zero, false, empty, and null input from
omission; ordinary Go values alone need not encode those distinctions.
Application-owned resolution belongs within the recorded loading lifecycle, so
consumers need no subsequent generic merging, defaulting, or provenance
reconstruction. This does not transfer domain execution or runtime resource
management into ConfigKit.
1. **Reduce application infrastructure:** Can applications delete substantial 1. **Reduce application infrastructure:** Can applications delete substantial
configuration infrastructure while retaining clear ownership of their domain configuration infrastructure while retaining clear ownership of their domain
rules? rules?
@@ -41,8 +58,11 @@ Do not promote incidental implementation choices into architectural invariants.
configuration model, with only the flexibility needed to express genuine configuration model, with only the flexibility needed to express genuine
application requirements? application requirements?
Both criteria must hold. ConfigKit should not preserve divergent consumer Both criteria must hold. Minimize consumer loading and processing code subject
conventions merely for compatibility, nor impose restrictions that prevent useful to the consistency, presence, and provenance principles above, rather than
trading those principles for consumer-specific shortcuts. ConfigKit should not
preserve divergent consumer conventions merely for compatibility, nor impose
restrictions that prevent useful
domain information from being expressed. Prefer a canonical representation using domain information from being expressed. Prefer a canonical representation using
existing primitives before adding another mechanism. Extend the model when a existing primitives before adding another mechanism. Extend the model when a
concrete requirement cannot be accommodated cleanly, not simply because a concrete requirement cannot be accommodated cleanly, not simply because a
@@ -88,6 +108,22 @@ must never run implicitly as part of configuration loading.
Remote data can enter through application-supplied input without introducing a Remote data can enter through application-supplied input without introducing a
network provider into the library. network provider into the library.
## Loading Lifecycle
Configuration loading is a synchronous, complete-on-return operation. A successful
load provides the final configuration and inspection metadata after the configured
processing and validation phases complete. A failed load reports diagnostics and
returns no usable partial result. The normal application pattern is one load near
startup; applications may explicitly initiate additional loads.
ConfigKit releases its library-owned loading handles before returning and remains
inactive until the application explicitly invokes it again. It performs no
background watching, refresh, or mutation of published results. Caller-owned
resources and application callback effects remain the caller's responsibility.
The [configuration contract](../config.md#completion-and-resource-lifetime) owns
the precise completion and ownership behavior. The proposed decision record is
[ADR 0008](../adr/0008-complete-on-return-loading.md).
## Public API And Dependency Direction ## Public API And Dependency Direction
Keep one small, coherent public surface for the supported lifecycle. Simple Keep one small, coherent public surface for the supported lifecycle. Simple
@@ -116,18 +152,41 @@ framework on this library.
## Configuration Integrity ## Configuration Integrity
Composition and intentional overlay are distinct operations. Composition must Within a configuration tier, independent contributions use strict composition:
not silently become last-wins override through file ordering. The decision and shared mappings combine recursively, while overlapping non-mapping definitions
alternatives are recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md). are conflicts, including equal values. Between precedence tiers, intentional
overlay combines mappings recursively, retains omitted lower-priority children,
and replaces other present values according to the existing overlay rules.
Tier precedence is explicit; traversal order within a tier establishes stable
reporting, not permission to override. The distinction and its rationale are
recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md).
The main document and its imports form a strictly composed tier. Explicit
underlay groups compose strictly within each group, then combine recursively
from lowest to highest beneath the main tier. Source acquisition kind does not
determine its precedence role. Input-specific constraints, including CLI path
rules and opt-in version ownership, remain part of the configuration contract. Consistent combination
semantics do not require identical acquisition or lifetime behavior from every
source kind.
After source precedence is complete, built-in sibling `extends` relationships
resolve structurally in the same parser-independent representation, using the
existing recursive overlay rules. This adds neither an acquired source nor a
consumer registration mechanism. Enclosing relationships precede nested
destination-local resolution; typed defaults, application resolution, and
secret lookup retain their existing lifecycle roles. The [configuration
contract](../config.md#sibling-inheritance) owns the exact syntax and presence
rules, and [ADR 0009](../adr/0009-sibling-inheritance.md) records the rationale.
ConfigKit must define supported YAML semantics explicitly rather than inherit ConfigKit must define supported YAML semantics explicitly rather than inherit
parser defaults. Processing order and diagnostics must be deterministic for the parser defaults. Processing order and diagnostics must be deterministic for the
same inputs and options. Invalid input must not produce a successful usable same inputs and options. Invalid input must not produce a successful usable
configuration. Do not run application hooks on invalid typed configuration. configuration. Do not run application hooks on invalid typed configuration.
User-controlled precedence is application defaults, composed base, selected User-controlled precedence is application defaults, ordered underlays, composed
overlay, then explicitly supplied CLI overrides. Applications own flag parsing; main input, selected overlay, then explicitly supplied CLI overrides.
ConfigKit owns overlay semantics and provenance. Domain resolution follows these Applications own flag parsing; ConfigKit owns overlay semantics and provenance.
Domain resolution follows these
inputs and must remain recorded, not become a hidden override channel. inputs and must remain recorded, not become a hidden override channel.
Preserve node-based composition until the effective input is ready for typed Preserve node-based composition until the effective input is ready for typed

53
docs/releases/v0.5.0.md Normal file
View File

@@ -0,0 +1,53 @@
# ConfigKit v0.5.0
This minor release adds source-backed layered configuration and built-in sibling
inheritance, so applications can load one complete configuration while retaining
presence, source context, and explainable derivation.
## Summary
- `LoadLayered` combines ordered underlay groups beneath the main input, selected
overlay, and explicit CLI overrides. Inputs within a group still compose
strictly; precedence tiers combine mappings recursively. The main input
remains the owner of the optional schema-version check.
- `NewFilesystem(...).File(...)` accepts caller-owned `fs.FS` sources, and
`Mapping(...)` snapshots supported built-in values as a labeled source.
`Result.SourceContext` and `Resolution.SourceContext` expose passive resource
contexts for values from file-backed sources.
- `extends` now resolves a mapping against one named sibling after source
precedence and before typed decoding. Inherited values count as explicit
input. Origins, history, explanations, and redacted rendering retain the
source and the structural path by which a value reached its destination.
Review corrections preserve immutable nested-history evidence and bounded
expansion work.
See the [configuration contract](../config.md#layered-loading),
[inheritance rules](../config.md#sibling-inheritance), and
[consumer examples](../consumers/README.md) for current behavior.
## Compatibility and upgrade
This is an intentional pre-v1 incompatible minor release for configurations
that use `extends` as ordinary data. That key is now reserved wherever an input
mapping occurs; it is consumed before decoding. Rename any such YAML key and
corresponding Go field, then rebuild. No collection registration or opt-in flag
is available. Configurations without `extends` retain their existing meaning.
Applications adopting the new features may move source-backed defaults into
explicit `Underlay` groups, use `SourceContext` for relative resources, or
replace application-owned sibling merging with one `extends` relationship per
mapping. Adoption is optional. The maintained
[named-preset example](../consumers/README.md#named-sibling-inheritance)
shows inherited `false` and destination-local nested resolution.
Select the new module version:
```sh
go get gitea.maximumdirect.net/eric/ConfigKit@v0.5.0
```
The minimum Go line remains 1.25. Linux amd64 and arm64 remain the
release-validated targets; macOS remains best-effort. The YAML dependency is
unchanged at `go.yaml.in/yaml/v4 v4.0.0-rc.6`.
[COPYRIGHT](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/COPYRIGHT) | [LICENSE](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/LICENSE) | [ATTRIBUTION](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.5.0/docs/oss/attribution.md)

View File

@@ -11,11 +11,13 @@ General environment-variable configuration overlays are not supported. Typed sec
references and explicit directory bootstrap are the environment convention. Literal references and explicit directory bootstrap are the environment convention. Literal
credential configuration and implicit secret-directory discovery are unsupported. credential configuration and implicit secret-directory discovery are unsupported.
The initial release deliberately excludes: The following remain deliberately excluded:
- automatic `conf.d` discovery; - automatic `conf.d` discovery;
- remote imports, arbitrary providers, and additional configuration formats; - remote imports, arbitrary providers, and additional configuration formats;
- custom merge strategies, sequence merging, reset operators, and overlay inheritance; - custom merge strategies, sequence merging, reset operators, multiple-parent
inheritance, and cross-scope references;
- named overlay inheritance;
- parser-specific or custom structural-decoding extension APIs; - parser-specific or custom structural-decoding extension APIs;
- generic dynamic-subtree conversion and module-registry APIs; - generic dynamic-subtree conversion and module-registry APIs;
- automatic schema migrations; - automatic schema migrations;
@@ -26,3 +28,7 @@ The initial release deliberately excludes:
Applications may build their own validate, effective-value, source, bootstrap, and Applications may build their own validate, effective-value, source, bootstrap, and
explanation commands around the library. Broadening these boundaries requires explanation commands around the library. Broadening these boundaries requires
concrete consumer evidence, compatibility review, and an explicit design change. concrete consumer evidence, compatibility review, and an explicit design change.
Built-in sibling inheritance using `extends` is implemented as described in the
[configuration contract](../config.md#sibling-inheritance). It is distinct from
inheritance between selected overlay files, which remains excluded above.

View File

@@ -52,6 +52,22 @@ func Example_seriatim() {
// Output: visitor true the event occurred <nil> <nil> // Output: visitor true the event occurred <nil> <nil>
} }
func Example_inheritedPresets() {
result, err := consumers.LoadPresets(configkit.File(fixture("inheritance", "presets.yml")))
if err != nil {
fmt.Println(err)
return
}
path := configkit.Keys("presets", "custom", "enabled")
info, _ := result.Metadata().Lookup(path)
custom := result.Config.Presets["custom"]
fmt.Println(custom.Enabled, result.Metadata().IsExplicit(path), custom.Settings["secondary"]["width"])
fmt.Println(info.Origin.Locations[0].SourceID, info.Origin.Inheritance[0].From)
// Output:
// false true 20
// source-1 /presets/base/enabled
}
func Example_weatherStartup() { func Example_weatherStartup() {
const blockedName = "CONFIGKIT_WEATHER_EXAMPLE_BLOCKED" const blockedName = "CONFIGKIT_WEATHER_EXAMPLE_BLOCKED"
const tokenName = "CONFIGKIT_WEATHER_EXAMPLE_TOKEN" const tokenName = "CONFIGKIT_WEATHER_EXAMPLE_TOKEN"

View File

@@ -0,0 +1,20 @@
package consumers
import configkit "gitea.maximumdirect.net/eric/ConfigKit"
// PresetCatalog is ordinary application data; no inheritance fields or
// registration hooks are required in the typed configuration.
type PresetCatalog struct {
Presets map[string]Preset `yaml:"presets"`
}
type Preset struct {
Enabled bool `yaml:"enabled"`
Settings map[string]map[string]int `yaml:"settings"`
}
// LoadPresets performs one load; callers may inspect inherited presence and
// source origins on the returned result before choosing a preset to execute.
func LoadPresets(source configkit.Source) (*configkit.Result[PresetCatalog], error) {
return configkit.Load[PresetCatalog](source)
}

View File

@@ -0,0 +1,35 @@
package consumers_test
import (
"testing"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/ConfigKit/examples/consumers"
)
func TestPresetCatalogOneLoadAndDerivation(t *testing.T) {
result, err := consumers.LoadPresets(configkit.File(fixture("inheritance", "presets.yml")))
if err != nil {
t.Fatal(err)
}
if result.Config.Presets["base"].Settings["secondary"]["width"] != 10 ||
result.Config.Presets["custom"].Settings["secondary"]["width"] != 20 ||
result.Config.Presets["custom"].Enabled {
t.Fatalf("expanded presets = %#v", result.Config.Presets)
}
enabled := configkit.Keys("presets", "custom", "enabled")
if !result.Metadata().IsExplicit(enabled) {
t.Fatal("inherited false lost explicit presence")
}
info, found := result.Metadata().Lookup(enabled)
if !found || info.Origin.Locations[0].SourceID != "source-1" ||
len(info.Origin.Inheritance) != 1 || info.Origin.Inheritance[0].From.String() != "/presets/base/enabled" {
t.Fatalf("origin and derivation = %#v, %v", info, found)
}
nested := configkit.Keys("presets", "custom", "settings", "secondary", "width")
info, found = result.Metadata().Lookup(nested)
if !found || len(info.Origin.Inheritance) != 1 ||
info.Origin.Inheritance[0].From.String() != "/presets/custom/settings/primary/width" {
t.Fatalf("destination-local derivation = %#v, %v", info, found)
}
}

View File

@@ -0,0 +1,95 @@
package consumers
import (
"embed"
"errors"
"io/fs"
"path"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
//go:embed testdata/native-source testdata/layered-bundle
var bundledFiles embed.FS
// BundledConfig is a small configuration loaded directly from an embedded file.
type BundledConfig struct {
Name string `yaml:"name"`
Enabled bool `yaml:"enabled"`
}
// LoadBundled demonstrates a native filesystem source without layered options.
func LoadBundled() (*configkit.Result[BundledConfig], error) {
files, err := fs.Sub(bundledFiles, "testdata/native-source")
if err != nil {
return nil, err
}
return configkit.Load[BundledConfig](configkit.NewFilesystem("bundled", files).File("config.yml"))
}
// CatalogConfig keeps resource names and domain rules in the application.
type CatalogConfig struct {
Mode string `yaml:"mode"`
Enabled bool `yaml:"enabled"`
Workers int `yaml:"workers"`
Templates map[string]string `yaml:"templates"`
Labels map[string]string `yaml:"labels"`
}
// LoadCatalog combines two underlay tiers beneath an independently located
// host main file. ConfigKit assembles and validates the complete result once.
func LoadCatalog(mainPath string) (*configkit.Result[CatalogConfig], error) {
files, err := fs.Sub(bundledFiles, "testdata/layered-bundle")
if err != nil {
return nil, err
}
bundle := configkit.NewFilesystem("catalog bundle", files)
return configkit.LoadLayered(configkit.Input{Source: configkit.File(mainPath)},
configkit.LayeredOptions[CatalogConfig]{
Underlays: []configkit.Underlay{
{Inputs: []configkit.Input{{Source: bundle.File("defaults/config.yml"), Imports: true}}},
{Inputs: []configkit.Input{{Source: configkit.Mapping("local policy", map[string]any{
"workers": 2,
"labels": map[string]any{"tenant": "example"},
})}}},
},
Definition: configkit.Definition[CatalogConfig]{
Defaults: func() CatalogConfig { return CatalogConfig{Mode: "demo"} },
Validate: validateCatalog,
},
})
}
func validateCatalog(configuration *CatalogConfig, validation *configkit.Validation) {
if configuration.Mode != "demo" && configuration.Mode != "production" {
validation.Error(configkit.Keys("mode"), "must be demo or production")
}
if configuration.Workers < 0 {
validation.Error(configkit.Keys("workers"), "must be nonnegative")
}
if name := configuration.Templates["welcome"]; name == "." || !fs.ValidPath(name) {
validation.Error(configkit.Keys("templates", "welcome"), "must be a relative resource name")
}
}
// ReadWelcome follows the effective template field's one declaring source to
// its retained resource context. Opening and interpreting the resource remain
// application work; this does not reopen configuration or change metadata.
func ReadWelcome(result *configkit.Result[CatalogConfig]) (string, error) {
if result == nil {
return "", errors.New("catalog result is nil")
}
info, found := result.Metadata().Lookup(configkit.Keys("templates", "welcome"))
if !found || len(info.Origin.Locations) != 1 {
return "", errors.New("welcome template has no single declaring source")
}
context, found := result.SourceContext(info.Origin.Locations[0].SourceID)
if !found {
return "", errors.New("welcome template has no resource context")
}
data, err := fs.ReadFile(context.FS, path.Join(context.BaseDir, result.Config.Templates["welcome"]))
if err != nil {
return "", err
}
return string(data), nil
}

View File

@@ -0,0 +1,71 @@
package consumers_test
import (
"errors"
"fmt"
"strings"
"testing"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
"gitea.maximumdirect.net/eric/ConfigKit/examples/consumers"
)
func Example_nativeFilesystem() {
result, err := consumers.LoadBundled()
if err != nil {
fmt.Println(err)
return
}
fmt.Println(result.Config.Name, result.Config.Enabled)
// Output: embedded false
}
func Example_layeredCatalog() {
result, err := consumers.LoadCatalog(fixture("layered-main", "config.yml"))
if err != nil {
fmt.Println(err)
return
}
welcome, err := consumers.ReadWelcome(result)
fmt.Println(result.Config.Mode, result.Config.Enabled, result.Config.Workers,
result.Config.Labels["origin"], result.Config.Labels["tenant"], strings.TrimSpace(welcome), err)
// Output: production false 0 bundle example Welcome from the bundle. <nil>
}
func TestCatalogRetainsPresenceAndSourceContext(t *testing.T) {
result, err := consumers.LoadCatalog(fixture("layered-main", "config.yml"))
if err != nil {
t.Fatal(err)
}
for _, key := range []string{"enabled", "workers"} {
info, found := result.Metadata().Lookup(configkit.Keys(key))
if !found || !info.Explicit || info.Origin.Layer != configkit.LayerBase {
t.Fatalf("%s presence = %#v, %v", key, info, found)
}
}
info, found := result.Metadata().Lookup(configkit.Keys("templates", "welcome"))
if !found || info.Origin.Layer != configkit.LayerUnderlay || len(info.Origin.Locations) != 1 {
t.Fatalf("welcome origin = %#v, %v", info, found)
}
context, found := result.SourceContext(info.Origin.Locations[0].SourceID)
if !found || context.BaseDir != "defaults" {
t.Fatalf("welcome context = %#v, %v", context, found)
}
welcome, err := consumers.ReadWelcome(result)
if err != nil || strings.TrimSpace(welcome) != "Welcome from the bundle." {
t.Fatalf("welcome = %q, %v", welcome, err)
}
if result.Config.Labels["tenant"] != "example" || result.Config.Labels["origin"] != "bundle" {
t.Fatalf("recursive underlay labels = %#v", result.Config.Labels)
}
}
func TestCatalogValidationRejectsInvalidMode(t *testing.T) {
_, err := consumers.LoadCatalog(fixture("layered-main", "invalid.yml"))
var loadError *configkit.LoadError
if !errors.As(err, &loadError) || len(loadError.Diagnostics()) == 0 ||
loadError.Diagnostics()[0].Code != configkit.CodeValidateError ||
loadError.Diagnostics()[0].Path.String() != "/mode" {
t.Fatalf("invalid mode error = %v", err)
}
}

View File

@@ -0,0 +1,10 @@
presets:
base:
enabled: false
settings:
primary: {width: 10}
secondary: {extends: primary}
custom:
extends: base
settings:
primary: {width: 20}

View File

@@ -0,0 +1,7 @@
configkit:
imports:
- path: ../shared.yml
enabled: true
workers: 4
templates:
welcome: messages/welcome.txt

View File

@@ -0,0 +1 @@
Welcome from the bundle.

View File

@@ -0,0 +1,2 @@
labels:
origin: bundle

View File

@@ -0,0 +1,3 @@
mode: production
enabled: false
workers: 0

View File

@@ -0,0 +1 @@
mode: unsupported

View File

@@ -0,0 +1,2 @@
name: embedded
enabled: false

View File

@@ -31,6 +31,9 @@ func TestFacadeValuesRemainRootOwned(t *testing.T) {
reflect.TypeFor[configkit.Resolution](), reflect.TypeFor[configkit.Validation](), reflect.TypeFor[configkit.Resolution](), reflect.TypeFor[configkit.Validation](),
reflect.TypeFor[configkit.Version](), reflect.TypeFor[configkit.VersionRule](), reflect.TypeFor[configkit.Version](), reflect.TypeFor[configkit.VersionRule](),
reflect.TypeFor[configkit.FilesystemContext](), reflect.TypeFor[configkit.Override](), reflect.TypeFor[configkit.FilesystemContext](), reflect.TypeFor[configkit.Override](),
reflect.TypeFor[configkit.Filesystem](), reflect.TypeFor[configkit.ResourceContext](),
reflect.TypeFor[configkit.Input](), reflect.TypeFor[configkit.Underlay](),
reflect.TypeFor[configkit.LayeredOptions[struct{}]](),
reflect.TypeFor[configkit.Definition[struct{}]](), reflect.TypeFor[configkit.Options[struct{}]](), reflect.TypeFor[configkit.Definition[struct{}]](), reflect.TypeFor[configkit.Options[struct{}]](),
reflect.TypeFor[configkit.Result[struct{}]](), reflect.TypeFor[configkit.Result[struct{}]](),
reflect.TypeFor[configkit.ValueSnapshot](), reflect.TypeFor[configkit.ValueChange](), reflect.TypeFor[configkit.ValueSnapshot](), reflect.TypeFor[configkit.ValueChange](),

View File

@@ -26,6 +26,7 @@ func TestFacadeInactiveContextsPreserveClosedFailure(t *testing.T) {
{"IsExplicit", func() { state.resolution.IsExplicit(path) }}, {"IsExplicit", func() { state.resolution.IsExplicit(path) }},
{"InputInfo", func() { state.resolution.InputInfo(path) }}, {"InputInfo", func() { state.resolution.InputInfo(path) }},
{"Source", func() { state.resolution.Source("source") }}, {"Source", func() { state.resolution.Source("source") }},
{"SourceContext", func() { state.resolution.SourceContext("source") }},
{"Record", func() { state.resolution.Record(path, configkit.ChangeNormalization, "reason") }}, {"Record", func() { state.resolution.Record(path, configkit.ChangeNormalization, "reason") }},
{"Resolution.Error", func() { state.resolution.Error(path, "message") }}, {"Resolution.Error", func() { state.resolution.Error(path, "message") }},
{"Resolution.Warning", func() { state.resolution.Warning(path, "message") }}, {"Resolution.Warning", func() { state.resolution.Warning(path, "message") }},

164
facade_equivalence_test.go Normal file
View File

@@ -0,0 +1,164 @@
package configkit_test
import (
"bytes"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
type equivalentConfig struct {
Version uint64 `yaml:"version"`
Name string `yaml:"name"`
Enabled bool `yaml:"enabled"`
Count int `yaml:"count"`
Items []string `yaml:"items"`
Optional *string `yaml:"optional"`
}
func TestEquivalentSourceFormsPreserveInputSemantics(t *testing.T) {
const yaml = "version: 1\nname: retained\nenabled: false\ncount: 0\nitems: []\noptional: null\n"
hostPath := filepath.Join(t.TempDir(), "main.yml")
if err := os.WriteFile(hostPath, []byte(yaml), 0o600); err != nil {
t.Fatal(err)
}
namespace := configkit.NewFilesystem("bundle", fstest.MapFS{"main.yml": &fstest.MapFile{Data: []byte(yaml)}})
context := &configkit.FilesystemContext{Root: ".", BaseDir: ".", FS: &namespace}
for _, test := range []struct {
name string
source configkit.Source
context *configkit.FilesystemContext
kind configkit.SourceKind
}{
{"host", configkit.File(hostPath), nil, configkit.SourceKindFile},
{"native", namespace.File("main.yml"), nil, configkit.SourceKindFilesystem},
{"bytes", configkit.Bytes("memory", []byte(yaml)), context, configkit.SourceKindBytes},
{"reader", configkit.Reader("stream", strings.NewReader(yaml)), context, configkit.SourceKindReader},
{"mapping", configkit.Mapping("programmatic", map[string]any{
"version": uint64(1), "name": "retained", "enabled": false, "count": 0,
"items": []any{}, "optional": nil,
}), context, configkit.SourceKindMapping},
} {
t.Run(test.name, func(t *testing.T) {
checks := 0
result, err := configkit.LoadLayered(configkit.Input{Source: test.source, Filesystem: test.context}, configkit.LayeredOptions[equivalentConfig]{
Definition: configkit.Definition[equivalentConfig]{Version: &configkit.VersionRule{
Path: configkit.Keys("version"), Check: func(version configkit.Version, validation *configkit.Validation) error {
checks++
if !version.Present || version.Text != "1" {
t.Errorf("version = %#v", version)
}
return nil
},
}},
})
if err != nil {
t.Fatal(err)
}
want := equivalentConfig{Version: 1, Name: "retained", Items: []string{}}
if !reflect.DeepEqual(result.Config, want) || checks != 1 {
t.Fatalf("config = %#v, checks %d", result.Config, checks)
}
for _, field := range []string{"version", "name", "enabled", "count", "items", "optional"} {
info, ok := result.Metadata().Lookup(configkit.Keys(field))
if !ok || !info.Explicit || info.Origin.Layer != configkit.LayerBase || info.Null != (field == "optional") {
t.Errorf("%s input info = %#v, %v", field, info, ok)
}
}
sources := result.Sources()
if len(sources) != 1 || sources[0].Kind != test.kind || sources[0].ID != "source-1" || sources[0].UnderlayIndex != 0 {
t.Fatalf("sources = %#v", sources)
}
resource, ok := result.SourceContext("source-1")
if !ok || resource.BaseDir != "." || resource.FS == nil {
t.Fatalf("source context = %#v, %v", resource, ok)
}
})
}
}
func TestRepeatedImportsAcrossTiersAndReaderConsumption(t *testing.T) {
namespace := configkit.NewFilesystem("bundle", fstest.MapFS{
"left.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: shared.yml}]}\n")},
"right.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: shared.yml}]}\n")},
"shared.yml": &fstest.MapFile{Data: []byte("name: retained\n")},
})
type config struct {
Name string `yaml:"name"`
}
left := configkit.Input{Source: namespace.File("left.yml"), Imports: true}
right := configkit.Input{Source: namespace.File("right.yml"), Imports: true}
_, err := configkit.LoadLayered(configkit.Input{Source: configkit.NoSource()}, configkit.LayeredOptions[config]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{left, right}}},
})
if got := layeredErrorCode(t, err); got != configkit.CodeImportRepeated {
t.Fatalf("same-tier repeated import = %s", got)
}
result, err := configkit.LoadLayered(configkit.Input{Source: configkit.NoSource()}, configkit.LayeredOptions[config]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{left}}, {Inputs: []configkit.Input{right}}},
})
if err != nil || result.Config.Name != "retained" || len(result.Sources()) != 4 {
t.Fatalf("cross-tier repeated import = %#v, %v", result, err)
}
reader := configkit.Reader("single-use", strings.NewReader("name: retained\n"))
_, err = configkit.LoadLayered(configkit.Input{Source: configkit.NoSource()}, configkit.LayeredOptions[config]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{Source: reader}}}, {Inputs: []configkit.Input{{Source: reader}}}},
})
if got := layeredErrorCode(t, err); got != configkit.CodeSourceReaderUsed {
t.Fatalf("reader re-consumption = %s", got)
}
}
func TestReplacedSecretRemainsAbsentFromSafeOutput(t *testing.T) {
type config struct {
Secret string `yaml:"secret"`
}
result, err := configkit.LoadLayered(configkit.Input{Source: configkit.Bytes("main", []byte("secret: new-secret-canary\n"))}, configkit.LayeredOptions[config]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{Source: configkit.Mapping("lower", map[string]any{"secret": "old-secret-canary"})}}}},
Definition: configkit.Definition[config]{Secret: func(path configkit.Path) bool { return path.String() == "/secret" }},
})
if err != nil || result.Config.Secret != "new-secret-canary" {
t.Fatalf("secret load = %#v, %v", result, err)
}
var effective, history bytes.Buffer
if err := result.WriteEffective(&effective); err != nil {
t.Fatal(err)
}
if err := result.WriteExplanation(&history, configkit.Keys("secret")); err != nil {
t.Fatal(err)
}
if strings.Contains(effective.String()+history.String(), "old-secret-canary") || strings.Contains(effective.String()+history.String(), "new-secret-canary") {
t.Fatalf("secret output leaked: %q %q", effective.String(), history.String())
}
_, err = configkit.LoadLayered(configkit.Input{Source: configkit.Bytes("main", []byte("secret: new-secret-canary\n"))}, configkit.LayeredOptions[config]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{Source: configkit.Mapping("lower", map[string]any{"secret": "old-secret-canary"})}}}},
Definition: configkit.Definition[config]{Validate: func(config *config, validation *configkit.Validation) {
validation.Error(configkit.Keys("secret"), "invalid secret")
}},
})
loadError, ok := err.(*configkit.LoadError)
if !ok {
t.Fatalf("validation error = %#v", err)
}
var diagnostics bytes.Buffer
if err := configkit.WriteDiagnostics(&diagnostics, loadError.Diagnostics()); err != nil {
t.Fatal(err)
}
if strings.Contains(diagnostics.String(), "secret-canary") {
t.Fatalf("diagnostic leaked secret: %q", diagnostics.String())
}
}
func layeredErrorCode(t *testing.T, err error) configkit.Code {
t.Helper()
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 {
t.Fatalf("load error = %#v", err)
}
return loadError.Diagnostics()[0].Code
}

142
facade_filesystem_test.go Normal file
View File

@@ -0,0 +1,142 @@
package configkit_test
import (
"embed"
"io/fs"
"reflect"
"strconv"
"strings"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
//go:embed testdata/filesystem
var filesystemFixtures embed.FS
type filesystemFacadeConfig struct {
Items map[string]string `yaml:"items"`
}
func TestFacadeFilesystemSourceImportsOverlayAndReuse(t *testing.T) {
fixture, err := fs.Sub(filesystemFixtures, "testdata/filesystem")
if err != nil {
t.Fatal(err)
}
namespace := configkit.NewFilesystem("embedded", fixture)
copyOfNamespace := namespace
source := namespace.File("config/main.yml")
for range 2 {
result, err := configkit.Load(source, configkit.Options[filesystemFacadeConfig]{
Imports: true, Overlay: "testing", Filesystem: &configkit.FilesystemContext{FS: &copyOfNamespace},
})
if err != nil {
t.Fatal(err)
}
if want := (map[string]string{"main": "selected", "shared": "default"}); !reflect.DeepEqual(result.Config.Items, want) {
t.Fatalf("items = %#v, want %#v", result.Config.Items, want)
}
sources := result.Sources()
if len(sources) != 3 {
t.Fatalf("sources = %#v", sources)
}
for index, want := range []string{"config/main.yml", "shared.yml", "overlays/testing.yml"} {
if sources[index].ID != "source-"+strconv.Itoa(index+1) || sources[index].Path != want || sources[index].Kind != configkit.SourceKindFilesystem {
t.Errorf("source %d = %#v", index, sources[index])
}
if sources[index].Label != "embedded/"+want {
t.Errorf("source label = %q", sources[index].Label)
}
}
if sources[1].ImportedFrom == nil || sources[1].ImportedFrom.SourceID != sources[0].ID || sources[1].ImportedFrom.Line == 0 {
t.Fatalf("import location = %#v", sources[1].ImportedFrom)
}
}
}
func TestFacadeFilesystemContextsAndIdentity(t *testing.T) {
first := configkit.NewFilesystem("same", fstest.MapFS{
"root/config/main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: ../shared.yml}]}\n")},
"root/shared.yml": &fstest.MapFile{Data: []byte("items: {value: first}\n")},
"root/overlays/testing.yml": &fstest.MapFile{Data: []byte("items: {overlay: first}\n")},
})
second := configkit.NewFilesystem("same", fstest.MapFS{
"root/config/main.yml": &fstest.MapFile{Data: []byte("items: {value: second}\n")},
})
context := &configkit.FilesystemContext{Root: "root", FS: &first}
result, err := configkit.Load(first.File("root/config/main.yml"), configkit.Options[filesystemFacadeConfig]{Imports: true, Overlay: "testing", Filesystem: context})
if err != nil || result.Config.Items["value"] != "first" || result.Config.Items["overlay"] != "first" {
t.Fatalf("contained load = %#v, %v", result, err)
}
result, err = configkit.Load[filesystemFacadeConfig](second.File("root/config/main.yml"))
if err != nil || result.Config.Items["value"] != "second" {
t.Fatalf("same-label independent namespace = %#v, %v", result, err)
}
checkCode(t, second.File("root/config/main.yml"), configkit.Options[filesystemFacadeConfig]{Filesystem: context}, configkit.CodeOptionsInvalid)
checkCode(t, first.File("root/config/main.yml"), configkit.Options[filesystemFacadeConfig]{Filesystem: &configkit.FilesystemContext{Root: "root/config", FS: &first}, Imports: true}, configkit.CodeImportOutsideRoot)
checkCode(t, configkit.File("testdata/filesystem/config/main.yml"), configkit.Options[filesystemFacadeConfig]{Filesystem: context}, configkit.CodeOptionsInvalid)
for _, input := range []struct {
name string
source configkit.Source
kind configkit.SourceKind
}{
{"bytes", configkit.Bytes("memory", []byte("configkit: {imports: [{path: shared.yml}]}\n")), configkit.SourceKindBytes},
{"reader", configkit.Reader("stream", strings.NewReader("configkit: {imports: [{path: shared.yml}]}\n")), configkit.SourceKindReader},
} {
t.Run(input.name, func(t *testing.T) {
result, err := configkit.Load(input.source, configkit.Options[filesystemFacadeConfig]{Imports: true, Filesystem: &configkit.FilesystemContext{Root: "root", BaseDir: "root", FS: &first}})
if err != nil || result.Config.Items["value"] != "first" {
t.Fatalf("synthetic import = %#v, %v", result, err)
}
if got := result.Sources(); len(got) != 2 || got[0].Kind != input.kind || got[0].Path != "" || got[1].Kind != configkit.SourceKindFilesystem {
t.Fatalf("sources = %#v", got)
}
})
}
checkCode(t, configkit.Bytes("memory", []byte("items: {}\n")), configkit.Options[filesystemFacadeConfig]{Filesystem: &configkit.FilesystemContext{Root: "root", BaseDir: ".", FS: &first}}, configkit.CodeImportOutsideRoot)
result, err = configkit.Load(configkit.Bytes("memory", []byte("items: {value: main}\n")), configkit.Options[filesystemFacadeConfig]{
Overlay: "testing", Filesystem: &configkit.FilesystemContext{Root: "root", BaseDir: "root", FS: &first},
})
if err != nil || result.Config.Items["value"] != "main" || result.Config.Items["overlay"] != "first" {
t.Fatalf("synthetic selected overlay = %#v, %v", result, err)
}
}
func TestFacadeSourceKindsIncludeHostAndCLI(t *testing.T) {
result, err := configkit.Load(configkit.File("testdata/filesystem/shared.yml"), configkit.Options[filesystemFacadeConfig]{
Overrides: []configkit.Override{{Path: configkit.Keys("items", "shared"), Value: "changed", Flag: "--shared"}},
})
if err != nil {
t.Fatal(err)
}
sources := result.Sources()
if len(sources) != 2 || sources[0].Kind != configkit.SourceKindFile || sources[0].Path == "" || sources[1].Kind != configkit.SourceKindCLI || sources[1].Path != "" {
t.Fatalf("host and CLI sources = %#v", sources)
}
}
func TestFacadeFilesystemConstructorDiagnostics(t *testing.T) {
var nilFilesystem fs.FS
var zero configkit.Filesystem
for _, source := range []configkit.Source{
zero.File("main.yml"),
configkit.NewFilesystem("", fstest.MapFS{}).File("main.yml"),
configkit.NewFilesystem("bad\nlabel", fstest.MapFS{}).File("main.yml"),
configkit.NewFilesystem("good", nilFilesystem).File("main.yml"),
configkit.NewFilesystem("good", fstest.MapFS{}).File("."),
configkit.NewFilesystem("good", fstest.MapFS{}).File("../outside.yml"),
} {
checkCode(t, source, configkit.Options[filesystemFacadeConfig]{}, configkit.CodeSourceInvalid)
}
}
func checkCode(t *testing.T, source configkit.Source, options configkit.Options[filesystemFacadeConfig], want configkit.Code) {
t.Helper()
_, err := configkit.Load(source, options)
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != want {
t.Fatalf("error = %#v, want %s", err, want)
}
}

284
facade_inheritance_test.go Normal file
View File

@@ -0,0 +1,284 @@
package configkit_test
import (
"bytes"
"io/fs"
"reflect"
"strings"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
type inheritedEntry struct {
Enabled bool `yaml:"enabled"`
Count int `yaml:"count"`
Resource string `yaml:"resource"`
Settings map[string]map[string]string `yaml:"settings"`
}
type inheritedConfiguration struct {
Version string `yaml:"version"`
Entries map[string]inheritedEntry `yaml:"entries"`
}
func TestFacadeInheritancePreservesPresenceOriginsAndHistory(t *testing.T) {
input := []byte(`entries:
parent:
enabled: false
count: 0
settings:
primary: {value: original}
secondary: {extends: primary}
child:
extends: parent
settings:
primary: {value: custom}
`)
defaultRan := false
result, err := configkit.Load(configkit.Bytes("config.yml", input), configkit.Options[inheritedConfiguration]{Definition: configkit.Definition[inheritedConfiguration]{
Resolve: func(config *inheritedConfiguration, resolution *configkit.Resolution) error {
if !resolution.IsExplicit(configkit.Keys("entries", "child", "enabled")) {
t.Error("inherited false is not explicit")
}
info, found := resolution.InputInfo(configkit.Keys("entries", "child", "enabled"))
if !found || len(info.Origin.Inheritance) != 1 {
t.Errorf("resolver inherited origin = %#v, %v", info, found)
}
child := config.Entries["child"]
configkit.Default(resolution, configkit.Keys("entries", "child", "enabled"), &child.Enabled, true, "use fallback")
defaultRan = child.Enabled
config.Entries["child"] = child
return nil
},
}})
if err != nil {
t.Fatal(err)
}
child := result.Config.Entries["child"]
if defaultRan || child.Enabled || child.Count != 0 || child.Settings["secondary"]["value"] != "custom" {
t.Fatalf("inherited values = %#v, default ran %v", child, defaultRan)
}
path := configkit.Keys("entries", "child", "settings", "secondary", "value")
info, found := result.Metadata().Lookup(path)
if !found || !info.Explicit || len(info.Origin.Inheritance) != 1 ||
info.Origin.Inheritance[0].From.String() != "/entries/child/settings/primary/value" ||
info.Origin.Inheritance[0].To.String() != path.String() {
t.Fatalf("nested destination origin = %#v, %v", info, found)
}
info.Origin.Inheritance[0].Locations[0].SourceID = "mutated"
info, _ = result.Metadata().Lookup(path)
if info.Origin.Inheritance[0].Locations[0].SourceID == "mutated" {
t.Fatal("public inheritance record aliases inspection state")
}
if _, found := result.Metadata().Lookup(configkit.Keys("entries", "child", "extends")); found {
t.Fatal("consumed directive remained effective")
}
control, found := result.Explain(configkit.Keys("entries", "child", "extends"))
if !found || control.Effective.Present || len(control.History) < 2 ||
control.History[len(control.History)-1].Origin.Operation != configkit.ChangeInheritance {
t.Fatalf("consumed directive = %#v, %v", control, found)
}
explanation, found := result.Explain(path)
if !found || explanation.Effective.YAML != "custom\n" {
t.Fatalf("nested explanation = %#v, %v", explanation, found)
}
for _, change := range explanation.History {
if change.Origin.Operation == configkit.ChangeDecoding {
t.Fatalf("inherited value attributed to decoding: %#v", explanation.History)
}
}
}
func TestFacadeInheritanceLayeredPrecedenceAndSourceContext(t *testing.T) {
filesystem := configkit.NewFilesystem("bundle", fstest.MapFS{
"config/main.yml": &fstest.MapFile{Data: []byte("entries: {child: {extends: parent}}\n")},
"shared/parent.yml": &fstest.MapFile{Data: []byte("entries: {parent: {resource: icon.txt, count: 1}}\n")},
"shared/flags.yml": &fstest.MapFile{Data: []byte("entries: {parent: {enabled: false}}\n")},
"shared/icon.txt": &fstest.MapFile{Data: []byte("parent-resource")},
"config/overlays/test.yml": &fstest.MapFile{Data: []byte("entries: {parent: {count: 2}}\n")},
})
result, err := configkit.LoadLayered(configkit.Input{Source: filesystem.File("config/main.yml")}, configkit.LayeredOptions[inheritedConfiguration]{
Underlays: []configkit.Underlay{
{Inputs: []configkit.Input{{Source: filesystem.File("shared/parent.yml")}}},
{Inputs: []configkit.Input{{Source: filesystem.File("shared/flags.yml")}}},
},
Overlay: "test", OverlayFilesystem: &configkit.FilesystemContext{Root: "config", BaseDir: "config", FS: &filesystem},
Overrides: []configkit.Override{{Path: configkit.Keys("entries", "child", "count"), Value: 3, Flag: "--count"}},
})
if err != nil {
t.Fatal(err)
}
child := result.Config.Entries["child"]
if child.Count != 3 || child.Resource != "icon.txt" {
t.Fatalf("layered inheritance = %#v", child)
}
info, found := result.Metadata().Lookup(configkit.Keys("entries", "child", "resource"))
if !found || len(info.Origin.Inheritance) != 1 || len(info.Origin.Locations) != 1 {
t.Fatalf("resource origin = %#v, %v", info, found)
}
context, found := result.SourceContext(info.Origin.Locations[0].SourceID)
if !found || context.BaseDir != "shared" || context.FS == nil {
t.Fatalf("source context = %#v, %v", context, found)
}
data, err := fs.ReadFile(context.FS, context.BaseDir+"/"+child.Resource)
if err != nil || string(data) != "parent-resource" {
t.Fatalf("source-relative resource = %q, %v", data, err)
}
count, found := result.Metadata().Lookup(configkit.Keys("entries", "child", "count"))
if !found || count.Origin.Layer != configkit.LayerCLI || len(count.Origin.Inheritance) != 0 {
t.Fatalf("CLI replacement origin = %#v, %v", count, found)
}
var rendered bytes.Buffer
if err := result.WriteEffective(&rendered); err != nil || strings.Contains(rendered.String(), "extends:") {
t.Fatalf("effective rendering = %q, %v", rendered.String(), err)
}
}
func TestFacadeInheritanceMappingMatchesBytes(t *testing.T) {
values := map[string]any{"entries": map[string]any{
"parent": map[string]any{"enabled": false, "count": 0},
"child": map[string]any{"extends": "parent"},
}}
mapped, err := configkit.Load[inheritedConfiguration](configkit.Mapping("mapping", values))
if err != nil {
t.Fatal(err)
}
fromYAML, err := configkit.Load[inheritedConfiguration](configkit.Bytes("config.yml", []byte("entries: {parent: {enabled: false, count: 0}, child: {extends: parent}}\n")))
if err != nil || !reflect.DeepEqual(mapped.Config, fromYAML.Config) {
t.Fatalf("mapping and bytes differ: %#v, %#v, %v", mapped, fromYAML, err)
}
}
func TestFacadeInheritanceProtectsVersionOwnership(t *testing.T) {
for _, test := range []struct {
name, input string
path configkit.Path
wantFailure bool
}{
{"inherited version", "entries: {parent: {count: 1}, child: {extends: parent}}\n", configkit.Keys("entries", "child", "count"), true},
{"overridden version", "entries: {parent: {count: 1}, child: {extends: parent, count: 1}}\n", configkit.Keys("entries", "child", "count"), true},
{"direct unchanged version", "entries: {parent: {enabled: false}, child: {extends: parent, count: 1}}\n", configkit.Keys("entries", "child", "count"), false},
{"consumed checked directive", "entries: {parent: {count: 1}, child: {extends: parent}}\n", configkit.Keys("entries", "child", "extends"), true},
{"independent top-level version", "version: v1\nentries: {parent: {count: 1}, child: {extends: parent}}\n", configkit.Keys("version"), false},
} {
t.Run(test.name, func(t *testing.T) {
checks := struct{ version, defaults, resolution, validation int }{}
result, err := configkit.Load(configkit.Bytes("version.yml", []byte(test.input)), configkit.Options[inheritedConfiguration]{Definition: configkit.Definition[inheritedConfiguration]{
Version: &configkit.VersionRule{Path: test.path, Check: func(configkit.Version, *configkit.Validation) error {
checks.version++
return nil
}},
Defaults: func() inheritedConfiguration { checks.defaults++; return inheritedConfiguration{} },
Resolve: func(*inheritedConfiguration, *configkit.Resolution) error { checks.resolution++; return nil },
Validate: func(*inheritedConfiguration, *configkit.Validation) { checks.validation++ },
}})
if test.wantFailure {
if result != nil || err == nil {
t.Fatalf("protected version = %#v, %v", result, err)
}
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != configkit.CodeInheritanceVersion ||
checks != (struct{ version, defaults, resolution, validation int }{version: 1}) {
t.Fatalf("version failure = %v, callbacks %#v", err, checks)
}
} else if err != nil || result == nil || checks != (struct{ version, defaults, resolution, validation int }{1, 1, 1, 1}) {
t.Fatalf("unchanged version = %#v, %v, callbacks %#v", result, err, checks)
}
})
}
}
func TestFacadeInheritanceFailureStopsLaterCallbacks(t *testing.T) {
checks := struct{ defaults, resolution, validation int }{}
result, err := configkit.Load(configkit.Bytes("invalid.yml", []byte("entries: {child: {extends: PRIVATE-REFERENCE-CANARY}}\n")), configkit.Options[inheritedConfiguration]{Definition: configkit.Definition[inheritedConfiguration]{
Defaults: func() inheritedConfiguration { checks.defaults++; return inheritedConfiguration{} },
Resolve: func(*inheritedConfiguration, *configkit.Resolution) error { checks.resolution++; return nil },
Validate: func(*inheritedConfiguration, *configkit.Validation) { checks.validation++ },
}})
if result != nil || err == nil || checks != (struct{ defaults, resolution, validation int }{}) || strings.Contains(err.Error(), "PRIVATE-REFERENCE-CANARY") {
t.Fatalf("invalid inheritance = %#v, %v, callbacks %#v", result, err, checks)
}
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != configkit.CodeInheritanceMissing {
t.Fatalf("missing reference diagnostics = %v", err)
}
}
func TestFacadeInheritanceResolvesAfterImportsAndCLIReferenceChange(t *testing.T) {
filesystem := configkit.NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: parent.yml}]}\nentries: {child: {extends: first}}\n")},
"parent.yml": &fstest.MapFile{Data: []byte("entries: {first: {count: 1}, second: {count: 2}}\n")},
})
result, err := configkit.Load(filesystem.File("main.yml"), configkit.Options[inheritedConfiguration]{
Imports: true, Overrides: []configkit.Override{{Path: configkit.Keys("entries", "child", "extends"), Value: "second", Flag: "--extends"}},
})
if err != nil || result.Config.Entries["child"].Count != 2 {
t.Fatalf("import and CLI reference = %#v, %v", result, err)
}
history := result.Metadata().History(configkit.Keys("entries", "child", "extends"))
if len(history) < 3 || history[len(history)-1].Origin.Operation != configkit.ChangeInheritance {
t.Fatalf("superseded reference history = %#v", history)
}
conflicting := configkit.NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: parent.yml}]}\nentries: {child: {extends: first}, first: {count: 3}}\n")},
"parent.yml": &fstest.MapFile{Data: []byte("entries: {first: {count: 1}}\n")},
})
_, err = configkit.Load[inheritedConfiguration](conflicting.File("main.yml"), configkit.Options[inheritedConfiguration]{Imports: true})
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != configkit.CodeComposeConflict {
t.Fatalf("strict conflict before expansion = %v", err)
}
}
func TestFacadeInheritanceSelectedOverlayCancelsRelationship(t *testing.T) {
filesystem := configkit.NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("entries: {parent: {count: 7}, child: {extends: parent, enabled: false}}\n")},
"overlays/test.yml": &fstest.MapFile{Data: []byte("entries: {child: {extends: null}}\n")},
})
result, err := configkit.Load(filesystem.File("main.yml"), configkit.Options[inheritedConfiguration]{
Overlay: "test", Filesystem: &configkit.FilesystemContext{Root: ".", FS: &filesystem},
})
if err != nil || result.Config.Entries["child"].Count != 0 || result.Config.Entries["child"].Enabled {
if loadError, ok := err.(*configkit.LoadError); ok {
t.Fatalf("cancelled relationship = %#v, %v, diagnostics %#v", result, err, loadError.Diagnostics())
}
t.Fatalf("cancelled relationship = %#v, %v", result, err)
}
history := result.Metadata().History(configkit.Keys("entries", "child", "extends"))
if len(history) < 3 || history[len(history)-1].Origin.Operation != configkit.ChangeInheritance {
t.Fatalf("cancellation history = %#v", history)
}
}
func TestFacadeInheritedSecretsStayRedactedAfterLookup(t *testing.T) {
const canary = "INHERITED-RELEASE-TEST-CREDENTIAL"
t.Setenv("CONFIGKIT_INHERITED_TEST_SECRET", canary)
type entry struct {
Credential configkit.Secret `yaml:"credential"`
Plain string `yaml:"plain"`
}
type configuration struct {
Entries map[string]entry `yaml:"entries"`
}
result, err := configkit.Load[configuration](configkit.Bytes("secret.yml", []byte("entries: {parent: {credential: CONFIGKIT_INHERITED_TEST_SECRET, plain: visible}, child: {extends: parent}}\n")), configkit.Options[configuration]{Definition: configkit.Definition[configuration]{
Secret: func(path configkit.Path) bool { return path.String() == "/entries/parent/credential" },
}})
if err != nil || result.Config.Entries["child"].Credential.Value() != canary {
t.Fatalf("inherited credential = %#v, %v", result, err)
}
path := configkit.Keys("entries", "child", "credential")
explanation, found := result.Explain(path)
if !found || !explanation.Effective.Redacted || strings.Contains(explanation.Effective.YAML, canary) {
t.Fatalf("credential explanation = %#v, %v", explanation, found)
}
var rendered bytes.Buffer
if err := result.WriteEffective(&rendered); err != nil || strings.Contains(rendered.String(), canary) || !strings.Contains(rendered.String(), "visible") {
t.Fatalf("effective secret rendering = %q, %v", rendered.String(), err)
}
rendered.Reset()
if err := result.WriteExplanation(&rendered, path); err != nil || strings.Contains(rendered.String(), canary) {
t.Fatalf("historical secret rendering = %q, %v", rendered.String(), err)
}
}

243
facade_layered_test.go Normal file
View File

@@ -0,0 +1,243 @@
package configkit_test
import (
"bytes"
"os"
"path/filepath"
"reflect"
"strconv"
"strings"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
type layeredFacadeConfig struct {
Settings map[string]string `yaml:"settings"`
Count int `yaml:"count"`
Enabled bool `yaml:"enabled"`
List []string `yaml:"list"`
Optional *string `yaml:"optional"`
Derived string `yaml:"derived"`
}
func TestFacadeLayeredMixedSourcesAndInspection(t *testing.T) {
mainPath := filepath.Join(t.TempDir(), "main.yml")
if err := os.WriteFile(mainPath, []byte("settings: {shared: main, secret: new-canary}\ncount: 0\nenabled: false\nlist: []\noptional: null\n"), 0o600); err != nil {
t.Fatal(err)
}
lower := configkit.NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: defaults.yml}]}\nsettings: {shared: lower, native: retained, secret: old-canary}\nlist: [lower]\n")},
"defaults.yml": &fstest.MapFile{Data: []byte("settings: {imported: retained}\n")},
"overlays/testing.yml": &fstest.MapFile{Data: []byte("settings: {from_lower_namespace: retained}\n")},
})
selected := configkit.NewFilesystem("bundle", fstest.MapFS{
"overlays/testing.yml": &fstest.MapFile{Data: []byte("settings: {shared: overlay}\n")},
})
checks := struct{ defaults, resolution, validation int }{}
result, err := configkit.LoadLayered(configkit.Input{Source: configkit.File(mainPath)}, configkit.LayeredOptions[layeredFacadeConfig]{
Underlays: []configkit.Underlay{
{Inputs: []configkit.Input{
{Source: lower.File("main.yml"), Imports: true},
{Source: configkit.Mapping("programmatic-low", map[string]any{"settings": map[string]any{"mapping": "retained"}, "count": 7, "enabled": true})},
{Source: lower.File("overlays/testing.yml")},
}},
{Inputs: []configkit.Input{{Source: configkit.Mapping("programmatic-high", map[string]any{
"settings": map[string]any{"shared": "middle", "second": "retained"}, "list": []any{"middle"},
})}}},
},
Overlay: "testing", OverlayFilesystem: &configkit.FilesystemContext{Root: ".", BaseDir: ".", FS: &selected},
Overrides: []configkit.Override{{Path: configkit.Keys("settings", "shared"), Value: "cli", Flag: "--shared"}},
Definition: configkit.Definition[layeredFacadeConfig]{
Defaults: func() layeredFacadeConfig { checks.defaults++; return layeredFacadeConfig{} },
Resolve: func(config *layeredFacadeConfig, resolution *configkit.Resolution) error {
checks.resolution++
if source, ok := resolution.Source("source-2"); !ok || source.Layer != configkit.LayerUnderlay || source.UnderlayIndex != 1 || source.Kind != configkit.SourceKindFilesystem {
t.Errorf("resolver source = %#v, %v", source, ok)
}
configkit.Default(resolution, configkit.Keys("derived"), &config.Derived, config.Settings["native"], "derive from native", configkit.Keys("settings", "native"))
return nil
},
Validate: func(config *layeredFacadeConfig, validation *configkit.Validation) { checks.validation++ },
Secret: func(path configkit.Path) bool { return path.String() == "/settings/secret" },
},
})
if err != nil {
t.Fatalf("layered load: %v, diagnostics %#v", err, err.(*configkit.LoadError).Diagnostics())
}
wantSettings := map[string]string{
"shared": "cli", "secret": "new-canary", "native": "retained", "imported": "retained",
"mapping": "retained", "from_lower_namespace": "retained", "second": "retained",
}
if !reflect.DeepEqual(result.Config.Settings, wantSettings) || result.Config.Count != 0 || result.Config.Enabled || result.Config.List == nil || len(result.Config.List) != 0 || result.Config.Optional != nil || result.Config.Derived != "retained" {
t.Fatalf("configuration = %#v", result.Config)
}
if checks != (struct{ defaults, resolution, validation int }{1, 1, 1}) {
t.Fatalf("lifecycle calls = %#v", checks)
}
sources := result.Sources()
if len(sources) != 8 {
t.Fatalf("sources = %#v", sources)
}
wantLayers := []configkit.Layer{configkit.LayerBase, configkit.LayerUnderlay, configkit.LayerUnderlay, configkit.LayerUnderlay, configkit.LayerUnderlay, configkit.LayerUnderlay, configkit.LayerOverlay, configkit.LayerCLI}
wantIndexes := []int{0, 1, 1, 1, 1, 2, 0, 0}
wantKinds := []configkit.SourceKind{configkit.SourceKindFile, configkit.SourceKindFilesystem, configkit.SourceKindFilesystem, configkit.SourceKindMapping, configkit.SourceKindFilesystem, configkit.SourceKindMapping, configkit.SourceKindFilesystem, configkit.SourceKindCLI}
for index, source := range sources {
if source.ID != "source-"+strconv.Itoa(index+1) || source.Layer != wantLayers[index] || source.UnderlayIndex != wantIndexes[index] || source.Kind != wantKinds[index] {
t.Errorf("source %d = %#v", index, source)
}
}
if sources[4].Path != sources[6].Path || sources[4].Label != sources[6].Label {
t.Fatalf("namespace-collision fixture changed: %#v, %#v", sources[4], sources[6])
}
sources[1].Label = "changed"
if result.Sources()[1].Label != "bundle/main.yml" {
t.Fatal("source records alias retained metadata")
}
for _, test := range []struct {
path string
want configkit.Layer
}{
{"/settings/native", configkit.LayerUnderlay},
{"/settings/shared", configkit.LayerCLI},
{"/derived", configkit.LayerResolution},
} {
path, err := configkit.ParsePath(test.path)
if err != nil {
t.Fatal(err)
}
info, ok := result.Metadata().Lookup(path)
if !ok || info.Origin.Layer != test.want {
t.Errorf("%s metadata = %#v, %v", test.path, info, ok)
}
}
container, ok := result.Metadata().Lookup(configkit.Keys("settings"))
if !ok || len(container.Origin.Locations) < 4 {
t.Fatalf("composite mapping origin = %#v, %v", container, ok)
}
explanation, ok := result.Explain(configkit.Keys("settings", "shared"))
if !ok || len(explanation.History) < 5 || explanation.Origin.Layer != configkit.LayerCLI {
t.Fatalf("shared history = %#v, %v", explanation, ok)
}
var effective, rendered bytes.Buffer
if err := result.WriteEffective(&effective); err != nil {
t.Fatal(err)
}
if err := result.WriteExplanation(&rendered, configkit.Keys("settings", "secret")); err != nil {
t.Fatal(err)
}
if strings.Contains(effective.String()+rendered.String(), "old-canary") || strings.Contains(effective.String()+rendered.String(), "new-canary") || !strings.Contains(rendered.String(), "redacted") {
t.Fatalf("secret rendering leaked: effective %q, explanation %q", effective.String(), rendered.String())
}
}
func TestFacadeLayeredDisplacedDescendantRemainsInspectable(t *testing.T) {
type config struct {
Choice string `yaml:"choice"`
}
result, err := configkit.LoadLayered(configkit.Input{Source: configkit.Mapping("main", map[string]any{"choice": "flat"})}, configkit.LayeredOptions[config]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{Source: configkit.Mapping("lower", map[string]any{
"choice": map[string]any{"nested": "historical"},
})}}}},
})
if err != nil || result.Config.Choice != "flat" {
t.Fatalf("replacement load = %#v, %v", result, err)
}
path := configkit.Keys("choice", "nested")
explanation, ok := result.Explain(path)
if !ok || explanation.Effective.Present || len(explanation.History) == 0 || explanation.History[0].Origin.Layer != configkit.LayerUnderlay {
t.Fatalf("deleted descendant = %#v, %v", explanation, ok)
}
}
func TestFacadeLayeredZeroUnderlaysMatchesOrdinaryLoad(t *testing.T) {
filesystem := configkit.NewFilesystem("bundle", fstest.MapFS{
"overlays/testing.yml": &fstest.MapFile{Data: []byte("value: selected\n")},
})
context := &configkit.FilesystemContext{Root: ".", BaseDir: ".", FS: &filesystem}
source := configkit.Bytes("main", []byte("value: base\n"))
type config struct {
Value string `yaml:"value"`
}
ordinary, err := configkit.Load(source, configkit.Options[config]{Overlay: "testing", Filesystem: context})
if err != nil {
t.Fatal(err)
}
layered, err := configkit.LoadLayered(configkit.Input{Source: source}, configkit.LayeredOptions[config]{Overlay: "testing", OverlayFilesystem: context})
if err != nil {
t.Fatal(err)
}
if ordinary.Config != layered.Config || !reflect.DeepEqual(ordinary.Sources(), layered.Sources()) {
t.Fatalf("ordinary = %#v, %#v; layered = %#v, %#v", ordinary.Config, ordinary.Sources(), layered.Config, layered.Sources())
}
var first, second bytes.Buffer
if err := ordinary.WriteEffective(&first); err != nil {
t.Fatal(err)
}
if err := layered.WriteEffective(&second); err != nil {
t.Fatal(err)
}
if first.String() != second.String() {
t.Fatalf("effective ordinary %q, layered %q", first.String(), second.String())
}
if ordinaryExplanation, ok := ordinary.Explain(configkit.Keys("value")); !ok {
t.Fatal("ordinary explanation missing")
} else if layeredExplanation, ok := layered.Explain(configkit.Keys("value")); !ok || !reflect.DeepEqual(ordinaryExplanation, layeredExplanation) {
t.Fatalf("explanations differ: %#v, %#v", ordinaryExplanation, layeredExplanation)
}
}
func TestFacadeLayeredOptionAndVersionFailures(t *testing.T) {
type config struct {
Version string `yaml:"version"`
}
namespace := configkit.NewFilesystem("bundle", fstest.MapFS{
"overlays/testing.yml": &fstest.MapFile{Data: []byte("{}\n")},
})
for _, test := range []struct {
name string
main configkit.Input
options []configkit.LayeredOptions[config]
want configkit.Code
}{
{"zero source", configkit.Input{}, nil, configkit.CodeSourceInvalid},
{"multiple options", configkit.Input{Source: configkit.NoSource()}, []configkit.LayeredOptions[config]{{}, {}}, configkit.CodeOptionsInvalid},
{"missing overlay context", configkit.Input{Source: configkit.NoSource()}, []configkit.LayeredOptions[config]{{Overlay: "testing"}}, configkit.CodeOptionsInvalid},
{"selected file already acquired", configkit.Input{Source: configkit.NoSource()}, []configkit.LayeredOptions[config]{{
Overlay: "testing", OverlayFilesystem: &configkit.FilesystemContext{Root: ".", BaseDir: ".", FS: &namespace},
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{Source: namespace.File("overlays/testing.yml")}}}},
}}, configkit.CodeOverlayRepeated},
{"underlay version", configkit.Input{Source: configkit.NoSource()}, []configkit.LayeredOptions[config]{{
Definition: configkit.Definition[config]{Version: &configkit.VersionRule{Path: configkit.Keys("version"), Check: func(configkit.Version, *configkit.Validation) error { return nil }}},
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{Source: configkit.Mapping("lower", map[string]any{"version": "forbidden"})}}}},
}}, configkit.CodeUnderlayVersion},
} {
t.Run(test.name, func(t *testing.T) {
_, err := configkit.LoadLayered(test.main, test.options...)
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != test.want {
t.Fatalf("error = %#v, want %s", err, test.want)
}
})
}
}
func TestFacadeLayeredExplicitHostOverlayWithoutMain(t *testing.T) {
root := t.TempDir()
if err := os.Mkdir(filepath.Join(root, "overlays"), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, "overlays", "testing.yml"), []byte("value: selected\n"), 0o600); err != nil {
t.Fatal(err)
}
type config struct {
Value string `yaml:"value"`
}
result, err := configkit.LoadLayered(configkit.Input{Source: configkit.NoSource()}, configkit.LayeredOptions[config]{
Overlay: "testing", OverlayFilesystem: &configkit.FilesystemContext{Root: root, BaseDir: root},
})
if err != nil || result.Config.Value != "selected" || len(result.Sources()) != 1 || result.Sources()[0].Kind != configkit.SourceKindFile {
t.Fatalf("explicit host overlay = %#v, %v", result, err)
}
}

133
facade_mapping_test.go Normal file
View File

@@ -0,0 +1,133 @@
package configkit_test
import (
"math"
"reflect"
"strings"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
type mappingFacadeConfig struct {
Enabled bool `yaml:"enabled"`
Count uint64 `yaml:"count"`
Optional *string `yaml:"optional"`
Items []string `yaml:"items"`
Name string `yaml:"name"`
}
func TestFacadeMappingSnapshotsAndMatchesYAML(t *testing.T) {
items := []any{"one", "two"}
values := map[string]any{
"name": "original", "enabled": false, "count": ^uint64(0),
"optional": nil, "items": items,
}
source := configkit.Mapping("programmatic", values)
values["name"] = "mutated"
items[0] = "mutated"
for range 2 {
result, err := configkit.Load[mappingFacadeConfig](source)
if err != nil {
t.Fatal(err)
}
want := mappingFacadeConfig{Count: ^uint64(0), Items: []string{"one", "two"}, Name: "original"}
if !reflect.DeepEqual(result.Config, want) {
t.Fatalf("mapping config = %#v, want %#v", result.Config, want)
}
if sources := result.Sources(); len(sources) != 1 || sources[0].ID != "source-1" || sources[0].Kind != configkit.SourceKindMapping || sources[0].Layer != configkit.LayerBase || sources[0].Path != "" {
t.Fatalf("mapping sources = %#v", sources)
}
info, ok := result.Metadata().Lookup(configkit.Keys("name"))
if !ok || len(info.Origin.Locations) != 1 || info.Origin.Locations[0].SourceID != "source-1" || info.Origin.Locations[0].Line != 0 {
t.Fatalf("mapping name origin = %#v, %v", info, ok)
}
fromYAML, err := configkit.Load[mappingFacadeConfig](configkit.Bytes("equivalent", []byte("enabled: false\ncount: 18446744073709551615\noptional: null\nitems: [one, two]\nname: original\n")))
if err != nil || !reflect.DeepEqual(result.Config, fromYAML.Config) {
t.Fatalf("equivalent YAML = %#v, %v", fromYAML, err)
}
}
}
func TestFacadeMappingNullEmptyAndVersion(t *testing.T) {
result, err := configkit.Load[mappingFacadeConfig](configkit.Mapping("empty", map[string]any{}))
if err != nil || len(result.Sources()) != 1 {
t.Fatalf("empty mapping = %#v, %v", result, err)
}
var nilSequence []any
var nilMap map[string]any
var checked string
result, err = configkit.Load(configkit.Mapping("nullable", map[string]any{
"items": nilSequence, "optional": nilMap, "enabled": false, "count": uint64(0),
}), configkit.Options[mappingFacadeConfig]{Definition: configkit.Definition[mappingFacadeConfig]{
Version: &configkit.VersionRule{Path: configkit.Keys("count"), Check: func(version configkit.Version, validation *configkit.Validation) error {
checked = version.Text
return nil
}},
}})
if err != nil || checked != "0" || result.Config.Items != nil || result.Config.Optional != nil {
t.Fatalf("null and zero mapping = %#v, checked %q, %v", result, checked, err)
}
info, ok := result.Metadata().Lookup(configkit.Keys("items"))
if !ok || !info.Null || !info.Explicit {
t.Fatalf("null metadata = %#v, %v", info, ok)
}
}
func TestFacadeMappingImportsUseExplicitContext(t *testing.T) {
namespace := configkit.NewFilesystem("bundle", fstest.MapFS{
"root/shared.yml": &fstest.MapFile{Data: []byte("name: imported\n")},
})
source := configkit.Mapping("programmatic", map[string]any{
"configkit": map[string]any{"imports": []any{map[string]any{"path": "shared.yml"}}},
"enabled": true,
})
result, err := configkit.Load(source, configkit.Options[mappingFacadeConfig]{
Imports: true, Filesystem: &configkit.FilesystemContext{Root: "root", BaseDir: "root", FS: &namespace},
})
if err != nil || result.Config.Name != "imported" || !result.Config.Enabled {
t.Fatalf("mapping import = %#v, %v", result, err)
}
if sources := result.Sources(); len(sources) != 2 || sources[0].Kind != configkit.SourceKindMapping || sources[1].Kind != configkit.SourceKindFilesystem || sources[1].Layer != configkit.LayerBase {
t.Fatalf("mapping import sources = %#v", sources)
}
checkMappingCode(t, source, configkit.Options[mappingFacadeConfig]{Imports: true}, configkit.CodeOptionsInvalid)
checkMappingCode(t, source, configkit.Options[mappingFacadeConfig]{}, configkit.CodeImportDisabled)
}
func TestFacadeMappingDeferredValueDiagnostics(t *testing.T) {
cycle := map[string]any{}
cycle["secret"] = cycle
for _, test := range []struct {
name string
source configkit.Source
want configkit.Code
path string
}{
{"nil root", configkit.Mapping("nil", nil), configkit.CodeSourceInvalid, ""},
{"bad label", configkit.Mapping("bad\nlabel", map[string]any{}), configkit.CodeSourceInvalid, ""},
{"unsupported", configkit.Mapping("bad", map[string]any{"secret": struct{ Value string }{"canary-value"}}), configkit.CodeSourceInvalid, "/secret"},
{"nonfinite", configkit.Mapping("bad", map[string]any{"count": math.NaN()}), configkit.CodeSourceInvalid, "/count"},
{"cycle", configkit.Mapping("bad", cycle), configkit.CodeSourceInvalid, "/secret"},
{"too many bytes", configkit.Mapping("bad", map[string]any{"name": strings.Repeat("x", 8<<20+1)}), configkit.CodeSourceLimitExceeded, "/name"},
} {
t.Run(test.name, func(t *testing.T) {
loadError := checkMappingCode(t, test.source, configkit.Options[mappingFacadeConfig]{}, test.want)
diagnostic := loadError.Diagnostics()[0]
if diagnostic.Path.String() != test.path || diagnostic.Primary == nil || diagnostic.Primary.SourceID != "source-1" || strings.Contains(diagnostic.Message, "canary-value") {
t.Fatalf("diagnostic = %#v", diagnostic)
}
})
}
}
func checkMappingCode(t *testing.T, source configkit.Source, options configkit.Options[mappingFacadeConfig], want configkit.Code) *configkit.LoadError {
t.Helper()
_, err := configkit.Load(source, options)
loadError, ok := err.(*configkit.LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != want {
t.Fatalf("error = %#v, want %s", err, want)
}
return loadError
}

View File

@@ -0,0 +1,221 @@
package configkit_test
import (
"errors"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
type observedResourceFS struct {
files fstest.MapFS
opens int
closed bool
}
func (f *observedResourceFS) Open(name string) (fs.File, error) {
f.opens++
if f.closed {
return nil, fs.ErrClosed
}
return f.files.Open(name)
}
type resourceFacadeConfig struct {
Native string `yaml:"native"`
Shared string `yaml:"shared"`
Host string `yaml:"host"`
Selected string `yaml:"selected"`
}
func TestFacadeSourceContextsFollowOriginsWithoutQueryIO(t *testing.T) {
host := t.TempDir()
if err := os.WriteFile(filepath.Join(host, "main.yml"), []byte("host: host.txt\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(host, "host.txt"), []byte("host-v1"), 0o600); err != nil {
t.Fatal(err)
}
borrowed := &observedResourceFS{files: fstest.MapFS{
"bundle/config/defaults.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: ../shared.yml}]}\nnative: icon.txt\n")},
"bundle/shared.yml": &fstest.MapFile{Data: []byte("shared: shared.txt\n")},
"bundle/overlays/test.yml": &fstest.MapFile{Data: []byte("selected: overlay.txt\n")},
"bundle/config/icon.txt": &fstest.MapFile{Data: []byte("icon-v1")},
"bundle/shared.txt": &fstest.MapFile{Data: []byte("shared-v1")},
"bundle/overlays/overlay.txt": &fstest.MapFile{Data: []byte("overlay-v1")},
}}
namespace := configkit.NewFilesystem("bundle", borrowed)
var resolverContext configkit.ResourceContext
result, err := configkit.LoadLayered(configkit.Input{Source: configkit.File(filepath.Join(host, "main.yml"))}, configkit.LayeredOptions[resourceFacadeConfig]{
Underlays: []configkit.Underlay{{Inputs: []configkit.Input{{
Source: namespace.File("bundle/config/defaults.yml"), Imports: true,
Filesystem: &configkit.FilesystemContext{Root: "bundle", FS: &namespace},
}}}},
Overlay: "test", OverlayFilesystem: &configkit.FilesystemContext{Root: "bundle", BaseDir: "bundle", FS: &namespace},
Definition: configkit.Definition[resourceFacadeConfig]{Resolve: func(config *resourceFacadeConfig, resolution *configkit.Resolution) error {
before := borrowed.opens
var ok bool
resolverContext, ok = resolution.SourceContext("source-2")
if !ok || resolverContext.BaseDir != "config" || resolverContext.FS == nil {
t.Errorf("resolver context = %#v, %v", resolverContext, ok)
}
if _, ok := resolution.SourceContext("missing"); ok || borrowed.opens != before {
t.Error("resolution context query performed I/O or guessed an unknown source")
}
return nil
}},
})
if err != nil {
t.Fatalf("load: %v, diagnostics %#v", err, err.(*configkit.LoadError).Diagnostics())
}
queriesBefore := borrowed.opens
for _, test := range []struct {
field, wantBase, wantData string
}{
{"native", "config", "icon-v1"},
{"shared", ".", "shared-v1"},
{"selected", "overlays", "overlay-v1"},
{"host", ".", "host-v1"},
} {
origin, ok := result.Metadata().Lookup(configkit.Keys(test.field))
if !ok || len(origin.Origin.Locations) != 1 {
t.Fatalf("%s origin = %#v, %v", test.field, origin, ok)
}
context, ok := result.SourceContext(origin.Origin.Locations[0].SourceID)
if !ok || context.FS == nil || context.BaseDir != test.wantBase {
t.Fatalf("%s context = %#v, %v", test.field, context, ok)
}
if borrowed.opens != queriesBefore {
t.Fatal("result SourceContext query performed filesystem I/O")
}
resourceName := path.Join(context.BaseDir, map[string]string{
"native": result.Config.Native, "shared": result.Config.Shared,
"selected": result.Config.Selected, "host": result.Config.Host,
}[test.field])
data, err := fs.ReadFile(context.FS, resourceName)
if err != nil || string(data) != test.wantData {
t.Fatalf("%s resource = %q, %v", test.field, data, err)
}
queriesBefore = borrowed.opens
}
if _, ok := result.SourceContext(""); ok {
t.Fatal("empty source ID returned a context")
}
var nilResult *configkit.Result[resourceFacadeConfig]
if _, ok := nilResult.SourceContext("source-2"); ok {
t.Fatal("nil result returned a context")
}
if _, err := fs.ReadFile(resolverContext.FS, "../shared.txt"); !errors.Is(err, fs.ErrInvalid) {
t.Fatalf("native root escape = %v", err)
}
borrowed.files["bundle/config/icon.txt"].Data = []byte("icon-v2")
data, err := fs.ReadFile(resolverContext.FS, "config/icon.txt")
if err != nil || string(data) != "icon-v2" {
t.Fatalf("updated native resource = %q, %v", data, err)
}
borrowed.closed = true
if result.Config.Native != "icon.txt" {
t.Fatal("closing caller filesystem altered loaded configuration")
}
if _, err := fs.ReadFile(resolverContext.FS, "config/icon.txt"); !errors.Is(err, fs.ErrClosed) {
t.Fatalf("closed caller filesystem error = %v", err)
}
}
func TestFacadeHostContextSurvivesConfigurationDeletion(t *testing.T) {
root := t.TempDir()
main := filepath.Join(root, "main.yml")
resource := filepath.Join(root, "asset.txt")
if err := os.WriteFile(main, []byte("native: asset.txt\n"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(resource, []byte("first"), 0o600); err != nil {
t.Fatal(err)
}
result, err := configkit.Load[resourceFacadeConfig](configkit.File(main))
if err != nil {
t.Fatal(err)
}
context, ok := result.SourceContext("source-1")
if !ok || context.BaseDir != "." {
t.Fatalf("host context = %#v, %v", context, ok)
}
if err := os.Remove(main); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(resource, []byte("second"), 0o600); err != nil {
t.Fatal(err)
}
data, err := fs.ReadFile(context.FS, path.Join(context.BaseDir, result.Config.Native))
if err != nil || string(data) != "second" || result.Config.Native != "asset.txt" {
t.Fatalf("resource after source deletion = %q, %v", data, err)
}
if _, err := fs.ReadFile(context.FS, "../asset.txt"); !errors.Is(err, fs.ErrInvalid) {
t.Fatalf("invalid host path = %v", err)
}
if err := os.Remove(resource); err != nil {
t.Fatal(err)
}
if _, err := fs.ReadFile(context.FS, "asset.txt"); !errors.Is(err, fs.ErrNotExist) {
t.Fatalf("missing later resource = %v", err)
}
outside := filepath.Join(t.TempDir(), "outside.txt")
if err := os.WriteFile(outside, []byte("outside"), 0o600); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Join(root, "escape")); err == nil {
if _, err := fs.ReadFile(context.FS, "escape"); !errors.Is(err, fs.ErrPermission) {
t.Fatalf("host symlink escape = %v", err)
}
}
}
func TestFacadeSyntheticSourceContextsAndContextlessSources(t *testing.T) {
namespace := configkit.NewFilesystem("bundle", fstest.MapFS{
"bundle/config/icon.txt": &fstest.MapFile{Data: []byte("icon")},
})
context := &configkit.FilesystemContext{Root: "bundle", BaseDir: "bundle/config", FS: &namespace}
for _, test := range []struct {
name string
source configkit.Source
}{
{"bytes", configkit.Bytes("memory", []byte("native: icon.txt\n"))},
{"reader", configkit.Reader("stream", strings.NewReader("native: icon.txt\n"))},
{"mapping", configkit.Mapping("values", map[string]any{"native": "icon.txt"})},
} {
t.Run(test.name, func(t *testing.T) {
result, err := configkit.Load(test.source, configkit.Options[resourceFacadeConfig]{Filesystem: context})
if err != nil {
t.Fatal(err)
}
resource, ok := result.SourceContext("source-1")
if !ok || resource.BaseDir != "config" {
t.Fatalf("synthetic context = %#v, %v", resource, ok)
}
data, err := fs.ReadFile(resource.FS, path.Join(resource.BaseDir, result.Config.Native))
if err != nil || string(data) != "icon" {
t.Fatalf("synthetic resource = %q, %v", data, err)
}
})
}
result, err := configkit.Load[resourceFacadeConfig](configkit.Bytes("plain", []byte("native: none\n")))
if err != nil {
t.Fatal(err)
}
if _, ok := result.SourceContext("source-1"); ok {
t.Fatal("contextless byte source returned a context")
}
result, err = configkit.Load(configkit.NoSource(), configkit.Options[resourceFacadeConfig]{Overrides: []configkit.Override{{Path: configkit.Keys("native"), Value: "from-cli", Flag: "--native"}}})
if err != nil {
t.Fatal(err)
}
if _, ok := result.SourceContext("source-1"); ok {
t.Fatal("CLI source returned a context")
}
}

View File

@@ -0,0 +1,107 @@
package configkit_test
import (
"reflect"
"testing"
"testing/fstest"
configkit "gitea.maximumdirect.net/eric/ConfigKit"
)
func TestUnderlaySequenceDescendantOrigins(t *testing.T) {
for _, later := range []bool{false, true} {
name := "first tier"
if later {
name = "later tier"
}
t.Run(name, func(t *testing.T) {
bundle := configkit.NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: items.yml}]}\n")},
"items.yml": &fstest.MapFile{Data: []byte("items: [{name: first, nested: [false]}]\n")},
})
tiers := []configkit.Underlay{}
if later {
tiers = append(tiers, configkit.Underlay{Inputs: []configkit.Input{{Source: configkit.Mapping("lower", map[string]any{"other": true})}}})
}
tiers = append(tiers, configkit.Underlay{Inputs: []configkit.Input{{Source: bundle.File("main.yml"), Imports: true}}})
paths := []string{"/items", "/items/0", "/items/0/name", "/items/0/nested", "/items/0/nested/0"}
entry := map[string]configkit.Origin{}
result, err := configkit.LoadLayered[map[string]any](configkit.Input{Source: configkit.Mapping("main", map[string]any{"main": true})}, configkit.LayeredOptions[map[string]any]{
Underlays: tiers,
Definition: configkit.Definition[map[string]any]{Resolve: func(_ *map[string]any, resolution *configkit.Resolution) error {
for _, pointer := range paths {
path, _ := configkit.ParsePath(pointer)
info, ok := resolution.InputInfo(path)
if !ok {
t.Errorf("missing resolver input %s", pointer)
}
entry[pointer] = info.Origin
}
return nil
}},
})
if err != nil {
t.Fatal(err)
}
var sourceID string
for _, source := range result.Sources() {
if source.Path == "items.yml" {
sourceID = source.ID
}
}
if sourceID == "" {
t.Fatal("missing imported source")
}
for _, pointer := range paths {
path, _ := configkit.ParsePath(pointer)
info, ok := result.Metadata().Lookup(path)
if !ok || info.Origin.Layer != configkit.LayerUnderlay || len(info.Origin.Locations) != 1 || info.Origin.Locations[0].SourceID != sourceID {
t.Errorf("%s origin = %+v", pointer, info.Origin)
}
if !reflect.DeepEqual(entry[pointer], info.Origin) {
t.Errorf("%s resolver origin differs", pointer)
}
explanation, ok := result.Explain(path)
if !ok || !reflect.DeepEqual(explanation.Origin, info.Origin) {
t.Errorf("%s explanation origin differs", pointer)
}
history := result.Metadata().History(path)
if len(history) == 0 || !reflect.DeepEqual(history[len(history)-1].Origin, info.Origin) {
t.Errorf("%s history = %+v", pointer, history)
}
}
})
}
}
func TestUnderlayHistoryRetainsLaterMappingContributors(t *testing.T) {
result, err := configkit.LoadLayered[map[string]any](configkit.Input{Source: configkit.NoSource()}, configkit.LayeredOptions[map[string]any]{Underlays: []configkit.Underlay{
{Inputs: []configkit.Input{{Source: configkit.Mapping("one", map[string]any{"x": 1})}}},
{Inputs: []configkit.Input{{Source: configkit.Mapping("two", map[string]any{"x": map[string]any{}})}}},
{Inputs: []configkit.Input{{Source: configkit.Mapping("three", map[string]any{"x": map[string]any{"y": 2}})}}},
}})
if err != nil {
t.Fatal(err)
}
path := configkit.Keys("x")
info, _ := result.Metadata().Lookup(path)
wantOrigin := configkit.Origin{Layer: configkit.LayerUnderlay, Locations: []configkit.Location{{SourceID: "source-2"}, {SourceID: "source-3"}}}
if !reflect.DeepEqual(info.Origin, wantOrigin) {
t.Fatalf("effective origin = %+v, want %+v", info.Origin, wantOrigin)
}
history := result.Metadata().History(path)
if len(history) != 3 || !reflect.DeepEqual(history[len(history)-1].Origin, info.Origin) {
t.Fatalf("history = %+v, effective origin = %+v", history, info.Origin)
}
explanation, ok := result.Explain(path)
if !ok || !reflect.DeepEqual(explanation.Origin, info.Origin) {
t.Fatal("explanation and metadata disagree")
}
if len(explanation.History) != 3 {
t.Fatalf("explanation history = %+v", explanation.History)
}
last := explanation.History[len(explanation.History)-1].Origin
if last.Layer != configkit.LayerUnderlay || len(last.Locations) != 1 || last.Locations[0].SourceID != "source-3" {
t.Fatalf("last explanation origin = %+v", last)
}
}

View File

@@ -39,7 +39,8 @@ type Explanation struct {
} }
// Explain returns safe effective and chronological values for a known final or // Explain returns safe effective and chronological values for a known final or
// historical path. Unknown and malformed paths return false. // historical path. Structural inheritance appears between source applications
// and typed reconciliation. Unknown and malformed paths return false.
func (r *Result[T]) Explain(path Path) (Explanation, bool) { func (r *Result[T]) Explain(path Path) (Explanation, bool) {
if r == nil { if r == nil {
return Explanation{}, false return Explanation{}, false

View File

@@ -0,0 +1,119 @@
package engine
import (
"strconv"
"testing"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func BenchmarkComposeWideMappings(b *testing.B) {
for _, width := range []int{128, 256, 512} {
b.Run(strconv.Itoa(width), func(b *testing.B) {
left := benchmarkCatalog("source-1", 0, width)
right := benchmarkCatalog("source-2", width, width)
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, diagnostics := composeDocuments(left, right); len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
}
})
}
}
func BenchmarkOverlayRecursiveMappings(b *testing.B) {
for _, width := range []int{128, 256, 512} {
b.Run(strconv.Itoa(width), func(b *testing.B) {
base, diagnostics := composeDocuments(benchmarkCatalog("source-1", 0, width))
if len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
overlay := benchmarkCatalog("source-2", width/2, width)
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, diagnostics := overlayDocuments(base, overlay); len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
}
})
}
}
func BenchmarkOverlayManyApplications(b *testing.B) {
for _, count := range []int{32, 64, 128} {
inputs := make([]*document.Node, count)
for index := range inputs {
inputs[index] = benchmarkCatalog("source-2", index, 1)
}
benchmarkOverlayApplications(b, count, inputs)
}
}
func BenchmarkOverlayRepeatedReplacement(b *testing.B) {
for _, count := range []int{32, 64, 128} {
inputs := make([]*document.Node, count)
for index := range inputs {
location := document.Location{SourceID: "source-2"}
value := &document.Node{Kind: document.String, Text: "replacement", Location: location}
if index%2 == 0 {
value = &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{{
Key: "enabled", Value: &document.Node{Kind: document.Boolean, Text: "false", Location: location},
}}}
}
inputs[index] = &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{{
Key: "catalog", Value: &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{{
Key: "item-0", Value: value,
}}},
}}}
}
benchmarkOverlayApplications(b, count, inputs)
}
}
func benchmarkOverlayApplications(b *testing.B, count int, inputs []*document.Node) {
base, diagnostics := composeDocuments(benchmarkCatalog("source-1", 0, 1))
if len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
for _, mode := range []string{"owned", "copy-preserving"} {
b.Run(mode+"/"+strconv.Itoa(count), func(b *testing.B) {
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if mode == "owned" {
assembly := newAssemblyAccumulator(cloneComposedDocument(base))
for _, input := range inputs {
if diagnostics := assembly.applyOverlay(input, nil, nil, LayerOverlay); len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
}
continue
}
current := base
for _, input := range inputs {
current, diagnostics = overlayDocuments(current, input)
if len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
}
}
})
}
}
func benchmarkCatalog(source string, first, width int) *document.Node {
location := document.Location{SourceID: source}
entries := make([]document.Entry, width)
for index := range entries {
entries[index] = document.Entry{
Key: "item-" + strconv.Itoa(first+index),
Value: &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{{Key: "enabled", Value: &document.Node{Kind: document.Boolean, Text: "true", Location: location}}}},
}
}
return &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{{
Key: "catalog", Value: &document.Node{Kind: document.Mapping, Location: location, Mapping: entries},
}}}
}

View File

@@ -15,6 +15,17 @@ type composedDocument struct {
inputNodes int inputNodes int
} }
// assemblyAccumulator owns one mutable working document. Acquired trees held by
// history applications are separate from its root and remain unchanged.
type assemblyAccumulator struct {
current *composedDocument
index mappingIndex
}
func newAssemblyAccumulator(owned *composedDocument) *assemblyAccumulator {
return &assemblyAccumulator{current: owned}
}
// cloneComposedDocument detaches mutable composition and provenance state; // cloneComposedDocument detaches mutable composition and provenance state;
// history application trees are immutable and retain their existing sharing. // history application trees are immutable and retain their existing sharing.
func cloneComposedDocument(base *composedDocument) *composedDocument { func cloneComposedDocument(base *composedDocument) *composedDocument {
@@ -58,8 +69,14 @@ func composeDocuments(inputs ...*document.Node) (*composedDocument, []Diagnostic
collected.append(input, Path{}) collected.append(input, Path{})
} }
checked := make(map[string]struct{}, len(collected.byPath))
for _, claim := range collected.ordered { for _, claim := range collected.ordered {
claims := collected.byPath[claim.path.String()] pointer := claim.path.String()
if _, seen := checked[pointer]; seen {
continue
}
checked[pointer] = struct{}{}
claims := collected.byPath[pointer]
if len(claims) < 2 || allMappingClaims(claims) { if len(claims) < 2 || allMappingClaims(claims) {
continue continue
} }
@@ -67,21 +84,43 @@ func composeDocuments(inputs ...*document.Node) (*composedDocument, []Diagnostic
} }
root := document.Clone(inputs[0]) root := document.Clone(inputs[0])
index := new(mappingIndex)
for _, input := range inputs[1:] { for _, input := range inputs[1:] {
mergeDisjointMappings(root, input) mergeDisjointMappings(root, input, index)
} }
applications := make([]historyApplication, 0, len(inputs)) applications := make([]historyApplication, 0, len(inputs))
for _, input := range inputs { for _, input := range inputs {
applications = append(applications, historyApplication{layer: LayerBase, value: input}) applications = append(applications, historyApplication{layer: LayerBase, value: input})
} }
origins := compositionOrigins(collected.ordered)
completeCompositionOrigins(root, Path{}, origins)
return &composedDocument{ return &composedDocument{
root: root, root: root,
claims: cloneCompositionClaims(collected.ordered), claims: cloneCompositionClaims(collected.ordered),
origins: compositionOrigins(collected.ordered), origins: origins,
applications: applications, applications: applications,
}, nil }, nil
} }
// Conflict claims treat sequences atomically, but inspection needs origins for
// every descendant. Preserve composite mapping claims while filling those gaps.
func completeCompositionOrigins(node *document.Node, path Path, origins map[string]Origin) {
pointer := path.String()
if _, present := origins[pointer]; !present {
origins[pointer] = Origin{Layer: LayerBase, Locations: []Location{publicLocation(node.Location)}}
}
switch node.Kind {
case document.Mapping:
for _, entry := range node.Mapping {
completeCompositionOrigins(entry.Value, path.Key(entry.Key), origins)
}
case document.Sequence:
for index, child := range node.Sequence {
completeCompositionOrigins(child, path.Index(index), origins)
}
}
}
func (c *compositionClaims) append(node *document.Node, path Path) { func (c *compositionClaims) append(node *document.Node, path Path) {
claim := compositionClaim{ claim := compositionClaim{
path: path.clone(), path: path.clone(),
@@ -109,18 +148,18 @@ func allMappingClaims(claims []compositionClaim) bool {
return true return true
} }
func mergeDisjointMappings(target, incoming *document.Node) { func mergeDisjointMappings(target, incoming *document.Node, index *mappingIndex) {
for _, incomingEntry := range incoming.Mapping { for _, incomingEntry := range incoming.Mapping {
index := documentEntryIndex(target.Mapping, incomingEntry.Key) position := index.find(target, incomingEntry.Key)
if index < 0 { if position < 0 {
cloned := incomingEntry cloned := incomingEntry
cloned.Value = document.Clone(incomingEntry.Value) cloned.Value = document.Clone(incomingEntry.Value)
target.Mapping = append(target.Mapping, cloned) index.append(target, cloned)
continue continue
} }
// Claim validation guarantees that the only remaining overlap is a // Claim validation guarantees that the only remaining overlap is a
// mapping pair. Empty mappings naturally contribute no child entries. // mapping pair. Empty mappings naturally contribute no child entries.
mergeDisjointMappings(target.Mapping[index].Value, incomingEntry.Value) mergeDisjointMappings(target.Mapping[position].Value, incomingEntry.Value, index)
} }
} }

View File

@@ -116,6 +116,31 @@ func TestComposeDocumentsReportsAllClaimantsAtExactLocations(t *testing.T) {
assertCompositionConflict(t, diagnostics, "/group", []string{"source-1", "source-2", "source-3"}) assertCompositionConflict(t, diagnostics, "/group", []string{"source-1", "source-2", "source-3"})
} }
func TestComposeDocumentsKeepsFirstConflictAndAppendedEntryOrder(t *testing.T) {
first := parseCompositionDocument(t, "source-1", "catalog:\n first: 1\n shared: one\n later: one\n")
second := parseCompositionDocument(t, "source-2", "catalog:\n second: 2\n shared: two\n later: two\n")
third := parseCompositionDocument(t, "source-3", "catalog:\n third: 3\n shared: three\n")
_, diagnostics := composeDocuments(first, second, third)
assertCompositionConflict(t, diagnostics, "/catalog/shared", []string{"source-1", "source-2", "source-3"})
composed, diagnostics := composeDocuments(
parseCompositionDocument(t, "source-1", "catalog:\n first: 1\n"),
parseCompositionDocument(t, "source-2", "catalog:\n second: 2\n"),
parseCompositionDocument(t, "source-3", "catalog:\n third: 3\n"),
)
if len(diagnostics) != 0 {
t.Fatalf("composeDocuments() diagnostics = %#v", diagnostics)
}
entries := composed.root.Mapping[0].Value.Mapping
got := make([]string, len(entries))
for index, entry := range entries {
got[index] = entry.Key
}
if want := []string{"first", "second", "third"}; !reflect.DeepEqual(got, want) {
t.Fatalf("catalog keys = %#v, want %#v", got, want)
}
}
func TestComposeDocumentsTreatsEmptyMappingsAsChildlessContributions(t *testing.T) { func TestComposeDocumentsTreatsEmptyMappingsAsChildlessContributions(t *testing.T) {
empty, diagnostics := composeDocuments( empty, diagnostics := composeDocuments(
parseCompositionDocument(t, "source-1", "{}\n"), parseCompositionDocument(t, "source-1", "{}\n"),

View File

@@ -51,13 +51,14 @@ func (r *Inspection) Explain(path Path) (Explanation, bool) {
After: valueSnapshot(working, path, r.secrets), Origin: Origin{Layer: LayerDefault}, After: valueSnapshot(working, path, r.secrets), Origin: Origin{Layer: LayerDefault},
}) })
} }
replayIndex := new(mappingIndex)
for _, application := range r.applications { for _, application := range r.applications {
operationPath, relevant := applicationOperationPath(application, path) operationPath, relevant := applicationOperationPath(application, path)
var before ValueSnapshot var before ValueSnapshot
if relevant { if relevant {
before = valueSnapshot(working, path, r.secrets) before = valueSnapshot(working, path, r.secrets)
} }
applyHistoryApplication(&working, application) applyHistoryApplication(&working, application, replayIndex)
if !relevant { if !relevant {
continue continue
} }
@@ -71,6 +72,7 @@ func (r *Inspection) Explain(path Path) (Explanation, bool) {
Origin: historyApplicationOrigin(application, operationPath), Origin: historyApplicationOrigin(application, operationPath),
}) })
} }
appendInheritanceExplanation(&explanation, &working, path, r.inheritance, r.secrets)
// Intermediate input replacements do not permanently suppress defaults: // Intermediate input replacements do not permanently suppress defaults:
// decoding applies them to the final input tree. Show that reconciliation // decoding applies them to the final input tree. Show that reconciliation

View File

@@ -23,10 +23,14 @@ func (r *Inspection) WriteExplanation(output io.Writer, path Path) error {
return errExplanationUnknownPath return errExplanationUnknownPath
} }
var rendered strings.Builder var rendered strings.Builder
labels := make(map[string]string, len(r.sources))
for _, source := range r.sources {
labels[source.ID] = source.Label
}
rendered.WriteString("path: ") rendered.WriteString("path: ")
rendered.WriteString(strconv.Quote(explanation.Path.String())) rendered.WriteString(strconv.Quote(explanation.Path.String()))
rendered.WriteByte('\n') rendered.WriteByte('\n')
writeExplanationOrigin(&rendered, "effective source", explanation.Origin, r.sources) writeExplanationOrigin(&rendered, "effective source", explanation.Origin, labels)
writeExplanationSnapshot(&rendered, "effective (canonical typed)", explanation.Effective, 0) writeExplanationSnapshot(&rendered, "effective (canonical typed)", explanation.Effective, 0)
if explanation.SecretReference != nil { if explanation.SecretReference != nil {
rendered.WriteString("secret reference: ") rendered.WriteString("secret reference: ")
@@ -38,12 +42,14 @@ func (r *Inspection) WriteExplanation(output io.Writer, path Path) error {
rendered.WriteString(" - representation: ") rendered.WriteString(" - representation: ")
if change.Origin.Operation == ChangeDecoding { if change.Origin.Operation == ChangeDecoding {
rendered.WriteString("input to canonical typed\n") rendered.WriteString("input to canonical typed\n")
} else if change.Origin.Layer == LayerBase || change.Origin.Layer == LayerOverlay || change.Origin.Layer == LayerCLI { } else if change.Origin.Operation == ChangeInheritance {
rendered.WriteString("structural inheritance\n")
} else if change.Origin.Layer == LayerBase || change.Origin.Layer == LayerUnderlay || change.Origin.Layer == LayerOverlay || change.Origin.Layer == LayerCLI {
rendered.WriteString("policy-validated input\n") rendered.WriteString("policy-validated input\n")
} else { } else {
rendered.WriteString("canonical typed\n") rendered.WriteString("canonical typed\n")
} }
writeExplanationOrigin(&rendered, " source", change.Origin, r.sources) writeExplanationOrigin(&rendered, " source", change.Origin, labels)
writeExplanationSnapshot(&rendered, " before", change.Before, 4) writeExplanationSnapshot(&rendered, " before", change.Before, 4)
writeExplanationSnapshot(&rendered, " after", change.After, 4) writeExplanationSnapshot(&rendered, " after", change.After, 4)
for _, reason := range change.Reasons { for _, reason := range change.Reasons {
@@ -65,7 +71,7 @@ func (r *Inspection) WriteExplanation(output io.Writer, path Path) error {
return err return err
} }
func writeExplanationOrigin(output *strings.Builder, label string, origin Origin, sources []SourceInfo) { func writeExplanationOrigin(output *strings.Builder, label string, origin Origin, labels map[string]string) {
output.WriteString(label) output.WriteString(label)
output.WriteString(": layer=") output.WriteString(": layer=")
output.WriteString(strconv.Quote(string(origin.Layer))) output.WriteString(strconv.Quote(string(origin.Layer)))
@@ -77,7 +83,7 @@ func writeExplanationOrigin(output *strings.Builder, label string, origin Origin
for _, location := range origin.Locations { for _, location := range origin.Locations {
output.WriteString(" location: source=") output.WriteString(" location: source=")
output.WriteString(strconv.Quote(location.SourceID)) output.WriteString(strconv.Quote(location.SourceID))
if label := sourceLabel(sources, location.SourceID); label != "" { if label := labels[location.SourceID]; label != "" {
output.WriteString(" label=") output.WriteString(" label=")
output.WriteString(strconv.Quote(label)) output.WriteString(strconv.Quote(label))
} }
@@ -87,15 +93,15 @@ func writeExplanationOrigin(output *strings.Builder, label string, origin Origin
output.WriteString(strconv.Itoa(location.Column)) output.WriteString(strconv.Itoa(location.Column))
output.WriteByte('\n') output.WriteByte('\n')
} }
} for _, step := range origin.Inheritance {
output.WriteString(" inherited: from=")
func sourceLabel(sources []SourceInfo, sourceID string) string { output.WriteString(strconv.Quote(step.From.String()))
for _, source := range sources { output.WriteString(" to=")
if source.ID == sourceID { output.WriteString(strconv.Quote(step.To.String()))
return source.Label output.WriteString(" reference=")
output.WriteString(strconv.Quote(step.Reference.String()))
output.WriteByte('\n')
} }
}
return ""
} }
func writeExplanationSnapshot(output *strings.Builder, label string, snapshot ValueSnapshot, indent int) { func writeExplanationSnapshot(output *strings.Builder, label string, snapshot ValueSnapshot, indent int) {

View File

@@ -1,12 +1,21 @@
package engine package engine
import ( import (
"io/fs"
"os" "os"
"path"
"path/filepath" "path/filepath"
"strings" "strings"
) )
func validateFilesystemContext(source Source, context *FilesystemContext, main parsedSource) *Diagnostic { func validateFilesystemContext(source Source, context *FilesystemContext, main parsedSource) *Diagnostic {
if nativeContext(source, context) {
access, basePath, mainPath, diagnostic := nativeFilesystemAccess(source, context)
if diagnostic != nil {
return diagnostic
}
return validateNativeDirectories(access, basePath, mainPath)
}
rootPath, _, _, diagnostic := importFilesystem(source, context, main) rootPath, _, _, diagnostic := importFilesystem(source, context, main)
if diagnostic != nil { if diagnostic != nil {
return diagnostic return diagnostic
@@ -29,20 +38,87 @@ func filesystemOptionDiagnostics(source Source, context *FilesystemContext) []Di
case sourceNone: case sourceNone:
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "NoSource cannot use filesystem context")} return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "NoSource cannot use filesystem context")}
case sourceFile: case sourceFile:
if context.FS != nil {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "host file sources cannot use a native filesystem context")}
}
if context.BaseDir != "" { if context.BaseDir != "" {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "file sources determine their own import base directory")} return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "file sources determine their own import base directory")}
} }
case sourceBytes, sourceReader: case sourceFilesystem:
if context.FS == nil || context.FS.namespace == nil || source.namespace == nil || context.FS.namespace.token != source.namespace.token {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "filesystem file context must use the source namespace")}
}
if context.BaseDir != "" {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "file sources determine their own import base directory")}
}
if context.Root != "" && !fs.ValidPath(context.Root) {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "filesystem root must be a slash-relative valid path")}
}
case sourceBytes, sourceReader, sourceMapping:
if context.BaseDir == "" || context.Root == "" { if context.BaseDir == "" || context.Root == "" {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem context requires base and root directories")} return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem context requires base and root directories")}
} }
if !filepath.IsAbs(context.BaseDir) || !filepath.IsAbs(context.Root) { if context.FS != nil {
if context.FS.namespace == nil || !fs.ValidPath(context.BaseDir) || !fs.ValidPath(context.Root) {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "filesystem base and root must be slash-relative valid paths")}
}
} else if !filepath.IsAbs(context.BaseDir) || !filepath.IsAbs(context.Root) {
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem base and root directories must be absolute")} return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem base and root directories must be absolute")}
} }
} }
return nil return nil
} }
func nativeContext(source Source, context *FilesystemContext) bool {
return source.kind == sourceFilesystem || context != nil && context.FS != nil
}
// nativeFilesystemAccess resolves namespace context without opening caller
// resources. File inputs always derive their base from the selected file.
func nativeFilesystemAccess(source Source, context *FilesystemContext) (*filesystemAccess, string, string, *Diagnostic) {
var namespace *nativeNamespace
rootPath := "."
basePath := "."
mainPath := ""
if source.kind == sourceFilesystem {
namespace = source.namespace
mainPath = source.path
basePath = path.Dir(source.path)
if context != nil && context.Root != "" {
rootPath = context.Root
}
} else if context != nil && context.FS != nil {
namespace = context.FS.namespace
rootPath, basePath = context.Root, context.BaseDir
}
if namespace == nil || nilFilesystem(namespace.filesystem) || !fs.ValidPath(rootPath) || !fs.ValidPath(basePath) {
value := optionDiagnostic(CodeOptionsInvalid, "filesystem context is invalid")
return nil, "", "", &value
}
access := newNativeAccess(namespace.filesystem, rootPath, namespace.token)
access.label = namespace.label
if !access.contains(basePath) || mainPath != "" && !access.contains(mainPath) {
value := importDiagnostic(CodeImportOutsideRoot, "selected source and import base must stay inside the configuration root", nil)
return nil, "", "", &value
}
return access, basePath, mainPath, nil
}
func validateNativeDirectories(access *filesystemAccess, basePath, mainPath string) *Diagnostic {
if !access.contains(basePath) || mainPath != "" && !access.contains(mainPath) {
value := importDiagnostic(CodeImportOutsideRoot, "selected source and import base must stay inside the configuration root", nil)
return &value
}
for _, directory := range []string{access.rootPath, basePath} {
info, err := access.stat(directory)
if err != nil || !info.IsDir() {
value := importDiagnostic(CodeImportInvalid, "configuration root and import base must be directories", nil)
return &value
}
}
return nil
}
func importFilesystem(source Source, context *FilesystemContext, main parsedSource) (rootPath, basePath, mainPath string, diagnostic *Diagnostic) { func importFilesystem(source Source, context *FilesystemContext, main parsedSource) (rootPath, basePath, mainPath string, diagnostic *Diagnostic) {
if main.source == nil || main.document == nil { if main.source == nil || main.document == nil {
value := importDiagnostic(CodeImportInvalid, "imports require a selected configuration source", nil) value := importDiagnostic(CodeImportInvalid, "imports require a selected configuration source", nil)

View File

@@ -0,0 +1,185 @@
package engine
import (
"errors"
"io"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
)
// namespaceToken identifies one caller-selected filesystem without comparing
// arbitrary fs.FS values. A token can be shared by copies of a source wrapper.
type namespaceToken struct{ marker byte }
type nativeNamespace struct {
filesystem fs.FS
label string
token *namespaceToken
}
func newNativeNamespace(label string, filesystem fs.FS) *nativeNamespace {
return &nativeNamespace{filesystem: filesystem, label: label, token: &namespaceToken{}}
}
type resourceIdentity struct {
namespace *namespaceToken
path string
occurrence *namespaceToken
}
// filesystemAccess is a load-scoped view of either a contained host root or a
// borrowed native filesystem. Names are canonical host paths or fs.FS paths.
type filesystemAccess struct {
rootPath string
root *os.Root
native fs.FS
token *namespaceToken
label string
}
func openHostAccess(rootPath string) (*filesystemAccess, error) {
root, err := os.OpenRoot(rootPath)
if err != nil {
return nil, err
}
return &filesystemAccess{rootPath: rootPath, root: root, token: &namespaceToken{}}, nil
}
func newNativeAccess(filesystem fs.FS, root string, token *namespaceToken) *filesystemAccess {
return &filesystemAccess{rootPath: root, native: filesystem, token: token}
}
func (a *filesystemAccess) close() error {
if a.root != nil {
return a.root.Close()
}
return nil
}
func (a *filesystemAccess) identity(name string) resourceIdentity {
return resourceIdentity{namespace: a.token, path: name}
}
func (a *filesystemAccess) syntheticIdentity() resourceIdentity {
return resourceIdentity{namespace: a.token, occurrence: &namespaceToken{}}
}
func (a *filesystemAccess) join(base, name string) string {
if a.native != nil {
return path.Join(base, name)
}
return filepath.Join(base, filepath.FromSlash(path.Clean(name)))
}
func (a *filesystemAccess) dir(name string) string {
if a.native != nil {
return path.Dir(name)
}
return filepath.Dir(name)
}
func (a *filesystemAccess) contains(name string) bool {
if a.native == nil {
return containedPath(a.rootPath, filepath.Clean(name))
}
if !fs.ValidPath(name) || !fs.ValidPath(a.rootPath) {
return false
}
return a.rootPath == "." || name == a.rootPath || strings.HasPrefix(name, a.rootPath+"/")
}
func (a *filesystemAccess) canonical(name string) (string, error) {
if a.native == nil {
return canonicalExistingPath(name)
}
if !fs.ValidPath(name) {
return "", fs.ErrInvalid
}
_, err := fs.Stat(a.native, name)
if err != nil {
return "", err
}
return name, nil
}
func (a *filesystemAccess) relative(name string) (string, error) {
if a.native != nil {
if !a.contains(name) {
return "", fs.ErrInvalid
}
return name, nil
}
relative, err := filepath.Rel(a.rootPath, name)
if err != nil || filepath.IsAbs(relative) || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return "", fs.ErrInvalid
}
return relative, nil
}
func (a *filesystemAccess) stat(name string) (fs.FileInfo, error) {
if a.native != nil {
return fs.Stat(a.native, name)
}
relative, err := a.relative(name)
if err != nil {
return nil, err
}
return a.root.Stat(relative)
}
func (a *filesystemAccess) open(name string) (fs.File, error) {
if a.native != nil {
return a.native.Open(name)
}
relative, err := a.relative(name)
if err != nil {
return nil, err
}
return a.root.Open(relative)
}
func (a *filesystemAccess) readDirectory(name string, visit func(fs.DirEntry) bool) error {
file, err := a.open(name)
if err != nil {
return err
}
if reader, ok := file.(fs.ReadDirFile); ok {
for {
entries, readErr := reader.ReadDir(globReadBatch)
for _, entry := range entries {
if !visit(entry) {
return file.Close()
}
}
if errors.Is(readErr, io.EOF) {
return file.Close()
}
if readErr != nil {
_ = file.Close()
return readErr
}
if len(entries) == 0 {
_ = file.Close()
return io.ErrNoProgress
}
}
}
if err := file.Close(); err != nil {
return err
}
// fs.ReadDir provides the standard fallback for implementations whose opened
// files do not implement fs.ReadDirFile. It may allocate all entries itself.
entries, err := fs.ReadDir(a.native, name)
if err != nil {
return err
}
for _, entry := range entries {
if !visit(entry) {
break
}
}
return nil
}

View File

@@ -2,11 +2,8 @@ package engine
import ( import (
"errors" "errors"
"io"
"io/fs" "io/fs"
"os"
"path" "path"
"path/filepath"
"slices" "slices"
"strings" "strings"
) )
@@ -42,29 +39,29 @@ func (s *importTraversal) expandImportGlob(basePath string, declaration importDe
names = []string{literal} names = []string{literal}
} }
for _, name := range names { for _, name := range names {
absolute := filepath.Join(candidate.absolute, filepath.FromSlash(name)) absolute := s.access.join(candidate.absolute, name)
relative := path.Join(candidate.relative, name) relative := path.Join(candidate.relative, name)
if !containedPath(s.rootPath, filepath.Clean(absolute)) { if !s.access.contains(absolute) {
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal is outside the configuration root", &location)} return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal is outside the configuration root", &location)}
} }
canonical, err := canonicalExistingPath(absolute) canonical, err := s.access.canonical(absolute)
if err != nil { if err != nil {
if errors.Is(err, fs.ErrNotExist) && !patterned { if errors.Is(err, fs.ErrNotExist) && !patterned {
continue continue
} }
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob match could not be canonicalized", location)} return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob match could not be canonicalized", location)}
} }
if !containedPath(s.rootPath, canonical) { if !s.access.contains(canonical) {
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal resolves outside the configuration root", &location)} return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal resolves outside the configuration root", &location)}
} }
if last { if last {
next = append(next, globCandidate{absolute: canonical, relative: relative}) next = append(next, globCandidate{absolute: canonical, relative: relative})
if len(s.sources)+len(next) > s.limits.sources { if s.accounting.sources+len(next) > s.limits.sources {
return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob matches exceed the remaining source-count limit", &location)} return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob matches exceed the remaining source-count limit", &location)}
} }
continue continue
} }
info, err := os.Stat(canonical) info, err := s.access.stat(canonical)
if err != nil { if err != nil {
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob traversal target could not be inspected", location)} return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob traversal target could not be inspected", location)}
} }
@@ -88,56 +85,47 @@ func (s *importTraversal) expandImportGlob(basePath string, declaration importDe
} }
func (s *importTraversal) readGlobMatches(directory, pattern string, location Location) ([]string, []Diagnostic) { func (s *importTraversal) readGlobMatches(directory, pattern string, location Location) ([]string, []Diagnostic) {
canonical, err := canonicalExistingPath(directory) canonical, err := s.access.canonical(directory)
if err != nil { if err != nil {
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
return nil, nil return nil, nil
} }
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be canonicalized", location)} return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be canonicalized", location)}
} }
if !containedPath(s.rootPath, canonical) { if !s.access.contains(canonical) {
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory resolves outside the configuration root", &location)} return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory resolves outside the configuration root", &location)}
} }
relative, err := filepath.Rel(s.rootPath, canonical) if _, err := s.access.relative(canonical); err != nil {
if err != nil || filepath.IsAbs(relative) || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory could not be addressed inside the configuration root", &location)} return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory could not be addressed inside the configuration root", &location)}
} }
directoryFile, err := s.root.Open(relative)
if err != nil {
if errors.Is(err, fs.ErrNotExist) {
return nil, nil
}
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be opened", location)}
}
var matches []string var matches []string
for { var problem *Diagnostic
entries, readErr := directoryFile.ReadDir(globReadBatch) err = s.access.readDirectory(canonical, func(entry fs.DirEntry) bool {
for _, entry := range entries { s.accounting.entries++
s.usedEntries++ if s.accounting.entries > s.limits.entries {
if s.usedEntries > s.limits.entries { value := importDiagnostic(CodeImportLimitExceeded, "glob traversal exceeds the directory-entry visit limit", &location)
_ = directoryFile.Close() problem = &value
return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob traversal exceeds the directory-entry visit limit", &location)} return false
} }
matched, matchErr := path.Match(pattern, entry.Name()) matched, matchErr := path.Match(pattern, entry.Name())
if matchErr != nil { if matchErr != nil {
_ = directoryFile.Close() value := importDiagnostic(CodeImportInvalid, "import glob pattern is invalid", &location)
return nil, []Diagnostic{importDiagnostic(CodeImportInvalid, "import glob pattern is invalid", &location)} problem = &value
return false
} }
if matched { if matched {
matches = append(matches, entry.Name()) matches = append(matches, entry.Name())
} }
return true
})
if problem != nil {
return nil, []Diagnostic{*problem}
} }
if errors.Is(readErr, io.EOF) { if errors.Is(err, fs.ErrNotExist) {
break return nil, nil
} }
if readErr != nil { if err != nil {
_ = directoryFile.Close() return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be traversed", location)}
return nil, []Diagnostic{globFilesystemDiagnostic(readErr, "glob directory could not be read", location)}
}
}
if err := directoryFile.Close(); err != nil {
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be closed", location)}
} }
slices.Sort(matches) slices.Sort(matches)
return matches, nil return matches, nil

View File

@@ -6,6 +6,40 @@ import (
const historyNodeLimit = 4_000_000 const historyNodeLimit = 4_000_000
type retainedNodeCounter struct {
seen map[*document.Node]bool
count int
limit int
}
func newRetainedNodeCounter(limit int) *retainedNodeCounter {
return &retainedNodeCounter{seen: make(map[*document.Node]bool), limit: limit}
}
func (c *retainedNodeCounter) add(root *document.Node) bool {
if c == nil {
return true
}
stack := []*document.Node{root}
for len(stack) != 0 {
node := stack[len(stack)-1]
stack = stack[:len(stack)-1]
if node == nil || c.seen[node] {
continue
}
c.seen[node] = true
c.count++
if c.limit >= 0 && c.count > c.limit {
return false
}
for _, entry := range node.Mapping {
stack = append(stack, entry.Value)
}
stack = append(stack, node.Sequence...)
}
return true
}
// History applications borrow immutable policy-validated input trees. Metadata // History applications borrow immutable policy-validated input trees. Metadata
// records answer presence and origin queries; replay reconstructs value changes. // records answer presence and origin queries; replay reconstructs value changes.
// Replay reconciles with canonical decoding before domain resolution so both // Replay reconciles with canonical decoding before domain resolution so both
@@ -83,15 +117,15 @@ func pathsIntersect(left, right Path) bool {
return pathContains(left, right) || pathContains(right, left) return pathContains(left, right) || pathContains(right, left)
} }
func applyHistoryApplication(root **document.Node, application historyApplication) { func applyHistoryApplication(root **document.Node, application historyApplication, index *mappingIndex) {
if application.path.String() == "" { if application.path.String() == "" {
mergeHistoryDocument(root, application.value) mergeHistoryDocument(root, application.value, index)
return return
} }
mergeHistoryAtPath(root, application.path, application.value) mergeHistoryAtPath(root, application.path, application.value, index)
} }
func mergeHistoryDocument(target **document.Node, incoming *document.Node) { func mergeHistoryDocument(target **document.Node, incoming *document.Node, index *mappingIndex) {
if incoming == nil { if incoming == nil {
return return
} }
@@ -100,39 +134,39 @@ func mergeHistoryDocument(target **document.Node, incoming *document.Node) {
return return
} }
for _, entry := range incoming.Mapping { for _, entry := range incoming.Mapping {
position := documentEntryIndex((*target).Mapping, entry.Key) position := index.find(*target, entry.Key)
if position < 0 { if position < 0 {
cloned := entry cloned := entry
cloned.Value = document.Clone(entry.Value) cloned.Value = document.Clone(entry.Value)
(*target).Mapping = append((*target).Mapping, cloned) index.append(*target, cloned)
continue continue
} }
mergeHistoryDocument(&(*target).Mapping[position].Value, entry.Value) mergeHistoryDocument(&(*target).Mapping[position].Value, entry.Value, index)
} }
} }
func mergeHistoryAtPath(root **document.Node, path Path, incoming *document.Node) { func mergeHistoryAtPath(root **document.Node, path Path, incoming *document.Node, index *mappingIndex) {
if root == nil || *root == nil || len(path.segments) == 0 { if root == nil || *root == nil || len(path.segments) == 0 {
mergeHistoryDocument(root, incoming) mergeHistoryDocument(root, incoming, index)
return return
} }
current := *root current := *root
for index, segment := range path.segments { for segmentIndex, segment := range path.segments {
if current.Kind != document.Mapping || segment.kind == pathSegmentIndex { if current.Kind != document.Mapping || segment.kind == pathSegmentIndex {
return return
} }
position := documentEntryIndex(current.Mapping, segment.key) position := index.find(current, segment.key)
if index == len(path.segments)-1 { if segmentIndex == len(path.segments)-1 {
if position < 0 { if position < 0 {
current.Mapping = append(current.Mapping, document.Entry{Key: segment.key, Value: document.Clone(incoming)}) index.append(current, document.Entry{Key: segment.key, Value: document.Clone(incoming)})
} else { } else {
mergeHistoryDocument(&current.Mapping[position].Value, incoming) mergeHistoryDocument(&current.Mapping[position].Value, incoming, index)
} }
return return
} }
if position < 0 { if position < 0 {
child := &document.Node{Kind: document.Mapping} child := &document.Node{Kind: document.Mapping}
current.Mapping = append(current.Mapping, document.Entry{Key: segment.key, Value: child}) index.append(current, document.Entry{Key: segment.key, Value: child})
current = child current = child
} else { } else {
current = current.Mapping[position].Value current = current.Mapping[position].Value
@@ -230,11 +264,11 @@ func replaceHistoryPath(root **document.Node, path Path, replacement *document.N
} }
} }
func historyNodesWithinLimit(baseline, input, final *document.Node, applications []historyApplication, resolutions []resolutionSnapshot) bool { func historyNodesWithinLimit(baseline, input, final *document.Node, applications []historyApplication, resolutions []resolutionSnapshot, inherited ...*inheritanceScope) bool {
return historyNodesWithin(historyNodeLimit, baseline, input, final, applications, resolutions) return historyNodesWithin(historyNodeLimit, baseline, input, final, applications, resolutions, inherited...)
} }
func historyNodesWithin(limit int, baseline, input, final *document.Node, applications []historyApplication, resolutions []resolutionSnapshot) bool { func historyNodesWithin(limit int, baseline, input, final *document.Node, applications []historyApplication, resolutions []resolutionSnapshot, inherited ...*inheritanceScope) bool {
seen := make(map[*document.Node]bool) seen := make(map[*document.Node]bool)
count := 0 count := 0
var visit func(*document.Node) bool var visit func(*document.Node) bool
@@ -272,5 +306,23 @@ func historyNodesWithin(limit int, baseline, input, final *document.Node, applic
return false return false
} }
} }
for _, expansion := range inherited {
if expansion == nil {
continue
}
if !visit(expansion.root) {
return false
}
for _, operation := range expansion.operations {
if !visit(operation.parent) || !visit(operation.effective) {
return false
}
}
for _, replacement := range expansion.replacements {
if !visit(replacement.before) || !visit(replacement.after) {
return false
}
}
}
return true return true
} }

View File

@@ -4,9 +4,6 @@ import (
"bytes" "bytes"
"errors" "errors"
"io/fs" "io/fs"
"os"
"path"
"path/filepath"
"strconv" "strconv"
"strings" "strings"
@@ -40,47 +37,96 @@ type importRoute struct {
locations []Location locations []Location
} }
type importAccounting struct {
sources int
bytes int64
nodes int
entries int
}
type importTraversal struct { type importTraversal struct {
rootPath string access *filesystemAccess
root *os.Root accounting *importAccounting
limits importLimits limits importLimits
rule *VersionRule rule *VersionRule
documents []*document.Node documents []*document.Node
sources []SourceInfo sources []SourceInfo
active map[string]int active map[resourceIdentity]int
visited map[string]importRoute visited map[resourceIdentity]importRoute
stack []importRoute stack []importRoute
usedBytes int64
usedNodes int
usedEntries int
} }
func loadImportedDocuments(source Source, context *FilesystemContext, main parsedSource, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) { func loadImportedDocuments(source Source, context *FilesystemContext, main parsedSource, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) {
return loadImportedDocumentsTracked(source, context, main, rule, limits, &importAccounting{}, nil, nil)
}
func loadImportedDocumentsTracked(source Source, context *FilesystemContext, main parsedSource, rule *VersionRule, limits importLimits, accounting *importAccounting, visited map[resourceIdentity]importRoute, hostToken *namespaceToken) (*composedDocument, []SourceInfo, []Diagnostic) {
if nativeContext(source, context) {
access, basePath, mainPath, diagnostic := nativeFilesystemAccess(source, context)
if diagnostic != nil {
return nil, sourceInfos(main.source), []Diagnostic{*diagnostic}
}
return loadImportedWithAccessTracked(access, basePath, mainPath, main, rule, limits, accounting, visited)
}
rootPath, basePath, mainPath, diagnostic := importFilesystem(source, context, main) rootPath, basePath, mainPath, diagnostic := importFilesystem(source, context, main)
if diagnostic != nil { if diagnostic != nil {
return nil, sourceInfos(main.source), []Diagnostic{*diagnostic} return nil, sourceInfos(main.source), []Diagnostic{*diagnostic}
} }
root, err := os.OpenRoot(rootPath) access, err := openHostAccess(rootPath)
if err != nil { if err != nil {
diagnostic := globFilesystemDiagnostic(err, "configuration root could not be opened", Location{}) diagnostic := globFilesystemDiagnostic(err, "configuration root could not be opened", Location{})
diagnostic.Primary = nil diagnostic.Primary = nil
return nil, sourceInfos(main.source), []Diagnostic{diagnostic} return nil, sourceInfos(main.source), []Diagnostic{diagnostic}
} }
if hostToken != nil {
access.token = hostToken
}
return loadImportedWithAccessTracked(access, basePath, mainPath, main, rule, limits, accounting, visited)
}
func loadImportedWithAccess(access *filesystemAccess, basePath, mainPath string, main parsedSource, rule *VersionRule, limits importLimits, accounting *importAccounting) (*composedDocument, []SourceInfo, []Diagnostic) {
return loadImportedWithAccessTracked(access, basePath, mainPath, main, rule, limits, accounting, nil)
}
func loadImportedWithAccessTracked(access *filesystemAccess, basePath, mainPath string, main parsedSource, rule *VersionRule, limits importLimits, accounting *importAccounting, visited map[resourceIdentity]importRoute) (*composedDocument, []SourceInfo, []Diagnostic) {
if !access.contains(basePath) || (mainPath != "" && !access.contains(mainPath)) {
_ = access.close()
return nil, sourceInfos(main.source), []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "selected source and import base must stay inside the configuration root", nil)}
}
for _, directory := range []string{access.rootPath, basePath} {
info, err := access.stat(directory)
if err != nil || !info.IsDir() {
_ = access.close()
return nil, sourceInfos(main.source), []Diagnostic{importDiagnostic(CodeImportInvalid, "configuration root and import base must be directories", nil)}
}
}
mainInfo := cloneSourceInfo(*main.source) mainInfo := cloneSourceInfo(*main.source)
if mainPath != "" { if mainPath != "" {
mainInfo.Path = mainPath mainInfo.Path = mainPath
} }
mainKey := access.identity(mainPath)
if mainPath == "" {
mainKey = access.syntheticIdentity()
}
if previous, repeated := visited[mainKey]; repeated {
_ = access.close()
return nil, sourceInfos(main.source), []Diagnostic{importRepeatedDiagnostic(Location{SourceID: main.source.ID}, previous, importRoute{})}
}
accounting.sources++
accounting.bytes += main.bytes
accounting.nodes += countInputNodes(main.document)
state := importTraversal{ state := importTraversal{
rootPath: rootPath, root: root, limits: limits, rule: rule, access: access, accounting: accounting, limits: limits, rule: rule,
documents: make([]*document.Node, 0, 1), documents: make([]*document.Node, 0, 1),
sources: []SourceInfo{mainInfo}, sources: []SourceInfo{mainInfo},
active: make(map[string]int), visited: make(map[string]importRoute), active: make(map[resourceIdentity]int), visited: visited,
usedBytes: main.bytes, usedNodes: countInputNodes(main.document), }
if state.visited == nil {
state.visited = make(map[resourceIdentity]importRoute)
} }
if diagnostic := state.checkInitialLimits(); diagnostic != nil { if diagnostic := state.checkInitialLimits(); diagnostic != nil {
_ = root.Close() _ = access.close()
return nil, cloneSourceInfos(state.sources), []Diagnostic{*diagnostic} return nil, cloneSourceInfos(state.sources), []Diagnostic{*diagnostic}
} }
@@ -89,8 +135,8 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse
mainIdentity = "<selected-source>" mainIdentity = "<selected-source>"
} }
mainRoute := importRoute{canonical: mainIdentity} mainRoute := importRoute{canonical: mainIdentity}
state.active[mainIdentity] = 0 state.active[mainKey] = 0
state.visited[mainIdentity] = mainRoute state.visited[mainKey] = mainRoute
state.stack = append(state.stack, mainRoute) state.stack = append(state.stack, mainRoute)
body, declarations, diagnostics := splitImportDirectives(main.document) body, declarations, diagnostics := splitImportDirectives(main.document)
if len(diagnostics) == 0 { if len(diagnostics) == 0 {
@@ -98,7 +144,7 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse
diagnostics = state.walkDeclarations(basePath, declarations, 0) diagnostics = state.walkDeclarations(basePath, declarations, 0)
} }
closeErr := root.Close() closeErr := access.close()
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return nil, cloneSourceInfos(state.sources), diagnostics return nil, cloneSourceInfos(state.sources), diagnostics
} }
@@ -107,22 +153,22 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse
} }
composed, diagnostics := composeDocuments(state.documents...) composed, diagnostics := composeDocuments(state.documents...)
if composed != nil { if composed != nil {
composed.inputBytes = state.usedBytes composed.inputBytes = accounting.bytes
composed.inputNodes = state.usedNodes composed.inputNodes = accounting.nodes
} }
return composed, cloneSourceInfos(state.sources), diagnostics return composed, cloneSourceInfos(state.sources), diagnostics
} }
func (s *importTraversal) checkInitialLimits() *Diagnostic { func (s *importTraversal) checkInitialLimits() *Diagnostic {
if s.limits.sources < 1 { if s.accounting.sources > s.limits.sources {
value := importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the source-count limit", nil) value := importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the source-count limit", nil)
return &value return &value
} }
if s.usedBytes > s.limits.bytes { if s.accounting.bytes > s.limits.bytes {
value := importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the aggregate byte limit", nil) value := importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the aggregate byte limit", nil)
return &value return &value
} }
if s.usedNodes > s.limits.nodes { if s.accounting.nodes > s.limits.nodes {
value := importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the aggregate node limit", nil) value := importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the aggregate node limit", nil)
return &value return &value
} }
@@ -169,11 +215,11 @@ func (s *importTraversal) visitPath(basePath string, declaration importDeclarati
return []Diagnostic{importDiagnostic(CodeImportInvalid, "import path must be a nonempty slash-relative path without a network scheme", &location)} return []Diagnostic{importDiagnostic(CodeImportInvalid, "import path must be a nonempty slash-relative path without a network scheme", &location)}
} }
candidate := filepath.Join(basePath, filepath.FromSlash(path.Clean(declaration.path))) candidate := s.access.join(basePath, declaration.path)
if !containedPath(s.rootPath, filepath.Clean(candidate)) { if !s.access.contains(candidate) {
return []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "import target is outside the configuration root", &location)} return []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "import target is outside the configuration root", &location)}
} }
canonical, err := canonicalExistingPath(candidate) canonical, err := s.access.canonical(candidate)
if err != nil { if err != nil {
if declaration.optional && errors.Is(err, fs.ErrNotExist) { if declaration.optional && errors.Is(err, fs.ErrNotExist) {
return nil return nil
@@ -187,30 +233,39 @@ func (s *importTraversal) visitPath(basePath string, declaration importDeclarati
} }
return []Diagnostic{importDiagnostic(code, message, &location)} return []Diagnostic{importDiagnostic(code, message, &location)}
} }
if !containedPath(s.rootPath, canonical) { if !s.access.contains(canonical) {
return []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "import target resolves outside the configuration root", &location)} return []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "import target resolves outside the configuration root", &location)}
} }
currentRoute := s.currentRoute(location) currentRoute := s.currentRoute(location)
if activeIndex, ok := s.active[canonical]; ok { identity := s.access.identity(canonical)
if activeIndex, ok := s.active[identity]; ok {
cycle := append([]importRoute(nil), s.stack[activeIndex:]...) cycle := append([]importRoute(nil), s.stack[activeIndex:]...)
return []Diagnostic{importCycleDiagnostic(canonical, location, cycle)} return []Diagnostic{importCycleDiagnostic(canonical, location, cycle)}
} }
if previous, ok := s.visited[canonical]; ok { if previous, ok := s.visited[identity]; ok {
return []Diagnostic{importRepeatedDiagnostic(location, previous, currentRoute)} return []Diagnostic{importRepeatedDiagnostic(location, previous, currentRoute)}
} }
if len(s.sources)+1 > s.limits.sources { if s.accounting.sources+1 > s.limits.sources {
return []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the source-count limit", &location)} return []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the source-count limit", &location)}
} }
relative, err := filepath.Rel(s.rootPath, canonical) if _, err := s.access.relative(canonical); err != nil {
if err != nil || filepath.IsAbs(relative) || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "import target could not be addressed inside the configuration root", &location)} return []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "import target could not be addressed inside the configuration root", &location)}
} }
sourceID := "source-" + strconv.Itoa(len(s.sources)+1) sourceID := "source-" + strconv.Itoa(s.accounting.sources+1)
info := SourceInfo{ID: sourceID, Label: declaration.path, Path: canonical, Layer: LayerBase, ImportedFrom: &location} kind := SourceKindFile
label := declaration.path
if s.access.native != nil {
kind = SourceKindFilesystem
label = s.access.label + "/" + canonical
}
info := SourceInfo{ID: sourceID, Label: label, Path: canonical, Layer: LayerBase, Kind: kind, ImportedFrom: &location}
if kind == SourceKindFilesystem {
info.namespace = s.access.token
}
s.sources = append(s.sources, info) s.sources = append(s.sources, info)
data, skipped, diagnostic := s.readImport(relative, sourceID, declaration.optional) data, skipped, diagnostic := s.readImport(canonical, sourceID, declaration.optional)
if diagnostic != nil { if diagnostic != nil {
return []Diagnostic{*diagnostic} return []Diagnostic{*diagnostic}
} }
@@ -218,6 +273,7 @@ func (s *importTraversal) visitPath(basePath string, declaration importDeclarati
s.sources = s.sources[:len(s.sources)-1] s.sources = s.sources[:len(s.sources)-1]
return nil return nil
} }
s.accounting.sources++
root, issues := yamladapter.Parse(bytes.NewReader(data), sourceID, yamladapter.Limits{ root, issues := yamladapter.Parse(bytes.NewReader(data), sourceID, yamladapter.Limits{
Depth: maxDocumentDepth, Nodes: maxDocumentNodes, Depth: maxDocumentDepth, Nodes: maxDocumentNodes,
@@ -226,10 +282,10 @@ func (s *importTraversal) visitPath(basePath string, declaration importDeclarati
return parserDiagnostics(sourceID, issues) return parserDiagnostics(sourceID, issues)
} }
nodeCount := countInputNodes(root) nodeCount := countInputNodes(root)
if s.usedNodes+nodeCount > s.limits.nodes { if s.accounting.nodes+nodeCount > s.limits.nodes {
return []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the aggregate node limit", &location)} return []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "configuration exceeds the aggregate node limit", &location)}
} }
s.usedNodes += nodeCount s.accounting.nodes += nodeCount
body, declarations, diagnostics := splitImportDirectives(root) body, declarations, diagnostics := splitImportDirectives(root)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
@@ -245,22 +301,22 @@ func (s *importTraversal) visitPath(basePath string, declaration importDeclarati
route := importRoute{canonical: canonical, locations: currentRoute.locations} route := importRoute{canonical: canonical, locations: currentRoute.locations}
s.documents = append(s.documents, body) s.documents = append(s.documents, body)
s.active[canonical] = len(s.stack) s.active[identity] = len(s.stack)
s.visited[canonical] = route s.visited[identity] = route
s.stack = append(s.stack, route) s.stack = append(s.stack, route)
diagnostics = s.walkDeclarations(filepath.Dir(canonical), declarations, depth) diagnostics = s.walkDeclarations(s.access.dir(canonical), declarations, depth)
delete(s.active, canonical) delete(s.active, identity)
s.stack = s.stack[:len(s.stack)-1] s.stack = s.stack[:len(s.stack)-1]
return diagnostics return diagnostics
} }
func (s *importTraversal) readImport(relative, sourceID string, optional bool) ([]byte, bool, *Diagnostic) { func (s *importTraversal) readImport(name, sourceID string, optional bool) ([]byte, bool, *Diagnostic) {
remaining := s.limits.bytes - s.usedBytes remaining := s.limits.bytes - s.accounting.bytes
limit := min(int64(maxSourceBytes), remaining) limit := min(int64(maxSourceBytes), remaining)
if limit < 0 { if limit < 0 {
limit = 0 limit = 0
} }
data, failure := readRegularFile(relative, s.root.Stat, s.root.Open, limit) data, failure := readRegularFile(name, s.access.stat, s.access.open, limit)
if failure != nil { if failure != nil {
if optional && failure.missingTarget { if optional && failure.missingTarget {
return nil, true, nil return nil, true, nil
@@ -268,7 +324,7 @@ func (s *importTraversal) readImport(relative, sourceID string, optional bool) (
value := sourceReadDiagnostic(failure, sourceID, CodeImportLimitExceeded, "configuration exceeds a source or aggregate byte limit") value := sourceReadDiagnostic(failure, sourceID, CodeImportLimitExceeded, "configuration exceeds a source or aggregate byte limit")
return nil, false, &value return nil, false, &value
} }
s.usedBytes += int64(len(data)) s.accounting.bytes += int64(len(data))
return data, false, nil return data, false, nil
} }

View File

@@ -0,0 +1,313 @@
package engine
import (
"slices"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
const inheritanceKey = "extends"
// These private records keep structural evidence separate from acquired source
// records. Later inspection can project a transfer onto individual descendants.
type inheritanceTransfer struct {
from Path
to Path
reference Path
locations []Location
}
type inheritanceControl struct {
path Path
locations []Location
cancelled bool
}
type inheritanceScope struct {
root *document.Node
origins map[string]Origin
transfers []inheritanceTransfer
controls []inheritanceControl
replacements []inheritanceReplacement
operations []inheritanceOperation
}
type inheritanceOperation struct {
control *inheritanceControl
transfer *inheritanceTransfer
controlOrigin Origin
parent *document.Node
effective *document.Node
parentOrigins map[string]Origin
effectiveOrigins map[string]Origin
}
// A replacement retains the displaced source/derivation record while the
// effective origin belongs to the child. Value trees are borrowed immutable
// inputs or detached templates and are budgeted with retained history later.
type inheritanceReplacement struct {
path Path
before *document.Node
after *document.Node
previous []displacedDefinition
effective Origin
}
// A link represents one shared relationship edge. Each contributor retains
// its original semantic path independently of source line/column coordinates.
type inheritanceTrace struct {
previous *inheritanceTrace
transfer *inheritanceTransfer
}
type inheritanceContributor struct {
source Path
location Location
trace *inheritanceTrace
}
type siblingFrame struct {
name string
parent string
hasParent bool
entered bool
}
// expandSiblingDefinitions resolves only the mappings directly in scope.
// Nested instructions remain in the returned templates for destination-local
// resolution. This private entry point does not participate in public loading.
func expandSiblingDefinitions(scope *document.Node, scopePath Path) (*inheritanceScope, []Diagnostic) {
return expandSiblingDefinitionsWithLimits(scope, scopePath, defaultProjectionLimits())
}
func expandSiblingDefinitionsWithLimits(scope *document.Node, scopePath Path, limits projectionLimits) (*inheritanceScope, []Diagnostic) {
return expandSiblingDefinitionsWithOrigins(scope, scopePath, limits, nil)
}
func expandSiblingDefinitionsWithOrigins(scope *document.Node, scopePath Path, limits projectionLimits, inputOrigins map[string]Origin, budget ...*retainedNodeCounter) (*inheritanceScope, []Diagnostic) {
if scope == nil || scope.Kind != document.Mapping {
return nil, []Diagnostic{{
Code: CodeInheritanceInvalid, Severity: SeverityError, Path: scopePath,
Message: "inheritance scope must be a mapping",
}}
}
byName := make(map[string]document.Entry, len(scope.Mapping))
names := make([]string, 0, len(scope.Mapping))
for _, entry := range scope.Mapping {
byName[entry.Key] = entry
names = append(names, entry.Key)
}
slices.Sort(names)
resolved := make(map[string]*document.Node, len(byName))
resolvedOrigins := make(map[string]map[string]Origin, len(byName))
state := make(map[string]uint8, len(byName))
result := &inheritanceScope{}
counter := newRetainedNodeCounter(historyNodeLimit)
if len(budget) != 0 {
counter = budget[0]
}
retained := 1 // scope mapping node
for _, entry := range scope.Mapping {
if entry.Value.Kind == document.Mapping {
continue
}
remaining := limits.nodes - retained - 1
if limits.nodes >= 0 && remaining < 0 {
return nil, []Diagnostic{inheritanceBudgetDiagnostic(scopePath.Key(entry.Key), entry.Value.Location)}
}
count, within := inheritanceLowerBoundCount(entry.Value, remaining)
if !within {
return nil, []Diagnostic{inheritanceBudgetDiagnostic(scopePath.Key(entry.Key), entry.Value.Location)}
}
retained += 1 + count // key and value subtree
}
for _, name := range names {
if byName[name].Value.Kind != document.Mapping || state[name] == 2 {
continue
}
stack := []siblingFrame{{name: name}}
for len(stack) != 0 {
frame := &stack[len(stack)-1]
entry := byName[frame.name]
path := scopePath.Key(frame.name)
if !frame.entered {
state[frame.name] = 1
frame.entered = true
reference, found := inheritanceDirective(entry.Value)
if found {
referencePath := path.Key(inheritanceKey)
location := publicLocation(reference.Value.Location)
result.controls = append(result.controls, inheritanceControl{
path: referencePath, locations: []Location{location},
cancelled: reference.Value.Kind == document.Null,
})
switch reference.Value.Kind {
case document.Null:
case document.String:
frame.parent, frame.hasParent = reference.Value.Text, true
parent, exists := byName[frame.parent]
if !exists {
return nil, []Diagnostic{inheritanceDiagnostic(CodeInheritanceMissing, referencePath, location, nil,
"referenced sibling does not exist")}
}
if parent.Value.Kind != document.Mapping {
related := []Location{publicLocation(parent.Value.Location)}
return nil, []Diagnostic{inheritanceDiagnostic(CodeInheritanceInvalid, referencePath, location, related,
"referenced sibling is not a mapping")}
}
if state[frame.parent] == 1 {
related := []Location{publicLocation(parent.Value.Location)}
return nil, []Diagnostic{inheritanceDiagnostic(CodeInheritanceCycle, referencePath, location, related,
"sibling inheritance contains a cycle")}
}
if state[frame.parent] == 0 {
stack = append(stack, siblingFrame{name: frame.parent})
continue
}
default:
return nil, []Diagnostic{inheritanceDiagnostic(CodeInheritanceInvalid, referencePath, location, nil,
"inheritance reference must be a sibling name or null")}
}
}
}
payload := withoutInheritanceDirective(entry.Value)
operationStart, replacementStart := len(result.operations), len(result.replacements)
childOrigins := directInheritanceOrigins(payload, path, inputOrigins)
if reference, found := inheritanceDirective(entry.Value); found {
control := inheritanceControl{
path: path.Key(inheritanceKey),
locations: []Location{publicLocation(reference.Value.Location)},
cancelled: reference.Value.Kind == document.Null,
}
controlOrigins := directInheritanceOrigins(reference.Value, control.path, inputOrigins)
result.operations = append(result.operations, inheritanceOperation{
control: &control, controlOrigin: controlOrigins[control.path.String()],
})
}
if limits.nodes >= 0 {
remaining := limits.nodes - retained - 1 // the sibling key
within := false
if frame.hasParent {
within = inheritanceOverlayLowerBoundWithin(resolved[frame.parent], payload, remaining)
} else {
within = inheritanceLowerBoundWithin(payload, remaining)
}
if !within {
return nil, []Diagnostic{inheritanceBudgetDiagnostic(path, entry.Value.Location)}
}
}
if frame.hasParent {
parent := resolved[frame.parent]
merged := document.Clone(parent)
control := inheritanceDirectiveLocation(entry.Value)
transfer := inheritanceTransfer{
from: scopePath.Key(frame.parent), to: path,
reference: path.Key(inheritanceKey), locations: []Location{control},
}
mergedOrigins := transferredInheritanceOrigins(resolvedOrigins[frame.parent], transfer.from, path, transfer)
mergeInheritanceOrigin(mergedOrigins, path, childOrigins[path.String()])
mergeRecursiveMappings(merged, payload, path, new(mappingIndex), recursiveMergeHooks{
added: func(childPath Path, incoming *document.Node) {
copyInheritanceSubtree(mergedOrigins, childOrigins, incoming, childPath)
},
combined: func(childPath Path, _ *document.Node) {
mergeInheritanceOrigin(mergedOrigins, childPath, childOrigins[childPath.String()])
},
replaced: func(childPath Path, existing, incoming *document.Node) {
previous := displacedOrigins(existing, mergedOrigins, childPath)
result.replacements = append(result.replacements, inheritanceReplacement{
path: childPath.clone(), before: existing, after: incoming,
previous: previous, effective: cloneOrigin(childOrigins[childPath.String()]),
})
deleteOriginSubtree(mergedOrigins, childPath, existing)
copyInheritanceSubtree(mergedOrigins, childOrigins, incoming, childPath)
},
})
merged.Location = entry.Value.Location
resolved[frame.name] = merged
resolvedOrigins[frame.name] = mergedOrigins
result.transfers = append(result.transfers, transfer)
result.operations = append(result.operations, inheritanceOperation{
transfer: &transfer, parent: parent, effective: merged,
parentOrigins: resolvedOrigins[frame.parent], effectiveOrigins: mergedOrigins,
})
} else {
resolved[frame.name] = document.Clone(payload)
resolvedOrigins[frame.name] = childOrigins
}
count, _ := inheritanceLowerBoundCount(resolved[frame.name], limits.nodes)
retained += 1 + count
for _, operation := range result.operations[operationStart:] {
if !counter.add(operation.parent) || !counter.add(operation.effective) {
return nil, []Diagnostic{inheritanceHistoryBudgetDiagnostic(path, entry.Value.Location)}
}
}
for _, replacement := range result.replacements[replacementStart:] {
if !counter.add(replacement.before) || !counter.add(replacement.after) {
return nil, []Diagnostic{inheritanceHistoryBudgetDiagnostic(path, entry.Value.Location)}
}
}
state[frame.name] = 2
stack = stack[:len(stack)-1]
}
}
root := *scope
root.Mapping = make([]document.Entry, len(scope.Mapping))
rootOrigin := directInheritanceOrigins(&document.Node{Kind: document.Mapping, Location: scope.Location}, scopePath, inputOrigins)
result.origins = rootOrigin
for index, entry := range scope.Mapping {
root.Mapping[index] = entry
if resolved[entry.Key] != nil {
root.Mapping[index].Value = resolved[entry.Key]
for pointer, origin := range resolvedOrigins[entry.Key] {
result.origins[pointer] = origin
}
} else {
root.Mapping[index].Value = document.Clone(entry.Value)
for pointer, origin := range directInheritanceOrigins(entry.Value, scopePath.Key(entry.Key), inputOrigins) {
result.origins[pointer] = origin
}
}
}
result.root = &root
if !inheritanceLowerBoundWithin(result.root, limits.nodes) {
return nil, []Diagnostic{inheritanceBudgetDiagnostic(scopePath, scope.Location)}
}
return result, nil
}
func inheritanceDirective(node *document.Node) (document.Entry, bool) {
for _, entry := range node.Mapping {
if entry.Key == inheritanceKey {
return entry, true
}
}
return document.Entry{}, false
}
func inheritanceDirectiveLocation(node *document.Node) Location {
entry, _ := inheritanceDirective(node)
return publicLocation(entry.Value.Location)
}
func withoutInheritanceDirective(node *document.Node) *document.Node {
payload := *node
payload.Mapping = make([]document.Entry, 0, len(node.Mapping))
for _, entry := range node.Mapping {
if entry.Key != inheritanceKey {
payload.Mapping = append(payload.Mapping, entry)
}
}
return &payload
}
func inheritanceDiagnostic(code Code, path Path, primary Location, related []Location, message string) Diagnostic {
return Diagnostic{
Code: code, Severity: SeverityError, Path: path, Message: message,
Primary: &primary, Related: related,
}
}

View File

@@ -0,0 +1,110 @@
package engine
import (
"strconv"
"strings"
"testing"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func BenchmarkInheritanceExpansion(b *testing.B) {
for _, shape := range []string{"chain", "wide", "nested", "replacement"} {
for _, size := range []int{16, 32, 64} {
b.Run(shape+"/"+strconv.Itoa(size), func(b *testing.B) {
input := inheritanceBenchmarkInput(shape, size)
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, diagnostics := expandInheritanceWithOrigins(input, nil, defaultProjectionLimits()); len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
}
})
}
}
}
func BenchmarkInheritanceInspectedLoad(b *testing.B) {
type entry struct {
Value string `yaml:"value"`
}
type configuration struct {
Entries map[string]entry `yaml:"entries"`
}
for _, size := range []int{16, 32, 64} {
b.Run(strconv.Itoa(size), func(b *testing.B) {
var input strings.Builder
input.WriteString("entries:\n item-0: {value: base}\n")
for index := 1; index <= size; index++ {
input.WriteString(" item-")
input.WriteString(strconv.Itoa(index))
input.WriteString(": {extends: item-0}\n")
}
source := Bytes("inheritance-benchmark", []byte(input.String()))
query := Keys("entries", "item-"+strconv.Itoa(size), "value")
b.ReportAllocs()
b.ResetTimer()
for range b.N {
loaded, err := Load[configuration](source, testSecretBinding())
if err != nil {
b.Fatal(err)
}
if _, found := loaded.Inspection.Explain(query); !found {
b.Fatal("inherited value is not explainable")
}
}
})
}
}
// Each scope has constant-sized output and evidence. This distinguishes budget
// accounting overhead from legitimate growth in expanded parent payloads.
func BenchmarkInheritanceIndependentScopes(b *testing.B) {
for _, size := range []int{256, 512, 1024} {
b.Run(strconv.Itoa(size), func(b *testing.B) {
input := &document.Node{Kind: document.Mapping}
for index := 0; index < size; index++ {
input.Mapping = append(input.Mapping, document.Entry{
Key: strconv.Itoa(index), Value: inheritanceBenchmarkInput("wide", 1),
})
}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, diagnostics := expandInheritanceTree(input, defaultProjectionLimits()); len(diagnostics) != 0 {
b.Fatal(diagnostics)
}
}
})
}
}
func inheritanceBenchmarkInput(shape string, size int) *document.Node {
value := func(text string) *document.Node { return &document.Node{Kind: document.String, Text: text} }
mapping := func(entries ...document.Entry) *document.Node {
return &document.Node{Kind: document.Mapping, Mapping: entries}
}
entry := func(key string, node *document.Node) document.Entry { return document.Entry{Key: key, Value: node} }
base := mapping(entry("value", value("base")), entry("settings", mapping(entry("enabled", &document.Node{Kind: document.Boolean, Text: "false"}))))
if shape == "nested" {
base = mapping(entry("settings", mapping(entry("first", mapping(entry("value", value("base")))), entry("second", mapping(entry("extends", value("first")))))))
}
definitions := make([]document.Entry, 0, size+1)
definitions = append(definitions, entry("item-0", base))
for index := 1; index <= size; index++ {
parent := "item-0"
if shape == "chain" {
parent = "item-" + strconv.Itoa(index-1)
}
payload := []document.Entry{entry("extends", value(parent))}
if shape == "nested" {
payload = append(payload, entry("settings", mapping(entry("first", mapping(entry("value", value("custom")))))))
}
if shape == "replacement" {
payload = append(payload, entry("value", value("replacement")))
}
definitions = append(definitions, entry("item-"+strconv.Itoa(index), mapping(payload...)))
}
return mapping(entry("entries", mapping(definitions...)))
}

View File

@@ -0,0 +1,60 @@
package engine
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document"
func appendInheritanceExplanation(explanation *Explanation, working **document.Node, query Path, operations []inheritanceOperation, secrets map[string]bool) {
for _, operation := range operations {
changes := projectedInheritanceChanges(query, operation)
if operation.control != nil {
if !pathsIntersect(operation.control.path, query) {
continue
}
before := valueSnapshot(*working, query, secrets)
replaceHistoryPath(working, operation.control.path, nil)
after := valueSnapshot(*working, query, secrets)
if len(changes) != 0 {
explanation.History = append(explanation.History, ValueChange{
Before: before, After: after, Origin: cloneOrigin(changes[0].Origin),
Dependencies: clonePaths(changes[0].Dependencies),
})
}
continue
}
if operation.transfer == nil {
continue
}
before := valueSnapshot(*working, query, secrets)
replaceHistoryPath(working, operation.transfer.to, operation.parent)
afterParent := valueSnapshot(*working, query, secrets)
parent, parentPresent := transferredOperationOrigin(operation, operationPathForQuery(operation.transfer.to, query))
if pathsIntersect(operation.transfer.to, query) && parentPresent {
parent.Operation = ChangeInheritance
parent.Inheritance = projectInheritanceSteps(parent)
explanation.History = append(explanation.History, ValueChange{
Before: before, After: afterParent, Origin: parent,
Dependencies: []Path{operation.transfer.from.clone(), operation.transfer.reference.clone()},
})
}
replaceHistoryPath(working, operation.transfer.to, operation.effective)
if !pathsIntersect(operation.transfer.to, query) {
continue
}
afterChild := valueSnapshot(*working, query, secrets)
if len(changes) != 0 {
last := changes[len(changes)-1]
if !parentPresent || len(changes) > 1 {
explanation.History = append(explanation.History, ValueChange{
Before: afterParent, After: afterChild, Origin: cloneOrigin(last.Origin),
Dependencies: clonePaths(last.Dependencies),
})
}
}
}
}
func operationPathForQuery(root, query Path) Path {
if pathContains(query, root) {
return root
}
return query
}

View File

@@ -0,0 +1,119 @@
package engine
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document"
// Retain source history at original paths. Structural operations are held once
// per relationship and projected onto a requested destination path only when
// History or Explain is queried.
func applyInheritanceMetadata(metadata *Metadata, raw *document.Node, rawOrigins map[string]Origin, overlays []overlayRecord, expansion *inheritanceScope) {
if metadata == nil || expansion == nil {
return
}
sourceView := Metadata{
fields: make(map[string]FieldInfo),
history: metadata.history,
}
walkDocument(raw, Path{}, func(path Path, node *document.Node) {
origin, found := rawOrigins[path.String()]
if !found {
origin = Origin{Layer: LayerBase, Locations: []Location{publicLocation(node.Location)}}
}
sourceView.fields[path.String()] = FieldInfo{Explicit: true, Null: node.Kind == document.Null, Origin: cloneOrigin(origin)}
})
applyOverlayMetadata(&sourceView, overlays)
metadata.history = sourceView.history
metadata.inheritance = append([]inheritanceOperation(nil), expansion.operations...)
}
func historyWithInheritance(path Path, stored []Change, operations []inheritanceOperation) []Change {
split := len(stored)
for index, change := range stored {
if change.Origin.Layer == LayerResolution {
split = index
break
}
}
history := cloneChanges(stored[:split])
for _, operation := range operations {
for _, change := range projectedInheritanceChanges(path, operation) {
if len(history) != 0 {
change.Previous = []Origin{cloneOrigin(history[len(history)-1].Origin)}
}
history = append(history, change)
}
}
for _, change := range stored[split:] {
cloned := cloneChange(change)
if len(history) != 0 {
cloned.Previous = []Origin{cloneOrigin(history[len(history)-1].Origin)}
}
history = append(history, cloned)
}
return history
}
func projectedInheritanceChanges(query Path, operation inheritanceOperation) []Change {
if operation.control != nil {
if !pathsIntersect(operation.control.path, query) {
return nil
}
origin := cloneOrigin(operation.controlOrigin)
origin.Operation = ChangeInheritance
origin.Inheritance = projectInheritanceSteps(origin)
return []Change{{Path: operation.control.path.clone(), Origin: origin,
Dependencies: []Path{operation.control.path.clone()}}}
}
if operation.transfer == nil || !pathsIntersect(operation.transfer.to, query) {
return nil
}
path := query
if pathContains(query, operation.transfer.to) {
path = operation.transfer.to
}
var changes []Change
parent, parentFound := transferredOperationOrigin(operation, path)
if parentFound {
parent.Operation = ChangeInheritance
parent.Inheritance = projectInheritanceSteps(parent)
changes = append(changes, Change{
Path: path.clone(), Origin: parent,
Dependencies: []Path{operation.transfer.from.clone(), operation.transfer.reference.clone()},
})
}
effective, effectiveFound := operation.effectiveOrigins[path.String()]
if effectiveFound {
effective = cloneOrigin(effective)
effective.Operation = ChangeInheritance
}
if effectiveFound && (!parentFound || !originContainedBy(effective, parent)) {
effective.Inheritance = projectInheritanceSteps(effective)
changes = append(changes, Change{
Path: path.clone(), Origin: effective,
Dependencies: []Path{operation.transfer.reference.clone()},
})
} else if parentFound && !effectiveFound {
removed := cloneOrigin(operation.effectiveOrigins[operation.transfer.to.String()])
removed.Operation = ChangeInheritance
removed.Inheritance = projectInheritanceSteps(removed)
changes = append(changes, Change{
Path: operation.transfer.to.clone(), Origin: removed,
Dependencies: []Path{operation.transfer.reference.clone()},
})
}
return changes
}
func transferredOperationOrigin(operation inheritanceOperation, destination Path) (Origin, bool) {
relative, ok := relativeHistoryPath(operation.transfer.to, destination)
if !ok {
return Origin{}, false
}
source := appendInheritancePath(operation.transfer.from, relative)
origin, found := operation.parentOrigins[source.String()]
if !found {
return Origin{}, false
}
transferred := transferredInheritanceOrigins(map[string]Origin{source.String(): origin},
operation.transfer.from, operation.transfer.to, *operation.transfer)
return transferred[destination.String()], true
}

View File

@@ -0,0 +1,363 @@
package engine
import (
"bytes"
"strings"
"testing"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func TestNestedInheritanceHistoryRetainsUnresolvedTemplates(t *testing.T) {
input := `presets:
base:
settings:
primary: {width: 10}
secondary: {extends: primary}
custom:
extends: base
settings:
primary: {width: 20}
grandchild:
extends: custom
settings:
primary: {width: 30}
`
loaded, err := Load[map[string]any](Bytes("nested", []byte(input)), testSecretBinding())
if err != nil {
t.Fatal(err)
}
for name, width := range map[string]string{"custom": "20", "grandchild": "30"} {
t.Run(name, func(t *testing.T) {
root := Keys("presets", name)
value, _ := loaded.Inspection.Explain(root.Key("settings").Key("secondary").Key("width"))
introduced := false
for _, change := range value.History {
if change.Origin.Operation == ChangeInheritance && !change.Before.Present && change.After.YAML == width+"\n" {
introduced = true
}
}
if !introduced || value.Effective.YAML != width+"\n" {
t.Fatalf("missing nested introduction: %#v", value)
}
controlPath := root.Key("settings").Key("secondary").Key("extends")
control, _ := loaded.Inspection.Explain(controlPath)
arrived, consumed := false, false
for _, change := range control.History {
arrived = arrived || change.After.YAML == "primary\n"
consumed = consumed || (change.Before.YAML == "primary\n" && !change.After.Present)
}
if !arrived || !consumed || control.Effective.Present {
t.Fatalf("missing control history: %#v", control)
}
container, _ := loaded.Inspection.Explain(root)
unresolved := false
for _, change := range container.History {
if change.Origin.Operation == ChangeInheritance && strings.Contains(change.After.YAML, "extends: primary") {
unresolved = true
}
}
if !unresolved || strings.Contains(container.Effective.YAML, "extends:") {
t.Fatalf("invalid enclosing snapshot: %#v", container)
}
if _, present := loaded.Inspection.Metadata().Lookup(controlPath); present {
t.Fatal("control is effective")
}
if len(loaded.Inspection.Metadata().History(controlPath)) != len(control.History) {
t.Fatal("metadata and explanation disagree")
}
})
}
}
func TestNestedInheritanceHistoryInsideSequence(t *testing.T) {
input := `base:
items:
- first: {value: inherited}
second: {extends: first}
child: {extends: base}
`
loaded, err := Load[map[string]any](Bytes("sequence", []byte(input)), testSecretBinding())
if err != nil {
t.Fatal(err)
}
path := Keys("child", "items").Index(0).Key("second")
control, _ := loaded.Inspection.Explain(path.Key("extends"))
consumed := false
for _, change := range control.History {
consumed = consumed || (change.Before.YAML == "first\n" && !change.After.Present)
}
if !consumed || control.Effective.Present {
t.Fatalf("sequence control history = %#v", control)
}
value, _ := loaded.Inspection.Explain(path.Key("value"))
introduced := false
for _, change := range value.History {
introduced = introduced || (!change.Before.Present && change.After.YAML == "inherited\n")
}
if !introduced || value.Effective.YAML != "inherited\n" {
t.Fatalf("sequence value history = %#v", value)
}
}
func inheritanceInspectionForTest(t *testing.T, sources ...*document.Node) (*Inspection, *inheritanceScope) {
t.Helper()
composed, diagnostics := composeDocuments(sources...)
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
return inheritanceInspectionFromComposition(t, composed)
}
func inheritanceInspectionFromComposition(t *testing.T, composed *composedDocument) (*Inspection, *inheritanceScope) {
t.Helper()
expanded, diagnostics := expandInheritanceWithOrigins(composed.root, composed.origins, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
baseline := &document.Node{Kind: document.Mapping}
presence := make(map[string]inputPresence)
collectPresence(expanded.root, Path{}, presence)
decoded := decodedConfiguration{
baseline: baseline, final: expanded.root, presence: presence, origins: expanded.origins,
}
metadata := metadataFromDecoded(decoded)
applyInheritanceMetadata(&metadata, composed.root, composed.origins, composed.overlays, expanded)
finalizeMetadataPaths(&metadata, expanded.root, composed.applications, baseline, composed.root)
return &Inspection{
metadata: metadata, baseline: baseline, input: expanded.root,
inputOrigins: expanded.origins, final: expanded.root,
applications: composed.applications, inheritance: expanded.operations,
}, expanded
}
func TestInheritanceHistoryFollowsFinalParentAfterSourceOverlay(t *testing.T) {
base, diagnostics := composeDocuments(parseCompositionDocument(t, "source-1", "parent: {value: original}\nchild: {extends: parent}\n"))
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
assembly := newAssemblyAccumulator(base)
if diagnostics := assembly.applyOverlay(parseCompositionDocument(t, "source-2", "parent: {value: latest}\n"), nil, nil, LayerOverlay); len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
inspection, _ := inheritanceInspectionFromComposition(t, assembly.current)
path := Keys("child", "value")
explanation, found := inspection.Explain(path)
if !found || explanation.Effective.YAML != "latest\n" ||
len(explanation.History) == 0 || explanation.History[len(explanation.History)-1].Origin.Locations[0].SourceID != "source-2" {
t.Fatalf("overlaid parent contribution = %#v, %v", explanation, found)
}
parentHistory := inspection.Metadata().History(Keys("parent", "value"))
if len(parentHistory) < 2 || parentHistory[0].Origin.Layer != LayerBase ||
parentHistory[len(parentHistory)-1].Origin.Layer != LayerOverlay {
t.Fatalf("parent source history = %#v", parentHistory)
}
}
func TestInheritanceHistoryPreservesCancellationAndSupersededDirective(t *testing.T) {
base, diagnostics := composeDocuments(parseCompositionDocument(t, "source-1", "parent: {value: one}\nchild: {extends: missing, value: own}\n"))
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
assembly := newAssemblyAccumulator(base)
if diagnostics := assembly.applyOverlay(parseCompositionDocument(t, "source-2", "child: {extends: null}\n"), nil, nil, LayerOverlay); len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
inspection, _ := inheritanceInspectionFromComposition(t, assembly.current)
control := Keys("child", inheritanceKey)
history := inspection.Metadata().History(control)
if len(history) < 3 || history[0].Origin.Locations[0].SourceID != "source-1" ||
history[1].Origin.Locations[0].SourceID != "source-2" ||
history[len(history)-1].Origin.Operation != ChangeInheritance {
t.Fatalf("superseded and cancelled directive history = %#v", history)
}
explanation, found := inspection.Explain(control)
if !found || explanation.Effective.Present || len(explanation.History) < 3 ||
explanation.History[len(explanation.History)-1].After.Present {
t.Fatalf("cancelled directive explanation = %#v, %v", explanation, found)
}
value, found := inspection.Explain(Keys("child", "value"))
if !found || value.Effective.YAML != "own\n" {
t.Fatalf("cancelled relationship changed own value = %#v, %v", value, found)
}
}
func TestInheritanceHistoryProjectsNestedDestinationValues(t *testing.T) {
inspection, _ := inheritanceInspectionForTest(t, parseCompositionDocument(t, "source-1", `base:
settings:
primary: {value: old}
secondary: {extends: primary}
custom:
extends: base
settings:
primary: {value: new}
`))
path := Keys("custom", "settings", "secondary", "value")
history := inspection.Metadata().History(path)
structural := 0
for _, change := range history {
if change.Origin.Operation == ChangeInheritance {
structural++
}
}
if structural == 0 {
t.Fatalf("nested destination history = %#v", history)
}
explanation, found := inspection.Explain(path)
if !found || explanation.Effective.YAML != "new\n" ||
len(explanation.Origin.Inheritance) == 0 ||
explanation.Origin.Inheritance[len(explanation.Origin.Inheritance)-1].To.String() != path.String() {
t.Fatalf("nested destination explanation = %#v, %v", explanation, found)
}
for _, change := range explanation.History {
if change.Origin.Operation == ChangeDecoding {
t.Fatalf("nested inheritance was attributed to decoding: %#v", explanation.History)
}
}
}
func TestInheritanceHistoryShowsParentAndEqualChildContribution(t *testing.T) {
inspection, _ := inheritanceInspectionForTest(t,
parseCompositionDocument(t, "source-1", "parent: {value: same}\n"),
parseCompositionDocument(t, "source-2", "child: {extends: parent, value: same}\n"))
path := Keys("child", "value")
history := inspection.Metadata().History(path)
var structural []Change
for _, change := range history {
if change.Origin.Operation == ChangeInheritance {
structural = append(structural, change)
}
}
if len(structural) != 2 || structural[0].Origin.Locations[0].SourceID != "source-1" ||
structural[1].Origin.Locations[0].SourceID != "source-2" ||
len(structural[0].Dependencies) != 2 || structural[0].Dependencies[0].String() != "/parent" ||
len(structural[0].Origin.Inheritance) != 1 {
t.Fatalf("equal-valued structural history = %#v", history)
}
explanation, found := inspection.Explain(path)
if !found || explanation.Effective.YAML != "same\n" {
t.Fatalf("explanation = %#v, %v", explanation, found)
}
var equalEvents int
for _, change := range explanation.History {
if change.Origin.Operation == ChangeDecoding {
t.Fatalf("inheritance was attributed to decoding: %#v", explanation.History)
}
if change.Origin.Operation == ChangeInheritance && change.Before.YAML == change.After.YAML {
equalEvents++
}
}
if equalEvents != 2 {
t.Fatalf("equal parent and child operations = %#v", explanation.History)
}
parentHistory := inspection.Metadata().History(Keys("parent", "value"))
for _, change := range parentHistory {
if change.Origin.Operation == ChangeInheritance {
t.Fatalf("source definition acquired a destination operation: %#v", parentHistory)
}
}
}
func TestInheritanceHistoryRetainsControlsAndDisplacedPaths(t *testing.T) {
inspection, _ := inheritanceInspectionForTest(t,
parseCompositionDocument(t, "source-1", "parent: {branch: {detail: inherited}}\n"),
parseCompositionDocument(t, "source-2", "child: {extends: parent, branch: replaced}\n"))
control := Keys("child", inheritanceKey)
controlHistory := inspection.Metadata().History(control)
if len(controlHistory) < 2 || controlHistory[len(controlHistory)-1].Origin.Operation != ChangeInheritance {
t.Fatalf("control history = %#v", controlHistory)
}
controlExplanation, found := inspection.Explain(control)
if !found || controlExplanation.Effective.Present ||
controlExplanation.History[len(controlExplanation.History)-1].After.Present {
t.Fatalf("consumed control = %#v, %v", controlExplanation, found)
}
path := Keys("child", "branch", "detail")
if _, found := inspection.Metadata().Lookup(path); found {
t.Fatal("displaced descendant remained effective")
}
listed := false
for _, historical := range inspection.Metadata().HistoricalPaths() {
if historical.String() == path.String() {
listed = true
}
}
if !listed {
t.Fatal("displaced inherited descendant was not enumerable")
}
history := inspection.Metadata().History(path)
if len(history) < 2 || history[len(history)-1].Origin.Operation != ChangeInheritance {
t.Fatalf("displaced descendant history = %#v", history)
}
explanation, found := inspection.Explain(path)
if !found || explanation.Effective.Present {
t.Fatalf("displaced descendant explanation = %#v, %v", explanation, found)
}
var sawInherited, sawRemoval bool
for _, change := range explanation.History {
if change.Origin.Operation != ChangeInheritance {
continue
}
sawInherited = sawInherited || change.After.Present
sawRemoval = sawRemoval || (change.Before.Present && !change.After.Present)
}
if !sawInherited || !sawRemoval {
t.Fatalf("displaced value sequence = %#v", explanation.History)
}
var rendered bytes.Buffer
if err := inspection.WriteExplanation(&rendered, path); err != nil ||
!strings.Contains(rendered.String(), "structural inheritance") {
t.Fatalf("rendered structural explanation = %q, %v", rendered.String(), err)
}
}
func TestInheritanceHistoryPrecedesApplicationResolution(t *testing.T) {
inspection, _ := inheritanceInspectionForTest(t,
parseCompositionDocument(t, "source-1", "parent: {value: original}\nchild: {extends: parent}\n"))
path := Keys("child", "value")
replacement := &document.Node{Kind: document.String, Text: "derived"}
snapshot := resolutionSnapshot{
change: Change{Path: path, Origin: Origin{Layer: LayerResolution, Operation: ChangeNormalization}},
after: replacement,
}
final := document.Clone(inspection.final)
replaceHistoryPath(&final, path, replacement)
inspection.final = final
inspection.resolution = []resolutionSnapshot{snapshot}
inspection.metadata.history[path.String()] = append(inspection.metadata.history[path.String()], snapshot.change)
inspection.metadata.fields[path.String()] = FieldInfo{Explicit: true, Origin: snapshot.change.Origin}
history := inspection.Metadata().History(path)
if len(history) < 2 || history[len(history)-2].Origin.Operation != ChangeInheritance ||
history[len(history)-1].Origin.Operation != ChangeNormalization {
t.Fatalf("resolution order = %#v", history)
}
explanation, found := inspection.Explain(path)
if !found || explanation.Effective.YAML != "derived\n" ||
explanation.History[len(explanation.History)-1].Origin.Operation != ChangeNormalization {
t.Fatalf("resolved explanation = %#v, %v", explanation, found)
}
}
func TestHistoryNodeBudgetCountsSharedInheritanceEvidenceOnce(t *testing.T) {
shared := &document.Node{Kind: document.String, Text: "shared"}
extra := &document.Node{Kind: document.String, Text: "extra"}
expansion := &inheritanceScope{
root: shared,
operations: []inheritanceOperation{{parent: shared, effective: shared}},
replacements: []inheritanceReplacement{{before: shared, after: extra}},
}
if !historyNodesWithin(2, nil, nil, nil, nil, nil, expansion) ||
historyNodesWithin(1, nil, nil, nil, nil, nil, expansion) {
t.Fatal("shared evidence node accounting is incorrect")
}
}
func TestInheritanceExpansionEnforcesRetainedHistoryBudget(t *testing.T) {
input := parseCompositionDocument(t, "source-1", "parent: {value: original}\nchild: {extends: parent, value: replacement}\n")
if _, diagnostics := expandInheritanceWithBudgets(input, nil, defaultProjectionLimits(), 2); len(diagnostics) != 1 || diagnostics[0].Code != CodeExplainLimitExceeded {
t.Fatalf("retained history limit = %#v", diagnostics)
}
if _, diagnostics := expandInheritanceWithBudgets(input, nil, defaultProjectionLimits(), 100); len(diagnostics) != 0 {
t.Fatalf("sufficient retained history budget = %#v", diagnostics)
}
}

View File

@@ -0,0 +1,53 @@
package engine
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document"
func containsInheritanceDirective(root *document.Node) bool {
found := false
walkDocument(root, Path{}, func(path Path, _ *document.Node) {
if len(path.segments) != 0 && path.segments[len(path.segments)-1].kind == pathSegmentKey &&
path.segments[len(path.segments)-1].key == inheritanceKey {
found = true
}
})
return found
}
func inheritanceVersionDiagnostic(rule *VersionRule, expanded *inheritanceScope) *Diagnostic {
if rule == nil || expanded == nil {
return nil
}
for _, operation := range expanded.operations {
var location Location
var affected bool
if operation.control != nil {
affected = pathsIntersect(operation.control.path, rule.Path)
if len(operation.control.locations) != 0 {
location = operation.control.locations[0]
}
} else if operation.transfer != nil {
if relative, within := relativeHistoryPath(operation.transfer.to, rule.Path); within {
_, affected = documentAtPath(operation.parent, relative)
} else if pathContains(rule.Path, operation.transfer.to) {
affected = true
}
if len(operation.transfer.locations) != 0 {
location = operation.transfer.locations[0]
}
}
if affected {
return &Diagnostic{
Code: CodeInheritanceVersion, Severity: SeverityError, Path: rule.Path.clone(),
Message: "inheritance cannot change the schema-version path", Primary: &location,
}
}
}
return nil
}
func inheritanceOperations(expanded *inheritanceScope) []inheritanceOperation {
if expanded == nil {
return nil
}
return expanded.operations
}

View File

@@ -0,0 +1,127 @@
package engine
import (
"slices"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func directInheritanceOrigins(node *document.Node, root Path, input map[string]Origin) map[string]Origin {
result := make(map[string]Origin)
walkDocument(node, root, func(path Path, value *document.Node) {
origin, found := input[path.String()]
if !found {
origin = Origin{Layer: LayerBase, Locations: []Location{publicLocation(value.Location)}}
}
origin = cloneOrigin(origin)
if len(origin.contributors) == 0 {
origin.contributors = make([]inheritanceContributor, len(origin.Locations))
for index, location := range origin.Locations {
origin.contributors[index] = inheritanceContributor{
source: path.clone(), location: location,
}
}
}
result[path.String()] = origin
})
return result
}
func transferredInheritanceOrigins(parent map[string]Origin, from, to Path, control inheritanceTransfer) map[string]Origin {
result := make(map[string]Origin, len(parent))
transfer := control // one immutable edge shared by every matching contributor
links := make(map[*inheritanceTrace]*inheritanceTrace)
for pointer, origin := range parent {
path, err := ParsePath(pointer)
if err != nil {
continue
}
relative, ok := relativeHistoryPath(from, path)
if !ok {
continue
}
destination := appendInheritancePath(to, relative)
origin = cloneOrigin(origin)
for index, contributor := range origin.contributors {
link, found := links[contributor.trace]
if !found {
link = &inheritanceTrace{previous: contributor.trace, transfer: &transfer}
links[contributor.trace] = link
}
origin.contributors[index].trace = link
}
result[destination.String()] = origin
}
return result
}
func appendInheritancePath(root, relative Path) Path {
return Path{segments: append(clonePathSegments(root.segments, len(relative.segments)), relative.segments...)}
}
func mergeInheritanceOrigin(origins map[string]Origin, path Path, incoming Origin) {
pointer := path.String()
current := origins[pointer]
current.Layer = incoming.Layer
current.Locations = append(current.Locations, incoming.Locations...)
current.contributors = append(current.contributors, incoming.contributors...)
origins[pointer] = current
}
func copyInheritanceSubtree(target, incoming map[string]Origin, node *document.Node, root Path) {
walkDocument(node, root, func(path Path, _ *document.Node) {
target[path.String()] = cloneOrigin(incoming[path.String()])
})
}
// Materialize concrete steps only for an inspection record. The linked private
// traces remain compact when one parent supplies many descendant values.
func projectInheritanceSteps(origin Origin) []InheritanceStep {
steps := cloneInheritanceSteps(origin.Inheritance)
for _, contributor := range origin.contributors {
var reverse []*inheritanceTrace
for link := contributor.trace; link != nil; link = link.previous {
reverse = append(reverse, link)
}
current := contributor.source
for index := len(reverse) - 1; index >= 0; index-- {
transfer := reverse[index].transfer
relative, ok := relativeHistoryPath(transfer.from, current)
if !ok {
continue
}
next := appendInheritancePath(transfer.to, relative)
step := InheritanceStep{
From: current.clone(), To: next.clone(), Reference: transfer.reference.clone(),
Locations: cloneLocations(transfer.locations),
}
if !slices.ContainsFunc(steps, func(existing InheritanceStep) bool {
return existing.From.String() == step.From.String() &&
existing.To.String() == step.To.String() &&
existing.Reference.String() == step.Reference.String() &&
slices.Equal(existing.Locations, step.Locations)
}) {
steps = append(steps, step)
}
current = next
}
}
return steps
}
func sameInheritanceTrace(left, right *inheritanceTrace) bool {
for left != nil && right != nil {
if left == right {
return true
}
l, r := left.transfer, right.transfer
if !slices.Equal(l.from.segments, r.from.segments) ||
!slices.Equal(l.to.segments, r.to.segments) ||
!slices.Equal(l.reference.segments, r.reference.segments) ||
!slices.Equal(l.locations, r.locations) {
return false
}
left, right = left.previous, right.previous
}
return left == nil && right == nil
}

View File

@@ -0,0 +1,141 @@
package engine
import (
"io/fs"
"testing"
"testing/fstest"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func TestInheritedOriginsKeepSourcePathsPresenceAndIndependentReplacement(t *testing.T) {
parent := parseCompositionDocument(t, "source-1", `parent:
enabled: false
count: 0
optional: null
items: [{file: message.txt}]
settings: {shared: parent}
`)
child := parseCompositionDocument(t, "source-2", `child:
extends: parent
count: 3
settings: {own: child}
`)
composed, diagnostics := composeDocuments(parent, child)
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
result, diagnostics := expandInheritanceWithOrigins(composed.root, composed.origins, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
for _, key := range []string{"enabled", "optional"} {
path := Keys("child", key)
value, found := documentAtPath(result.root, path)
origin, hasOrigin := result.origins[path.String()]
if !found || !hasOrigin || value == nil || len(origin.Locations) != 1 || origin.Locations[0].SourceID != "source-1" ||
len(projectInheritanceSteps(origin)) != 1 {
t.Fatalf("inherited %s = %#v, %#v", key, value, origin)
}
}
count := result.origins["/child/count"]
if len(count.Locations) != 1 || count.Locations[0].SourceID != "source-2" || len(projectInheritanceSteps(count)) != 0 {
t.Fatalf("direct replacement origin = %#v", count)
}
if len(result.replacements) != 1 || result.replacements[0].path.String() != "/child/count" ||
result.replacements[0].before.Text != "0" || result.replacements[0].after.Text != "3" ||
len(result.replacements[0].previous) != 1 ||
len(projectInheritanceSteps(result.replacements[0].previous[0].origin)) != 1 {
t.Fatalf("displaced inheritance = %#v", result.replacements)
}
settings := result.origins["/child/settings"]
if len(settings.Locations) != 2 || settings.Locations[0].SourceID != "source-1" ||
settings.Locations[1].SourceID != "source-2" || len(projectInheritanceSteps(settings)) != 1 {
t.Fatalf("composite mapping origin = %#v", settings)
}
if originContainedBy(result.origins["/child/enabled"], result.origins["/parent/enabled"]) ||
originContainedBy(result.origins["/parent/enabled"], result.origins["/child/enabled"]) {
t.Fatal("direct and inherited evidence collapsed despite matching source coordinates")
}
original := result.origins["/child/enabled"]
detached := cloneOrigin(original)
detached.Locations[0].SourceID = "changed"
detached.contributors[0].source.segments[0].key = "changed"
stepsCopy := projectInheritanceSteps(original)
stepsCopy[0].Locations[0].SourceID = "changed"
if original.Locations[0].SourceID != "source-1" ||
original.contributors[0].source.String() != "/parent/enabled" ||
projectInheritanceSteps(original)[0].Locations[0].SourceID != "source-2" {
t.Fatal("cloned origin or projected steps changed retained evidence")
}
resource := result.origins["/child/items/0/file"]
steps := projectInheritanceSteps(resource)
if len(steps) != 1 || steps[0].From.String() != "/parent/items/0/file" ||
steps[0].To.String() != "/child/items/0/file" || steps[0].Reference.String() != "/child/extends" ||
len(resource.Locations) != 1 || resource.Locations[0].SourceID != "source-1" {
t.Fatalf("sequence resource origin = %#v, steps = %#v", resource, steps)
}
contexts := map[string]fs.FS{
"source-1": fstest.MapFS{"message.txt": &fstest.MapFile{Data: []byte("from parent")}},
"source-2": fstest.MapFS{"message.txt": &fstest.MapFile{Data: []byte("from child")}},
}
data, err := fs.ReadFile(contexts[resource.Locations[0].SourceID], "message.txt")
if err != nil || string(data) != "from parent" {
t.Fatalf("inherited resource = %q, %v", data, err)
}
}
func TestInheritedOriginsRetainChainsAndNestedDestinations(t *testing.T) {
first := parseCompositionDocument(t, "source-1", `first:
enabled: false
settings:
primary: {value: original}
secondary: {extends: primary}
`)
second := parseCompositionDocument(t, "source-2", `first: {other: present}
middle: {extends: first}
last:
extends: middle
settings:
primary: {value: replaced}
`)
composed, diagnostics := composeDocuments(first, second)
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
result, diagnostics := expandInheritanceWithOrigins(composed.root, composed.origins, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
chain := projectInheritanceSteps(result.origins["/last/enabled"])
if len(chain) != 2 || chain[0].From.String() != "/first/enabled" ||
chain[0].To.String() != "/middle/enabled" || chain[1].From.String() != "/middle/enabled" ||
chain[1].To.String() != "/last/enabled" {
t.Fatalf("chain = %#v", chain)
}
composite := result.origins["/last"]
if len(composite.Locations) < 2 || len(composite.contributors) != len(composite.Locations) {
t.Fatalf("composite contributors = %#v", composite)
}
seenSources := make(map[string]bool)
for _, contributor := range composite.contributors {
seenSources[contributor.location.SourceID] = true
}
if !seenSources["source-1"] || !seenSources["source-2"] {
t.Fatalf("same-layer source contributors = %#v", composite.contributors)
}
nested := projectInheritanceSteps(result.origins["/last/settings/secondary/value"])
if len(nested) != 1 || nested[0].From.String() != "/last/settings/primary/value" ||
nested[0].To.String() != "/last/settings/secondary/value" {
t.Fatalf("destination-specific nested transfer = %#v", nested)
}
if got := result.origins["/last/settings/secondary/value"].Locations[0].SourceID; got != "source-2" {
t.Fatalf("nested replacement source = %q", got)
}
if original, found := documentAtPath(first, Keys("first", "settings", "primary", "value")); !found || original.Text != "original" {
t.Fatal("expansion changed its input")
}
if inherited, found := documentAtPath(result.root, Keys("last", "settings", "secondary", "value")); !found || inherited.Kind != document.String || inherited.Text != "replaced" {
t.Fatalf("nested inherited value = %#v, %v", inherited, found)
}
}

View File

@@ -0,0 +1,136 @@
package engine
import (
"bytes"
"reflect"
"strings"
"testing"
)
func TestInheritanceSensitivityFollowsCopiesWithoutMaskingNeighbors(t *testing.T) {
const canary = "INHERITED-CANARY-CREDENTIAL"
input := parseCompositionDocument(t, "source-1", "parent: {token: "+canary+", plain: visible}\nchild: {extends: parent}\nnext: {extends: child}\nother: {extends: parent}\n")
for _, classified := range []Path{Keys("parent", "token"), Keys("next", "token")} {
t.Run(classified.String(), func(t *testing.T) {
inspection, _ := inheritanceInspectionForTest(t, input)
inspection.secrets = classifySecretPaths(nil, func(path Path) bool {
return path.String() == classified.String()
}, inspection.metadata, inspection.final, inspection.applications, inspection.input)
for _, path := range []Path{Keys("parent", "token"), Keys("child", "token"), Keys("next", "token"), Keys("other", "token")} {
explanation, found := inspection.Explain(path)
if !found || !explanation.Effective.Redacted || strings.Contains(explanation.Effective.YAML, canary) {
t.Fatalf("sensitive copy %s = %#v, %v", path, explanation, found)
}
for _, change := range explanation.History {
if strings.Contains(change.Before.YAML, canary) || strings.Contains(change.After.YAML, canary) {
t.Fatalf("historical canary at %s: %#v", path, change)
}
}
}
plain, found := inspection.Explain(Keys("other", "plain"))
if !found || plain.Effective.Redacted || plain.Effective.YAML != "visible\n" {
t.Fatalf("unrelated field = %#v, %v", plain, found)
}
var output bytes.Buffer
if err := inspection.WriteEffective(&output); err != nil || strings.Contains(output.String(), canary) || !strings.Contains(output.String(), "visible") {
t.Fatalf("effective rendering = %q, %v", output.String(), err)
}
})
}
}
func TestInheritanceSensitivityPreservesDisplacedHistoryAndControl(t *testing.T) {
const canary = "DISPLACED-CANARY-CREDENTIAL"
inspection, _ := inheritanceInspectionForTest(t, parseCompositionDocument(t, "source-1", "parent: {branch: {token: "+canary+"}}\nchild: {extends: parent, branch: public}\n"))
inspection.secrets = classifySecretPaths(nil, func(path Path) bool {
return path.String() == Keys("parent", "branch", "token").String() || path.String() == Keys("child", inheritanceKey).String()
}, inspection.metadata, inspection.final, inspection.applications, inspection.input)
path := Keys("child", "branch", "token")
explanation, found := inspection.Explain(path)
if !found || explanation.Effective.Present {
t.Fatalf("displaced sensitive path = %#v, %v", explanation, found)
}
for _, change := range explanation.History {
if strings.Contains(change.Before.YAML, canary) || strings.Contains(change.After.YAML, canary) ||
(change.After.Present && !change.After.Redacted) {
t.Fatalf("displaced secret leaked: %#v", explanation.History)
}
}
var rendered bytes.Buffer
if err := inspection.WriteExplanation(&rendered, path); err != nil || strings.Contains(rendered.String(), canary) {
t.Fatalf("displaced explanation rendering = %q, %v", rendered.String(), err)
}
control, found := inspection.Explain(Keys("child", inheritanceKey))
if !found || control.Effective.Present || len(control.History) == 0 || !control.History[0].After.Redacted {
t.Fatalf("sensitive control history = %#v, %v", control, found)
}
info, found := inspection.Metadata().Lookup(Keys("child", "branch"))
if !found || len(info.Origin.Inheritance) != 0 {
t.Fatalf("independent replacement provenance = %#v, %v", info, found)
}
}
func TestInheritanceSensitivityIncludesSchemaTaggedSource(t *testing.T) {
type entry struct {
Token string `yaml:"token"`
}
type configuration struct {
Parent struct {
Token string `yaml:"token" configkit:"secret"`
} `yaml:"parent"`
Child entry `yaml:"child"`
}
schema, issue := buildSchema(reflect.TypeFor[configuration](), testSecretBinding())
if issue != nil {
t.Fatal(issue)
}
inspection, _ := inheritanceInspectionForTest(t, parseCompositionDocument(t, "source-1", "parent: {token: TAGGED-CANARY}\nchild: {extends: parent}\n"))
inspection.secrets = classifySecretPaths(schema, nil, inspection.metadata, inspection.final, inspection.applications, inspection.input)
child, found := inspection.Explain(Keys("child", "token"))
if !found || !child.Effective.Redacted || strings.Contains(child.Effective.YAML, "TAGGED-CANARY") {
t.Fatalf("tagged source copy = %#v, %v", child, found)
}
}
func TestInheritedSecretReferencesResolveOnlyAfterExpansion(t *testing.T) {
type entry struct {
Credential Secret `yaml:"credential"`
}
type configuration struct {
Entries map[string]entry `yaml:"entries"`
}
schema, issue := buildSchema(reflect.TypeFor[configuration](), testSecretBinding())
if issue != nil {
t.Fatal(issue)
}
composed, diagnostics := composeDocuments(parseCompositionDocument(t, "source-1", "entries:\n parent: {credential: INHERITED_SECRET_NAME}\n child: {extends: parent}\n"))
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
expanded, diagnostics := expandInheritanceWithOrigins(composed.root, composed.origins, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
lookupCount := 0
decoded, issues := decodeConfiguration(schema, reflect.Zero(reflect.TypeFor[configuration]()), expanded.root)
if len(issues) != 0 || lookupCount != 0 {
t.Fatalf("decoding and lookup order = %#v, %d", issues, lookupCount)
}
resolved := resolveSecretReferences(decoded.value, schema, expanded.root, func(name string) (string, bool) {
lookupCount++
if name != "INHERITED_SECRET_NAME" {
t.Fatalf("inherited reference name = %q", name)
}
return "RESOLVED-CREDENTIAL-CANARY", true
})
if len(resolved.diagnostics) != 0 || lookupCount != 1 ||
decoded.value.Interface().(configuration).Entries["child"].Credential.Value() != "RESOLVED-CREDENTIAL-CANARY" {
t.Fatalf("inherited reference resolution = %#v, %d", resolved.diagnostics, lookupCount)
}
inspection, _ := inheritanceInspectionFromComposition(t, composed)
inspection.secrets = classifySecretPaths(schema, nil, inspection.metadata, inspection.final, inspection.applications, inspection.input)
var rendered bytes.Buffer
if err := inspection.WriteEffective(&rendered); err != nil || strings.Contains(rendered.String(), "RESOLVED-CREDENTIAL-CANARY") {
t.Fatalf("reference rendering = %q, %v", rendered.String(), err)
}
}

View File

@@ -0,0 +1,141 @@
package engine
import (
"reflect"
"strconv"
"strings"
"testing"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func TestSiblingInheritanceResolvesReferencesAndKeepsInputs(t *testing.T) {
tests := []struct {
name, input string
path Path
text string
transfers int
controls int
}{
{
name: "forward reference", input: "child: {extends: parent, value: own}\nparent: {value: parent, kept: preserved}\n",
path: Keys("child", "kept"), text: "preserved", transfers: 1, controls: 1,
},
{
name: "backward reference", input: "parent: {value: parent, kept: preserved}\nchild: {extends: parent, value: own}\n",
path: Keys("child", "value"), text: "own", transfers: 1, controls: 1,
},
{
name: "chain and shared parent", input: "last: {extends: middle, enabled: false}\nfirst: {value: original}\nmiddle: {extends: first, extra: middle}\nother: {extends: first}\n",
path: Keys("last", "value"), text: "original", transfers: 3, controls: 3,
},
{
name: "exact empty sibling name", input: "child: {extends: \"\"}\n\"\": {value: empty}\n",
path: Keys("child", "value"), text: "empty", transfers: 1, controls: 1,
},
{
name: "exact unusual sibling name", input: "child: {extends: \"a/b~c\"}\n\"a/b~c\": {value: unusual}\n",
path: Keys("child", "value"), text: "unusual", transfers: 1, controls: 1,
},
{
name: "null cancellation", input: "child: {extends: null, value: own}\nparent: {value: parent}\n",
path: Keys("child", "value"), text: "own", controls: 1,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
input := parseCompositionDocument(t, "source-1", test.input)
before := document.Clone(input)
result, diagnostics := expandSiblingDefinitions(input, Path{})
if len(diagnostics) != 0 {
t.Fatalf("diagnostics = %#v", diagnostics)
}
if len(result.transfers) != test.transfers || len(result.controls) != test.controls {
t.Fatalf("evidence = %#v, %#v", result.transfers, result.controls)
}
node, found := documentAtPath(result.root, test.path)
if !found || node.Text != test.text {
t.Fatalf("%s = %#v, %v", test.path.String(), node, found)
}
if _, found := documentAtPath(result.root, Keys("child", inheritanceKey)); found {
t.Fatal("child directive remained in expanded payload")
}
if !reflect.DeepEqual(input, before) {
t.Fatal("resolver changed input")
}
node.Text = "changed"
if !reflect.DeepEqual(input, before) {
t.Fatal("expanded payload aliases input")
}
})
}
}
func TestSiblingInheritanceTracksTransferAndIndependentChildren(t *testing.T) {
input := parseCompositionDocument(t, "source-1", "first: {value: parent}\nsecond: {extends: first}\nthird: {extends: first}\n")
result, diagnostics := expandSiblingDefinitions(input, Keys("catalog"))
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
if len(result.transfers) != 2 || result.transfers[0].from.String() != "/catalog/first" ||
result.transfers[0].to.String() != "/catalog/second" ||
result.transfers[0].reference.String() != "/catalog/second/extends" ||
len(result.transfers[0].locations) != 1 || result.transfers[0].locations[0].SourceID != "source-1" {
t.Fatalf("transfers = %#v", result.transfers)
}
second, _ := documentAtPath(result.root, Keys("second", "value"))
third, _ := documentAtPath(result.root, Keys("third", "value"))
second.Text = "changed"
if third.Text != "parent" {
t.Fatal("siblings share mutable expanded values")
}
}
func TestSiblingInheritanceRejectsInvalidRelationships(t *testing.T) {
tests := []struct {
name, input, path string
code Code
related bool
}{
{"missing", "child: {extends: private-reference-canary}\n", "/child/extends", CodeInheritanceMissing, false},
{"nonmapping parent", "child: {extends: parent}\nparent: null\n", "/child/extends", CodeInheritanceInvalid, true},
{"boolean reference", "child: {extends: true}\n", "/child/extends", CodeInheritanceInvalid, false},
{"sequence reference", "child: {extends: [parent]}\n", "/child/extends", CodeInheritanceInvalid, false},
{"self reference", "child: {extends: child}\n", "/child/extends", CodeInheritanceCycle, true},
{"longer cycle", "a: {extends: b}\nb: {extends: c}\nc: {extends: a}\n", "/c/extends", CodeInheritanceCycle, true},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
input := parseCompositionDocument(t, "source-1", test.input)
result, diagnostics := expandSiblingDefinitions(input, Path{})
if result != nil || len(diagnostics) != 1 {
t.Fatalf("result and diagnostics = %#v, %#v", result, diagnostics)
}
problem := diagnostics[0]
if problem.Code != test.code || problem.Path.String() != test.path || problem.Primary == nil ||
problem.Primary.SourceID != "source-1" || problem.Primary.Line < 1 ||
(len(problem.Related) != 0) != test.related || strings.Contains(problem.Message, "private-reference-canary") {
t.Fatalf("diagnostic = %#v", problem)
}
})
}
}
func TestSiblingInheritanceDetectsDeepCycleWithoutRecursiveDependencyWalk(t *testing.T) {
const count = 2048
entries := make([]document.Entry, count)
for index := range entries {
location := document.Location{SourceID: "source-1", Line: index + 1, Column: 1}
parent := strconv.Itoa((index + 1) % count)
entries[index] = document.Entry{
Key: strconv.Itoa(index), Value: &document.Node{Kind: document.Mapping, Location: location,
Mapping: []document.Entry{{Key: inheritanceKey, Value: &document.Node{
Kind: document.String, Text: parent, Location: location,
}}}},
}
}
result, diagnostics := expandSiblingDefinitions(&document.Node{Kind: document.Mapping, Mapping: entries}, Path{})
if result != nil || len(diagnostics) != 1 || diagnostics[0].Code != CodeInheritanceCycle {
t.Fatalf("deep cycle result = %#v, diagnostics = %#v", result, diagnostics)
}
}

View File

@@ -0,0 +1,309 @@
package engine
import (
"cmp"
"slices"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
type inheritanceWalker struct {
nodes int // logical non-directive nodes in the entire working tree
origins map[string]Origin
limits projectionLimits
retained *retainedNodeCounter
transfers []inheritanceTransfer
controls []inheritanceControl
replacements []inheritanceReplacement
operations []inheritanceOperation
}
// expandInheritanceTree resolves each destination scope after its enclosing
// relationships. The loading lifecycle uses the origin-aware entry point.
func expandInheritanceTree(input *document.Node, limits projectionLimits) (*inheritanceScope, []Diagnostic) {
return expandInheritanceWithOrigins(input, nil, limits)
}
func expandInheritanceWithOrigins(input *document.Node, inputOrigins map[string]Origin, limits projectionLimits) (*inheritanceScope, []Diagnostic) {
return expandInheritanceWithBudgets(input, inputOrigins, limits, historyNodeLimit)
}
func expandInheritanceWithBudgets(input *document.Node, inputOrigins map[string]Origin, limits projectionLimits, historyLimit int) (*inheritanceScope, []Diagnostic) {
if input == nil || input.Kind != document.Mapping {
return nil, []Diagnostic{{Code: CodeInheritanceInvalid, Severity: SeverityError,
Message: "inheritance requires a mapping document root"}}
}
root := document.Clone(input)
origins := cloneOriginIndex(inputOrigins)
if inputOrigins == nil {
origins = directInheritanceOrigins(input, Path{}, nil)
}
nodes, within := inheritanceLowerBoundCount(root, limits.nodes)
if !within {
return nil, []Diagnostic{inheritanceBudgetDiagnostic(Path{}, root.Location)}
}
walker := inheritanceWalker{
nodes: nodes, limits: limits, origins: origins,
retained: newRetainedNodeCounter(historyLimit),
}
if problem := walker.visit(&root, Path{}, true); problem != nil {
return nil, []Diagnostic{*problem}
}
if problem := inheritanceFinalBudget(root, limits); problem != nil {
return nil, []Diagnostic{*problem}
}
return &inheritanceScope{
root: root, origins: walker.origins, transfers: walker.transfers,
controls: walker.controls, replacements: walker.replacements, operations: walker.operations,
}, nil
}
func (w *inheritanceWalker) visit(node **document.Node, path Path, unscoped bool) *Diagnostic {
switch (*node).Kind {
case document.Mapping:
original := *node
if unscoped {
reference, found := inheritanceDirective(*node)
if found {
referencePath := path.Key(inheritanceKey)
location := publicLocation(reference.Value.Location)
if reference.Value.Kind != document.Null {
problem := inheritanceDiagnostic(CodeInheritanceInvalid, referencePath, location, nil,
"inheritance reference has no enclosing sibling scope")
return &problem
}
w.controls = append(w.controls, inheritanceControl{
path: referencePath, locations: []Location{location}, cancelled: true,
})
origin := directInheritanceOrigins(reference.Value, referencePath, w.origins)[referencePath.String()]
control := inheritanceControl{path: referencePath, locations: []Location{location}, cancelled: true}
w.operations = append(w.operations, inheritanceOperation{control: &control, controlOrigin: origin})
deleteOriginSubtree(w.origins, referencePath, reference.Value)
*node = withoutInheritanceDirective(*node)
}
}
if scopeHasInheritanceDefinitions(*node) {
// Only this scope can change the logical total. Reserve the nodes
// outside it so expansion checks the remaining budget before cloning.
before, _ := inheritanceLowerBoundCount(*node, -1)
limits := w.limits
if limits.nodes >= 0 {
limits.nodes -= w.nodes - before
}
resolved, diagnostics := expandSiblingDefinitionsWithOrigins(*node, path, limits, w.origins, w.retained)
if len(diagnostics) != 0 {
return &diagnostics[0]
}
*node = resolved.root
deleteOriginSubtree(w.origins, path, original)
for pointer, origin := range resolved.origins {
w.origins[pointer] = origin
}
w.transfers = append(w.transfers, resolved.transfers...)
w.controls = append(w.controls, resolved.controls...)
w.replacements = append(w.replacements, resolved.replacements...)
w.operations = append(w.operations, resolved.operations...)
after, _ := inheritanceLowerBoundCount(*node, -1)
w.nodes += after - before
}
// Operations retain immutable enclosing templates. Detach just this
// container before replacing descendants during destination resolution;
// untouched subtrees remain shared with the retained evidence.
working := **node
working.Mapping = slices.Clone(working.Mapping)
*node = &working
indices := make([]int, len((*node).Mapping))
for index := range indices {
indices[index] = index
}
slices.SortFunc(indices, func(left, right int) int {
return cmp.Compare((*node).Mapping[left].Key, (*node).Mapping[right].Key)
})
for _, index := range indices {
entry := &(*node).Mapping[index]
if problem := w.visit(&entry.Value, path.Key(entry.Key), false); problem != nil {
return problem
}
}
case document.Sequence:
working := **node
working.Sequence = slices.Clone(working.Sequence)
*node = &working
for index := range (*node).Sequence {
child := &(*node).Sequence[index]
if problem := w.visit(child, path.Index(index), (*child).Kind == document.Mapping); problem != nil {
return problem
}
}
}
return nil
}
func scopeHasInheritanceDefinitions(scope *document.Node) bool {
for _, entry := range scope.Mapping {
if entry.Value.Kind != document.Mapping {
continue
}
if _, found := inheritanceDirective(entry.Value); found {
return true
}
}
return false
}
// A directive may disappear during later destination resolution, so only
// non-directive nodes form a safe early lower bound on final output size.
func inheritanceLowerBoundWithin(root *document.Node, limit int) bool {
_, within := inheritanceLowerBoundCount(root, limit)
return within
}
func inheritanceLowerBoundCount(root *document.Node, limit int) (int, bool) {
if limit < 0 {
limit = int(^uint(0) >> 1)
}
stack := []*document.Node{root}
count := 0
for len(stack) != 0 {
node := stack[len(stack)-1]
stack = stack[:len(stack)-1]
if node == nil {
continue
}
count++
if count > limit {
return count, false
}
switch node.Kind {
case document.Mapping:
for _, entry := range node.Mapping {
if entry.Key == inheritanceKey {
continue
}
count++ // mapping key
stack = append(stack, entry.Value)
}
case document.Sequence:
stack = append(stack, node.Sequence...)
}
if count > limit {
return count, false
}
}
return count, true
}
// Count the materialized mapping shape before cloning a memoized parent.
// Relationship fields are excluded because later destination resolution
// consumes them; their omission makes this a safe lower bound.
func inheritanceOverlayLowerBoundWithin(lower, upper *document.Node, limit int) bool {
if limit < 0 {
return false
}
index := new(mappingIndex)
count := 0
var visit func(*document.Node, *document.Node) bool
visit = func(base, incoming *document.Node) bool {
if base.Kind != document.Mapping || incoming.Kind != document.Mapping {
added, within := inheritanceLowerBoundCount(incoming, limit-count)
count += added
return within
}
count++ // mapping node
if count > limit {
return false
}
for _, entry := range base.Mapping {
if entry.Key == inheritanceKey {
continue
}
count++ // mapping key
if count > limit {
return false
}
position := index.find(incoming, entry.Key)
if position >= 0 {
if !visit(entry.Value, incoming.Mapping[position].Value) {
return false
}
} else {
added, within := inheritanceLowerBoundCount(entry.Value, limit-count)
count += added
if !within {
return false
}
}
}
for _, entry := range incoming.Mapping {
if entry.Key == inheritanceKey || index.find(base, entry.Key) >= 0 {
continue
}
count++ // mapping key
if count > limit {
return false
}
added, within := inheritanceLowerBoundCount(entry.Value, limit-count)
count += added
if !within || count > limit {
return false
}
}
return true
}
return visit(lower, upper)
}
func inheritanceFinalBudget(root *document.Node, limits projectionLimits) *Diagnostic {
type pending struct {
node *document.Node
path Path
}
stack := []pending{{node: root}}
count := 0
for len(stack) != 0 {
item := stack[len(stack)-1]
stack = stack[:len(stack)-1]
if len(item.path.segments) > limits.depth {
problem := inheritanceBudgetDiagnostic(item.path, item.node.Location)
return &problem
}
count++
if limits.nodes >= 0 && count > limits.nodes {
problem := inheritanceBudgetDiagnostic(item.path, item.node.Location)
return &problem
}
switch item.node.Kind {
case document.Mapping:
count += len(item.node.Mapping) // mapping keys
for index := len(item.node.Mapping) - 1; index >= 0; index-- {
entry := item.node.Mapping[index]
stack = append(stack, pending{node: entry.Value, path: item.path.Key(entry.Key)})
}
case document.Sequence:
for index := len(item.node.Sequence) - 1; index >= 0; index-- {
stack = append(stack, pending{node: item.node.Sequence[index], path: item.path.Index(index)})
}
}
if limits.nodes >= 0 && count > limits.nodes {
problem := inheritanceBudgetDiagnostic(item.path, item.node.Location)
return &problem
}
}
return nil
}
func inheritanceBudgetDiagnostic(path Path, location document.Location) Diagnostic {
primary := publicLocation(location)
return Diagnostic{
Code: CodeSchemaProjectionLimit, Severity: SeverityError, Path: path,
Message: "expanded configuration exceeds the projection limit", Primary: &primary,
}
}
func inheritanceHistoryBudgetDiagnostic(path Path, location document.Location) Diagnostic {
primary := publicLocation(location)
return Diagnostic{
Code: CodeExplainLimitExceeded, Severity: SeverityError, Path: path,
Message: "retained inheritance history exceeds the node limit", Primary: &primary,
}
}

View File

@@ -0,0 +1,219 @@
package engine
import (
"reflect"
"testing"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func TestNestedInheritanceUsesEachDestinationScope(t *testing.T) {
input := parseCompositionDocument(t, "source-1", `presets:
base:
settings:
primary: {width: 10}
secondary: {extends: primary}
custom:
extends: base
settings:
primary: {width: 20}
`)
before := document.Clone(input)
result, diagnostics := expandInheritanceTree(input, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
for path, want := range map[string]string{
"/presets/base/settings/secondary/width": "10",
"/presets/custom/settings/secondary/width": "20",
} {
parsed, _ := ParsePath(path)
node, found := documentAtPath(result.root, parsed)
if !found || node.Text != want {
t.Errorf("%s = %#v, found %v", path, node, found)
}
}
if len(result.transfers) != 3 || !reflect.DeepEqual(input, before) {
t.Fatalf("transfers = %#v; input changed = %v", result.transfers, !reflect.DeepEqual(input, before))
}
if node, found := documentAtPath(result.root, Keys("presets", "custom", "settings", "secondary", inheritanceKey)); found {
t.Fatalf("nested directive survived: %#v", node)
}
}
func TestNestedInheritanceSupportsChainsCancellationAndContainerReplacement(t *testing.T) {
input := parseCompositionDocument(t, "source-1", `catalog:
alpha:
settings:
first: {value: one}
second: {extends: first}
beta:
extends: alpha
settings:
first: {value: two}
gamma:
extends: beta
settings:
first: {value: three}
cancelled:
extends: alpha
settings:
second: {extends: null, value: own}
replaced:
extends: alpha
settings: none
`)
result, diagnostics := expandInheritanceTree(input, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
checks := map[string]string{
"/catalog/alpha/settings/second/value": "one",
"/catalog/beta/settings/second/value": "two",
"/catalog/gamma/settings/second/value": "three",
"/catalog/cancelled/settings/second/value": "own",
"/catalog/replaced/settings": "none",
}
for pointer, want := range checks {
path, _ := ParsePath(pointer)
node, found := documentAtPath(result.root, path)
if !found || node.Text != want {
t.Errorf("%s = %#v, found %v", pointer, node, found)
}
}
if _, found := documentAtPath(result.root, Keys("catalog", "replaced", "settings", "second")); found {
t.Fatal("replaced container retained an active child")
}
if _, found := documentAtPath(result.root, Keys("catalog", "cancelled", "settings", "second", inheritanceKey)); found {
t.Fatal("cancelled nested directive survived")
}
}
func TestNestedInheritanceCanReplaceAnInheritedRelationship(t *testing.T) {
input := parseCompositionDocument(t, "source-1", `base:
settings:
first: {value: old}
other: {value: new}
selected: {extends: first}
custom:
extends: base
settings:
selected: {extends: other}
`)
result, diagnostics := expandInheritanceTree(input, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
base, found := documentAtPath(result.root, Keys("base", "settings", "selected", "value"))
if !found || base.Text != "old" {
t.Fatalf("base relationship = %#v, %v", base, found)
}
custom, found := documentAtPath(result.root, Keys("custom", "settings", "selected", "value"))
if !found || custom.Text != "new" {
t.Fatalf("replaced relationship = %#v, %v", custom, found)
}
}
func TestNestedInheritanceRejectsDestinationCycleAndInvalidPositions(t *testing.T) {
tests := []struct {
name, input, path string
code Code
}{
{
name: "destination-only cycle", input: `base:
settings:
a: {value: one}
b: {extends: a}
custom:
extends: base
settings:
a: {extends: b}
`, path: "/custom/settings/b/extends", code: CodeInheritanceCycle,
},
{"root reference", "extends: anything\nentry: {value: one}\n", "/extends", CodeInheritanceInvalid},
{"list element reference", "items: [{extends: anything, value: one}]\n", "/items/0/extends", CodeInheritanceInvalid},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
input := parseCompositionDocument(t, "source-1", test.input)
result, diagnostics := expandInheritanceTree(input, defaultProjectionLimits())
if result != nil || len(diagnostics) != 1 || diagnostics[0].Code != test.code ||
diagnostics[0].Path.String() != test.path || diagnostics[0].Primary == nil {
t.Fatalf("result = %#v, diagnostics = %#v", result, diagnostics)
}
})
}
}
func TestNestedInheritanceResolvesScopesInsideSequences(t *testing.T) {
input := parseCompositionDocument(t, "source-1", `items:
- primary: {value: first}
secondary: {extends: primary}
- extends: null
zeta: {extends: alpha}
alpha: {value: second}
`)
result, diagnostics := expandInheritanceTree(input, defaultProjectionLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
first, found := documentAtPath(result.root, Keys("items").Index(0).Key("secondary").Key("value"))
if !found || first.Text != "first" {
t.Fatalf("first sequence scope = %#v, %v", first, found)
}
second, found := documentAtPath(result.root, Keys("items").Index(1).Key("zeta").Key("value"))
if !found || second.Text != "second" {
t.Fatalf("second sequence scope = %#v, %v", second, found)
}
if _, found := documentAtPath(result.root, Keys("items").Index(1).Key(inheritanceKey)); found {
t.Fatal("sequence-element cancellation survived")
}
}
func TestInheritanceExpansionHonorsOutputNodeAndDepthBudgets(t *testing.T) {
input := parseCompositionDocument(t, "source-1", "base: {value: one}\nchild: {extends: base}\n")
for _, test := range []struct {
name string
limits projectionLimits
valid bool
}{
{name: "at node boundary", limits: projectionLimits{nodes: 9, depth: 2}, valid: true},
{name: "beyond node boundary", limits: projectionLimits{nodes: 8, depth: 2}},
{name: "at depth boundary", limits: projectionLimits{nodes: 20, depth: 2}, valid: true},
{name: "beyond depth boundary", limits: projectionLimits{nodes: 20, depth: 1}},
} {
t.Run(test.name, func(t *testing.T) {
result, diagnostics := expandInheritanceTree(input, test.limits)
if test.valid {
if result == nil || len(diagnostics) != 0 {
t.Fatalf("result = %#v, diagnostics = %#v", result, diagnostics)
}
} else if result != nil || len(diagnostics) != 1 || diagnostics[0].Code != CodeSchemaProjectionLimit {
t.Fatalf("result = %#v, diagnostics = %#v", result, diagnostics)
}
})
}
}
func TestInheritanceBudgetIncludesOtherScopesAndSequenceItems(t *testing.T) {
input := parseCompositionDocument(t, "source-1", `extends: null
groups:
- a: {x: one}
b: {extends: a}
- a: {x: two}
b: {extends: a}
unchanged: {x: three}
`)
// Root (1), groups key/list (2), two expanded scopes (18), and
// unchanged key/mapping/key/value (4). Consumed controls cost nothing.
for _, limit := range []int{24, 25, 26} {
result, diagnostics := expandInheritanceTree(input, projectionLimits{nodes: limit, depth: 5})
if limit < 25 {
if result != nil || len(diagnostics) != 1 || diagnostics[0].Code != CodeSchemaProjectionLimit {
t.Fatalf("limit %d accepted or wrong error: %#v", limit, diagnostics)
}
} else if result == nil || len(diagnostics) != 0 {
t.Fatalf("limit %d rejected: %#v", limit, diagnostics)
}
}
}

View File

@@ -0,0 +1,50 @@
package engine
import "os"
// Input selects one source and its independent import policy.
type Input struct {
Source Source
Imports bool
Filesystem *FilesystemContext
}
// Underlay contains strictly composed contributions at one precedence tier.
type Underlay struct{ Inputs []Input }
// LayeredOptions selects lower-priority tiers and final application rules.
type LayeredOptions[T any] struct {
Definition Definition[T]
Underlays []Underlay
Overlay string
OverlayFilesystem *FilesystemContext
Overrides []Override
}
// LoadLayered loads a main input and lower-priority grouped inputs through one
// final decode, resolution, and validation lifecycle.
func LoadLayered[T any](main Input, secrets SecretBinding, supplied ...LayeredOptions[T]) (*Loaded[T], error) {
if len(supplied) > 1 {
return nil, (&diagnosticAccumulator{diagnostics: []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "LoadLayered accepts at most one options value")}}).loadError()
}
var options LayeredOptions[T]
if len(supplied) != 0 {
options = supplied[0]
}
request := loadRequest[T]{
main: loadInput{source: main.Source, imports: main.Imports, filesystem: main.Filesystem},
definition: options.Definition, overlay: options.Overlay,
overlayFilesystem: options.OverlayFilesystem,
overrides: append([]Override(nil), options.Overrides...),
underlays: make([]loadGroup, len(options.Underlays)),
}
for index, underlay := range options.Underlays {
request.underlays[index].inputs = make([]loadInput, len(underlay.Inputs))
for inputIndex, input := range underlay.Inputs {
request.underlays[index].inputs[inputIndex] = loadInput{
source: input.Source, imports: input.Imports, filesystem: input.Filesystem,
}
}
}
return loadGroupedWithSecretLookup(request, secrets, os.LookupEnv)
}

View File

@@ -33,6 +33,7 @@ type Options[T any] struct {
type FilesystemContext struct { type FilesystemContext struct {
BaseDir string BaseDir string
Root string Root string
FS *Filesystem
} }
// Override describes one explicitly supplied application override. Value uses // Override describes one explicitly supplied application override. Value uses
@@ -77,6 +78,7 @@ type Loaded[T any] struct {
type Inspection struct { type Inspection struct {
metadata Metadata metadata Metadata
sources []SourceInfo sources []SourceInfo
contexts map[string]ResourceContext
diagnostics []Diagnostic diagnostics []Diagnostic
baseline *document.Node baseline *document.Node
input *document.Node input *document.Node
@@ -84,6 +86,7 @@ type Inspection struct {
final *document.Node final *document.Node
resolution []resolutionSnapshot resolution []resolutionSnapshot
applications []historyApplication applications []historyApplication
inheritance []inheritanceOperation
secrets map[string]bool secrets map[string]bool
secretReferences map[string]string secretReferences map[string]string
} }
@@ -104,6 +107,16 @@ func (r *Inspection) Sources() []SourceInfo {
return cloneSourceInfos(r.sources) return cloneSourceInfos(r.sources)
} }
// SourceContext returns a passive filesystem view for an acquired source.
// Unknown IDs and sources without a filesystem context return false.
func (r *Inspection) SourceContext(id string) (ResourceContext, bool) {
if r == nil {
return ResourceContext{}, false
}
context, ok := r.contexts[id]
return context, ok
}
// Diagnostics returns detached nonfatal diagnostics from the load. // Diagnostics returns detached nonfatal diagnostics from the load.
func (r *Inspection) Diagnostics() []Diagnostic { func (r *Inspection) Diagnostics() []Diagnostic {
if r == nil { if r == nil {
@@ -120,14 +133,27 @@ func Load[T any](source Source, secrets SecretBinding, supplied ...Options[T]) (
} }
func loadWithSecretLookup[T any](source Source, secrets SecretBinding, lookup secretLookup, supplied ...Options[T]) (*Loaded[T], error) { func loadWithSecretLookup[T any](source Source, secrets SecretBinding, lookup secretLookup, supplied ...Options[T]) (*Loaded[T], error) {
accumulator := diagnosticAccumulator{}
options, diagnostics := selectOptions(source, supplied) options, diagnostics := selectOptions(source, supplied)
accumulator.add(diagnostics...)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return nil, accumulator.loadError() return nil, (&diagnosticAccumulator{diagnostics: diagnostics}).loadError()
}
request := loadRequest[T]{
main: loadInput{source: source, imports: options.Imports, filesystem: options.Filesystem},
definition: options.Definition, overlay: options.Overlay,
overrides: options.Overrides, ordinary: true,
}
return executeLoadRequest(request, secrets, lookup)
}
func executeLoadRequest[T any](request loadRequest[T], secrets SecretBinding, lookup secretLookup) (*Loaded[T], error) {
accumulator := diagnosticAccumulator{}
accounting := &importAccounting{}
mainOptions := Options[T]{Definition: request.definition, Imports: request.main.imports, Filesystem: request.main.filesystem}
if request.ordinary {
mainOptions.Overlay = request.overlay
} }
input, diagnostics := assembleBaseInput(source, options) input, diagnostics := assembleBaseInputTracked(request.main.source, mainOptions, accounting, nil, nil)
accumulator.sources = input.sources accumulator.sources = input.sources
accumulator.add(diagnostics...) accumulator.add(diagnostics...)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
@@ -138,29 +164,68 @@ func loadWithSecretLookup[T any](source Source, secrets SecretBinding, lookup se
accumulator.add(diagnosticFromSchemaIssue(schemaIssue)) accumulator.add(diagnosticFromSchemaIssue(schemaIssue))
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
versionDiagnostics := checkVersion(options.Definition.Version, input.document) versionDiagnostics := checkVersion(request.definition.Version, input.document)
accumulator.add(versionDiagnostics...) accumulator.add(versionDiagnostics...)
if hasErrors(versionDiagnostics) { if hasErrors(versionDiagnostics) {
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
input, diagnostics = applyInputLayers(source, options, input) if request.ordinary {
input, diagnostics = applyInputLayers(request.main.source, Options[T]{Definition: request.definition, Overlay: request.overlay, Filesystem: request.main.filesystem, Overrides: request.overrides}, input)
} else {
input, diagnostics = assembleLayeredInput(request, input, accounting)
}
accumulator.sources = input.sources accumulator.sources = input.sources
accumulator.add(diagnostics...) accumulator.add(diagnostics...)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
if request.ordinary && input.parsed.source != nil {
baseSources := make([]SourceInfo, 0, len(input.sources))
for _, source := range input.sources {
if source.Layer == LayerBase {
baseSources = append(baseSources, source)
}
}
if len(baseSources) != 0 && input.contexts[baseSources[0].ID].FS == nil {
contexts, diagnostic := contextsForAcquiredSources(request.main.source, request.main.filesystem, input.parsed.source, baseSources)
if diagnostic != nil {
accumulator.add(*diagnostic)
return nil, accumulator.loadError()
}
mergeResourceContexts(&input.contexts, contexts)
}
}
if input.document != nil && containsInheritanceDirective(input.document) {
composition, problems := compositionForInput(input)
if len(problems) != 0 {
accumulator.add(problems...)
return nil, accumulator.loadError()
}
expanded, problems := expandInheritanceWithOrigins(composition.root, composition.origins, defaultProjectionLimits())
if len(problems) != 0 {
accumulator.add(problems...)
return nil, accumulator.loadError()
}
if problem := inheritanceVersionDiagnostic(request.definition.Version, expanded); problem != nil {
accumulator.add(*problem)
return nil, accumulator.loadError()
}
input.composition = composition
input.inheritance = expanded
input.document = expanded.root
}
decoded, diagnostics := decodeLoadInput(schema, options.Definition, input) decoded, diagnostics := decodeLoadInput(schema, request.definition, input)
accumulator.add(diagnostics...) accumulator.add(diagnostics...)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
resolved, diagnostics := resolveLoadInput(schema, options.Definition, input, decoded) resolved, diagnostics := resolveLoadInput(schema, request.definition, input, decoded)
accumulator.add(diagnostics...) accumulator.add(diagnostics...)
if hasErrors(diagnostics) { if hasErrors(diagnostics) {
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
if !historyNodesWithinLimit(decoded.baseline, decoded.final, resolved.final, resolved.applications, resolved.snapshots) { if !historyNodesWithinLimit(decoded.baseline, decoded.final, resolved.final, resolved.applications, resolved.snapshots, input.inheritance) {
accumulator.add(Diagnostic{Code: CodeExplainLimitExceeded, Severity: SeverityError, Message: "retained configuration history exceeds the node limit"}) accumulator.add(Diagnostic{Code: CodeExplainLimitExceeded, Severity: SeverityError, Message: "retained configuration history exceeds the node limit"})
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
@@ -176,8 +241,8 @@ func loadWithSecretLookup[T any](source Source, secrets SecretBinding, lookup se
if hasErrors(secretResolution.diagnostics) { if hasErrors(secretResolution.diagnostics) {
return nil, accumulator.loadError() return nil, accumulator.loadError()
} }
if options.Definition.Validate != nil { if request.definition.Validate != nil {
diagnostics = runValidation(&configuration, options.Definition.Validate, schema, resolved.final, input.document, secretResolution.lookups) diagnostics = runValidation(&configuration, request.definition.Validate, schema, resolved.final, input.document, secretResolution.lookups)
accumulator.add(diagnostics...) accumulator.add(diagnostics...)
if hasErrors(diagnostics) { if hasErrors(diagnostics) {
return nil, accumulator.loadError() return nil, accumulator.loadError()
@@ -186,5 +251,5 @@ func loadWithSecretLookup[T any](source Source, secrets SecretBinding, lookup se
resolved.configuration = configuration resolved.configuration = configuration
// Publication detaches equal-valued validator replacements, preserving the // Publication detaches equal-valued validator replacements, preserving the
// validated secret state, then finalizes metadata and sensitivity. // validated secret state, then finalizes metadata and sensitivity.
return publishLoadResult(schema, options.Definition, resolved, secretResolution.references, &accumulator) return publishLoadResult(schema, request.definition, resolved, secretResolution.references, &accumulator)
} }

View File

@@ -1,24 +1,37 @@
package engine package engine
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document" import (
"strconv"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
// assembledInput is the source/document handoff across schema and version checks. // assembledInput is the source/document handoff across schema and version checks.
type assembledInput struct { type assembledInput struct {
parsed parsedSource parsed parsedSource
document *document.Node document *document.Node
composition *composedDocument composition *composedDocument
inheritance *inheritanceScope
sources []SourceInfo sources []SourceInfo
contexts map[string]ResourceContext
} }
func assembleBaseInput[T any](source Source, options Options[T]) (assembledInput, []Diagnostic) { func assembleBaseInput[T any](source Source, options Options[T]) (assembledInput, []Diagnostic) {
parsed, diagnostics := parseSource(source, "source-1", defaultParseLimits()) return assembleBaseInputTracked(source, options, &importAccounting{}, nil, nil)
}
func assembleBaseInputTracked[T any](source Source, options Options[T], accounting *importAccounting, visited map[resourceIdentity]importRoute, hostToken *namespaceToken) (assembledInput, []Diagnostic) {
limits := defaultImportLimits()
parseLimits := defaultParseLimits()
parseLimits.source.remaining = limits.bytes - accounting.bytes
parsed, diagnostics := parseSource(source, "source-"+strconv.Itoa(accounting.sources+1), parseLimits)
input := assembledInput{parsed: parsed, document: parsed.document} input := assembledInput{parsed: parsed, document: parsed.document}
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
input.sources = sourceInfos(parsed.source) input.sources = sourceInfos(parsed.source)
return input, diagnostics return input, diagnostics
} }
if options.Imports { if options.Imports {
input.composition, input.sources, diagnostics = loadImportedDocuments(source, options.Filesystem, parsed, options.Definition.Version, defaultImportLimits()) input.composition, input.sources, diagnostics = loadImportedDocumentsTracked(source, options.Filesystem, parsed, options.Definition.Version, limits, accounting, visited, hostToken)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return input, diagnostics return input, diagnostics
} }
@@ -35,10 +48,44 @@ func assembleBaseInput[T any](source Source, options Options[T]) (assembledInput
if diagnostics := disabledDirectiveDiagnostics(input.document, Path{}); len(diagnostics) != 0 { if diagnostics := disabledDirectiveDiagnostics(input.document, Path{}); len(diagnostics) != 0 {
return input, diagnostics return input, diagnostics
} }
if accounting.sources+1 > limits.sources || accounting.bytes+parsed.bytes > limits.bytes || accounting.nodes+countInputNodes(input.document) > limits.nodes {
return input, []Diagnostic{sourceDiagnostic(CodeSourceLimitExceeded, parsed.source.ID, "configuration exceeds the aggregate input limit")}
}
if diagnostic := registerAcquiredSource(source, options.Filesystem, parsed.source, visited, hostToken); diagnostic != nil {
return input, []Diagnostic{*diagnostic}
}
accounting.sources++
accounting.bytes += parsed.bytes
accounting.nodes += countInputNodes(input.document)
}
if !activeOverlay(options.Overlay) || options.Imports {
contexts, diagnostic := contextsForAcquiredSources(source, options.Filesystem, parsed.source, input.sources)
if diagnostic != nil {
return input, []Diagnostic{*diagnostic}
}
input.contexts = contexts
} }
return input, nil return input, nil
} }
func registerAcquiredSource(source Source, context *FilesystemContext, info *SourceInfo, visited map[resourceIdentity]importRoute, hostToken *namespaceToken) *Diagnostic {
if visited == nil || info == nil || info.Path == "" {
return nil
}
identity := resourceIdentity{namespace: hostToken, path: info.Path}
if source.kind == sourceFilesystem {
identity.namespace = source.namespace.token
} else if context != nil && context.FS != nil {
identity.namespace = context.FS.namespace.token
}
if previous, repeated := visited[identity]; repeated {
value := importRepeatedDiagnostic(Location{SourceID: info.ID}, previous, importRoute{})
return &value
}
visited[identity] = importRoute{canonical: info.Path}
return nil
}
func applyInputLayers[T any](source Source, options Options[T], input assembledInput) (assembledInput, []Diagnostic) { func applyInputLayers[T any](source Source, options Options[T], input assembledInput) (assembledInput, []Diagnostic) {
if !activeOverlay(options.Overlay) && len(options.Overrides) == 0 { if !activeOverlay(options.Overlay) && len(options.Overrides) == 0 {
return input, nil return input, nil
@@ -47,25 +94,31 @@ func applyInputLayers[T any](source Source, options Options[T], input assembledI
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return input, diagnostics return input, diagnostics
} }
assembly := newAssemblyAccumulator(composition)
if activeOverlay(options.Overlay) { if activeOverlay(options.Overlay) {
composition, input.sources, diagnostics = loadSelectedOverlayDocument( input.sources, diagnostics = loadSelectedOverlayInto(
source, options.Filesystem, input.parsed, composition, input.sources, source, options.Filesystem, input.parsed, assembly, input.sources,
options.Overlay, options.Definition.Version, defaultImportLimits(), options.Overlay, options.Definition.Version, defaultImportLimits(),
) )
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return input, diagnostics return input, diagnostics
} }
contexts, diagnostic := contextsForAcquiredSources(source, options.Filesystem, input.parsed.source, input.sources[len(input.sources)-1:])
if diagnostic != nil {
return input, []Diagnostic{*diagnostic}
}
mergeResourceContexts(&input.contexts, contexts)
} }
if len(options.Overrides) != 0 { if len(options.Overrides) != 0 {
composition, input.sources, diagnostics = applyOverrides( input.sources, diagnostics = assembly.applyOverrides(
composition, input.sources, options.Overrides, options.Definition.Version, defaultImportLimits(), input.sources, options.Overrides, options.Definition.Version, defaultImportLimits(),
) )
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return input, diagnostics return input, diagnostics
} }
} }
input.composition = composition input.composition = assembly.current
input.document = composition.root input.document = assembly.current.root
return input, nil return input, nil
} }

View File

@@ -12,9 +12,10 @@ import (
type resolutionInput struct { type resolutionInput struct {
metadata Metadata metadata Metadata
sources map[string]SourceInfo sources map[string]SourceInfo
contexts map[string]ResourceContext
} }
func newResolutionInput(metadata Metadata, final *document.Node, sources []SourceInfo) resolutionInput { func newResolutionInput(metadata Metadata, final *document.Node, sources []SourceInfo, contexts ...map[string]ResourceContext) resolutionInput {
frozen := Metadata{ frozen := Metadata{
fields: make(map[string]FieldInfo, len(metadata.fields)), fields: make(map[string]FieldInfo, len(metadata.fields)),
effectivePaths: effectivePathIndex(final), effectivePaths: effectivePathIndex(final),
@@ -32,7 +33,11 @@ func newResolutionInput(metadata Metadata, final *document.Node, sources []Sourc
} }
indexed[source.ID] = cloneSourceInfo(source) indexed[source.ID] = cloneSourceInfo(source)
} }
return resolutionInput{metadata: frozen, sources: indexed} result := resolutionInput{metadata: frozen, sources: indexed}
if len(contexts) != 0 {
result.contexts = cloneResourceContexts(contexts[0])
}
return result
} }
func (input resolutionInput) lookup(path Path) (FieldInfo, bool) { func (input resolutionInput) lookup(path Path) (FieldInfo, bool) {
@@ -47,6 +52,11 @@ func (input resolutionInput) source(id string) (SourceInfo, bool) {
return cloneSourceInfo(source), true return cloneSourceInfo(source), true
} }
func (input resolutionInput) sourceContext(id string) (ResourceContext, bool) {
context, ok := input.contexts[id]
return context, ok
}
func metadataFromDecoded(decoded decodedConfiguration) Metadata { func metadataFromDecoded(decoded decodedConfiguration) Metadata {
metadata := Metadata{ metadata := Metadata{
fields: make(map[string]FieldInfo), fields: make(map[string]FieldInfo),
@@ -115,9 +125,6 @@ func applyOverlayMetadata(metadata *Metadata, records []overlayRecord) {
if err != nil { if err != nil {
continue continue
} }
if history := events[pointer]; len(history) != 0 && history[len(history)-1].Origin.Layer == info.Origin.Layer && history[len(history)-1].Origin.Operation == info.Origin.Operation {
continue
}
appendInputOriginEvent(events, baseline, pointer, path, info.Origin) appendInputOriginEvent(events, baseline, pointer, path, info.Origin)
} }
metadata.history = make(map[string][]Change, len(baseline)+len(events)) metadata.history = make(map[string][]Change, len(baseline)+len(events))
@@ -134,12 +141,37 @@ func appendInputOriginEvent(events map[string][]Change, baseline map[string][]Ch
return return
} }
history := events[pointer] history := events[pointer]
if len(history) != 0 && originContainedBy(history[len(history)-1].Origin, origin) { if len(history) != 0 && originRepresentedByChange(path, origin, history[len(history)-1]) {
return return
} }
appendInputChangeEvent(events, baseline, pointer, Change{Path: path.clone(), Origin: cloneOrigin(origin)}) appendInputChangeEvent(events, baseline, pointer, Change{Path: path.clone(), Origin: cloneOrigin(origin)})
} }
func originRepresentedByChange(path Path, origin Origin, change Change) bool {
if change.Path.String() == path.String() {
return originContainedBy(origin, change.Origin)
}
if !pathContains(change.Path, path) || origin.Layer != change.Origin.Layer || origin.Operation != change.Origin.Operation {
return false
}
// A subtree replacement already records its descendants' contributions.
// Their coordinates differ from the container's, but their sources must be
// covered by the operation; matching only the precedence role is insufficient.
for _, location := range origin.Locations {
found := false
for _, recorded := range change.Origin.Locations {
if location.SourceID == recorded.SourceID {
found = true
break
}
}
if !found {
return false
}
}
return true
}
func appendInputChangeEvent(events map[string][]Change, baseline map[string][]Change, pointer string, value Change) { func appendInputChangeEvent(events map[string][]Change, baseline map[string][]Change, pointer string, value Change) {
change := cloneChange(value) change := cloneChange(value)
history := events[pointer] history := events[pointer]
@@ -169,6 +201,23 @@ func originContainedBy(candidate, container Origin) bool {
return false return false
} }
} }
if (len(candidate.contributors) == 0) != (len(container.contributors) == 0) {
return false
}
for _, contributor := range candidate.contributors {
found := false
for _, available := range container.contributors {
if contributor.location == available.location &&
slices.Equal(contributor.source.segments, available.source.segments) &&
sameInheritanceTrace(contributor.trace, available.trace) {
found = true
break
}
}
if !found {
return false
}
}
return true return true
} }
@@ -207,6 +256,7 @@ func cloneMetadata(metadata Metadata) Metadata {
cloned := Metadata{ cloned := Metadata{
fields: make(map[string]FieldInfo, len(metadata.fields)), fields: make(map[string]FieldInfo, len(metadata.fields)),
history: make(map[string][]Change, len(metadata.history)), history: make(map[string][]Change, len(metadata.history)),
inheritance: append([]inheritanceOperation(nil), metadata.inheritance...),
} }
cloned.effectivePaths = make(map[string]Path, len(metadata.effectivePaths)) cloned.effectivePaths = make(map[string]Path, len(metadata.effectivePaths))
for pointer, path := range metadata.effectivePaths { for pointer, path := range metadata.effectivePaths {

View File

@@ -0,0 +1,182 @@
package engine
import (
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
// loadRequest keeps source acquisition separate from precedence assembly.
// The ordinary entry point and grouped entry point share the final lifecycle.
type loadRequest[T any] struct {
main loadInput
underlays []loadGroup
definition Definition[T]
overlay string
overlayFilesystem *FilesystemContext
overrides []Override
ordinary bool
}
type loadInput struct {
source Source
imports bool
filesystem *FilesystemContext
}
type loadGroup struct{ inputs []loadInput }
func loadGroupedWithSecretLookup[T any](request loadRequest[T], secrets SecretBinding, lookup secretLookup) (*Loaded[T], error) {
request.ordinary = false
var diagnostics []Diagnostic
_, mainDiagnostics := selectOptions(request.main.source, []Options[T]{{
Definition: request.definition, Imports: request.main.imports, Filesystem: request.main.filesystem,
}})
diagnostics = append(diagnostics, mainDiagnostics...)
for _, group := range request.underlays {
for _, input := range group.inputs {
_, inputDiagnostics := selectOptions(input.source, []Options[T]{{Imports: input.imports, Filesystem: input.filesystem}})
diagnostics = append(diagnostics, inputDiagnostics...)
}
}
if !validOverlayName(request.overlay) {
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "overlay name must contain only ASCII letters, digits, underscore, or hyphen"))
}
if activeOverlay(request.overlay) {
if request.overlayFilesystem == nil {
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "selected overlay requires explicit filesystem context"))
} else {
diagnostics = append(diagnostics, filesystemOptionDiagnostics(Source{kind: sourceBytes}, request.overlayFilesystem)...)
}
}
if len(diagnostics) != 0 {
return nil, (&diagnosticAccumulator{diagnostics: canonicalDiagnostics(diagnostics)}).loadError()
}
return executeLoadRequest(request, secrets, lookup)
}
func assembleLayeredInput[T any](request loadRequest[T], input assembledInput, accounting *importAccounting) (assembledInput, []Diagnostic) {
groups := make([]*composedDocument, 0, len(request.underlays))
for index, group := range request.underlays {
composition, sources, contexts, diagnostics := acquireUnderlayGroup(group, index+1, request.definition.Version, accounting)
input.sources = append(input.sources, sources...)
mergeResourceContexts(&input.contexts, contexts)
if len(diagnostics) != 0 {
return input, diagnostics
}
if composition != nil {
groups = append(groups, composition)
}
}
var assembly *assemblyAccumulator
for _, group := range groups {
if assembly == nil {
assembly = newAssemblyAccumulator(group)
continue
}
if diagnostics := assembly.applyOverlay(group.root, group.origins, group.applications, LayerUnderlay); len(diagnostics) != 0 {
return input, diagnostics
}
}
if input.document != nil {
mainComposition, diagnostics := compositionForInput(input)
if len(diagnostics) != 0 {
return input, diagnostics
}
if assembly == nil {
assembly = newAssemblyAccumulator(mainComposition)
} else if diagnostics := assembly.applyOverlay(mainComposition.root, mainComposition.origins, mainComposition.applications, LayerBase); len(diagnostics) != 0 {
return input, diagnostics
}
}
if assembly == nil && (activeOverlay(request.overlay) || len(request.overrides) != 0) {
composition, _ := compositionForInput(assembledInput{})
assembly = newAssemblyAccumulator(composition)
}
if assembly == nil {
return input, nil
}
assembly.current.inputBytes = accounting.bytes
assembly.current.inputNodes = accounting.nodes
if activeOverlay(request.overlay) {
// The selected overlay context is independent of the main input. This
// synthetic descriptor creates no source record or input contribution.
selection := Source{kind: sourceBytes}
selected := parsedSource{document: &document.Node{Kind: document.Mapping}, source: &SourceInfo{}}
var diagnostics []Diagnostic
input.sources, diagnostics = loadSelectedOverlayInto(selection, request.overlayFilesystem, selected, assembly, input.sources, request.overlay, request.definition.Version, defaultImportLimits())
if len(diagnostics) != 0 {
return input, diagnostics
}
contexts, diagnostic := contextsForAcquiredSources(selection, request.overlayFilesystem, selected.source, input.sources[len(input.sources)-1:])
if diagnostic != nil {
return input, []Diagnostic{*diagnostic}
}
mergeResourceContexts(&input.contexts, contexts)
}
if len(request.overrides) != 0 {
var diagnostics []Diagnostic
input.sources, diagnostics = assembly.applyOverrides(input.sources, request.overrides, request.definition.Version, defaultImportLimits())
if len(diagnostics) != 0 {
return input, diagnostics
}
}
input.composition = assembly.current
input.document = assembly.current.root
return input, nil
}
func acquireUnderlayGroup(group loadGroup, index int, rule *VersionRule, accounting *importAccounting) (*composedDocument, []SourceInfo, map[string]ResourceContext, []Diagnostic) {
initialBytes, initialNodes := accounting.bytes, accounting.nodes
visited := make(map[resourceIdentity]importRoute)
hostToken := &namespaceToken{}
var documents []*document.Node
var sources []SourceInfo
var contexts map[string]ResourceContext
for _, contribution := range group.inputs {
options := Options[struct{}]{Imports: contribution.imports, Filesystem: contribution.filesystem, Definition: Definition[struct{}]{Version: rule}}
acquired, diagnostics := assembleBaseInputTracked(contribution.source, options, accounting, visited, hostToken)
for _, source := range acquired.sources {
source.Layer = LayerUnderlay
source.UnderlayIndex = index
sources = append(sources, source)
}
mergeResourceContexts(&contexts, acquired.contexts)
if len(diagnostics) != 0 {
return nil, sources, contexts, diagnostics
}
if acquired.document == nil {
continue
}
if node := fragmentVersionClaim(acquired.document, rule); node != nil {
location := publicLocation(node.Location)
return nil, sources, contexts, []Diagnostic{{
Code: CodeUnderlayVersion, Severity: SeverityError, Path: rule.Path.clone(),
Message: "underlay inputs cannot contribute to the schema-version path", Primary: &location,
}}
}
if acquired.composition != nil {
for _, application := range acquired.composition.applications {
documents = append(documents, application.value)
}
} else {
documents = append(documents, acquired.document)
}
}
if len(documents) == 0 {
return nil, sources, contexts, nil
}
composition, diagnostics := composeDocuments(documents...)
if len(diagnostics) != 0 {
return nil, sources, contexts, diagnostics
}
composition.inputBytes = accounting.bytes - initialBytes
composition.inputNodes = accounting.nodes - initialNodes
for pointer, origin := range composition.origins {
origin.Layer = LayerUnderlay
composition.origins[pointer] = origin
}
for index := range composition.applications {
composition.applications[index].layer = LayerUnderlay
}
return composition, sources, contexts, nil
}

View File

@@ -0,0 +1,280 @@
package engine
import (
"reflect"
"strings"
"testing"
"testing/fstest"
)
type groupedConfig struct {
Version string `yaml:"version"`
Settings map[string]string `yaml:"settings"`
Port int `yaml:"port"`
Derived string `yaml:"derived"`
}
func testGroupedLoad[T any](request loadRequest[T]) (*testResult[T], error) {
loaded, err := loadGroupedWithSecretLookup(request, testSecretBinding(), func(string) (string, bool) { return "", false })
return testLoaded(loaded), err
}
func TestGroupedLoadPrecedenceSourceOrderAndSingleLifecycle(t *testing.T) {
filesystem := NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: imported.yml}]}\nversion: v1\nport: 5\nsettings: {shared: main}\n")},
"imported.yml": &fstest.MapFile{Data: []byte("settings: {imported: included}\n")},
"overlays/test.yml": &fstest.MapFile{Data: []byte("settings: {shared: overlay}\n")},
})
checks := struct{ version, defaults, resolution, validation int }{}
request := loadRequest[groupedConfig]{
main: loadInput{source: filesystem.File("main.yml"), imports: true},
underlays: []loadGroup{
{inputs: []loadInput{
{source: Mapping("low-a", map[string]any{"settings": map[string]any{"shared": "low", "a": "one"}, "port": "invalid until replaced"})},
{source: Bytes("low-b", []byte("settings: {b: two}\n"))},
}},
{inputs: []loadInput{{source: Mapping("middle", map[string]any{"settings": map[string]any{"shared": "middle", "c": "three"}})}}},
},
definition: Definition[groupedConfig]{
Version: &VersionRule{Path: Keys("version"), Check: func(version Version, validation *Validation) error {
checks.version++
if version.Text != "v1" {
t.Errorf("version = %#v", version)
}
return nil
}},
Defaults: func() groupedConfig { checks.defaults++; return groupedConfig{Derived: "initial"} },
Resolve: func(config *groupedConfig, resolution *Resolution) error {
checks.resolution++
Default(resolution, Keys("derived"), &config.Derived, "resolved", "set derived")
return nil
},
Validate: func(config *groupedConfig, validation *Validation) { checks.validation++ },
},
overlay: "test", overlayFilesystem: &FilesystemContext{Root: ".", BaseDir: ".", FS: &filesystem},
overrides: []Override{{Path: Keys("settings", "shared"), Value: "cli", Flag: "--shared"}},
}
result, err := testGroupedLoad(request)
if err != nil {
t.Fatalf("load: %v, diagnostics %#v", err, err.(*LoadError).Diagnostics())
}
wantSettings := map[string]string{"shared": "cli", "a": "one", "b": "two", "c": "three", "imported": "included"}
if result.Config.Port != 5 || !reflect.DeepEqual(result.Config.Settings, wantSettings) {
t.Fatalf("config = %#v", result.Config)
}
if checks != (struct{ version, defaults, resolution, validation int }{1, 1, 1, 1}) {
t.Fatalf("lifecycle calls = %#v", checks)
}
sources := result.Sources()
if len(sources) != 7 {
t.Fatalf("sources = %#v", sources)
}
wantLayers := []Layer{LayerBase, LayerBase, LayerUnderlay, LayerUnderlay, LayerUnderlay, LayerOverlay, LayerCLI}
wantIndexes := []int{0, 0, 1, 1, 2, 0, 0}
for index, source := range sources {
if source.ID != "source-"+string(rune('1'+index)) || source.Layer != wantLayers[index] || source.UnderlayIndex != wantIndexes[index] {
t.Errorf("source %d = %#v", index, source)
}
}
info, ok := result.Metadata().Lookup(Keys("settings", "shared"))
if !ok || info.Origin.Layer != LayerCLI {
t.Fatalf("effective shared metadata = %#v, %v", info, ok)
}
explanation, ok := result.Explain(Keys("settings", "shared"))
if !ok || len(explanation.History) < 5 {
t.Fatalf("shared explanation = %#v, %v", explanation, ok)
}
var layers []Layer
for _, change := range explanation.History {
if change.Origin.Layer != LayerDefault && (len(layers) == 0 || layers[len(layers)-1] != change.Origin.Layer) {
layers = append(layers, change.Origin.Layer)
}
}
if !reflect.DeepEqual(layers, []Layer{LayerUnderlay, LayerBase, LayerOverlay, LayerCLI}) {
t.Fatalf("history layers = %#v", layers)
}
}
func TestGroupedLoadAbsentMainAndVersionProtection(t *testing.T) {
var observed Version
request := loadRequest[groupedConfig]{
main: loadInput{source: NoSource()},
underlays: []loadGroup{{}, {inputs: []loadInput{{source: Bytes("underlay", []byte("port: 7\n"))}}}},
definition: Definition[groupedConfig]{Version: &VersionRule{Path: Keys("version"), Check: func(version Version, validation *Validation) error {
observed = version
return nil
}}},
}
result, err := testGroupedLoad(request)
if err != nil || result.Config.Port != 7 || observed.Present || len(result.Sources()) != 1 || result.Sources()[0].UnderlayIndex != 2 {
t.Fatalf("underlay-only load = %#v, version %#v, %v", result, observed, err)
}
request.underlays[1].inputs[0].source = Bytes("underlay", []byte("version: forbidden\n"))
_, err = testGroupedLoad(request)
if got := groupedErrorCode(t, err); got != CodeUnderlayVersion {
t.Fatalf("underlay version code = %s", got)
}
filesystem := NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: fragment.yml}]}\n")},
"fragment.yml": &fstest.MapFile{Data: []byte("version: forbidden\n")},
})
request.underlays[1].inputs[0] = loadInput{source: filesystem.File("main.yml"), imports: true}
_, err = testGroupedLoad(request)
if got := groupedErrorCode(t, err); got != CodeImportVersion {
t.Fatalf("import version code = %s", got)
}
request.underlays = []loadGroup{{}}
request.definition.Defaults = func() groupedConfig { return groupedConfig{Port: 11} }
result, err = testGroupedLoad(request)
if err != nil || result.Config.Port != 11 || len(result.Sources()) != 0 {
t.Fatalf("defaults-only load = %#v, %v", result, err)
}
}
func TestGroupedLoadStrictConflictsRepetitionAndBudgets(t *testing.T) {
main := loadInput{source: Bytes("main", []byte("port: 5\n"))}
request := loadRequest[groupedConfig]{
main: main,
underlays: []loadGroup{{inputs: []loadInput{
{source: Bytes("left", []byte("port: 1\n"))},
{source: Bytes("right", []byte("port: 2\n"))},
}}},
}
_, err := testGroupedLoad(request)
if got := groupedErrorCode(t, err); got != CodeComposeConflict {
t.Fatalf("strict conflict code = %s", got)
}
conflicting := NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: fragment.yml}]}\nport: 1\n")},
"fragment.yml": &fstest.MapFile{Data: []byte("port: 2\n")},
})
request.main = loadInput{source: conflicting.File("main.yml"), imports: true}
request.underlays = []loadGroup{{inputs: []loadInput{{source: Bytes("lower", []byte("port: 3\n"))}}}}
_, err = testGroupedLoad(request)
if got := groupedErrorCode(t, err); got != CodeComposeConflict {
t.Fatalf("main/import conflict code = %s", got)
}
request.main = main
filesystem := NewFilesystem("bundle", fstest.MapFS{"same.yml": &fstest.MapFile{Data: []byte("port: 1\n")}})
file := filesystem.File("same.yml")
request.underlays = []loadGroup{{inputs: []loadInput{{source: file}, {source: file}}}}
_, err = testGroupedLoad(request)
if got := groupedErrorCode(t, err); got != CodeImportRepeated {
t.Fatalf("same-group repetition code = %s", got)
}
request.underlays = []loadGroup{{inputs: []loadInput{{source: file}}}, {inputs: []loadInput{{source: file}}}}
if result, err := testGroupedLoad(request); err != nil || result.Config.Port != 5 || len(result.Sources()) != 3 {
t.Fatalf("cross-group reuse = %#v, %v", result, err)
}
accounting := &importAccounting{sources: maxImportSources - 1}
group := loadGroup{inputs: []loadInput{{source: Mapping("first", map[string]any{})}, {source: Mapping("second", map[string]any{})}}}
_, _, _, diagnostics := acquireUnderlayGroup(group, 1, nil, accounting)
if len(diagnostics) == 0 || diagnostics[0].Code != CodeSourceLimitExceeded {
t.Fatalf("shared source budget = %#v", diagnostics)
}
accounting = &importAccounting{bytes: maxTotalBytes - 2}
group.inputs = []loadInput{{source: Mapping("first", map[string]any{"port": "one"})}}
_, _, _, diagnostics = acquireUnderlayGroup(group, 1, nil, accounting)
if len(diagnostics) == 0 || diagnostics[0].Code != CodeSourceLimitExceeded {
t.Fatalf("shared byte budget = %#v", diagnostics)
}
}
func TestGroupedLoadInvalidIntermediateTypeCanBeReplaced(t *testing.T) {
request := loadRequest[groupedConfig]{
main: loadInput{source: Bytes("main", []byte("port: 9\n"))},
underlays: []loadGroup{{inputs: []loadInput{{source: Bytes("low", []byte("port: wrong\n"))}}}},
}
result, err := testGroupedLoad(request)
if err != nil || result.Config.Port != 9 {
t.Fatalf("final typed decode = %#v, %v", result, err)
}
request.main.source = Bytes("main", []byte("port: wrong\n"))
_, err = testGroupedLoad(request)
if got := groupedErrorCode(t, err); got != CodeDecodeType && got != CodeDecodeRange {
t.Fatalf("final invalid type code = %s", got)
}
}
func groupedErrorCode(t *testing.T, err error) Code {
t.Helper()
loadError, ok := err.(*LoadError)
if !ok || len(loadError.Diagnostics()) == 0 {
t.Fatalf("error = %#v", err)
}
if strings.Contains(loadError.Diagnostics()[0].Message, "wrong") {
t.Fatal("raw value escaped diagnostics")
}
return loadError.Diagnostics()[0].Code
}
func TestGroupedAccountingChargesMixedOccurrencesOnce(t *testing.T) {
const (
one = "configkit: {imports: [{path: child-one.yml}]}\nleft: first\n"
child = "extra: imported\n"
two = "configkit: {imports: [{path: child-two.yml}]}\nright: second\n"
)
filesystem := NewFilesystem("bundle", fstest.MapFS{
"one.yml": &fstest.MapFile{Data: []byte(one)},
"child-one.yml": &fstest.MapFile{Data: []byte(child)},
"two.yml": &fstest.MapFile{Data: []byte(two)},
"child-two.yml": &fstest.MapFile{Data: []byte(child)},
})
accounting := &importAccounting{}
first, sources, _, diagnostics := acquireUnderlayGroup(loadGroup{inputs: []loadInput{{source: filesystem.File("one.yml"), imports: true}}}, 1, nil, accounting)
if len(diagnostics) != 0 || first == nil || len(sources) != 2 || accounting.sources != 2 || accounting.bytes != int64(len(one)+len(child)) {
t.Fatalf("first acquisition = %#v, %#v, %#v, accounting %#v", first, sources, diagnostics, accounting)
}
second, sources, _, diagnostics := acquireUnderlayGroup(loadGroup{inputs: []loadInput{{source: filesystem.File("two.yml"), imports: true}}}, 2, nil, accounting)
if len(diagnostics) != 0 || second == nil || len(sources) != 2 || sources[0].ID != "source-3" || sources[1].ID != "source-4" || accounting.sources != 4 || accounting.bytes != int64(len(one)+len(two)+2*len(child)) {
t.Fatalf("second acquisition = %#v, %#v, %#v, accounting %#v", second, sources, diagnostics, accounting)
}
var wantNodes int
for _, text := range []string{one, child, two, child} {
parsed, diagnostics := parseSource(Bytes("fixture", []byte(text)), "source-1", defaultParseLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
wantNodes += countInputNodes(parsed.document)
}
if accounting.nodes != wantNodes {
t.Fatalf("charged nodes = %d, want %d", accounting.nodes, wantNodes)
}
if first.inputBytes != int64(len(one)+len(child)) || second.inputBytes != int64(len(two)+len(child)) {
t.Fatalf("composition costs = %d, %d, accounting %d", first.inputBytes, second.inputBytes, accounting.bytes)
}
}
func TestGroupedGlobEntryBudgetIsShared(t *testing.T) {
filesystem := NewFilesystem("bundle", fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{glob: 'parts/*.yml'}]}\n")},
"parts/a.yml": &fstest.MapFile{Data: []byte("left: one\n")},
"parts/b.yml": &fstest.MapFile{Data: []byte("right: two\n")},
})
accounting := &importAccounting{entries: maxGlobEntries - 1}
_, _, _, diagnostics := acquireUnderlayGroup(loadGroup{inputs: []loadInput{{source: filesystem.File("main.yml"), imports: true}}}, 1, nil, accounting)
if len(diagnostics) == 0 || diagnostics[0].Code != CodeImportLimitExceeded || accounting.entries <= maxGlobEntries {
t.Fatalf("shared glob budget = %#v, accounting %#v", diagnostics, accounting)
}
}
func TestGroupedMixedSourcesShareCLIRecordLimit(t *testing.T) {
filesystem := NewFilesystem("bundle", fstest.MapFS{"main.yml": &fstest.MapFile{Data: []byte("value: base\n")}})
inputs := make([]Input, maxImportSources-1)
inputs[0] = Input{Source: Bytes("empty-bytes", []byte("{}\n"))}
for index := 1; index < len(inputs); index++ {
inputs[index] = Input{Source: Mapping("empty-map", map[string]any{})}
}
type config struct {
Value string `yaml:"value"`
}
_, err := LoadLayered(Input{Source: filesystem.File("main.yml")}, testSecretBinding(), LayeredOptions[config]{
Underlays: []Underlay{{Inputs: inputs}},
Overrides: []Override{{Path: Keys("value"), Value: "cli", Flag: "--value"}},
})
loadError, ok := err.(*LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != CodeOverrideLimitExceeded || len(loadError.Sources()) != maxImportSources {
t.Fatalf("mixed source/CLI limit = %#v", err)
}
}

View File

@@ -15,6 +15,8 @@ type resolvedLoad[T any] struct {
final *document.Node final *document.Node
snapshots []resolutionSnapshot snapshots []resolutionSnapshot
applications []historyApplication applications []historyApplication
inheritance *inheritanceScope
contexts map[string]ResourceContext
} }
func decodeLoadInput[T any](schema *schemaDescriptor, definition Definition[T], input assembledInput) (decodedConfiguration, []Diagnostic) { func decodeLoadInput[T any](schema *schemaDescriptor, definition Definition[T], input assembledInput) (decodedConfiguration, []Diagnostic) {
@@ -34,7 +36,9 @@ func decodeLoadInput[T any](schema *schemaDescriptor, definition Definition[T],
} }
return decodedConfiguration{}, diagnostics return decodedConfiguration{}, diagnostics
} }
if input.composition != nil { if input.inheritance != nil {
applyCompositionOrigins(&decoded, input.inheritance.origins)
} else if input.composition != nil {
applyCompositionOrigins(&decoded, input.composition.origins) applyCompositionOrigins(&decoded, input.composition.origins)
} }
@@ -45,13 +49,15 @@ func resolveLoadInput[T any](schema *schemaDescriptor, definition Definition[T],
var diagnostics []Diagnostic var diagnostics []Diagnostic
configuration := decoded.value.Interface().(T) configuration := decoded.value.Interface().(T)
metadata := metadataFromDecoded(decoded) metadata := metadataFromDecoded(decoded)
if input.composition != nil { if input.inheritance != nil {
applyInheritanceMetadata(&metadata, input.composition.root, input.composition.origins, input.composition.overlays, input.inheritance)
} else if input.composition != nil {
applyOverlayMetadata(&metadata, input.composition.overlays) applyOverlayMetadata(&metadata, input.composition.overlays)
} }
final := decoded.final final := decoded.final
var resolutionSnapshots []resolutionSnapshot var resolutionSnapshots []resolutionSnapshot
if definition.Resolve != nil { if definition.Resolve != nil {
entry := newResolutionInput(metadata, decoded.final, input.sources) entry := newResolutionInput(metadata, decoded.final, input.sources, input.contexts)
resolved := runResolution(&configuration, definition.Resolve, schema, decoded, input.document, entry) resolved := runResolution(&configuration, definition.Resolve, schema, decoded, input.document, entry)
diagnostics = append(diagnostics, resolved.diagnostics...) diagnostics = append(diagnostics, resolved.diagnostics...)
if hasErrors(resolved.diagnostics) { if hasErrors(resolved.diagnostics) {
@@ -67,7 +73,7 @@ func resolveLoadInput[T any](schema *schemaDescriptor, definition Definition[T],
} else if input.document != nil { } else if input.document != nil {
applications = []historyApplication{{layer: LayerBase, value: input.document}} applications = []historyApplication{{layer: LayerBase, value: input.document}}
} }
return resolvedLoad[T]{configuration: configuration, decoded: decoded, metadata: metadata, final: final, snapshots: resolutionSnapshots, applications: applications}, diagnostics return resolvedLoad[T]{configuration: configuration, decoded: decoded, metadata: metadata, final: final, snapshots: resolutionSnapshots, applications: applications, inheritance: input.inheritance, contexts: input.contexts}, diagnostics
} }
func detachLoadConfiguration[T any](schema *schemaDescriptor, configuration T, mode secretCloneMode) (T, *Diagnostic) { func detachLoadConfiguration[T any](schema *schemaDescriptor, configuration T, mode secretCloneMode) (T, *Diagnostic) {
@@ -95,6 +101,7 @@ func publishLoadResult[T any](schema *schemaDescriptor, definition Definition[T]
inspection := &Inspection{ inspection := &Inspection{
metadata: metadata, metadata: metadata,
sources: cloneSourceInfos(accumulator.sources), sources: cloneSourceInfos(accumulator.sources),
contexts: cloneResourceContexts(resolved.contexts),
diagnostics: canonicalDiagnostics(accumulator.diagnostics), diagnostics: canonicalDiagnostics(accumulator.diagnostics),
baseline: decoded.baseline, baseline: decoded.baseline,
input: decoded.final, input: decoded.final,
@@ -102,6 +109,7 @@ func publishLoadResult[T any](schema *schemaDescriptor, definition Definition[T]
final: final, final: final,
resolution: resolutionSnapshots, resolution: resolutionSnapshots,
applications: applications, applications: applications,
inheritance: inheritanceOperations(resolved.inheritance),
secrets: secrets, secrets: secrets,
secretReferences: cloneSecretReferences(references), secretReferences: cloneSecretReferences(references),
} }

View File

@@ -0,0 +1,37 @@
package engine
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document"
// mappingIndex keeps ordered document entries and their lookup positions in
// sync while one composition or overlay is assembled.
type mappingIndex struct {
byNode map[*document.Node]map[string]int
}
func (m *mappingIndex) entries(node *document.Node) map[string]int {
if m.byNode == nil {
m.byNode = make(map[*document.Node]map[string]int)
}
if entries, ok := m.byNode[node]; ok {
return entries
}
entries := make(map[string]int, len(node.Mapping))
for index, entry := range node.Mapping {
entries[entry.Key] = index
}
m.byNode[node] = entries
return entries
}
func (m *mappingIndex) find(node *document.Node, key string) int {
if index, ok := m.entries(node)[key]; ok {
return index
}
return -1
}
func (m *mappingIndex) append(node *document.Node, entry document.Entry) {
entries := m.entries(node)
entries[entry.Key] = len(node.Mapping)
node.Mapping = append(node.Mapping, entry)
}

View File

@@ -13,6 +13,7 @@ type Metadata struct {
historicalPaths map[string][]Path historicalPaths map[string][]Path
fields map[string]FieldInfo fields map[string]FieldInfo
history map[string][]Change history map[string][]Change
inheritance []inheritanceOperation
} }
// Lookup returns information for an effective path. Unknown and invalid paths // Lookup returns information for an effective path. Unknown and invalid paths
@@ -25,17 +26,19 @@ func (m Metadata) Lookup(path Path) (FieldInfo, bool) {
if !ok { if !ok {
return FieldInfo{}, false return FieldInfo{}, false
} }
return cloneFieldInfo(info), true detached := cloneFieldInfo(info)
detached.Origin.Inheritance = projectInheritanceSteps(detached.Origin)
return detached, true
} }
// IsExplicit reports whether a surviving input contribution explicitly defined // IsExplicit reports whether a surviving input contribution, including one
// path. // inherited from a sibling, explicitly defined path.
func (m Metadata) IsExplicit(path Path) bool { func (m Metadata) IsExplicit(path Path) bool {
info, ok := m.Lookup(path) info, ok := m.Lookup(path)
return ok && info.Explicit return ok && info.Explicit
} }
// History returns detached definition and resolution records for path. A record's // History returns detached definition, inheritance, and resolution records for path. A record's
// Path can be an ancestor when that container operation replaced or removed path. // Path can be an ancestor when that container operation replaced or removed path.
// Removed historical paths can have history even though Lookup returns false. // Removed historical paths can have history even though Lookup returns false.
// Unknown and invalid paths return nil. // Unknown and invalid paths return nil.
@@ -43,19 +46,12 @@ func (m Metadata) History(path Path) []Change {
if !m.hasHistoricalPath(path) { if !m.hasHistoricalPath(path) {
return nil return nil
} }
history, ok := m.history[path.String()] history := m.history[path.String()]
if !ok { return historyWithInheritance(path, history, m.inheritance)
return nil
}
cloned := make([]Change, len(history))
for i, change := range history {
cloned[i] = cloneChange(change)
}
return cloned
} }
// HistoricalPaths returns the paths observed during loading, including effective // HistoricalPaths returns the paths observed during loading, including effective
// paths and paths removed by overlays or recorded resolution. It includes default // paths and paths removed by overlays, inheritance, or recorded resolution. It includes default
// paths and the root, but not paths that never existed. The zero Metadata returns // paths and the root, but not paths that never existed. The zero Metadata returns
// an empty slice. // an empty slice.
// //
@@ -136,6 +132,16 @@ func finalizeMetadataPaths(m *Metadata, final *document.Node, applications []his
} }
walkDocument(application.value, prefix, add) walkDocument(application.value, prefix, add)
} }
for _, operation := range m.inheritance {
if operation.control != nil {
add(operation.control.path, nil)
continue
}
if operation.transfer != nil {
walkDocument(operation.parent, operation.transfer.to, add)
walkDocument(operation.effective, operation.transfer.to, add)
}
}
} }
// effectivePathIndex records document-derived paths with their contextual segment // effectivePathIndex records document-derived paths with their contextual segment

View File

@@ -0,0 +1,284 @@
package engine
import (
"errors"
"io/fs"
"reflect"
"strings"
"testing"
"testing/fstest"
)
func TestNativeFilesystemImportsAndOverlay(t *testing.T) {
filesystem := fstest.MapFS{
"config/main.yml": &fstest.MapFile{Data: []byte(`configkit:
imports:
- path: ../shared.yml
- glob: fragments/*.yml
items: {main: main}
`)},
"shared.yml": &fstest.MapFile{Data: []byte("items: {shared: shared}\n")},
"config/fragments/b.yml": &fstest.MapFile{Data: []byte("items: {beta: beta}\n")},
"config/fragments/a.yml": &fstest.MapFile{Data: []byte("items: {alpha: alpha}\n")},
"overlays/testing.yml": &fstest.MapFile{Data: []byte("items: {main: overlaid}\n")},
}
namespace := newNativeNamespace("bundle", filesystem)
result, err := testLoad[importTestConfig](filesystemFile(namespace, "config/main.yml"), Options[importTestConfig]{Imports: true, Overlay: "testing"})
if err != nil {
t.Fatal(err)
}
want := map[string]string{"main": "overlaid", "shared": "shared", "alpha": "alpha", "beta": "beta"}
if !reflect.DeepEqual(result.Config.Items, want) {
t.Fatalf("items = %#v, want %#v", result.Config.Items, want)
}
sources := result.Sources()
wantPaths := []string{"config/main.yml", "shared.yml", "config/fragments/a.yml", "config/fragments/b.yml", "overlays/testing.yml"}
gotPaths := make([]string, len(sources))
for index, source := range sources {
gotPaths[index] = source.Path
}
if !reflect.DeepEqual(gotPaths, wantPaths) {
t.Fatalf("source paths = %#v, want %#v", gotPaths, wantPaths)
}
}
func TestNativeFilesystemNamespaceAndPathFailures(t *testing.T) {
first := fstest.MapFS{"main.yml": &fstest.MapFile{Data: []byte("items: {name: first}\n")}}
second := fstest.MapFS{"main.yml": &fstest.MapFile{Data: []byte("items: {name: second}\n")}}
firstNamespace := newNativeNamespace("same", first)
secondNamespace := newNativeNamespace("same", second)
if reflect.TypeOf(first).Comparable() {
t.Fatal("test filesystem must have a non-comparable dynamic value")
}
if firstNamespace.token == secondNamespace.token {
t.Fatal("independent filesystem constructors shared identity")
}
if newNativeAccess(first, ".", firstNamespace.token).identity("main.yml") == newNativeAccess(second, ".", secondNamespace.token).identity("main.yml") {
t.Fatal("equal paths from separate filesystems have the same identity")
}
access := newNativeAccess(first, ".", firstNamespace.token)
if access.syntheticIdentity() == access.syntheticIdentity() || access.syntheticIdentity() == access.identity("<selected-source>") {
t.Fatal("independent synthetic sources shared a file identity")
}
for _, test := range []struct {
name string
namespace *nativeNamespace
want string
}{
{name: "first", namespace: firstNamespace, want: "first"},
{name: "second", namespace: secondNamespace, want: "second"},
} {
t.Run(test.name, func(t *testing.T) {
result, err := testLoad[importTestConfig](filesystemFile(test.namespace, "main.yml"))
if err != nil || result.Config.Items["name"] != test.want {
t.Fatalf("result = %#v, %v", result, err)
}
})
}
for _, name := range []string{".", "../main.yml", "/main.yml", "missing.yml"} {
_, diagnostics := readSource(filesystemFile(firstNamespace, name), "source-1", defaultSourceLimits())
if len(diagnostics) != 1 {
t.Errorf("%q diagnostics = %#v", name, diagnostics)
}
}
missingImport := fstest.MapFS{"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: host-only.yml}]}\n")}}
_, err := testLoad[importTestConfig](filesystemFile(newNativeNamespace("bundle", missingImport), "main.yml"), Options[importTestConfig]{Imports: true})
if err == nil {
t.Fatal("missing virtual import fell back to the host filesystem")
}
loadError, ok := err.(*LoadError)
if !ok || loadError.Diagnostics()[0].Code != CodeImportMissing {
t.Fatalf("missing import error = %#v", err)
}
}
func TestNativeFilesystemClosesFilesOnSuccessAndFailure(t *testing.T) {
for _, fallback := range []bool{false, true} {
t.Run(map[bool]string{false: "ReadDirFile", true: "ReadDir fallback"}[fallback], func(t *testing.T) {
filesystem := &observedFilesystem{files: fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{glob: 'parts/*.yml'}]}\nitems: {main: main}\n")},
"parts/one.yml": &fstest.MapFile{Data: []byte("items: {one: one}\n")},
}, fallback: fallback}
result, err := testLoad[importTestConfig](filesystemFile(newNativeNamespace("bundle", filesystem), "main.yml"), Options[importTestConfig]{Imports: true})
if err != nil || result.Config.Items["one"] != "one" {
t.Fatalf("result = %#v, %v", result, err)
}
filesystem.assertClosed(t)
})
}
for _, failure := range []struct {
name string
file string
failRead string
failStat string
failDir string
}{
{name: "read", file: "items: {one: one}\n", failRead: "parts/one.yml"},
{name: "stat", file: "items: {one: one}\n", failStat: "parts/one.yml"},
{name: "parse", file: "items: [unterminated\n"},
{name: "directory", file: "items: {one: one}\n", failDir: "parts"},
} {
t.Run(failure.name, func(t *testing.T) {
filesystem := &observedFilesystem{files: fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{glob: 'parts/*.yml'}]}\n")},
"parts/one.yml": &fstest.MapFile{Data: []byte(failure.file)},
}, failRead: failure.failRead, failStat: failure.failStat, failDir: failure.failDir}
_, err := testLoad[importTestConfig](filesystemFile(newNativeNamespace("bundle", filesystem), "main.yml"), Options[importTestConfig]{Imports: true})
if err == nil {
t.Fatal("expected a load failure")
}
if strings.Contains(err.Error(), "tracking failure canary") {
t.Fatal("unsafe filesystem error escaped")
}
filesystem.assertClosed(t)
})
}
}
func TestNativeFilesystemRootAndOptionalImports(t *testing.T) {
filesystem := fstest.MapFS{
"config/sub/main.yml": &fstest.MapFile{Data: []byte(`configkit:
imports:
- path: ../shared.yml
- path: absent.yml
optional: true
- glob: absent/*.yml
optional: true
`)},
"config/shared.yml": &fstest.MapFile{Data: []byte("items: {shared: retained}\n")},
"outside.yml": &fstest.MapFile{Data: []byte("items: {outside: blocked}\n")},
}
namespace := newNativeNamespace("bundle", filesystem)
source := filesystemFile(namespace, "config/sub/main.yml")
parsed, diagnostics := parseSource(source, "source-1", defaultParseLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
access := newNativeAccess(filesystem, "config", namespace.token)
composed, sources, diagnostics := loadImportedWithAccess(access, "config/sub", source.path, parsed, nil, defaultImportLimits(), &importAccounting{})
if len(diagnostics) != 0 || composed == nil || len(sources) != 2 || sources[1].Path != "config/shared.yml" {
t.Fatalf("contained imports = (%#v, %#v, %#v)", composed, sources, diagnostics)
}
outside := fstest.MapFS{
"config/sub/main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: ../../outside.yml}]}\n")},
"outside.yml": &fstest.MapFile{Data: []byte("items: {outside: blocked}\n")},
}
outsideNamespace := newNativeNamespace("bundle", outside)
parsed, diagnostics = parseSource(filesystemFile(outsideNamespace, "config/sub/main.yml"), "source-1", defaultParseLimits())
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
_, _, diagnostics = loadImportedWithAccess(newNativeAccess(outside, "config", outsideNamespace.token), "config/sub", "config/sub/main.yml", parsed, nil, defaultImportLimits(), &importAccounting{})
if len(diagnostics) != 1 || diagnostics[0].Code != CodeImportOutsideRoot {
t.Fatalf("escaping import diagnostics = %#v", diagnostics)
}
}
func TestNativeFilesystemDetectsRepeatedAndCyclicImports(t *testing.T) {
for _, test := range []struct {
name string
files fstest.MapFS
code Code
}{
{
name: "repeated",
files: fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: child.yml}, {path: child.yml}]}\n")},
"child.yml": &fstest.MapFile{Data: []byte("items: {child: child}\n")},
},
code: CodeImportRepeated,
},
{
name: "cycle",
files: fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: child.yml}]}\n")},
"child.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{path: main.yml}]}\n")},
},
code: CodeImportCycle,
},
} {
t.Run(test.name, func(t *testing.T) {
_, err := testLoad[importTestConfig](filesystemFile(newNativeNamespace("bundle", test.files), "main.yml"), Options[importTestConfig]{Imports: true})
loadError, ok := err.(*LoadError)
if !ok || len(loadError.Diagnostics()) == 0 || loadError.Diagnostics()[0].Code != test.code {
t.Fatalf("load error = %#v, want %q", err, test.code)
}
})
}
}
type observedFilesystem struct {
files fstest.MapFS
opened int
closed int
fallback bool
failRead string
failStat string
failDir string
}
func (f *observedFilesystem) Open(name string) (fs.File, error) {
inner, err := f.files.Open(name)
if err != nil {
return nil, err
}
f.opened++
file := &observedFile{File: inner, owner: f, name: name}
if directory, ok := inner.(fs.ReadDirFile); ok && !f.fallback {
return &observedDirectoryFile{observedFile: file, directory: directory}, nil
}
return file, nil
}
func (f *observedFilesystem) ReadDir(name string) ([]fs.DirEntry, error) {
if name == f.failDir {
return nil, errors.New("tracking failure canary")
}
return fs.ReadDir(f.files, name)
}
func (f *observedFilesystem) assertClosed(t *testing.T) {
t.Helper()
if f.opened == 0 || f.opened != f.closed {
t.Fatalf("filesystem opens=%d closes=%d", f.opened, f.closed)
}
}
type observedFile struct {
fs.File
owner *observedFilesystem
name string
}
func (f *observedFile) Read(data []byte) (int, error) {
if f.name == f.owner.failRead {
return 0, errors.New("tracking failure canary")
}
return f.File.Read(data)
}
func (f *observedFile) Stat() (fs.FileInfo, error) {
if f.name == f.owner.failStat {
return nil, errors.New("tracking failure canary")
}
return f.File.Stat()
}
func (f *observedFile) Close() error {
f.owner.closed++
return f.File.Close()
}
type observedDirectoryFile struct {
*observedFile
directory fs.ReadDirFile
}
func (f *observedDirectoryFile) ReadDir(count int) ([]fs.DirEntry, error) {
if f.name == f.owner.failDir {
return nil, errors.New("tracking failure canary")
}
return f.directory.ReadDir(count)
}

View File

@@ -22,8 +22,8 @@ func selectOptions[T any](source Source, supplied []Options[T]) (Options[T], []D
if source.kind == sourceNone && options.Overlay != "" && options.Overlay != "default" { if source.kind == sourceNone && options.Overlay != "" && options.Overlay != "default" {
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "NoSource cannot select an active overlay")) diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "NoSource cannot select an active overlay"))
} }
if options.Filesystem == nil && (source.kind == sourceBytes || source.kind == sourceReader) && (options.Imports || activeOverlay(options.Overlay)) { if options.Filesystem == nil && (source.kind == sourceBytes || source.kind == sourceReader || source.kind == sourceMapping) && (options.Imports || activeOverlay(options.Overlay)) {
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "byte and reader imports or overlays require filesystem context")) diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsInvalid, "synthetic-source imports or overlays require filesystem context"))
} }
if definition.Version != nil { if definition.Version != nil {
if !definition.Version.Path.valid() { if !definition.Version.Path.valid() {

View File

@@ -18,78 +18,105 @@ type displacedDefinition struct {
} }
func overlayDocuments(base *composedDocument, overlay *document.Node) (*composedDocument, []Diagnostic) { func overlayDocuments(base *composedDocument, overlay *document.Node) (*composedDocument, []Diagnostic) {
if base == nil || base.root == nil || base.root.Kind != document.Mapping || overlay == nil || overlay.Kind != document.Mapping { if base == nil {
return nil, []Diagnostic{{ return nil, invalidOverlayDocument()
}
assembly := newAssemblyAccumulator(cloneComposedDocument(base))
if diagnostics := assembly.applyOverlay(overlay, nil, nil, LayerOverlay); len(diagnostics) != 0 {
return nil, diagnostics
}
return assembly.current, nil
}
func invalidOverlayDocument() []Diagnostic {
return []Diagnostic{{
Code: CodeOverlayInvalid, Severity: SeverityError, Code: CodeOverlayInvalid, Severity: SeverityError,
Message: "overlay requires mapping document roots", Message: "overlay requires mapping document roots",
}} }}
}
result := cloneComposedDocument(base)
mergeOverlayOrigin(result.origins, Path{}, overlay.Location, LayerOverlay)
mergeOverlayMappings(result, result.root, overlay, Path{}, LayerOverlay, true)
result.applications = append(result.applications, historyApplication{layer: LayerOverlay, value: overlay})
return result, nil
} }
func mergeOverlayMappings(result *composedDocument, target, overlay *document.Node, current Path, layer Layer, recordReplacements bool) { func (a *assemblyAccumulator) applyOverlay(overlay *document.Node, incomingOrigins map[string]Origin, applications []historyApplication, layer Layer) []Diagnostic {
for _, incoming := range overlay.Mapping { if a == nil || a.current == nil || a.current.root == nil || a.current.root.Kind != document.Mapping || overlay == nil || overlay.Kind != document.Mapping {
path := current.Key(incoming.Key) return invalidOverlayDocument()
index := documentEntryIndex(target.Mapping, incoming.Key)
if index < 0 {
cloned := incoming
cloned.Value = document.Clone(incoming.Value)
target.Mapping = append(target.Mapping, cloned)
collectOverlayOrigins(result.origins, incoming.Value, path, layer)
continue
} }
mergeOriginContribution(a.current.origins, Path{}, incomingOrigin(incomingOrigins, Path{}, overlay, layer))
existing := target.Mapping[index].Value mergeOverlayMappings(a.current, a.current.root, overlay, Path{}, layer, true, incomingOrigins, &a.index)
if existing.Kind == document.Mapping && incoming.Value.Kind == document.Mapping { if len(applications) == 0 {
mergeOverlayOrigin(result.origins, path, incoming.Value.Location, layer) a.current.applications = append(a.current.applications, historyApplication{layer: layer, value: overlay})
mergeOverlayMappings(result, existing, incoming.Value, path, layer, recordReplacements) } else {
continue for _, application := range applications {
application.layer = layer
a.current.applications = append(a.current.applications, application)
} }
}
return nil
}
func mergeOverlayMappings(result *composedDocument, target, overlay *document.Node, current Path, layer Layer, recordReplacements bool, incomingOrigins map[string]Origin, index *mappingIndex) {
mergeRecursiveMappings(target, overlay, current, index, recursiveMergeHooks{
added: func(path Path, incoming *document.Node) {
collectOverlayOriginsFrom(result.origins, incomingOrigins, incoming, path, layer)
},
combined: func(path Path, incoming *document.Node) {
mergeOriginContribution(result.origins, path, incomingOrigin(incomingOrigins, path, incoming, layer))
},
replaced: func(path Path, existing, incoming *document.Node) {
displaced := displacedOrigins(existing, result.origins, path) displaced := displacedOrigins(existing, result.origins, path)
previous := cloneOrigin(result.origins[path.String()]) previous := cloneOrigin(result.origins[path.String()])
deleteOriginSubtree(result.origins, path) deleteOriginSubtree(result.origins, path, existing)
target.Mapping[index] = incoming collectOverlayOriginsFrom(result.origins, incomingOrigins, incoming, path, layer)
target.Mapping[index].Value = document.Clone(incoming.Value)
collectOverlayOrigins(result.origins, incoming.Value, path, layer)
if recordReplacements { if recordReplacements {
result.overlays = append(result.overlays, overlayRecord{ result.overlays = append(result.overlays, overlayRecord{
change: Change{ change: Change{
Path: path.clone(), Origin: overlayOrigin(layer, incoming.Value.Location), Path: path.clone(), Origin: incomingOrigin(incomingOrigins, path, incoming, layer),
Previous: previousOrigins(previous), Previous: previousOrigins(previous),
}, },
displaced: displaced, displaced: displaced,
}) })
} }
} },
})
} }
func collectOverlayOrigins(origins map[string]Origin, node *document.Node, path Path, layer Layer) { func collectOverlayOrigins(origins map[string]Origin, node *document.Node, path Path, layer Layer) {
origins[path.String()] = overlayOrigin(layer, node.Location) collectOverlayOriginsFrom(origins, nil, node, path, layer)
}
func collectOverlayOriginsFrom(origins, incomingOrigins map[string]Origin, node *document.Node, path Path, layer Layer) {
origins[path.String()] = incomingOrigin(incomingOrigins, path, node, layer)
switch node.Kind { switch node.Kind {
case document.Mapping: case document.Mapping:
for _, entry := range node.Mapping { for _, entry := range node.Mapping {
collectOverlayOrigins(origins, entry.Value, path.Key(entry.Key), layer) collectOverlayOriginsFrom(origins, incomingOrigins, entry.Value, path.Key(entry.Key), layer)
} }
case document.Sequence: case document.Sequence:
for index, child := range node.Sequence { for index, child := range node.Sequence {
collectOverlayOrigins(origins, child, path.Index(index), layer) collectOverlayOriginsFrom(origins, incomingOrigins, child, path.Index(index), layer)
} }
} }
} }
func mergeOverlayOrigin(origins map[string]Origin, path Path, location document.Location, layer Layer) { func mergeOverlayOrigin(origins map[string]Origin, path Path, location document.Location, layer Layer) {
mergeOriginContribution(origins, path, overlayOrigin(layer, location))
}
func mergeOriginContribution(origins map[string]Origin, path Path, incoming Origin) {
pointer := path.String() pointer := path.String()
origin := cloneOrigin(origins[pointer]) origin := origins[pointer]
origin.Layer = layer origin.Layer = incoming.Layer
origin.Locations = append(origin.Locations, publicLocation(location)) origin.Locations = append(origin.Locations, incoming.Locations...)
origins[pointer] = origin origins[pointer] = origin
} }
func incomingOrigin(origins map[string]Origin, path Path, node *document.Node, layer Layer) Origin {
if origin, ok := origins[path.String()]; ok {
origin = cloneOrigin(origin)
origin.Layer = layer
return origin
}
return overlayOrigin(layer, node.Location)
}
func overlayOrigin(layer Layer, location document.Location) Origin { func overlayOrigin(layer Layer, location document.Location) Origin {
return Origin{Layer: layer, Locations: []Location{publicLocation(location)}} return Origin{Layer: layer, Locations: []Location{publicLocation(location)}}
} }
@@ -129,11 +156,16 @@ func collectDisplacedOrigins(displaced *[]displacedDefinition, node *document.No
} }
} }
func deleteOriginSubtree(origins map[string]Origin, path Path) { func deleteOriginSubtree(origins map[string]Origin, path Path, node *document.Node) {
prefix := path.String() delete(origins, path.String())
for pointer := range origins { switch node.Kind {
if pointer == prefix || strings.HasPrefix(pointer, prefix+"/") { case document.Mapping:
delete(origins, pointer) for _, entry := range node.Mapping {
deleteOriginSubtree(origins, path.Key(entry.Key), entry.Value)
}
case document.Sequence:
for index, child := range node.Sequence {
deleteOriginSubtree(origins, path.Index(index), child)
} }
} }
} }

View File

@@ -76,3 +76,44 @@ to_scalar: scalar
t.Fatal("overlay result aliases an input document") t.Fatal("overlay result aliases an input document")
} }
} }
func TestOwnedOverlayKeepsCompositeContributorsAndAcquiredInputs(t *testing.T) {
baseInput := parseCompositionDocument(t, "source-1", "catalog:\n base: true\n")
base, diagnostics := composeDocuments(baseInput)
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
first := parseCompositionDocument(t, "source-2", "catalog:\n first: true\n")
second := parseCompositionDocument(t, "source-3", "catalog:\n second: true\n")
firstBefore, secondBefore := document.Clone(first), document.Clone(second)
incoming, diagnostics := composeDocuments(first, second)
if len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
assembly := newAssemblyAccumulator(base)
if diagnostics := assembly.applyOverlay(incoming.root, incoming.origins, incoming.applications, LayerOverlay); len(diagnostics) != 0 {
t.Fatal(diagnostics)
}
for _, key := range []string{"base", "first", "second"} {
if _, ok := documentAtPath(assembly.current.root, Keys("catalog", key)); !ok {
t.Fatalf("catalog is missing %q", key)
}
}
origin := assembly.current.origins[Keys("catalog").String()]
if got, want := origin.Locations, []Location{{SourceID: "source-1", Line: 2, Column: 3}, {SourceID: "source-2", Line: 2, Column: 3}, {SourceID: "source-3", Line: 2, Column: 3}}; !reflect.DeepEqual(got, want) {
t.Fatalf("catalog origin locations = %#v, want %#v", got, want)
}
if len(assembly.current.applications) != 3 {
t.Fatalf("history applications = %d, want three source contributions", len(assembly.current.applications))
}
for _, application := range assembly.current.applications[1:] {
if application.layer != LayerOverlay {
t.Fatalf("incoming application layer = %q", application.layer)
}
}
assembly.current.root.Mapping[0].Value.Mapping[1].Value.Text = "false"
if !reflect.DeepEqual(first, firstBefore) || !reflect.DeepEqual(second, secondBefore) ||
!reflect.DeepEqual(first, assembly.current.applications[1].value) || !reflect.DeepEqual(second, assembly.current.applications[2].value) {
t.Fatal("working tree mutation changed an acquired input")
}
}

View File

@@ -44,7 +44,7 @@ func prepareOverrides(base *composedDocument, sources []SourceInfo, overrides []
allSources := cloneSourceInfos(sources) allSources := cloneSourceInfos(sources)
for index, override := range overrides { for index, override := range overrides {
sourceID := "source-" + strconv.Itoa(len(sources)+index+1) sourceID := "source-" + strconv.Itoa(len(sources)+index+1)
info := SourceInfo{ID: sourceID, Label: override.Flag, Layer: LayerCLI} info := SourceInfo{ID: sourceID, Label: override.Flag, Layer: LayerCLI, Kind: SourceKindCLI}
allSources = append(allSources, info) allSources = append(allSources, info)
prepared[index].source = info prepared[index].source = info
} }
@@ -94,11 +94,12 @@ func prepareOverrides(base *composedDocument, sources []SourceInfo, overrides []
} }
pathAccepted[index] = true pathAccepted[index] = true
builder := overrideValueBuilder{sourceID: prepared[index].source.ID, active: make(map[valueVisit]bool)} builder := valueBuilder{sourceID: prepared[index].source.ID, active: make(map[valueVisit]bool)}
value, issue := builder.build(override.Value, override.Path, len(override.Path.segments)) value, issue := builder.build(override.Value, override.Path, len(override.Path.segments))
if issue != nil { if issue != nil {
issue.Primary = location diagnostic := issue.overrideDiagnostic()
diagnostics = append(diagnostics, *issue) diagnostic.Primary = location
diagnostics = append(diagnostics, diagnostic)
continue continue
} }
if relative, ok := overrideDirectivePath(value, Path{}); ok { if relative, ok := overrideDirectivePath(value, Path{}); ok {

View File

@@ -11,31 +11,42 @@ func applyOverrides(base *composedDocument, sources []SourceInfo, overrides []Ov
if len(overrides) == 0 { if len(overrides) == 0 {
return base, cloneSourceInfos(sources), nil return base, cloneSourceInfos(sources), nil
} }
prepared, allSources, diagnostics := prepareOverrides(base, sources, overrides, rule, limits) assembly := newAssemblyAccumulator(cloneComposedDocument(base))
allSources, diagnostics := assembly.applyOverrides(sources, overrides, rule, limits)
if len(diagnostics) != 0 { if len(diagnostics) != 0 {
return nil, allSources, diagnostics return nil, allSources, diagnostics
} }
result := cloneComposedDocument(base) return assembly.current, allSources, nil
for _, override := range prepared {
applyPreparedOverride(result, override)
result.inputBytes += override.bytes
result.inputNodes += override.nodes
}
return result, allSources, nil
} }
func applyPreparedOverride(result *composedDocument, override preparedOverride) { func (a *assemblyAccumulator) applyOverrides(sources []SourceInfo, overrides []Override, rule *VersionRule, limits importLimits) ([]SourceInfo, []Diagnostic) {
if len(overrides) == 0 {
return cloneSourceInfos(sources), nil
}
prepared, allSources, diagnostics := prepareOverrides(a.current, sources, overrides, rule, limits)
if len(diagnostics) != 0 {
return allSources, diagnostics
}
for _, override := range prepared {
applyPreparedOverride(a.current, override, &a.index)
a.current.inputBytes += override.bytes
a.current.inputNodes += override.nodes
}
return allSources, nil
}
func applyPreparedOverride(result *composedDocument, override preparedOverride, index *mappingIndex) {
location := document.Location{SourceID: override.source.ID} location := document.Location{SourceID: override.source.ID}
mergeOverlayOrigin(result.origins, Path{}, location, LayerCLI) mergeOverlayOrigin(result.origins, Path{}, location, LayerCLI)
current := result.root current := result.root
currentPath := Path{} currentPath := Path{}
for index, segment := range override.path.segments { for segmentIndex, segment := range override.path.segments {
path := currentPath.Key(segment.key) path := currentPath.Key(segment.key)
position := documentEntryIndex(current.Mapping, segment.key) position := index.find(current, segment.key)
if index < len(override.path.segments)-1 { if segmentIndex < len(override.path.segments)-1 {
if position < 0 { if position < 0 {
child := &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{}} child := &document.Node{Kind: document.Mapping, Location: location, Mapping: []document.Entry{}}
current.Mapping = append(current.Mapping, document.Entry{Key: segment.key, KeyLocation: location, ValueLocation: location, Value: child}) index.append(current, document.Entry{Key: segment.key, KeyLocation: location, ValueLocation: location, Value: child})
collectOverlayOrigins(result.origins, child, path, LayerCLI) collectOverlayOrigins(result.origins, child, path, LayerCLI)
current = child current = child
} else { } else {
@@ -50,16 +61,16 @@ func applyPreparedOverride(result *composedDocument, override preparedOverride)
var displaced []displacedDefinition var displaced []displacedDefinition
if position < 0 { if position < 0 {
incoming := document.Clone(override.value) incoming := document.Clone(override.value)
current.Mapping = append(current.Mapping, document.Entry{Key: segment.key, KeyLocation: location, ValueLocation: location, Value: incoming}) index.append(current, document.Entry{Key: segment.key, KeyLocation: location, ValueLocation: location, Value: incoming})
collectOverlayOrigins(result.origins, incoming, path, LayerCLI) collectOverlayOrigins(result.origins, incoming, path, LayerCLI)
} else { } else {
existing := current.Mapping[position].Value existing := current.Mapping[position].Value
displaced = affectedOverrideOrigins(existing, override.value, result.origins, path) displaced = affectedOverrideOrigins(existing, override.value, result.origins, path, index)
if existing.Kind == document.Mapping && override.value.Kind == document.Mapping { if existing.Kind == document.Mapping && override.value.Kind == document.Mapping {
mergeOverlayOrigin(result.origins, path, location, LayerCLI) mergeOverlayOrigin(result.origins, path, location, LayerCLI)
mergeOverlayMappings(result, existing, override.value, path, LayerCLI, false) mergeOverlayMappings(result, existing, override.value, path, LayerCLI, false, nil, index)
} else { } else {
deleteOriginSubtree(result.origins, path) deleteOriginSubtree(result.origins, path, existing)
current.Mapping[position] = document.Entry{Key: segment.key, KeyLocation: location, ValueLocation: location, Value: document.Clone(override.value)} current.Mapping[position] = document.Entry{Key: segment.key, KeyLocation: location, ValueLocation: location, Value: document.Clone(override.value)}
collectOverlayOrigins(result.origins, current.Mapping[position].Value, path, LayerCLI) collectOverlayOrigins(result.origins, current.Mapping[position].Value, path, LayerCLI)
} }
@@ -74,18 +85,18 @@ func applyPreparedOverride(result *composedDocument, override preparedOverride)
} }
} }
func affectedOverrideOrigins(existing, incoming *document.Node, origins map[string]Origin, path Path) []displacedDefinition { func affectedOverrideOrigins(existing, incoming *document.Node, origins map[string]Origin, path Path, index *mappingIndex) []displacedDefinition {
displaced := []displacedDefinition{{path: path.clone(), origin: cloneOrigin(origins[path.String()])}} displaced := []displacedDefinition{{path: path.clone(), origin: cloneOrigin(origins[path.String()])}}
if existing.Kind == document.Mapping && incoming.Kind == document.Mapping { if existing.Kind == document.Mapping && incoming.Kind == document.Mapping {
for _, entry := range incoming.Mapping { for _, entry := range incoming.Mapping {
position := documentEntryIndex(existing.Mapping, entry.Key) position := index.find(existing, entry.Key)
if position < 0 { if position < 0 {
continue continue
} }
childPath := path.Key(entry.Key) childPath := path.Key(entry.Key)
childExisting := existing.Mapping[position].Value childExisting := existing.Mapping[position].Value
if childExisting.Kind == document.Mapping && entry.Value.Kind == document.Mapping { if childExisting.Kind == document.Mapping && entry.Value.Kind == document.Mapping {
displaced = append(displaced, affectedOverrideOrigins(childExisting, entry.Value, origins, childPath)...) displaced = append(displaced, affectedOverrideOrigins(childExisting, entry.Value, origins, childPath, index)...)
} else { } else {
displaced = append(displaced, displacedOrigins(childExisting, origins, childPath)...) displaced = append(displaced, displacedOrigins(childExisting, origins, childPath)...)
} }

View File

@@ -1,150 +1,21 @@
package engine package engine
import ( func (issue *valueIssue) overrideDiagnostic() Diagnostic {
"math" code := CodeOverrideInvalidValue
"reflect" var message string
"slices" switch issue.kind {
"strconv" case valueDepthLimit:
code, message = CodeOverrideLimitExceeded, "configuration override exceeds the depth limit"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document" case valueNodeLimit:
) code, message = CodeOverrideLimitExceeded, "configuration override exceeds the node limit"
case valueByteLimit:
type overrideValueBuilder struct { code, message = CodeOverrideLimitExceeded, "configuration override exceeds the record string-byte limit"
sourceID string case valueNonFinite:
active map[valueVisit]bool message = "configuration override float must be finite"
bytes int64 case valueCycle:
nodes int message = "configuration override values must not contain cycles"
}
func (b *overrideValueBuilder) build(value any, path Path, depth int) (*document.Node, *Diagnostic) {
if depth > maxDocumentDepth {
return nil, overrideValueDiagnostic(CodeOverrideLimitExceeded, path, "configuration override exceeds the depth limit")
}
b.nodes++
if b.nodes > maxDocumentNodes {
return nil, overrideValueDiagnostic(CodeOverrideLimitExceeded, path, "configuration override exceeds the node limit")
}
location := document.Location{SourceID: b.sourceID}
switch value := value.(type) {
case nil:
return &document.Node{Kind: document.Null, Location: location}, nil
case string:
if !b.addBytes(len(value)) {
return nil, overrideValueDiagnostic(CodeOverrideLimitExceeded, path, "configuration override exceeds the record string-byte limit")
}
return &document.Node{Kind: document.String, Text: value, Location: location}, nil
case bool:
return &document.Node{Kind: document.Boolean, Text: strconv.FormatBool(value), Location: location}, nil
case int:
return overrideNumber(strconv.FormatInt(int64(value), 10), location), nil
case int8:
return overrideNumber(strconv.FormatInt(int64(value), 10), location), nil
case int16:
return overrideNumber(strconv.FormatInt(int64(value), 10), location), nil
case int32:
return overrideNumber(strconv.FormatInt(int64(value), 10), location), nil
case int64:
return overrideNumber(strconv.FormatInt(value, 10), location), nil
case uint:
return overrideNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint8:
return overrideNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint16:
return overrideNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint32:
return overrideNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint64:
return overrideNumber(strconv.FormatUint(value, 10), location), nil
case float32:
if math.IsNaN(float64(value)) || math.IsInf(float64(value), 0) {
return nil, overrideValueDiagnostic(CodeOverrideInvalidValue, path, "configuration override float must be finite")
}
return overrideNumber(strconv.FormatFloat(float64(value), 'g', -1, 32), location), nil
case float64:
if math.IsNaN(value) || math.IsInf(value, 0) {
return nil, overrideValueDiagnostic(CodeOverrideInvalidValue, path, "configuration override float must be finite")
}
return overrideNumber(strconv.FormatFloat(value, 'g', -1, 64), location), nil
case []any:
if value == nil {
return &document.Node{Kind: document.Null, Location: location}, nil
}
leave, diagnostic := b.enter(reflect.ValueOf(value), path)
if diagnostic != nil {
return nil, diagnostic
}
defer leave()
if len(value) > maxDocumentNodes-b.nodes {
return nil, overrideValueDiagnostic(CodeOverrideLimitExceeded, path, "configuration override exceeds the node limit")
}
node := &document.Node{Kind: document.Sequence, Location: location, Sequence: make([]*document.Node, len(value))}
for index, child := range value {
built, diagnostic := b.build(child, path.Index(index), depth+1)
if diagnostic != nil {
return nil, diagnostic
}
node.Sequence[index] = built
}
return node, nil
case map[string]any:
if value == nil {
return &document.Node{Kind: document.Null, Location: location}, nil
}
leave, diagnostic := b.enter(reflect.ValueOf(value), path)
if diagnostic != nil {
return nil, diagnostic
}
defer leave()
if len(value) > (maxDocumentNodes-b.nodes)/2 {
return nil, overrideValueDiagnostic(CodeOverrideLimitExceeded, path, "configuration override exceeds the node limit")
}
keys := make([]string, 0, len(value))
for key := range value {
keys = append(keys, key)
}
slices.Sort(keys)
node := &document.Node{Kind: document.Mapping, Location: location, Mapping: make([]document.Entry, 0, len(keys))}
for _, key := range keys {
b.nodes++ // mapping key
if !b.addBytes(len(key)) {
return nil, overrideValueDiagnostic(CodeOverrideLimitExceeded, path.Key(key), "configuration override exceeds the record string-byte limit")
}
built, diagnostic := b.build(value[key], path.Key(key), depth+1)
if diagnostic != nil {
return nil, diagnostic
}
node.Mapping = append(node.Mapping, document.Entry{Key: key, KeyLocation: location, ValueLocation: location, Value: built})
}
return node, nil
default: default:
return nil, overrideValueDiagnostic(CodeOverrideInvalidValue, path, "configuration override value has an unsupported type") message = "configuration override value has an unsupported type"
} }
} return Diagnostic{Code: code, Severity: SeverityError, Path: issue.path.clone(), Message: message}
func (b *overrideValueBuilder) addBytes(count int) bool {
if count < 0 || int64(count) > maxSourceBytes-b.bytes {
return false
}
b.bytes += int64(count)
return true
}
func (b *overrideValueBuilder) enter(value reflect.Value, path Path) (func(), *Diagnostic) {
visit, ok := visitFor(value)
if !ok {
return func() {}, nil
}
if b.active[visit] {
return nil, overrideValueDiagnostic(CodeOverrideInvalidValue, path, "configuration override values must not contain cycles")
}
b.active[visit] = true
return func() { delete(b.active, visit) }, nil
}
func overrideValueDiagnostic(code Code, path Path, message string) *Diagnostic {
return &Diagnostic{Code: code, Severity: SeverityError, Path: path.clone(), Message: message}
}
func overrideNumber(text string, location document.Location) *document.Node {
return &document.Node{Kind: document.Number, Text: text, Location: location}
} }

View File

@@ -53,6 +53,9 @@ func parseSource(source Source, sourceID string, limits parseLimits) (parsedSour
if len(diagnostics) != 0 || read.info == nil { if len(diagnostics) != 0 || read.info == nil {
return parsedSource{source: read.info}, diagnostics return parsedSource{source: read.info}, diagnostics
} }
if read.document != nil {
return parsedSource{document: read.document, source: read.info, bytes: read.bytes}, nil
}
root, issues := yamladapter.Parse(bytes.NewReader(read.data), sourceID, limits.document) root, issues := yamladapter.Parse(bytes.NewReader(read.data), sourceID, limits.document)
if len(issues) != 0 { if len(issues) != 0 {

View File

@@ -6,11 +6,24 @@ type Layer string
const ( const (
LayerDefault Layer = "default" LayerDefault Layer = "default"
LayerBase Layer = "base" LayerBase Layer = "base"
LayerUnderlay Layer = "underlay"
LayerOverlay Layer = "overlay" LayerOverlay Layer = "overlay"
LayerCLI Layer = "cli" LayerCLI Layer = "cli"
LayerResolution Layer = "resolution" LayerResolution Layer = "resolution"
) )
// SourceKind identifies source acquisition independently of precedence.
type SourceKind string
const (
SourceKindFile SourceKind = "file"
SourceKindFilesystem SourceKind = "filesystem"
SourceKindBytes SourceKind = "bytes"
SourceKindReader SourceKind = "reader"
SourceKindMapping SourceKind = "mapping"
SourceKindCLI SourceKind = "cli"
)
// ChangeKind identifies a recorded operation. The zero value is // ChangeKind identifies a recorded operation. The zero value is
// used for input and default origins. // used for input and default origins.
type ChangeKind string type ChangeKind string
@@ -19,6 +32,9 @@ const (
// ChangeDecoding records the built-in transition to canonical typed values. // ChangeDecoding records the built-in transition to canonical typed values.
// It is inspection metadata, not an operation accepted by Resolution.Record. // It is inspection metadata, not an operation accepted by Resolution.Record.
ChangeDecoding ChangeKind = "decoding" ChangeDecoding ChangeKind = "decoding"
// ChangeInheritance records structural input expansion. It is inspection
// metadata, not an operation accepted by Resolution.Record.
ChangeInheritance ChangeKind = "inheritance"
ChangeDerivedDefault ChangeKind = "derived_default" ChangeDerivedDefault ChangeKind = "derived_default"
ChangeNormalization ChangeKind = "normalization" ChangeNormalization ChangeKind = "normalization"
) )
@@ -89,6 +105,7 @@ const (
CodeImportRepeated Code = "import.repeated" CodeImportRepeated Code = "import.repeated"
CodeImportLimitExceeded Code = "import.limit_exceeded" CodeImportLimitExceeded Code = "import.limit_exceeded"
CodeImportVersion Code = "import.version" CodeImportVersion Code = "import.version"
CodeUnderlayVersion Code = "underlay.version"
CodeOverlayInvalid Code = "overlay.invalid" CodeOverlayInvalid Code = "overlay.invalid"
CodeOverlayMissing Code = "overlay.missing" CodeOverlayMissing Code = "overlay.missing"
@@ -146,6 +163,11 @@ const (
CodeOverrideLimitExceeded Code = "override.limit_exceeded" CodeOverrideLimitExceeded Code = "override.limit_exceeded"
CodeExplainLimitExceeded Code = "explain.limit_exceeded" CodeExplainLimitExceeded Code = "explain.limit_exceeded"
CodeInheritanceInvalid Code = "inheritance.invalid"
CodeInheritanceMissing Code = "inheritance.missing"
CodeInheritanceCycle Code = "inheritance.cycle"
CodeInheritanceVersion Code = "inheritance.version"
) )
// Location identifies a position in a configuration source. Zero line and column // Location identifies a position in a configuration source. Zero line and column
@@ -156,15 +178,19 @@ type Location struct {
Column int Column int
} }
// SourceInfo describes a source discovered during a load. Path is a canonical // SourceInfo describes a source discovered during a load. Kind identifies
// local path when one exists. ImportedFrom identifies the declaration that // acquisition independently of Layer. Path is a canonical host path or a
// included this source. // namespace-relative filesystem path; synthetic sources have no Path.
// ImportedFrom identifies the declaration that included this source.
type SourceInfo struct { type SourceInfo struct {
ID string ID string
Label string Label string
Path string Path string
Layer Layer Layer Layer
Kind SourceKind
UnderlayIndex int
ImportedFrom *Location ImportedFrom *Location
namespace *namespaceToken
} }
// Diagnostic is a safe structured problem report. Message and source labels must // Diagnostic is a safe structured problem report. Message and source labels must
@@ -178,13 +204,25 @@ type Diagnostic struct {
Related []Location Related []Location
} }
// InheritanceStep identifies a concrete payload transfer controlled by a
// relationship directive. Locations belong to that directive, not its value.
type InheritanceStep struct {
From Path
To Path
Reference Path
Locations []Location
}
// Origin describes where an effective definition came from. Multiple locations // Origin describes where an effective definition came from. Multiple locations
// represent a composite origin; resolution origins are synthetic and identify their // represent a composite origin; resolution origins are synthetic and identify their
// operation instead of a file location. // operation instead of a file location. Inheritance is projected from private
// contributor traces at inspection boundaries.
type Origin struct { type Origin struct {
Layer Layer Layer Layer
Locations []Location Locations []Location
Operation ChangeKind Operation ChangeKind
Inheritance []InheritanceStep
contributors []inheritanceContributor
} }
// FieldInfo describes an effective configuration path. // FieldInfo describes an effective configuration path.
@@ -238,9 +276,32 @@ func cloneSourceInfos(sources []SourceInfo) []SourceInfo {
func cloneOrigin(origin Origin) Origin { func cloneOrigin(origin Origin) Origin {
origin.Locations = cloneLocations(origin.Locations) origin.Locations = cloneLocations(origin.Locations)
origin.Inheritance = cloneInheritanceSteps(origin.Inheritance)
if origin.contributors != nil {
contributors := make([]inheritanceContributor, len(origin.contributors))
for index, contributor := range origin.contributors {
contributors[index] = contributor
contributors[index].source = contributor.source.clone()
}
origin.contributors = contributors
}
return origin return origin
} }
func cloneInheritanceSteps(steps []InheritanceStep) []InheritanceStep {
if steps == nil {
return nil
}
cloned := make([]InheritanceStep, len(steps))
for index, step := range steps {
cloned[index] = InheritanceStep{
From: step.From.clone(), To: step.To.clone(), Reference: step.Reference.clone(),
Locations: cloneLocations(step.Locations),
}
}
return cloned
}
func cloneFieldInfo(info FieldInfo) FieldInfo { func cloneFieldInfo(info FieldInfo) FieldInfo {
info.Origin = cloneOrigin(info.Origin) info.Origin = cloneOrigin(info.Origin)
return info return info

View File

@@ -0,0 +1,45 @@
package engine
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document"
// recursiveMergeHooks lets an assembly owner account for the values changed by
// a recursive mapping overlay without making source roles part of the merge.
// Hooks run before replacement and receive the still-owned incoming subtree.
type recursiveMergeHooks struct {
added func(Path, *document.Node)
combined func(Path, *document.Node)
replaced func(Path, *document.Node, *document.Node)
}
// mergeRecursiveMappings changes an owned mapping and never aliases incoming
// values into it. Entry order follows the target, then new incoming entries.
func mergeRecursiveMappings(target, incoming *document.Node, current Path, index *mappingIndex, hooks recursiveMergeHooks) {
for _, entry := range incoming.Mapping {
path := current.Key(entry.Key)
position := index.find(target, entry.Key)
if position < 0 {
cloned := entry
cloned.Value = document.Clone(entry.Value)
index.append(target, cloned)
if hooks.added != nil {
hooks.added(path, entry.Value)
}
continue
}
existing := target.Mapping[position].Value
if existing.Kind == document.Mapping && entry.Value.Kind == document.Mapping {
if hooks.combined != nil {
hooks.combined(path, entry.Value)
}
mergeRecursiveMappings(existing, entry.Value, path, index, hooks)
continue
}
if hooks.replaced != nil {
hooks.replaced(path, existing, entry.Value)
}
target.Mapping[position] = entry
target.Mapping[position].Value = document.Clone(entry.Value)
}
}

View File

@@ -0,0 +1,90 @@
package engine
import (
"reflect"
"testing"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func TestRecursiveMappingMergePreservesValuesPathsAndInputs(t *testing.T) {
type expectedNode struct {
path Path
kind document.Kind
text string
entries int
}
tests := []struct {
name, base, incoming string
want []expectedNode
events []string
}{
{
name: "nested mappings", base: "shared: {kept: base, changed: old}\n", incoming: "shared: {changed: new, added: child}\n",
want: []expectedNode{
{path: Keys("shared"), kind: document.Mapping, entries: 3},
{path: Keys("shared", "kept"), kind: document.String, text: "base"},
{path: Keys("shared", "changed"), kind: document.String, text: "new"},
{path: Keys("shared", "added"), kind: document.String, text: "child"},
},
events: []string{"combine:/shared", "replace:/shared/changed", "add:/shared/added"},
},
{
name: "sequence replacement", base: "value: [old]\n", incoming: "value: []\n",
want: []expectedNode{{path: Keys("value"), kind: document.Sequence}},
events: []string{"replace:/value"},
},
{
name: "scalar replacement", base: "value: old\n", incoming: "value: new\n",
want: []expectedNode{{path: Keys("value"), kind: document.String, text: "new"}},
events: []string{"replace:/value"},
},
{
name: "null replacement", base: "value: old\n", incoming: "value: null\n",
want: []expectedNode{{path: Keys("value"), kind: document.Null}},
events: []string{"replace:/value"},
},
{
name: "empty mapping retains children", base: "shared: {kept: base}\n", incoming: "shared: {}\n",
want: []expectedNode{
{path: Keys("shared"), kind: document.Mapping, entries: 1},
{path: Keys("shared", "kept"), kind: document.String, text: "base"},
},
events: []string{"combine:/shared"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
base := parseCompositionDocument(t, "source-1", test.base)
incoming := parseCompositionDocument(t, "source-2", test.incoming)
baseBefore, incomingBefore := document.Clone(base), document.Clone(incoming)
owned := document.Clone(base)
var events []string
mergeRecursiveMappings(owned, incoming, Path{}, new(mappingIndex), recursiveMergeHooks{
added: func(path Path, _ *document.Node) { events = append(events, "add:"+path.String()) },
combined: func(path Path, _ *document.Node) {
events = append(events, "combine:"+path.String())
},
replaced: func(path Path, _, _ *document.Node) {
events = append(events, "replace:"+path.String())
},
})
if !reflect.DeepEqual(events, test.events) {
t.Fatalf("merge events = %#v, want %#v", events, test.events)
}
for _, expected := range test.want {
node, found := documentAtPath(owned, expected.path)
if !found || node.Kind != expected.kind || node.Text != expected.text || len(node.Mapping) != expected.entries {
t.Errorf("merged %s = %#v, found %v", expected.path.String(), node, found)
}
}
if !reflect.DeepEqual(base, baseBefore) || !reflect.DeepEqual(incoming, incomingBefore) {
t.Fatal("merge changed an input")
}
owned.Mapping[0].Value.Location.SourceID = "changed"
if !reflect.DeepEqual(base, baseBefore) || !reflect.DeepEqual(incoming, incomingBefore) {
t.Fatal("merged result aliases an input")
}
})
}
}

View File

@@ -41,7 +41,7 @@ type resolutionResult struct {
diagnostics []Diagnostic diagnostics []Diagnostic
} }
// IsExplicit reports whether surviving input explicitly defined path. // IsExplicit reports whether surviving direct or inherited input defined path.
func (r *Resolution) IsExplicit(path Path) bool { func (r *Resolution) IsExplicit(path Path) bool {
state := r.activeState() state := r.activeState()
if !state.validPath(path) { if !state.validPath(path) {
@@ -72,6 +72,12 @@ func (r *Resolution) Source(id string) (SourceInfo, bool) {
return r.activeState().input.source(id) return r.activeState().input.source(id)
} }
// SourceContext returns a passive context during the active resolution
// callback. Retaining the callback context after return is unsupported.
func (r *Resolution) SourceContext(id string) (ResourceContext, bool) {
return r.activeState().input.sourceContext(id)
}
// Record declares the intent for a resolver mutation. A container path covers // Record declares the intent for a resolver mutation. A container path covers
// mutations to its descendants. // mutations to its descendants.
func (r *Resolution) Record(path Path, kind ChangeKind, reason string, dependencies ...Path) { func (r *Resolution) Record(path Path, kind ChangeKind, reason string, dependencies ...Path) {

View File

@@ -0,0 +1,133 @@
package engine
import (
"errors"
"io/fs"
"os"
"path"
"path/filepath"
"strings"
)
// ResourceContext is a passive copy of one acquired source's rooted filesystem
// and slash-relative base directory. FS is borrowed and opens resources only
// when the application explicitly requests them.
type ResourceContext struct {
FS fs.FS
BaseDir string
}
func cloneResourceContexts(contexts map[string]ResourceContext) map[string]ResourceContext {
if contexts == nil {
return nil
}
cloned := make(map[string]ResourceContext, len(contexts))
for id, context := range contexts {
cloned[id] = context
}
return cloned
}
// rootedNativeFS borrows a namespace without invoking SubFS or opening it at
// query time. All resource names remain relative to its configured root.
type rootedNativeFS struct {
filesystem fs.FS
root string
}
func (r rootedNativeFS) name(name string) (string, error) {
if !fs.ValidPath(name) {
return "", &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid}
}
joined := path.Join(r.root, name)
if !fs.ValidPath(joined) || r.root != "." && joined != r.root && !strings.HasPrefix(joined, r.root+"/") {
return "", &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
}
return joined, nil
}
func (r rootedNativeFS) Open(name string) (fs.File, error) {
joined, err := r.name(name)
if err != nil {
return nil, err
}
return r.filesystem.Open(joined)
}
func (r rootedNativeFS) ReadDir(name string) ([]fs.DirEntry, error) {
joined, err := r.name(name)
if err != nil {
return nil, err
}
return fs.ReadDir(r.filesystem, joined)
}
// rootedHostFS stores only a canonical root path. No os.Root is retained after
// loading or created by a SourceContext query.
type hostResourceRoot interface {
Open(string) (*os.File, error)
Close() error
}
type rootedHostFS struct {
root string
openRoot func(string) (hostResourceRoot, error)
}
func (r rootedHostFS) Open(name string) (fs.File, error) {
localName, err := filepath.Localize(name)
if err != nil {
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid}
}
opener := r.openRoot
if opener == nil {
opener = func(name string) (hostResourceRoot, error) { return os.OpenRoot(name) }
}
root, err := opener(r.root)
if err != nil {
return nil, &fs.PathError{Op: "open", Path: name, Err: err}
}
canonical, err := canonicalExistingPath(filepath.Join(r.root, localName))
if err != nil {
_ = root.Close()
return nil, &fs.PathError{Op: "open", Path: name, Err: err}
}
if !containedPath(r.root, canonical) {
_ = root.Close()
return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrPermission}
}
relative, err := filepath.Rel(r.root, canonical)
if err != nil {
_ = root.Close()
return nil, &fs.PathError{Op: "open", Path: name, Err: err}
}
file, err := root.Open(relative)
if err != nil {
_ = root.Close()
return nil, &fs.PathError{Op: "open", Path: name, Err: err}
}
owned := rootedHostFile{File: file, root: root}
var opened fs.File = file
if directory, ok := opened.(fs.ReadDirFile); ok {
return &rootedHostDirFile{rootedHostFile: owned, directory: directory}, nil
}
return &owned, nil
}
type rootedHostFile struct {
fs.File
root hostResourceRoot
}
func (f *rootedHostFile) Close() error {
return errors.Join(f.File.Close(), f.root.Close())
}
type rootedHostDirFile struct {
rootedHostFile
directory fs.ReadDirFile
}
func (f *rootedHostDirFile) ReadDir(count int) ([]fs.DirEntry, error) {
return f.directory.ReadDir(count)
}

View File

@@ -0,0 +1,169 @@
package engine
import (
"errors"
"io/fs"
"os"
"path/filepath"
"runtime"
"testing"
"testing/fstest"
)
func TestResourceContextsPreserveLiteralBackslashes(t *testing.T) {
const directory = `assets\literal`
const name = directory + `/text\file`
check := func(t *testing.T, filesystem fs.FS) {
t.Helper()
data, err := fs.ReadFile(filesystem, name)
if err != nil || string(data) != "content" {
t.Fatalf("read = %q, %v", data, err)
}
entries, err := fs.ReadDir(filesystem, directory)
if err != nil || len(entries) != 1 || entries[0].Name() != `text\file` {
t.Fatalf("entries = %v, %v", entries, err)
}
if _, err := filesystem.Open("../outside"); !errors.Is(err, fs.ErrInvalid) {
t.Fatalf("escape = %v", err)
}
}
t.Run("native", func(t *testing.T) {
check(t, rootedNativeFS{filesystem: fstest.MapFS{"bundle/" + name: &fstest.MapFile{Data: []byte("content")}}, root: "bundle"})
})
t.Run("host", func(t *testing.T) {
if runtime.GOOS == "windows" {
t.Skip("backslash is a host separator on Windows; native case remains covered")
}
root := t.TempDir()
if err := os.Mkdir(filepath.Join(root, directory), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(root, filepath.FromSlash(name)), []byte("content"), 0o600); err != nil {
t.Fatal(err)
}
check(t, rootedHostFS{root: root})
})
}
type trackedHostRoot struct {
root *os.Root
closed *int
failOpen bool
}
func (r *trackedHostRoot) Open(name string) (*os.File, error) {
if r.failOpen {
return nil, fs.ErrPermission
}
return r.root.Open(name)
}
func (r *trackedHostRoot) Close() error {
(*r.closed)++
return r.root.Close()
}
func TestRootedHostFSClosesRootOnSuccessfulAndFailedOpen(t *testing.T) {
rootPath := t.TempDir()
if err := os.WriteFile(filepath.Join(rootPath, "resource.txt"), []byte("resource"), 0o600); err != nil {
t.Fatal(err)
}
opened, closed := 0, 0
failOpen := false
filesystem := rootedHostFS{root: rootPath, openRoot: func(name string) (hostResourceRoot, error) {
root, err := os.OpenRoot(name)
if err != nil {
return nil, err
}
opened++
return &trackedHostRoot{root: root, closed: &closed, failOpen: failOpen}, nil
}}
if _, err := filesystem.Open("../resource.txt"); !errors.Is(err, fs.ErrInvalid) || opened != 0 {
t.Fatalf("invalid path = %v; roots opened %d", err, opened)
}
if _, err := filesystem.Open("missing.txt"); !errors.Is(err, fs.ErrNotExist) || opened != 1 || closed != 1 {
t.Fatalf("missing resource = %v; opened %d, closed %d", err, opened, closed)
}
failOpen = true
if _, err := filesystem.Open("resource.txt"); !errors.Is(err, fs.ErrPermission) || opened != 2 || closed != 2 {
t.Fatalf("failed inner open = %v; opened %d, closed %d", err, opened, closed)
}
failOpen = false
file, err := filesystem.Open("resource.txt")
if err != nil || opened != 3 || closed != 2 {
t.Fatalf("resource open = %v; opened %d, closed %d", err, opened, closed)
}
if err := file.Close(); err != nil || closed != 3 {
t.Fatalf("resource close = %v; opened %d, closed %d", err, opened, closed)
}
if _, err := file.Read(make([]byte, 1)); err == nil {
t.Fatal("closed resource file remained readable")
}
directory, err := filesystem.Open(".")
if err != nil {
t.Fatal(err)
}
reader, ok := directory.(fs.ReadDirFile)
if !ok {
t.Fatal("rooted host directory did not expose ReadDirFile")
}
entries, err := reader.ReadDir(-1)
if err != nil || len(entries) != 1 || entries[0].Name() != "resource.txt" {
t.Fatalf("directory entries = %#v, %v", entries, err)
}
if err := directory.Close(); err != nil || opened != 4 || closed != 4 {
t.Fatalf("directory close = %v; opened %d, closed %d", err, opened, closed)
}
}
func TestNativeResourceViewIsBorrowedAndRooted(t *testing.T) {
filesystem := &observedFilesystem{files: fstest.MapFS{
"bundle/config/resource.txt": &fstest.MapFile{Data: []byte("content")},
}}
view := rootedNativeFS{filesystem: filesystem, root: "bundle"}
if filesystem.opened != 0 {
t.Fatal("constructing a native resource view opened files")
}
data, err := fs.ReadFile(view, "config/resource.txt")
if err != nil || string(data) != "content" {
t.Fatalf("resource = %q, %v", data, err)
}
filesystem.assertClosed(t)
if _, err := fs.ReadFile(view, "../outside.txt"); !errors.Is(err, fs.ErrInvalid) {
t.Fatalf("resource escape = %v", err)
}
}
func TestLoadingClosesNativeFilesAcrossPipelineOutcomes(t *testing.T) {
type config struct {
Value string `yaml:"value"`
Version string `yaml:"version"`
}
for _, test := range []struct {
name string
main string
child string
failRead string
options Options[config]
wantErr bool
}{
{name: "success", main: "value: retained\n"},
{name: "acquisition", main: "value: retained\n", failRead: "main.yml", wantErr: true},
{name: "parse", main: "value: [broken\n", wantErr: true},
{name: "composition", main: "configkit: {imports: [{path: child.yml}]}\nvalue: first\n", child: "value: second\n", options: Options[config]{Imports: true}, wantErr: true},
{name: "version", main: "version: wrong\n", options: Options[config]{Definition: Definition[config]{Version: &VersionRule{Path: Keys("version"), Check: func(Version, *Validation) error { return fs.ErrInvalid }}}}, wantErr: true},
{name: "validation", main: "value: retained\n", options: Options[config]{Definition: Definition[config]{Validate: func(_ *config, validation *Validation) { validation.Error(Keys("value"), "rejected") }}}, wantErr: true},
} {
t.Run(test.name, func(t *testing.T) {
files := fstest.MapFS{"main.yml": &fstest.MapFile{Data: []byte(test.main)}}
if test.child != "" {
files["child.yml"] = &fstest.MapFile{Data: []byte(test.child)}
}
filesystem := &observedFilesystem{files: files, failRead: test.failRead}
_, err := testLoad(filesystemFile(newNativeNamespace("bundle", filesystem), "main.yml"), test.options)
if (err != nil) != test.wantErr {
t.Fatalf("load error = %v, want failure %v", err, test.wantErr)
}
filesystem.assertClosed(t)
})
}
}

View File

@@ -34,6 +34,35 @@ func classifySecretPaths(schema *schemaDescriptor, predicate func(Path) bool, me
} }
} }
} }
var inherited []secretDependency
for _, operation := range metadata.inheritance {
if operation.control != nil {
control := operation.control.path
paths[control.String()] = control.clone()
for _, contributor := range operation.controlOrigin.contributors {
paths[contributor.source.String()] = contributor.source.clone()
inherited = append(inherited, secretDependency{target: control.clone(), dependency: contributor.source.clone()})
}
continue
}
if operation.transfer == nil {
continue
}
for pointer := range operation.parentOrigins {
source, err := ParsePath(pointer)
if err != nil {
continue
}
relative, ok := relativeHistoryPath(operation.transfer.from, source)
if !ok {
continue
}
destination := appendInheritancePath(operation.transfer.to, relative)
paths[pointer] = source
paths[destination.String()] = destination.clone()
inherited = append(inherited, secretDependency{target: destination, dependency: source})
}
}
secret := make(map[string]bool) secret := make(map[string]bool)
for _, tree := range append(projections, final) { for _, tree := range append(projections, final) {
@@ -62,6 +91,16 @@ func classifySecretPaths(schema *schemaDescriptor, predicate func(Path) bool, me
changed = true changed = true
} }
} }
for _, link := range inherited {
if pathIsSecret(secret, link.dependency) && !secret[link.target.String()] {
secret[link.target.String()] = true
changed = true
}
if pathIsSecret(secret, link.target) && !secret[link.dependency.String()] {
secret[link.dependency.String()] = true
changed = true
}
}
if !changed { if !changed {
return secret return secret
} }

View File

@@ -4,10 +4,7 @@ import (
"bytes" "bytes"
"errors" "errors"
"io/fs" "io/fs"
"os"
"path/filepath"
"strconv" "strconv"
"strings"
yamladapter "gitea.maximumdirect.net/eric/ConfigKit/internal/yaml" yamladapter "gitea.maximumdirect.net/eric/ConfigKit/internal/yaml"
) )
@@ -32,12 +29,47 @@ func validOverlayName(name string) bool {
} }
func loadSelectedOverlayDocument(source Source, context *FilesystemContext, main parsedSource, base *composedDocument, sources []SourceInfo, name string, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) { func loadSelectedOverlayDocument(source Source, context *FilesystemContext, main parsedSource, base *composedDocument, sources []SourceInfo, name string, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) {
assembly := newAssemblyAccumulator(cloneComposedDocument(base))
allSources, diagnostics := loadSelectedOverlayInto(source, context, main, assembly, sources, name, rule, limits)
if len(diagnostics) != 0 {
return nil, allSources, diagnostics
}
return assembly.current, allSources, nil
}
func loadSelectedOverlayInto(source Source, context *FilesystemContext, main parsedSource, base *assemblyAccumulator, sources []SourceInfo, name string, rule *VersionRule, limits importLimits) ([]SourceInfo, []Diagnostic) {
if nativeContext(source, context) {
access, basePath, mainPath, diagnostic := nativeFilesystemAccess(source, context)
if diagnostic != nil {
return cloneSourceInfos(sources), []Diagnostic{*diagnostic}
}
if diagnostic := validateNativeDirectories(access, basePath, mainPath); diagnostic != nil {
return cloneSourceInfos(sources), []Diagnostic{*diagnostic}
}
return loadSelectedOverlayWithAccess(access, base, sources, name, rule, limits)
}
rootPath, _, _, diagnostic := importFilesystem(source, context, main) rootPath, _, _, diagnostic := importFilesystem(source, context, main)
if diagnostic != nil { if diagnostic != nil {
return nil, cloneSourceInfos(sources), []Diagnostic{*diagnostic} return cloneSourceInfos(sources), []Diagnostic{*diagnostic}
} }
overlayPath := filepath.Join(rootPath, "overlays", name+".yml") access, err := openHostAccess(rootPath)
canonical, err := canonicalExistingPath(overlayPath) if err != nil {
value := globFilesystemDiagnostic(err, "configuration root could not be opened for the selected overlay", Location{})
value.Primary = nil
return cloneSourceInfos(sources), []Diagnostic{value}
}
return loadSelectedOverlayWithAccess(access, base, sources, name, rule, limits)
}
func loadSelectedOverlayWithAccess(access *filesystemAccess, base *assemblyAccumulator, sources []SourceInfo, name string, rule *VersionRule, limits importLimits) ([]SourceInfo, []Diagnostic) {
closed := false
defer func() {
if !closed {
_ = access.close()
}
}()
overlayPath := access.join(access.rootPath, "overlays/"+name+".yml")
canonical, err := access.canonical(overlayPath)
if err != nil { if err != nil {
code := CodeOverlayInvalid code := CodeOverlayInvalid
message := "selected overlay could not be resolved" message := "selected overlay could not be resolved"
@@ -46,83 +78,81 @@ func loadSelectedOverlayDocument(source Source, context *FilesystemContext, main
} else if errors.Is(err, fs.ErrPermission) { } else if errors.Is(err, fs.ErrPermission) {
code, message = CodeImportPermission, "selected overlay cannot be accessed" code, message = CodeImportPermission, "selected overlay cannot be accessed"
} }
return nil, cloneSourceInfos(sources), []Diagnostic{importDiagnostic(code, message, nil)} return cloneSourceInfos(sources), []Diagnostic{importDiagnostic(code, message, nil)}
} }
if !containedPath(rootPath, canonical) { if !access.contains(canonical) {
return nil, cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayOutsideRoot, "selected overlay resolves outside the configuration root", nil)} return cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayOutsideRoot, "selected overlay resolves outside the configuration root", nil)}
} }
for _, existing := range sources { for _, existing := range sources {
if existing.Path != "" && existing.Path == canonical { if existing.Path != "" && existing.Path == canonical &&
return nil, cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayRepeated, "selected overlay is already present in the base source graph", nil)} ((access.native == nil && existing.Kind == SourceKindFile) ||
(access.native != nil && existing.Kind == SourceKindFilesystem && existing.namespace == access.token)) {
return cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayRepeated, "selected overlay is already present in the base source graph", nil)}
} }
} }
if len(sources)+1 > limits.sources { if len(sources)+1 > limits.sources {
return nil, cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayLimitExceeded, "selected overlay exceeds the source-count limit", nil)} return cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayLimitExceeded, "selected overlay exceeds the source-count limit", nil)}
} }
root, err := os.OpenRoot(rootPath) if _, err := access.relative(canonical); err != nil {
if err != nil { return cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayOutsideRoot, "selected overlay could not be addressed inside the configuration root", nil)}
value := globFilesystemDiagnostic(err, "configuration root could not be opened for the selected overlay", Location{})
value.Primary = nil
return nil, cloneSourceInfos(sources), []Diagnostic{value}
}
rootClosed := false
defer func() {
if !rootClosed {
_ = root.Close()
}
}()
relative, err := filepath.Rel(rootPath, canonical)
if err != nil || filepath.IsAbs(relative) || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
return nil, cloneSourceInfos(sources), []Diagnostic{importDiagnostic(CodeOverlayOutsideRoot, "selected overlay could not be addressed inside the configuration root", nil)}
} }
sourceID := "source-" + strconv.Itoa(len(sources)+1) sourceID := "source-" + strconv.Itoa(len(sources)+1)
overlayLocation := &Location{SourceID: sourceID} overlayLocation := &Location{SourceID: sourceID}
overlaySource := SourceInfo{ID: sourceID, Label: filepath.ToSlash(filepath.Join("overlays", name+".yml")), Path: canonical, Layer: LayerOverlay} kind := SourceKindFile
label := "overlays/" + name + ".yml"
if access.native != nil {
kind = SourceKindFilesystem
label = access.label + "/" + canonical
}
overlaySource := SourceInfo{ID: sourceID, Label: label, Path: canonical, Layer: LayerOverlay, Kind: kind}
if kind == SourceKindFilesystem {
overlaySource.namespace = access.token
}
sources = append(cloneSourceInfos(sources), overlaySource) sources = append(cloneSourceInfos(sources), overlaySource)
data, diagnostic := readOverlayFile(root, relative, sourceID, limits.bytes-base.inputBytes) data, diagnostic := readOverlayFile(access, canonical, sourceID, limits.bytes-base.current.inputBytes)
if diagnostic != nil { if diagnostic != nil {
return nil, sources, []Diagnostic{*diagnostic} return sources, []Diagnostic{*diagnostic}
} }
overlayRoot, issues := yamladapter.Parse(bytes.NewReader(data), sourceID, yamladapter.Limits{Depth: maxDocumentDepth, Nodes: maxDocumentNodes}) overlayRoot, issues := yamladapter.Parse(bytes.NewReader(data), sourceID, yamladapter.Limits{Depth: maxDocumentDepth, Nodes: maxDocumentNodes})
if len(issues) != 0 { if len(issues) != 0 {
return nil, sources, parserDiagnostics(sourceID, issues) return sources, parserDiagnostics(sourceID, issues)
} }
nodeCount := countInputNodes(overlayRoot) nodeCount := countInputNodes(overlayRoot)
if base.inputNodes+nodeCount > limits.nodes { if base.current.inputNodes+nodeCount > limits.nodes {
return nil, sources, []Diagnostic{importDiagnostic(CodeOverlayLimitExceeded, "selected overlay exceeds the aggregate node limit", overlayLocation)} return sources, []Diagnostic{importDiagnostic(CodeOverlayLimitExceeded, "selected overlay exceeds the aggregate node limit", overlayLocation)}
} }
if diagnostics := overlayDirectiveDiagnostics(overlayRoot, Path{}); len(diagnostics) != 0 { if diagnostics := overlayDirectiveDiagnostics(overlayRoot, Path{}); len(diagnostics) != 0 {
return nil, sources, diagnostics return sources, diagnostics
} }
if versionNode := fragmentVersionClaim(overlayRoot, rule); versionNode != nil { if versionNode := fragmentVersionClaim(overlayRoot, rule); versionNode != nil {
location := publicLocation(versionNode.Location) location := publicLocation(versionNode.Location)
return nil, sources, []Diagnostic{{ return sources, []Diagnostic{{
Code: CodeOverlayVersion, Severity: SeverityError, Path: rule.Path.clone(), Code: CodeOverlayVersion, Severity: SeverityError, Path: rule.Path.clone(),
Message: "overlays cannot contribute to or replace the schema-version path", Primary: &location, Message: "overlays cannot contribute to or replace the schema-version path", Primary: &location,
}} }}
} }
closeErr := root.Close() closeErr := access.close()
rootClosed = true closed = true
if closeErr != nil { if closeErr != nil {
return nil, sources, []Diagnostic{importDiagnostic(CodeImportRead, "configuration root could not be closed after reading the selected overlay", nil)} return sources, []Diagnostic{importDiagnostic(CodeImportRead, "configuration root could not be closed after reading the selected overlay", nil)}
} }
overlayd, diagnostics := overlayDocuments(base, overlayRoot) if diagnostics := base.applyOverlay(overlayRoot, nil, nil, LayerOverlay); len(diagnostics) != 0 {
if overlayd != nil { return sources, diagnostics
overlayd.inputBytes = base.inputBytes + int64(len(data))
overlayd.inputNodes = base.inputNodes + nodeCount
} }
return overlayd, sources, diagnostics base.current.inputBytes += int64(len(data))
base.current.inputNodes += nodeCount
return sources, nil
} }
func readOverlayFile(root *os.Root, relative, sourceID string, remaining int64) ([]byte, *Diagnostic) { func readOverlayFile(access *filesystemAccess, name, sourceID string, remaining int64) ([]byte, *Diagnostic) {
limit := min(int64(maxSourceBytes), remaining) limit := min(int64(maxSourceBytes), remaining)
if limit < 0 { if limit < 0 {
limit = 0 limit = 0
} }
data, failure := readRegularFile(relative, root.Stat, root.Open, limit) data, failure := readRegularFile(name, access.stat, access.open, limit)
if failure != nil { if failure != nil {
value := sourceReadDiagnostic(failure, sourceID, CodeOverlayLimitExceeded, "selected overlay exceeds a source or aggregate byte limit") value := sourceReadDiagnostic(failure, sourceID, CodeOverlayLimitExceeded, "selected overlay exceeds a source or aggregate byte limit")
return nil, &value return nil, &value

View File

@@ -9,6 +9,8 @@ import (
"path/filepath" "path/filepath"
"reflect" "reflect"
"sync/atomic" "sync/atomic"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
) )
const ( const (
@@ -16,14 +18,19 @@ const (
maxTotalBytes = 64 << 20 maxTotalBytes = 64 << 20
) )
// Source is an opaque configuration input. Construct sources with File, Bytes, // Source is an opaque configuration input. Construct sources with File,
// Reader, or NoSource; the zero value is invalid. // Filesystem.File, Bytes, Reader, or NoSource; the zero value is invalid.
type Source struct { type Source struct {
kind sourceKind kind sourceKind
path string path string
label string label string
namespace *nativeNamespace
data []byte data []byte
reader *readerSource reader *readerSource
mapping *document.Node
mapIssue *valueIssue
mapBytes int64
mapNodes int
createErr sourceCreationError createErr sourceCreationError
} }
@@ -32,8 +39,10 @@ type sourceKind uint8
const ( const (
sourceInvalid sourceKind = iota sourceInvalid sourceKind = iota
sourceFile sourceFile
sourceFilesystem
sourceBytes sourceBytes
sourceReader sourceReader
sourceMapping
sourceNone sourceNone
) )
@@ -45,8 +54,53 @@ const (
creationInvalidLabel creationInvalidLabel
creationInvalidReader creationInvalidReader
creationTooLarge creationTooLarge
creationInvalidFilesystem
creationInvalidFilesystemLabel
creationInvalidFilesystemPath
creationInvalidMappingRoot
) )
// Filesystem is a reusable opaque namespace for caller-owned fs.FS inputs.
// Copies preserve the namespace identity used for explicit contexts.
type Filesystem struct {
namespace *nativeNamespace
createErr sourceCreationError
}
// NewFilesystem records a borrowed filesystem without opening it. Invalid
// arguments are reported when a File source is consumed.
func NewFilesystem(label string, filesystem fs.FS) Filesystem {
result := Filesystem{}
if !validOverrideLabel(label) {
result.createErr = creationInvalidFilesystemLabel
} else if nilFilesystem(filesystem) {
result.createErr = creationInvalidFilesystem
} else {
result.namespace = newNativeNamespace(label, filesystem)
}
return result
}
// File selects a non-root slash-relative file within this namespace.
func (f Filesystem) File(name string) Source {
if f.createErr != creationValid {
return Source{kind: sourceFilesystem, createErr: f.createErr}
}
return filesystemFile(f.namespace, name)
}
func nilFilesystem(filesystem fs.FS) bool {
if filesystem == nil {
return true
}
value := reflect.ValueOf(filesystem)
switch value.Kind() {
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice:
return value.IsNil()
}
return false
}
type readerSource struct { type readerSource struct {
input io.Reader input io.Reader
used atomic.Bool used atomic.Bool
@@ -60,6 +114,16 @@ func File(path string) Source {
return Source{kind: sourceFile, path: path, label: path} return Source{kind: sourceFile, path: path, label: path}
} }
func filesystemFile(namespace *nativeNamespace, name string) Source {
if namespace == nil || nilFilesystem(namespace.filesystem) {
return Source{kind: sourceFilesystem, createErr: creationInvalidFilesystem}
}
if name == "." || !fs.ValidPath(name) {
return Source{kind: sourceFilesystem, createErr: creationInvalidFilesystemPath}
}
return Source{kind: sourceFilesystem, path: name, label: namespace.label + "/" + name, namespace: namespace}
}
// Bytes snapshots data immediately. Label must be nonempty and safe to display; // Bytes snapshots data immediately. Label must be nonempty and safe to display;
// constructor problems are reported when the source is consumed. // constructor problems are reported when the source is consumed.
func Bytes(label string, data []byte) Source { func Bytes(label string, data []byte) Source {
@@ -92,6 +156,24 @@ func Reader(label string, input io.Reader) Source {
return source return source
} }
// Mapping snapshots supported built-in values into an owned document at
// construction. Invalid values are retained as safe deferred diagnostics.
func Mapping(label string, values map[string]any) Source {
source := Source{kind: sourceMapping, label: label}
if !validOverrideLabel(label) {
source.createErr = creationInvalidLabel
return source
}
if values == nil {
source.createErr = creationInvalidMappingRoot
return source
}
builder := valueBuilder{active: make(map[valueVisit]bool)}
source.mapping, source.mapIssue = builder.build(values, Path{}, 0)
source.mapBytes, source.mapNodes = builder.bytes, builder.nodes
return source
}
// NoSource explicitly selects a defaults-only input. It creates no source record. // NoSource explicitly selects a defaults-only input. It creates no source record.
func NoSource() Source { func NoSource() Source {
return Source{kind: sourceNone} return Source{kind: sourceNone}
@@ -108,13 +190,23 @@ func defaultSourceLimits() sourceLimits {
type sourceRead struct { type sourceRead struct {
data []byte data []byte
document *document.Node
bytes int64
info *SourceInfo info *SourceInfo
} }
func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead, []Diagnostic) { func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead, []Diagnostic) {
if diagnostic := sourceConstructorDiagnostic(source); diagnostic != nil { if diagnostic := sourceConstructorDiagnostic(source); diagnostic != nil {
if source.kind == sourceMapping {
diagnostic.Primary = &Location{SourceID: sourceID}
return sourceRead{info: mappingSourceInfo(source, sourceID)}, []Diagnostic{*diagnostic}
}
return sourceRead{}, []Diagnostic{*diagnostic} return sourceRead{}, []Diagnostic{*diagnostic}
} }
if source.kind == sourceMapping && source.mapIssue != nil {
diagnostic := source.mapIssue.sourceDiagnostic(sourceID)
return sourceRead{info: mappingSourceInfo(source, sourceID)}, []Diagnostic{diagnostic}
}
if source.kind == sourceNone { if source.kind == sourceNone {
return sourceRead{}, nil return sourceRead{}, nil
} }
@@ -123,7 +215,7 @@ func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead
if limit < 0 { if limit < 0 {
limit = 0 limit = 0
} }
info := &SourceInfo{ID: sourceID, Label: source.label, Layer: LayerBase} info := &SourceInfo{ID: sourceID, Label: source.label, Layer: LayerBase, Kind: sourceSourceKind(source.kind)}
var input io.Reader var input io.Reader
switch source.kind { switch source.kind {
@@ -143,6 +235,20 @@ func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead
return sourceRead{info: info}, []Diagnostic{sourceReadDiagnostic(failure, sourceID, CodeSourceLimitExceeded, "source exceeds the configured byte limit")} return sourceRead{info: info}, []Diagnostic{sourceReadDiagnostic(failure, sourceID, CodeSourceLimitExceeded, "source exceeds the configured byte limit")}
} }
return sourceRead{data: bytes.Clone(data), info: info}, nil return sourceRead{data: bytes.Clone(data), info: info}, nil
case sourceFilesystem:
access := newNativeAccess(source.namespace.filesystem, ".", source.namespace.token)
info.Path = source.path
info.namespace = source.namespace.token
canonical, err := access.canonical(source.path)
if err != nil {
return sourceRead{info: info}, []Diagnostic{classifySourceError(err, sourceID)}
}
info.Path = canonical
data, failure := readRegularFile(canonical, access.stat, access.open, limit)
if failure != nil {
return sourceRead{info: info}, []Diagnostic{sourceReadDiagnostic(failure, sourceID, CodeSourceLimitExceeded, "source exceeds the configured byte limit")}
}
return sourceRead{data: bytes.Clone(data), info: info}, nil
case sourceBytes: case sourceBytes:
input = bytes.NewReader(source.data) input = bytes.NewReader(source.data)
case sourceReader: case sourceReader:
@@ -150,6 +256,13 @@ func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceReaderUsed, sourceID, "reader source has already been consumed")} return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceReaderUsed, sourceID, "reader source has already been consumed")}
} }
input = source.reader.input input = source.reader.input
case sourceMapping:
if source.mapBytes > limit || source.mapNodes > maxDocumentNodes {
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceLimitExceeded, sourceID, "mapping source exceeds the configured input limit")}
}
owned := document.Clone(source.mapping)
setDocumentSourceID(owned, sourceID)
return sourceRead{document: owned, bytes: source.mapBytes, info: info}, nil
default: default:
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceInvalid, sourceID, "source is invalid")} return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceInvalid, sourceID, "source is invalid")}
} }
@@ -160,6 +273,14 @@ func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead
return sourceRead{data: bytes.Clone(data), info: info}, nil return sourceRead{data: bytes.Clone(data), info: info}, nil
} }
func mappingSourceInfo(source Source, sourceID string) *SourceInfo {
label := source.label
if !validOverrideLabel(label) {
label = "mapping"
}
return &SourceInfo{ID: sourceID, Label: label, Layer: LayerBase, Kind: SourceKindMapping}
}
func sourceConstructorDiagnostic(source Source) *Diagnostic { func sourceConstructorDiagnostic(source Source) *Diagnostic {
var message string var message string
code := CodeSourceInvalid code := CodeSourceInvalid
@@ -173,16 +294,60 @@ func sourceConstructorDiagnostic(source Source) *Diagnostic {
message = "source path must not be empty" message = "source path must not be empty"
case creationInvalidLabel: case creationInvalidLabel:
message = "source label must not be empty" message = "source label must not be empty"
if source.kind == sourceMapping {
message = "mapping source label must be nonempty and safe to display"
}
case creationInvalidReader: case creationInvalidReader:
message = "source reader must not be nil" message = "source reader must not be nil"
case creationTooLarge: case creationTooLarge:
code = CodeSourceLimitExceeded code = CodeSourceLimitExceeded
message = "source exceeds the configured byte limit" message = "source exceeds the configured byte limit"
case creationInvalidFilesystem:
message = "filesystem must not be nil"
case creationInvalidFilesystemLabel:
message = "filesystem label must be nonempty and safe to display"
case creationInvalidFilesystemPath:
message = "filesystem source path must be a non-root slash-relative valid path"
case creationInvalidMappingRoot:
message = "mapping source root must be a non-nil mapping"
} }
diagnostic := sourceDiagnostic(code, "", message) diagnostic := sourceDiagnostic(code, "", message)
return &diagnostic return &diagnostic
} }
func sourceSourceKind(kind sourceKind) SourceKind {
switch kind {
case sourceFile:
return SourceKindFile
case sourceFilesystem:
return SourceKindFilesystem
case sourceBytes:
return SourceKindBytes
case sourceReader:
return SourceKindReader
case sourceMapping:
return SourceKindMapping
default:
return ""
}
}
func setDocumentSourceID(node *document.Node, sourceID string) {
if node == nil {
return
}
node.Location.SourceID = sourceID
for index := range node.Mapping {
entry := &node.Mapping[index]
entry.KeyLocation.SourceID = sourceID
entry.ValueLocation.SourceID = sourceID
setDocumentSourceID(entry.Value, sourceID)
}
for _, child := range node.Sequence {
setDocumentSourceID(child, sourceID)
}
}
func classifySourceError(err error, sourceID string) Diagnostic { func classifySourceError(err error, sourceID string) Diagnostic {
switch { switch {
case errors.Is(err, fs.ErrNotExist): case errors.Is(err, fs.ErrNotExist):

View File

@@ -0,0 +1,112 @@
package engine
import (
"io/fs"
"path"
"path/filepath"
"strings"
)
func contextsForAcquiredSources(source Source, context *FilesystemContext, baseSource *SourceInfo, sources []SourceInfo) (map[string]ResourceContext, *Diagnostic) {
if len(sources) == 0 {
return nil, nil
}
var rootPath, basePath string
var filesystem fs.FS
if source.kind == sourceFilesystem {
filesystem = source.namespace.filesystem
rootPath, basePath = ".", path.Dir(source.path)
if context != nil && context.Root != "" {
rootPath = context.Root
}
} else if context != nil && context.FS != nil {
filesystem = context.FS.namespace.filesystem
rootPath, basePath = context.Root, context.BaseDir
} else if source.kind == sourceFile {
filePath := sources[0].Path
if baseSource != nil {
filePath = baseSource.Path
}
basePath = filepath.Dir(filePath)
rootPath = basePath
if context != nil && context.Root != "" {
canonical, err := canonicalExistingPath(context.Root)
if err != nil {
value := importDiagnostic(CodeImportInvalid, "configuration root could not be canonicalized", nil)
return nil, &value
}
rootPath = canonical
}
} else if context != nil {
var err error
rootPath, err = canonicalExistingPath(context.Root)
if err != nil {
value := importDiagnostic(CodeImportInvalid, "configuration root could not be canonicalized", nil)
return nil, &value
}
basePath, err = canonicalExistingPath(context.BaseDir)
if err != nil {
value := importDiagnostic(CodeImportInvalid, "import base directory could not be canonicalized", nil)
return nil, &value
}
} else {
return nil, nil
}
var rooted fs.FS
if filesystem != nil {
rooted = rootedNativeFS{filesystem: filesystem, root: rootPath}
} else {
rooted = rootedHostFS{root: rootPath}
}
result := make(map[string]ResourceContext, len(sources))
for _, acquired := range sources {
base := basePath
if acquired.Path != "" {
if filesystem != nil {
base = path.Dir(acquired.Path)
} else {
base = filepath.Dir(acquired.Path)
}
}
var relative string
if filesystem != nil {
if !fs.ValidPath(rootPath) || !fs.ValidPath(base) || rootPath != "." && base != rootPath && !strings.HasPrefix(base, rootPath+"/") {
value := importDiagnostic(CodeImportOutsideRoot, "source base is outside the configuration root", nil)
return nil, &value
}
relative = base
if rootPath != "." {
relative = strings.TrimPrefix(strings.TrimPrefix(base, rootPath), "/")
if relative == "" {
relative = "."
}
}
} else {
if !containedPath(rootPath, base) {
value := importDiagnostic(CodeImportOutsideRoot, "source base is outside the configuration root", nil)
return nil, &value
}
var err error
relative, err = filepath.Rel(rootPath, base)
if err != nil {
value := importDiagnostic(CodeImportInvalid, "source base could not be addressed in the configuration root", nil)
return nil, &value
}
relative = filepath.ToSlash(relative)
}
result[acquired.ID] = ResourceContext{FS: rooted, BaseDir: relative}
}
return result, nil
}
func mergeResourceContexts(target *map[string]ResourceContext, incoming map[string]ResourceContext) {
if len(incoming) == 0 {
return
}
if *target == nil {
*target = make(map[string]ResourceContext, len(incoming))
}
for id, context := range incoming {
(*target)[id] = context
}
}

View File

@@ -4,7 +4,6 @@ import (
"errors" "errors"
"io" "io"
"io/fs" "io/fs"
"os"
) )
type sourceReadFailureKind uint8 type sourceReadFailureKind uint8
@@ -26,7 +25,7 @@ type sourceReadFailure struct {
// readRegularFile borrows stat/open operations so root-bound callers retain their // readRegularFile borrows stat/open operations so root-bound callers retain their
// containment policy. Preflight rejects FIFOs before open can block. The opened // containment policy. Preflight rejects FIFOs before open can block. The opened
// handle is checked again and always closed; read-only close errors are ignored. // handle is checked again and always closed; read-only close errors are ignored.
func readRegularFile(name string, stat func(string) (fs.FileInfo, error), open func(string) (*os.File, error), limit int64) ([]byte, *sourceReadFailure) { func readRegularFile[F fs.File](name string, stat func(string) (fs.FileInfo, error), open func(string) (F, error), limit int64) ([]byte, *sourceReadFailure) {
info, err := stat(name) info, err := stat(name)
if err != nil { if err != nil {
return nil, &sourceReadFailure{cause: err, missingTarget: errors.Is(err, fs.ErrNotExist)} return nil, &sourceReadFailure{cause: err, missingTarget: errors.Is(err, fs.ErrNotExist)}

View File

@@ -0,0 +1,19 @@
package engine
func (issue *valueIssue) sourceDiagnostic(sourceID string) Diagnostic {
code := CodeSourceInvalid
message := "mapping source has an unsupported value"
switch issue.kind {
case valueDepthLimit:
code, message = CodeSourceLimitExceeded, "mapping source exceeds the depth limit"
case valueNodeLimit:
code, message = CodeSourceLimitExceeded, "mapping source exceeds the node limit"
case valueByteLimit:
code, message = CodeSourceLimitExceeded, "mapping source exceeds the string-byte limit"
case valueNonFinite:
message = "mapping source float must be finite"
case valueCycle:
message = "mapping source values must not contain cycles"
}
return Diagnostic{Code: code, Severity: SeverityError, Path: issue.path.clone(), Message: message, Primary: &Location{SourceID: sourceID}}
}

View File

@@ -0,0 +1,156 @@
package engine
import (
"io"
"strconv"
"testing"
"testing/fstest"
)
type benchmarkCatalogConfig struct {
Catalog map[string]string `yaml:"catalog"`
}
type benchmarkNodeConfig struct {
Node string `yaml:"node"`
}
func BenchmarkUnderlayBroadCatalog(b *testing.B) {
for _, width := range []int{128, 256, 512} {
b.Run(strconv.Itoa(width), func(b *testing.B) {
inputs := make([]Input, 2)
for half := range inputs {
catalog := make(map[string]any, width/2)
for index := range width / 2 {
catalog["item-"+strconv.Itoa(half*width/2+index)] = "value"
}
inputs[half] = Input{Source: Mapping("catalog-"+strconv.Itoa(half), map[string]any{"catalog": catalog})}
}
options := LayeredOptions[benchmarkCatalogConfig]{Underlays: []Underlay{{Inputs: inputs}}}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, err := LoadLayered(Input{Source: NoSource()}, testSecretBinding(), options); err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkUnderlayManySmallTiers(b *testing.B) {
for _, count := range []int{32, 64, 128} {
b.Run(strconv.Itoa(count), func(b *testing.B) {
groups := make([]Underlay, count)
for index := range groups {
groups[index].Inputs = []Input{{Source: Mapping("item", map[string]any{
"catalog": map[string]any{"item-" + strconv.Itoa(index): "value"},
})}}
}
options := LayeredOptions[benchmarkCatalogConfig]{Underlays: groups}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, err := LoadLayered(Input{Source: NoSource()}, testSecretBinding(), options); err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkUnderlaySubtreeReplacement(b *testing.B) {
for _, count := range []int{32, 64, 128} {
b.Run(strconv.Itoa(count), func(b *testing.B) {
groups := make([]Underlay, count)
for index := range groups {
var node any = "temporary"
if index%2 == 0 {
node = map[string]any{"left": "one", "right": "two", "nested": map[string]any{"enabled": false}}
}
groups[index].Inputs = []Input{{Source: Mapping("replacement", map[string]any{"node": node})}}
}
options := LayeredOptions[benchmarkNodeConfig]{Underlays: groups}
main := Input{Source: Mapping("main", map[string]any{"node": "final"})}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, err := LoadLayered(main, testSecretBinding(), options); err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkUnderlayNativeImports(b *testing.B) {
for _, count := range []int{16, 32, 64} {
b.Run(strconv.Itoa(count), func(b *testing.B) {
files := fstest.MapFS{
"main.yml": &fstest.MapFile{Data: []byte("configkit: {imports: [{glob: 'defaults/*.yml'}]}\n")},
}
for index := range count {
name := "item-" + strconv.Itoa(index)
files["defaults/"+name+".yml"] = &fstest.MapFile{Data: []byte("catalog: {" + name + ": value}\n")}
}
filesystem := NewFilesystem("bundle", files)
main := Input{Source: filesystem.File("main.yml"), Imports: true}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if _, err := LoadLayered[benchmarkCatalogConfig](main, testSecretBinding()); err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkUnderlayRetainedHistory(b *testing.B) {
for _, count := range []int{16, 32, 64} {
b.Run(strconv.Itoa(count), func(b *testing.B) {
groups := make([]Underlay, count)
for index := range groups {
groups[index].Inputs = []Input{{Source: Mapping("value", map[string]any{
"catalog": map[string]any{"shared": strconv.Itoa(index)},
})}}
}
options := LayeredOptions[benchmarkCatalogConfig]{Underlays: groups}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
result, err := LoadLayered(Input{Source: NoSource()}, testSecretBinding(), options)
if err != nil {
b.Fatal(err)
}
if _, ok := result.Inspection.Explain(Keys("catalog", "shared")); !ok {
b.Fatal("retained path was not explained")
}
}
})
}
}
func BenchmarkUnderlayRenderedHistory(b *testing.B) {
for _, count := range []int{32, 64, 128} {
b.Run(strconv.Itoa(count), func(b *testing.B) {
groups := make([]Underlay, count)
for index := range groups {
groups[index].Inputs = []Input{{Source: Mapping("value", map[string]any{
"catalog": map[string]any{"shared": strconv.Itoa(index)},
})}}
}
result, err := LoadLayered(Input{Source: NoSource()}, testSecretBinding(), LayeredOptions[benchmarkCatalogConfig]{Underlays: groups})
if err != nil {
b.Fatal(err)
}
b.ReportAllocs()
b.ResetTimer()
for range b.N {
if err := result.Inspection.WriteExplanation(io.Discard, Keys("catalog", "shared")); err != nil {
b.Fatal(err)
}
}
})
}
}

166
internal/engine/value.go Normal file
View File

@@ -0,0 +1,166 @@
package engine
import (
"math"
"reflect"
"slices"
"strconv"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
type valueBuilder struct {
sourceID string
active map[valueVisit]bool
bytes int64
nodes int
}
type valueIssueKind uint8
const (
valueDepthLimit valueIssueKind = iota
valueNodeLimit
valueByteLimit
valueNonFinite
valueUnsupported
valueCycle
)
type valueIssue struct {
kind valueIssueKind
path Path
}
func newValueIssue(kind valueIssueKind, path Path) *valueIssue {
return &valueIssue{kind: kind, path: path.clone()}
}
func (b *valueBuilder) build(value any, path Path, depth int) (*document.Node, *valueIssue) {
if depth > maxDocumentDepth {
return nil, newValueIssue(valueDepthLimit, path)
}
b.nodes++
if b.nodes > maxDocumentNodes {
return nil, newValueIssue(valueNodeLimit, path)
}
location := document.Location{SourceID: b.sourceID}
switch value := value.(type) {
case nil:
return &document.Node{Kind: document.Null, Location: location}, nil
case string:
if !b.addBytes(len(value)) {
return nil, newValueIssue(valueByteLimit, path)
}
return &document.Node{Kind: document.String, Text: value, Location: location}, nil
case bool:
return &document.Node{Kind: document.Boolean, Text: strconv.FormatBool(value), Location: location}, nil
case int:
return valueNumber(strconv.FormatInt(int64(value), 10), location), nil
case int8:
return valueNumber(strconv.FormatInt(int64(value), 10), location), nil
case int16:
return valueNumber(strconv.FormatInt(int64(value), 10), location), nil
case int32:
return valueNumber(strconv.FormatInt(int64(value), 10), location), nil
case int64:
return valueNumber(strconv.FormatInt(value, 10), location), nil
case uint:
return valueNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint8:
return valueNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint16:
return valueNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint32:
return valueNumber(strconv.FormatUint(uint64(value), 10), location), nil
case uint64:
return valueNumber(strconv.FormatUint(value, 10), location), nil
case float32:
if math.IsNaN(float64(value)) || math.IsInf(float64(value), 0) {
return nil, newValueIssue(valueNonFinite, path)
}
return valueNumber(strconv.FormatFloat(float64(value), 'g', -1, 32), location), nil
case float64:
if math.IsNaN(value) || math.IsInf(value, 0) {
return nil, newValueIssue(valueNonFinite, path)
}
return valueNumber(strconv.FormatFloat(value, 'g', -1, 64), location), nil
case []any:
if value == nil {
return &document.Node{Kind: document.Null, Location: location}, nil
}
leave, diagnostic := b.enter(reflect.ValueOf(value), path)
if diagnostic != nil {
return nil, diagnostic
}
defer leave()
if len(value) > maxDocumentNodes-b.nodes {
return nil, newValueIssue(valueNodeLimit, path)
}
node := &document.Node{Kind: document.Sequence, Location: location, Sequence: make([]*document.Node, len(value))}
for index, child := range value {
built, diagnostic := b.build(child, path.Index(index), depth+1)
if diagnostic != nil {
return nil, diagnostic
}
node.Sequence[index] = built
}
return node, nil
case map[string]any:
if value == nil {
return &document.Node{Kind: document.Null, Location: location}, nil
}
leave, diagnostic := b.enter(reflect.ValueOf(value), path)
if diagnostic != nil {
return nil, diagnostic
}
defer leave()
if len(value) > (maxDocumentNodes-b.nodes)/2 {
return nil, newValueIssue(valueNodeLimit, path)
}
keys := make([]string, 0, len(value))
for key := range value {
keys = append(keys, key)
}
slices.Sort(keys)
node := &document.Node{Kind: document.Mapping, Location: location, Mapping: make([]document.Entry, 0, len(keys))}
for _, key := range keys {
b.nodes++ // mapping key
if !b.addBytes(len(key)) {
return nil, newValueIssue(valueByteLimit, path.Key(key))
}
built, diagnostic := b.build(value[key], path.Key(key), depth+1)
if diagnostic != nil {
return nil, diagnostic
}
node.Mapping = append(node.Mapping, document.Entry{Key: key, KeyLocation: location, ValueLocation: location, Value: built})
}
return node, nil
default:
return nil, newValueIssue(valueUnsupported, path)
}
}
func (b *valueBuilder) addBytes(count int) bool {
if count < 0 || int64(count) > maxSourceBytes-b.bytes {
return false
}
b.bytes += int64(count)
return true
}
func (b *valueBuilder) enter(value reflect.Value, path Path) (func(), *valueIssue) {
visit, ok := visitFor(value)
if !ok {
return func() {}, nil
}
if b.active[visit] {
return nil, newValueIssue(valueCycle, path)
}
b.active[visit] = true
return func() { delete(b.active, visit) }, nil
}
func valueNumber(text string, location document.Location) *document.Node {
return &document.Node{Kind: document.Number, Text: text, Location: location}
}

78
layered.go Normal file
View File

@@ -0,0 +1,78 @@
package configkit
import "gitea.maximumdirect.net/eric/ConfigKit/internal/engine"
// Input selects one source, whether its imports are enabled, and any explicit
// filesystem context. A zero Source is invalid; use NoSource() for an omitted
// main input. Sources keep their documented reuse and reader ownership rules.
type Input struct {
Source Source
Imports bool
Filesystem *FilesystemContext
}
// Underlay contains inputs that must compose strictly within one tier. Tiers
// appear in lowest-to-highest precedence order; an empty tier is a no-op.
type Underlay struct{ Inputs []Input }
// LayeredOptions selects lower-priority tiers and final application rules.
// An active Overlay requires an explicit OverlayFilesystem; an inactive overlay
// ignores it. Overrides remain explicit CLI records, not general mappings.
type LayeredOptions[T any] struct {
Definition Definition[T]
Underlays []Underlay
Overlay string
OverlayFilesystem *FilesystemContext
Overrides []Override
}
// LoadLayered acquires the main input first and strictly composes enabled
// imports with each input's tier. It strictly composes contributions within
// each underlay, then recursively combines underlays in listed order beneath
// main, a selected overlay, and CLI overrides. Defaults, decoding, resolution,
// and validation run once after sibling `extends` expansion of the assembled
// input. Source IDs follow acquisition order, not precedence order. Zero or
// one options value is accepted; any error
// returns a nil result.
func LoadLayered[T any](main Input, supplied ...LayeredOptions[T]) (*Result[T], error) {
options := make([]engine.LayeredOptions[T], len(supplied))
for index := range supplied {
options[index] = toEngineLayeredOptions(supplied[index])
}
loaded, err := engine.LoadLayered(toEngineInput(main), secretBinding(), options...)
return fromEngineLoaded(loaded, err)
}
func toEngineInput(input Input) engine.Input {
return engine.Input{
Source: input.Source.toEngine(), Imports: input.Imports,
Filesystem: toEngineFilesystemContext(input.Filesystem),
}
}
func toEngineLayeredOptions[T any](options LayeredOptions[T]) engine.LayeredOptions[T] {
result := engine.LayeredOptions[T]{
Definition: toEngineDefinition(options.Definition), Overlay: options.Overlay,
OverlayFilesystem: toEngineFilesystemContext(options.OverlayFilesystem),
}
if options.Underlays != nil {
result.Underlays = make([]engine.Underlay, len(options.Underlays))
for index, group := range options.Underlays {
if group.Inputs != nil {
result.Underlays[index].Inputs = make([]engine.Input, len(group.Inputs))
for inputIndex, input := range group.Inputs {
result.Underlays[index].Inputs[inputIndex] = toEngineInput(input)
}
}
}
}
if options.Overrides != nil {
result.Overrides = make([]engine.Override, len(options.Overrides))
for index, override := range options.Overrides {
result.Overrides[index] = engine.Override{
Path: toEnginePath(override.Path), Value: override.Value, Flag: override.Flag,
}
}
}
return result
}

View File

@@ -28,10 +28,14 @@ type Options[T any] struct {
Overrides []Override Overrides []Override
} }
// FilesystemContext supplies filesystem roots for imports and overlays. // FilesystemContext supplies roots for imports and overlays. With FS nil,
// BaseDir and Root use absolute host paths. With FS set, they use slash-relative
// paths in that namespace ("." denotes its root). File sources derive BaseDir
// from the selected file; a native file's explicit Root limits its scope.
type FilesystemContext struct { type FilesystemContext struct {
BaseDir string BaseDir string
Root string Root string
FS *Filesystem
} }
// Override describes one explicitly supplied application override. Value uses // Override describes one explicitly supplied application override. Value uses
@@ -75,14 +79,19 @@ type Result[T any] struct {
} }
// Load parses and strictly decodes zero or one selected source, explicitly enabled // Load parses and strictly decodes zero or one selected source, explicitly enabled
// imports, and one selected overlay. It accepts at most one Options value and // imports, and one selected overlay. After source precedence, built-in sibling
// returns no result when any error is reported. // `extends` relationships resolve before defaults and typed decoding. It accepts
// at most one Options value and returns no result when any error is reported.
func Load[T any](source Source, supplied ...Options[T]) (*Result[T], error) { func Load[T any](source Source, supplied ...Options[T]) (*Result[T], error) {
options := make([]engine.Options[T], len(supplied)) options := make([]engine.Options[T], len(supplied))
for index := range supplied { for index := range supplied {
options[index] = toEngineOptions(supplied[index]) options[index] = toEngineOptions(supplied[index])
} }
loaded, err := engine.Load(source.toEngine(), secretBinding(), options...) loaded, err := engine.Load(source.toEngine(), secretBinding(), options...)
return fromEngineLoaded(loaded, err)
}
func fromEngineLoaded[T any](loaded *engine.Loaded[T], err error) (*Result[T], error) {
if err != nil { if err != nil {
var loadError *engine.LoadError var loadError *engine.LoadError
if errors.As(err, &loadError) { if errors.As(err, &loadError) {
@@ -133,7 +142,7 @@ func (r *Result[T]) Diagnostics() []Diagnostic {
return result return result
} }
// IsExplicit reports whether surviving input explicitly defined path. // IsExplicit reports whether surviving direct or inherited input defined path.
func (r *Resolution) IsExplicit(path Path) bool { func (r *Resolution) IsExplicit(path Path) bool {
return r.engineContext().IsExplicit(toEnginePath(path)) return r.engineContext().IsExplicit(toEnginePath(path))
} }
@@ -222,7 +231,7 @@ func toEngineOptions[T any](options Options[T]) engine.Options[T] {
Overlay: options.Overlay, Overlay: options.Overlay,
} }
if options.Filesystem != nil { if options.Filesystem != nil {
result.Filesystem = &engine.FilesystemContext{BaseDir: options.Filesystem.BaseDir, Root: options.Filesystem.Root} result.Filesystem = toEngineFilesystemContext(options.Filesystem)
} }
if options.Overrides != nil { if options.Overrides != nil {
result.Overrides = make([]engine.Override, len(options.Overrides)) result.Overrides = make([]engine.Override, len(options.Overrides))
@@ -235,6 +244,18 @@ func toEngineOptions[T any](options Options[T]) engine.Options[T] {
return result return result
} }
func toEngineFilesystemContext(context *FilesystemContext) *engine.FilesystemContext {
if context == nil {
return nil
}
result := &engine.FilesystemContext{BaseDir: context.BaseDir, Root: context.Root}
if context.FS != nil {
filesystem := context.FS.value
result.FS = &filesystem
}
return result
}
func toEngineDefinition[T any](definition Definition[T]) engine.Definition[T] { func toEngineDefinition[T any](definition Definition[T]) engine.Definition[T] {
result := engine.Definition[T]{Defaults: definition.Defaults} result := engine.Definition[T]{Defaults: definition.Defaults}
if definition.Resolve != nil { if definition.Resolve != nil {

View File

@@ -13,13 +13,13 @@ func (m Metadata) Lookup(path Path) (FieldInfo, bool) {
return fromEngineFieldInfo(value), ok return fromEngineFieldInfo(value), ok
} }
// IsExplicit reports whether a surviving input contribution explicitly defined // IsExplicit reports whether a surviving input contribution, including one
// path. // inherited from a sibling, explicitly defined path.
func (m Metadata) IsExplicit(path Path) bool { func (m Metadata) IsExplicit(path Path) bool {
return m.value.IsExplicit(toEnginePath(path)) return m.value.IsExplicit(toEnginePath(path))
} }
// History returns detached definition and resolution records for path. A record's // History returns detached definition, inheritance, and resolution records for path. A record's
// Path can be an ancestor when that container operation replaced or removed path. // Path can be an ancestor when that container operation replaced or removed path.
// Removed historical paths can have history even though Lookup returns false. // Removed historical paths can have history even though Lookup returns false.
// Unknown and invalid paths return nil. // Unknown and invalid paths return nil.
@@ -36,7 +36,7 @@ func (m Metadata) History(path Path) []Change {
} }
// HistoricalPaths returns the paths observed during loading, including effective // HistoricalPaths returns the paths observed during loading, including effective
// paths and paths removed by overlays or recorded resolution. It includes default // paths and paths removed by overlays, inheritance, or recorded resolution. It includes default
// paths and the root, but not paths that never existed. The zero Metadata returns // paths and the root, but not paths that never existed. The zero Metadata returns
// an empty slice. // an empty slice.
// //

View File

@@ -6,11 +6,25 @@ type Layer string
const ( const (
LayerDefault Layer = "default" LayerDefault Layer = "default"
LayerBase Layer = "base" LayerBase Layer = "base"
LayerUnderlay Layer = "underlay"
LayerOverlay Layer = "overlay" LayerOverlay Layer = "overlay"
LayerCLI Layer = "cli" LayerCLI Layer = "cli"
LayerResolution Layer = "resolution" LayerResolution Layer = "resolution"
) )
// SourceKind identifies how a configuration source was acquired, independent
// of its precedence layer.
type SourceKind string
const (
SourceKindFile SourceKind = "file"
SourceKindFilesystem SourceKind = "filesystem"
SourceKindBytes SourceKind = "bytes"
SourceKindReader SourceKind = "reader"
SourceKindMapping SourceKind = "mapping"
SourceKindCLI SourceKind = "cli"
)
// ChangeKind identifies a recorded operation. The zero value is // ChangeKind identifies a recorded operation. The zero value is
// used for input and default origins. // used for input and default origins.
type ChangeKind string type ChangeKind string
@@ -19,6 +33,9 @@ const (
// ChangeDecoding records the built-in transition to canonical typed values. // ChangeDecoding records the built-in transition to canonical typed values.
// It is inspection metadata, not an operation accepted by Resolution.Record. // It is inspection metadata, not an operation accepted by Resolution.Record.
ChangeDecoding ChangeKind = "decoding" ChangeDecoding ChangeKind = "decoding"
// ChangeInheritance records structural input expansion. It is inspection
// metadata, not an operation accepted by Resolution.Record.
ChangeInheritance ChangeKind = "inheritance"
ChangeDerivedDefault ChangeKind = "derived_default" ChangeDerivedDefault ChangeKind = "derived_default"
ChangeNormalization ChangeKind = "normalization" ChangeNormalization ChangeKind = "normalization"
) )
@@ -89,6 +106,7 @@ const (
CodeImportRepeated Code = "import.repeated" CodeImportRepeated Code = "import.repeated"
CodeImportLimitExceeded Code = "import.limit_exceeded" CodeImportLimitExceeded Code = "import.limit_exceeded"
CodeImportVersion Code = "import.version" CodeImportVersion Code = "import.version"
CodeUnderlayVersion Code = "underlay.version"
CodeOverlayInvalid Code = "overlay.invalid" CodeOverlayInvalid Code = "overlay.invalid"
CodeOverlayMissing Code = "overlay.missing" CodeOverlayMissing Code = "overlay.missing"
@@ -146,6 +164,11 @@ const (
CodeOverrideLimitExceeded Code = "override.limit_exceeded" CodeOverrideLimitExceeded Code = "override.limit_exceeded"
CodeExplainLimitExceeded Code = "explain.limit_exceeded" CodeExplainLimitExceeded Code = "explain.limit_exceeded"
CodeInheritanceInvalid Code = "inheritance.invalid"
CodeInheritanceMissing Code = "inheritance.missing"
CodeInheritanceCycle Code = "inheritance.cycle"
CodeInheritanceVersion Code = "inheritance.version"
) )
// Location identifies a position in a configuration source. Zero line and column // Location identifies a position in a configuration source. Zero line and column
@@ -156,14 +179,18 @@ type Location struct {
Column int Column int
} }
// SourceInfo describes a source discovered during a load. Path is a canonical // SourceInfo describes a source discovered during a load. Kind identifies its
// local path when one exists. ImportedFrom identifies the declaration that // acquisition, independent of Layer. Path is canonical for a host file and
// included this source. // namespace-relative for a filesystem file; synthetic sources have no Path.
// UnderlayIndex is the one-based original underlay position, or zero for other
// sources. ImportedFrom identifies the declaration that included this source.
type SourceInfo struct { type SourceInfo struct {
ID string ID string
Label string Label string
Path string Path string
Layer Layer Layer Layer
Kind SourceKind
UnderlayIndex int
ImportedFrom *Location ImportedFrom *Location
} }
@@ -178,13 +205,25 @@ type Diagnostic struct {
Related []Location Related []Location
} }
// InheritanceStep identifies a concrete payload transfer. From and To name
// semantic value paths; Reference names the controlling destination `extends`
// path. Locations belong to that directive, not the transferred value.
type InheritanceStep struct {
From Path
To Path
Reference Path
Locations []Location
}
// Origin describes where an effective definition came from. Multiple locations // Origin describes where an effective definition came from. Multiple locations
// represent a composite origin; resolution origins are synthetic and identify their // represent a composite origin; resolution origins are synthetic and identify their
// operation instead of a file location. // operation instead of a file location. Inheritance records the concrete
// derivation path while Layer and Locations continue to identify value sources.
type Origin struct { type Origin struct {
Layer Layer Layer Layer
Locations []Location Locations []Location
Operation ChangeKind Operation ChangeKind
Inheritance []InheritanceStep
} }
// FieldInfo describes an effective configuration path. // FieldInfo describes an effective configuration path.
@@ -238,6 +277,16 @@ func cloneSourceInfos(sources []SourceInfo) []SourceInfo {
func cloneOrigin(origin Origin) Origin { func cloneOrigin(origin Origin) Origin {
origin.Locations = cloneLocations(origin.Locations) origin.Locations = cloneLocations(origin.Locations)
if origin.Inheritance != nil {
steps := make([]InheritanceStep, len(origin.Inheritance))
for index, step := range origin.Inheritance {
steps[index] = InheritanceStep{
From: step.From.clone(), To: step.To.clone(), Reference: step.Reference.clone(),
Locations: cloneLocations(step.Locations),
}
}
origin.Inheritance = steps
}
return origin return origin
} }

33
resource_context.go Normal file
View File

@@ -0,0 +1,33 @@
package configkit
import "io/fs"
// ResourceContext provides a passive rooted filesystem view and a
// slash-relative BaseDir for one acquired source. FS is borrowed: caller-owned
// filesystems remain under caller control, while a host context opens a fresh
// contained root for each resource open rather than retaining a loading handle.
// Querying this record performs no I/O. Resource contents may change after
// loading; callers must close files they open through FS.
type ResourceContext struct {
FS fs.FS
BaseDir string
}
// SourceContext returns the context of an acquired source ID. Unknown IDs and
// sources without context return false. It is nil-safe and performs no I/O.
// Follow individual origin locations when an origin has multiple sources.
func (r *Result[T]) SourceContext(id string) (ResourceContext, bool) {
if r == nil || r.inspection == nil {
return ResourceContext{}, false
}
context, ok := r.inspection.SourceContext(id)
return ResourceContext{FS: context.FS, BaseDir: context.BaseDir}, ok
}
// SourceContext returns a passive source context during the active resolution
// callback. Like other Resolution queries, it cannot be used after callback
// return and performs no filesystem operations itself.
func (r *Resolution) SourceContext(id string) (ResourceContext, bool) {
context, ok := r.engineContext().SourceContext(id)
return ResourceContext{FS: context.FS, BaseDir: context.BaseDir}, ok
}

View File

@@ -2,14 +2,31 @@ package configkit
import ( import (
"io" "io"
"io/fs"
"gitea.maximumdirect.net/eric/ConfigKit/internal/engine" "gitea.maximumdirect.net/eric/ConfigKit/internal/engine"
) )
// Source is an opaque configuration input. Construct sources with File, Bytes, // Source is an opaque configuration input. Construct sources with File,
// Reader, or NoSource; the zero value is invalid. // Filesystem.File, Bytes, Reader, Mapping, or NoSource; the zero value is invalid.
type Source struct{ value engine.Source } type Source struct{ value engine.Source }
// Filesystem is a reusable opaque namespace for a caller-owned fs.FS. Copies
// preserve identity; separately constructed values are distinct even if their
// labels or underlying filesystems match. The filesystem is never closed.
type Filesystem struct{ value engine.Filesystem }
// NewFilesystem borrows filesystem without opening it. Label must be nonempty
// and safe to display; a nil filesystem or invalid label is reported when a
// source from the returned namespace is consumed.
func NewFilesystem(label string, filesystem fs.FS) Filesystem {
return Filesystem{value: engine.NewFilesystem(label, filesystem)}
}
// File selects a non-root fs.ValidPath name in this filesystem. Each load
// reopens the file; constructor problems are reported when the source is used.
func (f Filesystem) File(name string) Source { return Source{value: f.value.File(name)} }
// File returns a source that reopens path for every load attempt. // File returns a source that reopens path for every load attempt.
func File(path string) Source { return Source{value: engine.File(path)} } func File(path string) Source { return Source{value: engine.File(path)} }
@@ -21,6 +38,15 @@ func Bytes(label string, data []byte) Source { return Source{value: engine.Bytes
// Copies of the returned Source share the same single-use claim. // Copies of the returned Source share the same single-use claim.
func Reader(label string, input io.Reader) Source { return Source{value: engine.Reader(label, input)} } func Reader(label string, input io.Reader) Source { return Source{value: engine.Reader(label, input)} }
// Mapping snapshots supported built-in values immediately. A nil root map is
// invalid; an empty map is a valid empty document. Values may be nil, built-in
// scalars, []any, or map[string]any recursively. Unsupported values and cycles
// are reported when the source is consumed. Label must be nonempty and safe to
// display. Later caller mutation cannot change the captured document.
func Mapping(label string, values map[string]any) Source {
return Source{value: engine.Mapping(label, values)}
}
// NoSource explicitly selects a defaults-only input. It creates no source record. // NoSource explicitly selects a defaults-only input. It creates no source record.
func NoSource() Source { return Source{value: engine.NoSource()} } func NoSource() Source { return Source{value: engine.NoSource()} }

5
testdata/filesystem/config/main.yml vendored Normal file
View File

@@ -0,0 +1,5 @@
configkit:
imports:
- path: ../shared.yml
items:
main: base

View File

@@ -0,0 +1,2 @@
items:
main: selected

2
testdata/filesystem/shared.yml vendored Normal file
View File

@@ -0,0 +1,2 @@
items:
shared: default