Complete bounded glob imports
This commit is contained in:
@@ -3,12 +3,13 @@
|
|||||||
An opinionated Go library for applications that use YAML configuration files.
|
An opinionated Go library for applications that use YAML configuration files.
|
||||||
|
|
||||||
ConfigKit is in early implementation. Its current loader accepts one strict YAML
|
ConfigKit is in early implementation. Its current loader accepts one strict YAML
|
||||||
source (or explicit defaults-only input), optionally traverses explicit path imports,
|
source (or explicit defaults-only input), optionally traverses explicit path and glob
|
||||||
strictly composes their disjoint definitions, applies typed defaults, and can check
|
imports, strictly composes their disjoint definitions, applies typed defaults, and
|
||||||
an application schema version. It returns detached metadata and safe diagnostics.
|
can check an application schema version. It returns detached metadata and safe
|
||||||
|
diagnostics.
|
||||||
Applications may perform audited defaulting and normalization through a resolver,
|
Applications may perform audited defaulting and normalization through a resolver,
|
||||||
then report domain errors and warnings through read-only semantic validation.
|
then report domain errors and warnings through read-only semantic validation.
|
||||||
Glob imports, profiles, overrides, secrets, and rendering are still under development.
|
Profiles, overrides, secrets, and rendering are still under development.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
type Config struct {
|
type Config struct {
|
||||||
|
|||||||
@@ -22,9 +22,10 @@ File sources can observe file changes between consumptions; byte sources cannot.
|
|||||||
|
|
||||||
Each source is limited to 8 MiB. An imported load is limited to 64 MiB across all
|
Each source is limited to 8 MiB. An imported load is limited to 64 MiB across all
|
||||||
accepted sources, 1,024 accepted sources, 64 active import edges, and 1,000,000
|
accepted sources, 1,024 accepted sources, 64 active import edges, and 1,000,000
|
||||||
input nodes in aggregate. Reads request at most the remaining allowance plus one byte
|
input nodes in aggregate. Glob expansion separately permits 1,000,000 visited
|
||||||
so excess data is rejected rather than silently truncated. Data returned together
|
directory entries, including nonmatches. Reads request at most the remaining
|
||||||
with a read error is not accepted as partial input.
|
allowance plus one byte so excess data is rejected rather than silently truncated.
|
||||||
|
Data returned together with a read error is not accepted as partial input.
|
||||||
|
|
||||||
## Single-source loading
|
## Single-source loading
|
||||||
|
|
||||||
@@ -50,10 +51,10 @@ contributes no source, and continues through defaults, decode, and projection. A
|
|||||||
missing file remains an error; it is never interpreted as no input.
|
missing file remains an error; it is never interpreted as no input.
|
||||||
|
|
||||||
The current `Options` boundary accepts a defaults factory, `VersionRule`, resolver,
|
The current `Options` boundary accepts a defaults factory, `VersionRule`, resolver,
|
||||||
semantic validator, explicit path imports, and import filesystem context. Active
|
semantic validator, explicit path and glob imports, and filesystem context. Active
|
||||||
profiles, standalone filesystem context, overrides, and dynamic secret classification
|
profiles, overrides, and dynamic secret classification return `options.unavailable`
|
||||||
return `options.unavailable` because their lifecycle stages are not implemented.
|
because their lifecycle stages are not implemented. The reserved profile spelling
|
||||||
The reserved profile spelling `default` selects no active profile. `NoSource`
|
`default` selects no active profile. `NoSource`
|
||||||
additionally rejects imports, active profiles, and filesystem context as invalid
|
additionally rejects imports, active profiles, and filesystem context as invalid
|
||||||
combinations. Input containing a reserved `_configkit` mapping key is rejected while
|
combinations. Input containing a reserved `_configkit` mapping key is rejected while
|
||||||
imports are disabled.
|
imports are disabled.
|
||||||
@@ -180,16 +181,14 @@ traversal order. Traversal order never authorizes replacement. Composition clone
|
|||||||
the resulting tree and its ordered claim records rather than mutating or aliasing an
|
the resulting tree and its ordered claim records rather than mutating or aliasing an
|
||||||
input document.
|
input document.
|
||||||
|
|
||||||
## Explicit path imports
|
## Explicit imports
|
||||||
|
|
||||||
Imports are opt-in. With `Options.Imports` enabled, a document may reserve its root
|
Imports are opt-in. With `Options.Imports` enabled, a document may reserve its root
|
||||||
`_configkit` mapping for an `imports` sequence. ConfigKit removes that mapping before
|
`_configkit` mapping for an `imports` sequence. ConfigKit removes that mapping before
|
||||||
application decoding. Each sequence item is a mapping with exactly one nonempty
|
application decoding. Each sequence item is a mapping with exactly one nonempty
|
||||||
string `path` or `glob` plus an optional boolean `optional`; all other shapes and
|
string `path` or `glob` plus an optional boolean `optional`; all other shapes and
|
||||||
fields are `import.invalid`. `glob` is recognized but temporarily returns
|
fields are `import.invalid`. A `_configkit` key below the root is invalid. With
|
||||||
`import.unsupported` until bounded glob expansion is implemented. A `_configkit`
|
imports disabled, any occurrence remains `import.disabled`.
|
||||||
key below the root is invalid. With imports disabled, any occurrence remains
|
|
||||||
`import.disabled`.
|
|
||||||
|
|
||||||
Path declarations use slash-separated relative paths. Absolute paths and network
|
Path declarations use slash-separated relative paths. Absolute paths and network
|
||||||
URLs are invalid. Each path resolves from its declaring file's canonical directory,
|
URLs are invalid. Each path resolves from its declaring file's canonical directory,
|
||||||
@@ -199,16 +198,29 @@ permits replacement. Required missing paths fail; `optional: true` suppresses on
|
|||||||
target nonexistence, not invalid paths, containment failures, directories,
|
target nonexistence, not invalid paths, containment failures, directories,
|
||||||
permissions, parse errors, or composition errors.
|
permissions, parse errors, or composition errors.
|
||||||
|
|
||||||
|
Glob declarations use the slash-based Go `path.Match` grammar: `*`, `?`, character
|
||||||
|
classes, and backslash escapes are supported, while recursive `**` syntax is invalid.
|
||||||
|
Backslashes escape pattern characters rather than acting as directory separators.
|
||||||
|
ConfigKit validates the complete pattern before filesystem traversal, walks only
|
||||||
|
directories selected by preceding segments, and reads wildcard directories in
|
||||||
|
bounded batches. Every examined entry consumes the separate visit budget, including
|
||||||
|
nonmatches. Matches are sorted lexicographically by slash-relative spelling within
|
||||||
|
each declaration before depth-first source traversal. A required zero-match glob is
|
||||||
|
`import.missing`; `optional: true` suppresses only zero matches. Invalid patterns,
|
||||||
|
walk failures, matched directories, invalid documents, escapes, and repeated targets
|
||||||
|
remain errors. Final candidates are bounded by the remaining source limit before
|
||||||
|
their contents are accumulated.
|
||||||
|
|
||||||
The configuration root defaults to a selected file's canonical directory. A file
|
The configuration root defaults to a selected file's canonical directory. A file
|
||||||
source may supply `FilesystemContext.Root` to widen that root but may not supply
|
source may supply `FilesystemContext.Root` to widen that root but may not supply
|
||||||
`BaseDir`. Byte and reader sources must supply both `BaseDir` and `Root` as absolute
|
`BaseDir`. Whenever byte and reader sources supply filesystem context, they must
|
||||||
existing directories when imports are enabled; their diagnostic labels, including
|
provide both `BaseDir` and `Root` as absolute existing directories; their diagnostic
|
||||||
URI-looking labels, never imply filesystem paths. The base and selected file must be
|
labels, including URI-looking labels, never imply filesystem paths. The base and
|
||||||
inside the canonical root. ConfigKit canonicalizes symlinks, checks containment with
|
selected file must be inside the canonical root. ConfigKit canonicalizes symlinks,
|
||||||
relative paths, and opens the canonical root-relative target through a library-owned
|
checks containment with relative paths, and opens the canonical root-relative target
|
||||||
root handle. Contained relative or absolute symlink targets are accepted; escapes
|
through a library-owned root handle. Contained relative or absolute symlink targets
|
||||||
are rejected. Canonical paths define identity, so hard links at distinct paths remain
|
are accepted; escapes are rejected. Canonical paths define identity, so hard links at
|
||||||
distinct sources.
|
distinct paths remain distinct sources.
|
||||||
|
|
||||||
An active-stack repeat is `import.cycle` and reports the complete cycle's import
|
An active-stack repeat is `import.cycle` and reports the complete cycle's import
|
||||||
sites. A previously visited canonical path is `import.repeated` and reports both
|
sites. A previously visited canonical path is `import.repeated` and reports both
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
This is the contributor entry point for ConfigKit. Read the task-specific owners
|
This is the contributor entry point for ConfigKit. Read the task-specific owners
|
||||||
below before making changes. The repository is in early implementation: its public
|
below before making changes. The repository is in early implementation: its public
|
||||||
loader currently covers strict selected sources, explicit path-import graphs,
|
loader currently covers strict selected sources, explicit path/glob import graphs,
|
||||||
composition, typed defaults, schema-version checks, recorded application resolution,
|
composition, typed defaults, schema-version checks, recorded application resolution,
|
||||||
semantic validation, and metadata. The
|
semantic validation, and metadata. The
|
||||||
[component overview](internal/overview.md) owns the implementation inventory.
|
[component overview](internal/overview.md) owns the implementation inventory.
|
||||||
|
|||||||
@@ -124,6 +124,16 @@ fragment version-placement enforcement. The planned glob syntax is parsed but
|
|||||||
remains explicitly unavailable for its next owning work; hard links intentionally
|
remains explicitly unavailable for its next owning work; hard links intentionally
|
||||||
replace Narratio's same-file rejection with path identity.
|
replace Narratio's same-file rejection with path identity.
|
||||||
|
|
||||||
|
Glob-import implementation record (2026-09-06): ConfigKit retains the extracted
|
||||||
|
declaration order, import failures, composition, and identity cases while adding the
|
||||||
|
new `expandImportGlob`, `readGlobMatches`, and pattern-validation mechanisms in
|
||||||
|
`glob.go`. Focused cases cover lexical ordering, multiple declarations, overlap,
|
||||||
|
directories, hard links, contained and escaping symlinks, unmatched declarations,
|
||||||
|
malformed and recursive patterns, escaped metacharacters, character classes,
|
||||||
|
permission failures, explicit `conf.d` use, and entry/candidate budgets. This work
|
||||||
|
has no Narratio production equivalent; it uses the planned standard-library
|
||||||
|
`path.Match` and bounded `File.ReadDir` adaptation rather than its flat loader.
|
||||||
|
|
||||||
### Profiles and overlay
|
### Profiles and overlay
|
||||||
|
|
||||||
Assigned to selected-profile overlay:
|
Assigned to selected-profile overlay:
|
||||||
|
|||||||
@@ -20,8 +20,9 @@ module and package documentation boundary. Its implemented public primitives are
|
|||||||
- strict schema-driven decoding over detached defaults in `decode.go`, including
|
- strict schema-driven decoding over detached defaults in `decode.go`, including
|
||||||
mapping/sequence semantics, null and numeric policy, safe codec errors, presence,
|
mapping/sequence semantics, null and numeric policy, safe codec errors, presence,
|
||||||
baseline/final trees, and initial default/input origins; and
|
baseline/final trees, and initial default/input origins; and
|
||||||
- public lifecycle coordination in `load.go`, bounded canonical path-import DFS in
|
- public lifecycle coordination in `load.go`, bounded canonical import DFS in
|
||||||
`imports.go`, version-report contexts in `validation.go`, and detached
|
`imports.go`, pattern-directed glob expansion in `glob.go`, version-report contexts
|
||||||
|
in `validation.go`, and detached
|
||||||
base/default metadata assembly in `load_metadata.go`; and
|
base/default metadata assembly in `load_metadata.go`; and
|
||||||
- recorded application defaulting and normalization in `resolution.go`, including
|
- recorded application defaulting and normalization in `resolution.go`, including
|
||||||
explicitness queries, helper verification, structural mutation audit, coalesced
|
explicitness queries, helper verification, structural mutation audit, coalesced
|
||||||
|
|||||||
@@ -878,7 +878,7 @@ Completion evidence (2026-09-06):
|
|||||||
tests cover nested/list-order traversal, composition conflicts, version placement,
|
tests cover nested/list-order traversal, composition conflicts, version placement,
|
||||||
directives, optional/required paths, explicit roots, cycles, diamonds, symlinks,
|
directives, optional/required paths, explicit roots, cycles, diamonds, symlinks,
|
||||||
hard links, parse/decode attribution, file types, context rules, and reduced limit
|
hard links, parse/decode attribution, file types, context rules, and reduced limit
|
||||||
boundaries. The [current path-import contract](../config.md#explicit-path-imports)
|
boundaries. The [current path-import contract](../config.md#explicit-imports)
|
||||||
owns implemented behavior; glob expansion remains pending. Verified `go test ./...`,
|
owns implemented behavior; glob expansion remains pending. Verified `go test ./...`,
|
||||||
`go test -race ./...`, `go vet ./...`, `go build ./...`, `go mod tidy -diff`, and
|
`go test -race ./...`, `go vet ./...`, `go build ./...`, `go mod tidy -diff`, and
|
||||||
`git diff --check` with Go 1.26.7 on linux/amd64.
|
`git diff --check` with Go 1.26.7 on linux/amd64.
|
||||||
@@ -917,7 +917,36 @@ Acceptance:
|
|||||||
|
|
||||||
## Stage 12 — Complete glob imports and filesystem option handling
|
## Stage 12 — Complete glob imports and filesystem option handling
|
||||||
|
|
||||||
Status: pending.
|
Status: complete.
|
||||||
|
|
||||||
|
Completion evidence (2026-09-06):
|
||||||
|
|
||||||
|
- Removed the temporary glob gate and added prevalidated Go `path.Match` semantics
|
||||||
|
for `*`, `?`, character classes, and escapes, with recursive unescaped `**`, URLs,
|
||||||
|
absolute paths, empty segments, and malformed classes rejected before any walk.
|
||||||
|
- Added a pattern-directed walker that resolves literal segments directly and reads
|
||||||
|
wildcard directories through positive bounded `File.ReadDir` batches. It charges
|
||||||
|
every examined entry, including nonmatches, canonicalizes and contains traversed
|
||||||
|
directories and final targets, sorts slash-relative matches, and bounds final
|
||||||
|
candidates against remaining source capacity before traversal.
|
||||||
|
- Reused the path-import DFS for every sorted match, preserving declaration ordering,
|
||||||
|
nested depth-first imports, canonical alias/repetition handling, hard-link path
|
||||||
|
identity, source IDs, composition, version placement, and all existing byte/node/
|
||||||
|
source/depth budgets. Optional glob declarations suppress only zero matches, never
|
||||||
|
invalid syntax, walk/permission failures, matched directories, or symlink escapes.
|
||||||
|
- Completed filesystem option validation independently of import/profile selection:
|
||||||
|
file inputs reject `BaseDir` and may select a containing root; byte/readers require
|
||||||
|
existing absolute base/root pairs whenever context is supplied; `NoSource` rejects
|
||||||
|
context while the reserved `default` profile remains inactive.
|
||||||
|
- Tests cover lexical and multi-declaration ordering, deterministic repeated loads,
|
||||||
|
overlapping matches, directories, hard links, contained absolute and escaping
|
||||||
|
symlink directories, required/optional no-match behavior, invalid patterns in empty
|
||||||
|
directories, escaped metacharacters, character classes, visible permission errors,
|
||||||
|
explicit `conf.d` imports without discovery, nonmatching-entry exhaustion, candidate
|
||||||
|
bounds, and filesystem combinations. The
|
||||||
|
[current import contract](../config.md#explicit-imports) owns the completed behavior.
|
||||||
|
Verified `go test ./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`,
|
||||||
|
`go mod tidy -diff`, and `git diff --check` with Go 1.26.7 on linux/amd64.
|
||||||
|
|
||||||
Read: feature import syntax and source-context API.
|
Read: feature import syntax and source-context API.
|
||||||
|
|
||||||
|
|||||||
@@ -475,30 +475,9 @@ the surviving sequence, not an attempted correspondence with previous indices.
|
|||||||
|
|
||||||
## Explicit imports and source identity
|
## Explicit imports and source identity
|
||||||
|
|
||||||
Explicit path-import traversal, source identity, containment, budgets, recursive
|
Explicit path and glob imports, recursive traversal, source identity, containment,
|
||||||
directives, and the temporary glob gate are implemented and owned by the
|
filesystem context, and budgets are implemented and now owned by the
|
||||||
[current configuration contract](../config.md#explicit-path-imports). Bounded glob
|
[current configuration contract](../config.md#explicit-imports).
|
||||||
expansion remains target behavior. Illustrative future fragment:
|
|
||||||
|
|
||||||
```yaml
|
|
||||||
_configkit:
|
|
||||||
imports:
|
|
||||||
- glob: fragments/outputs/*.yml
|
|
||||||
optional: true
|
|
||||||
```
|
|
||||||
|
|
||||||
- Expand declarations in list order, sorting each glob's matches
|
|
||||||
lexicographically by slash-separated relative path, with no locale sorting.
|
|
||||||
Use the slash-based [Go `path.Match` grammar](https://pkg.go.dev/path#Match)
|
|
||||||
(`*`, `?`, character classes, and backslash escapes), with recursive `**`
|
|
||||||
syntax forbidden. Backslashes escape pattern characters, not directory separators.
|
|
||||||
- A required unmatched glob is an error. Optional suppresses only zero matches.
|
|
||||||
Invalid patterns, permission
|
|
||||||
failures, directories, and invalid matched documents remain errors.
|
|
||||||
- Glob traversal counts examined directory entries, including nonmatches, against
|
|
||||||
its separate work budget; only accepted matches contribute to sorted source order.
|
|
||||||
Expansion remains pattern-directed and must preserve the current path-import
|
|
||||||
identity, containment, repetition, and depth-first traversal rules.
|
|
||||||
|
|
||||||
## Profiles
|
## Profiles
|
||||||
|
|
||||||
|
|||||||
205
glob.go
Normal file
205
glob.go
Normal file
@@ -0,0 +1,205 @@
|
|||||||
|
package configkit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"errors"
|
||||||
|
"io"
|
||||||
|
"io/fs"
|
||||||
|
"os"
|
||||||
|
"path"
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
const globReadBatch = 128
|
||||||
|
|
||||||
|
type globCandidate struct {
|
||||||
|
absolute string
|
||||||
|
relative string
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *importTraversal) expandImportGlob(basePath string, declaration importDeclaration) ([]string, []Diagnostic) {
|
||||||
|
location := declaration.location
|
||||||
|
segments, valid := validGlobPattern(declaration.glob)
|
||||||
|
if !valid {
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportInvalid, "import glob pattern is invalid or uses recursive ** syntax", &location)}
|
||||||
|
}
|
||||||
|
|
||||||
|
candidates := []globCandidate{{absolute: basePath}}
|
||||||
|
for index, segment := range segments {
|
||||||
|
last := index == len(segments)-1
|
||||||
|
literal, patterned := globLiteralSegment(segment)
|
||||||
|
var next []globCandidate
|
||||||
|
for _, candidate := range candidates {
|
||||||
|
var names []string
|
||||||
|
if patterned {
|
||||||
|
matched, diagnostics := s.readGlobMatches(candidate.absolute, segment, location)
|
||||||
|
if len(diagnostics) != 0 {
|
||||||
|
return nil, diagnostics
|
||||||
|
}
|
||||||
|
names = matched
|
||||||
|
} else {
|
||||||
|
names = []string{literal}
|
||||||
|
}
|
||||||
|
for _, name := range names {
|
||||||
|
absolute := filepath.Join(candidate.absolute, filepath.FromSlash(name))
|
||||||
|
relative := path.Join(candidate.relative, name)
|
||||||
|
if !containedPath(s.rootPath, filepath.Clean(absolute)) {
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal is outside the configuration root", &location)}
|
||||||
|
}
|
||||||
|
canonical, err := canonicalExistingPath(absolute)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, fs.ErrNotExist) && !patterned {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob match could not be canonicalized", location)}
|
||||||
|
}
|
||||||
|
if !containedPath(s.rootPath, canonical) {
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal resolves outside the configuration root", &location)}
|
||||||
|
}
|
||||||
|
if last {
|
||||||
|
next = append(next, globCandidate{absolute: canonical, relative: relative})
|
||||||
|
if len(s.sources)+len(next) > s.limits.sources {
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob matches exceed the remaining source-count limit", &location)}
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
info, err := os.Stat(canonical)
|
||||||
|
if err != nil {
|
||||||
|
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob traversal target could not be inspected", location)}
|
||||||
|
}
|
||||||
|
if info.IsDir() {
|
||||||
|
next = append(next, globCandidate{absolute: canonical, relative: relative})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
candidates = next
|
||||||
|
if len(candidates) == 0 {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
matches := make([]string, len(candidates))
|
||||||
|
for index, candidate := range candidates {
|
||||||
|
matches[index] = candidate.relative
|
||||||
|
}
|
||||||
|
slices.Sort(matches)
|
||||||
|
return matches, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (s *importTraversal) readGlobMatches(directory, pattern string, location Location) ([]string, []Diagnostic) {
|
||||||
|
canonical, err := canonicalExistingPath(directory)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be canonicalized", location)}
|
||||||
|
}
|
||||||
|
if !containedPath(s.rootPath, canonical) {
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory resolves outside the configuration root", &location)}
|
||||||
|
}
|
||||||
|
relative, err := filepath.Rel(s.rootPath, canonical)
|
||||||
|
if err != nil || filepath.IsAbs(relative) || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) {
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory could not be addressed inside the configuration root", &location)}
|
||||||
|
}
|
||||||
|
directoryFile, err := s.root.Open(relative)
|
||||||
|
if err != nil {
|
||||||
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be opened", location)}
|
||||||
|
}
|
||||||
|
|
||||||
|
var matches []string
|
||||||
|
for {
|
||||||
|
entries, readErr := directoryFile.ReadDir(globReadBatch)
|
||||||
|
for _, entry := range entries {
|
||||||
|
s.usedEntries++
|
||||||
|
if s.usedEntries > s.limits.entries {
|
||||||
|
_ = directoryFile.Close()
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob traversal exceeds the directory-entry visit limit", &location)}
|
||||||
|
}
|
||||||
|
matched, matchErr := path.Match(pattern, entry.Name())
|
||||||
|
if matchErr != nil {
|
||||||
|
_ = directoryFile.Close()
|
||||||
|
return nil, []Diagnostic{importDiagnostic(CodeImportInvalid, "import glob pattern is invalid", &location)}
|
||||||
|
}
|
||||||
|
if matched {
|
||||||
|
matches = append(matches, entry.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if errors.Is(readErr, io.EOF) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if readErr != nil {
|
||||||
|
_ = directoryFile.Close()
|
||||||
|
return nil, []Diagnostic{globFilesystemDiagnostic(readErr, "glob directory could not be read", location)}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := directoryFile.Close(); err != nil {
|
||||||
|
return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be closed", location)}
|
||||||
|
}
|
||||||
|
slices.Sort(matches)
|
||||||
|
return matches, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func validGlobPattern(pattern string) ([]string, bool) {
|
||||||
|
if !validImportPath(pattern) {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
segments := strings.Split(pattern, "/")
|
||||||
|
for _, segment := range segments {
|
||||||
|
if segment == "" || hasRecursiveStars(segment) {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
if _, err := path.Match(segment, ""); err != nil {
|
||||||
|
return nil, false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return segments, true
|
||||||
|
}
|
||||||
|
|
||||||
|
func hasRecursiveStars(segment string) bool {
|
||||||
|
inClass := false
|
||||||
|
for index := 0; index < len(segment); index++ {
|
||||||
|
switch segment[index] {
|
||||||
|
case '\\':
|
||||||
|
index++
|
||||||
|
case '[':
|
||||||
|
inClass = true
|
||||||
|
case ']':
|
||||||
|
inClass = false
|
||||||
|
case '*':
|
||||||
|
if !inClass && index+1 < len(segment) && segment[index+1] == '*' {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
func globLiteralSegment(segment string) (string, bool) {
|
||||||
|
var literal strings.Builder
|
||||||
|
for index := 0; index < len(segment); index++ {
|
||||||
|
switch segment[index] {
|
||||||
|
case '*', '?', '[':
|
||||||
|
return "", true
|
||||||
|
case '\\':
|
||||||
|
index++
|
||||||
|
if index < len(segment) {
|
||||||
|
literal.WriteByte(segment[index])
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
literal.WriteByte(segment[index])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return literal.String(), false
|
||||||
|
}
|
||||||
|
|
||||||
|
func globFilesystemDiagnostic(err error, message string, location Location) Diagnostic {
|
||||||
|
code := CodeImportRead
|
||||||
|
if errors.Is(err, fs.ErrPermission) {
|
||||||
|
code = CodeImportPermission
|
||||||
|
}
|
||||||
|
return importDiagnostic(code, message, &location)
|
||||||
|
}
|
||||||
273
glob_test.go
Normal file
273
glob_test.go
Normal file
@@ -0,0 +1,273 @@
|
|||||||
|
package configkit
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
|
"runtime"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestLoadGlobImportsUseLexicalDeclarationOrderAndNestedDFS(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
mainPath := writeImportFile(t, directory, "main.yml", `_configkit:
|
||||||
|
imports:
|
||||||
|
- glob: fragments/*.yml
|
||||||
|
- path: after.yml
|
||||||
|
items: {root: root}
|
||||||
|
`)
|
||||||
|
aPath := writeImportFile(t, directory, "fragments/a.yml", `_configkit:
|
||||||
|
imports: [{path: nested/child.yml}]
|
||||||
|
items: {alpha: alpha}
|
||||||
|
`)
|
||||||
|
childPath := writeImportFile(t, directory, "fragments/nested/child.yml", "items: {child: child}\n")
|
||||||
|
bPath := writeImportFile(t, directory, "fragments/b.yml", "items: {beta: beta}\n")
|
||||||
|
afterPath := writeImportFile(t, directory, "after.yml", "items: {after: after}\n")
|
||||||
|
|
||||||
|
var previous []SourceInfo
|
||||||
|
for run := 0; run < 2; run++ {
|
||||||
|
result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
wantItems := map[string]string{"root": "root", "alpha": "alpha", "child": "child", "beta": "beta", "after": "after"}
|
||||||
|
if !reflect.DeepEqual(result.Config.Items, wantItems) {
|
||||||
|
t.Fatalf("items = %#v, want %#v", result.Config.Items, wantItems)
|
||||||
|
}
|
||||||
|
sources := result.Sources()
|
||||||
|
wantPaths := []string{
|
||||||
|
canonicalTestPath(t, mainPath), canonicalTestPath(t, aPath),
|
||||||
|
canonicalTestPath(t, childPath), canonicalTestPath(t, bPath),
|
||||||
|
canonicalTestPath(t, afterPath),
|
||||||
|
}
|
||||||
|
gotPaths := make([]string, len(sources))
|
||||||
|
for index, source := range sources {
|
||||||
|
gotPaths[index] = source.Path
|
||||||
|
}
|
||||||
|
if !reflect.DeepEqual(gotPaths, wantPaths) {
|
||||||
|
t.Fatalf("source paths = %#v, want %#v", gotPaths, wantPaths)
|
||||||
|
}
|
||||||
|
if run != 0 && !reflect.DeepEqual(sources, previous) {
|
||||||
|
t.Fatalf("repeated traversal changed sources: %#v != %#v", sources, previous)
|
||||||
|
}
|
||||||
|
previous = sources
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadGlobImportsSupportEscapesClassesAndExplicitConfD(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
mainPath := writeImportFile(t, directory, "main.yml", `_configkit:
|
||||||
|
imports:
|
||||||
|
- glob: 'conf.d/literal\*.yml'
|
||||||
|
- glob: 'conf.d/item-[ab].yml'
|
||||||
|
- glob: 'conf.d/question?.yml'
|
||||||
|
items: {root: root}
|
||||||
|
`)
|
||||||
|
writeImportFile(t, directory, "conf.d/literal*.yml", "items: {literal: literal}\n")
|
||||||
|
writeImportFile(t, directory, "conf.d/item-b.yml", "items: {b: b}\n")
|
||||||
|
writeImportFile(t, directory, "conf.d/item-a.yml", "items: {a: a}\n")
|
||||||
|
writeImportFile(t, directory, "conf.d/question1.yml", "items: {question: question}\n")
|
||||||
|
|
||||||
|
result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if want := map[string]string{"root": "root", "literal": "literal", "a": "a", "b": "b", "question": "question"}; !reflect.DeepEqual(result.Config.Items, want) {
|
||||||
|
t.Fatalf("items = %#v, want %#v", result.Config.Items, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
ignoredPath := writeImportFile(t, directory, "conf.d/not-discovered.yml", "unknown: value\n")
|
||||||
|
_ = ignoredPath
|
||||||
|
plainPath := writeImportFile(t, directory, "plain.yml", "items: {root: only}\n")
|
||||||
|
plain, err := Load[importTestConfig](File(plainPath), Options[importTestConfig]{Imports: true})
|
||||||
|
if err != nil || len(plain.Sources()) != 1 || plain.Config.Items["root"] != "only" {
|
||||||
|
t.Fatalf("implicit discovery result = %#v, %v", plain, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadGlobImportsRejectInvalidAndUnmatchedPatterns(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
pattern string
|
||||||
|
code Code
|
||||||
|
}{
|
||||||
|
{name: "bad class", pattern: "empty/[", code: CodeImportInvalid},
|
||||||
|
{name: "recursive star segment", pattern: "empty/**/value.yml", code: CodeImportInvalid},
|
||||||
|
{name: "recursive adjacent stars", pattern: "empty/value**.yml", code: CodeImportInvalid},
|
||||||
|
{name: "trailing separator", pattern: "empty/", code: CodeImportInvalid},
|
||||||
|
{name: "absolute", pattern: "/tmp/*.yml", code: CodeImportInvalid},
|
||||||
|
{name: "URL", pattern: "https://example.invalid/*.yml", code: CodeImportInvalid},
|
||||||
|
{name: "missing parent escape", pattern: "../missing/*.yml", code: CodeImportOutsideRoot},
|
||||||
|
{name: "unmatched", pattern: "empty/*.yml", code: CodeImportMissing},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
if err := os.Mkdir(filepath.Join(directory, "empty"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, directory, test.pattern, false)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, test.code)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
directory := t.TempDir()
|
||||||
|
if err := os.Mkdir(filepath.Join(directory, "empty"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, directory, "empty/*.yml", true)
|
||||||
|
result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true})
|
||||||
|
if err != nil || len(result.Sources()) != 1 {
|
||||||
|
t.Fatalf("optional unmatched glob = %#v, %v", result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadGlobImportsPreserveIdentityAndFileRules(t *testing.T) {
|
||||||
|
t.Run("overlapping declarations", func(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
writeImportFile(t, directory, "conf/value.yml", "items: {value: value}\n")
|
||||||
|
mainPath := writeImportFile(t, directory, "main.yml", `_configkit:
|
||||||
|
imports:
|
||||||
|
- glob: conf/*.yml
|
||||||
|
- glob: conf/value.*
|
||||||
|
`)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeImportRepeated)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("matched directory", func(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
if err := os.MkdirAll(filepath.Join(directory, "conf/matched.yml"), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, directory, "conf/*.yml", true)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeSourceNotRegular)
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("hard links stay distinct", func(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
one := writeImportFile(t, directory, "conf/one.yml", "items: {same: value}\n")
|
||||||
|
if err := os.Link(one, filepath.Join(directory, "conf/two.yml")); err != nil {
|
||||||
|
t.Skipf("hard links unavailable: %v", err)
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, directory, "conf/*.yml", false)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeComposeConflict)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestLoadGlobImportsConfineSymlinkDirectories(t *testing.T) {
|
||||||
|
if runtime.GOOS != "linux" && runtime.GOOS != "darwin" {
|
||||||
|
t.Skip("initial filesystem contract covers Linux and macOS")
|
||||||
|
}
|
||||||
|
t.Run("contained absolute target", func(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
target := filepath.Join(root, "target")
|
||||||
|
writeImportFile(t, target, "value.yml", "items: {linked: linked}\n")
|
||||||
|
if err := os.Symlink(target, filepath.Join(root, "linked")); err != nil {
|
||||||
|
t.Skipf("symlinks unavailable: %v", err)
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, root, "linked/*.yml", false)
|
||||||
|
result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true})
|
||||||
|
if err != nil || result.Config.Items["linked"] != "linked" {
|
||||||
|
t.Fatalf("contained directory symlink = %#v, %v", result, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
t.Run("escaping target", func(t *testing.T) {
|
||||||
|
root := t.TempDir()
|
||||||
|
outside := t.TempDir()
|
||||||
|
writeImportFile(t, outside, "value.yml", "items: {escaped: escaped}\n")
|
||||||
|
if err := os.Symlink(outside, filepath.Join(root, "linked")); err != nil {
|
||||||
|
t.Skipf("symlinks unavailable: %v", err)
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, root, "linked/*.yml", true)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeImportOutsideRoot)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestOptionalGlobDoesNotSuppressDirectoryPermissionFailure(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
locked := filepath.Join(directory, "locked")
|
||||||
|
if err := os.Mkdir(locked, 0o700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Chmod(locked, 0); err != nil {
|
||||||
|
t.Skipf("cannot create unreadable directory fixture: %v", err)
|
||||||
|
}
|
||||||
|
defer func() { _ = os.Chmod(locked, 0o700) }()
|
||||||
|
if opened, err := os.Open(locked); err == nil {
|
||||||
|
_ = opened.Close()
|
||||||
|
t.Skip("filesystem does not enforce the unreadable-directory fixture")
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, directory, "locked/*.yml", true)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeImportPermission)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGlobTraversalBoundsNonmatchesAndAcceptedCandidates(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
for _, name := range []string{"one.txt", "two.txt", "three.txt"} {
|
||||||
|
writeImportFile(t, directory, "many/"+name, "ignored\n")
|
||||||
|
}
|
||||||
|
mainPath := writeGlobRoot(t, directory, "many/*.yml", true)
|
||||||
|
main := parseImportMain(t, mainPath)
|
||||||
|
limits := defaultImportLimits()
|
||||||
|
limits.entries = 2
|
||||||
|
_, _, diagnostics := loadImportedDocuments(File(mainPath), nil, main, nil, limits)
|
||||||
|
if len(diagnostics) != 1 || diagnostics[0].Code != CodeImportLimitExceeded {
|
||||||
|
t.Fatalf("entry-budget diagnostics = %#v", diagnostics)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, name := range []string{"a.yml", "b.yml"} {
|
||||||
|
writeImportFile(t, directory, "matches/"+name, "items: {"+name[:1]+": value}\n")
|
||||||
|
}
|
||||||
|
mainPath = writeGlobRoot(t, directory, "matches/*.yml", false)
|
||||||
|
main = parseImportMain(t, mainPath)
|
||||||
|
limits = defaultImportLimits()
|
||||||
|
limits.sources = 2
|
||||||
|
_, _, diagnostics = loadImportedDocuments(File(mainPath), nil, main, nil, limits)
|
||||||
|
if len(diagnostics) != 1 || diagnostics[0].Code != CodeImportLimitExceeded {
|
||||||
|
t.Fatalf("candidate-budget diagnostics = %#v", diagnostics)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFilesystemContextIsValidatedWithoutImportsOrActiveProfile(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
mainPath := writeImportFile(t, directory, "main.yml", "items: {root: root}\n")
|
||||||
|
result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{
|
||||||
|
Profile: "default", Filesystem: &FilesystemContext{Root: directory},
|
||||||
|
})
|
||||||
|
if err != nil || result.Config.Items["root"] != "root" {
|
||||||
|
t.Fatalf("file filesystem context = %#v, %v", result, err)
|
||||||
|
}
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Filesystem: &FilesystemContext{BaseDir: directory, Root: directory}}, CodeOptionsInvalid)
|
||||||
|
requireLoadCode(t, File(mainPath), Options[importTestConfig]{Filesystem: &FilesystemContext{Root: filepath.Join(directory, "missing")}}, CodeImportInvalid)
|
||||||
|
|
||||||
|
input := Bytes("memory.yml", []byte("items: {root: memory}\n"))
|
||||||
|
result, err = Load[importTestConfig](input, Options[importTestConfig]{Filesystem: &FilesystemContext{BaseDir: directory, Root: directory}})
|
||||||
|
if err != nil || result.Config.Items["root"] != "memory" {
|
||||||
|
t.Fatalf("byte filesystem context = %#v, %v", result, err)
|
||||||
|
}
|
||||||
|
requireLoadCode(t, Bytes("memory.yml", []byte("{}\n")), Options[importTestConfig]{Filesystem: &FilesystemContext{BaseDir: directory}}, CodeOptionsInvalid)
|
||||||
|
defaultsOnly, err := Load[importTestConfig](NoSource(), Options[importTestConfig]{Profile: "default"})
|
||||||
|
if err != nil || defaultsOnly == nil {
|
||||||
|
t.Fatalf("default profile with NoSource = %#v, %v", defaultsOnly, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if runtime.GOOS == "linux" || runtime.GOOS == "darwin" {
|
||||||
|
selectedLink := filepath.Join(directory, "selected-link.yml")
|
||||||
|
if err := os.Symlink(mainPath, selectedLink); err == nil {
|
||||||
|
linked, err := Load[importTestConfig](File(selectedLink), Options[importTestConfig]{Filesystem: &FilesystemContext{Root: directory}})
|
||||||
|
if err != nil || linked.Sources()[0].Path != canonicalTestPath(t, mainPath) {
|
||||||
|
t.Fatalf("selected symlink identity = %#v, %v", linked, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func writeGlobRoot(t *testing.T, directory, pattern string, optional bool) string {
|
||||||
|
t.Helper()
|
||||||
|
optionalLine := ""
|
||||||
|
if optional {
|
||||||
|
optionalLine = "\n optional: true"
|
||||||
|
}
|
||||||
|
return writeImportFile(t, directory, "main.yml", "_configkit:\n imports:\n - glob: '"+pattern+"'"+optionalLine+"\n")
|
||||||
|
}
|
||||||
94
imports.go
94
imports.go
@@ -5,7 +5,6 @@ import (
|
|||||||
"errors"
|
"errors"
|
||||||
"io"
|
"io"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
"net/url"
|
|
||||||
"os"
|
"os"
|
||||||
"path"
|
"path"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
@@ -19,6 +18,7 @@ import (
|
|||||||
const (
|
const (
|
||||||
maxImportDepth = 64
|
maxImportDepth = 64
|
||||||
maxImportSources = 1024
|
maxImportSources = 1024
|
||||||
|
maxGlobEntries = 1_000_000
|
||||||
)
|
)
|
||||||
|
|
||||||
type importLimits struct {
|
type importLimits struct {
|
||||||
@@ -26,12 +26,13 @@ type importLimits struct {
|
|||||||
sources int
|
sources int
|
||||||
bytes int64
|
bytes int64
|
||||||
nodes int
|
nodes int
|
||||||
|
entries int
|
||||||
}
|
}
|
||||||
|
|
||||||
func defaultImportLimits() importLimits {
|
func defaultImportLimits() importLimits {
|
||||||
return importLimits{
|
return importLimits{
|
||||||
depth: maxImportDepth, sources: maxImportSources,
|
depth: maxImportDepth, sources: maxImportSources,
|
||||||
bytes: maxTotalBytes, nodes: maxDocumentNodes,
|
bytes: maxTotalBytes, nodes: maxDocumentNodes, entries: maxGlobEntries,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -60,6 +61,7 @@ type importTraversal struct {
|
|||||||
stack []importRoute
|
stack []importRoute
|
||||||
usedBytes int64
|
usedBytes int64
|
||||||
usedNodes int
|
usedNodes int
|
||||||
|
usedEntries int
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadImportedDocuments(source Source, context *FilesystemContext, main parsedSource, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) {
|
func loadImportedDocuments(source Source, context *FilesystemContext, main parsedSource, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) {
|
||||||
@@ -69,7 +71,9 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse
|
|||||||
}
|
}
|
||||||
root, err := os.OpenRoot(rootPath)
|
root, err := os.OpenRoot(rootPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, sourceInfos(main.source), []Diagnostic{importDiagnostic(CodeImportInvalid, "configuration root could not be opened", nil)}
|
diagnostic := globFilesystemDiagnostic(err, "configuration root could not be opened", Location{})
|
||||||
|
diagnostic.Primary = nil
|
||||||
|
return nil, sourceInfos(main.source), []Diagnostic{diagnostic}
|
||||||
}
|
}
|
||||||
|
|
||||||
mainInfo := cloneSourceInfo(*main.source)
|
mainInfo := cloneSourceInfo(*main.source)
|
||||||
@@ -107,12 +111,49 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse
|
|||||||
return nil, cloneSourceInfos(state.sources), diagnostics
|
return nil, cloneSourceInfos(state.sources), diagnostics
|
||||||
}
|
}
|
||||||
if closeErr != nil {
|
if closeErr != nil {
|
||||||
return nil, cloneSourceInfos(state.sources), []Diagnostic{importDiagnostic(CodeImportInvalid, "configuration root could not be closed", nil)}
|
return nil, cloneSourceInfos(state.sources), []Diagnostic{importDiagnostic(CodeImportRead, "configuration root could not be closed", nil)}
|
||||||
}
|
}
|
||||||
composed, diagnostics := composeDocuments(state.documents...)
|
composed, diagnostics := composeDocuments(state.documents...)
|
||||||
return composed, cloneSourceInfos(state.sources), diagnostics
|
return composed, cloneSourceInfos(state.sources), diagnostics
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateFilesystemContext(source Source, context *FilesystemContext, main parsedSource) *Diagnostic {
|
||||||
|
rootPath, _, _, diagnostic := importFilesystem(source, context, main)
|
||||||
|
if diagnostic != nil {
|
||||||
|
return diagnostic
|
||||||
|
}
|
||||||
|
root, err := os.OpenRoot(rootPath)
|
||||||
|
if err != nil {
|
||||||
|
value := globFilesystemDiagnostic(err, "configuration root could not be opened", Location{})
|
||||||
|
value.Primary = nil
|
||||||
|
return &value
|
||||||
|
}
|
||||||
|
if err := root.Close(); err != nil {
|
||||||
|
value := importDiagnostic(CodeImportRead, "configuration root could not be closed", nil)
|
||||||
|
return &value
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func filesystemOptionDiagnostics(source Source, context *FilesystemContext) []Diagnostic {
|
||||||
|
switch source.kind {
|
||||||
|
case sourceNone:
|
||||||
|
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "NoSource cannot use filesystem context")}
|
||||||
|
case sourceFile:
|
||||||
|
if context.BaseDir != "" {
|
||||||
|
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "file sources determine their own import base directory")}
|
||||||
|
}
|
||||||
|
case sourceBytes, sourceReader:
|
||||||
|
if context.BaseDir == "" || context.Root == "" {
|
||||||
|
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem context requires base and root directories")}
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(context.BaseDir) || !filepath.IsAbs(context.Root) {
|
||||||
|
return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem base and root directories must be absolute")}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func importFilesystem(source Source, context *FilesystemContext, main parsedSource) (rootPath, basePath, mainPath string, diagnostic *Diagnostic) {
|
func importFilesystem(source Source, context *FilesystemContext, main parsedSource) (rootPath, basePath, mainPath string, diagnostic *Diagnostic) {
|
||||||
if main.source == nil || main.document == nil {
|
if main.source == nil || main.document == nil {
|
||||||
value := importDiagnostic(CodeImportInvalid, "imports require a selected configuration source", nil)
|
value := importDiagnostic(CodeImportInvalid, "imports require a selected configuration source", nil)
|
||||||
@@ -209,8 +250,26 @@ func (s *importTraversal) checkInitialLimits() *Diagnostic {
|
|||||||
func (s *importTraversal) walkDeclarations(basePath string, declarations []importDeclaration, depth int) []Diagnostic {
|
func (s *importTraversal) walkDeclarations(basePath string, declarations []importDeclaration, depth int) []Diagnostic {
|
||||||
for _, declaration := range declarations {
|
for _, declaration := range declarations {
|
||||||
if declaration.glob != "" {
|
if declaration.glob != "" {
|
||||||
|
matches, diagnostics := s.expandImportGlob(basePath, declaration)
|
||||||
|
if len(diagnostics) != 0 {
|
||||||
|
return diagnostics
|
||||||
|
}
|
||||||
|
if len(matches) == 0 {
|
||||||
|
if declaration.optional {
|
||||||
|
continue
|
||||||
|
}
|
||||||
location := declaration.location
|
location := declaration.location
|
||||||
return []Diagnostic{importDiagnostic(CodeImportUnsupported, "glob imports are not available in this release boundary", &location)}
|
return []Diagnostic{importDiagnostic(CodeImportMissing, "required import glob matched no files", &location)}
|
||||||
|
}
|
||||||
|
for _, match := range matches {
|
||||||
|
matched := declaration
|
||||||
|
matched.path = match
|
||||||
|
matched.glob = ""
|
||||||
|
if diagnostics := s.visitPath(basePath, matched, depth+1); len(diagnostics) != 0 {
|
||||||
|
return diagnostics
|
||||||
|
}
|
||||||
|
}
|
||||||
|
continue
|
||||||
}
|
}
|
||||||
if diagnostics := s.visitPath(basePath, declaration, depth+1); len(diagnostics) != 0 {
|
if diagnostics := s.visitPath(basePath, declaration, depth+1); len(diagnostics) != 0 {
|
||||||
return diagnostics
|
return diagnostics
|
||||||
@@ -518,8 +577,29 @@ func validImportPath(value string) bool {
|
|||||||
if value == "" || strings.TrimSpace(value) != value || path.IsAbs(value) || strings.HasPrefix(value, "//") {
|
if value == "" || strings.TrimSpace(value) != value || path.IsAbs(value) || strings.HasPrefix(value, "//") {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
parsed, err := url.Parse(value)
|
return !hasImportScheme(value)
|
||||||
return err == nil && parsed.Scheme == ""
|
}
|
||||||
|
|
||||||
|
func hasImportScheme(value string) bool {
|
||||||
|
colon := strings.IndexByte(value, ':')
|
||||||
|
slash := strings.IndexByte(value, '/')
|
||||||
|
if colon <= 0 || (slash >= 0 && slash < colon) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
for index := 0; index < colon; index++ {
|
||||||
|
character := value[index]
|
||||||
|
if index == 0 {
|
||||||
|
if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') &&
|
||||||
|
(character < '0' || character > '9') && character != '+' && character != '-' && character != '.' {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
func sourceInfos(source *SourceInfo) []SourceInfo {
|
func sourceInfos(source *SourceInfo) []SourceInfo {
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ type importTestConfig struct {
|
|||||||
Second string `yaml:"second"`
|
Second string `yaml:"second"`
|
||||||
} `yaml:"settings"`
|
} `yaml:"settings"`
|
||||||
Version string `yaml:"version"`
|
Version string `yaml:"version"`
|
||||||
|
Items map[string]string `yaml:"items"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLoadTraversesPathImportsDepthFirstAndComposesOrigins(t *testing.T) {
|
func TestLoadTraversesPathImportsDepthFirstAndComposesOrigins(t *testing.T) {
|
||||||
@@ -121,7 +122,6 @@ func TestLoadImportDirectiveValidationAndTemporaryGlobGate(t *testing.T) {
|
|||||||
{name: "two targets", body: "_configkit: {imports: [{path: a.yml, glob: '*.yml'}]}\n", code: CodeImportInvalid},
|
{name: "two targets", body: "_configkit: {imports: [{path: a.yml, glob: '*.yml'}]}\n", code: CodeImportInvalid},
|
||||||
{name: "unknown declaration field", body: "_configkit: {imports: [{path: a.yml, extra: true}]}\n", code: CodeImportInvalid},
|
{name: "unknown declaration field", body: "_configkit: {imports: [{path: a.yml, extra: true}]}\n", code: CodeImportInvalid},
|
||||||
{name: "optional kind", body: "_configkit: {imports: [{path: a.yml, optional: 'yes'}]}\n", code: CodeImportInvalid},
|
{name: "optional kind", body: "_configkit: {imports: [{path: a.yml, optional: 'yes'}]}\n", code: CodeImportInvalid},
|
||||||
{name: "glob gate", body: "_configkit: {imports: [{glob: '*.yml'}]}\n", code: CodeImportUnsupported},
|
|
||||||
{name: "nested directive", body: "nested: {_configkit: {}}\n", code: CodeImportInvalid},
|
{name: "nested directive", body: "nested: {_configkit: {}}\n", code: CodeImportInvalid},
|
||||||
}
|
}
|
||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
|
|||||||
20
load.go
20
load.go
@@ -17,8 +17,8 @@ type Definition[T any] struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Options contains the application definition and per-load input selections.
|
// Options contains the application definition and per-load input selections.
|
||||||
// Imports enables explicit path imports; glob imports and later precedence layers
|
// Imports enables explicit path and glob imports; later precedence layers remain
|
||||||
// remain unavailable until their owning features are implemented.
|
// unavailable until their owning features are implemented.
|
||||||
type Options[T any] struct {
|
type Options[T any] struct {
|
||||||
Definition Definition[T]
|
Definition Definition[T]
|
||||||
Imports bool
|
Imports bool
|
||||||
@@ -127,6 +127,12 @@ func Load[T any](source Source, supplied ...Options[T]) (*Result[T], error) {
|
|||||||
if parsed.source != nil {
|
if parsed.source != nil {
|
||||||
accumulator.sources = append(accumulator.sources, *parsed.source)
|
accumulator.sources = append(accumulator.sources, *parsed.source)
|
||||||
}
|
}
|
||||||
|
if options.Filesystem != nil {
|
||||||
|
if diagnostic := validateFilesystemContext(source, options.Filesystem, parsed); diagnostic != nil {
|
||||||
|
accumulator.add(*diagnostic)
|
||||||
|
return nil, accumulator.loadError()
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
if !options.Imports && inputDocument != nil {
|
if !options.Imports && inputDocument != nil {
|
||||||
directiveDiagnostics := disabledDirectiveDiagnostics(inputDocument, Path{})
|
directiveDiagnostics := disabledDirectiveDiagnostics(inputDocument, Path{})
|
||||||
@@ -210,14 +216,8 @@ func selectOptions[T any](source Source, supplied []Options[T]) (Options[T], []D
|
|||||||
if options.Profile != "" && options.Profile != "default" {
|
if options.Profile != "" && options.Profile != "default" {
|
||||||
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "configuration profiles are not available in this release boundary"))
|
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "configuration profiles are not available in this release boundary"))
|
||||||
}
|
}
|
||||||
if options.Filesystem != nil && !options.Imports {
|
if options.Filesystem != nil {
|
||||||
code := CodeOptionsUnavailable
|
diagnostics = append(diagnostics, filesystemOptionDiagnostics(source, options.Filesystem)...)
|
||||||
message := "filesystem context is not available in this release boundary"
|
|
||||||
if source.kind == sourceNone {
|
|
||||||
code = CodeOptionsInvalid
|
|
||||||
message = "NoSource cannot use filesystem context"
|
|
||||||
}
|
|
||||||
diagnostics = append(diagnostics, optionDiagnostic(code, message))
|
|
||||||
}
|
}
|
||||||
if len(options.Overrides) != 0 {
|
if len(options.Overrides) != 0 {
|
||||||
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "configuration overrides are not available in this release boundary"))
|
diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "configuration overrides are not available in this release boundary"))
|
||||||
|
|||||||
@@ -225,7 +225,7 @@ func TestLoadRejectsUnavailableOptionsAndDisabledDirectives(t *testing.T) {
|
|||||||
{name: "secret classifier", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Secret: func(Path) bool { return false }}}}, code: CodeOptionsUnavailable},
|
{name: "secret classifier", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Secret: func(Path) bool { return false }}}}, code: CodeOptionsUnavailable},
|
||||||
{name: "imports without filesystem context", source: validSource(), options: []Options[config]{{Imports: true}}, code: CodeOptionsInvalid},
|
{name: "imports without filesystem context", source: validSource(), options: []Options[config]{{Imports: true}}, code: CodeOptionsInvalid},
|
||||||
{name: "profile", source: validSource(), options: []Options[config]{{Profile: "testing"}}, code: CodeOptionsUnavailable},
|
{name: "profile", source: validSource(), options: []Options[config]{{Profile: "testing"}}, code: CodeOptionsUnavailable},
|
||||||
{name: "filesystem", source: validSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsUnavailable},
|
{name: "incomplete filesystem", source: validSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsInvalid},
|
||||||
{name: "overrides", source: validSource(), options: []Options[config]{{Overrides: []Override{{Path: Keys("name"), Value: "other", Flag: "--name"}}}}, code: CodeOptionsUnavailable},
|
{name: "overrides", source: validSource(), options: []Options[config]{{Overrides: []Override{{Path: Keys("name"), Value: "other", Flag: "--name"}}}}, code: CodeOptionsUnavailable},
|
||||||
{name: "missing version checker", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Version: &VersionRule{Path: Keys("version")}}}}, code: CodeOptionsInvalid},
|
{name: "missing version checker", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Version: &VersionRule{Path: Keys("version")}}}}, code: CodeOptionsInvalid},
|
||||||
{name: "no-source filesystem", source: NoSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsInvalid},
|
{name: "no-source filesystem", source: NoSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsInvalid},
|
||||||
|
|||||||
@@ -78,8 +78,9 @@ const (
|
|||||||
|
|
||||||
CodeImportDisabled Code = "import.disabled"
|
CodeImportDisabled Code = "import.disabled"
|
||||||
CodeImportInvalid Code = "import.invalid"
|
CodeImportInvalid Code = "import.invalid"
|
||||||
CodeImportUnsupported Code = "import.unsupported"
|
|
||||||
CodeImportMissing Code = "import.missing"
|
CodeImportMissing Code = "import.missing"
|
||||||
|
CodeImportPermission Code = "import.permission"
|
||||||
|
CodeImportRead Code = "import.read"
|
||||||
CodeImportOutsideRoot Code = "import.outside_root"
|
CodeImportOutsideRoot Code = "import.outside_root"
|
||||||
CodeImportCycle Code = "import.cycle"
|
CodeImportCycle Code = "import.cycle"
|
||||||
CodeImportRepeated Code = "import.repeated"
|
CodeImportRepeated Code = "import.repeated"
|
||||||
|
|||||||
@@ -150,6 +150,12 @@ func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead
|
|||||||
_ = file.Close()
|
_ = file.Close()
|
||||||
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceNotRegular, sourceID, "source is not a regular file")}
|
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceNotRegular, sourceID, "source is not a regular file")}
|
||||||
}
|
}
|
||||||
|
canonical, err := filepath.EvalSymlinks(info.Path)
|
||||||
|
if err != nil {
|
||||||
|
_ = file.Close()
|
||||||
|
return sourceRead{info: info}, []Diagnostic{classifySourceError(err, sourceID)}
|
||||||
|
}
|
||||||
|
info.Path = canonical
|
||||||
if stat.Size() > limit {
|
if stat.Size() > limit {
|
||||||
_ = file.Close()
|
_ = file.Close()
|
||||||
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceLimitExceeded, sourceID, "source exceeds the configured byte limit")}
|
return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceLimitExceeded, sourceID, "source exceeds the configured byte limit")}
|
||||||
|
|||||||
Reference in New Issue
Block a user