diff --git a/README.md b/README.md index 281ee40..34b0855 100644 --- a/README.md +++ b/README.md @@ -3,12 +3,13 @@ An opinionated Go library for applications that use YAML configuration files. ConfigKit is in early implementation. Its current loader accepts one strict YAML -source (or explicit defaults-only input), optionally traverses explicit path imports, -strictly composes their disjoint definitions, applies typed defaults, and can check -an application schema version. It returns detached metadata and safe diagnostics. +source (or explicit defaults-only input), optionally traverses explicit path and glob +imports, strictly composes their disjoint definitions, applies typed defaults, and +can check an application schema version. It returns detached metadata and safe +diagnostics. Applications may perform audited defaulting and normalization through a resolver, then report domain errors and warnings through read-only semantic validation. -Glob imports, profiles, overrides, secrets, and rendering are still under development. +Profiles, overrides, secrets, and rendering are still under development. ```go type Config struct { diff --git a/docs/config.md b/docs/config.md index 8fa61a8..3ebd288 100644 --- a/docs/config.md +++ b/docs/config.md @@ -22,9 +22,10 @@ File sources can observe file changes between consumptions; byte sources cannot. Each source is limited to 8 MiB. An imported load is limited to 64 MiB across all accepted sources, 1,024 accepted sources, 64 active import edges, and 1,000,000 -input nodes in aggregate. Reads request at most the remaining allowance plus one byte -so excess data is rejected rather than silently truncated. Data returned together -with a read error is not accepted as partial input. +input nodes in aggregate. Glob expansion separately permits 1,000,000 visited +directory entries, including nonmatches. Reads request at most the remaining +allowance plus one byte so excess data is rejected rather than silently truncated. +Data returned together with a read error is not accepted as partial input. ## Single-source loading @@ -50,10 +51,10 @@ contributes no source, and continues through defaults, decode, and projection. A missing file remains an error; it is never interpreted as no input. The current `Options` boundary accepts a defaults factory, `VersionRule`, resolver, -semantic validator, explicit path imports, and import filesystem context. Active -profiles, standalone filesystem context, overrides, and dynamic secret classification -return `options.unavailable` because their lifecycle stages are not implemented. -The reserved profile spelling `default` selects no active profile. `NoSource` +semantic validator, explicit path and glob imports, and filesystem context. Active +profiles, overrides, and dynamic secret classification return `options.unavailable` +because their lifecycle stages are not implemented. The reserved profile spelling +`default` selects no active profile. `NoSource` additionally rejects imports, active profiles, and filesystem context as invalid combinations. Input containing a reserved `_configkit` mapping key is rejected while imports are disabled. @@ -180,16 +181,14 @@ traversal order. Traversal order never authorizes replacement. Composition clone the resulting tree and its ordered claim records rather than mutating or aliasing an input document. -## Explicit path imports +## Explicit imports Imports are opt-in. With `Options.Imports` enabled, a document may reserve its root `_configkit` mapping for an `imports` sequence. ConfigKit removes that mapping before application decoding. Each sequence item is a mapping with exactly one nonempty string `path` or `glob` plus an optional boolean `optional`; all other shapes and -fields are `import.invalid`. `glob` is recognized but temporarily returns -`import.unsupported` until bounded glob expansion is implemented. A `_configkit` -key below the root is invalid. With imports disabled, any occurrence remains -`import.disabled`. +fields are `import.invalid`. A `_configkit` key below the root is invalid. With +imports disabled, any occurrence remains `import.disabled`. Path declarations use slash-separated relative paths. Absolute paths and network URLs are invalid. Each path resolves from its declaring file's canonical directory, @@ -199,16 +198,29 @@ permits replacement. Required missing paths fail; `optional: true` suppresses on target nonexistence, not invalid paths, containment failures, directories, permissions, parse errors, or composition errors. +Glob declarations use the slash-based Go `path.Match` grammar: `*`, `?`, character +classes, and backslash escapes are supported, while recursive `**` syntax is invalid. +Backslashes escape pattern characters rather than acting as directory separators. +ConfigKit validates the complete pattern before filesystem traversal, walks only +directories selected by preceding segments, and reads wildcard directories in +bounded batches. Every examined entry consumes the separate visit budget, including +nonmatches. Matches are sorted lexicographically by slash-relative spelling within +each declaration before depth-first source traversal. A required zero-match glob is +`import.missing`; `optional: true` suppresses only zero matches. Invalid patterns, +walk failures, matched directories, invalid documents, escapes, and repeated targets +remain errors. Final candidates are bounded by the remaining source limit before +their contents are accumulated. + The configuration root defaults to a selected file's canonical directory. A file source may supply `FilesystemContext.Root` to widen that root but may not supply -`BaseDir`. Byte and reader sources must supply both `BaseDir` and `Root` as absolute -existing directories when imports are enabled; their diagnostic labels, including -URI-looking labels, never imply filesystem paths. The base and selected file must be -inside the canonical root. ConfigKit canonicalizes symlinks, checks containment with -relative paths, and opens the canonical root-relative target through a library-owned -root handle. Contained relative or absolute symlink targets are accepted; escapes -are rejected. Canonical paths define identity, so hard links at distinct paths remain -distinct sources. +`BaseDir`. Whenever byte and reader sources supply filesystem context, they must +provide both `BaseDir` and `Root` as absolute existing directories; their diagnostic +labels, including URI-looking labels, never imply filesystem paths. The base and +selected file must be inside the canonical root. ConfigKit canonicalizes symlinks, +checks containment with relative paths, and opens the canonical root-relative target +through a library-owned root handle. Contained relative or absolute symlink targets +are accepted; escapes are rejected. Canonical paths define identity, so hard links at +distinct paths remain distinct sources. An active-stack repeat is `import.cycle` and reports the complete cycle's import sites. A previously visited canonical path is `import.repeated` and reports both diff --git a/docs/development.md b/docs/development.md index 18981c7..c76b2ab 100644 --- a/docs/development.md +++ b/docs/development.md @@ -2,7 +2,7 @@ This is the contributor entry point for ConfigKit. Read the task-specific owners below before making changes. The repository is in early implementation: its public -loader currently covers strict selected sources, explicit path-import graphs, +loader currently covers strict selected sources, explicit path/glob import graphs, composition, typed defaults, schema-version checks, recorded application resolution, semantic validation, and metadata. The [component overview](internal/overview.md) owns the implementation inventory. diff --git a/docs/internal/extraction.md b/docs/internal/extraction.md index fed25fb..6f8ad90 100644 --- a/docs/internal/extraction.md +++ b/docs/internal/extraction.md @@ -124,6 +124,16 @@ fragment version-placement enforcement. The planned glob syntax is parsed but remains explicitly unavailable for its next owning work; hard links intentionally replace Narratio's same-file rejection with path identity. +Glob-import implementation record (2026-09-06): ConfigKit retains the extracted +declaration order, import failures, composition, and identity cases while adding the +new `expandImportGlob`, `readGlobMatches`, and pattern-validation mechanisms in +`glob.go`. Focused cases cover lexical ordering, multiple declarations, overlap, +directories, hard links, contained and escaping symlinks, unmatched declarations, +malformed and recursive patterns, escaped metacharacters, character classes, +permission failures, explicit `conf.d` use, and entry/candidate budgets. This work +has no Narratio production equivalent; it uses the planned standard-library +`path.Match` and bounded `File.ReadDir` adaptation rather than its flat loader. + ### Profiles and overlay Assigned to selected-profile overlay: diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 8b3b411..da55aec 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -20,8 +20,9 @@ module and package documentation boundary. Its implemented public primitives are - strict schema-driven decoding over detached defaults in `decode.go`, including mapping/sequence semantics, null and numeric policy, safe codec errors, presence, baseline/final trees, and initial default/input origins; and -- public lifecycle coordination in `load.go`, bounded canonical path-import DFS in - `imports.go`, version-report contexts in `validation.go`, and detached +- public lifecycle coordination in `load.go`, bounded canonical import DFS in + `imports.go`, pattern-directed glob expansion in `glob.go`, version-report contexts + in `validation.go`, and detached base/default metadata assembly in `load_metadata.go`; and - recorded application defaulting and normalization in `resolution.go`, including explicitness queries, helper verification, structural mutation audit, coalesced diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index b879a69..917c2c1 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -878,7 +878,7 @@ Completion evidence (2026-09-06): tests cover nested/list-order traversal, composition conflicts, version placement, directives, optional/required paths, explicit roots, cycles, diamonds, symlinks, hard links, parse/decode attribution, file types, context rules, and reduced limit - boundaries. The [current path-import contract](../config.md#explicit-path-imports) + boundaries. The [current path-import contract](../config.md#explicit-imports) owns implemented behavior; glob expansion remains pending. Verified `go test ./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`, `go mod tidy -diff`, and `git diff --check` with Go 1.26.7 on linux/amd64. @@ -917,7 +917,36 @@ Acceptance: ## Stage 12 — Complete glob imports and filesystem option handling -Status: pending. +Status: complete. + +Completion evidence (2026-09-06): + +- Removed the temporary glob gate and added prevalidated Go `path.Match` semantics + for `*`, `?`, character classes, and escapes, with recursive unescaped `**`, URLs, + absolute paths, empty segments, and malformed classes rejected before any walk. +- Added a pattern-directed walker that resolves literal segments directly and reads + wildcard directories through positive bounded `File.ReadDir` batches. It charges + every examined entry, including nonmatches, canonicalizes and contains traversed + directories and final targets, sorts slash-relative matches, and bounds final + candidates against remaining source capacity before traversal. +- Reused the path-import DFS for every sorted match, preserving declaration ordering, + nested depth-first imports, canonical alias/repetition handling, hard-link path + identity, source IDs, composition, version placement, and all existing byte/node/ + source/depth budgets. Optional glob declarations suppress only zero matches, never + invalid syntax, walk/permission failures, matched directories, or symlink escapes. +- Completed filesystem option validation independently of import/profile selection: + file inputs reject `BaseDir` and may select a containing root; byte/readers require + existing absolute base/root pairs whenever context is supplied; `NoSource` rejects + context while the reserved `default` profile remains inactive. +- Tests cover lexical and multi-declaration ordering, deterministic repeated loads, + overlapping matches, directories, hard links, contained absolute and escaping + symlink directories, required/optional no-match behavior, invalid patterns in empty + directories, escaped metacharacters, character classes, visible permission errors, + explicit `conf.d` imports without discovery, nonmatching-entry exhaustion, candidate + bounds, and filesystem combinations. The + [current import contract](../config.md#explicit-imports) owns the completed behavior. + Verified `go test ./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`, + `go mod tidy -diff`, and `git diff --check` with Go 1.26.7 on linux/amd64. Read: feature import syntax and source-context API. diff --git a/docs/roadmap/initial.md b/docs/roadmap/initial.md index ca80fe5..17d7397 100644 --- a/docs/roadmap/initial.md +++ b/docs/roadmap/initial.md @@ -475,30 +475,9 @@ the surviving sequence, not an attempted correspondence with previous indices. ## Explicit imports and source identity -Explicit path-import traversal, source identity, containment, budgets, recursive -directives, and the temporary glob gate are implemented and owned by the -[current configuration contract](../config.md#explicit-path-imports). Bounded glob -expansion remains target behavior. Illustrative future fragment: - -```yaml -_configkit: - imports: - - glob: fragments/outputs/*.yml - optional: true -``` - -- Expand declarations in list order, sorting each glob's matches - lexicographically by slash-separated relative path, with no locale sorting. - Use the slash-based [Go `path.Match` grammar](https://pkg.go.dev/path#Match) - (`*`, `?`, character classes, and backslash escapes), with recursive `**` - syntax forbidden. Backslashes escape pattern characters, not directory separators. -- A required unmatched glob is an error. Optional suppresses only zero matches. - Invalid patterns, permission - failures, directories, and invalid matched documents remain errors. -- Glob traversal counts examined directory entries, including nonmatches, against - its separate work budget; only accepted matches contribute to sorted source order. - Expansion remains pattern-directed and must preserve the current path-import - identity, containment, repetition, and depth-first traversal rules. +Explicit path and glob imports, recursive traversal, source identity, containment, +filesystem context, and budgets are implemented and now owned by the +[current configuration contract](../config.md#explicit-imports). ## Profiles diff --git a/glob.go b/glob.go new file mode 100644 index 0000000..84a2fd6 --- /dev/null +++ b/glob.go @@ -0,0 +1,205 @@ +package configkit + +import ( + "errors" + "io" + "io/fs" + "os" + "path" + "path/filepath" + "slices" + "strings" +) + +const globReadBatch = 128 + +type globCandidate struct { + absolute string + relative string +} + +func (s *importTraversal) expandImportGlob(basePath string, declaration importDeclaration) ([]string, []Diagnostic) { + location := declaration.location + segments, valid := validGlobPattern(declaration.glob) + if !valid { + return nil, []Diagnostic{importDiagnostic(CodeImportInvalid, "import glob pattern is invalid or uses recursive ** syntax", &location)} + } + + candidates := []globCandidate{{absolute: basePath}} + for index, segment := range segments { + last := index == len(segments)-1 + literal, patterned := globLiteralSegment(segment) + var next []globCandidate + for _, candidate := range candidates { + var names []string + if patterned { + matched, diagnostics := s.readGlobMatches(candidate.absolute, segment, location) + if len(diagnostics) != 0 { + return nil, diagnostics + } + names = matched + } else { + names = []string{literal} + } + for _, name := range names { + absolute := filepath.Join(candidate.absolute, filepath.FromSlash(name)) + relative := path.Join(candidate.relative, name) + if !containedPath(s.rootPath, filepath.Clean(absolute)) { + return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal is outside the configuration root", &location)} + } + canonical, err := canonicalExistingPath(absolute) + if err != nil { + if errors.Is(err, fs.ErrNotExist) && !patterned { + continue + } + return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob match could not be canonicalized", location)} + } + if !containedPath(s.rootPath, canonical) { + return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob traversal resolves outside the configuration root", &location)} + } + if last { + next = append(next, globCandidate{absolute: canonical, relative: relative}) + if len(s.sources)+len(next) > s.limits.sources { + return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob matches exceed the remaining source-count limit", &location)} + } + continue + } + info, err := os.Stat(canonical) + if err != nil { + return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob traversal target could not be inspected", location)} + } + if info.IsDir() { + next = append(next, globCandidate{absolute: canonical, relative: relative}) + } + } + } + candidates = next + if len(candidates) == 0 { + return nil, nil + } + } + + matches := make([]string, len(candidates)) + for index, candidate := range candidates { + matches[index] = candidate.relative + } + slices.Sort(matches) + return matches, nil +} + +func (s *importTraversal) readGlobMatches(directory, pattern string, location Location) ([]string, []Diagnostic) { + canonical, err := canonicalExistingPath(directory) + if err != nil { + if errors.Is(err, fs.ErrNotExist) { + return nil, nil + } + return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be canonicalized", location)} + } + if !containedPath(s.rootPath, canonical) { + return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory resolves outside the configuration root", &location)} + } + relative, err := filepath.Rel(s.rootPath, canonical) + if err != nil || filepath.IsAbs(relative) || relative == ".." || strings.HasPrefix(relative, ".."+string(filepath.Separator)) { + return nil, []Diagnostic{importDiagnostic(CodeImportOutsideRoot, "glob directory could not be addressed inside the configuration root", &location)} + } + directoryFile, err := s.root.Open(relative) + if err != nil { + if errors.Is(err, fs.ErrNotExist) { + return nil, nil + } + return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be opened", location)} + } + + var matches []string + for { + entries, readErr := directoryFile.ReadDir(globReadBatch) + for _, entry := range entries { + s.usedEntries++ + if s.usedEntries > s.limits.entries { + _ = directoryFile.Close() + return nil, []Diagnostic{importDiagnostic(CodeImportLimitExceeded, "glob traversal exceeds the directory-entry visit limit", &location)} + } + matched, matchErr := path.Match(pattern, entry.Name()) + if matchErr != nil { + _ = directoryFile.Close() + return nil, []Diagnostic{importDiagnostic(CodeImportInvalid, "import glob pattern is invalid", &location)} + } + if matched { + matches = append(matches, entry.Name()) + } + } + if errors.Is(readErr, io.EOF) { + break + } + if readErr != nil { + _ = directoryFile.Close() + return nil, []Diagnostic{globFilesystemDiagnostic(readErr, "glob directory could not be read", location)} + } + } + if err := directoryFile.Close(); err != nil { + return nil, []Diagnostic{globFilesystemDiagnostic(err, "glob directory could not be closed", location)} + } + slices.Sort(matches) + return matches, nil +} + +func validGlobPattern(pattern string) ([]string, bool) { + if !validImportPath(pattern) { + return nil, false + } + segments := strings.Split(pattern, "/") + for _, segment := range segments { + if segment == "" || hasRecursiveStars(segment) { + return nil, false + } + if _, err := path.Match(segment, ""); err != nil { + return nil, false + } + } + return segments, true +} + +func hasRecursiveStars(segment string) bool { + inClass := false + for index := 0; index < len(segment); index++ { + switch segment[index] { + case '\\': + index++ + case '[': + inClass = true + case ']': + inClass = false + case '*': + if !inClass && index+1 < len(segment) && segment[index+1] == '*' { + return true + } + } + } + return false +} + +func globLiteralSegment(segment string) (string, bool) { + var literal strings.Builder + for index := 0; index < len(segment); index++ { + switch segment[index] { + case '*', '?', '[': + return "", true + case '\\': + index++ + if index < len(segment) { + literal.WriteByte(segment[index]) + } + default: + literal.WriteByte(segment[index]) + } + } + return literal.String(), false +} + +func globFilesystemDiagnostic(err error, message string, location Location) Diagnostic { + code := CodeImportRead + if errors.Is(err, fs.ErrPermission) { + code = CodeImportPermission + } + return importDiagnostic(code, message, &location) +} diff --git a/glob_test.go b/glob_test.go new file mode 100644 index 0000000..c13c4cc --- /dev/null +++ b/glob_test.go @@ -0,0 +1,273 @@ +package configkit + +import ( + "os" + "path/filepath" + "reflect" + "runtime" + "testing" +) + +func TestLoadGlobImportsUseLexicalDeclarationOrderAndNestedDFS(t *testing.T) { + directory := t.TempDir() + mainPath := writeImportFile(t, directory, "main.yml", `_configkit: + imports: + - glob: fragments/*.yml + - path: after.yml +items: {root: root} +`) + aPath := writeImportFile(t, directory, "fragments/a.yml", `_configkit: + imports: [{path: nested/child.yml}] +items: {alpha: alpha} +`) + childPath := writeImportFile(t, directory, "fragments/nested/child.yml", "items: {child: child}\n") + bPath := writeImportFile(t, directory, "fragments/b.yml", "items: {beta: beta}\n") + afterPath := writeImportFile(t, directory, "after.yml", "items: {after: after}\n") + + var previous []SourceInfo + for run := 0; run < 2; run++ { + result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true}) + if err != nil { + t.Fatal(err) + } + wantItems := map[string]string{"root": "root", "alpha": "alpha", "child": "child", "beta": "beta", "after": "after"} + if !reflect.DeepEqual(result.Config.Items, wantItems) { + t.Fatalf("items = %#v, want %#v", result.Config.Items, wantItems) + } + sources := result.Sources() + wantPaths := []string{ + canonicalTestPath(t, mainPath), canonicalTestPath(t, aPath), + canonicalTestPath(t, childPath), canonicalTestPath(t, bPath), + canonicalTestPath(t, afterPath), + } + gotPaths := make([]string, len(sources)) + for index, source := range sources { + gotPaths[index] = source.Path + } + if !reflect.DeepEqual(gotPaths, wantPaths) { + t.Fatalf("source paths = %#v, want %#v", gotPaths, wantPaths) + } + if run != 0 && !reflect.DeepEqual(sources, previous) { + t.Fatalf("repeated traversal changed sources: %#v != %#v", sources, previous) + } + previous = sources + } +} + +func TestLoadGlobImportsSupportEscapesClassesAndExplicitConfD(t *testing.T) { + directory := t.TempDir() + mainPath := writeImportFile(t, directory, "main.yml", `_configkit: + imports: + - glob: 'conf.d/literal\*.yml' + - glob: 'conf.d/item-[ab].yml' + - glob: 'conf.d/question?.yml' +items: {root: root} +`) + writeImportFile(t, directory, "conf.d/literal*.yml", "items: {literal: literal}\n") + writeImportFile(t, directory, "conf.d/item-b.yml", "items: {b: b}\n") + writeImportFile(t, directory, "conf.d/item-a.yml", "items: {a: a}\n") + writeImportFile(t, directory, "conf.d/question1.yml", "items: {question: question}\n") + + result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true}) + if err != nil { + t.Fatal(err) + } + if want := map[string]string{"root": "root", "literal": "literal", "a": "a", "b": "b", "question": "question"}; !reflect.DeepEqual(result.Config.Items, want) { + t.Fatalf("items = %#v, want %#v", result.Config.Items, want) + } + + ignoredPath := writeImportFile(t, directory, "conf.d/not-discovered.yml", "unknown: value\n") + _ = ignoredPath + plainPath := writeImportFile(t, directory, "plain.yml", "items: {root: only}\n") + plain, err := Load[importTestConfig](File(plainPath), Options[importTestConfig]{Imports: true}) + if err != nil || len(plain.Sources()) != 1 || plain.Config.Items["root"] != "only" { + t.Fatalf("implicit discovery result = %#v, %v", plain, err) + } +} + +func TestLoadGlobImportsRejectInvalidAndUnmatchedPatterns(t *testing.T) { + tests := []struct { + name string + pattern string + code Code + }{ + {name: "bad class", pattern: "empty/[", code: CodeImportInvalid}, + {name: "recursive star segment", pattern: "empty/**/value.yml", code: CodeImportInvalid}, + {name: "recursive adjacent stars", pattern: "empty/value**.yml", code: CodeImportInvalid}, + {name: "trailing separator", pattern: "empty/", code: CodeImportInvalid}, + {name: "absolute", pattern: "/tmp/*.yml", code: CodeImportInvalid}, + {name: "URL", pattern: "https://example.invalid/*.yml", code: CodeImportInvalid}, + {name: "missing parent escape", pattern: "../missing/*.yml", code: CodeImportOutsideRoot}, + {name: "unmatched", pattern: "empty/*.yml", code: CodeImportMissing}, + } + for _, test := range tests { + t.Run(test.name, func(t *testing.T) { + directory := t.TempDir() + if err := os.Mkdir(filepath.Join(directory, "empty"), 0o755); err != nil { + t.Fatal(err) + } + mainPath := writeGlobRoot(t, directory, test.pattern, false) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, test.code) + }) + } + + directory := t.TempDir() + if err := os.Mkdir(filepath.Join(directory, "empty"), 0o755); err != nil { + t.Fatal(err) + } + mainPath := writeGlobRoot(t, directory, "empty/*.yml", true) + result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true}) + if err != nil || len(result.Sources()) != 1 { + t.Fatalf("optional unmatched glob = %#v, %v", result, err) + } +} + +func TestLoadGlobImportsPreserveIdentityAndFileRules(t *testing.T) { + t.Run("overlapping declarations", func(t *testing.T) { + directory := t.TempDir() + writeImportFile(t, directory, "conf/value.yml", "items: {value: value}\n") + mainPath := writeImportFile(t, directory, "main.yml", `_configkit: + imports: + - glob: conf/*.yml + - glob: conf/value.* +`) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeImportRepeated) + }) + + t.Run("matched directory", func(t *testing.T) { + directory := t.TempDir() + if err := os.MkdirAll(filepath.Join(directory, "conf/matched.yml"), 0o755); err != nil { + t.Fatal(err) + } + mainPath := writeGlobRoot(t, directory, "conf/*.yml", true) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeSourceNotRegular) + }) + + t.Run("hard links stay distinct", func(t *testing.T) { + directory := t.TempDir() + one := writeImportFile(t, directory, "conf/one.yml", "items: {same: value}\n") + if err := os.Link(one, filepath.Join(directory, "conf/two.yml")); err != nil { + t.Skipf("hard links unavailable: %v", err) + } + mainPath := writeGlobRoot(t, directory, "conf/*.yml", false) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeComposeConflict) + }) +} + +func TestLoadGlobImportsConfineSymlinkDirectories(t *testing.T) { + if runtime.GOOS != "linux" && runtime.GOOS != "darwin" { + t.Skip("initial filesystem contract covers Linux and macOS") + } + t.Run("contained absolute target", func(t *testing.T) { + root := t.TempDir() + target := filepath.Join(root, "target") + writeImportFile(t, target, "value.yml", "items: {linked: linked}\n") + if err := os.Symlink(target, filepath.Join(root, "linked")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + mainPath := writeGlobRoot(t, root, "linked/*.yml", false) + result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{Imports: true}) + if err != nil || result.Config.Items["linked"] != "linked" { + t.Fatalf("contained directory symlink = %#v, %v", result, err) + } + }) + + t.Run("escaping target", func(t *testing.T) { + root := t.TempDir() + outside := t.TempDir() + writeImportFile(t, outside, "value.yml", "items: {escaped: escaped}\n") + if err := os.Symlink(outside, filepath.Join(root, "linked")); err != nil { + t.Skipf("symlinks unavailable: %v", err) + } + mainPath := writeGlobRoot(t, root, "linked/*.yml", true) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeImportOutsideRoot) + }) +} + +func TestOptionalGlobDoesNotSuppressDirectoryPermissionFailure(t *testing.T) { + directory := t.TempDir() + locked := filepath.Join(directory, "locked") + if err := os.Mkdir(locked, 0o700); err != nil { + t.Fatal(err) + } + if err := os.Chmod(locked, 0); err != nil { + t.Skipf("cannot create unreadable directory fixture: %v", err) + } + defer func() { _ = os.Chmod(locked, 0o700) }() + if opened, err := os.Open(locked); err == nil { + _ = opened.Close() + t.Skip("filesystem does not enforce the unreadable-directory fixture") + } + mainPath := writeGlobRoot(t, directory, "locked/*.yml", true) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Imports: true}, CodeImportPermission) +} + +func TestGlobTraversalBoundsNonmatchesAndAcceptedCandidates(t *testing.T) { + directory := t.TempDir() + for _, name := range []string{"one.txt", "two.txt", "three.txt"} { + writeImportFile(t, directory, "many/"+name, "ignored\n") + } + mainPath := writeGlobRoot(t, directory, "many/*.yml", true) + main := parseImportMain(t, mainPath) + limits := defaultImportLimits() + limits.entries = 2 + _, _, diagnostics := loadImportedDocuments(File(mainPath), nil, main, nil, limits) + if len(diagnostics) != 1 || diagnostics[0].Code != CodeImportLimitExceeded { + t.Fatalf("entry-budget diagnostics = %#v", diagnostics) + } + + for _, name := range []string{"a.yml", "b.yml"} { + writeImportFile(t, directory, "matches/"+name, "items: {"+name[:1]+": value}\n") + } + mainPath = writeGlobRoot(t, directory, "matches/*.yml", false) + main = parseImportMain(t, mainPath) + limits = defaultImportLimits() + limits.sources = 2 + _, _, diagnostics = loadImportedDocuments(File(mainPath), nil, main, nil, limits) + if len(diagnostics) != 1 || diagnostics[0].Code != CodeImportLimitExceeded { + t.Fatalf("candidate-budget diagnostics = %#v", diagnostics) + } +} + +func TestFilesystemContextIsValidatedWithoutImportsOrActiveProfile(t *testing.T) { + directory := t.TempDir() + mainPath := writeImportFile(t, directory, "main.yml", "items: {root: root}\n") + result, err := Load[importTestConfig](File(mainPath), Options[importTestConfig]{ + Profile: "default", Filesystem: &FilesystemContext{Root: directory}, + }) + if err != nil || result.Config.Items["root"] != "root" { + t.Fatalf("file filesystem context = %#v, %v", result, err) + } + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Filesystem: &FilesystemContext{BaseDir: directory, Root: directory}}, CodeOptionsInvalid) + requireLoadCode(t, File(mainPath), Options[importTestConfig]{Filesystem: &FilesystemContext{Root: filepath.Join(directory, "missing")}}, CodeImportInvalid) + + input := Bytes("memory.yml", []byte("items: {root: memory}\n")) + result, err = Load[importTestConfig](input, Options[importTestConfig]{Filesystem: &FilesystemContext{BaseDir: directory, Root: directory}}) + if err != nil || result.Config.Items["root"] != "memory" { + t.Fatalf("byte filesystem context = %#v, %v", result, err) + } + requireLoadCode(t, Bytes("memory.yml", []byte("{}\n")), Options[importTestConfig]{Filesystem: &FilesystemContext{BaseDir: directory}}, CodeOptionsInvalid) + defaultsOnly, err := Load[importTestConfig](NoSource(), Options[importTestConfig]{Profile: "default"}) + if err != nil || defaultsOnly == nil { + t.Fatalf("default profile with NoSource = %#v, %v", defaultsOnly, err) + } + + if runtime.GOOS == "linux" || runtime.GOOS == "darwin" { + selectedLink := filepath.Join(directory, "selected-link.yml") + if err := os.Symlink(mainPath, selectedLink); err == nil { + linked, err := Load[importTestConfig](File(selectedLink), Options[importTestConfig]{Filesystem: &FilesystemContext{Root: directory}}) + if err != nil || linked.Sources()[0].Path != canonicalTestPath(t, mainPath) { + t.Fatalf("selected symlink identity = %#v, %v", linked, err) + } + } + } +} + +func writeGlobRoot(t *testing.T, directory, pattern string, optional bool) string { + t.Helper() + optionalLine := "" + if optional { + optionalLine = "\n optional: true" + } + return writeImportFile(t, directory, "main.yml", "_configkit:\n imports:\n - glob: '"+pattern+"'"+optionalLine+"\n") +} diff --git a/imports.go b/imports.go index fce6668..14d801d 100644 --- a/imports.go +++ b/imports.go @@ -5,7 +5,6 @@ import ( "errors" "io" "io/fs" - "net/url" "os" "path" "path/filepath" @@ -19,6 +18,7 @@ import ( const ( maxImportDepth = 64 maxImportSources = 1024 + maxGlobEntries = 1_000_000 ) type importLimits struct { @@ -26,12 +26,13 @@ type importLimits struct { sources int bytes int64 nodes int + entries int } func defaultImportLimits() importLimits { return importLimits{ depth: maxImportDepth, sources: maxImportSources, - bytes: maxTotalBytes, nodes: maxDocumentNodes, + bytes: maxTotalBytes, nodes: maxDocumentNodes, entries: maxGlobEntries, } } @@ -53,13 +54,14 @@ type importTraversal struct { limits importLimits rule *VersionRule - documents []*document.Node - sources []SourceInfo - active map[string]int - visited map[string]importRoute - stack []importRoute - usedBytes int64 - usedNodes int + documents []*document.Node + sources []SourceInfo + active map[string]int + visited map[string]importRoute + stack []importRoute + usedBytes int64 + usedNodes int + usedEntries int } func loadImportedDocuments(source Source, context *FilesystemContext, main parsedSource, rule *VersionRule, limits importLimits) (*composedDocument, []SourceInfo, []Diagnostic) { @@ -69,7 +71,9 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse } root, err := os.OpenRoot(rootPath) if err != nil { - return nil, sourceInfos(main.source), []Diagnostic{importDiagnostic(CodeImportInvalid, "configuration root could not be opened", nil)} + diagnostic := globFilesystemDiagnostic(err, "configuration root could not be opened", Location{}) + diagnostic.Primary = nil + return nil, sourceInfos(main.source), []Diagnostic{diagnostic} } mainInfo := cloneSourceInfo(*main.source) @@ -107,12 +111,49 @@ func loadImportedDocuments(source Source, context *FilesystemContext, main parse return nil, cloneSourceInfos(state.sources), diagnostics } if closeErr != nil { - return nil, cloneSourceInfos(state.sources), []Diagnostic{importDiagnostic(CodeImportInvalid, "configuration root could not be closed", nil)} + return nil, cloneSourceInfos(state.sources), []Diagnostic{importDiagnostic(CodeImportRead, "configuration root could not be closed", nil)} } composed, diagnostics := composeDocuments(state.documents...) return composed, cloneSourceInfos(state.sources), diagnostics } +func validateFilesystemContext(source Source, context *FilesystemContext, main parsedSource) *Diagnostic { + rootPath, _, _, diagnostic := importFilesystem(source, context, main) + if diagnostic != nil { + return diagnostic + } + root, err := os.OpenRoot(rootPath) + if err != nil { + value := globFilesystemDiagnostic(err, "configuration root could not be opened", Location{}) + value.Primary = nil + return &value + } + if err := root.Close(); err != nil { + value := importDiagnostic(CodeImportRead, "configuration root could not be closed", nil) + return &value + } + return nil +} + +func filesystemOptionDiagnostics(source Source, context *FilesystemContext) []Diagnostic { + switch source.kind { + case sourceNone: + return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "NoSource cannot use filesystem context")} + case sourceFile: + if context.BaseDir != "" { + return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "file sources determine their own import base directory")} + } + case sourceBytes, sourceReader: + if context.BaseDir == "" || context.Root == "" { + return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem context requires base and root directories")} + } + if !filepath.IsAbs(context.BaseDir) || !filepath.IsAbs(context.Root) { + return []Diagnostic{optionDiagnostic(CodeOptionsInvalid, "byte and reader filesystem base and root directories must be absolute")} + } + } + return nil +} + func importFilesystem(source Source, context *FilesystemContext, main parsedSource) (rootPath, basePath, mainPath string, diagnostic *Diagnostic) { if main.source == nil || main.document == nil { value := importDiagnostic(CodeImportInvalid, "imports require a selected configuration source", nil) @@ -209,8 +250,26 @@ func (s *importTraversal) checkInitialLimits() *Diagnostic { func (s *importTraversal) walkDeclarations(basePath string, declarations []importDeclaration, depth int) []Diagnostic { for _, declaration := range declarations { if declaration.glob != "" { - location := declaration.location - return []Diagnostic{importDiagnostic(CodeImportUnsupported, "glob imports are not available in this release boundary", &location)} + matches, diagnostics := s.expandImportGlob(basePath, declaration) + if len(diagnostics) != 0 { + return diagnostics + } + if len(matches) == 0 { + if declaration.optional { + continue + } + location := declaration.location + return []Diagnostic{importDiagnostic(CodeImportMissing, "required import glob matched no files", &location)} + } + for _, match := range matches { + matched := declaration + matched.path = match + matched.glob = "" + if diagnostics := s.visitPath(basePath, matched, depth+1); len(diagnostics) != 0 { + return diagnostics + } + } + continue } if diagnostics := s.visitPath(basePath, declaration, depth+1); len(diagnostics) != 0 { return diagnostics @@ -518,8 +577,29 @@ func validImportPath(value string) bool { if value == "" || strings.TrimSpace(value) != value || path.IsAbs(value) || strings.HasPrefix(value, "//") { return false } - parsed, err := url.Parse(value) - return err == nil && parsed.Scheme == "" + return !hasImportScheme(value) +} + +func hasImportScheme(value string) bool { + colon := strings.IndexByte(value, ':') + slash := strings.IndexByte(value, '/') + if colon <= 0 || (slash >= 0 && slash < colon) { + return false + } + for index := 0; index < colon; index++ { + character := value[index] + if index == 0 { + if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') { + return false + } + continue + } + if (character < 'a' || character > 'z') && (character < 'A' || character > 'Z') && + (character < '0' || character > '9') && character != '+' && character != '-' && character != '.' { + return false + } + } + return true } func sourceInfos(source *SourceInfo) []SourceInfo { diff --git a/imports_test.go b/imports_test.go index 7b60db7..5327958 100644 --- a/imports_test.go +++ b/imports_test.go @@ -18,7 +18,8 @@ type importTestConfig struct { Nested string `yaml:"nested"` Second string `yaml:"second"` } `yaml:"settings"` - Version string `yaml:"version"` + Version string `yaml:"version"` + Items map[string]string `yaml:"items"` } func TestLoadTraversesPathImportsDepthFirstAndComposesOrigins(t *testing.T) { @@ -121,7 +122,6 @@ func TestLoadImportDirectiveValidationAndTemporaryGlobGate(t *testing.T) { {name: "two targets", body: "_configkit: {imports: [{path: a.yml, glob: '*.yml'}]}\n", code: CodeImportInvalid}, {name: "unknown declaration field", body: "_configkit: {imports: [{path: a.yml, extra: true}]}\n", code: CodeImportInvalid}, {name: "optional kind", body: "_configkit: {imports: [{path: a.yml, optional: 'yes'}]}\n", code: CodeImportInvalid}, - {name: "glob gate", body: "_configkit: {imports: [{glob: '*.yml'}]}\n", code: CodeImportUnsupported}, {name: "nested directive", body: "nested: {_configkit: {}}\n", code: CodeImportInvalid}, } for _, test := range tests { diff --git a/load.go b/load.go index 32fbf37..0b5144e 100644 --- a/load.go +++ b/load.go @@ -17,8 +17,8 @@ type Definition[T any] struct { } // Options contains the application definition and per-load input selections. -// Imports enables explicit path imports; glob imports and later precedence layers -// remain unavailable until their owning features are implemented. +// Imports enables explicit path and glob imports; later precedence layers remain +// unavailable until their owning features are implemented. type Options[T any] struct { Definition Definition[T] Imports bool @@ -127,6 +127,12 @@ func Load[T any](source Source, supplied ...Options[T]) (*Result[T], error) { if parsed.source != nil { accumulator.sources = append(accumulator.sources, *parsed.source) } + if options.Filesystem != nil { + if diagnostic := validateFilesystemContext(source, options.Filesystem, parsed); diagnostic != nil { + accumulator.add(*diagnostic) + return nil, accumulator.loadError() + } + } } if !options.Imports && inputDocument != nil { directiveDiagnostics := disabledDirectiveDiagnostics(inputDocument, Path{}) @@ -210,14 +216,8 @@ func selectOptions[T any](source Source, supplied []Options[T]) (Options[T], []D if options.Profile != "" && options.Profile != "default" { diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "configuration profiles are not available in this release boundary")) } - if options.Filesystem != nil && !options.Imports { - code := CodeOptionsUnavailable - message := "filesystem context is not available in this release boundary" - if source.kind == sourceNone { - code = CodeOptionsInvalid - message = "NoSource cannot use filesystem context" - } - diagnostics = append(diagnostics, optionDiagnostic(code, message)) + if options.Filesystem != nil { + diagnostics = append(diagnostics, filesystemOptionDiagnostics(source, options.Filesystem)...) } if len(options.Overrides) != 0 { diagnostics = append(diagnostics, optionDiagnostic(CodeOptionsUnavailable, "configuration overrides are not available in this release boundary")) diff --git a/load_test.go b/load_test.go index 15af2c2..efb91ad 100644 --- a/load_test.go +++ b/load_test.go @@ -225,7 +225,7 @@ func TestLoadRejectsUnavailableOptionsAndDisabledDirectives(t *testing.T) { {name: "secret classifier", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Secret: func(Path) bool { return false }}}}, code: CodeOptionsUnavailable}, {name: "imports without filesystem context", source: validSource(), options: []Options[config]{{Imports: true}}, code: CodeOptionsInvalid}, {name: "profile", source: validSource(), options: []Options[config]{{Profile: "testing"}}, code: CodeOptionsUnavailable}, - {name: "filesystem", source: validSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsUnavailable}, + {name: "incomplete filesystem", source: validSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsInvalid}, {name: "overrides", source: validSource(), options: []Options[config]{{Overrides: []Override{{Path: Keys("name"), Value: "other", Flag: "--name"}}}}, code: CodeOptionsUnavailable}, {name: "missing version checker", source: validSource(), options: []Options[config]{{Definition: Definition[config]{Version: &VersionRule{Path: Keys("version")}}}}, code: CodeOptionsInvalid}, {name: "no-source filesystem", source: NoSource(), options: []Options[config]{{Filesystem: &FilesystemContext{}}}, code: CodeOptionsInvalid}, diff --git a/records.go b/records.go index 0ddabcb..9582d2e 100644 --- a/records.go +++ b/records.go @@ -78,8 +78,9 @@ const ( CodeImportDisabled Code = "import.disabled" CodeImportInvalid Code = "import.invalid" - CodeImportUnsupported Code = "import.unsupported" CodeImportMissing Code = "import.missing" + CodeImportPermission Code = "import.permission" + CodeImportRead Code = "import.read" CodeImportOutsideRoot Code = "import.outside_root" CodeImportCycle Code = "import.cycle" CodeImportRepeated Code = "import.repeated" diff --git a/source.go b/source.go index c786593..29d43fc 100644 --- a/source.go +++ b/source.go @@ -150,6 +150,12 @@ func readSource(source Source, sourceID string, limits sourceLimits) (sourceRead _ = file.Close() return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceNotRegular, sourceID, "source is not a regular file")} } + canonical, err := filepath.EvalSymlinks(info.Path) + if err != nil { + _ = file.Close() + return sourceRead{info: info}, []Diagnostic{classifySourceError(err, sourceID)} + } + info.Path = canonical if stat.Size() > limit { _ = file.Close() return sourceRead{info: info}, []Diagnostic{sourceDiagnostic(CodeSourceLimitExceeded, sourceID, "source exceeds the configured byte limit")}