From f6bf4454b006fbcc2fb2a1b842fb5e23d45c404e Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 19 Oct 2026 13:43:56 +0000 Subject: [PATCH] Settle import selector lists and exclusions --- docs/roadmap/imports.md | 274 ++++++++++++++++++++++++++++------------ 1 file changed, 192 insertions(+), 82 deletions(-) diff --git a/docs/roadmap/imports.md b/docs/roadmap/imports.md index 83f77a9..8d53166 100644 --- a/docs/roadmap/imports.md +++ b/docs/roadmap/imports.md @@ -59,9 +59,9 @@ identity retain their distinct roles. ## Proposed Import Vocabulary -Use a direct-file form and a directory-search form. The latter separates the -search root from its filename filter rather than inferring traversal from a -combined path pattern: +Use direct-file, directory-search, and path-glob forms. Directory searches +separate literal search roots from filename filters; path globs additionally +support wildcard directory segments and selection at particular depths: ```yaml configkit: @@ -84,14 +84,16 @@ The fields have distinct jobs: | Field | Proposed meaning | | --- | --- | -| `path` | Import one named file, relative to the declaring file | -| `dir` | Search a directory, relative to the declaring file | -| `pattern` | Filter candidate files by filename within the searched directories | +| `path` | Import a named file or list of named files, relative to the declaring file | +| `dir` | Search a literal directory or list of literal directories, relative to the declaring file | +| `pattern` | A filename wildcard or list of wildcards; select files matching any pattern within the searched directories | +| `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file | +| `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results | | `recursive` | Default false; include descendant directories when true | | `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive | | `follow` | Follow symlinks by default; false disables following within the discovery scope defined below | | `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below | -| `required` | Fail when the declaration contributes no matching file | +| `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search | | `destination`, `placement` | Apply the existing placement rules to each selected source | This follows the familiar separation of traversal and filtering in Ansible's @@ -118,10 +120,11 @@ interfaces, without adopting all their spellings, defaults, or matching modes. shared dependency-ready scheduler defined below. This is settled independently of the existing implementations and includes refactoring both resolvers. -OD-1 (recursion default), OD-4 (discovery boundaries and cycles), OD-5 (policy -precedence), and OD-6 (absent-import evidence) are settled in the sections below. -OD-2 and OD-3 remain open. `follow` and `hidden` are the proposed field spellings; -their applicability to a retained `glob` selector remains part of OD-3. +OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-4 +(discovery boundaries and cycles), OD-5 (policy precedence), and OD-6 +(absent-import evidence) are settled in the sections below. Only OD-3 remains +open. `follow` and `hidden` are the proposed field spellings; their applicability +to `glob` remains part of OD-3. Applying `exclude` to `glob` is settled. ## Decisions Resolved From Ansible @@ -263,11 +266,71 @@ preserve each resolver's domain-specific admissibility and combination rules. Recursion changes which directories are visited; it does not change what `*` means. A nonrecursive search considers only files directly inside `dir`. -Recommend retaining the familiar `path.Match` wildcard grammar for filename -filters. The name `pattern` does not by itself imply regular expressions or -automatic dialect detection. Omitting `pattern` selects all filenames, with no -implicit extension restriction. Matching grammar and whether the final API -needs path-glob expressiveness remain open below. +**OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard +grammar for filename filters. Omitting `pattern` selects all filenames, with no +implicit extension restriction. Do not add regular expressions, automatic +dialect detection, or recursive `**` syntax as part of this work. + +Retain `glob` as a permanent, separate selector using the existing slash-based +path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects +files in a particular directory layout without selecting all descendants. +Literal `dir` roots and basename filters do not express that same search. +This capability justifies retaining `glob` independently of compatibility; +it uses the same new absence policy as other selectors. `glob` remains a +single string; list support is being added to `path`, `dir`, and the filename +filters. + +### Selector Lists And Set Operations + +Accept a string or a list of strings for `path`, `dir`, and `pattern`. Normalize +a scalar to a singleton list so both forms use the same selection logic. +Listed paths and directory roots resolve relative to the declaring file. +List order does not introduce precedence or replace the shared dependency-ready +resolution schedule. + +For directory roots D and filename patterns P, search the **Cartesian product** +D × P and take the **union** of the matching candidate files: + +`S = ⋃ {matches(d, p) | (d, p) ∈ D × P}` + +Each root uses the declaration's recursion, depth, and applicable discovery +controls. When `pattern` is omitted, select all filenames in the visited +directories instead. This describes selection semantics, not a requirement to +materialize the product or traverse a directory again for each pattern. + +Overlapping patterns or directory searches within one declaration select the +same candidate once. This local set union is not global source deduplication: +existing repeated-source and alias checks still apply to distinct inclusion +routes. In particular, it does not authorize silently merging separate alias +paths to an imported source. + +For `path`, `required: true` requires every listed file to be present. For `dir`, +it requires at least one selected file across the complete union after +exclusions; it does not require a match per directory, pattern, or pair. A +missing directory can therefore contribute nothing while another supplies the +required match. The equivalent `glob` requirement is at least one selected file +after exclusions. Invalid supplied configuration still fails under the existing +rules. + +### Exclusions + +Accept `exclude` as a string or list of strings with the same case-sensitive +basename wildcard grammar as `pattern`. Apply it to results from `path`, `dir`, +and `glob`. A candidate matching any exclusion is removed before its content is +loaded or parsed. An omitted exclusion removes nothing. + +This is **set difference**: for candidate set S and its subset X matching any +exclusion, the selected set is `S ∖ X`. Exclusions win over positive matches. +They filter filenames, not directory paths, and do not prune traversal into a +directory whose name happens to match. For example, `exclude: '*.draft.yaml'` +removes matching files at every visited depth, including results of a path glob. + +For searches, evaluate `required` on the selected set after exclusions. For an +explicit `path` list, exclusions filter what is loaded but do not waive the +settled requirement that every listed file be present when `required: true`. +Excluding a candidate is not permission to suppress discovery, containment, +or resource-limit errors. Exclusions are local to the declaration; they do not +prohibit another declaration from explicitly selecting the same filename. ### Recursive Discovery And Depth Limits @@ -315,9 +378,9 @@ direct-file imports retain their existing source-access semantics. The controls do not implicitly propagate to declarations inside imported files. These are discovery-selection controls, not a load-wide access prohibition. -If OD-2 retains `glob` and OD-3 enables these controls on it, filtering applies +If OD-3 enables these controls on `glob`, filtering applies to wildcard-selected entries; explicitly named path components remain explicit -access. This conditional scope rule does not settle either open decision. +access. This conditional scope rule does not settle that open decision. When following links, detect a canonical directory repeated on the active traversal path and fail with a cycle diagnostic. Do not silently prune the loop @@ -352,6 +415,20 @@ Import `defaults.yaml` into the declaring source's mapping scope. Under the proposed default, a missing file contributes nothing. Add `required: true` when its presence is necessary. A present but invalid file still fails loading. +### Require Several Named Files + +```yaml +configkit: + imports: + - path: [defaults.yaml, site.yaml] + required: true +``` + +Both files must be present. Listing them together shares the declaration's +options; their textual order does not establish overlay precedence. Their +contributions follow the shared dependency-ready schedule and strict +composition rules. + ### Recursively Discover Mapping Fragments ```yaml @@ -386,37 +463,62 @@ search, the declaration can also include: These controls narrow discovery below the explicitly named root; the root continues to resolve under the existing source-access rules. -### Require A Recursive Prompt Collection +### Search Several Prompt Collections And Filename Patterns ```yaml configkit: imports: - - dir: prompts - pattern: '*.prompt.yaml' + - dir: [prompts, shared-prompts] + pattern: ['*.prompt.yaml', '*.prompt.yml'] + exclude: ['*.draft.prompt.yaml', '*.draft.prompt.yml'] recursive: true required: true destination: [prompts] placement: sequence-item ``` -This selects `prompts/welcome.prompt.yaml` and -`prompts/weather/daily/summary.prompt.yaml`, but not -`prompts/weather/helpers.yaml`. Each selected file contains one prompt mapping, +The two directories and two patterns form four Cartesian-product pairs; the +selected candidates are the union of their matches, minus excluded filenames. +This selects `prompts/welcome.prompt.yaml`, +`prompts/weather/daily/summary.prompt.yaml`, and +`shared-prompts/greeting.prompt.yml`, but not `prompts/weather/helpers.yaml` +or `shared-prompts/greeting.draft.prompt.yml`. +Each selected file contains one prompt mapping, such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the `prompts` sequence with identity field `id` through the existing `Definition.SequenceCompositions` API. Distinct IDs compose into one sequence; duplicate IDs fail strict composition. Overlays continue replacing sequences. -Here `required: true` means at least one file must match across the entire -search, not that every visited directory must contain a matching file. The +Here `required: true` means at least one file must remain across the entire +search after exclusions, not that every directory or pattern must produce a +match. One missing root is acceptable if the other supplies a selected file. The filename convention keeps helper files out of this discovery declaration; it does not prevent a selected file from explicitly importing a helper. Existing repeated-source rules still apply if declarations select overlapping sources. +### Select A Directory Layout With A Glob And Exclusions + +```yaml +configkit: + imports: + - glob: 'collections/*/prompts/*.yaml' + exclude: '*.draft.yaml' + required: true + destination: [prompts] + placement: sequence-item +``` + +This selects files such as `collections/support/prompts/welcome.yaml`, excluding +`collections/support/prompts/welcome.draft.yaml`. It does not select deeper +files such as `collections/support/prompts/internal/help.yaml`. At least one +file must remain after exclusions. The same prompt-item composition setup as +the preceding example applies. Exclusion patterns match basenames even though +the positive glob matches paths. + ## Optional Imports By Default -The preferred direction is optional-by-default for both explicit files and -directory searches, with `required: true` opting into failure on absence. This +The selected direction is optional-by-default for explicit files, directory +searches, and globs, with `required: true` opting into failure on absence. This deliberately supersedes the earlier suggestion to retain required-by-default imports. @@ -425,15 +527,17 @@ a different default using the settled precedence rules below. | Situation | Proposed behavior | | --- | --- | -| Missing path; `required` omitted or false | Contribute nothing | -| Missing search directory or no matching files; `required` omitted or false | Contribute nothing | -| Missing file, missing search directory, or zero matches with `required: true` | Fail with an import diagnostic | +| Missing `path` entry; `required` omitted or false | That entry contributes nothing; other listed files may still contribute | +| Zero selected files across a complete search; `required` omitted or false | Contribute nothing | +| Any listed `path` file missing with `required: true` | Fail with an import diagnostic | +| Zero selected files across a complete `dir` or `glob` search after exclusions with `required: true` | Fail with an import diagnostic | +| One `dir` root missing, but another supplies a selected file with `required: true` | Satisfy the declaration's presence requirement | | Invalid pattern or malformed import declaration | Fail | -| Permission failure, unreadable matched file, or malformed YAML | Fail | +| Permission failure during discovery, unreadable selected file, or malformed YAML in a selected file | Fail | | Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules | An optional declaration is permission for absence, not permission to ignore a -present contribution that cannot be read or interpreted. Optionality must not +selected contribution that cannot be read or interpreted. Optionality must not turn processing failures into apparently successful partial configuration. This default supports installations with different optional components without @@ -485,51 +589,38 @@ no transition release, legacy parser path, compatibility switch, or temporary `optional` alias. Release notes should describe the changed behavior and final syntax; this documentation does not imply migration machinery. -Whether to support path globs in the final interface remains a feature-scope -decision because they express directory selection that basename filters cannot. -Any retained form must be justified as a permanent capability, not compatibility -scaffolding. No promise is made to preserve every old declaration or use case. +Retaining `glob` is a settled capability decision, as described under filename +matching. It does not preserve the old required-by-default behavior or promise +compatibility with every old declaration or use case. + +## Named Algorithms And Implementation Documentation + +Use the common names below in implementation comments and maintained +documentation when the feature lands. Keep them beside the relevant logic and +in the appropriate owning documents so human readers can recognize the design +without reconstructing it from code. This does not call for new public API +terminology or duplicate contracts across documents. + +| Operation | Name and intended use | +| --- | --- | +| Shared import and `extends` scheduling | **Kahn's algorithm with a stable ordering key**, using a min-heap for the ready set; the same dependency-ready topological sort serves both resolvers | +| Multiple directory roots and filename patterns | **Cartesian product** of roots and patterns, followed by **set union** of matching candidates within the declaration | +| Exclusion filtering | **Set difference** between positive candidates and those matching any exclusion | +| Directory symlink cycles | **Active-path cycle detection**, using canonical directories on the current traversal path rather than treating every previously visited directory as a cycle | + +Scalar-to-singleton normalization lets scalar and list forms share selection +logic. Set operations describe membership, not resolution order; the shared +stable topological schedule supplies that order. Preserve the scheduler's +complexity explanation and distinction between cycles and blocked dependents +in its implementation documentation. Do not label implementation-specific +matching or cycle-path helpers with an algorithm name until that algorithm is +actually chosen. ## Open Decisions -Only OD-2 and OD-3 remain open. Their identifiers are preserved for continued -discussion. The recommendations below reflect the latest review against the -architecture principles; they are not settled decisions. In particular, -multiple patterns, retaining `glob`, and applying discovery controls to `glob` -remain proposals. OD-1, OD-4, OD-5, and OD-6 are settled above. - -### OD-2. Matching Language And Selector Expressiveness - -**Why still open:** Permissiveness does not select a matching grammar. Ansible -uses different wildcard and regex languages across its discovery interfaces. -Also, basename filters and path globs express different searches; the absence -of compatibility support does not decide which capabilities the final API needs. - -**Recommendation (not yet accepted):** Use the existing case-sensitive Go -`path.Match` grammar for basename matching. Accept a scalar `pattern` or a list -of patterns; a candidate matching multiple patterns in one declaration is -selected once. This does not suppress repeated-source errors from separate -inclusion routes. Keep `dir` as a literal root and retain `glob` as a permanent, -separate selector for wildcard directory segments -and exact-depth path selection. Its justification is useful expressiveness, -not old-document compatibility. Apply the new absence policy to every selector; -retaining `glob` must not introduce a legacy behavior mode. This supports both -simple collection discovery and more selective searches, but leaves two -matching forms to explain and maintain. - -**Alternative:** Offer only `path` and `dir` with a basename `pattern`. This is -a smaller final interface and removes `glob` outright, with no alias or rewrite. -Users needing selected directory names or depths would use multiple explicit -imports or application-side source selection. That is a real capability -tradeoff, acceptable if those use cases do not justify a separate selector. - -**Filter alternatives:** Restrict `pattern` to a scalar, keeping the parser -smaller but requiring separate declarations for suffix unions; overlapping -selections remain subject to source-repeat errors. An explicit Go regex mode -would support richer filters but add another dialect and validation rules. -Recommend deferring regex until a concrete requirement justifies it. Regex -compatibility with Python or PowerShell is not implied. None of these choices -changes the settled all-filenames default when a filter is omitted. +Only OD-3 remains open. Its identifier is preserved for continued discussion; +the recommendations below are not settled decisions. OD-1, OD-2, OD-4, OD-5, +and OD-6 are settled above. ### OD-3. Directive Validation @@ -539,17 +630,28 @@ boundary remains: invalid declarations fail. We still need to define their valid shapes, rather than infer extra restrictions from that boundary. **Recommendation (not yet accepted):** Accept nonnegative YAML integers for -`depth`; zero remains unlimited. Treat negative, null, string, boolean, and fractional values as -malformed, even when recursion is disabled. Use exactly one selector per -import: `path`, `dir`, or `glob` if selected for the final API. Keep +`depth`; zero remains unlimited. Treat negative, null, string, boolean, and +fractional values as malformed, even when recursion is disabled. Use exactly one selector per +import: `path`, `dir`, or `glob`. Keep `pattern`, `recursive`, and `depth` specific to `dir`; allow `follow` and `hidden` -on both `dir` and `glob` if the latter is retained. Diagnose unknown fields and -options that have no meaning for the chosen selector. This makes likely mistakes -visible, but shared templates must omit inapplicable fields. It imposes no arbitrary maximum -depth. The settled combination `recursive: false` with a valid larger `depth` +on both `dir` and `glob`. `exclude` applies to all three selectors as settled +above. Diagnose unknown fields and options that have no meaning for the chosen +selector. This makes likely mistakes visible, but shared templates must omit +inapplicable fields. It imposes no arbitrary maximum depth. The settled +combination `recursive: false` with a valid larger `depth` remains accepted and selects only direct files. The removed `optional` spelling is not accepted under any variant. +The accepted list forms also need explicit validation rules for empty lists, +null values, empty strings, and non-string elements. Recommend allowing empty +lists as empty selections (and `exclude: []` as no exclusions), while rejecting +null values and non-string elements rather than silently coercing them. Under +that recommendation, an explicit `pattern: []` matches nothing, unlike omission; +an empty `path` list has no files whose presence is required. Empty-string +semantics should be reviewed against the existing source-path and wildcard +rules rather than introducing a blanket prohibition. These remain OD-3 choices, +not additional settled restrictions. + **Alternative:** Normalize null depth to omission and negative integers to unlimited, and permit unused directory controls on a direct-file declaration. That eases generated configuration but can turn an erroneous bound into a wider @@ -558,3 +660,11 @@ chosen independently; they do not require accepting ambiguous selectors. Another viable selector shape is to allow multiple selectors as an explicit union, but it needs rules for constructing the graph and handling overlaps that one-selector declarations avoid. + +For list validation, another viable choice is to reject empty positive-selector +or pattern lists as likely authoring mistakes while allowing `exclude: []`. +That catches accidentally empty generated inputs, but requires conditional +omission of declarations and is less consistent with permissive empty searches. +Normalizing null to omission would ease generated configuration, but for +`pattern` it would broaden selection to all filenames. Neither alternative +changes the settled string-or-list support for nonempty valid values.