eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Settle import selector lists and exclusions

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-19 13:43:56 +00:00
parent a7ca2c59a2
commit f6bf4454b0
1 changed files with 192 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files

274
docs/roadmap/imports.md
View File

@@ -59,9 +59,9 @@ identity retain their distinct roles.
## Proposed Import Vocabulary
Use a direct-file form and a directory-search form. The latter separates the
search root from its filename filter rather than inferring traversal from a
combined path pattern:
Use direct-file, directory-search, and path-glob forms. Directory searches
separate literal search roots from filename filters; path globs additionally
support wildcard directory segments and selection at particular depths:
```yaml
configkit:
@@ -84,14 +84,16 @@ The fields have distinct jobs:
| Field | Proposed meaning |
| --- | --- |
| `path` | Import one named file, relative to the declaring file |
| `dir` | Search a directory, relative to the declaring file |
| `pattern` | Filter candidate files by filename within the searched directories |
| `path` | Import a named file or list of named files, relative to the declaring file |
| `dir` | Search a literal directory or list of literal directories, relative to the declaring file |
| `pattern` | A filename wildcard or list of wildcards; select files matching any pattern within the searched directories |
| `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file |
| `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results |
| `recursive` | Default false; include descendant directories when true |
| `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive |
| `follow` | Follow symlinks by default; false disables following within the discovery scope defined below |
| `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below |
| `required` | Fail when the declaration contributes no matching file |
| `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search |
| `destination`, `placement` | Apply the existing placement rules to each selected source |
This follows the familiar separation of traversal and filtering in Ansible's
@@ -118,10 +120,11 @@ interfaces, without adopting all their spellings, defaults, or matching modes.
shared dependency-ready scheduler defined below. This is settled independently
of the existing implementations and includes refactoring both resolvers.
OD-1 (recursion default), OD-4 (discovery boundaries and cycles), OD-5 (policy
precedence), and OD-6 (absent-import evidence) are settled in the sections below.
OD-2 and OD-3 remain open. `follow` and `hidden` are the proposed field spellings;
their applicability to a retained `glob` selector remains part of OD-3.
OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-4
(discovery boundaries and cycles), OD-5 (policy precedence), and OD-6
(absent-import evidence) are settled in the sections below. Only OD-3 remains
open. `follow` and `hidden` are the proposed field spellings; their applicability
to `glob` remains part of OD-3. Applying `exclude` to `glob` is settled.
## Decisions Resolved From Ansible
@@ -263,11 +266,71 @@ preserve each resolver's domain-specific admissibility and combination rules.
Recursion changes which directories are visited; it does not change what `*`
means. A nonrecursive search considers only files directly inside `dir`.
Recommend retaining the familiar `path.Match` wildcard grammar for filename
filters. The name `pattern` does not by itself imply regular expressions or
automatic dialect detection. Omitting `pattern` selects all filenames, with no
implicit extension restriction. Matching grammar and whether the final API
needs path-glob expressiveness remain open below.
**OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard
grammar for filename filters. Omitting `pattern` selects all filenames, with no
implicit extension restriction. Do not add regular expressions, automatic
dialect detection, or recursive `**` syntax as part of this work.
Retain `glob` as a permanent, separate selector using the existing slash-based
path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects
files in a particular directory layout without selecting all descendants.
Literal `dir` roots and basename filters do not express that same search.
This capability justifies retaining `glob` independently of compatibility;
it uses the same new absence policy as other selectors. `glob` remains a
single string; list support is being added to `path`, `dir`, and the filename
filters.
### Selector Lists And Set Operations
Accept a string or a list of strings for `path`, `dir`, and `pattern`. Normalize
a scalar to a singleton list so both forms use the same selection logic.
Listed paths and directory roots resolve relative to the declaring file.
List order does not introduce precedence or replace the shared dependency-ready
resolution schedule.
For directory roots D and filename patterns P, search the **Cartesian product**
D × P and take the **union** of the matching candidate files:
`S = ⋃ {matches(d, p) | (d, p) ∈ D × P}`
Each root uses the declaration's recursion, depth, and applicable discovery
controls. When `pattern` is omitted, select all filenames in the visited
directories instead. This describes selection semantics, not a requirement to
materialize the product or traverse a directory again for each pattern.
Overlapping patterns or directory searches within one declaration select the
same candidate once. This local set union is not global source deduplication:
existing repeated-source and alias checks still apply to distinct inclusion
routes. In particular, it does not authorize silently merging separate alias
paths to an imported source.
For `path`, `required: true` requires every listed file to be present. For `dir`,
it requires at least one selected file across the complete union after
exclusions; it does not require a match per directory, pattern, or pair. A
missing directory can therefore contribute nothing while another supplies the
required match. The equivalent `glob` requirement is at least one selected file
after exclusions. Invalid supplied configuration still fails under the existing
rules.
### Exclusions
Accept `exclude` as a string or list of strings with the same case-sensitive
basename wildcard grammar as `pattern`. Apply it to results from `path`, `dir`,
and `glob`. A candidate matching any exclusion is removed before its content is
loaded or parsed. An omitted exclusion removes nothing.
This is **set difference**: for candidate set S and its subset X matching any
exclusion, the selected set is `S ∖ X`. Exclusions win over positive matches.
They filter filenames, not directory paths, and do not prune traversal into a
directory whose name happens to match. For example, `exclude: '*.draft.yaml'`
removes matching files at every visited depth, including results of a path glob.
For searches, evaluate `required` on the selected set after exclusions. For an
explicit `path` list, exclusions filter what is loaded but do not waive the
settled requirement that every listed file be present when `required: true`.
Excluding a candidate is not permission to suppress discovery, containment,
or resource-limit errors. Exclusions are local to the declaration; they do not
prohibit another declaration from explicitly selecting the same filename.
### Recursive Discovery And Depth Limits
@@ -315,9 +378,9 @@ direct-file imports retain their existing source-access semantics. The controls
do not implicitly propagate to declarations inside imported files.
These are discovery-selection controls, not a load-wide access prohibition.
If OD-2 retains `glob` and OD-3 enables these controls on it, filtering applies
If OD-3 enables these controls on `glob`, filtering applies
to wildcard-selected entries; explicitly named path components remain explicit
access. This conditional scope rule does not settle either open decision.
access. This conditional scope rule does not settle that open decision.
When following links, detect a canonical directory repeated on the active
traversal path and fail with a cycle diagnostic. Do not silently prune the loop
@@ -352,6 +415,20 @@ Import `defaults.yaml` into the declaring source's mapping scope. Under the
proposed default, a missing file contributes nothing. Add `required: true` when
its presence is necessary. A present but invalid file still fails loading.
### Require Several Named Files
```yaml
configkit:
imports:
- path: [defaults.yaml, site.yaml]
required: true
```
Both files must be present. Listing them together shares the declaration's
options; their textual order does not establish overlay precedence. Their
contributions follow the shared dependency-ready schedule and strict
composition rules.
### Recursively Discover Mapping Fragments
```yaml
@@ -386,37 +463,62 @@ search, the declaration can also include:
These controls narrow discovery below the explicitly named root; the root
continues to resolve under the existing source-access rules.
### Require A Recursive Prompt Collection
### Search Several Prompt Collections And Filename Patterns
```yaml
configkit:
imports:
- dir: prompts
pattern: '*.prompt.yaml'
- dir: [prompts, shared-prompts]
pattern: ['*.prompt.yaml', '*.prompt.yml']
exclude: ['*.draft.prompt.yaml', '*.draft.prompt.yml']
recursive: true
required: true
destination: [prompts]
placement: sequence-item
```
This selects `prompts/welcome.prompt.yaml` and
`prompts/weather/daily/summary.prompt.yaml`, but not
`prompts/weather/helpers.yaml`. Each selected file contains one prompt mapping,
The two directories and two patterns form four Cartesian-product pairs; the
selected candidates are the union of their matches, minus excluded filenames.
This selects `prompts/welcome.prompt.yaml`,
`prompts/weather/daily/summary.prompt.yaml`, and
`shared-prompts/greeting.prompt.yml`, but not `prompts/weather/helpers.yaml`
or `shared-prompts/greeting.draft.prompt.yml`.
Each selected file contains one prompt mapping,
such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the
`prompts` sequence with identity field `id` through the existing
`Definition.SequenceCompositions` API. Distinct IDs compose into one sequence;
duplicate IDs fail strict composition. Overlays continue replacing sequences.
Here `required: true` means at least one file must match across the entire
search, not that every visited directory must contain a matching file. The
Here `required: true` means at least one file must remain across the entire
search after exclusions, not that every directory or pattern must produce a
match. One missing root is acceptable if the other supplies a selected file. The
filename convention keeps helper files out of this discovery declaration; it
does not prevent a selected file from explicitly importing a helper. Existing
repeated-source rules still apply if declarations select overlapping sources.
### Select A Directory Layout With A Glob And Exclusions
```yaml
configkit:
imports:
- glob: 'collections/*/prompts/*.yaml'
exclude: '*.draft.yaml'
required: true
destination: [prompts]
placement: sequence-item
```
This selects files such as `collections/support/prompts/welcome.yaml`, excluding
`collections/support/prompts/welcome.draft.yaml`. It does not select deeper
files such as `collections/support/prompts/internal/help.yaml`. At least one
file must remain after exclusions. The same prompt-item composition setup as
the preceding example applies. Exclusion patterns match basenames even though
the positive glob matches paths.
## Optional Imports By Default
The preferred direction is optional-by-default for both explicit files and
directory searches, with `required: true` opting into failure on absence. This
The selected direction is optional-by-default for explicit files, directory
searches, and globs, with `required: true` opting into failure on absence. This
deliberately supersedes the earlier suggestion to retain required-by-default
imports.
@@ -425,15 +527,17 @@ a different default using the settled precedence rules below.
| Situation | Proposed behavior |
| --- | --- |
| Missing path; `required` omitted or false | Contribute nothing |
| Missing search directory or no matching files; `required` omitted or false | Contribute nothing |
| Missing file, missing search directory, or zero matches with `required: true` | Fail with an import diagnostic |
| Missing `path` entry; `required` omitted or false | That entry contributes nothing; other listed files may still contribute |
| Zero selected files across a complete search; `required` omitted or false | Contribute nothing |
| Any listed `path` file missing with `required: true` | Fail with an import diagnostic |
| Zero selected files across a complete `dir` or `glob` search after exclusions with `required: true` | Fail with an import diagnostic |
| One `dir` root missing, but another supplies a selected file with `required: true` | Satisfy the declaration's presence requirement |
| Invalid pattern or malformed import declaration | Fail |
| Permission failure, unreadable matched file, or malformed YAML | Fail |
| Permission failure during discovery, unreadable selected file, or malformed YAML in a selected file | Fail |
| Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules |
An optional declaration is permission for absence, not permission to ignore a
present contribution that cannot be read or interpreted. Optionality must not
selected contribution that cannot be read or interpreted. Optionality must not
turn processing failures into apparently successful partial configuration.
This default supports installations with different optional components without
@@ -485,51 +589,38 @@ no transition release, legacy parser path, compatibility switch, or temporary
`optional` alias. Release notes should describe the changed behavior and final
syntax; this documentation does not imply migration machinery.
Whether to support path globs in the final interface remains a feature-scope
decision because they express directory selection that basename filters cannot.
Any retained form must be justified as a permanent capability, not compatibility
scaffolding. No promise is made to preserve every old declaration or use case.
Retaining `glob` is a settled capability decision, as described under filename
matching. It does not preserve the old required-by-default behavior or promise
compatibility with every old declaration or use case.
## Named Algorithms And Implementation Documentation
Use the common names below in implementation comments and maintained
documentation when the feature lands. Keep them beside the relevant logic and
in the appropriate owning documents so human readers can recognize the design
without reconstructing it from code. This does not call for new public API
terminology or duplicate contracts across documents.
| Operation | Name and intended use |
| --- | --- |
| Shared import and `extends` scheduling | **Kahn's algorithm with a stable ordering key**, using a min-heap for the ready set; the same dependency-ready topological sort serves both resolvers |
| Multiple directory roots and filename patterns | **Cartesian product** of roots and patterns, followed by **set union** of matching candidates within the declaration |
| Exclusion filtering | **Set difference** between positive candidates and those matching any exclusion |
| Directory symlink cycles | **Active-path cycle detection**, using canonical directories on the current traversal path rather than treating every previously visited directory as a cycle |
Scalar-to-singleton normalization lets scalar and list forms share selection
logic. Set operations describe membership, not resolution order; the shared
stable topological schedule supplies that order. Preserve the scheduler's
complexity explanation and distinction between cycles and blocked dependents
in its implementation documentation. Do not label implementation-specific
matching or cycle-path helpers with an algorithm name until that algorithm is
actually chosen.
## Open Decisions
Only OD-2 and OD-3 remain open. Their identifiers are preserved for continued
discussion. The recommendations below reflect the latest review against the
architecture principles; they are not settled decisions. In particular,
multiple patterns, retaining `glob`, and applying discovery controls to `glob`
remain proposals. OD-1, OD-4, OD-5, and OD-6 are settled above.
### OD-2. Matching Language And Selector Expressiveness
**Why still open:** Permissiveness does not select a matching grammar. Ansible
uses different wildcard and regex languages across its discovery interfaces.
Also, basename filters and path globs express different searches; the absence
of compatibility support does not decide which capabilities the final API needs.
**Recommendation (not yet accepted):** Use the existing case-sensitive Go
`path.Match` grammar for basename matching. Accept a scalar `pattern` or a list
of patterns; a candidate matching multiple patterns in one declaration is
selected once. This does not suppress repeated-source errors from separate
inclusion routes. Keep `dir` as a literal root and retain `glob` as a permanent,
separate selector for wildcard directory segments
and exact-depth path selection. Its justification is useful expressiveness,
not old-document compatibility. Apply the new absence policy to every selector;
retaining `glob` must not introduce a legacy behavior mode. This supports both
simple collection discovery and more selective searches, but leaves two
matching forms to explain and maintain.
**Alternative:** Offer only `path` and `dir` with a basename `pattern`. This is
a smaller final interface and removes `glob` outright, with no alias or rewrite.
Users needing selected directory names or depths would use multiple explicit
imports or application-side source selection. That is a real capability
tradeoff, acceptable if those use cases do not justify a separate selector.
**Filter alternatives:** Restrict `pattern` to a scalar, keeping the parser
smaller but requiring separate declarations for suffix unions; overlapping
selections remain subject to source-repeat errors. An explicit Go regex mode
would support richer filters but add another dialect and validation rules.
Recommend deferring regex until a concrete requirement justifies it. Regex
compatibility with Python or PowerShell is not implied. None of these choices
changes the settled all-filenames default when a filter is omitted.
Only OD-3 remains open. Its identifier is preserved for continued discussion;
the recommendations below are not settled decisions. OD-1, OD-2, OD-4, OD-5,
and OD-6 are settled above.
### OD-3. Directive Validation
@@ -539,17 +630,28 @@ boundary remains: invalid declarations fail. We still need to define their
valid shapes, rather than infer extra restrictions from that boundary.
**Recommendation (not yet accepted):** Accept nonnegative YAML integers for
`depth`; zero remains unlimited. Treat negative, null, string, boolean, and fractional values as
malformed, even when recursion is disabled. Use exactly one selector per
import: `path`, `dir`, or `glob` if selected for the final API. Keep
`depth`; zero remains unlimited. Treat negative, null, string, boolean, and
fractional values as malformed, even when recursion is disabled. Use exactly one selector per
import: `path`, `dir`, or `glob`. Keep
`pattern`, `recursive`, and `depth` specific to `dir`; allow `follow` and `hidden`
on both `dir` and `glob` if the latter is retained. Diagnose unknown fields and
options that have no meaning for the chosen selector. This makes likely mistakes
visible, but shared templates must omit inapplicable fields. It imposes no arbitrary maximum
depth. The settled combination `recursive: false` with a valid larger `depth`
on both `dir` and `glob`. `exclude` applies to all three selectors as settled
above. Diagnose unknown fields and options that have no meaning for the chosen
selector. This makes likely mistakes visible, but shared templates must omit
inapplicable fields. It imposes no arbitrary maximum depth. The settled
combination `recursive: false` with a valid larger `depth`
remains accepted and selects only direct files. The removed `optional` spelling
is not accepted under any variant.
The accepted list forms also need explicit validation rules for empty lists,
null values, empty strings, and non-string elements. Recommend allowing empty
lists as empty selections (and `exclude: []` as no exclusions), while rejecting
null values and non-string elements rather than silently coercing them. Under
that recommendation, an explicit `pattern: []` matches nothing, unlike omission;
an empty `path` list has no files whose presence is required. Empty-string
semantics should be reviewed against the existing source-path and wildcard
rules rather than introducing a blanket prohibition. These remain OD-3 choices,
not additional settled restrictions.
**Alternative:** Normalize null depth to omission and negative integers to
unlimited, and permit unused directory controls on a direct-file declaration.
That eases generated configuration but can turn an erroneous bound into a wider
@@ -558,3 +660,11 @@ chosen independently; they do not require accepting ambiguous selectors.
Another viable selector shape is to allow multiple selectors as an explicit
union, but it needs rules for constructing the graph and handling overlaps that
one-selector declarations avoid.
For list validation, another viable choice is to reject empty positive-selector
or pattern lists as likely authoring mistakes while allowing `exclude: []`.
That catches accidentally empty generated inputs, but requires conditional
omission of declarations and is less consistent with permissive empty searches.
Normalizing null to omission would ease generated configuration, but for
`pattern` it would broaden selection to all filenames. Neither alternative
changes the settled string-or-list support for nonempty valid values.