eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Settle import validation and require recursion for depth

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-19 14:05:18 +00:00
parent f6bf4454b0
commit c9f9c98fbe
1 changed files with 128 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files

215
docs/roadmap/imports.md
View File

@@ -5,7 +5,7 @@ direction for discussion, not an implemented contract or an implementation plan.
The selected direction separates directory searches, filename matching,
recursion, and absence handling. Optional-by-default imports are the intended
direction. The consumer-directed and Ansible-aligned decisions below are settled;
remaining choices have recommendations and alternatives under Open Decisions.
API spelling and implementation details remain for implementation planning.
All new syntax below is illustrative and unimplemented.
## Guiding Philosophy
@@ -86,11 +86,11 @@ The fields have distinct jobs:
| --- | --- |
| `path` | Import a named file or list of named files, relative to the declaring file |
| `dir` | Search a literal directory or list of literal directories, relative to the declaring file |
| `pattern` | A filename wildcard or list of wildcards; select files matching any pattern within the searched directories |
| `pattern` | A filename wildcard or list of wildcards; select files matching any effective pattern, using the default precedence below |
| `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file |
| `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results |
| `recursive` | Default false; include descendant directories when true |
| `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive |
| `recursive` | Library default false, overridable by an application default; include descendant directories when effectively true |
| `depth` | Bound directory-search depth; supplying it requires effective recursion to be enabled; 1 means direct files, 0 or omission means unlimited when recursive |
| `follow` | Follow symlinks by default; false disables following within the discovery scope defined below |
| `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below |
| `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search |
@@ -108,9 +108,10 @@ interfaces, without adopting all their spellings, defaults, or matching modes.
it as an alias or normalize old declarations at runtime.
- **Optional contributions by default.** `required: true` opts into failure on
absence. Supplied invalid configuration continues to fail.
- **No implicit filename restriction.** Omitting `pattern` selects all
filenames. Authors can narrow selection explicitly. There is no default YAML
extension list; extensionless sources remain discoverable.
- **No library-imposed filename restriction.** Omitting `pattern` selects all
filenames unless the application supplies a default pattern. Authors can
override that default explicitly. There is no library default YAML extension
list; extensionless sources remain discoverable.
- **Follow symlinks by default, with an opt-out.** Provide `follow: false` for
callers who want discovery without symlink following. Applicable containment
and filesystem contracts remain in force.
@@ -120,11 +121,12 @@ interfaces, without adopting all their spellings, defaults, or matching modes.
shared dependency-ready scheduler defined below. This is settled independently
of the existing implementations and includes refactoring both resolvers.
OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-4
(discovery boundaries and cycles), OD-5 (policy precedence), and OD-6
(absent-import evidence) are settled in the sections below. Only OD-3 remains
open. `follow` and `hidden` are the proposed field spellings; their applicability
to `glob` remains part of OD-3. Applying `exclude` to `glob` is settled.
OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-3
(directive validation), OD-4 (discovery boundaries and cycles), OD-5 (policy
precedence), and OD-6 (absent-import evidence) are settled in the sections below.
Supplying `depth` requires effective recursion to be enabled, including when
enabled through an application default. This supersedes the earlier direction
to accept but ignore depth when recursion is disabled.
## Decisions Resolved From Ansible
@@ -153,13 +155,8 @@ The following choices are settled:
and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1).
The latter two document unlimited omission more clearly than explicit zero;
this choice uses source-confirmed behavior, not a claimed universal documented
Ansible guarantee. Negative values remain a separate open decision.
- **Disabling recursion takes precedence over depth.** With `recursive: false`,
search only direct files, even if a valid larger or unlimited depth is supplied.
Depth does not enable recursion. Both
[find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters)
and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters)
explicitly use this rule where a separate recursion switch exists.
Ansible guarantee. Negative values are invalid under the settled directive
validation rules below.
- **Directory searches select files, not directory documents.** Directories are
traversal containers, even when their names match `pattern`. This agrees with
the default file selection in the find modules and the file-oriented
@@ -267,9 +264,10 @@ Recursion changes which directories are visited; it does not change what `*`
means. A nonrecursive search considers only files directly inside `dir`.
**OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard
grammar for filename filters. Omitting `pattern` selects all filenames, with no
implicit extension restriction. Do not add regular expressions, automatic
dialect detection, or recursive `**` syntax as part of this work.
grammar for filename filters. When neither the declaration nor the application
supplies `pattern`, select all filenames, with no implicit extension restriction.
Do not add regular expressions, automatic dialect detection, or recursive `**`
syntax as part of this work.
Retain `glob` as a permanent, separate selector using the existing slash-based
path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects
@@ -294,12 +292,15 @@ D × P and take the **union** of the matching candidate files:
`S = ⋃ {matches(d, p) | (d, p) ∈ D × P}`
Each root uses the declaration's recursion, depth, and applicable discovery
controls. When `pattern` is omitted, select all filenames in the visited
directories instead. This describes selection semantics, not a requirement to
controls. Resolve an omitted `pattern` from the application default first;
if neither supplies patterns, select all filenames in the visited directories
instead. This describes selection semantics, not a requirement to
materialize the product or traverse a directory again for each pattern.
Overlapping patterns or directory searches within one declaration select the
same candidate once. This local set union is not global source deduplication:
same candidate once. Repeated identical entries in one `path` list likewise
select that candidate once; duplicate patterns or exclusions have no additional
effect. This local deduplication is not global source deduplication:
existing repeated-source and alias checks still apply to distinct inclusion
routes. In particular, it does not authorize silently merging separate alias
paths to an imported source.
@@ -334,7 +335,8 @@ prohibit another declaration from explicitly selecting the same filename.
### Recursive Discovery And Depth Limits
Use `recursive: true` for recursive discovery, optionally bounded by `depth`.
Enable recursion explicitly or through an application default for recursive
discovery, optionally bounded by `depth`.
The motivation is to let authors organize definitions in subdirectories without
updating import declarations for every depth. Directory organization should not
change resource identity or the effective configuration destination.
@@ -342,10 +344,10 @@ change resource identity or the effective configuration destination.
The selected model uses an explicit search root and recursion flag, rather than
introducing recursive `**` syntax or inferring a root from a wildcard prefix.
**OD-1: settled.** Omitting `recursive` means false: search only direct files.
Authors explicitly opt into descendants. This makes the search scope visible
while keeping recursion independent of the permissive defaults for absence,
symlinks, and hidden entries.
**OD-1: settled.** The library default for `recursive` is false: search only
direct files. An application may supply a different default; an explicit
declaration value overrides it, including `recursive: false`. Recursion remains
independent of the permissive defaults for absence, symlinks, and hidden entries.
The settled depth convention is:
@@ -353,11 +355,21 @@ The settled depth convention is:
- Each additional directory increases depth by one.
- Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit
when recursion is enabled; existing loader resource limits still apply.
- `recursive: false` selects only direct files regardless of a valid depth value.
- Supplying `depth` requires effective `recursive: true`. If recursion is false
after resolving declaration, application, and library defaults, any supplied
`depth` is an error, including `0` or `1`. Depth never enables recursion itself.
- With recursion disabled and `depth` omitted, select only direct files.
- Depth controls discovery only, not namespaces, identities, or destinations.
Remaining directive validation choices are listed below. Recursive discovery
must fit the existing filesystem containment, repeated-source, and load-budget
Resolve the effective recursion value before checking depth's applicability.
For example, `depth: 2` with no explicit `recursive` is valid if the application
default enables recursion, but invalid with the library default. Explicit
`recursive: false` overrides an application default of true and makes any
supplied depth invalid. Depth must also have the valid type and range defined
under directive validation; enabling recursion does not excuse malformed depth.
Recursive discovery must fit the existing filesystem containment,
repeated-source, and load-budget
contracts rather than silently changing them. In particular, examined entries,
including nonmatches, still consume the visit budget; a small number of matches
does not authorize an unbounded filesystem scan. This is an existing ConfigKit
@@ -378,9 +390,8 @@ direct-file imports retain their existing source-access semantics. The controls
do not implicitly propagate to declarations inside imported files.
These are discovery-selection controls, not a load-wide access prohibition.
If OD-3 enables these controls on `glob`, filtering applies
to wildcard-selected entries; explicitly named path components remain explicit
access. This conditional scope rule does not settle that open decision.
On `glob`, filtering applies to wildcard-selected entries; explicitly named
path components remain explicit access.
When following links, detect a canonical directory repeated on the active
traversal path and fail with a cycle diagnostic. Do not silently prune the loop
@@ -449,8 +460,8 @@ keys. Missing directories or no matching files contribute nothing.
Adding `depth: 2` limits discovery to direct files and files one directory below
`services`: it still selects `services/regional/worker.yaml`, but not
`services/regional/east/worker.yaml`. With `recursive: false`, only direct files
are selected, even if `depth: 2` remains present.
`services/regional/east/worker.yaml`. Switching to `recursive: false` requires
omitting `depth`; leaving `depth: 2` present is a declaration error.
Hidden entries and contained symlinks participate by default. To narrow the
search, the declaration can also include:
@@ -550,12 +561,22 @@ contribution.
## Who Chooses Strictness?
**OD-5: settled.** Provide load-level defaults for `required`, `follow`, and
`hidden`. Resolve each applicable field independently using this precedence:
**OD-5: settled.** Provide load-level defaults for `required`, `follow`, `hidden`,
`recursive`, and `pattern`. Resolve each applicable field independently using
this precedence:
1. Explicit value on the import declaration.
2. Application-selected default, when supplied.
3. Library default: false for `required`, true for `follow` and `hidden`.
3. Library default: false for `required` and `recursive`, true for `follow` and
`hidden`, and no filename restriction for `pattern`.
An explicit `pattern` replaces the application default rather than combining
with it. In particular, `pattern: []` selects nothing, even when the application
supplies patterns. An application can default to `['*.yml', '*.yaml']`; without
either an application default or an explicit pattern, a directory search
attempts to load all files in its search scope. Application defaults apply only
to selectors that support the field; they do not add inapplicable options to
other selectors.
Apply this rule to transitive declarations too. A declaration does not inherit
these field values from its importing parent. Applications may expose the
@@ -566,7 +587,7 @@ Explicit declarations remain authoritative, permitting intentional exceptions
to stricter application defaults. Defaults are not enforcement controls. Do not
add options that override explicit declarations as part of this work without a
separate concrete requirement and scope decision. The fields' applicability to
each selector remains part of OD-3.
each selector is defined under directive validation.
## Evidence For Absent Optional Imports
@@ -616,55 +637,75 @@ in its implementation documentation. Do not label implementation-specific
matching or cycle-path helpers with an algorithm name until that algorithm is
actually chosen.
## Open Decisions
## Directive Validation
Only OD-3 remains open. Its identifier is preserved for continued discussion;
the recommendations below are not settled decisions. OD-1, OD-2, OD-4, OD-5,
and OD-6 are settled above.
**OD-3: settled.** Validate declaration shapes and values independently of
whether discovery selects any files. Invalid declarations fail even when a
search root is missing or a list is empty. This does not require opening or
parsing excluded files or establish a new first-error ordering guarantee.
### OD-3. Directive Validation
### Selectors And Applicable Options
**Why still open:** Permissive discovery does not determine whether malformed or
inapplicable option values should be accepted. The draft's existing error
boundary remains: invalid declarations fail. We still need to define their
valid shapes, rather than infer extra restrictions from that boundary.
Require exactly one selector per declaration: `path`, `dir`, or `glob`. A list
inside `path` or `dir` still counts as one selector. Use separate declarations
when combining different selector kinds. Diagnose unknown fields and options
that have no meaning for the chosen selector. The removed `optional` spelling
is not accepted.
**Recommendation (not yet accepted):** Accept nonnegative YAML integers for
`depth`; zero remains unlimited. Treat negative, null, string, boolean, and
fractional values as malformed, even when recursion is disabled. Use exactly one selector per
import: `path`, `dir`, or `glob`. Keep
`pattern`, `recursive`, and `depth` specific to `dir`; allow `follow` and `hidden`
on both `dir` and `glob`. `exclude` applies to all three selectors as settled
above. Diagnose unknown fields and options that have no meaning for the chosen
selector. This makes likely mistakes visible, but shared templates must omit
inapplicable fields. It imposes no arbitrary maximum depth. The settled
combination `recursive: false` with a valid larger `depth`
remains accepted and selects only direct files. The removed `optional` spelling
is not accepted under any variant.
| Option | `path` | `dir` | `glob` |
| --- | --- | --- | --- |
| `required`, `exclude` | Yes | Yes | Yes |
| `destination`, `placement` | Yes | Yes | Yes |
| `pattern` | No | Yes | No |
| `recursive`, `depth` | No | Yes | No |
| `follow`, `hidden` | No | Yes | Yes |
The accepted list forms also need explicit validation rules for empty lists,
null values, empty strings, and non-string elements. Recommend allowing empty
lists as empty selections (and `exclude: []` as no exclusions), while rejecting
null values and non-string elements rather than silently coercing them. Under
that recommendation, an explicit `pattern: []` matches nothing, unlike omission;
an empty `path` list has no files whose presence is required. Empty-string
semantics should be reviewed against the existing source-path and wildcard
rules rather than introducing a blanket prohibition. These remain OD-3 choices,
not additional settled restrictions.
### Types And Depth Applicability
**Alternative:** Normalize null depth to omission and negative integers to
unlimited, and permit unused directory controls on a direct-file declaration.
That eases generated configuration but can turn an erroneous bound into a wider
search or make a direct-file import appear recursive. These relaxations can be
chosen independently; they do not require accepting ambiguous selectors.
Another viable selector shape is to allow multiple selectors as an explicit
union, but it needs rules for constructing the graph and handling overlaps that
one-selector declarations avoid.
- `path`, `dir`, `pattern`, and `exclude` accept strings or flat lists of strings.
- `glob` accepts a string.
- `required`, `recursive`, `follow`, and `hidden` accept booleans.
- `depth` accepts nonnegative integers under ConfigKit's existing YAML scalar
rules. Zero means unlimited when recursion is enabled; negative, string,
boolean, and fractional values are invalid. Do not introduce an arbitrary
maximum depth.
- Explicit null is invalid for these fields, including null list members.
Do not coerce non-string members to strings or flatten nested lists.
- Existing `destination` and `placement` validation remains unchanged.
For list validation, another viable choice is to reject empty positive-selector
or pattern lists as likely authoring mistakes while allowing `exclude: []`.
That catches accidentally empty generated inputs, but requires conditional
omission of declarations and is less consistent with permissive empty searches.
Normalizing null to omission would ease generated configuration, but for
`pattern` it would broaden selection to all filenames. Neither alternative
changes the settled string-or-list support for nonempty valid values.
A supplied `depth` additionally requires effective recursion to be enabled,
using the default precedence defined above. Valid type and range alone do not
make depth applicable. This rule applies even to empty directory or pattern
lists. With recursion enabled, malformed depth values still fail validation.
### Empty Lists, Empty Strings, And Duplicates
Empty lists are valid and retain their set meanings:
| Declaration | Meaning | Effect of `required: true` |
| --- | --- | --- |
| `path: []` | No named files | Presence requirement satisfied: no listed file is missing |
| `dir: []` | No search roots | Fail: no selected file |
| `dir: fragments` with `pattern: []` | No positive patterns; select nothing | Fail: no selected file |
| `exclude: []` | Exclude nothing | Otherwise unchanged |
An empty pattern list differs from omission: omission uses the application
default if supplied, otherwise all filenames. An explicit empty list selects
nothing and overrides any application default.
Empty strings are invalid for `path`, `dir`, and `glob`, including empty entries
in path or directory lists. Use `.` to explicitly name the declaring directory
as a search root. Empty strings in `pattern` and `exclude` are valid wildcard
expressions that match no actual filename. Other source-path and wildcard
validation rules still apply.
Repeated identical path entries within one declaration select that candidate
once. Repeated patterns or exclusions have no additional effect, and directory
search overlaps follow the settled set-union rule. These rules do not suppress
existing repeated-source errors across separate declarations or collapse
separate symlink aliases.
Validate every supplied wildcard expression even if it is inactive because
another list is empty or discovery finds no files. Likewise, malformed list
members remain errors even when other entries are valid. Do not silently drop
invalid members or treat malformed filters as absent.