Settle import validation and require recursion for depth

docs/roadmap/imports.md

@@ -5,7 +5,7 @@ direction for discussion, not an implemented contract or an implementation plan.
 The selected direction separates directory searches, filename matching,
 recursion, and absence handling. Optional-by-default imports are the intended
 direction. The consumer-directed and Ansible-aligned decisions below are settled;
-remaining choices have recommendations and alternatives under Open Decisions.
+API spelling and implementation details remain for implementation planning.
 All new syntax below is illustrative and unimplemented.
 
 ## Guiding Philosophy
@@ -86,11 +86,11 @@ The fields have distinct jobs:
 | --- | --- |
 | `path` | Import a named file or list of named files, relative to the declaring file |
 | `dir` | Search a literal directory or list of literal directories, relative to the declaring file |
-| `pattern` | A filename wildcard or list of wildcards; select files matching any pattern within the searched directories |
+| `pattern` | A filename wildcard or list of wildcards; select files matching any effective pattern, using the default precedence below |
 | `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file |
 | `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results |
-| `recursive` | Default false; include descendant directories when true |
+| `recursive` | Library default false, overridable by an application default; include descendant directories when effectively true |
-| `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive |
+| `depth` | Bound directory-search depth; supplying it requires effective recursion to be enabled; 1 means direct files, 0 or omission means unlimited when recursive |
 | `follow` | Follow symlinks by default; false disables following within the discovery scope defined below |
 | `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below |
 | `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search |
@@ -108,9 +108,10 @@ interfaces, without adopting all their spellings, defaults, or matching modes.
   it as an alias or normalize old declarations at runtime.
 - **Optional contributions by default.** `required: true` opts into failure on
   absence. Supplied invalid configuration continues to fail.
-- **No implicit filename restriction.** Omitting `pattern` selects all
+- **No library-imposed filename restriction.** Omitting `pattern` selects all
-  filenames. Authors can narrow selection explicitly. There is no default YAML
+  filenames unless the application supplies a default pattern. Authors can
-  extension list; extensionless sources remain discoverable.
+  override that default explicitly. There is no library default YAML extension
+  list; extensionless sources remain discoverable.
 - **Follow symlinks by default, with an opt-out.** Provide `follow: false` for
   callers who want discovery without symlink following. Applicable containment
   and filesystem contracts remain in force.
@@ -120,11 +121,12 @@ interfaces, without adopting all their spellings, defaults, or matching modes.
   shared dependency-ready scheduler defined below. This is settled independently
   of the existing implementations and includes refactoring both resolvers.
 
-OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-4
+OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-3
-(discovery boundaries and cycles), OD-5 (policy precedence), and OD-6
+(directive validation), OD-4 (discovery boundaries and cycles), OD-5 (policy
-(absent-import evidence) are settled in the sections below. Only OD-3 remains
+precedence), and OD-6 (absent-import evidence) are settled in the sections below.
-open. `follow` and `hidden` are the proposed field spellings; their applicability
+Supplying `depth` requires effective recursion to be enabled, including when
-to `glob` remains part of OD-3. Applying `exclude` to `glob` is settled.
+enabled through an application default. This supersedes the earlier direction
+to accept but ignore depth when recursion is disabled.
 
 ## Decisions Resolved From Ansible
 
@@ -153,13 +155,8 @@ The following choices are settled:
   and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1).
   The latter two document unlimited omission more clearly than explicit zero;
   this choice uses source-confirmed behavior, not a claimed universal documented
-  Ansible guarantee. Negative values remain a separate open decision.
+  Ansible guarantee. Negative values are invalid under the settled directive
-- **Disabling recursion takes precedence over depth.** With `recursive: false`,
+  validation rules below.
-  search only direct files, even if a valid larger or unlimited depth is supplied.
-  Depth does not enable recursion. Both
-  [find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters)
-  and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters)
-  explicitly use this rule where a separate recursion switch exists.
 - **Directory searches select files, not directory documents.** Directories are
   traversal containers, even when their names match `pattern`. This agrees with
   the default file selection in the find modules and the file-oriented
@@ -267,9 +264,10 @@ Recursion changes which directories are visited; it does not change what `*`
 means. A nonrecursive search considers only files directly inside `dir`.
 
 **OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard
-grammar for filename filters. Omitting `pattern` selects all filenames, with no
+grammar for filename filters. When neither the declaration nor the application
-implicit extension restriction. Do not add regular expressions, automatic
+supplies `pattern`, select all filenames, with no implicit extension restriction.
-dialect detection, or recursive `**` syntax as part of this work.
+Do not add regular expressions, automatic dialect detection, or recursive `**`
+syntax as part of this work.
 
 Retain `glob` as a permanent, separate selector using the existing slash-based
 path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects
@@ -294,12 +292,15 @@ D × P and take the **union** of the matching candidate files:
 `S = ⋃ {matches(d, p) | (d, p) ∈ D × P}`
 
 Each root uses the declaration's recursion, depth, and applicable discovery
-controls. When `pattern` is omitted, select all filenames in the visited
+controls. Resolve an omitted `pattern` from the application default first;
-directories instead. This describes selection semantics, not a requirement to
+if neither supplies patterns, select all filenames in the visited directories
+instead. This describes selection semantics, not a requirement to
 materialize the product or traverse a directory again for each pattern.
 
 Overlapping patterns or directory searches within one declaration select the
-same candidate once. This local set union is not global source deduplication:
+same candidate once. Repeated identical entries in one `path` list likewise
+select that candidate once; duplicate patterns or exclusions have no additional
+effect. This local deduplication is not global source deduplication:
 existing repeated-source and alias checks still apply to distinct inclusion
 routes. In particular, it does not authorize silently merging separate alias
 paths to an imported source.
@@ -334,7 +335,8 @@ prohibit another declaration from explicitly selecting the same filename.
 
 ### Recursive Discovery And Depth Limits
 
-Use `recursive: true` for recursive discovery, optionally bounded by `depth`.
+Enable recursion explicitly or through an application default for recursive
+discovery, optionally bounded by `depth`.
 The motivation is to let authors organize definitions in subdirectories without
 updating import declarations for every depth. Directory organization should not
 change resource identity or the effective configuration destination.
@@ -342,10 +344,10 @@ change resource identity or the effective configuration destination.
 The selected model uses an explicit search root and recursion flag, rather than
 introducing recursive `**` syntax or inferring a root from a wildcard prefix.
 
-**OD-1: settled.** Omitting `recursive` means false: search only direct files.
+**OD-1: settled.** The library default for `recursive` is false: search only
-Authors explicitly opt into descendants. This makes the search scope visible
+direct files. An application may supply a different default; an explicit
-while keeping recursion independent of the permissive defaults for absence,
+declaration value overrides it, including `recursive: false`. Recursion remains
-symlinks, and hidden entries.
+independent of the permissive defaults for absence, symlinks, and hidden entries.
 
 The settled depth convention is:
 
@@ -353,11 +355,21 @@ The settled depth convention is:
 - Each additional directory increases depth by one.
 - Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit
   when recursion is enabled; existing loader resource limits still apply.
-- `recursive: false` selects only direct files regardless of a valid depth value.
+- Supplying `depth` requires effective `recursive: true`. If recursion is false
+  after resolving declaration, application, and library defaults, any supplied
+  `depth` is an error, including `0` or `1`. Depth never enables recursion itself.
+- With recursion disabled and `depth` omitted, select only direct files.
 - Depth controls discovery only, not namespaces, identities, or destinations.
 
-Remaining directive validation choices are listed below. Recursive discovery
+Resolve the effective recursion value before checking depth's applicability.
-must fit the existing filesystem containment, repeated-source, and load-budget
+For example, `depth: 2` with no explicit `recursive` is valid if the application
+default enables recursion, but invalid with the library default. Explicit
+`recursive: false` overrides an application default of true and makes any
+supplied depth invalid. Depth must also have the valid type and range defined
+under directive validation; enabling recursion does not excuse malformed depth.
+
+Recursive discovery must fit the existing filesystem containment,
+repeated-source, and load-budget
 contracts rather than silently changing them. In particular, examined entries,
 including nonmatches, still consume the visit budget; a small number of matches
 does not authorize an unbounded filesystem scan. This is an existing ConfigKit
@@ -378,9 +390,8 @@ direct-file imports retain their existing source-access semantics. The controls
 do not implicitly propagate to declarations inside imported files.
 
 These are discovery-selection controls, not a load-wide access prohibition.
-If OD-3 enables these controls on `glob`, filtering applies
+On `glob`, filtering applies to wildcard-selected entries; explicitly named
-to wildcard-selected entries; explicitly named path components remain explicit
+path components remain explicit access.
-access. This conditional scope rule does not settle that open decision.
 
 When following links, detect a canonical directory repeated on the active
 traversal path and fail with a cycle diagnostic. Do not silently prune the loop
@@ -449,8 +460,8 @@ keys. Missing directories or no matching files contribute nothing.
 
 Adding `depth: 2` limits discovery to direct files and files one directory below
 `services`: it still selects `services/regional/worker.yaml`, but not
-`services/regional/east/worker.yaml`. With `recursive: false`, only direct files
+`services/regional/east/worker.yaml`. Switching to `recursive: false` requires
-are selected, even if `depth: 2` remains present.
+omitting `depth`; leaving `depth: 2` present is a declaration error.
 
 Hidden entries and contained symlinks participate by default. To narrow the
 search, the declaration can also include:
@@ -550,12 +561,22 @@ contribution.
 
 ## Who Chooses Strictness?
 
-**OD-5: settled.** Provide load-level defaults for `required`, `follow`, and
+**OD-5: settled.** Provide load-level defaults for `required`, `follow`, `hidden`,
-`hidden`. Resolve each applicable field independently using this precedence:
+`recursive`, and `pattern`. Resolve each applicable field independently using
+this precedence:
 
 1. Explicit value on the import declaration.
 2. Application-selected default, when supplied.
-3. Library default: false for `required`, true for `follow` and `hidden`.
+3. Library default: false for `required` and `recursive`, true for `follow` and
+   `hidden`, and no filename restriction for `pattern`.
+
+An explicit `pattern` replaces the application default rather than combining
+with it. In particular, `pattern: []` selects nothing, even when the application
+supplies patterns. An application can default to `['*.yml', '*.yaml']`; without
+either an application default or an explicit pattern, a directory search
+attempts to load all files in its search scope. Application defaults apply only
+to selectors that support the field; they do not add inapplicable options to
+other selectors.
 
 Apply this rule to transitive declarations too. A declaration does not inherit
 these field values from its importing parent. Applications may expose the
@@ -566,7 +587,7 @@ Explicit declarations remain authoritative, permitting intentional exceptions
 to stricter application defaults. Defaults are not enforcement controls. Do not
 add options that override explicit declarations as part of this work without a
 separate concrete requirement and scope decision. The fields' applicability to
-each selector remains part of OD-3.
+each selector is defined under directive validation.
 
 ## Evidence For Absent Optional Imports
 
@@ -616,55 +637,75 @@ in its implementation documentation. Do not label implementation-specific
 matching or cycle-path helpers with an algorithm name until that algorithm is
 actually chosen.
 
-## Open Decisions
+## Directive Validation
 
-Only OD-3 remains open. Its identifier is preserved for continued discussion;
+**OD-3: settled.** Validate declaration shapes and values independently of
-the recommendations below are not settled decisions. OD-1, OD-2, OD-4, OD-5,
+whether discovery selects any files. Invalid declarations fail even when a
-and OD-6 are settled above.
+search root is missing or a list is empty. This does not require opening or
+parsing excluded files or establish a new first-error ordering guarantee.
 
-### OD-3. Directive Validation
+### Selectors And Applicable Options
 
-**Why still open:** Permissive discovery does not determine whether malformed or
+Require exactly one selector per declaration: `path`, `dir`, or `glob`. A list
-inapplicable option values should be accepted. The draft's existing error
+inside `path` or `dir` still counts as one selector. Use separate declarations
-boundary remains: invalid declarations fail. We still need to define their
+when combining different selector kinds. Diagnose unknown fields and options
-valid shapes, rather than infer extra restrictions from that boundary.
+that have no meaning for the chosen selector. The removed `optional` spelling
+is not accepted.
 
-**Recommendation (not yet accepted):** Accept nonnegative YAML integers for
+| Option | `path` | `dir` | `glob` |
-`depth`; zero remains unlimited. Treat negative, null, string, boolean, and
+| --- | --- | --- | --- |
-fractional values as malformed, even when recursion is disabled. Use exactly one selector per
+| `required`, `exclude` | Yes | Yes | Yes |
-import: `path`, `dir`, or `glob`. Keep
+| `destination`, `placement` | Yes | Yes | Yes |
-`pattern`, `recursive`, and `depth` specific to `dir`; allow `follow` and `hidden`
+| `pattern` | No | Yes | No |
-on both `dir` and `glob`. `exclude` applies to all three selectors as settled
+| `recursive`, `depth` | No | Yes | No |
-above. Diagnose unknown fields and options that have no meaning for the chosen
+| `follow`, `hidden` | No | Yes | Yes |
-selector. This makes likely mistakes visible, but shared templates must omit
-inapplicable fields. It imposes no arbitrary maximum depth. The settled
-combination `recursive: false` with a valid larger `depth`
-remains accepted and selects only direct files. The removed `optional` spelling
-is not accepted under any variant.
 
-The accepted list forms also need explicit validation rules for empty lists,
+### Types And Depth Applicability
-null values, empty strings, and non-string elements. Recommend allowing empty
-lists as empty selections (and `exclude: []` as no exclusions), while rejecting
-null values and non-string elements rather than silently coercing them. Under
-that recommendation, an explicit `pattern: []` matches nothing, unlike omission;
-an empty `path` list has no files whose presence is required. Empty-string
-semantics should be reviewed against the existing source-path and wildcard
-rules rather than introducing a blanket prohibition. These remain OD-3 choices,
-not additional settled restrictions.
 
-**Alternative:** Normalize null depth to omission and negative integers to
+- `path`, `dir`, `pattern`, and `exclude` accept strings or flat lists of strings.
-unlimited, and permit unused directory controls on a direct-file declaration.
+- `glob` accepts a string.
-That eases generated configuration but can turn an erroneous bound into a wider
+- `required`, `recursive`, `follow`, and `hidden` accept booleans.
-search or make a direct-file import appear recursive. These relaxations can be
+- `depth` accepts nonnegative integers under ConfigKit's existing YAML scalar
-chosen independently; they do not require accepting ambiguous selectors.
+  rules. Zero means unlimited when recursion is enabled; negative, string,
-Another viable selector shape is to allow multiple selectors as an explicit
+  boolean, and fractional values are invalid. Do not introduce an arbitrary
-union, but it needs rules for constructing the graph and handling overlaps that
+  maximum depth.
-one-selector declarations avoid.
+- Explicit null is invalid for these fields, including null list members.
+  Do not coerce non-string members to strings or flatten nested lists.
+- Existing `destination` and `placement` validation remains unchanged.
 
-For list validation, another viable choice is to reject empty positive-selector
+A supplied `depth` additionally requires effective recursion to be enabled,
-or pattern lists as likely authoring mistakes while allowing `exclude: []`.
+using the default precedence defined above. Valid type and range alone do not
-That catches accidentally empty generated inputs, but requires conditional
+make depth applicable. This rule applies even to empty directory or pattern
-omission of declarations and is less consistent with permissive empty searches.
+lists. With recursion enabled, malformed depth values still fail validation.
-Normalizing null to omission would ease generated configuration, but for
+
-`pattern` it would broaden selection to all filenames. Neither alternative
+### Empty Lists, Empty Strings, And Duplicates
-changes the settled string-or-list support for nonempty valid values.
+
+Empty lists are valid and retain their set meanings:
+
+| Declaration | Meaning | Effect of `required: true` |
+| --- | --- | --- |
+| `path: []` | No named files | Presence requirement satisfied: no listed file is missing |
+| `dir: []` | No search roots | Fail: no selected file |
+| `dir: fragments` with `pattern: []` | No positive patterns; select nothing | Fail: no selected file |
+| `exclude: []` | Exclude nothing | Otherwise unchanged |
+
+An empty pattern list differs from omission: omission uses the application
+default if supplied, otherwise all filenames. An explicit empty list selects
+nothing and overrides any application default.
+
+Empty strings are invalid for `path`, `dir`, and `glob`, including empty entries
+in path or directory lists. Use `.` to explicitly name the declaring directory
+as a search root. Empty strings in `pattern` and `exclude` are valid wildcard
+expressions that match no actual filename. Other source-path and wildcard
+validation rules still apply.
+
+Repeated identical path entries within one declaration select that candidate
+once. Repeated patterns or exclusions have no additional effect, and directory
+search overlaps follow the settled set-union rule. These rules do not suppress
+existing repeated-source errors across separate declarations or collapse
+separate symlink aliases.
+
+Validate every supplied wildcard expression even if it is inactive because
+another list is empty or discovery finds no files. Likewise, malformed list
+members remain errors even when other entries are valid. Do not silently drop
+invalid members or treat malformed filters as absent.