ConfigKit/docs/roadmap/imports.md

Import Ergonomics And Permissive Defaults

Status: working draft / brainstorming notes. This document captures a proposed direction for discussion, not an implemented contract or an implementation plan. The selected direction separates directory searches, filename matching, recursion, and absence handling. Optional-by-default imports are the intended direction. The Ansible-aligned decisions below are settled for this roadmap; remaining choices have recommendations and alternatives under Open Decisions. All new syntax below is illustrative and unimplemented.

Guiding Philosophy

ConfigKit should provide predictable configuration mechanisms and avoid imposing application policy. Its defaults should permit optional contributions to be absent, with explicit options for increased strictness when needed. That choice may come from an operator or the consuming application; authored import declarations can also express requirements on contributions.

Permissiveness should distinguish absence from invalid supplied configuration:

Optional contributions may be absent by default. Consumers can explicitly require them. Supplied configuration must still be structurally valid and interpretable under the selected composition rules.

This is a general design direction worth making more explicit and consistent across ConfigKit. It is not blanket authorization to weaken existing contracts, ignore errors, or add a global permissiveness switch. Other defaults should be evaluated individually against this principle.

Current Baseline

The current import contract uses path or glob, with required imports by default and optional: true to permit absence. Glob matching uses slash-based Go path.Match syntax. Recursive ** syntax is not supported; authors can match particular depths with patterns such as prompts/*.yaml and prompts/*/*.yaml.

The proposed changes concern naming, discovery, and absence handling. Import destinations, composition, precedence, and resource identity remain separate concerns.

Proposed Import Vocabulary

Use a direct-file form and a directory-search form. The latter separates the search root from its filename filter rather than inferring traversal from a combined path pattern:

configkit:
  imports:
    - dir: prompts
      pattern: '*.yaml'
      recursive: true
      destination: [prompts]
      placement: sequence-item
    - path: defaults.yaml
      required: true

The first declaration searches prompts and its descendants for matching filenames, placing each source mapping as one prompt item, and permits no matches. The second explicitly requires its target file. Sequence composition still requires the consumer's existing registration, as explained below.

The fields have distinct jobs:

Field	Proposed meaning
path	Import one named file, relative to the declaring file
dir	Search a directory, relative to the declaring file
pattern	Filter candidate files by filename within the searched directories
recursive	Include descendant directories when true
depth	Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive
required	Fail when the declaration contributes no matching file
destination, placement	Apply the existing placement rules to each selected source

This follows the familiar separation of traversal and filtering in Ansible's find and include_vars interfaces, without adopting all their spellings, defaults, or matching modes.

Decisions Resolved From Ansible

The comparison covers ansible.builtin.find, ansible.builtin.include_vars, ansible.builtin.fileglob, and ansible.windows.win_find. A consistent approach means agreement among the relevant interfaces that offer the capability, not that every Ansible module exposes it. Existing ConfigKit contracts and the already selected optional-by-default direction remain authoritative; this is not a general Ansible compatibility mode.

The following choices are settled:

• Separate traversal from filename filtering. Directory roots and filename filters have separate roles; filters operate on basenames in visited directories. A directory failing the filename filter does not prune its descendants. This agrees with the directory-search interfaces above. See find's parameters and the include_vars implementation.
• Include an optional integer depth control. All three directory-search interfaces use that spelling. Direct files are at depth 1, and each additional directory adds one. Omission permits unlimited depth when recursion is enabled. See include_vars and win_find.
• Treat depth: 0 as unlimited. This follows include_vars and agrees with the current find implementation and win_find implementation. The latter two document unlimited omission more clearly than explicit zero; this choice uses source-confirmed behavior, not a claimed universal documented Ansible guarantee. Negative values remain a separate open decision.
• Disabling recursion takes precedence over depth. With recursive: false, search only direct files, even if a valid larger or unlimited depth is supplied. Depth does not enable recursion. Both find and win_find explicitly use this rule where a separate recursion switch exists.
• Directory searches select files, not directory documents. Directories are traversal containers, even when their names match pattern. This agrees with the default file selection in the find modules and the file-oriented fileglob lookup. Existing regular-file and symlink rules still govern opening selected sources; the legacy glob contract is not implicitly changed by this new search form.

The accepted recursive spelling is retained. The formerly tentative max_depth spelling becomes depth. Resource budgets remain separate from depth, and the current limits still apply to an unlimited-depth search.

Discovery Semantics

Filename Matching

pattern applies to a file's basename, not its full relative path. For example, *.yaml matches both prompts/one.yaml and prompts/weather/daily/summary.yaml when the latter's directory is visited. Recursion changes which directories are visited; it does not change what * means. A nonrecursive search considers only files directly inside dir.

Recommend retaining the familiar path.Match wildcard grammar for filename filters. The name pattern does not by itself imply regular expressions or automatic dialect detection. Exact filter defaults and the treatment of existing path-glob use cases remain open below.

Recursive Discovery And Depth Limits

Use recursive: true for recursive discovery, optionally bounded by depth. The motivation is to let authors organize definitions in subdirectories without updating import declarations for every depth. Directory organization should not change resource identity or the effective configuration destination.

The selected model uses an explicit search root and recursion flag, rather than introducing recursive ** syntax or inferring a root from a wildcard prefix. The default when recursive is omitted remains open; examples specify it.

The settled depth convention is:

• Files directly inside the search root are at depth 1.
• Each additional directory increases depth by one.
• Omitting depth, or setting it to 0, imposes no caller-selected depth limit when recursion is enabled; existing loader resource limits still apply.
• recursive: false selects only direct files regardless of a valid depth value.
• Depth controls discovery only, not namespaces, identities, or destinations.

Remaining validation and traversal choices are listed below. Recursive discovery must fit the existing filesystem containment, repeated-source, and load-budget contracts rather than silently changing them. In particular, examined entries, including nonmatches, still consume the visit budget; a small number of matches does not authorize an unbounded filesystem scan. This is an existing ConfigKit constraint, not a new Ansible-derived decision.

Illustrative Examples

These examples describe the proposed interface, not configuration accepted by the current release. They assume the consumer has enabled imports. File paths and directory roots resolve from the declaring file; destinations use the existing relative-placement rules.

Import One File

configkit:
  imports:
    - path: defaults.yaml

Import defaults.yaml into the declaring source's mapping scope. Under the proposed default, a missing file contributes nothing. Add required: true when its presence is necessary. A present but invalid file still fails loading.

Recursively Discover Mapping Fragments

configkit:
  imports:
    - dir: services
      pattern: '*.yaml'
      recursive: true
      destination: [services]

This discovers files such as services/api.yaml and services/regional/worker.yaml, but not services/README.md. Each file contains named service entries, for example api: {port: 8080} or worker: {concurrency: 4}. They compose beneath services under the existing strict composition rules. Directory names do not create additional mapping keys. Missing directories or no matching files contribute nothing.

Adding depth: 2 limits discovery to direct files and files one directory below services: it still selects services/regional/worker.yaml, but not services/regional/east/worker.yaml. With recursive: false, only direct files are selected, even if depth: 2 remains present.

Require A Recursive Prompt Collection

configkit:
  imports:
    - dir: prompts
      pattern: '*.prompt.yaml'
      recursive: true
      required: true
      destination: [prompts]
      placement: sequence-item

This selects prompts/welcome.prompt.yaml and prompts/weather/daily/summary.prompt.yaml, but not prompts/weather/helpers.yaml. Each selected file contains one prompt mapping, such as id: prompt.welcome and text: Welcome. The consumer registers the prompts sequence with identity field id through the existing Definition.SequenceCompositions API. Distinct IDs compose into one sequence; duplicate IDs fail strict composition. Overlays continue replacing sequences.

Here required: true means at least one file must match across the entire search, not that every visited directory must contain a matching file. The filename convention keeps helper files out of this discovery declaration; it does not prevent a selected file from explicitly importing a helper. Existing repeated-source rules still apply if declarations select overlapping sources.

Optional Imports By Default

The preferred direction is optional-by-default for both explicit files and directory searches, with required: true opting into failure on absence. This deliberately supersedes the earlier suggestion to retain required-by-default imports.

The table assumes the library's default absence policy. A consumer-selected default or enforcement option remains open below.

Situation	Proposed behavior
Missing path; required omitted or false	Contribute nothing
Missing search directory or no matching files; required omitted or false	Contribute nothing
Missing file, missing search directory, or zero matches with required: true	Fail with an import diagnostic
Invalid pattern or malformed import declaration	Fail
Permission failure, unreadable matched file, or malformed YAML	Fail
Composition conflict or other invalid supplied configuration	Fail under the applicable existing rules

An optional declaration is permission for absence, not permission to ignore a present contribution that cannot be read or interpreted. Optionality must not turn processing failures into apparently successful partial configuration.

This default supports installations with different optional components without requiring ceremony for every potentially absent directory or file. Its tradeoff is that a misspelled path or pattern can also contribute nothing. Authors and applications that need an import to exist can state that requirement explicitly; applications remain responsible for validating their necessary final values. Final validation does not automatically detect every unintentionally omitted contribution.

Who Chooses Strictness?

The general philosophy permits operators and consuming applications to choose stricter behavior. Import authors can express local requirements through required: true. This does not yet specify an application-wide override API.

Keep the distinction between choosing defaults and overriding explicit intent. If an operator or application should be able to strengthen or relax a requirement declared in an imported document, define that capability and its precedence separately. Optional-by-default alone does not imply that an authored requirement can be overridden, or that requirements propagate to transitive imports.

Migration

Changing absence defaults changes behavior even when existing YAML contains no optionality field. Renaming keys also changes the accepted syntax. Pre-1.0 is a useful time to reconsider these contracts, but adoption still needs explicit migration guidance.

To preserve current intent, currently required imports would acquire required: true; existing optional: true declarations could omit the new field or use required: false. A simple glob: prompts/*.yaml becomes dir: prompts plus pattern: '*.yaml' and recursive: false, preserving its discovery scope. This is not a mechanical rename of glob to pattern: current globs can also select particular directory names or depths. Their compatibility path must be settled before removing any existing form.

Open Decisions

The recommendations below are not settled decisions. Each concerns either inconsistent Ansible behavior or a ConfigKit-specific policy without a direct Ansible counterpart. They do not reopen the accepted absence policy, placement, strict composition, or source-version rules.

1. Recursion Default

Ansible evidence: find and win_find default to nonrecursive searches; include_vars recursively loads directories by default. There is no common default across these interfaces. See their parameter references above.

Recommendation: Default recursive to false. This keeps discovery local unless the declaration explicitly requests descendants and makes migration from simple current globs straightforward. The cost is one extra field for recursive collections.

Alternative: Default to true, following include_vars. Collection authors could reorganize files into subdirectories without revising declarations, but adding a subdirectory could also add contributions to an existing import.

2. Matching Language And Filter Shape

Ansible evidence: find supports shell patterns and optional regex; include_vars.files_matching is a regex; win_find uses PowerShell wildcards or regex. Their case and matching details differ. There is no single exact grammar.

Recommendation: Keep one scalar pattern using ConfigKit's existing case-sensitive Go path.Match grammar, applied to basenames. This reuses a familiar contract, including escapes and character classes, without introducing regex mode or platform-dependent case rules. It cannot express every regex or an arbitrary union of suffixes in one declaration; separate declarations remain available, subject to existing repeated-source rules.

Alternative: Add an explicit regex mode and/or a list of patterns whose matches are unioned within one declaration. This better supports filters such as both .yaml and .yml without overlapping imports, but adds syntax, matching rules, and validation. Regex would need a specified Go grammar rather than a promise of Python or PowerShell compatibility. Exclusion filters could accompany that design, but are not assumed in the initial recommendation.

3. Omitted Filename Filter

Ansible evidence: The find modules permit unrestricted filename selection. include_vars also has an extension policy, defaulting to json, yaml, and yml, with separate handling of unknown extensions. Its loading behavior is not equivalent to a general-purpose find operation.

Recommendation: Omitted pattern means all filenames, with no implicit extension filter. This keeps selection independent of filename conventions and supports extensionless configuration. A selected README or other invalid file still causes a load error; authors of mixed directories should supply a filter.

Alternative: Default discovery to .yaml and .yml, with an explicit way to select all files or other extensions. This suits typical configuration directories and avoids incidental README selection, but adds an extension policy and an override mechanism that explicit path imports do not need.

4. Invalid Depth Values

Ansible evidence: Positive depth, unlimited omission, and disabled-recursion precedence have useful common semantics. The implementations do not provide a consistent negative-depth contract; accepting a value is not evidence of a useful shared meaning. See the implementation links above.

Recommendation: Accept nonnegative YAML integers only. Zero means unlimited; negative, null, string, boolean, or fractional values are malformed declarations, even when recursion is disabled. This follows ConfigKit's typed directive style and gives likely mistakes a direct diagnostic without imposing an arbitrary maximum depth.

Alternative: Treat negative integers as unlimited too, and null as omission. That can ease generated configurations but adds redundant spellings and risks turning an erroneous bound into a wider search. Neither interpretation is required by the settled rule that disabled recursion ignores a valid bound.

5. Hidden Files And Directories

Ansible evidence: The find modules expose hidden-file filtering, while include_vars has no equivalent hidden-entry exclusion in its directory walk. Also, filtering a hidden entry is not necessarily the same as pruning a hidden directory's descendants. See find and include_vars.

Recommendation: Give dot-prefixed names no special treatment: visit their directories within the selected depth and select files by pattern. This fits the existing matcher and avoids an implicit exclusion. Broad recursive searches may include configuration in hidden directories, so the search root and filter should reflect the intended collection.

Alternative: Add hidden, defaulting to false, with an explicit definition covering both hidden-file selection and hidden-directory pruning. This can exclude incidental metadata directories, but adds policy and a platform choice between dot prefixes and filesystem hidden attributes. Callers would need to opt in when their collections intentionally use hidden names.

6. Directory Symlinks And Traversal Cycles

Ansible evidence: The find modules default to not following directory links; the inspected include_vars implementation walks with link following enabled. There is no common default to adopt.

Recommendation: Follow contained directory symlinks using ConfigKit's existing canonicalization and containment rules, preserving the current ability to select through contained symlinks. For the new recursive walk, detect a canonical directory repeated on the active traversal path and report a cycle. Do not silently deduplicate imported files reached through aliases; existing source-repeat errors remain applicable. This supports linked collections but requires directory-cycle handling in addition to import-cycle handling.

Alternative: Add a traversal follow option defaulting to false, skipping directory symlinks encountered below the search root unless enabled. Keep explicitly named roots and file targets under existing source-opening rules. This reduces accidental traversal through linked trees, but introduces a new distinction between discovery and explicit access and requires callers to opt in for linked collections. Both choices retain native-filesystem semantics; neither promises symlink capabilities absent from the supplied filesystem.

7. Match Ordering

Ansible evidence: include_vars sorts directories and filenames; win_find documents results sorted by path. POSIX find describes top-down searching and does not promise the same global ordering. These are not identical ordering contracts.

Recommendation: Sort all selected slash-relative paths lexicographically within each declaration, then use existing depth-first import expansion. This extends ConfigKit's existing glob order and stabilizes sequence assembly and diagnostics. It requires collecting bounded candidates before expansion.

Alternative: Load direct files first, then visit sorted subdirectories recursively. This offers a natural directory-oriented traversal and can begin loading sooner, but differs from existing global match ordering and can change sequence item positions. Neither choice grants replacement precedence within strict composition.

8. Application And Operator Strictness Controls

Ansible evidence: Task failure controls and module-specific options are not a direct counterpart to defaults for ConfigKit's transitive import graph. Ansible cannot settle this public API choice.

Recommendation: Add a load-level default for whether imports are required, used only when a declaration omits required. The library default stays false; an application can choose true and expose that choice to operators through its own interface. Apply the default independently to each declaration, including transitive imports. Explicit true or false remains authoritative. This enables stricter deployments while preserving intentionally optional imports; it does not enforce that every authored optional declaration succeeds.

Alternative: Offer an enforcement mode requiring every declaration to match, including those explicitly marked required: false. That provides a stronger audit policy but overrides authors' intent and needs clear precedence and diagnostics. A smaller alternative is authored requirements only for now; it avoids a public option but leaves applications unable to choose a common default without editing source declarations.

9. Evidence For Absent Optional Imports

Ansible evidence: fileglob returns an empty result for no matches; include_vars reports included filenames; the find modules expose richer search results. There is no common inspection model for an absent import declaration.

Recommendation: Record minimal structured inspection evidence for an optional declaration that selects nothing: its declaring source/location, selector, and absent outcome. Emit no warning by default and do not fabricate a loaded-source record. This helps distinguish intentional absence from a typo, but requires an inspection API decision and associated ownership tests. Design the exact record shape during implementation planning, using existing evidence structures where appropriate.

Alternative: Add no new inspection surface initially; retain loaded-source evidence and let required: true turn absence into a diagnostic when needed. This keeps the feature smaller, but successful optional omissions are harder to investigate. Existing inspection contracts must remain intact either way.

10. Selector Combinations And Directory-Only Options

Ansible evidence: include_vars separates file-only and directory-only arguments, while the find modules are directory-search interfaces. This does not settle how ConfigKit should combine its new form with retained path globs.

Recommendation: Exactly one selector per declaration: path, dir, or glob if retained. Use a literal directory root; pattern, recursive, and depth apply only to dir. Diagnose irrelevant fields rather than silently ignoring them. This makes declarations unambiguous and keeps the roles of file selection and traversal visible, at the cost of templates needing to omit inapplicable fields. This is a proposed validation rule, not an already settled restriction.

Alternative: Permit directory-search controls on a path declaration and ignore them, retaining exclusive selectors. Shared templates become easier, but a declaration can appear recursive without being so. Wildcard directory roots are another viable extension: useful for selecting collections, but they need root-expansion, depth-origin, ordering, and overlap rules. Retaining glob is the simpler way to preserve existing wildcard-directory use cases.

11. Compatibility And Migration

Ansible evidence: There is no Ansible precedent that determines ConfigKit's compatibility commitments or migration schedule.

Recommendation: Retain glob alongside dir because it can select directory segments and exact depths that basename filters cannot express. Introduce required while accepting explicit optional as its inverse for one documented transition release. Reject declarations containing both fields to avoid two authorities for the same choice. This retains existing expressiveness and eases adoption, at the cost of temporary parser and documentation complexity.

The intended optional default applies when both fields are absent, including legacy path and glob declarations; old spelling alone does not preserve the old default. The application default, if adopted, also applies to this absent case. Explicit legacy optional: false maps to required: true, and explicit optional: true maps to required: false. Release notes must call out the behavior change and show how to preserve existing requirements. Under the release policy, this belongs in a pre-1.0 minor release, not a patch release.

Alternative: Remove optional immediately in that minor release while retaining glob. This establishes one vocabulary sooner, but every declaration using the old field needs migration before it can load. If a transition is chosen, accepting both fields when they agree is also viable; it helps generated documents at the cost of extra normalization rules. Removing glob too would need an expressiveness decision and migration for wildcard directory segments; it cannot be treated as a simple field rename.