# Import Ergonomics And Permissive Defaults Status: working draft / brainstorming notes. This document captures a proposed direction for discussion, not an implemented contract or an implementation plan. The selected direction separates directory searches, filename matching, recursion, and absence handling. Optional-by-default imports are the intended direction. The consumer-directed and Ansible-aligned decisions below are settled; remaining choices have recommendations and alternatives under Open Decisions. All new syntax below is illustrative and unimplemented. ## Guiding Philosophy ConfigKit should provide predictable configuration mechanisms and avoid imposing application policy. Its defaults should permit optional contributions to be absent, with explicit options for increased strictness when needed. That choice may come from an operator or the consuming application; authored import declarations can also express requirements on contributions. Permissiveness should distinguish absence from invalid supplied configuration: > Optional contributions may be absent by default. Consumers can explicitly > require them. Supplied configuration must still be structurally valid and > interpretable under the selected composition rules. This is a general design direction worth making more explicit and consistent across ConfigKit. It is not blanket authorization to weaken existing contracts, ignore errors, or add a global permissiveness switch. Other defaults should be evaluated individually against this principle. Discovery should accept hidden entries and follow symlinks by default, with options to narrow either behavior. An omitted filename filter should not impose an implicit extension restriction. These defaults do not waive containment, source validity, composition, or resource-limit rules. Recursion remains a separate choice about search scope, not a measure of permissiveness. This is a hard-breaking pre-1.0 change. Do not build temporary aliases, translation layers, compatibility modes, or staged deprecation support. Evaluate the final interface on its own merits, without retaining syntax solely to ease upgrades. ## Current Baseline The [current import contract](../config.md#explicit-imports) uses `path` or `glob`, with required imports by default and `optional: true` to permit absence. Glob matching uses slash-based Go `path.Match` syntax. Recursive `**` syntax is not supported; authors can match particular depths with patterns such as `prompts/*.yaml` and `prompts/*/*.yaml`. Current imports expand depth-first in declaration order, with each ordinary source contributing its body before its imports. The existing [sibling inheritance resolver](../config.md#sibling-inheritance) instead follows dependency chains before resolving dependents, using sorted sibling names and an explicit stack. Neither implements the shared ordering selected below. The proposed changes concern naming, discovery, absence handling, and uniform dependency ordering for imports and `extends`. Import destinations, strict composition, inheritance overlay semantics, tier precedence, and resource identity retain their distinct roles. ## Proposed Import Vocabulary Use a direct-file form and a directory-search form. The latter separates the search root from its filename filter rather than inferring traversal from a combined path pattern: ```yaml configkit: imports: - dir: prompts pattern: '*.yaml' recursive: true destination: [prompts] placement: sequence-item - path: defaults.yaml required: true ``` The first declaration searches `prompts` and its descendants for matching filenames, placing each source mapping as one prompt item, and permits no matches. The second explicitly requires its target file. Sequence composition still requires the consumer's existing registration, as explained below. The fields have distinct jobs: | Field | Proposed meaning | | --- | --- | | `path` | Import one named file, relative to the declaring file | | `dir` | Search a directory, relative to the declaring file | | `pattern` | Filter candidate files by filename within the searched directories | | `recursive` | Default false; include descendant directories when true | | `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive | | `follow` | Follow symlinks by default; false disables following within the discovery scope defined below | | `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below | | `required` | Fail when the declaration contributes no matching file | | `destination`, `placement` | Apply the existing placement rules to each selected source | This follows the familiar separation of traversal and filtering in Ansible's [find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html) and [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html) interfaces, without adopting all their spellings, defaults, or matching modes. ## Decisions Resolved By Project Direction - **No transition support.** Use the final interface immediately in the selected pre-1.0 minor release. Remove `optional` in favor of `required`; do not accept it as an alias or normalize old declarations at runtime. - **Optional contributions by default.** `required: true` opts into failure on absence. Supplied invalid configuration continues to fail. - **No implicit filename restriction.** Omitting `pattern` selects all filenames. Authors can narrow selection explicitly. There is no default YAML extension list; extensionless sources remain discoverable. - **Follow symlinks by default, with an opt-out.** Provide `follow: false` for callers who want discovery without symlink following. Applicable containment and filesystem contracts remain in force. - **Include hidden entries by default, with an opt-out.** Provide `hidden: false` to exclude hidden files and prune hidden directories from discovery. - **One dependency-ordering algorithm for imports and `extends`.** Both use the shared dependency-ready scheduler defined below. This is settled independently of the existing implementations and includes refactoring both resolvers. OD-1 (recursion default), OD-4 (discovery boundaries and cycles), OD-5 (policy precedence), and OD-6 (absent-import evidence) are settled in the sections below. OD-2 and OD-3 remain open. `follow` and `hidden` are the proposed field spellings; their applicability to a retained `glob` selector remains part of OD-3. ## Decisions Resolved From Ansible The comparison covers `ansible.builtin.find`, `ansible.builtin.include_vars`, `ansible.builtin.fileglob`, and `ansible.windows.win_find`. A consistent approach means agreement among the relevant interfaces that offer the capability, not that every Ansible module exposes it. Existing ConfigKit contracts and the already selected optional-by-default direction remain authoritative; this is not a general Ansible compatibility mode. The following choices are settled: - **Separate traversal from filename filtering.** Directory roots and filename filters have separate roles; filters operate on basenames in visited directories. A directory failing the filename filter does not prune its descendants. This agrees with the directory-search interfaces above. See [find's parameters](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters) and the [include_vars implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py). - **Include an optional integer `depth` control.** All three directory-search interfaces use that spelling. Direct files are at depth 1, and each additional directory adds one. Omission permits unlimited depth when recursion is enabled. See [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html#parameters) and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters). - **Treat `depth: 0` as unlimited.** This follows `include_vars` and agrees with the current [find implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py) and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1). The latter two document unlimited omission more clearly than explicit zero; this choice uses source-confirmed behavior, not a claimed universal documented Ansible guarantee. Negative values remain a separate open decision. - **Disabling recursion takes precedence over depth.** With `recursive: false`, search only direct files, even if a valid larger or unlimited depth is supplied. Depth does not enable recursion. Both [find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters) and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters) explicitly use this rule where a separate recursion switch exists. - **Directory searches select files, not directory documents.** Directories are traversal containers, even when their names match `pattern`. This agrees with the default file selection in the find modules and the file-oriented [fileglob lookup](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/fileglob_lookup.html). Existing regular-file and containment rules still govern opening selected sources, together with the selected discovery controls. The accepted `recursive` spelling is retained. The formerly tentative `max_depth` spelling becomes `depth`. Resource budgets remain separate from depth, and the current limits still apply to an unlimited-depth search. ## Shared Dependency-Ready Ordering **Status: settled for imports and `extends`; not yet implemented.** Use deterministic dependency-ready ordering: Kahn's topological-sort algorithm with a min-heap of eligible nodes. Both resolvers must use the same algorithm in the same way, through one shared internal scheduling implementation, common tie-breaking rules, and common cycle-path logic. ### Scheduling Rule A dependency edge means that the dependent cannot complete until its dependency has completed. For each dependency graph: 1. Count each node's outstanding dependencies and identify initially eligible nodes: those with none. 2. Choose the eligible node with the smallest stable ordering key. 3. Append it to the schedule, decrement its dependents' outstanding counts, and immediately make newly eligible nodes available for the next selection. 4. Continue until all nodes are scheduled. If unfinished nodes remain with no eligible node, reject the graph and identify an actual dependency cycle. Use stable source or definition identities to construct ordering keys, with the same deterministic comparison rule in both resolvers. Map iteration order and discovery-assigned source IDs must not supply the tie-breaker. Specify key construction for source namespaces and logical definition paths during implementation planning; it must distinguish nodes within each scheduled graph. Construct and validate a complete schedule for the available graph before applying its resolution operations. A blocked dependent is not necessarily a cycle member: diagnostics should identify the actual cycle using shared, deterministic cycle-path logic. Source discovery and parsing may already have occurred; this rule is not a promise to avoid reading invalid configurations. ### Scope And Integration Discovery and resolution are separate operations. An importing source must be read to discover its imports, but its contribution need not be assembled first. Discovery order is not a substitute for the selected resolution schedule. Both resolvers construct dependency graphs and apply resolver-specific operations through the shared scheduler. Imports retain strict composition; `extends` retains inheritance overlay behavior. Scheduling does not authorize replacement within a strict composition tier or change precedence between tiers. Shared dependencies can be scheduled once where valid; this does not relax the existing prohibition on repeated source inclusion for imports. Preserve item-local helper scopes and enclosing-before-nested inheritance requirements. An imported item completes its helper contributions before being placed as an item. Enclosing inheritance establishes the context in which nested relationships are discovered and resolved. Use the same scheduler at those scope boundaries, with the actual prerequisites represented; do not flatten raw files or mappings in a way that loses their scope. Inheritance still resolves after source assembly and precedence have established the effective document. The integration must account for resulting sequence positions, provenance, history, and diagnostic ordering. It must distinguish source acquisition records from assembly order rather than assuming they are the same. Review both resolvers' contracts and behavioral tests when implementing the new schedule; preserving the old traversal is not a design objective. ### Rationale And Validation Dependency-ready ordering selects the smallest currently eligible node without adding branch preference, batch barriers, or distance-from-root priority. With a fixed total ordering key, it gives the lexicographically smallest valid dependency-first schedule. It can interleave independent branches; branch contiguity is not part of the chosen ordering contract. The comparison considered deterministic depth-first postorder, dependency rounds, and descending longest-path depth as well. All can satisfy dependency correctness, but their additional ordering policies did not justify choosing them over the uniform eligibility rule. This decision gives no preference to the existing implementation or to minimizing pre-1.0 refactoring. The scheduler requires O(E + V log V) time and O(V + E) memory for V nodes and E dependency edges, assuming constant-time key comparisons. Source access, parsing, merging, and provenance work are separate costs. Validate independent branches, unequal chains, branches with multiple dependencies, shared dependencies at equal and unequal depths, cycles with blocked non-cycle dependents, and nested scopes. Equivalent valid graphs and ordering keys must produce identical schedules through the import and inheritance integrations. Verify independence from graph insertion order and preserve each resolver's domain-specific admissibility and combination rules. ## Discovery Semantics ### Filename Matching `pattern` applies to a file's basename, not its full relative path. For example, `*.yaml` matches both `prompts/one.yaml` and `prompts/weather/daily/summary.yaml` when the latter's directory is visited. Recursion changes which directories are visited; it does not change what `*` means. A nonrecursive search considers only files directly inside `dir`. Recommend retaining the familiar `path.Match` wildcard grammar for filename filters. The name `pattern` does not by itself imply regular expressions or automatic dialect detection. Omitting `pattern` selects all filenames, with no implicit extension restriction. Matching grammar and whether the final API needs path-glob expressiveness remain open below. ### Recursive Discovery And Depth Limits Use `recursive: true` for recursive discovery, optionally bounded by `depth`. The motivation is to let authors organize definitions in subdirectories without updating import declarations for every depth. Directory organization should not change resource identity or the effective configuration destination. The selected model uses an explicit search root and recursion flag, rather than introducing recursive `**` syntax or inferring a root from a wildcard prefix. **OD-1: settled.** Omitting `recursive` means false: search only direct files. Authors explicitly opt into descendants. This makes the search scope visible while keeping recursion independent of the permissive defaults for absence, symlinks, and hidden entries. The settled depth convention is: - Files directly inside the search root are at depth `1`. - Each additional directory increases depth by one. - Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit when recursion is enabled; existing loader resource limits still apply. - `recursive: false` selects only direct files regardless of a valid depth value. - Depth controls discovery only, not namespaces, identities, or destinations. Remaining directive validation choices are listed below. Recursive discovery must fit the existing filesystem containment, repeated-source, and load-budget contracts rather than silently changing them. In particular, examined entries, including nonmatches, still consume the visit budget; a small number of matches does not authorize an unbounded filesystem scan. This is an existing ConfigKit constraint, not a new Ansible-derived decision. ### Symlinks And Hidden Entries **OD-4: settled.** Discovery controls are local to the declaration. Directory searches default to `follow: true` and `hidden: true`, subject to the application defaults described below. Authors may disable either behavior independently of recursion, depth, matching, or absence handling. Define hidden entries by a leading dot in the portable source namespace. `hidden: false` skips discovered hidden files and prunes discovered hidden directories. `follow: false` skips discovered symlink files and prevents descent through discovered symlink directories. Explicitly named roots and separate direct-file imports retain their existing source-access semantics. The controls do not implicitly propagate to declarations inside imported files. These are discovery-selection controls, not a load-wide access prohibition. If OD-2 retains `glob` and OD-3 enables these controls on it, filtering applies to wildcard-selected entries; explicitly named path components remain explicit access. This conditional scope rule does not settle either open decision. When following links, detect a canonical directory repeated on the active traversal path and fail with a cycle diagnostic. Do not silently prune the loop and continue. Existing dependency-cycle and repeated-source errors remain in force; discovered aliases do not authorize silently deduplicating imported sources. Directory-cycle handling retains the existing containment and native-filesystem capabilities. These controls narrow selection rather than authorize ignoring errors in a selected source. `required: true` applies after discovery filters: if the search selects no file, it fails. Native filesystems retain their existing capabilities; this feature does not promise link inspection where a supplied filesystem does not expose it. ## Illustrative Examples These examples describe the proposed interface, not configuration accepted by the current release. They assume the consumer has enabled imports and has not changed the library's policy defaults. File paths and directory roots resolve from the declaring file; destinations use the existing relative-placement rules. ### Import One File ```yaml configkit: imports: - path: defaults.yaml ``` Import `defaults.yaml` into the declaring source's mapping scope. Under the proposed default, a missing file contributes nothing. Add `required: true` when its presence is necessary. A present but invalid file still fails loading. ### Recursively Discover Mapping Fragments ```yaml configkit: imports: - dir: services pattern: '*.yaml' recursive: true destination: [services] ``` This discovers files such as `services/api.yaml` and `services/regional/worker.yaml`, but not `services/README.md`. Each file contains named service entries, for example `api: {port: 8080}` or `worker: {concurrency: 4}`. They compose beneath `services` under the existing strict composition rules. Directory names do not create additional mapping keys. Missing directories or no matching files contribute nothing. Adding `depth: 2` limits discovery to direct files and files one directory below `services`: it still selects `services/regional/worker.yaml`, but not `services/regional/east/worker.yaml`. With `recursive: false`, only direct files are selected, even if `depth: 2` remains present. Hidden entries and contained symlinks participate by default. To narrow the search, the declaration can also include: ```yaml follow: false hidden: false ``` These controls narrow discovery below the explicitly named root; the root continues to resolve under the existing source-access rules. ### Require A Recursive Prompt Collection ```yaml configkit: imports: - dir: prompts pattern: '*.prompt.yaml' recursive: true required: true destination: [prompts] placement: sequence-item ``` This selects `prompts/welcome.prompt.yaml` and `prompts/weather/daily/summary.prompt.yaml`, but not `prompts/weather/helpers.yaml`. Each selected file contains one prompt mapping, such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the `prompts` sequence with identity field `id` through the existing `Definition.SequenceCompositions` API. Distinct IDs compose into one sequence; duplicate IDs fail strict composition. Overlays continue replacing sequences. Here `required: true` means at least one file must match across the entire search, not that every visited directory must contain a matching file. The filename convention keeps helper files out of this discovery declaration; it does not prevent a selected file from explicitly importing a helper. Existing repeated-source rules still apply if declarations select overlapping sources. ## Optional Imports By Default The preferred direction is optional-by-default for both explicit files and directory searches, with `required: true` opting into failure on absence. This deliberately supersedes the earlier suggestion to retain required-by-default imports. The table assumes the library's default absence policy. Applications may select a different default using the settled precedence rules below. | Situation | Proposed behavior | | --- | --- | | Missing path; `required` omitted or false | Contribute nothing | | Missing search directory or no matching files; `required` omitted or false | Contribute nothing | | Missing file, missing search directory, or zero matches with `required: true` | Fail with an import diagnostic | | Invalid pattern or malformed import declaration | Fail | | Permission failure, unreadable matched file, or malformed YAML | Fail | | Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules | An optional declaration is permission for absence, not permission to ignore a present contribution that cannot be read or interpreted. Optionality must not turn processing failures into apparently successful partial configuration. This default supports installations with different optional components without requiring ceremony for every potentially absent directory or file. Its tradeoff is that a misspelled path or pattern can also contribute nothing. Authors and applications that need an import to exist can state that requirement explicitly; applications remain responsible for validating their necessary final values. Final validation does not automatically detect every unintentionally omitted contribution. ## Who Chooses Strictness? **OD-5: settled.** Provide load-level defaults for `required`, `follow`, and `hidden`. Resolve each applicable field independently using this precedence: 1. Explicit value on the import declaration. 2. Application-selected default, when supplied. 3. Library default: false for `required`, true for `follow` and `hidden`. Apply this rule to transitive declarations too. A declaration does not inherit these field values from its importing parent. Applications may expose the defaults to operators through their own interfaces; simple consumers need no additional setup. Exact Go API spelling belongs to implementation planning. Explicit declarations remain authoritative, permitting intentional exceptions to stricter application defaults. Defaults are not enforcement controls. Do not add options that override explicit declarations as part of this work without a separate concrete requirement and scope decision. The fields' applicability to each selector remains part of OD-3. ## Evidence For Absent Optional Imports **OD-6: settled.** Record minimal structured inspection evidence when an optional declaration selects nothing: its declaring source/location, selector, and absent outcome. Emit no warning by default and do not fabricate a loaded-source record. This keeps optional contributions quiet during ordinary use while allowing applications to explain omissions during troubleshooting. Decide the exact inspection record shape during implementation planning, reusing existing structures where appropriate and retaining existing inspection contracts. Include ownership tests for the new evidence. This addition does not require applications to reconstruct absent-import provenance themselves. ## Release Boundary Ship one final interface with a hard break in a pre-1.0 minor release, following the [release policy](../policy/release.md#versioning-and-compatibility). There is no transition release, legacy parser path, compatibility switch, or temporary `optional` alias. Release notes should describe the changed behavior and final syntax; this documentation does not imply migration machinery. Whether to support path globs in the final interface remains a feature-scope decision because they express directory selection that basename filters cannot. Any retained form must be justified as a permanent capability, not compatibility scaffolding. No promise is made to preserve every old declaration or use case. ## Open Decisions Only OD-2 and OD-3 remain open. Their identifiers are preserved for continued discussion. The recommendations below reflect the latest review against the architecture principles; they are not settled decisions. In particular, multiple patterns, retaining `glob`, and applying discovery controls to `glob` remain proposals. OD-1, OD-4, OD-5, and OD-6 are settled above. ### OD-2. Matching Language And Selector Expressiveness **Why still open:** Permissiveness does not select a matching grammar. Ansible uses different wildcard and regex languages across its discovery interfaces. Also, basename filters and path globs express different searches; the absence of compatibility support does not decide which capabilities the final API needs. **Recommendation (not yet accepted):** Use the existing case-sensitive Go `path.Match` grammar for basename matching. Accept a scalar `pattern` or a list of patterns; a candidate matching multiple patterns in one declaration is selected once. This does not suppress repeated-source errors from separate inclusion routes. Keep `dir` as a literal root and retain `glob` as a permanent, separate selector for wildcard directory segments and exact-depth path selection. Its justification is useful expressiveness, not old-document compatibility. Apply the new absence policy to every selector; retaining `glob` must not introduce a legacy behavior mode. This supports both simple collection discovery and more selective searches, but leaves two matching forms to explain and maintain. **Alternative:** Offer only `path` and `dir` with a basename `pattern`. This is a smaller final interface and removes `glob` outright, with no alias or rewrite. Users needing selected directory names or depths would use multiple explicit imports or application-side source selection. That is a real capability tradeoff, acceptable if those use cases do not justify a separate selector. **Filter alternatives:** Restrict `pattern` to a scalar, keeping the parser smaller but requiring separate declarations for suffix unions; overlapping selections remain subject to source-repeat errors. An explicit Go regex mode would support richer filters but add another dialect and validation rules. Recommend deferring regex until a concrete requirement justifies it. Regex compatibility with Python or PowerShell is not implied. None of these choices changes the settled all-filenames default when a filter is omitted. ### OD-3. Directive Validation **Why still open:** Permissive discovery does not determine whether malformed or inapplicable option values should be accepted. The draft's existing error boundary remains: invalid declarations fail. We still need to define their valid shapes, rather than infer extra restrictions from that boundary. **Recommendation (not yet accepted):** Accept nonnegative YAML integers for `depth`; zero remains unlimited. Treat negative, null, string, boolean, and fractional values as malformed, even when recursion is disabled. Use exactly one selector per import: `path`, `dir`, or `glob` if selected for the final API. Keep `pattern`, `recursive`, and `depth` specific to `dir`; allow `follow` and `hidden` on both `dir` and `glob` if the latter is retained. Diagnose unknown fields and options that have no meaning for the chosen selector. This makes likely mistakes visible, but shared templates must omit inapplicable fields. It imposes no arbitrary maximum depth. The settled combination `recursive: false` with a valid larger `depth` remains accepted and selects only direct files. The removed `optional` spelling is not accepted under any variant. **Alternative:** Normalize null depth to omission and negative integers to unlimited, and permit unused directory controls on a direct-file declaration. That eases generated configuration but can turn an erroneous bound into a wider search or make a direct-file import appear recursive. These relaxations can be chosen independently; they do not require accepting ambiguous selectors. Another viable selector shape is to allow multiple selectors as an explicit union, but it needs rules for constructing the graph and handling overlaps that one-selector declarations avoid.