From 82b289f4fe0855c19c75cc22ecbb0abeacfcac87 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 18 Oct 2026 16:28:38 +0000 Subject: [PATCH] Clarify import roadmap decisions using Ansible conventions --- docs/roadmap/imports.md | 522 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 522 insertions(+) create mode 100644 docs/roadmap/imports.md diff --git a/docs/roadmap/imports.md b/docs/roadmap/imports.md new file mode 100644 index 0000000..b7898c5 --- /dev/null +++ b/docs/roadmap/imports.md @@ -0,0 +1,522 @@ +# Import Ergonomics And Permissive Defaults + +Status: working draft / brainstorming notes. This document captures a proposed +direction for discussion, not an implemented contract or an implementation plan. +The selected direction separates directory searches, filename matching, +recursion, and absence handling. Optional-by-default imports are the intended +direction. The Ansible-aligned decisions below are settled for this roadmap; +remaining choices have recommendations and alternatives under Open Decisions. +All new syntax below is illustrative and unimplemented. + +## Guiding Philosophy + +ConfigKit should provide predictable configuration mechanisms and avoid imposing +application policy. Its defaults should permit optional contributions to be +absent, with explicit options for increased strictness when needed. That choice +may come from an operator or the consuming application; authored import +declarations can also express requirements on contributions. + +Permissiveness should distinguish absence from invalid supplied configuration: + +> Optional contributions may be absent by default. Consumers can explicitly +> require them. Supplied configuration must still be structurally valid and +> interpretable under the selected composition rules. + +This is a general design direction worth making more explicit and consistent +across ConfigKit. It is not blanket authorization to weaken existing contracts, +ignore errors, or add a global permissiveness switch. Other defaults should be +evaluated individually against this principle. + +## Current Baseline + +The [current import contract](../config.md#explicit-imports) uses `path` or +`glob`, with required imports by default and `optional: true` to permit absence. +Glob matching uses slash-based Go `path.Match` syntax. Recursive `**` syntax is +not supported; authors can match particular depths with patterns such as +`prompts/*.yaml` and `prompts/*/*.yaml`. + +The proposed changes concern naming, discovery, and absence handling. Import +destinations, composition, precedence, and resource identity remain separate +concerns. + +## Proposed Import Vocabulary + +Use a direct-file form and a directory-search form. The latter separates the +search root from its filename filter rather than inferring traversal from a +combined path pattern: + +```yaml +configkit: + imports: + - dir: prompts + pattern: '*.yaml' + recursive: true + destination: [prompts] + placement: sequence-item + - path: defaults.yaml + required: true +``` + +The first declaration searches `prompts` and its descendants for matching +filenames, placing each source mapping as one prompt item, and permits no +matches. The second explicitly requires its target file. Sequence composition +still requires the consumer's existing registration, as explained below. + +The fields have distinct jobs: + +| Field | Proposed meaning | +| --- | --- | +| `path` | Import one named file, relative to the declaring file | +| `dir` | Search a directory, relative to the declaring file | +| `pattern` | Filter candidate files by filename within the searched directories | +| `recursive` | Include descendant directories when true | +| `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive | +| `required` | Fail when the declaration contributes no matching file | +| `destination`, `placement` | Apply the existing placement rules to each selected source | + +This follows the familiar separation of traversal and filtering in Ansible's +[find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html) +and [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html) +interfaces, without adopting all their spellings, defaults, or matching modes. + +## Decisions Resolved From Ansible + +The comparison covers `ansible.builtin.find`, `ansible.builtin.include_vars`, +`ansible.builtin.fileglob`, and `ansible.windows.win_find`. A consistent approach +means agreement among the relevant interfaces that offer the capability, not +that every Ansible module exposes it. Existing ConfigKit contracts and the +already selected optional-by-default direction remain authoritative; this is +not a general Ansible compatibility mode. + +The following choices are settled: + +- **Separate traversal from filename filtering.** Directory roots and filename + filters have separate roles; filters operate on basenames in visited + directories. A directory failing the filename filter does not prune its + descendants. This agrees with the directory-search interfaces above. + See [find's parameters](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters) + and the [include_vars implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py). +- **Include an optional integer `depth` control.** All three directory-search + interfaces use that spelling. Direct files are at depth 1, and each additional + directory adds one. Omission permits unlimited depth when recursion is + enabled. See [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html#parameters) + and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters). +- **Treat `depth: 0` as unlimited.** This follows `include_vars` and agrees with + the current [find implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py) + and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1). + The latter two document unlimited omission more clearly than explicit zero; + this choice uses source-confirmed behavior, not a claimed universal documented + Ansible guarantee. Negative values remain a separate open decision. +- **Disabling recursion takes precedence over depth.** With `recursive: false`, + search only direct files, even if a valid larger or unlimited depth is supplied. + Depth does not enable recursion. Both + [find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters) + and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters) + explicitly use this rule where a separate recursion switch exists. +- **Directory searches select files, not directory documents.** Directories are + traversal containers, even when their names match `pattern`. This agrees with + the default file selection in the find modules and the file-oriented + [fileglob lookup](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/fileglob_lookup.html). + Existing regular-file and symlink rules still govern opening selected sources; + the legacy `glob` contract is not implicitly changed by this new search form. + +The accepted `recursive` spelling is retained. The formerly tentative +`max_depth` spelling becomes `depth`. Resource budgets remain separate from +depth, and the current limits still apply to an unlimited-depth search. + +## Discovery Semantics + +### Filename Matching + +`pattern` applies to a file's basename, not its full relative path. For example, +`*.yaml` matches both `prompts/one.yaml` and +`prompts/weather/daily/summary.yaml` when the latter's directory is visited. +Recursion changes which directories are visited; it does not change what `*` +means. A nonrecursive search considers only files directly inside `dir`. + +Recommend retaining the familiar `path.Match` wildcard grammar for filename +filters. The name `pattern` does not by itself imply regular expressions or +automatic dialect detection. Exact filter defaults and the treatment of +existing path-glob use cases remain open below. + +### Recursive Discovery And Depth Limits + +Use `recursive: true` for recursive discovery, optionally bounded by `depth`. +The motivation is to let authors organize definitions in subdirectories without +updating import declarations for every depth. Directory organization should not +change resource identity or the effective configuration destination. + +The selected model uses an explicit search root and recursion flag, rather than +introducing recursive `**` syntax or inferring a root from a wildcard prefix. +The default when `recursive` is omitted remains open; examples specify it. + +The settled depth convention is: + +- Files directly inside the search root are at depth `1`. +- Each additional directory increases depth by one. +- Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit + when recursion is enabled; existing loader resource limits still apply. +- `recursive: false` selects only direct files regardless of a valid depth value. +- Depth controls discovery only, not namespaces, identities, or destinations. + +Remaining validation and traversal choices are listed below. Recursive discovery +must fit the existing filesystem containment, repeated-source, and load-budget +contracts rather than silently changing them. In particular, examined entries, +including nonmatches, still consume the visit budget; a small number of matches +does not authorize an unbounded filesystem scan. This is an existing ConfigKit +constraint, not a new Ansible-derived decision. + +## Illustrative Examples + +These examples describe the proposed interface, not configuration accepted by +the current release. They assume the consumer has enabled imports. File paths +and directory roots resolve from the declaring file; destinations use the +existing relative-placement rules. + +### Import One File + +```yaml +configkit: + imports: + - path: defaults.yaml +``` + +Import `defaults.yaml` into the declaring source's mapping scope. Under the +proposed default, a missing file contributes nothing. Add `required: true` when +its presence is necessary. A present but invalid file still fails loading. + +### Recursively Discover Mapping Fragments + +```yaml +configkit: + imports: + - dir: services + pattern: '*.yaml' + recursive: true + destination: [services] +``` + +This discovers files such as `services/api.yaml` and +`services/regional/worker.yaml`, but not `services/README.md`. Each file contains +named service entries, for example `api: {port: 8080}` or +`worker: {concurrency: 4}`. They compose beneath `services` under the existing +strict composition rules. Directory names do not create additional mapping +keys. Missing directories or no matching files contribute nothing. + +Adding `depth: 2` limits discovery to direct files and files one directory below +`services`: it still selects `services/regional/worker.yaml`, but not +`services/regional/east/worker.yaml`. With `recursive: false`, only direct files +are selected, even if `depth: 2` remains present. + +### Require A Recursive Prompt Collection + +```yaml +configkit: + imports: + - dir: prompts + pattern: '*.prompt.yaml' + recursive: true + required: true + destination: [prompts] + placement: sequence-item +``` + +This selects `prompts/welcome.prompt.yaml` and +`prompts/weather/daily/summary.prompt.yaml`, but not +`prompts/weather/helpers.yaml`. Each selected file contains one prompt mapping, +such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the +`prompts` sequence with identity field `id` through the existing +`Definition.SequenceCompositions` API. Distinct IDs compose into one sequence; +duplicate IDs fail strict composition. Overlays continue replacing sequences. + +Here `required: true` means at least one file must match across the entire +search, not that every visited directory must contain a matching file. The +filename convention keeps helper files out of this discovery declaration; it +does not prevent a selected file from explicitly importing a helper. Existing +repeated-source rules still apply if declarations select overlapping sources. + +## Optional Imports By Default + +The preferred direction is optional-by-default for both explicit files and +directory searches, with `required: true` opting into failure on absence. This +deliberately supersedes the earlier suggestion to retain required-by-default +imports. + +The table assumes the library's default absence policy. A consumer-selected +default or enforcement option remains open below. + +| Situation | Proposed behavior | +| --- | --- | +| Missing path; `required` omitted or false | Contribute nothing | +| Missing search directory or no matching files; `required` omitted or false | Contribute nothing | +| Missing file, missing search directory, or zero matches with `required: true` | Fail with an import diagnostic | +| Invalid pattern or malformed import declaration | Fail | +| Permission failure, unreadable matched file, or malformed YAML | Fail | +| Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules | + +An optional declaration is permission for absence, not permission to ignore a +present contribution that cannot be read or interpreted. Optionality must not +turn processing failures into apparently successful partial configuration. + +This default supports installations with different optional components without +requiring ceremony for every potentially absent directory or file. Its tradeoff +is that a misspelled path or pattern can also contribute nothing. Authors and +applications that need an import to exist can state that requirement explicitly; +applications remain responsible for validating their necessary final values. +Final validation does not automatically detect every unintentionally omitted +contribution. + +## Who Chooses Strictness? + +The general philosophy permits operators and consuming applications to choose +stricter behavior. Import authors can express local requirements through +`required: true`. This does not yet specify an application-wide override API. + +Keep the distinction between choosing defaults and overriding explicit intent. +If an operator or application should be able to strengthen or relax a requirement +declared in an imported document, define that capability and its precedence +separately. Optional-by-default alone does not imply that an authored requirement +can be overridden, or that requirements propagate to transitive imports. + +## Migration + +Changing absence defaults changes behavior even when existing YAML contains no +optionality field. Renaming keys also changes the accepted syntax. Pre-1.0 is a +useful time to reconsider these contracts, but adoption still needs explicit +migration guidance. + +To preserve current intent, currently required imports would acquire +`required: true`; existing `optional: true` declarations could omit the new field +or use `required: false`. A simple `glob: prompts/*.yaml` becomes `dir: prompts` +plus `pattern: '*.yaml'` and `recursive: false`, preserving its discovery scope. +This is not a mechanical rename of `glob` to `pattern`: current globs can also +select particular directory names or depths. Their compatibility path must be +settled before removing any existing form. + +## Open Decisions + +The recommendations below are not settled decisions. Each concerns either +inconsistent Ansible behavior or a ConfigKit-specific policy without a direct +Ansible counterpart. They do not reopen the accepted absence policy, placement, +strict composition, or source-version rules. + +### 1. Recursion Default + +**Ansible evidence:** `find` and `win_find` default to nonrecursive searches; +`include_vars` recursively loads directories by default. There is no common +default across these interfaces. See their parameter references above. + +**Recommendation:** Default `recursive` to false. This keeps discovery local +unless the declaration explicitly requests descendants and makes migration from +simple current globs straightforward. The cost is one extra field for recursive +collections. + +**Alternative:** Default to true, following `include_vars`. Collection authors +could reorganize files into subdirectories without revising declarations, but +adding a subdirectory could also add contributions to an existing import. + +### 2. Matching Language And Filter Shape + +**Ansible evidence:** `find` supports shell patterns and optional regex; +`include_vars.files_matching` is a regex; `win_find` uses PowerShell wildcards or +regex. Their case and matching details differ. There is no single exact grammar. + +**Recommendation:** Keep one scalar `pattern` using ConfigKit's existing +case-sensitive Go `path.Match` grammar, applied to basenames. This reuses a +familiar contract, including escapes and character classes, without introducing +regex mode or platform-dependent case rules. It cannot express every regex or +an arbitrary union of suffixes in one declaration; separate declarations remain +available, subject to existing repeated-source rules. + +**Alternative:** Add an explicit regex mode and/or a list of patterns whose +matches are unioned within one declaration. This better supports filters such +as both `.yaml` and `.yml` without overlapping imports, but adds syntax, matching +rules, and validation. Regex would need a specified Go grammar rather than a +promise of Python or PowerShell compatibility. Exclusion filters could accompany +that design, but are not assumed in the initial recommendation. + +### 3. Omitted Filename Filter + +**Ansible evidence:** The find modules permit unrestricted filename selection. +`include_vars` also has an extension policy, defaulting to `json`, `yaml`, and +`yml`, with separate handling of unknown extensions. Its loading behavior is +not equivalent to a general-purpose find operation. + +**Recommendation:** Omitted `pattern` means all filenames, with no implicit +extension filter. This keeps selection independent of filename conventions and +supports extensionless configuration. A selected README or other invalid file +still causes a load error; authors of mixed directories should supply a filter. + +**Alternative:** Default discovery to `.yaml` and `.yml`, with an explicit way +to select all files or other extensions. This suits typical configuration +directories and avoids incidental README selection, but adds an extension +policy and an override mechanism that explicit `path` imports do not need. + +### 4. Invalid Depth Values + +**Ansible evidence:** Positive depth, unlimited omission, and disabled-recursion +precedence have useful common semantics. The implementations do not provide a +consistent negative-depth contract; accepting a value is not evidence of a +useful shared meaning. See the implementation links above. + +**Recommendation:** Accept nonnegative YAML integers only. Zero means unlimited; +negative, null, string, boolean, or fractional values are malformed declarations, +even when recursion is disabled. This follows ConfigKit's typed directive style +and gives likely mistakes a direct diagnostic without imposing an arbitrary +maximum depth. + +**Alternative:** Treat negative integers as unlimited too, and null as omission. +That can ease generated configurations but adds redundant spellings and risks +turning an erroneous bound into a wider search. Neither interpretation is +required by the settled rule that disabled recursion ignores a valid bound. + +### 5. Hidden Files And Directories + +**Ansible evidence:** The find modules expose hidden-file filtering, while +`include_vars` has no equivalent hidden-entry exclusion in its directory walk. +Also, filtering a hidden entry is not necessarily the same as pruning a hidden +directory's descendants. See +[find](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py) +and [include_vars](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py). + +**Recommendation:** Give dot-prefixed names no special treatment: visit their +directories within the selected depth and select files by `pattern`. This fits +the existing matcher and avoids an implicit exclusion. Broad recursive searches +may include configuration in hidden directories, so the search root and filter +should reflect the intended collection. + +**Alternative:** Add `hidden`, defaulting to false, with an explicit definition +covering both hidden-file selection and hidden-directory pruning. This can +exclude incidental metadata directories, but adds policy and a platform choice +between dot prefixes and filesystem hidden attributes. Callers would need to +opt in when their collections intentionally use hidden names. + +### 6. Directory Symlinks And Traversal Cycles + +**Ansible evidence:** The find modules default to not following directory links; +the inspected `include_vars` implementation walks with link following enabled. +There is no common default to adopt. + +**Recommendation:** Follow contained directory symlinks using ConfigKit's +existing canonicalization and containment rules, preserving the current ability +to select through contained symlinks. For the new recursive walk, detect a +canonical directory repeated on the active traversal path and report a cycle. +Do not silently deduplicate imported files reached through aliases; existing +source-repeat errors remain applicable. This supports linked collections but +requires directory-cycle handling in addition to import-cycle handling. + +**Alternative:** Add a traversal `follow` option defaulting to false, skipping +directory symlinks encountered below the search root unless enabled. Keep +explicitly named roots and file targets under existing source-opening rules. +This reduces accidental traversal through linked trees, but introduces a new +distinction between discovery and explicit access and requires callers to opt +in for linked collections. Both choices retain native-filesystem semantics; +neither promises symlink capabilities absent from the supplied filesystem. + +### 7. Match Ordering + +**Ansible evidence:** `include_vars` sorts directories and filenames; +`win_find` documents results sorted by path. POSIX `find` describes top-down +searching and does not promise the same global ordering. These are not identical +ordering contracts. + +**Recommendation:** Sort all selected slash-relative paths lexicographically +within each declaration, then use existing depth-first import expansion. This +extends ConfigKit's existing glob order and stabilizes sequence assembly and +diagnostics. It requires collecting bounded candidates before expansion. + +**Alternative:** Load direct files first, then visit sorted subdirectories +recursively. This offers a natural directory-oriented traversal and can begin +loading sooner, but differs from existing global match ordering and can change +sequence item positions. Neither choice grants replacement precedence within +strict composition. + +### 8. Application And Operator Strictness Controls + +**Ansible evidence:** Task failure controls and module-specific options are not +a direct counterpart to defaults for ConfigKit's transitive import graph. +Ansible cannot settle this public API choice. + +**Recommendation:** Add a load-level default for whether imports are required, +used only when a declaration omits `required`. The library default stays false; +an application can choose true and expose that choice to operators through its +own interface. Apply the default independently to each declaration, including +transitive imports. Explicit true or false remains authoritative. This enables +stricter deployments while preserving intentionally optional imports; it does +not enforce that every authored optional declaration succeeds. + +**Alternative:** Offer an enforcement mode requiring every declaration to +match, including those explicitly marked `required: false`. That provides a +stronger audit policy but overrides authors' intent and needs clear precedence +and diagnostics. A smaller alternative is authored requirements only for now; +it avoids a public option but leaves applications unable to choose a common +default without editing source declarations. + +### 9. Evidence For Absent Optional Imports + +**Ansible evidence:** `fileglob` returns an empty result for no matches; +`include_vars` reports included filenames; the find modules expose richer search +results. There is no common inspection model for an absent import declaration. + +**Recommendation:** Record minimal structured inspection evidence for an +optional declaration that selects nothing: its declaring source/location, +selector, and absent outcome. Emit no warning by default and do not fabricate a +loaded-source record. This helps distinguish intentional absence from a typo, +but requires an inspection API decision and associated ownership tests. Design +the exact record shape during implementation planning, using existing evidence +structures where appropriate. + +**Alternative:** Add no new inspection surface initially; retain loaded-source +evidence and let `required: true` turn absence into a diagnostic when needed. +This keeps the feature smaller, but successful optional omissions are harder to +investigate. Existing inspection contracts must remain intact either way. + +### 10. Selector Combinations And Directory-Only Options + +**Ansible evidence:** `include_vars` separates file-only and directory-only +arguments, while the find modules are directory-search interfaces. This does +not settle how ConfigKit should combine its new form with retained path globs. + +**Recommendation:** Exactly one selector per declaration: `path`, `dir`, or +`glob` if retained. Use a literal directory root; `pattern`, `recursive`, and +`depth` apply only to `dir`. Diagnose irrelevant fields rather than silently +ignoring them. This makes declarations unambiguous and keeps the roles of file +selection and traversal visible, at the cost of templates needing to omit +inapplicable fields. This is a proposed validation rule, not an already settled +restriction. + +**Alternative:** Permit directory-search controls on a `path` declaration and +ignore them, retaining exclusive selectors. Shared templates become easier, +but a declaration can appear recursive without being so. Wildcard directory +roots are another viable extension: useful for selecting collections, but they +need root-expansion, depth-origin, ordering, and overlap rules. Retaining `glob` +is the simpler way to preserve existing wildcard-directory use cases. + +### 11. Compatibility And Migration + +**Ansible evidence:** There is no Ansible precedent that determines ConfigKit's +compatibility commitments or migration schedule. + +**Recommendation:** Retain `glob` alongside `dir` because it can select directory +segments and exact depths that basename filters cannot express. Introduce +`required` while accepting explicit `optional` as its inverse for one documented +transition release. Reject declarations containing both fields to avoid two +authorities for the same choice. This retains existing expressiveness and eases +adoption, at the cost of temporary parser and documentation complexity. + +The intended optional default applies when both fields are absent, including +legacy `path` and `glob` declarations; old spelling alone does not preserve the +old default. The application default, if adopted, also applies to this absent +case. Explicit legacy `optional: false` maps to `required: true`, and explicit +`optional: true` maps to `required: false`. Release notes must call out the +behavior change and show how to preserve existing requirements. Under the +[release policy](../policy/release.md#versioning-and-compatibility), this belongs +in a pre-1.0 minor release, not a patch release. + +**Alternative:** Remove `optional` immediately in that minor release while +retaining `glob`. This establishes one vocabulary sooner, but every declaration +using the old field needs migration before it can load. If a transition is +chosen, accepting both fields when they agree is also viable; it helps generated +documents at the cost of extra normalization rules. Removing `glob` too would +need an expressiveness decision and migration for wildcard directory segments; +it cannot be treated as a simple field rename.