ConfigKit/docs/roadmap/imports.md

Import Ergonomics And Permissive Defaults

Status: working draft / brainstorming notes. This document captures a proposed direction for discussion, not an implemented contract or an implementation plan. The selected direction separates directory searches, filename matching, recursion, and absence handling. Optional-by-default imports are the intended direction. The consumer-directed and Ansible-aligned decisions below are settled; remaining choices have recommendations and alternatives under Open Decisions. All new syntax below is illustrative and unimplemented.

Guiding Philosophy

ConfigKit should provide predictable configuration mechanisms and avoid imposing application policy. Its defaults should permit optional contributions to be absent, with explicit options for increased strictness when needed. That choice may come from an operator or the consuming application; authored import declarations can also express requirements on contributions.

Permissiveness should distinguish absence from invalid supplied configuration:

Optional contributions may be absent by default. Consumers can explicitly require them. Supplied configuration must still be structurally valid and interpretable under the selected composition rules.

This is a general design direction worth making more explicit and consistent across ConfigKit. It is not blanket authorization to weaken existing contracts, ignore errors, or add a global permissiveness switch. Other defaults should be evaluated individually against this principle.

Discovery should accept hidden entries and follow symlinks by default, with options to narrow either behavior. An omitted filename filter should not impose an implicit extension restriction. These defaults do not waive containment, source validity, composition, or resource-limit rules. Recursion remains a separate choice about search scope, not a measure of permissiveness.

This is a hard-breaking pre-1.0 change. Do not build temporary aliases, translation layers, compatibility modes, or staged deprecation support. Evaluate the final interface on its own merits, without retaining syntax solely to ease upgrades.

Current Baseline

The current import contract uses path or glob, with required imports by default and optional: true to permit absence. Glob matching uses slash-based Go path.Match syntax. Recursive ** syntax is not supported; authors can match particular depths with patterns such as prompts/*.yaml and prompts/*/*.yaml.

Current imports expand depth-first in declaration order, with each ordinary source contributing its body before its imports. The existing sibling inheritance resolver instead follows dependency chains before resolving dependents, using sorted sibling names and an explicit stack. Neither implements the shared ordering selected below.

The proposed changes concern naming, discovery, absence handling, and uniform dependency ordering for imports and extends. Import destinations, strict composition, inheritance overlay semantics, tier precedence, and resource identity retain their distinct roles.

Proposed Import Vocabulary

Use a direct-file form and a directory-search form. The latter separates the search root from its filename filter rather than inferring traversal from a combined path pattern:

configkit:
  imports:
    - dir: prompts
      pattern: '*.yaml'
      recursive: true
      destination: [prompts]
      placement: sequence-item
    - path: defaults.yaml
      required: true

The first declaration searches prompts and its descendants for matching filenames, placing each source mapping as one prompt item, and permits no matches. The second explicitly requires its target file. Sequence composition still requires the consumer's existing registration, as explained below.

The fields have distinct jobs:

Field	Proposed meaning
path	Import one named file, relative to the declaring file
dir	Search a directory, relative to the declaring file
pattern	Filter candidate files by filename within the searched directories
recursive	Include descendant directories when true
depth	Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive
follow	Follow symlinks by default; false disables following within the discovery scope defined below
hidden	Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below
required	Fail when the declaration contributes no matching file
destination, placement	Apply the existing placement rules to each selected source

This follows the familiar separation of traversal and filtering in Ansible's find and include_vars interfaces, without adopting all their spellings, defaults, or matching modes.

Decisions Resolved By Project Direction

• No transition support. Use the final interface immediately in the selected pre-1.0 minor release. Remove optional in favor of required; do not accept it as an alias or normalize old declarations at runtime.
• Optional contributions by default. required: true opts into failure on absence. Supplied invalid configuration continues to fail.
• No implicit filename restriction. Omitting pattern selects all filenames. Authors can narrow selection explicitly. There is no default YAML extension list; extensionless sources remain discoverable.
• Follow symlinks by default, with an opt-out. Provide follow: false for callers who want discovery without symlink following. Applicable containment and filesystem contracts remain in force.
• Include hidden entries by default, with an opt-out. Provide hidden: false to exclude hidden files and prune hidden directories from discovery.
• One dependency-ordering algorithm for imports and extends. Both use the shared dependency-ready scheduler defined below. This is settled independently of the existing implementations and includes refactoring both resolvers.

The defaults and availability of these controls are settled. Their exact scope at explicitly named roots, the definition of hidden names across filesystems, and application-level policy precedence are described under Open Decisions. follow and hidden are the proposed field spellings.

Decisions Resolved From Ansible

The comparison covers ansible.builtin.find, ansible.builtin.include_vars, ansible.builtin.fileglob, and ansible.windows.win_find. A consistent approach means agreement among the relevant interfaces that offer the capability, not that every Ansible module exposes it. Existing ConfigKit contracts and the already selected optional-by-default direction remain authoritative; this is not a general Ansible compatibility mode.

The following choices are settled:

• Separate traversal from filename filtering. Directory roots and filename filters have separate roles; filters operate on basenames in visited directories. A directory failing the filename filter does not prune its descendants. This agrees with the directory-search interfaces above. See find's parameters and the include_vars implementation.
• Include an optional integer depth control. All three directory-search interfaces use that spelling. Direct files are at depth 1, and each additional directory adds one. Omission permits unlimited depth when recursion is enabled. See include_vars and win_find.
• Treat depth: 0 as unlimited. This follows include_vars and agrees with the current find implementation and win_find implementation. The latter two document unlimited omission more clearly than explicit zero; this choice uses source-confirmed behavior, not a claimed universal documented Ansible guarantee. Negative values remain a separate open decision.
• Disabling recursion takes precedence over depth. With recursive: false, search only direct files, even if a valid larger or unlimited depth is supplied. Depth does not enable recursion. Both find and win_find explicitly use this rule where a separate recursion switch exists.
• Directory searches select files, not directory documents. Directories are traversal containers, even when their names match pattern. This agrees with the default file selection in the find modules and the file-oriented fileglob lookup. Existing regular-file and containment rules still govern opening selected sources, together with the selected discovery controls.

The accepted recursive spelling is retained. The formerly tentative max_depth spelling becomes depth. Resource budgets remain separate from depth, and the current limits still apply to an unlimited-depth search.

Shared Dependency-Ready Ordering

Status: settled for imports and extends; not yet implemented. Use deterministic dependency-ready ordering: Kahn's topological-sort algorithm with a min-heap of eligible nodes. Both resolvers must use the same algorithm in the same way, through one shared internal scheduling implementation, common tie-breaking rules, and common cycle-path logic.

Scheduling Rule

A dependency edge means that the dependent cannot complete until its dependency has completed. For each dependency graph:

1. Count each node's outstanding dependencies and identify initially eligible nodes: those with none.
2. Choose the eligible node with the smallest stable ordering key.
3. Append it to the schedule, decrement its dependents' outstanding counts, and immediately make newly eligible nodes available for the next selection.
4. Continue until all nodes are scheduled. If unfinished nodes remain with no eligible node, reject the graph and identify an actual dependency cycle.

Use stable source or definition identities to construct ordering keys, with the same deterministic comparison rule in both resolvers. Map iteration order and discovery-assigned source IDs must not supply the tie-breaker. Specify key construction for source namespaces and logical definition paths during implementation planning; it must distinguish nodes within each scheduled graph.

Construct and validate a complete schedule for the available graph before applying its resolution operations. A blocked dependent is not necessarily a cycle member: diagnostics should identify the actual cycle using shared, deterministic cycle-path logic. Source discovery and parsing may already have occurred; this rule is not a promise to avoid reading invalid configurations.

Scope And Integration

Discovery and resolution are separate operations. An importing source must be read to discover its imports, but its contribution need not be assembled first. Discovery order is not a substitute for the selected resolution schedule.

Both resolvers construct dependency graphs and apply resolver-specific operations through the shared scheduler. Imports retain strict composition; extends retains inheritance overlay behavior. Scheduling does not authorize replacement within a strict composition tier or change precedence between tiers. Shared dependencies can be scheduled once where valid; this does not relax the existing prohibition on repeated source inclusion for imports.

Preserve item-local helper scopes and enclosing-before-nested inheritance requirements. An imported item completes its helper contributions before being placed as an item. Enclosing inheritance establishes the context in which nested relationships are discovered and resolved. Use the same scheduler at those scope boundaries, with the actual prerequisites represented; do not flatten raw files or mappings in a way that loses their scope. Inheritance still resolves after source assembly and precedence have established the effective document.

The integration must account for resulting sequence positions, provenance, history, and diagnostic ordering. It must distinguish source acquisition records from assembly order rather than assuming they are the same. Review both resolvers' contracts and behavioral tests when implementing the new schedule; preserving the old traversal is not a design objective.

Rationale And Validation

Dependency-ready ordering selects the smallest currently eligible node without adding branch preference, batch barriers, or distance-from-root priority. With a fixed total ordering key, it gives the lexicographically smallest valid dependency-first schedule. It can interleave independent branches; branch contiguity is not part of the chosen ordering contract.

The comparison considered deterministic depth-first postorder, dependency rounds, and descending longest-path depth as well. All can satisfy dependency correctness, but their additional ordering policies did not justify choosing them over the uniform eligibility rule. This decision gives no preference to the existing implementation or to minimizing pre-1.0 refactoring.

The scheduler requires O(E + V log V) time and O(V + E) memory for V nodes and E dependency edges, assuming constant-time key comparisons. Source access, parsing, merging, and provenance work are separate costs.

Validate independent branches, unequal chains, branches with multiple dependencies, shared dependencies at equal and unequal depths, cycles with blocked non-cycle dependents, and nested scopes. Equivalent valid graphs and ordering keys must produce identical schedules through the import and inheritance integrations. Verify independence from graph insertion order and preserve each resolver's domain-specific admissibility and combination rules.

Discovery Semantics

Filename Matching

pattern applies to a file's basename, not its full relative path. For example, *.yaml matches both prompts/one.yaml and prompts/weather/daily/summary.yaml when the latter's directory is visited. Recursion changes which directories are visited; it does not change what * means. A nonrecursive search considers only files directly inside dir.

Recommend retaining the familiar path.Match wildcard grammar for filename filters. The name pattern does not by itself imply regular expressions or automatic dialect detection. Omitting pattern selects all filenames, with no implicit extension restriction. Matching grammar and whether the final API needs path-glob expressiveness remain open below.

Recursive Discovery And Depth Limits

Use recursive: true for recursive discovery, optionally bounded by depth. The motivation is to let authors organize definitions in subdirectories without updating import declarations for every depth. Directory organization should not change resource identity or the effective configuration destination.

The selected model uses an explicit search root and recursion flag, rather than introducing recursive ** syntax or inferring a root from a wildcard prefix. The default when recursive is omitted remains open; examples specify it.

The settled depth convention is:

• Files directly inside the search root are at depth 1.
• Each additional directory increases depth by one.
• Omitting depth, or setting it to 0, imposes no caller-selected depth limit when recursion is enabled; existing loader resource limits still apply.
• recursive: false selects only direct files regardless of a valid depth value.
• Depth controls discovery only, not namespaces, identities, or destinations.

Remaining validation and traversal choices are listed below. Recursive discovery must fit the existing filesystem containment, repeated-source, and load-budget contracts rather than silently changing them. In particular, examined entries, including nonmatches, still consume the visit budget; a small number of matches does not authorize an unbounded filesystem scan. This is an existing ConfigKit constraint, not a new Ansible-derived decision.

Symlinks And Hidden Entries

Directory searches default to follow: true and hidden: true. Authors may disable either behavior independently of recursion, depth, matching, or absence handling. hidden: false excludes hidden files and prunes hidden directories; follow: false disables following symlinks within the discovery scope. The precise boundary for explicit roots and file links remains open below.

These controls narrow selection rather than authorize ignoring errors in a selected source. required: true applies after discovery filters: if the search selects no file, it fails. Native filesystems retain their existing capabilities; this feature does not promise link inspection where a supplied filesystem does not expose it.

Illustrative Examples

These examples describe the proposed interface, not configuration accepted by the current release. They assume the consumer has enabled imports. File paths and directory roots resolve from the declaring file; destinations use the existing relative-placement rules.

Import One File

configkit:
  imports:
    - path: defaults.yaml

Import defaults.yaml into the declaring source's mapping scope. Under the proposed default, a missing file contributes nothing. Add required: true when its presence is necessary. A present but invalid file still fails loading.

Recursively Discover Mapping Fragments

configkit:
  imports:
    - dir: services
      pattern: '*.yaml'
      recursive: true
      destination: [services]

This discovers files such as services/api.yaml and services/regional/worker.yaml, but not services/README.md. Each file contains named service entries, for example api: {port: 8080} or worker: {concurrency: 4}. They compose beneath services under the existing strict composition rules. Directory names do not create additional mapping keys. Missing directories or no matching files contribute nothing.

Adding depth: 2 limits discovery to direct files and files one directory below services: it still selects services/regional/worker.yaml, but not services/regional/east/worker.yaml. With recursive: false, only direct files are selected, even if depth: 2 remains present.

Hidden entries and contained symlinks participate by default. To narrow the search, the declaration can also include:

      follow: false
      hidden: false

The exact treatment of an explicitly named root is an open scope decision, not an additional change to the settled defaults.

Require A Recursive Prompt Collection

configkit:
  imports:
    - dir: prompts
      pattern: '*.prompt.yaml'
      recursive: true
      required: true
      destination: [prompts]
      placement: sequence-item

This selects prompts/welcome.prompt.yaml and prompts/weather/daily/summary.prompt.yaml, but not prompts/weather/helpers.yaml. Each selected file contains one prompt mapping, such as id: prompt.welcome and text: Welcome. The consumer registers the prompts sequence with identity field id through the existing Definition.SequenceCompositions API. Distinct IDs compose into one sequence; duplicate IDs fail strict composition. Overlays continue replacing sequences.

Here required: true means at least one file must match across the entire search, not that every visited directory must contain a matching file. The filename convention keeps helper files out of this discovery declaration; it does not prevent a selected file from explicitly importing a helper. Existing repeated-source rules still apply if declarations select overlapping sources.

Optional Imports By Default

The preferred direction is optional-by-default for both explicit files and directory searches, with required: true opting into failure on absence. This deliberately supersedes the earlier suggestion to retain required-by-default imports.

The table assumes the library's default absence policy. A consumer-selected default or enforcement option remains open below.

Situation	Proposed behavior
Missing path; required omitted or false	Contribute nothing
Missing search directory or no matching files; required omitted or false	Contribute nothing
Missing file, missing search directory, or zero matches with required: true	Fail with an import diagnostic
Invalid pattern or malformed import declaration	Fail
Permission failure, unreadable matched file, or malformed YAML	Fail
Composition conflict or other invalid supplied configuration	Fail under the applicable existing rules

An optional declaration is permission for absence, not permission to ignore a present contribution that cannot be read or interpreted. Optionality must not turn processing failures into apparently successful partial configuration.

This default supports installations with different optional components without requiring ceremony for every potentially absent directory or file. Its tradeoff is that a misspelled path or pattern can also contribute nothing. Authors and applications that need an import to exist can state that requirement explicitly; applications remain responsible for validating their necessary final values. Final validation does not automatically detect every unintentionally omitted contribution.

Who Chooses Strictness?

The general philosophy permits operators and consuming applications to choose stricter behavior. Import authors can express local requirements through required: true. This does not yet specify an application-wide override API.

Keep the distinction between choosing defaults and overriding explicit intent. If an operator or application should be able to strengthen or relax a requirement declared in an imported document, define that capability and its precedence separately. Optional-by-default alone does not imply that an authored requirement can be overridden, or that requirements propagate to transitive imports.

Release Boundary

Ship one final interface with a hard break in a pre-1.0 minor release, following the release policy. There is no transition release, legacy parser path, compatibility switch, or temporary optional alias. Release notes should describe the changed behavior and final syntax; this documentation does not imply migration machinery.

Whether to support path globs in the final interface remains a feature-scope decision because they express directory selection that basename filters cannot. Any retained form must be justified as a permanent capability, not compatibility scaffolding. No promise is made to preserve every old declaration or use case.

Open Decisions

The recommendations below are not settled decisions. The new principles resolve transition support, omitted filename filtering, and the defaults and opt-out availability for symlinks and hidden entries. Shared dependency-ready ordering for imports and extends is also settled. Six decision areas remain; closely related validation details are grouped together without treating them as implicitly approved. Each has at least one viable alternative.

1. Recursion Default

Why still open: Recursion determines search scope independently of absence handling and permissiveness. Ansible's find and win_find default to nonrecursive searches; include_vars recursively loads directories by default.

Recommendation: Default recursive to false. A declaration explicitly chooses whether subdirectories contribute. This makes the search scope visible, at the cost of one extra field for recursive collections.

Alternative: Default to true, following include_vars. Collection authors could reorganize files into subdirectories without revising declarations, but adding a subdirectory could also add contributions to an existing import. Neither choice changes the settled follow, hidden, or required defaults.

2. Matching Language And Selector Expressiveness

Why still open: Permissiveness does not select a matching grammar. Ansible uses different wildcard and regex languages across its discovery interfaces. Also, basename filters and path globs express different searches; the absence of compatibility support does not decide which capabilities the final API needs.

Recommendation: Use one scalar pattern with the existing case-sensitive Go path.Match grammar, applied to basenames. Keep dir as a literal root and retain glob as a permanent, separate selector for wildcard directory segments and exact-depth path selection. Its justification is useful expressiveness, not old-document compatibility. Apply the new absence policy to every selector; retaining glob must not introduce a legacy behavior mode. This supports both simple collection discovery and more selective searches, but leaves two matching forms to explain and maintain.

Alternative: Offer only path and dir with a basename pattern. This is a smaller final interface and removes glob outright, with no alias or rewrite. Users needing selected directory names or depths would use multiple explicit imports or application-side source selection. That is a real capability tradeoff, acceptable if those use cases do not justify a separate selector.

Filter alternative: Allow a list of patterns unioned within one declaration, and/or an explicit Go regex mode. This supports suffix unions or richer filters without overlapping imports, but adds syntax, matching rules, and validation. Separate scalar declarations are simpler, though overlapping selections remain subject to source-repeat errors. Regex compatibility with Python or PowerShell is not implied. None of these choices changes the settled all-filenames default when a filter is omitted.

3. Directive Validation

Why still open: Permissive discovery does not determine whether malformed or inapplicable option values should be accepted. The draft's existing error boundary remains: invalid declarations fail. We still need to define their valid shapes, rather than infer extra restrictions from that boundary.

Recommendation: Accept nonnegative YAML integers for depth; zero remains unlimited. Treat negative, null, string, boolean, and fractional values as malformed, even when recursion is disabled. Use exactly one selector per import: path, dir, or glob if selected for the final API. Keep directory search options (pattern, recursive, depth, follow, and hidden) on dir and diagnose them on other forms. This makes likely mistakes visible, but shared templates must omit inapplicable fields. It imposes no arbitrary maximum depth. The removed optional spelling is not accepted under any variant.

Alternative: Normalize null depth to omission and negative integers to unlimited, and permit unused directory controls on a direct-file declaration. That eases generated configuration but can turn an erroneous bound into a wider search or make a direct-file import appear recursive. These relaxations can be chosen independently; they do not require accepting ambiguous selectors. Another viable selector shape is to allow multiple selectors as an explicit union, but it needs rules for constructing the graph and handling overlaps that one-selector declarations avoid.

4. Discovery Control Boundaries And Cycles

Why still open: Including hidden entries and following symlinks by default, with options to disable both, is settled. The principles do not define hidden names across filesystems, the reach of an opt-out at explicitly named roots, or the response to recursive directory-link cycles. Ansible does not provide one consistent set of answers across the inspected interfaces.

Recommendation: Define hidden entries by a leading dot for the portable source namespace. With hidden: false, skip such files and prune such directories encountered below the search root. With follow: false, skip encountered symlink files and do not descend through encountered symlink directories. An explicitly named dir root still resolves under existing source-access rules, as does a separate path import. The controls affect this declaration's discovery; they do not propagate to imports declared by selected files. This gives a simple selection rule and preserves deliberate explicit access, but is not a ban on all hidden or symlink access in the load.

When following links, detect a canonical directory repeated on the active traversal path and fail with a cycle diagnostic. Keep existing errors for imported sources reached repeatedly through aliases. This makes cycles visible without silently deduplicating configuration. It adds directory-cycle handling alongside the existing source-cycle rules; it does not change containment or native-filesystem capabilities.

Alternative: Make the controls govern the explicit search root and every component used to reach it too, and classify hidden entries using host attributes where available. This can serve a broader exclusion policy, but adds filesystem inspection requirements and differences between host and abstract filesystems. A consumer-wide enforcement policy would also need explicit precedence rather than being inferred from one declaration's flags.

Cycle alternative: Prune an edge back to an active ancestor directory and continue. This permits link-rich trees without failing for an irrelevant loop, but can hide a layout mistake and needs clearly defined inspection evidence. It would not justify suppressing repeated-source errors for files actually selected through distinct routes.

5. Application And Operator Policy Precedence

Why still open: The library defaults and authored opt-outs are settled. Whether consumers can set different defaults or enforce restrictions over explicit declarations remains an API choice. Ansible's task controls are not a direct counterpart to policy over ConfigKit's transitive import graph.

Recommendation: Provide load-level defaults for required, follow, and hidden, applied independently when a declaration omits the corresponding field. Library defaults remain false, true, and true respectively. Explicit per-declaration values win; imports do not inherit these values from their parent declaration. Applications can expose defaults to operators through their own interfaces. This enables stricter deployment choices while retaining intentional exceptions, but does not enforce a universal exclusion rule.

Alternative: Offer enforcement options that override explicit declarations, for example requiring every import to match or forbidding discovery through symlinks throughout the load. This supports centrally imposed restrictions but needs clear conflict diagnostics and scope, particularly for explicit sources. A smaller alternative is declaration-level controls only initially: simpler API, but consumers cannot choose common defaults without changing declarations.

6. Evidence For Absent Optional Imports

Why still open: Permitting absence does not settle how it should be exposed for inspection. Ansible's file lookup, variable loader, and find modules return different evidence, with no common absent-import model.

Recommendation: Record minimal structured inspection evidence when an optional declaration selects nothing: declaring source/location, selector, and absent outcome. Emit no warning by default and do not fabricate a loaded-source record. This helps distinguish intentional absence from a typo, but requires an inspection API decision and ownership tests. Decide its exact shape during implementation planning, reusing existing structures where appropriate.

Alternative: Add no new inspection surface initially; retain loaded-source evidence and let required: true turn absence into a diagnostic when needed. This keeps the feature smaller, but successful optional omissions are harder to investigate. Existing inspection contracts remain intact either way.