ConfigKit/docs/roadmap/imports.md

# Import Ergonomics And Permissive Defaults

Status: working draft / brainstorming notes. This document captures a proposed
direction for discussion, not an implemented contract or an implementation plan.
The selected direction separates directory searches, filename matching,
recursion, and absence handling. Optional-by-default imports are the intended
direction. The consumer-directed and Ansible-aligned decisions below are settled;
remaining choices have recommendations and alternatives under Open Decisions.
All new syntax below is illustrative and unimplemented.

## Guiding Philosophy

ConfigKit should provide predictable configuration mechanisms and avoid imposing
application policy. Its defaults should permit optional contributions to be
absent, with explicit options for increased strictness when needed. That choice
may come from an operator or the consuming application; authored import
declarations can also express requirements on contributions.

Permissiveness should distinguish absence from invalid supplied configuration:

> Optional contributions may be absent by default. Consumers can explicitly
> require them. Supplied configuration must still be structurally valid and
> interpretable under the selected composition rules.

This is a general design direction worth making more explicit and consistent
across ConfigKit. It is not blanket authorization to weaken existing contracts,
ignore errors, or add a global permissiveness switch. Other defaults should be
evaluated individually against this principle.

Discovery should accept hidden entries and follow symlinks by default, with
options to narrow either behavior. An omitted filename filter should not impose
an implicit extension restriction. These defaults do not waive containment,
source validity, composition, or resource-limit rules. Recursion remains a
separate choice about search scope, not a measure of permissiveness.

This is a hard-breaking pre-1.0 change. Do not build temporary aliases,
translation layers, compatibility modes, or staged deprecation support. Evaluate
the final interface on its own merits, without retaining syntax solely to ease
upgrades.

## Current Baseline

The [current import contract](../config.md#explicit-imports) uses `path` or
`glob`, with required imports by default and `optional: true` to permit absence.
Glob matching uses slash-based Go `path.Match` syntax. Recursive `**` syntax is
not supported; authors can match particular depths with patterns such as
`prompts/*.yaml` and `prompts/*/*.yaml`.

Current imports expand depth-first in declaration order, with each ordinary
source contributing its body before its imports. The existing
[sibling inheritance resolver](../config.md#sibling-inheritance) instead follows
dependency chains before resolving dependents, using sorted sibling names and
an explicit stack. Neither implements the shared ordering selected below.

The proposed changes concern naming, discovery, absence handling, and uniform
dependency ordering for imports and `extends`. Import destinations, strict
composition, inheritance overlay semantics, tier precedence, and resource
identity retain their distinct roles.

## Proposed Import Vocabulary

Use a direct-file form and a directory-search form. The latter separates the
search root from its filename filter rather than inferring traversal from a
combined path pattern:

```yaml
configkit:
  imports:
    - dir: prompts
      pattern: '*.yaml'
      recursive: true
      destination: [prompts]
      placement: sequence-item
    - path: defaults.yaml
      required: true
```

The first declaration searches `prompts` and its descendants for matching
filenames, placing each source mapping as one prompt item, and permits no
matches. The second explicitly requires its target file. Sequence composition
still requires the consumer's existing registration, as explained below.

The fields have distinct jobs:

| Field | Proposed meaning |
| --- | --- |
| `path` | Import one named file, relative to the declaring file |
| `dir` | Search a directory, relative to the declaring file |
| `pattern` | Filter candidate files by filename within the searched directories |
| `recursive` | Default false; include descendant directories when true |
| `depth` | Bound directory-search depth; 1 means direct files, 0 or omission means unlimited when recursive |
| `follow` | Follow symlinks by default; false disables following within the discovery scope defined below |
| `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below |
| `required` | Fail when the declaration contributes no matching file |
| `destination`, `placement` | Apply the existing placement rules to each selected source |

This follows the familiar separation of traversal and filtering in Ansible's
[find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html)
and [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html)
interfaces, without adopting all their spellings, defaults, or matching modes.

## Decisions Resolved By Project Direction

- **No transition support.** Use the final interface immediately in the selected
  pre-1.0 minor release. Remove `optional` in favor of `required`; do not accept
  it as an alias or normalize old declarations at runtime.
- **Optional contributions by default.** `required: true` opts into failure on
  absence. Supplied invalid configuration continues to fail.
- **No implicit filename restriction.** Omitting `pattern` selects all
  filenames. Authors can narrow selection explicitly. There is no default YAML
  extension list; extensionless sources remain discoverable.
- **Follow symlinks by default, with an opt-out.** Provide `follow: false` for
  callers who want discovery without symlink following. Applicable containment
  and filesystem contracts remain in force.
- **Include hidden entries by default, with an opt-out.** Provide `hidden: false`
  to exclude hidden files and prune hidden directories from discovery.
- **One dependency-ordering algorithm for imports and `extends`.** Both use the
  shared dependency-ready scheduler defined below. This is settled independently
  of the existing implementations and includes refactoring both resolvers.

OD-1 (recursion default), OD-4 (discovery boundaries and cycles), OD-5 (policy
precedence), and OD-6 (absent-import evidence) are settled in the sections below.
OD-2 and OD-3 remain open. `follow` and `hidden` are the proposed field spellings;
their applicability to a retained `glob` selector remains part of OD-3.

## Decisions Resolved From Ansible

The comparison covers `ansible.builtin.find`, `ansible.builtin.include_vars`,
`ansible.builtin.fileglob`, and `ansible.windows.win_find`. A consistent approach
means agreement among the relevant interfaces that offer the capability, not
that every Ansible module exposes it. Existing ConfigKit contracts and the
already selected optional-by-default direction remain authoritative; this is
not a general Ansible compatibility mode.

The following choices are settled:

- **Separate traversal from filename filtering.** Directory roots and filename
  filters have separate roles; filters operate on basenames in visited
  directories. A directory failing the filename filter does not prune its
  descendants. This agrees with the directory-search interfaces above.
  See [find's parameters](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters)
  and the [include_vars implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py).
- **Include an optional integer `depth` control.** All three directory-search
  interfaces use that spelling. Direct files are at depth 1, and each additional
  directory adds one. Omission permits unlimited depth when recursion is
  enabled. See [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html#parameters)
  and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters).
- **Treat `depth: 0` as unlimited.** This follows `include_vars` and agrees with
  the current [find implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py)
  and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1).
  The latter two document unlimited omission more clearly than explicit zero;
  this choice uses source-confirmed behavior, not a claimed universal documented
  Ansible guarantee. Negative values remain a separate open decision.
- **Disabling recursion takes precedence over depth.** With `recursive: false`,
  search only direct files, even if a valid larger or unlimited depth is supplied.
  Depth does not enable recursion. Both
  [find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters)
  and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters)
  explicitly use this rule where a separate recursion switch exists.
- **Directory searches select files, not directory documents.** Directories are
  traversal containers, even when their names match `pattern`. This agrees with
  the default file selection in the find modules and the file-oriented
  [fileglob lookup](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/fileglob_lookup.html).
  Existing regular-file and containment rules still govern opening selected
  sources, together with the selected discovery controls.

The accepted `recursive` spelling is retained. The formerly tentative
`max_depth` spelling becomes `depth`. Resource budgets remain separate from
depth, and the current limits still apply to an unlimited-depth search.

## Shared Dependency-Ready Ordering

**Status: settled for imports and `extends`; not yet implemented.** Use
deterministic dependency-ready ordering: Kahn's topological-sort algorithm with
a min-heap of eligible nodes. Both resolvers must use the same algorithm in the
same way, through one shared internal scheduling implementation, common
tie-breaking rules, and common cycle-path logic.

### Scheduling Rule

A dependency edge means that the dependent cannot complete until its dependency
has completed. For each dependency graph:

1. Count each node's outstanding dependencies and identify initially eligible
   nodes: those with none.
2. Choose the eligible node with the smallest stable ordering key.
3. Append it to the schedule, decrement its dependents' outstanding counts, and
   immediately make newly eligible nodes available for the next selection.
4. Continue until all nodes are scheduled. If unfinished nodes remain with no
   eligible node, reject the graph and identify an actual dependency cycle.

Use stable source or definition identities to construct ordering keys, with
the same deterministic comparison rule in both resolvers. Map iteration order
and discovery-assigned source IDs must not supply the tie-breaker. Specify key
construction for source namespaces and logical definition paths during
implementation planning; it must distinguish nodes within each scheduled graph.

Construct and validate a complete schedule for the available graph before
applying its resolution operations. A blocked dependent is not necessarily a
cycle member: diagnostics should identify the actual cycle using shared,
deterministic cycle-path logic. Source discovery and parsing may already have
occurred; this rule is not a promise to avoid reading invalid configurations.

### Scope And Integration

Discovery and resolution are separate operations. An importing source must be
read to discover its imports, but its contribution need not be assembled first.
Discovery order is not a substitute for the selected resolution schedule.

Both resolvers construct dependency graphs and apply resolver-specific operations
through the shared scheduler. Imports retain strict composition; `extends`
retains inheritance overlay behavior. Scheduling does not authorize replacement
within a strict composition tier or change precedence between tiers. Shared
dependencies can be scheduled once where valid; this does not relax the existing
prohibition on repeated source inclusion for imports.

Preserve item-local helper scopes and enclosing-before-nested inheritance
requirements. An imported item completes its helper contributions before being
placed as an item. Enclosing inheritance establishes the context in which nested
relationships are discovered and resolved. Use the same scheduler at those
scope boundaries, with the actual prerequisites represented; do not flatten raw
files or mappings in a way that loses their scope. Inheritance still resolves
after source assembly and precedence have established the effective document.

The integration must account for resulting sequence positions, provenance,
history, and diagnostic ordering. It must distinguish source acquisition records
from assembly order rather than assuming they are the same. Review both
resolvers' contracts and behavioral tests when implementing the new schedule;
preserving the old traversal is not a design objective.

### Rationale And Validation

Dependency-ready ordering selects the smallest currently eligible node without
adding branch preference, batch barriers, or distance-from-root priority. With
a fixed total ordering key, it gives the lexicographically smallest valid
dependency-first schedule. It can interleave independent branches; branch
contiguity is not part of the chosen ordering contract.

The comparison considered deterministic depth-first postorder, dependency
rounds, and descending longest-path depth as well. All can satisfy dependency
correctness, but their additional ordering policies did not justify choosing
them over the uniform eligibility rule. This decision gives no preference to
the existing implementation or to minimizing pre-1.0 refactoring.

The scheduler requires O(E + V log V) time and O(V + E) memory for V nodes and
E dependency edges, assuming constant-time key comparisons. Source access,
parsing, merging, and provenance work are separate costs.

Validate independent branches, unequal chains, branches with multiple
dependencies, shared dependencies at equal and unequal depths, cycles with
blocked non-cycle dependents, and nested scopes. Equivalent valid graphs and
ordering keys must produce identical schedules through the import and
inheritance integrations. Verify independence from graph insertion order and
preserve each resolver's domain-specific admissibility and combination rules.

## Discovery Semantics

### Filename Matching

`pattern` applies to a file's basename, not its full relative path. For example,
`*.yaml` matches both `prompts/one.yaml` and
`prompts/weather/daily/summary.yaml` when the latter's directory is visited.
Recursion changes which directories are visited; it does not change what `*`
means. A nonrecursive search considers only files directly inside `dir`.

Recommend retaining the familiar `path.Match` wildcard grammar for filename
filters. The name `pattern` does not by itself imply regular expressions or
automatic dialect detection. Omitting `pattern` selects all filenames, with no
implicit extension restriction. Matching grammar and whether the final API
needs path-glob expressiveness remain open below.

### Recursive Discovery And Depth Limits

Use `recursive: true` for recursive discovery, optionally bounded by `depth`.
The motivation is to let authors organize definitions in subdirectories without
updating import declarations for every depth. Directory organization should not
change resource identity or the effective configuration destination.

The selected model uses an explicit search root and recursion flag, rather than
introducing recursive `**` syntax or inferring a root from a wildcard prefix.

**OD-1: settled.** Omitting `recursive` means false: search only direct files.
Authors explicitly opt into descendants. This makes the search scope visible
while keeping recursion independent of the permissive defaults for absence,
symlinks, and hidden entries.

The settled depth convention is:

- Files directly inside the search root are at depth `1`.
- Each additional directory increases depth by one.
- Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit
  when recursion is enabled; existing loader resource limits still apply.
- `recursive: false` selects only direct files regardless of a valid depth value.
- Depth controls discovery only, not namespaces, identities, or destinations.

Remaining directive validation choices are listed below. Recursive discovery
must fit the existing filesystem containment, repeated-source, and load-budget
contracts rather than silently changing them. In particular, examined entries,
including nonmatches, still consume the visit budget; a small number of matches
does not authorize an unbounded filesystem scan. This is an existing ConfigKit
constraint, not a new Ansible-derived decision.

### Symlinks And Hidden Entries

**OD-4: settled.** Discovery controls are local to the declaration. Directory
searches default to `follow: true` and `hidden: true`, subject to the application
defaults described below. Authors may disable either behavior independently of
recursion, depth, matching, or absence handling.

Define hidden entries by a leading dot in the portable source namespace.
`hidden: false` skips discovered hidden files and prunes discovered hidden
directories. `follow: false` skips discovered symlink files and prevents descent
through discovered symlink directories. Explicitly named roots and separate
direct-file imports retain their existing source-access semantics. The controls
do not implicitly propagate to declarations inside imported files.

These are discovery-selection controls, not a load-wide access prohibition.
If OD-2 retains `glob` and OD-3 enables these controls on it, filtering applies
to wildcard-selected entries; explicitly named path components remain explicit
access. This conditional scope rule does not settle either open decision.

When following links, detect a canonical directory repeated on the active
traversal path and fail with a cycle diagnostic. Do not silently prune the loop
and continue. Existing dependency-cycle and repeated-source errors remain in
force; discovered aliases do not authorize silently deduplicating imported
sources. Directory-cycle handling retains the existing containment and
native-filesystem capabilities.

These controls narrow selection rather than authorize ignoring errors in a
selected source. `required: true` applies after discovery filters: if the search
selects no file, it fails. Native filesystems retain their existing capabilities;
this feature does not promise link inspection where a supplied filesystem does
not expose it.

## Illustrative Examples

These examples describe the proposed interface, not configuration accepted by
the current release. They assume the consumer has enabled imports and has not
changed the library's policy defaults. File paths and directory roots resolve
from the declaring file; destinations use the
existing relative-placement rules.

### Import One File

```yaml
configkit:
  imports:
    - path: defaults.yaml
```

Import `defaults.yaml` into the declaring source's mapping scope. Under the
proposed default, a missing file contributes nothing. Add `required: true` when
its presence is necessary. A present but invalid file still fails loading.

### Recursively Discover Mapping Fragments

```yaml
configkit:
  imports:
    - dir: services
      pattern: '*.yaml'
      recursive: true
      destination: [services]
```

This discovers files such as `services/api.yaml` and
`services/regional/worker.yaml`, but not `services/README.md`. Each file contains
named service entries, for example `api: {port: 8080}` or
`worker: {concurrency: 4}`. They compose beneath `services` under the existing
strict composition rules. Directory names do not create additional mapping
keys. Missing directories or no matching files contribute nothing.

Adding `depth: 2` limits discovery to direct files and files one directory below
`services`: it still selects `services/regional/worker.yaml`, but not
`services/regional/east/worker.yaml`. With `recursive: false`, only direct files
are selected, even if `depth: 2` remains present.

Hidden entries and contained symlinks participate by default. To narrow the
search, the declaration can also include:

```yaml
      follow: false
      hidden: false
```

These controls narrow discovery below the explicitly named root; the root
continues to resolve under the existing source-access rules.

### Require A Recursive Prompt Collection

```yaml
configkit:
  imports:
    - dir: prompts
      pattern: '*.prompt.yaml'
      recursive: true
      required: true
      destination: [prompts]
      placement: sequence-item
```

This selects `prompts/welcome.prompt.yaml` and
`prompts/weather/daily/summary.prompt.yaml`, but not
`prompts/weather/helpers.yaml`. Each selected file contains one prompt mapping,
such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the
`prompts` sequence with identity field `id` through the existing
`Definition.SequenceCompositions` API. Distinct IDs compose into one sequence;
duplicate IDs fail strict composition. Overlays continue replacing sequences.

Here `required: true` means at least one file must match across the entire
search, not that every visited directory must contain a matching file. The
filename convention keeps helper files out of this discovery declaration; it
does not prevent a selected file from explicitly importing a helper. Existing
repeated-source rules still apply if declarations select overlapping sources.

## Optional Imports By Default

The preferred direction is optional-by-default for both explicit files and
directory searches, with `required: true` opting into failure on absence. This
deliberately supersedes the earlier suggestion to retain required-by-default
imports.

The table assumes the library's default absence policy. Applications may select
a different default using the settled precedence rules below.

| Situation | Proposed behavior |
| --- | --- |
| Missing path; `required` omitted or false | Contribute nothing |
| Missing search directory or no matching files; `required` omitted or false | Contribute nothing |
| Missing file, missing search directory, or zero matches with `required: true` | Fail with an import diagnostic |
| Invalid pattern or malformed import declaration | Fail |
| Permission failure, unreadable matched file, or malformed YAML | Fail |
| Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules |

An optional declaration is permission for absence, not permission to ignore a
present contribution that cannot be read or interpreted. Optionality must not
turn processing failures into apparently successful partial configuration.

This default supports installations with different optional components without
requiring ceremony for every potentially absent directory or file. Its tradeoff
is that a misspelled path or pattern can also contribute nothing. Authors and
applications that need an import to exist can state that requirement explicitly;
applications remain responsible for validating their necessary final values.
Final validation does not automatically detect every unintentionally omitted
contribution.

## Who Chooses Strictness?

**OD-5: settled.** Provide load-level defaults for `required`, `follow`, and
`hidden`. Resolve each applicable field independently using this precedence:

1. Explicit value on the import declaration.
2. Application-selected default, when supplied.
3. Library default: false for `required`, true for `follow` and `hidden`.

Apply this rule to transitive declarations too. A declaration does not inherit
these field values from its importing parent. Applications may expose the
defaults to operators through their own interfaces; simple consumers need no
additional setup. Exact Go API spelling belongs to implementation planning.

Explicit declarations remain authoritative, permitting intentional exceptions
to stricter application defaults. Defaults are not enforcement controls. Do not
add options that override explicit declarations as part of this work without a
separate concrete requirement and scope decision. The fields' applicability to
each selector remains part of OD-3.

## Evidence For Absent Optional Imports

**OD-6: settled.** Record minimal structured inspection evidence when an
optional declaration selects nothing: its declaring source/location, selector,
and absent outcome. Emit no warning by default and do not fabricate a
loaded-source record. This keeps optional contributions quiet during ordinary
use while allowing applications to explain omissions during troubleshooting.

Decide the exact inspection record shape during implementation planning,
reusing existing structures where appropriate and retaining existing inspection
contracts. Include ownership tests for the new evidence. This addition does
not require applications to reconstruct absent-import provenance themselves.

## Release Boundary

Ship one final interface with a hard break in a pre-1.0 minor release, following
the [release policy](../policy/release.md#versioning-and-compatibility). There is
no transition release, legacy parser path, compatibility switch, or temporary
`optional` alias. Release notes should describe the changed behavior and final
syntax; this documentation does not imply migration machinery.

Whether to support path globs in the final interface remains a feature-scope
decision because they express directory selection that basename filters cannot.
Any retained form must be justified as a permanent capability, not compatibility
scaffolding. No promise is made to preserve every old declaration or use case.

## Open Decisions

Only OD-2 and OD-3 remain open. Their identifiers are preserved for continued
discussion. The recommendations below reflect the latest review against the
architecture principles; they are not settled decisions. In particular,
multiple patterns, retaining `glob`, and applying discovery controls to `glob`
remain proposals. OD-1, OD-4, OD-5, and OD-6 are settled above.

### OD-2. Matching Language And Selector Expressiveness

**Why still open:** Permissiveness does not select a matching grammar. Ansible
uses different wildcard and regex languages across its discovery interfaces.
Also, basename filters and path globs express different searches; the absence
of compatibility support does not decide which capabilities the final API needs.

**Recommendation (not yet accepted):** Use the existing case-sensitive Go
`path.Match` grammar for basename matching. Accept a scalar `pattern` or a list
of patterns; a candidate matching multiple patterns in one declaration is
selected once. This does not suppress repeated-source errors from separate
inclusion routes. Keep `dir` as a literal root and retain `glob` as a permanent,
separate selector for wildcard directory segments
and exact-depth path selection. Its justification is useful expressiveness,
not old-document compatibility. Apply the new absence policy to every selector;
retaining `glob` must not introduce a legacy behavior mode. This supports both
simple collection discovery and more selective searches, but leaves two
matching forms to explain and maintain.

**Alternative:** Offer only `path` and `dir` with a basename `pattern`. This is
a smaller final interface and removes `glob` outright, with no alias or rewrite.
Users needing selected directory names or depths would use multiple explicit
imports or application-side source selection. That is a real capability
tradeoff, acceptable if those use cases do not justify a separate selector.

**Filter alternatives:** Restrict `pattern` to a scalar, keeping the parser
smaller but requiring separate declarations for suffix unions; overlapping
selections remain subject to source-repeat errors. An explicit Go regex mode
would support richer filters but add another dialect and validation rules.
Recommend deferring regex until a concrete requirement justifies it. Regex
compatibility with Python or PowerShell is not implied. None of these choices
changes the settled all-filenames default when a filter is omitted.

### OD-3. Directive Validation

**Why still open:** Permissive discovery does not determine whether malformed or
inapplicable option values should be accepted. The draft's existing error
boundary remains: invalid declarations fail. We still need to define their
valid shapes, rather than infer extra restrictions from that boundary.

**Recommendation (not yet accepted):** Accept nonnegative YAML integers for
`depth`; zero remains unlimited. Treat negative, null, string, boolean, and fractional values as
malformed, even when recursion is disabled. Use exactly one selector per
import: `path`, `dir`, or `glob` if selected for the final API. Keep
`pattern`, `recursive`, and `depth` specific to `dir`; allow `follow` and `hidden`
on both `dir` and `glob` if the latter is retained. Diagnose unknown fields and
options that have no meaning for the chosen selector. This makes likely mistakes
visible, but shared templates must omit inapplicable fields. It imposes no arbitrary maximum
depth. The settled combination `recursive: false` with a valid larger `depth`
remains accepted and selects only direct files. The removed `optional` spelling
is not accepted under any variant.

**Alternative:** Normalize null depth to omission and negative integers to
unlimited, and permit unused directory controls on a direct-file declaration.
That eases generated configuration but can turn an erroneous bound into a wider
search or make a direct-file import appear recursive. These relaxations can be
chosen independently; they do not require accepting ambiguous selectors.
Another viable selector shape is to allow multiple selectors as an explicit
union, but it needs rules for constructing the graph and handling overlaps that
one-selector declarations avoid.