712 lines
37 KiB
Markdown
712 lines
37 KiB
Markdown
# Import Ergonomics And Permissive Defaults
|
||
|
||
Status: working draft / brainstorming notes. This document captures a proposed
|
||
direction for discussion, not an implemented contract or an implementation plan.
|
||
The selected direction separates directory searches, filename matching,
|
||
recursion, and absence handling. Optional-by-default imports are the intended
|
||
direction. The consumer-directed and Ansible-aligned decisions below are settled;
|
||
API spelling and implementation details remain for implementation planning.
|
||
All new syntax below is illustrative and unimplemented.
|
||
|
||
## Guiding Philosophy
|
||
|
||
ConfigKit should provide predictable configuration mechanisms and avoid imposing
|
||
application policy. Its defaults should permit optional contributions to be
|
||
absent, with explicit options for increased strictness when needed. That choice
|
||
may come from an operator or the consuming application; authored import
|
||
declarations can also express requirements on contributions.
|
||
|
||
Permissiveness should distinguish absence from invalid supplied configuration:
|
||
|
||
> Optional contributions may be absent by default. Consumers can explicitly
|
||
> require them. Supplied configuration must still be structurally valid and
|
||
> interpretable under the selected composition rules.
|
||
|
||
This is a general design direction worth making more explicit and consistent
|
||
across ConfigKit. It is not blanket authorization to weaken existing contracts,
|
||
ignore errors, or add a global permissiveness switch. Other defaults should be
|
||
evaluated individually against this principle.
|
||
|
||
Discovery should accept hidden entries and follow symlinks by default, with
|
||
options to narrow either behavior. An omitted filename filter should not impose
|
||
an implicit extension restriction. These defaults do not waive containment,
|
||
source validity, composition, or resource-limit rules. Recursion remains a
|
||
separate choice about search scope, not a measure of permissiveness.
|
||
|
||
This is a hard-breaking pre-1.0 change. Do not build temporary aliases,
|
||
translation layers, compatibility modes, or staged deprecation support. Evaluate
|
||
the final interface on its own merits, without retaining syntax solely to ease
|
||
upgrades.
|
||
|
||
## Current Baseline
|
||
|
||
The [current import contract](../config.md#explicit-imports) uses `path` or
|
||
`glob`, with required imports by default and `optional: true` to permit absence.
|
||
Glob matching uses slash-based Go `path.Match` syntax. Recursive `**` syntax is
|
||
not supported; authors can match particular depths with patterns such as
|
||
`prompts/*.yaml` and `prompts/*/*.yaml`.
|
||
|
||
Current imports expand depth-first in declaration order, with each ordinary
|
||
source contributing its body before its imports. The existing
|
||
[sibling inheritance resolver](../config.md#sibling-inheritance) instead follows
|
||
dependency chains before resolving dependents, using sorted sibling names and
|
||
an explicit stack. Neither implements the shared ordering selected below.
|
||
|
||
The proposed changes concern naming, discovery, absence handling, and uniform
|
||
dependency ordering for imports and `extends`. Import destinations, strict
|
||
composition, inheritance overlay semantics, tier precedence, and resource
|
||
identity retain their distinct roles.
|
||
|
||
## Proposed Import Vocabulary
|
||
|
||
Use direct-file, directory-search, and path-glob forms. Directory searches
|
||
separate literal search roots from filename filters; path globs additionally
|
||
support wildcard directory segments and selection at particular depths:
|
||
|
||
```yaml
|
||
configkit:
|
||
imports:
|
||
- dir: prompts
|
||
pattern: '*.yaml'
|
||
recursive: true
|
||
destination: [prompts]
|
||
placement: sequence-item
|
||
- path: defaults.yaml
|
||
required: true
|
||
```
|
||
|
||
The first declaration searches `prompts` and its descendants for matching
|
||
filenames, placing each source mapping as one prompt item, and permits no
|
||
matches. The second explicitly requires its target file. Sequence composition
|
||
still requires the consumer's existing registration, as explained below.
|
||
|
||
The fields have distinct jobs:
|
||
|
||
| Field | Proposed meaning |
|
||
| --- | --- |
|
||
| `path` | Import a named file or list of named files, relative to the declaring file |
|
||
| `dir` | Search a literal directory or list of literal directories, relative to the declaring file |
|
||
| `pattern` | A filename wildcard or list of wildcards; select files matching any effective pattern, using the default precedence below |
|
||
| `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file |
|
||
| `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results |
|
||
| `recursive` | Library default false, overridable by an application default; include descendant directories when effectively true |
|
||
| `depth` | Bound directory-search depth; supplying it requires effective recursion to be enabled; 1 means direct files, 0 or omission means unlimited when recursive |
|
||
| `follow` | Follow symlinks by default; false disables following within the discovery scope defined below |
|
||
| `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below |
|
||
| `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search |
|
||
| `destination`, `placement` | Apply the existing placement rules to each selected source |
|
||
|
||
This follows the familiar separation of traversal and filtering in Ansible's
|
||
[find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html)
|
||
and [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html)
|
||
interfaces, without adopting all their spellings, defaults, or matching modes.
|
||
|
||
## Decisions Resolved By Project Direction
|
||
|
||
- **No transition support.** Use the final interface immediately in the selected
|
||
pre-1.0 minor release. Remove `optional` in favor of `required`; do not accept
|
||
it as an alias or normalize old declarations at runtime.
|
||
- **Optional contributions by default.** `required: true` opts into failure on
|
||
absence. Supplied invalid configuration continues to fail.
|
||
- **No library-imposed filename restriction.** Omitting `pattern` selects all
|
||
filenames unless the application supplies a default pattern. Authors can
|
||
override that default explicitly. There is no library default YAML extension
|
||
list; extensionless sources remain discoverable.
|
||
- **Follow symlinks by default, with an opt-out.** Provide `follow: false` for
|
||
callers who want discovery without symlink following. Applicable containment
|
||
and filesystem contracts remain in force.
|
||
- **Include hidden entries by default, with an opt-out.** Provide `hidden: false`
|
||
to exclude hidden files and prune hidden directories from discovery.
|
||
- **One dependency-ordering algorithm for imports and `extends`.** Both use the
|
||
shared dependency-ready scheduler defined below. This is settled independently
|
||
of the existing implementations and includes refactoring both resolvers.
|
||
|
||
OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-3
|
||
(directive validation), OD-4 (discovery boundaries and cycles), OD-5 (policy
|
||
precedence), and OD-6 (absent-import evidence) are settled in the sections below.
|
||
Supplying `depth` requires effective recursion to be enabled, including when
|
||
enabled through an application default. This supersedes the earlier direction
|
||
to accept but ignore depth when recursion is disabled.
|
||
|
||
## Decisions Resolved From Ansible
|
||
|
||
The comparison covers `ansible.builtin.find`, `ansible.builtin.include_vars`,
|
||
`ansible.builtin.fileglob`, and `ansible.windows.win_find`. A consistent approach
|
||
means agreement among the relevant interfaces that offer the capability, not
|
||
that every Ansible module exposes it. Existing ConfigKit contracts and the
|
||
already selected optional-by-default direction remain authoritative; this is
|
||
not a general Ansible compatibility mode.
|
||
|
||
The following choices are settled:
|
||
|
||
- **Separate traversal from filename filtering.** Directory roots and filename
|
||
filters have separate roles; filters operate on basenames in visited
|
||
directories. A directory failing the filename filter does not prune its
|
||
descendants. This agrees with the directory-search interfaces above.
|
||
See [find's parameters](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters)
|
||
and the [include_vars implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py).
|
||
- **Include an optional integer `depth` control.** All three directory-search
|
||
interfaces use that spelling. Direct files are at depth 1, and each additional
|
||
directory adds one. Omission permits unlimited depth when recursion is
|
||
enabled. See [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html#parameters)
|
||
and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters).
|
||
- **Treat `depth: 0` as unlimited.** This follows `include_vars` and agrees with
|
||
the current [find implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py)
|
||
and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1).
|
||
The latter two document unlimited omission more clearly than explicit zero;
|
||
this choice uses source-confirmed behavior, not a claimed universal documented
|
||
Ansible guarantee. Negative values are invalid under the settled directive
|
||
validation rules below.
|
||
- **Directory searches select files, not directory documents.** Directories are
|
||
traversal containers, even when their names match `pattern`. This agrees with
|
||
the default file selection in the find modules and the file-oriented
|
||
[fileglob lookup](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/fileglob_lookup.html).
|
||
Existing regular-file and containment rules still govern opening selected
|
||
sources, together with the selected discovery controls.
|
||
|
||
The accepted `recursive` spelling is retained. The formerly tentative
|
||
`max_depth` spelling becomes `depth`. Resource budgets remain separate from
|
||
depth, and the current limits still apply to an unlimited-depth search.
|
||
|
||
## Shared Dependency-Ready Ordering
|
||
|
||
**Status: settled for imports and `extends`; not yet implemented.** Use
|
||
deterministic dependency-ready ordering: Kahn's topological-sort algorithm with
|
||
a min-heap of eligible nodes. Both resolvers must use the same algorithm in the
|
||
same way, through one shared internal scheduling implementation, common
|
||
tie-breaking rules, and common cycle-path logic.
|
||
|
||
### Scheduling Rule
|
||
|
||
A dependency edge means that the dependent cannot complete until its dependency
|
||
has completed. For each dependency graph:
|
||
|
||
1. Count each node's outstanding dependencies and identify initially eligible
|
||
nodes: those with none.
|
||
2. Choose the eligible node with the smallest stable ordering key.
|
||
3. Append it to the schedule, decrement its dependents' outstanding counts, and
|
||
immediately make newly eligible nodes available for the next selection.
|
||
4. Continue until all nodes are scheduled. If unfinished nodes remain with no
|
||
eligible node, reject the graph and identify an actual dependency cycle.
|
||
|
||
Use stable source or definition identities to construct ordering keys, with
|
||
the same deterministic comparison rule in both resolvers. Map iteration order
|
||
and discovery-assigned source IDs must not supply the tie-breaker. Specify key
|
||
construction for source namespaces and logical definition paths during
|
||
implementation planning; it must distinguish nodes within each scheduled graph.
|
||
|
||
Construct and validate a complete schedule for the available graph before
|
||
applying its resolution operations. A blocked dependent is not necessarily a
|
||
cycle member: diagnostics should identify the actual cycle using shared,
|
||
deterministic cycle-path logic. Source discovery and parsing may already have
|
||
occurred; this rule is not a promise to avoid reading invalid configurations.
|
||
|
||
### Scope And Integration
|
||
|
||
Discovery and resolution are separate operations. An importing source must be
|
||
read to discover its imports, but its contribution need not be assembled first.
|
||
Discovery order is not a substitute for the selected resolution schedule.
|
||
|
||
Both resolvers construct dependency graphs and apply resolver-specific operations
|
||
through the shared scheduler. Imports retain strict composition; `extends`
|
||
retains inheritance overlay behavior. Scheduling does not authorize replacement
|
||
within a strict composition tier or change precedence between tiers. Shared
|
||
dependencies can be scheduled once where valid; this does not relax the existing
|
||
prohibition on repeated source inclusion for imports.
|
||
|
||
Preserve item-local helper scopes and enclosing-before-nested inheritance
|
||
requirements. An imported item completes its helper contributions before being
|
||
placed as an item. Enclosing inheritance establishes the context in which nested
|
||
relationships are discovered and resolved. Use the same scheduler at those
|
||
scope boundaries, with the actual prerequisites represented; do not flatten raw
|
||
files or mappings in a way that loses their scope. Inheritance still resolves
|
||
after source assembly and precedence have established the effective document.
|
||
|
||
The integration must account for resulting sequence positions, provenance,
|
||
history, and diagnostic ordering. It must distinguish source acquisition records
|
||
from assembly order rather than assuming they are the same. Review both
|
||
resolvers' contracts and behavioral tests when implementing the new schedule;
|
||
preserving the old traversal is not a design objective.
|
||
|
||
### Rationale And Validation
|
||
|
||
Dependency-ready ordering selects the smallest currently eligible node without
|
||
adding branch preference, batch barriers, or distance-from-root priority. With
|
||
a fixed total ordering key, it gives the lexicographically smallest valid
|
||
dependency-first schedule. It can interleave independent branches; branch
|
||
contiguity is not part of the chosen ordering contract.
|
||
|
||
The comparison considered deterministic depth-first postorder, dependency
|
||
rounds, and descending longest-path depth as well. All can satisfy dependency
|
||
correctness, but their additional ordering policies did not justify choosing
|
||
them over the uniform eligibility rule. This decision gives no preference to
|
||
the existing implementation or to minimizing pre-1.0 refactoring.
|
||
|
||
The scheduler requires O(E + V log V) time and O(V + E) memory for V nodes and
|
||
E dependency edges, assuming constant-time key comparisons. Source access,
|
||
parsing, merging, and provenance work are separate costs.
|
||
|
||
Validate independent branches, unequal chains, branches with multiple
|
||
dependencies, shared dependencies at equal and unequal depths, cycles with
|
||
blocked non-cycle dependents, and nested scopes. Equivalent valid graphs and
|
||
ordering keys must produce identical schedules through the import and
|
||
inheritance integrations. Verify independence from graph insertion order and
|
||
preserve each resolver's domain-specific admissibility and combination rules.
|
||
|
||
## Discovery Semantics
|
||
|
||
### Filename Matching
|
||
|
||
`pattern` applies to a file's basename, not its full relative path. For example,
|
||
`*.yaml` matches both `prompts/one.yaml` and
|
||
`prompts/weather/daily/summary.yaml` when the latter's directory is visited.
|
||
Recursion changes which directories are visited; it does not change what `*`
|
||
means. A nonrecursive search considers only files directly inside `dir`.
|
||
|
||
**OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard
|
||
grammar for filename filters. When neither the declaration nor the application
|
||
supplies `pattern`, select all filenames, with no implicit extension restriction.
|
||
Do not add regular expressions, automatic dialect detection, or recursive `**`
|
||
syntax as part of this work.
|
||
|
||
Retain `glob` as a permanent, separate selector using the existing slash-based
|
||
path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects
|
||
files in a particular directory layout without selecting all descendants.
|
||
Literal `dir` roots and basename filters do not express that same search.
|
||
This capability justifies retaining `glob` independently of compatibility;
|
||
it uses the same new absence policy as other selectors. `glob` remains a
|
||
single string; list support is being added to `path`, `dir`, and the filename
|
||
filters.
|
||
|
||
### Selector Lists And Set Operations
|
||
|
||
Accept a string or a list of strings for `path`, `dir`, and `pattern`. Normalize
|
||
a scalar to a singleton list so both forms use the same selection logic.
|
||
Listed paths and directory roots resolve relative to the declaring file.
|
||
List order does not introduce precedence or replace the shared dependency-ready
|
||
resolution schedule.
|
||
|
||
For directory roots D and filename patterns P, search the **Cartesian product**
|
||
D × P and take the **union** of the matching candidate files:
|
||
|
||
`S = ⋃ {matches(d, p) | (d, p) ∈ D × P}`
|
||
|
||
Each root uses the declaration's recursion, depth, and applicable discovery
|
||
controls. Resolve an omitted `pattern` from the application default first;
|
||
if neither supplies patterns, select all filenames in the visited directories
|
||
instead. This describes selection semantics, not a requirement to
|
||
materialize the product or traverse a directory again for each pattern.
|
||
|
||
Overlapping patterns or directory searches within one declaration select the
|
||
same candidate once. Repeated identical entries in one `path` list likewise
|
||
select that candidate once; duplicate patterns or exclusions have no additional
|
||
effect. This local deduplication is not global source deduplication:
|
||
existing repeated-source and alias checks still apply to distinct inclusion
|
||
routes. In particular, it does not authorize silently merging separate alias
|
||
paths to an imported source.
|
||
|
||
For `path`, `required: true` requires every listed file to be present. For `dir`,
|
||
it requires at least one selected file across the complete union after
|
||
exclusions; it does not require a match per directory, pattern, or pair. A
|
||
missing directory can therefore contribute nothing while another supplies the
|
||
required match. The equivalent `glob` requirement is at least one selected file
|
||
after exclusions. Invalid supplied configuration still fails under the existing
|
||
rules.
|
||
|
||
### Exclusions
|
||
|
||
Accept `exclude` as a string or list of strings with the same case-sensitive
|
||
basename wildcard grammar as `pattern`. Apply it to results from `path`, `dir`,
|
||
and `glob`. A candidate matching any exclusion is removed before its content is
|
||
loaded or parsed. An omitted exclusion removes nothing.
|
||
|
||
This is **set difference**: for candidate set S and its subset X matching any
|
||
exclusion, the selected set is `S ∖ X`. Exclusions win over positive matches.
|
||
They filter filenames, not directory paths, and do not prune traversal into a
|
||
directory whose name happens to match. For example, `exclude: '*.draft.yaml'`
|
||
removes matching files at every visited depth, including results of a path glob.
|
||
|
||
For searches, evaluate `required` on the selected set after exclusions. For an
|
||
explicit `path` list, exclusions filter what is loaded but do not waive the
|
||
settled requirement that every listed file be present when `required: true`.
|
||
Excluding a candidate is not permission to suppress discovery, containment,
|
||
or resource-limit errors. Exclusions are local to the declaration; they do not
|
||
prohibit another declaration from explicitly selecting the same filename.
|
||
|
||
### Recursive Discovery And Depth Limits
|
||
|
||
Enable recursion explicitly or through an application default for recursive
|
||
discovery, optionally bounded by `depth`.
|
||
The motivation is to let authors organize definitions in subdirectories without
|
||
updating import declarations for every depth. Directory organization should not
|
||
change resource identity or the effective configuration destination.
|
||
|
||
The selected model uses an explicit search root and recursion flag, rather than
|
||
introducing recursive `**` syntax or inferring a root from a wildcard prefix.
|
||
|
||
**OD-1: settled.** The library default for `recursive` is false: search only
|
||
direct files. An application may supply a different default; an explicit
|
||
declaration value overrides it, including `recursive: false`. Recursion remains
|
||
independent of the permissive defaults for absence, symlinks, and hidden entries.
|
||
|
||
The settled depth convention is:
|
||
|
||
- Files directly inside the search root are at depth `1`.
|
||
- Each additional directory increases depth by one.
|
||
- Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit
|
||
when recursion is enabled; existing loader resource limits still apply.
|
||
- Supplying `depth` requires effective `recursive: true`. If recursion is false
|
||
after resolving declaration, application, and library defaults, any supplied
|
||
`depth` is an error, including `0` or `1`. Depth never enables recursion itself.
|
||
- With recursion disabled and `depth` omitted, select only direct files.
|
||
- Depth controls discovery only, not namespaces, identities, or destinations.
|
||
|
||
Resolve the effective recursion value before checking depth's applicability.
|
||
For example, `depth: 2` with no explicit `recursive` is valid if the application
|
||
default enables recursion, but invalid with the library default. Explicit
|
||
`recursive: false` overrides an application default of true and makes any
|
||
supplied depth invalid. Depth must also have the valid type and range defined
|
||
under directive validation; enabling recursion does not excuse malformed depth.
|
||
|
||
Recursive discovery must fit the existing filesystem containment,
|
||
repeated-source, and load-budget
|
||
contracts rather than silently changing them. In particular, examined entries,
|
||
including nonmatches, still consume the visit budget; a small number of matches
|
||
does not authorize an unbounded filesystem scan. This is an existing ConfigKit
|
||
constraint, not a new Ansible-derived decision.
|
||
|
||
### Symlinks And Hidden Entries
|
||
|
||
**OD-4: settled.** Discovery controls are local to the declaration. Directory
|
||
searches default to `follow: true` and `hidden: true`, subject to the application
|
||
defaults described below. Authors may disable either behavior independently of
|
||
recursion, depth, matching, or absence handling.
|
||
|
||
Define hidden entries by a leading dot in the portable source namespace.
|
||
`hidden: false` skips discovered hidden files and prunes discovered hidden
|
||
directories. `follow: false` skips discovered symlink files and prevents descent
|
||
through discovered symlink directories. Explicitly named roots and separate
|
||
direct-file imports retain their existing source-access semantics. The controls
|
||
do not implicitly propagate to declarations inside imported files.
|
||
|
||
These are discovery-selection controls, not a load-wide access prohibition.
|
||
On `glob`, filtering applies to wildcard-selected entries; explicitly named
|
||
path components remain explicit access.
|
||
|
||
When following links, detect a canonical directory repeated on the active
|
||
traversal path and fail with a cycle diagnostic. Do not silently prune the loop
|
||
and continue. Existing dependency-cycle and repeated-source errors remain in
|
||
force; discovered aliases do not authorize silently deduplicating imported
|
||
sources. Directory-cycle handling retains the existing containment and
|
||
native-filesystem capabilities.
|
||
|
||
These controls narrow selection rather than authorize ignoring errors in a
|
||
selected source. `required: true` applies after discovery filters: if the search
|
||
selects no file, it fails. Native filesystems retain their existing capabilities;
|
||
this feature does not promise link inspection where a supplied filesystem does
|
||
not expose it.
|
||
|
||
## Illustrative Examples
|
||
|
||
These examples describe the proposed interface, not configuration accepted by
|
||
the current release. They assume the consumer has enabled imports and has not
|
||
changed the library's policy defaults. File paths and directory roots resolve
|
||
from the declaring file; destinations use the
|
||
existing relative-placement rules.
|
||
|
||
### Import One File
|
||
|
||
```yaml
|
||
configkit:
|
||
imports:
|
||
- path: defaults.yaml
|
||
```
|
||
|
||
Import `defaults.yaml` into the declaring source's mapping scope. Under the
|
||
proposed default, a missing file contributes nothing. Add `required: true` when
|
||
its presence is necessary. A present but invalid file still fails loading.
|
||
|
||
### Require Several Named Files
|
||
|
||
```yaml
|
||
configkit:
|
||
imports:
|
||
- path: [defaults.yaml, site.yaml]
|
||
required: true
|
||
```
|
||
|
||
Both files must be present. Listing them together shares the declaration's
|
||
options; their textual order does not establish overlay precedence. Their
|
||
contributions follow the shared dependency-ready schedule and strict
|
||
composition rules.
|
||
|
||
### Recursively Discover Mapping Fragments
|
||
|
||
```yaml
|
||
configkit:
|
||
imports:
|
||
- dir: services
|
||
pattern: '*.yaml'
|
||
recursive: true
|
||
destination: [services]
|
||
```
|
||
|
||
This discovers files such as `services/api.yaml` and
|
||
`services/regional/worker.yaml`, but not `services/README.md`. Each file contains
|
||
named service entries, for example `api: {port: 8080}` or
|
||
`worker: {concurrency: 4}`. They compose beneath `services` under the existing
|
||
strict composition rules. Directory names do not create additional mapping
|
||
keys. Missing directories or no matching files contribute nothing.
|
||
|
||
Adding `depth: 2` limits discovery to direct files and files one directory below
|
||
`services`: it still selects `services/regional/worker.yaml`, but not
|
||
`services/regional/east/worker.yaml`. Switching to `recursive: false` requires
|
||
omitting `depth`; leaving `depth: 2` present is a declaration error.
|
||
|
||
Hidden entries and contained symlinks participate by default. To narrow the
|
||
search, the declaration can also include:
|
||
|
||
```yaml
|
||
follow: false
|
||
hidden: false
|
||
```
|
||
|
||
These controls narrow discovery below the explicitly named root; the root
|
||
continues to resolve under the existing source-access rules.
|
||
|
||
### Search Several Prompt Collections And Filename Patterns
|
||
|
||
```yaml
|
||
configkit:
|
||
imports:
|
||
- dir: [prompts, shared-prompts]
|
||
pattern: ['*.prompt.yaml', '*.prompt.yml']
|
||
exclude: ['*.draft.prompt.yaml', '*.draft.prompt.yml']
|
||
recursive: true
|
||
required: true
|
||
destination: [prompts]
|
||
placement: sequence-item
|
||
```
|
||
|
||
The two directories and two patterns form four Cartesian-product pairs; the
|
||
selected candidates are the union of their matches, minus excluded filenames.
|
||
This selects `prompts/welcome.prompt.yaml`,
|
||
`prompts/weather/daily/summary.prompt.yaml`, and
|
||
`shared-prompts/greeting.prompt.yml`, but not `prompts/weather/helpers.yaml`
|
||
or `shared-prompts/greeting.draft.prompt.yml`.
|
||
Each selected file contains one prompt mapping,
|
||
such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the
|
||
`prompts` sequence with identity field `id` through the existing
|
||
`Definition.SequenceCompositions` API. Distinct IDs compose into one sequence;
|
||
duplicate IDs fail strict composition. Overlays continue replacing sequences.
|
||
|
||
Here `required: true` means at least one file must remain across the entire
|
||
search after exclusions, not that every directory or pattern must produce a
|
||
match. One missing root is acceptable if the other supplies a selected file. The
|
||
filename convention keeps helper files out of this discovery declaration; it
|
||
does not prevent a selected file from explicitly importing a helper. Existing
|
||
repeated-source rules still apply if declarations select overlapping sources.
|
||
|
||
### Select A Directory Layout With A Glob And Exclusions
|
||
|
||
```yaml
|
||
configkit:
|
||
imports:
|
||
- glob: 'collections/*/prompts/*.yaml'
|
||
exclude: '*.draft.yaml'
|
||
required: true
|
||
destination: [prompts]
|
||
placement: sequence-item
|
||
```
|
||
|
||
This selects files such as `collections/support/prompts/welcome.yaml`, excluding
|
||
`collections/support/prompts/welcome.draft.yaml`. It does not select deeper
|
||
files such as `collections/support/prompts/internal/help.yaml`. At least one
|
||
file must remain after exclusions. The same prompt-item composition setup as
|
||
the preceding example applies. Exclusion patterns match basenames even though
|
||
the positive glob matches paths.
|
||
|
||
## Optional Imports By Default
|
||
|
||
The selected direction is optional-by-default for explicit files, directory
|
||
searches, and globs, with `required: true` opting into failure on absence. This
|
||
deliberately supersedes the earlier suggestion to retain required-by-default
|
||
imports.
|
||
|
||
The table assumes the library's default absence policy. Applications may select
|
||
a different default using the settled precedence rules below.
|
||
|
||
| Situation | Proposed behavior |
|
||
| --- | --- |
|
||
| Missing `path` entry; `required` omitted or false | That entry contributes nothing; other listed files may still contribute |
|
||
| Zero selected files across a complete search; `required` omitted or false | Contribute nothing |
|
||
| Any listed `path` file missing with `required: true` | Fail with an import diagnostic |
|
||
| Zero selected files across a complete `dir` or `glob` search after exclusions with `required: true` | Fail with an import diagnostic |
|
||
| One `dir` root missing, but another supplies a selected file with `required: true` | Satisfy the declaration's presence requirement |
|
||
| Invalid pattern or malformed import declaration | Fail |
|
||
| Permission failure during discovery, unreadable selected file, or malformed YAML in a selected file | Fail |
|
||
| Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules |
|
||
|
||
An optional declaration is permission for absence, not permission to ignore a
|
||
selected contribution that cannot be read or interpreted. Optionality must not
|
||
turn processing failures into apparently successful partial configuration.
|
||
|
||
This default supports installations with different optional components without
|
||
requiring ceremony for every potentially absent directory or file. Its tradeoff
|
||
is that a misspelled path or pattern can also contribute nothing. Authors and
|
||
applications that need an import to exist can state that requirement explicitly;
|
||
applications remain responsible for validating their necessary final values.
|
||
Final validation does not automatically detect every unintentionally omitted
|
||
contribution.
|
||
|
||
## Who Chooses Strictness?
|
||
|
||
**OD-5: settled.** Provide load-level defaults for `required`, `follow`, `hidden`,
|
||
`recursive`, and `pattern`. Resolve each applicable field independently using
|
||
this precedence:
|
||
|
||
1. Explicit value on the import declaration.
|
||
2. Application-selected default, when supplied.
|
||
3. Library default: false for `required` and `recursive`, true for `follow` and
|
||
`hidden`, and no filename restriction for `pattern`.
|
||
|
||
An explicit `pattern` replaces the application default rather than combining
|
||
with it. In particular, `pattern: []` selects nothing, even when the application
|
||
supplies patterns. An application can default to `['*.yml', '*.yaml']`; without
|
||
either an application default or an explicit pattern, a directory search
|
||
attempts to load all files in its search scope. Application defaults apply only
|
||
to selectors that support the field; they do not add inapplicable options to
|
||
other selectors.
|
||
|
||
Apply this rule to transitive declarations too. A declaration does not inherit
|
||
these field values from its importing parent. Applications may expose the
|
||
defaults to operators through their own interfaces; simple consumers need no
|
||
additional setup. Exact Go API spelling belongs to implementation planning.
|
||
|
||
Explicit declarations remain authoritative, permitting intentional exceptions
|
||
to stricter application defaults. Defaults are not enforcement controls. Do not
|
||
add options that override explicit declarations as part of this work without a
|
||
separate concrete requirement and scope decision. The fields' applicability to
|
||
each selector is defined under directive validation.
|
||
|
||
## Evidence For Absent Optional Imports
|
||
|
||
**OD-6: settled.** Record minimal structured inspection evidence when an
|
||
optional declaration selects nothing: its declaring source/location, selector,
|
||
and absent outcome. Emit no warning by default and do not fabricate a
|
||
loaded-source record. This keeps optional contributions quiet during ordinary
|
||
use while allowing applications to explain omissions during troubleshooting.
|
||
|
||
Decide the exact inspection record shape during implementation planning,
|
||
reusing existing structures where appropriate and retaining existing inspection
|
||
contracts. Include ownership tests for the new evidence. This addition does
|
||
not require applications to reconstruct absent-import provenance themselves.
|
||
|
||
## Release Boundary
|
||
|
||
Ship one final interface with a hard break in a pre-1.0 minor release, following
|
||
the [release policy](../policy/release.md#versioning-and-compatibility). There is
|
||
no transition release, legacy parser path, compatibility switch, or temporary
|
||
`optional` alias. Release notes should describe the changed behavior and final
|
||
syntax; this documentation does not imply migration machinery.
|
||
|
||
Retaining `glob` is a settled capability decision, as described under filename
|
||
matching. It does not preserve the old required-by-default behavior or promise
|
||
compatibility with every old declaration or use case.
|
||
|
||
## Named Algorithms And Implementation Documentation
|
||
|
||
Use the common names below in implementation comments and maintained
|
||
documentation when the feature lands. Keep them beside the relevant logic and
|
||
in the appropriate owning documents so human readers can recognize the design
|
||
without reconstructing it from code. This does not call for new public API
|
||
terminology or duplicate contracts across documents.
|
||
|
||
| Operation | Name and intended use |
|
||
| --- | --- |
|
||
| Shared import and `extends` scheduling | **Kahn's algorithm with a stable ordering key**, using a min-heap for the ready set; the same dependency-ready topological sort serves both resolvers |
|
||
| Multiple directory roots and filename patterns | **Cartesian product** of roots and patterns, followed by **set union** of matching candidates within the declaration |
|
||
| Exclusion filtering | **Set difference** between positive candidates and those matching any exclusion |
|
||
| Directory symlink cycles | **Active-path cycle detection**, using canonical directories on the current traversal path rather than treating every previously visited directory as a cycle |
|
||
|
||
Scalar-to-singleton normalization lets scalar and list forms share selection
|
||
logic. Set operations describe membership, not resolution order; the shared
|
||
stable topological schedule supplies that order. Preserve the scheduler's
|
||
complexity explanation and distinction between cycles and blocked dependents
|
||
in its implementation documentation. Do not label implementation-specific
|
||
matching or cycle-path helpers with an algorithm name until that algorithm is
|
||
actually chosen.
|
||
|
||
## Directive Validation
|
||
|
||
**OD-3: settled.** Validate declaration shapes and values independently of
|
||
whether discovery selects any files. Invalid declarations fail even when a
|
||
search root is missing or a list is empty. This does not require opening or
|
||
parsing excluded files or establish a new first-error ordering guarantee.
|
||
|
||
### Selectors And Applicable Options
|
||
|
||
Require exactly one selector per declaration: `path`, `dir`, or `glob`. A list
|
||
inside `path` or `dir` still counts as one selector. Use separate declarations
|
||
when combining different selector kinds. Diagnose unknown fields and options
|
||
that have no meaning for the chosen selector. The removed `optional` spelling
|
||
is not accepted.
|
||
|
||
| Option | `path` | `dir` | `glob` |
|
||
| --- | --- | --- | --- |
|
||
| `required`, `exclude` | Yes | Yes | Yes |
|
||
| `destination`, `placement` | Yes | Yes | Yes |
|
||
| `pattern` | No | Yes | No |
|
||
| `recursive`, `depth` | No | Yes | No |
|
||
| `follow`, `hidden` | No | Yes | Yes |
|
||
|
||
### Types And Depth Applicability
|
||
|
||
- `path`, `dir`, `pattern`, and `exclude` accept strings or flat lists of strings.
|
||
- `glob` accepts a string.
|
||
- `required`, `recursive`, `follow`, and `hidden` accept booleans.
|
||
- `depth` accepts nonnegative integers under ConfigKit's existing YAML scalar
|
||
rules. Zero means unlimited when recursion is enabled; negative, string,
|
||
boolean, and fractional values are invalid. Do not introduce an arbitrary
|
||
maximum depth.
|
||
- Explicit null is invalid for these fields, including null list members.
|
||
Do not coerce non-string members to strings or flatten nested lists.
|
||
- Existing `destination` and `placement` validation remains unchanged.
|
||
|
||
A supplied `depth` additionally requires effective recursion to be enabled,
|
||
using the default precedence defined above. Valid type and range alone do not
|
||
make depth applicable. This rule applies even to empty directory or pattern
|
||
lists. With recursion enabled, malformed depth values still fail validation.
|
||
|
||
### Empty Lists, Empty Strings, And Duplicates
|
||
|
||
Empty lists are valid and retain their set meanings:
|
||
|
||
| Declaration | Meaning | Effect of `required: true` |
|
||
| --- | --- | --- |
|
||
| `path: []` | No named files | Presence requirement satisfied: no listed file is missing |
|
||
| `dir: []` | No search roots | Fail: no selected file |
|
||
| `dir: fragments` with `pattern: []` | No positive patterns; select nothing | Fail: no selected file |
|
||
| `exclude: []` | Exclude nothing | Otherwise unchanged |
|
||
|
||
An empty pattern list differs from omission: omission uses the application
|
||
default if supplied, otherwise all filenames. An explicit empty list selects
|
||
nothing and overrides any application default.
|
||
|
||
Empty strings are invalid for `path`, `dir`, and `glob`, including empty entries
|
||
in path or directory lists. Use `.` to explicitly name the declaring directory
|
||
as a search root. Empty strings in `pattern` and `exclude` are valid wildcard
|
||
expressions that match no actual filename. Other source-path and wildcard
|
||
validation rules still apply.
|
||
|
||
Repeated identical path entries within one declaration select that candidate
|
||
once. Repeated patterns or exclusions have no additional effect, and directory
|
||
search overlaps follow the settled set-union rule. These rules do not suppress
|
||
existing repeated-source errors across separate declarations or collapse
|
||
separate symlink aliases.
|
||
|
||
Validate every supplied wildcard expression even if it is inactive because
|
||
another list is empty or discovery finds no files. Likewise, malformed list
|
||
members remain errors even when other entries are valid. Do not silently drop
|
||
invalid members or treat malformed filters as absent.
|