eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Finalize import discovery documentation and audits

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-19 21:23:47 +00:00
parent 898035d83a
commit f6a6b0184b
13 changed files with 172 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
README.md
View File

@@ -33,6 +33,15 @@ sequences as whole values. See the [sequence contract](docs/config.md#strict-doc
[maintained prompt example](docs/consumers/README.md#identified-prompt-sequences),
and [ADR 0014](docs/adr/0014-sequence-placement-and-composition.md).
Imports are also explicit: a source may name files, search declared directories,
or select files with path globs. Applications can set load-wide import defaults
while keeping declaration-specific exceptions. Files are acquired in authored
declaration order; dependency-ready composition uses a shared stable scheduler.
ConfigKit does not search conventional application paths implicitly. See the
[import contract](docs/config.md#explicit-imports), [consumer examples](docs/consumers/README.md),
and [ADR 0015](docs/adr/0015-shared-dependency-ready-ordering.md) and
[ADR 0016](docs/adr/0016-independent-import-controls.md).
```go
type Config struct {
Enabled bool `yaml:"enabled"`

4
docs/adr/0015-shared-dependency-ready-ordering.md
View File

@@ -2,7 +2,9 @@
## Status
Accepted.
Accepted. Implemented behavior is described in the
[configuration contract](../config.md#explicit-imports) and
[internal import overview](../internal/imports.md).
## Date

5
docs/adr/0016-independent-import-controls.md
View File

@@ -2,7 +2,10 @@
## Status
Accepted.
Accepted. Implemented behavior is described in the
[configuration contract](../config.md#explicit-imports), the
[public API declarations](../../lifecycle.go), and the
[internal import overview](../internal/imports.md).
## Date

39
docs/config.md
View File

@@ -961,6 +961,18 @@ budget, including nonmatches. Literal components are explicit path components;
`follow` and `hidden` filter only wildcard-discovered entries. `exclude` filters
final candidate basenames.
`Options.ImportDefaults` and `LayeredOptions.ImportDefaults` provide one
application-owned load-wide policy for enabled imports in the main source,
underlays, and transitive imports; they do not apply to selected overlays or
discover sources by themselves. Each applicable field resolves independently:
an explicit declaration value wins, then the application default, then the
library default (`required: false`, `follow: true`, `hidden: true`,
`recursive: false`, and unrestricted filenames). A declaration's `pattern`
replaces rather than extends the default pattern. Nil `Pattern` leaves directory
filenames unrestricted; a non-nil empty slice selects none. Defaults do not make
a field applicable to selectors that do not support it, and load calls snapshot
the supplied pointers and slice before source acquisition.
Acquisition visits declarations in authored order and each declaration's selected
paths in namespace-path lexical order, preserving source-record and version-callback
order. Ordinary mapping contributions in a main tier are assembled dependency-first:
@@ -976,11 +988,29 @@ in one strict scope. Independent groups remain separate precedence tiers; a high
tier still replaces a lower tier's sequence values wholesale. Assembly order never
permits replacement.
Acquisition order and resolution order serve different purposes: acquisition
assigns source records and callback order, while the shared scheduler determines
composition order among currently dependency-ready contributions. Inheritance
uses the same eligibility rule within each destination-local sibling scope; it
does not reorder the authored output mapping. The private namespace and stable-key
construction is described in the [import implementation overview](internal/imports.md).
An empty selection is allowed by default. `required: true` makes an empty directory
or glob selection fail with `import.missing`; it also makes missing named paths
fail. Requiredness does not suppress invalid paths, containment failures,
directories selected as files, permissions, parse errors, or composition errors.
For inspection, `SourceInfo.ImportSelections` records one outcome for each
completed optional declaration whose entire selected set is empty. The record is
attached to its declaring source, keeps the declaration path local to that source,
identifies the declaration mapping (including its `SourceID`) and selector, and
uses outcome `none`. Records remain in declaration order and are available
through the existing result, load
error, and callback source snapshots. They do not enumerate candidates, explain
why a path was absent, or describe partial omissions. Empty `path: []` succeeds
but does not produce an optional-empty record. Snapshots detach the record paths
and slices.
Each declaration may include `placement: mapping` or
`placement: sequence-item`. Omission means `mapping` for that declaration,
including every match of a glob; placement is not inherited. Sequence-item
@@ -1022,8 +1052,13 @@ ConfigKit canonicalizes symlinks,
checks containment with relative paths, and opens the canonical root-relative target
through a library-owned root handle. Contained relative or absolute symlink targets
are accepted; escapes are rejected. Canonical paths define identity, so hard links at
distinct paths remain distinct sources. Native filesystems use slash-path operations
and their own symlink behavior; imports stay in their selected namespace and root.
distinct paths remain distinct sources. Native filesystems use slash-path
operations. Where the supplied `fs.FS` implements `fs.ReadLinkFS`, ConfigKit uses
that capability to inspect and resolve discovered links; otherwise it follows
the filesystem's ordinary `fs.FS` behavior. ConfigKit does not require every
native filesystem to expose link inspection, and native filesystem behavior is
not a general sandbox guarantee. Imports stay in their selected namespace and
root.
An active-stack repeat is `import.cycle` and reports the complete cycle's import
sites. A previously visited canonical path is `import.repeated` and reports both

3
docs/development.md
View File

@@ -59,6 +59,8 @@ Accepted decisions:
- [Keep the public facade root-owned](adr/0006-public-facade.md)
- [Compare frozen effective configuration snapshots](adr/0010-effective-configuration-diff.md)
- [Place inputs at literal application mapping paths](adr/0011-input-destinations.md)
- [Share dependency-ready ordering](adr/0015-shared-dependency-ready-ordering.md)
- [Separate import selection controls](adr/0016-independent-import-controls.md)
For cross-cutting changes, follow every applicable row. Completed implementation
and remediation plans are preserved in Git history at `cb5b435`; use current
@@ -197,6 +199,7 @@ code changes. Use the component overview to find the corresponding implementatio
| --- | --- | --- |
| YAML syntax, diagnostics, parser limits | [Adapter tests](../internal/yaml/parse_test.go), [source translation](../internal/engine/parse_test.go) | `go test ./internal/yaml` and `go test ./internal/engine -run '^TestParseSource'` |
| File/reader access, imports, globs, containment | [Sources](../internal/engine/source_test.go), [FIFO](../internal/engine/source_fifo_test.go), [imports](../internal/engine/imports_test.go), [relative destinations](../internal/engine/destination_import_test.go), [globs](../internal/engine/glob_test.go) | `go test ./internal/engine -run 'Source\|Import\|Glob\|Filesystem\|Nonregular\|RegularFile'` |
| Import defaults, empty-selection evidence, and dependency-ready ordering | [Defaults and facade evidence](../facade_import_defaults_test.go), [engine import evidence](../internal/engine/import_evidence_test.go), [scheduler](../internal/engine/dependency_order_test.go), [scheduling integrations](../internal/engine/imports_test.go) | `go test ./internal/engine -run 'Import|ScheduleDependencies|Inheritance'`, `go test . -run 'ImportDefaults|FacadeImport'`, and `go test ./examples/consumers` |
| Native filesystem and mapping sources | [Native acquisition](../internal/engine/native_filesystem_test.go), [filesystem facade](../facade_filesystem_test.go), [mapping facade](../facade_mapping_test.go) | `go test ./internal/engine -run 'NativeFilesystem'` and `go test . -run 'TestFacade'` |
| Composition, selected overlays, CLI precedence | [Composition](../internal/engine/compose_test.go), [overlay](../internal/engine/overlay_test.go), [selected overlays](../internal/engine/selected_overlay_test.go), [overrides](../internal/engine/override_test.go) | `go test ./internal/engine -run 'Compose\|Overlay\|Override'` |
| Sibling inheritance, provenance, history, sensitivity | [Expansion](../internal/engine/inheritance_test.go), [destination scopes](../internal/engine/inheritance_walk_test.go), [origins](../internal/engine/inheritance_origin_test.go), [history](../internal/engine/inheritance_history_test.go), [secrets](../internal/engine/inheritance_secret_test.go), [public loading](../facade_inheritance_test.go) | `go test ./internal/engine -run 'Inheritance\|Inherited'` and `go test . -run '^TestFacadeInheritance\|^TestFacadeInheritedSecrets'` |

80
docs/internal/imports.md Normal file
View File

@@ -0,0 +1,80 @@
# Import Discovery and Scheduling Internals
The [configuration contract](../config.md#explicit-imports) owns selector syntax,
defaults, requiredness, placement, inspection evidence, and observable ordering.
This note records implementation flow and private identities for maintainers.
## Discovery and acquisition
Each source's root directive is parsed into presence-aware declarations and
resolved against load-wide defaults. Candidate selection preserves a normalized
namespace path separately from canonical access identity: overlapping roots
coalesce the same candidate spelling in that namespace, while aliases reached
through distinct spellings retain separate inclusion routes for admission.
Directory searches walk each distinct normalized root once, apply all filename
patterns to discovered names, and form a set union of matching candidates across
overlapping roots. Exclusions are applied as set difference after discovery and
candidate normalization. They do not suppress discovery or resource errors.
Directory traversal tracks canonical directories on the active traversal path.
It adds an identity when descending and removes it when the frame is popped, so
only a back-edge to an active directory is a traversal cycle; a later alias is
not mistaken for one. This is separate from import dependency-cycle detection.
For host sources, discovery uses rooted `Lstat` and canonical containment. Native
sources use `fs.ReadLinkFS` inspection when supplied; otherwise traversal follows
the capabilities and behavior of the provided `fs.FS`, without claiming a
general sandbox guarantee.
Acquisition remains authored: declarations are visited in source order and each
selected candidate set in namespace-path lexical order. This preserves source
records, IDs, source-local version checks, and callback ordering. The acquired
inclusion graph is retained separately from composition scopes so cycles that
cross item boundaries remain detectable. Source discovery does not itself choose
sequence assembly order.
## Stable keys and scheduling
The shared scheduler compares immutable tuples component by component using Go's
bytewise string order; if one tuple is a prefix of another, the shorter tuple
sorts first. The import key is
`(namespace-kind, namespace-anchor, resource-kind, resource-path)`:
- Namespace kind is `host` or `native`. Host resource paths are canonical
absolute paths rendered with slash separators; native paths retain their
namespace-relative identity.
- Anchors come from explicit input positions, not discovery order: main is
`(0,0)` and underlay/input positions are `(group+1,input+1)`, each encoded as
fixed-width decimal. Host accesses in one scheduled tier share its host anchor.
A native filesystem namespace uses its earliest explicit selecting position
in that tier. Filesystem tokens distinguish namespace equality, not sort order,
so separately constructed filesystems with the same label remain distinct.
- Resource kind is `file` or `synthetic`. A synthetic root uses its explicit
input anchor and an empty resource path rather than a display label or token
address.
Inheritance keys are
`("definition", scope-pointer, definition-pointer)`, where both pointers use
canonical escaped semantic path spelling. Source IDs are not part of the key.
Both integrations use the same Kahn scheduler: it deduplicates dependency edges,
keeps currently ready nodes in a min-heap, and selects the smallest ready key
after every completion. Its scheduling cost is `O(E + V log V)` time and
`O(V + E)` additional memory, excluding key construction, tuple copying, and
tuple comparisons. On a stalled graph, deterministic DFS over the residual graph
returns a closed cycle rotated to its smallest key; blocked dependents are not
reported as cycle members.
Import composition schedules a main tier or one underlay group as a single
strict-scope graph. Item contributions own nested local strict-mapping scopes;
their helpers complete before the item is placed in its outer scope. Inheritance
uses the same ready-node rule inside each sibling mapping scope, while preserving
the authored mapping order in its output. These scopes do not flatten into one
global graph, and independent precedence tiers remain separate.
The main owners are [`import_selection.go`](../../internal/engine/import_selection.go),
[`directory_walk.go`](../../internal/engine/directory_walk.go),
[`import_graph.go`](../../internal/engine/import_graph.go),
[`dependency_order.go`](../../internal/engine/dependency_order.go), and
[`inheritance_schedule.go`](../../internal/engine/inheritance_schedule.go).
Their focused behavior is covered by import-selection, scheduler, and scheduling
integration tests under `internal/engine`.

9
docs/internal/overview.md
View File

@@ -15,6 +15,10 @@ Implemented component owners:
sequences as whole values. The engine also owns dependency-ready,
destination-local sibling inheritance before typed decoding, plus shared
source-version checking with sanitized declaration evidence.
Explicit import discovery supports declared paths, directory searches, and
globs; load-wide defaults and source-local empty-selection evidence cross the
root facade without changing acquisition order. Import and inheritance
resolution share dependency-ready scheduling within their composition scopes.
It retains each source's effective placement alongside physical source
identity and owns schema
inspection, lifecycle coordination, callback auditing,
@@ -68,4 +72,7 @@ records the relative-import extension; [ADR 0013](../adr/0013-shared-source-vers
records the source-version decision.
[ADR 0014](../adr/0014-sequence-placement-and-composition.md) records explicit
sequence placement and composition. [ADR 0015](../adr/0015-shared-dependency-ready-ordering.md)
records the shared dependency-ready ordering decision.
records the shared dependency-ready ordering decision; [ADR
0016](../adr/0016-independent-import-controls.md) records the independent
selection controls. The [import internals](imports.md) document candidate and
namespace identities, discovery and scheduling, and the bounded algorithms.

8
docs/policy/architecture.md
View File

@@ -153,6 +153,14 @@ the import extension.
Library code must not depend on consuming applications or initialize their
runtime resources. It must not perform implicit network access, discover
application-specific paths, or change process-global state during ordinary `Load`.
Import discovery is available only through declarations in selected sources:
explicit `path`, `dir`, and `glob` selectors may search their declared scope, but
ConfigKit does not search conventional application paths or discover a main
configuration file implicitly. Imports and sibling inheritance share
dependency-ready scheduling within their existing scopes; acquisition order and
precedence-tier boundaries remain distinct. The [configuration contract](../config.md#explicit-imports)
owns observable behavior; [ADR 0015](../adr/0015-shared-dependency-ready-ordering.md)
and [ADR 0016](../adr/0016-independent-import-controls.md) record the decisions.
The explicit secrets-directory bootstrap is a narrow, opt-in exception: applications
invoke it during startup, and it reports writes, skips and partial failures. It
must never run implicitly as part of configuration loading.

2
internal/engine/glob.go
View File

@@ -31,6 +31,8 @@ func selectGlobImportCandidates(access *filesystemAccess, basePath string, spec
for index, segment := range segments {
last := index == len(segments)-1
literal, patterned := globLiteralSegment(segment)
// nextByPath is the set union of candidates reached from each matching
// parent at this segment; final exclusions below are set difference.
nextByPath := make(map[string]selectedImportCandidate)
for _, parent := range candidates {
if !patterned {

5
internal/engine/import_selection.go
View File

@@ -97,6 +97,9 @@ func selectDirectoryImportCandidates(access *filesystemAccess, basePath string,
}
discovered := make([]selectedImportCandidate, 0)
seenCandidates := make(map[string]struct{})
// Each root is traversed once. Filename patterns are tested together for
// every discovered file, and seenCandidates forms their set union across
// overlapping roots. Exclusions are applied once below as set difference.
for _, root := range roots {
missing, isDirectory, diagnostic := validateNamedImportTarget(access, root, false, true)
if diagnostic != nil {
@@ -121,6 +124,8 @@ func selectDirectoryImportCandidates(access *filesystemAccess, basePath string,
continue
}
stack := []importDirectoryFrame{{directory: root, depth: 0, entries: entries, identity: rootTarget.identity}}
// Active-path cycle detection removes each directory when its frame is
// popped, so a later non-active alias remains a separate traversal.
activeDirectories := map[string]struct{}{rootTarget.identity: {}}
for len(stack) != 0 {
frameIndex := len(stack) - 1

13
internal/engine/source_version.go
View File

@@ -59,15 +59,18 @@ func sourceInfosWithVersionEvidence(sources []SourceInfo, evidence []*sourceVers
return result
}
sensitivity := newDiffSensitivity(secrets, nil)
sourceIndexes := make(map[string]int, len(result))
for index := range result {
if _, found := sourceIndexes[result[index].ID]; !found {
sourceIndexes[result[index].ID] = index
}
}
for _, item := range evidence {
if item == nil || item.sourceID == "" {
continue
}
for index := range result {
if result[index].ID == item.sourceID {
result[index].VersionCheck = sourceVersionCheckForEvidence(item, sensitivity, conservative)
break
}
if index, found := sourceIndexes[item.sourceID]; found {
result[index].VersionCheck = sourceVersionCheckForEvidence(item, sensitivity, conservative)
}
}
return result

3
lifecycle.go
View File

@@ -78,7 +78,8 @@ const (
// in effect: Required false, Follow true, Hidden true, and Recursive false.
// Pattern applies to directory searches; nil leaves filenames unrestricted,
// while a non-nil empty slice selects none. Defaults apply to main, underlay,
// and transitive imports, not selected overlays.
// and transitive imports, not selected overlays. Load calls snapshot the
// pointed-to booleans and Pattern slice before acquiring sources.
type ImportDefaults struct {
// Required defaults absence handling. For path, true requires every named
// file; for dir and glob, true requires at least one selected file.

3
records.go
View File

@@ -212,6 +212,9 @@ type SourceInfo struct {
VersionCheck *SourceVersionCheck
// ImportSelections records completed optional declarations that selected no
// sources, in declaration order. Paths are local to this declaring source.
// Each record identifies the declaration mapping, selector, and empty outcome;
// it does not list candidates or explain why a path was absent. Result, error,
// and callback snapshots detach these records and their paths.
ImportSelections []ImportSelection
}