diff --git a/README.md b/README.md index 63e64e9..7e63bc1 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,15 @@ sequences as whole values. See the [sequence contract](docs/config.md#strict-doc [maintained prompt example](docs/consumers/README.md#identified-prompt-sequences), and [ADR 0014](docs/adr/0014-sequence-placement-and-composition.md). +Imports are also explicit: a source may name files, search declared directories, +or select files with path globs. Applications can set load-wide import defaults +while keeping declaration-specific exceptions. Files are acquired in authored +declaration order; dependency-ready composition uses a shared stable scheduler. +ConfigKit does not search conventional application paths implicitly. See the +[import contract](docs/config.md#explicit-imports), [consumer examples](docs/consumers/README.md), +and [ADR 0015](docs/adr/0015-shared-dependency-ready-ordering.md) and +[ADR 0016](docs/adr/0016-independent-import-controls.md). + ```go type Config struct { Enabled bool `yaml:"enabled"` diff --git a/docs/adr/0015-shared-dependency-ready-ordering.md b/docs/adr/0015-shared-dependency-ready-ordering.md index efb4bab..e75831f 100644 --- a/docs/adr/0015-shared-dependency-ready-ordering.md +++ b/docs/adr/0015-shared-dependency-ready-ordering.md @@ -2,7 +2,9 @@ ## Status -Accepted. +Accepted. Implemented behavior is described in the +[configuration contract](../config.md#explicit-imports) and +[internal import overview](../internal/imports.md). ## Date diff --git a/docs/adr/0016-independent-import-controls.md b/docs/adr/0016-independent-import-controls.md index bcb507d..42c83c8 100644 --- a/docs/adr/0016-independent-import-controls.md +++ b/docs/adr/0016-independent-import-controls.md @@ -2,7 +2,10 @@ ## Status -Accepted. +Accepted. Implemented behavior is described in the +[configuration contract](../config.md#explicit-imports), the +[public API declarations](../../lifecycle.go), and the +[internal import overview](../internal/imports.md). ## Date diff --git a/docs/config.md b/docs/config.md index cd1bd0e..e3ed78e 100644 --- a/docs/config.md +++ b/docs/config.md @@ -961,6 +961,18 @@ budget, including nonmatches. Literal components are explicit path components; `follow` and `hidden` filter only wildcard-discovered entries. `exclude` filters final candidate basenames. +`Options.ImportDefaults` and `LayeredOptions.ImportDefaults` provide one +application-owned load-wide policy for enabled imports in the main source, +underlays, and transitive imports; they do not apply to selected overlays or +discover sources by themselves. Each applicable field resolves independently: +an explicit declaration value wins, then the application default, then the +library default (`required: false`, `follow: true`, `hidden: true`, +`recursive: false`, and unrestricted filenames). A declaration's `pattern` +replaces rather than extends the default pattern. Nil `Pattern` leaves directory +filenames unrestricted; a non-nil empty slice selects none. Defaults do not make +a field applicable to selectors that do not support it, and load calls snapshot +the supplied pointers and slice before source acquisition. + Acquisition visits declarations in authored order and each declaration's selected paths in namespace-path lexical order, preserving source-record and version-callback order. Ordinary mapping contributions in a main tier are assembled dependency-first: @@ -976,11 +988,29 @@ in one strict scope. Independent groups remain separate precedence tiers; a high tier still replaces a lower tier's sequence values wholesale. Assembly order never permits replacement. +Acquisition order and resolution order serve different purposes: acquisition +assigns source records and callback order, while the shared scheduler determines +composition order among currently dependency-ready contributions. Inheritance +uses the same eligibility rule within each destination-local sibling scope; it +does not reorder the authored output mapping. The private namespace and stable-key +construction is described in the [import implementation overview](internal/imports.md). + An empty selection is allowed by default. `required: true` makes an empty directory or glob selection fail with `import.missing`; it also makes missing named paths fail. Requiredness does not suppress invalid paths, containment failures, directories selected as files, permissions, parse errors, or composition errors. +For inspection, `SourceInfo.ImportSelections` records one outcome for each +completed optional declaration whose entire selected set is empty. The record is +attached to its declaring source, keeps the declaration path local to that source, +identifies the declaration mapping (including its `SourceID`) and selector, and +uses outcome `none`. Records remain in declaration order and are available +through the existing result, load +error, and callback source snapshots. They do not enumerate candidates, explain +why a path was absent, or describe partial omissions. Empty `path: []` succeeds +but does not produce an optional-empty record. Snapshots detach the record paths +and slices. + Each declaration may include `placement: mapping` or `placement: sequence-item`. Omission means `mapping` for that declaration, including every match of a glob; placement is not inherited. Sequence-item @@ -1022,8 +1052,13 @@ ConfigKit canonicalizes symlinks, checks containment with relative paths, and opens the canonical root-relative target through a library-owned root handle. Contained relative or absolute symlink targets are accepted; escapes are rejected. Canonical paths define identity, so hard links at -distinct paths remain distinct sources. Native filesystems use slash-path operations -and their own symlink behavior; imports stay in their selected namespace and root. +distinct paths remain distinct sources. Native filesystems use slash-path +operations. Where the supplied `fs.FS` implements `fs.ReadLinkFS`, ConfigKit uses +that capability to inspect and resolve discovered links; otherwise it follows +the filesystem's ordinary `fs.FS` behavior. ConfigKit does not require every +native filesystem to expose link inspection, and native filesystem behavior is +not a general sandbox guarantee. Imports stay in their selected namespace and +root. An active-stack repeat is `import.cycle` and reports the complete cycle's import sites. A previously visited canonical path is `import.repeated` and reports both diff --git a/docs/development.md b/docs/development.md index 7d9a77f..68cd70c 100644 --- a/docs/development.md +++ b/docs/development.md @@ -59,6 +59,8 @@ Accepted decisions: - [Keep the public facade root-owned](adr/0006-public-facade.md) - [Compare frozen effective configuration snapshots](adr/0010-effective-configuration-diff.md) - [Place inputs at literal application mapping paths](adr/0011-input-destinations.md) +- [Share dependency-ready ordering](adr/0015-shared-dependency-ready-ordering.md) +- [Separate import selection controls](adr/0016-independent-import-controls.md) For cross-cutting changes, follow every applicable row. Completed implementation and remediation plans are preserved in Git history at `cb5b435`; use current @@ -197,6 +199,7 @@ code changes. Use the component overview to find the corresponding implementatio | --- | --- | --- | | YAML syntax, diagnostics, parser limits | [Adapter tests](../internal/yaml/parse_test.go), [source translation](../internal/engine/parse_test.go) | `go test ./internal/yaml` and `go test ./internal/engine -run '^TestParseSource'` | | File/reader access, imports, globs, containment | [Sources](../internal/engine/source_test.go), [FIFO](../internal/engine/source_fifo_test.go), [imports](../internal/engine/imports_test.go), [relative destinations](../internal/engine/destination_import_test.go), [globs](../internal/engine/glob_test.go) | `go test ./internal/engine -run 'Source\|Import\|Glob\|Filesystem\|Nonregular\|RegularFile'` | +| Import defaults, empty-selection evidence, and dependency-ready ordering | [Defaults and facade evidence](../facade_import_defaults_test.go), [engine import evidence](../internal/engine/import_evidence_test.go), [scheduler](../internal/engine/dependency_order_test.go), [scheduling integrations](../internal/engine/imports_test.go) | `go test ./internal/engine -run 'Import|ScheduleDependencies|Inheritance'`, `go test . -run 'ImportDefaults|FacadeImport'`, and `go test ./examples/consumers` | | Native filesystem and mapping sources | [Native acquisition](../internal/engine/native_filesystem_test.go), [filesystem facade](../facade_filesystem_test.go), [mapping facade](../facade_mapping_test.go) | `go test ./internal/engine -run 'NativeFilesystem'` and `go test . -run 'TestFacade'` | | Composition, selected overlays, CLI precedence | [Composition](../internal/engine/compose_test.go), [overlay](../internal/engine/overlay_test.go), [selected overlays](../internal/engine/selected_overlay_test.go), [overrides](../internal/engine/override_test.go) | `go test ./internal/engine -run 'Compose\|Overlay\|Override'` | | Sibling inheritance, provenance, history, sensitivity | [Expansion](../internal/engine/inheritance_test.go), [destination scopes](../internal/engine/inheritance_walk_test.go), [origins](../internal/engine/inheritance_origin_test.go), [history](../internal/engine/inheritance_history_test.go), [secrets](../internal/engine/inheritance_secret_test.go), [public loading](../facade_inheritance_test.go) | `go test ./internal/engine -run 'Inheritance\|Inherited'` and `go test . -run '^TestFacadeInheritance\|^TestFacadeInheritedSecrets'` | diff --git a/docs/internal/imports.md b/docs/internal/imports.md new file mode 100644 index 0000000..e73d349 --- /dev/null +++ b/docs/internal/imports.md @@ -0,0 +1,80 @@ +# Import Discovery and Scheduling Internals + +The [configuration contract](../config.md#explicit-imports) owns selector syntax, +defaults, requiredness, placement, inspection evidence, and observable ordering. +This note records implementation flow and private identities for maintainers. + +## Discovery and acquisition + +Each source's root directive is parsed into presence-aware declarations and +resolved against load-wide defaults. Candidate selection preserves a normalized +namespace path separately from canonical access identity: overlapping roots +coalesce the same candidate spelling in that namespace, while aliases reached +through distinct spellings retain separate inclusion routes for admission. +Directory searches walk each distinct normalized root once, apply all filename +patterns to discovered names, and form a set union of matching candidates across +overlapping roots. Exclusions are applied as set difference after discovery and +candidate normalization. They do not suppress discovery or resource errors. + +Directory traversal tracks canonical directories on the active traversal path. +It adds an identity when descending and removes it when the frame is popped, so +only a back-edge to an active directory is a traversal cycle; a later alias is +not mistaken for one. This is separate from import dependency-cycle detection. +For host sources, discovery uses rooted `Lstat` and canonical containment. Native +sources use `fs.ReadLinkFS` inspection when supplied; otherwise traversal follows +the capabilities and behavior of the provided `fs.FS`, without claiming a +general sandbox guarantee. + +Acquisition remains authored: declarations are visited in source order and each +selected candidate set in namespace-path lexical order. This preserves source +records, IDs, source-local version checks, and callback ordering. The acquired +inclusion graph is retained separately from composition scopes so cycles that +cross item boundaries remain detectable. Source discovery does not itself choose +sequence assembly order. + +## Stable keys and scheduling + +The shared scheduler compares immutable tuples component by component using Go's +bytewise string order; if one tuple is a prefix of another, the shorter tuple +sorts first. The import key is +`(namespace-kind, namespace-anchor, resource-kind, resource-path)`: + +- Namespace kind is `host` or `native`. Host resource paths are canonical + absolute paths rendered with slash separators; native paths retain their + namespace-relative identity. +- Anchors come from explicit input positions, not discovery order: main is + `(0,0)` and underlay/input positions are `(group+1,input+1)`, each encoded as + fixed-width decimal. Host accesses in one scheduled tier share its host anchor. + A native filesystem namespace uses its earliest explicit selecting position + in that tier. Filesystem tokens distinguish namespace equality, not sort order, + so separately constructed filesystems with the same label remain distinct. +- Resource kind is `file` or `synthetic`. A synthetic root uses its explicit + input anchor and an empty resource path rather than a display label or token + address. + +Inheritance keys are +`("definition", scope-pointer, definition-pointer)`, where both pointers use +canonical escaped semantic path spelling. Source IDs are not part of the key. + +Both integrations use the same Kahn scheduler: it deduplicates dependency edges, +keeps currently ready nodes in a min-heap, and selects the smallest ready key +after every completion. Its scheduling cost is `O(E + V log V)` time and +`O(V + E)` additional memory, excluding key construction, tuple copying, and +tuple comparisons. On a stalled graph, deterministic DFS over the residual graph +returns a closed cycle rotated to its smallest key; blocked dependents are not +reported as cycle members. + +Import composition schedules a main tier or one underlay group as a single +strict-scope graph. Item contributions own nested local strict-mapping scopes; +their helpers complete before the item is placed in its outer scope. Inheritance +uses the same ready-node rule inside each sibling mapping scope, while preserving +the authored mapping order in its output. These scopes do not flatten into one +global graph, and independent precedence tiers remain separate. + +The main owners are [`import_selection.go`](../../internal/engine/import_selection.go), +[`directory_walk.go`](../../internal/engine/directory_walk.go), +[`import_graph.go`](../../internal/engine/import_graph.go), +[`dependency_order.go`](../../internal/engine/dependency_order.go), and +[`inheritance_schedule.go`](../../internal/engine/inheritance_schedule.go). +Their focused behavior is covered by import-selection, scheduler, and scheduling +integration tests under `internal/engine`. diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 789eddc..3dcbf9f 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -15,6 +15,10 @@ Implemented component owners: sequences as whole values. The engine also owns dependency-ready, destination-local sibling inheritance before typed decoding, plus shared source-version checking with sanitized declaration evidence. + Explicit import discovery supports declared paths, directory searches, and + globs; load-wide defaults and source-local empty-selection evidence cross the + root facade without changing acquisition order. Import and inheritance + resolution share dependency-ready scheduling within their composition scopes. It retains each source's effective placement alongside physical source identity and owns schema inspection, lifecycle coordination, callback auditing, @@ -68,4 +72,7 @@ records the relative-import extension; [ADR 0013](../adr/0013-shared-source-vers records the source-version decision. [ADR 0014](../adr/0014-sequence-placement-and-composition.md) records explicit sequence placement and composition. [ADR 0015](../adr/0015-shared-dependency-ready-ordering.md) -records the shared dependency-ready ordering decision. +records the shared dependency-ready ordering decision; [ADR +0016](../adr/0016-independent-import-controls.md) records the independent +selection controls. The [import internals](imports.md) document candidate and +namespace identities, discovery and scheduling, and the bounded algorithms. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 5121fdd..a863610 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -153,6 +153,14 @@ the import extension. Library code must not depend on consuming applications or initialize their runtime resources. It must not perform implicit network access, discover application-specific paths, or change process-global state during ordinary `Load`. +Import discovery is available only through declarations in selected sources: +explicit `path`, `dir`, and `glob` selectors may search their declared scope, but +ConfigKit does not search conventional application paths or discover a main +configuration file implicitly. Imports and sibling inheritance share +dependency-ready scheduling within their existing scopes; acquisition order and +precedence-tier boundaries remain distinct. The [configuration contract](../config.md#explicit-imports) +owns observable behavior; [ADR 0015](../adr/0015-shared-dependency-ready-ordering.md) +and [ADR 0016](../adr/0016-independent-import-controls.md) record the decisions. The explicit secrets-directory bootstrap is a narrow, opt-in exception: applications invoke it during startup, and it reports writes, skips and partial failures. It must never run implicitly as part of configuration loading. diff --git a/internal/engine/glob.go b/internal/engine/glob.go index 021fe5e..37bddc4 100644 --- a/internal/engine/glob.go +++ b/internal/engine/glob.go @@ -31,6 +31,8 @@ func selectGlobImportCandidates(access *filesystemAccess, basePath string, spec for index, segment := range segments { last := index == len(segments)-1 literal, patterned := globLiteralSegment(segment) + // nextByPath is the set union of candidates reached from each matching + // parent at this segment; final exclusions below are set difference. nextByPath := make(map[string]selectedImportCandidate) for _, parent := range candidates { if !patterned { diff --git a/internal/engine/import_selection.go b/internal/engine/import_selection.go index f00c252..b3fdb9d 100644 --- a/internal/engine/import_selection.go +++ b/internal/engine/import_selection.go @@ -97,6 +97,9 @@ func selectDirectoryImportCandidates(access *filesystemAccess, basePath string, } discovered := make([]selectedImportCandidate, 0) seenCandidates := make(map[string]struct{}) + // Each root is traversed once. Filename patterns are tested together for + // every discovered file, and seenCandidates forms their set union across + // overlapping roots. Exclusions are applied once below as set difference. for _, root := range roots { missing, isDirectory, diagnostic := validateNamedImportTarget(access, root, false, true) if diagnostic != nil { @@ -121,6 +124,8 @@ func selectDirectoryImportCandidates(access *filesystemAccess, basePath string, continue } stack := []importDirectoryFrame{{directory: root, depth: 0, entries: entries, identity: rootTarget.identity}} + // Active-path cycle detection removes each directory when its frame is + // popped, so a later non-active alias remains a separate traversal. activeDirectories := map[string]struct{}{rootTarget.identity: {}} for len(stack) != 0 { frameIndex := len(stack) - 1 diff --git a/internal/engine/source_version.go b/internal/engine/source_version.go index 238a1a6..0ae1378 100644 --- a/internal/engine/source_version.go +++ b/internal/engine/source_version.go @@ -59,15 +59,18 @@ func sourceInfosWithVersionEvidence(sources []SourceInfo, evidence []*sourceVers return result } sensitivity := newDiffSensitivity(secrets, nil) + sourceIndexes := make(map[string]int, len(result)) + for index := range result { + if _, found := sourceIndexes[result[index].ID]; !found { + sourceIndexes[result[index].ID] = index + } + } for _, item := range evidence { if item == nil || item.sourceID == "" { continue } - for index := range result { - if result[index].ID == item.sourceID { - result[index].VersionCheck = sourceVersionCheckForEvidence(item, sensitivity, conservative) - break - } + if index, found := sourceIndexes[item.sourceID]; found { + result[index].VersionCheck = sourceVersionCheckForEvidence(item, sensitivity, conservative) } } return result diff --git a/lifecycle.go b/lifecycle.go index 0098e0a..19c8db1 100644 --- a/lifecycle.go +++ b/lifecycle.go @@ -78,7 +78,8 @@ const ( // in effect: Required false, Follow true, Hidden true, and Recursive false. // Pattern applies to directory searches; nil leaves filenames unrestricted, // while a non-nil empty slice selects none. Defaults apply to main, underlay, -// and transitive imports, not selected overlays. +// and transitive imports, not selected overlays. Load calls snapshot the +// pointed-to booleans and Pattern slice before acquiring sources. type ImportDefaults struct { // Required defaults absence handling. For path, true requires every named // file; for dir and glob, true requires at least one selected file. diff --git a/records.go b/records.go index c5dad51..73d1744 100644 --- a/records.go +++ b/records.go @@ -212,6 +212,9 @@ type SourceInfo struct { VersionCheck *SourceVersionCheck // ImportSelections records completed optional declarations that selected no // sources, in declaration order. Paths are local to this declaring source. + // Each record identifies the declaration mapping, selector, and empty outcome; + // it does not list candidates or explain why a path was absent. Result, error, + // and callback snapshots detach these records and their paths. ImportSelections []ImportSelection }