Retire completed import discovery plans
This commit is contained in:
@@ -1,708 +0,0 @@
|
||||
# Implementation Plan: Import Discovery and Dependency-Ready Ordering
|
||||
|
||||
Status: All 19 implementation stages complete.
|
||||
|
||||
## Authority, baseline, and execution
|
||||
|
||||
Implement the accepted [import roadmap](imports.md),
|
||||
[ADR 0015](../adr/0015-shared-dependency-ready-ordering.md), and
|
||||
[ADR 0016](../adr/0016-independent-import-controls.md), starting at
|
||||
`94381a67fd39635f448a5225e38adb2a7f43cc9f`. The roadmap owns target behavior and
|
||||
scope; this plan settles delivery order and the remaining implementation choices.
|
||||
The accepted ADRs already exist; do not draft replacements or rewrite their
|
||||
decision text.
|
||||
|
||||
The implementer is a Luna coding agent. Execute one numbered stage per focused
|
||||
prompt, in order. Each stage should end with a validated, nonempty commit and a
|
||||
completion entry here identifying the commit, checks, and any remaining work.
|
||||
Do not mark a stage complete while its acceptance conditions remain unmet.
|
||||
Preserve unrelated work. This planning pass changes documentation only.
|
||||
|
||||
Start with [development](../development.md) and read the applicable
|
||||
[agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||||
[testing](../policy/testing.md), and [documentation](../policy/documentation.md)
|
||||
policies. Prefer the code graph for discovery, then inspect current source and
|
||||
tests; indexed snippets can have stale line ranges. Use the actual working tree
|
||||
when an indexed snippet is incomplete or mismatched.
|
||||
|
||||
Keep each stage's public documentation accurate to its implemented boundary.
|
||||
Private groundwork may precede wiring, but do not expose options that silently
|
||||
do nothing. The final syntax cutover is Stage 14; there is no consumer-facing
|
||||
compatibility mode, alias, or alternate scheduler. Intermediate private helpers
|
||||
are scaffolding, not a second supported interface. Release publication and
|
||||
consumer dependency updates are outside this plan.
|
||||
|
||||
## Settled implementation choices
|
||||
|
||||
### Public defaults
|
||||
|
||||
Add a root-owned `ImportDefaults` struct and an `ImportDefaults ImportDefaults`
|
||||
field to both `Options[T]` and `LayeredOptions[T]`:
|
||||
|
||||
```go
|
||||
type ImportDefaults struct {
|
||||
Required *bool
|
||||
Follow *bool
|
||||
Hidden *bool
|
||||
Recursive *bool
|
||||
Pattern []string
|
||||
}
|
||||
```
|
||||
|
||||
Nil boolean pointers mean unspecified. Copy pointed-to values when loading
|
||||
begins. Nil `Pattern` means unspecified; non-nil empty means select nothing for
|
||||
directory searches. Snapshot the slice without collapsing this distinction.
|
||||
No new defaults field belongs on `Input`, `Underlay`, or `Definition`: one
|
||||
load-level selection applies to main and all underlay/transitive import
|
||||
declarations. Existing per-input `Imports` booleans still enable imports.
|
||||
|
||||
Resolve each applicable declaration field independently: explicit declaration,
|
||||
then application default, then library default. Library defaults are required
|
||||
false, recursive false, follow true, hidden true, and unrestricted filenames.
|
||||
No parent-declaration inheritance, enforced application override, default
|
||||
exclusion list, or application depth setting is added. `pattern` replaces rather
|
||||
than augments the default. Validate supplied default patterns once at option
|
||||
selection, reporting `options.invalid` before source reads or callbacks.
|
||||
Defaults do not make unsupported fields applicable to a selector.
|
||||
|
||||
Use a private presence-aware declaration representation, distinct from its
|
||||
resolved discovery policy. Normalize scalar/list selectors and filters once.
|
||||
Keep existing import path validity rules, with `.` accepted for literal directory
|
||||
roots. Match basenames with case-sensitive `path.Match`; path globs retain their
|
||||
existing grammar, including rejection of recursive `**`. A filename filter has
|
||||
no directory-recursion interpretation. Validate all expressions even in empty
|
||||
or otherwise inactive selections. Follow the roadmap's option applicability,
|
||||
empty-list, null, and depth rules exactly.
|
||||
|
||||
`depth` has no arbitrary machine-integer maximum. Accept nonnegative YAML
|
||||
integers under the existing scalar rules, including bounds larger than can be
|
||||
reached under the visit budget; an implementation may saturate its internal
|
||||
bound at the largest representable traversal depth without rejecting that input.
|
||||
Numeric `-0` denotes zero. Do not accept fractional/exponent numbers as integer
|
||||
depth or coerce strings.
|
||||
|
||||
### No-selection inspection
|
||||
|
||||
Extend root-owned `SourceInfo` with `ImportSelections []ImportSelection`. Add
|
||||
root-owned string types and constants:
|
||||
|
||||
- `ImportSelector`: `ImportSelectorPath = "path"`, `ImportSelectorDirectory =
|
||||
"dir"`, and `ImportSelectorGlob = "glob"`.
|
||||
- `ImportSelectionOutcome`: `ImportSelectionNone = "none"`.
|
||||
- `ImportSelection` fields: `Path Path`, `Location Location`,
|
||||
`Selector ImportSelector`, and `Outcome ImportSelectionOutcome`.
|
||||
|
||||
`Path` is the source-local declaration path, such as
|
||||
`/configkit/imports/0`; `Location` identifies the declaring mapping and carries
|
||||
its actual SourceID. Attach one record to that declaring source for each
|
||||
effectively optional declaration whose entire selected set is empty. Keep
|
||||
records in declaration order. Do not record partial omissions, candidate lists,
|
||||
raw selector values, inferred missing-file reasons, warnings, or fake sources.
|
||||
Required empty `path: []` succeeds but gets no optional-declaration record.
|
||||
|
||||
Reuse `Result.Sources`, `LoadError.Sources`, and callback source snapshots rather
|
||||
than adding another query surface. Snapshots contain only outcomes established
|
||||
by that point; no complete discovery record is promised after an earlier error.
|
||||
Records remain source-local through mounting and item placement. Deep-copy paths
|
||||
and slices through engine cloning and facade conversion. Existing version-text
|
||||
redaction remains independent and unchanged.
|
||||
|
||||
### Scheduling and identities
|
||||
|
||||
Keep the shared scheduler private to `internal/engine`, in a focused
|
||||
`dependency_order.go` owner; no new public scheduling API or dependency is needed.
|
||||
It accepts immutable node keys and dependency edges and returns a complete order
|
||||
or an actual closed cycle. Use Kahn's algorithm and `container/heap`; deduplicate
|
||||
edges before counting dependencies. Compute keys once. Compare key tuples by Go
|
||||
bytewise string order component by component, shorter tuple first on a prefix.
|
||||
|
||||
Use these key constructions in both integrations:
|
||||
|
||||
- Import nodes: `(namespace-kind, namespace-anchor, resource-kind, resource-path)`.
|
||||
Kind is `host` or `native`; resource kind is `file` or `synthetic`. Host paths
|
||||
are canonical absolute paths rendered with slash separators; native paths use
|
||||
their existing namespace-relative identity. Synthetic roots use their explicit
|
||||
input anchor, not a label or allocated token address.
|
||||
- Derive namespace anchors from explicit request positions, never discovery.
|
||||
Positions are main `(0,0)` and underlay/input `(group+1,input+1)`. Encode each
|
||||
numeric component in fixed-width decimal sufficient for a Go index. All host
|
||||
accesses within one scheduled tier share its host anchor; each native token
|
||||
uses the smallest explicit input position selecting that namespace in that
|
||||
tier. Same-label, separately constructed filesystems remain distinct. Use
|
||||
tokens to identify equality, not to order nodes.
|
||||
- Inheritance nodes: `("definition", scope-pointer, definition-pointer)` using
|
||||
canonical escaped semantic pointers. Within a sibling scope these distinguish
|
||||
exact names, including empty or escaped keys. No source ID participates.
|
||||
|
||||
The tuple comparator is shared; resolver-specific key construction is not a
|
||||
second ordering algorithm. Unique keys are an internal construction requirement,
|
||||
not a new user configuration restriction. Imports already reject repeated
|
||||
canonical inclusion; separate synthetic roots have distinct explicit anchors.
|
||||
|
||||
For a stalled graph, deterministically traverse residual dependencies using
|
||||
explicit DFS state, visiting nodes and neighbors in key order. Return an actual
|
||||
stack cycle, rotated to begin at its smallest key, with the start repeated at
|
||||
the end. Exclude blocked non-cycle dependents. Both resolvers translate these
|
||||
node/edge identities to existing domain diagnostic codes and authored sites;
|
||||
do not expose inheritance names or payload values in messages. Preserve import
|
||||
cycle-site evidence. Build deterministic adjacency from precomputed node ranks
|
||||
in linear edge work, rather than repeatedly sorting the ready set or scanning
|
||||
all nodes for each completion. Scheduling costs O(E + V log V), O(V + E) memory,
|
||||
apart from key construction/comparison and domain operations.
|
||||
|
||||
### Discovery, scopes, and assembly
|
||||
|
||||
Keep acquisition deterministic and separate from application. Enumerate
|
||||
declarations in authored order; normalize/deduplicate a declaration's selected
|
||||
candidate paths and visit them in namespace-path lexical order. Keep main-first
|
||||
acquisition records, IDs, source-local version checking, and physical import
|
||||
resolution. This discovery order does not determine sequence assembly order.
|
||||
There is no new first-error or no-read-before-error guarantee.
|
||||
|
||||
Represent dependencies while acquiring files. Ordinary source-body nodes depend
|
||||
on their imported contributions; importing bodies therefore assemble after
|
||||
their dependencies. Record source roles explicitly: the main body must not be
|
||||
identified as contribution index zero after scheduling. Keep all source IDs and
|
||||
authored locations attached to contributions when reordering them.
|
||||
|
||||
Each main tier or underlay group has one outer strict-scope graph, including all
|
||||
its independent input roots and ordinary imports. An item contribution is one
|
||||
outer node keyed by its owning source. It owns a nested strict mapping-scope
|
||||
graph containing that source body, ordinary helper contributions, and further
|
||||
item nodes. Prevalidate the available scope schedules before applying their
|
||||
operations. When the outer schedule selects an item node, execute its validated
|
||||
local schedule, compose the item mapping, then place the completed item into
|
||||
the outer scope. Do not globally flatten helper bodies into the collection.
|
||||
Resolve local paths and project final indexes using that actual scheduled
|
||||
contribution order. Main-item and underlay-item roots use the same model.
|
||||
|
||||
Retain the acquisition/inclusion graph independently of these composition scope
|
||||
boundaries so cycles crossing item boundaries remain detectable. On an active
|
||||
canonical repeat, retain the closing edge without rereading the source and use
|
||||
the shared cycle logic on the available inclusion graph. Non-active repeated
|
||||
inclusion still fails as `import.repeated`; no successful diamond import or
|
||||
global source deduplication is introduced.
|
||||
|
||||
Within inheritance, construct the graph of direct mapping siblings and their
|
||||
string parent edges, validate it, then produce templates in scheduled order.
|
||||
Null cancels a relationship without an edge. Keep enclosing-before-nested
|
||||
destination-local traversal and unresolved nested templates, as in ADR 0009.
|
||||
Do not reorder output mappings merely to mirror scheduling.
|
||||
|
||||
Directory discovery keeps candidate spellings separate from canonical access.
|
||||
One normalized candidate path per namespace is selected once per declaration;
|
||||
aliases retain separate candidates and existing admission rules. Directory
|
||||
filters run on discovered names. Exclusions never read content, but do not hide
|
||||
discovery, containment, or resource failures. For required named paths, verify
|
||||
every listed target exists even if excluded, without parsing excluded content.
|
||||
|
||||
Use existing bounded directory reads. For distinct roots, traverse each search
|
||||
with its applicable depth once and test the union of filename patterns per
|
||||
file; do not walk once per root/pattern pair. Overlapping roots may revisit
|
||||
entries under the existing visit accounting, but selected candidates deduplicate.
|
||||
Do not use global canonical-directory visitation to silently suppress aliases.
|
||||
Charge selected-source allowance after exclusions and local deduplication;
|
||||
discovery entries still count throughout, and excluded content costs no bytes.
|
||||
|
||||
Use host `Lstat`/root-relative operations for discovery controls and the standard
|
||||
native `fs.ReadLinkFS` capability where available. Follow the supplied filesystem's
|
||||
capabilities; do not require every `fs.FS` to implement link inspection or turn
|
||||
opaque native behavior into a universal sandbox guarantee. Host directory cycle
|
||||
keys are canonical paths; link-aware native traversal uses the corresponding
|
||||
resolved namespace directory identity. This traversal identity must not silently
|
||||
change the existing native source-admission identity. Track canonical directories
|
||||
on the active traversal path, not all previously visited directories. Distinguish
|
||||
directory traversal cycles from import dependency cycles; use `import.cycle`
|
||||
with the declaring site and a safe directory-cycle message for the former.
|
||||
|
||||
## Stage 1 — Build the shared scheduler
|
||||
|
||||
Create the private scheduler, tuple comparator, min-heap, and deterministic
|
||||
cycle finder specified above. Keep it independent of documents, filesystem
|
||||
handles, diagnostics, and callbacks. Document Kahn's algorithm, actual-cycle
|
||||
identification, and complexity beside its owner.
|
||||
|
||||
Tests own observable schedules: independent branches, unequal chains, multiple
|
||||
dependencies, shared dependencies at equal/unequal depths, a newly ready small
|
||||
key preceding an older large key, duplicate edges, self/multi-node cycles, and
|
||||
blocked non-cycle dependents. Permute insertion order while retaining keys and
|
||||
assert equal schedules/cycles. Do not assert heap internals.
|
||||
|
||||
Acceptance: focused scheduler tests pass; no loading behavior changes.
|
||||
|
||||
Completion: implemented in commit `167fcd4` (`Add deterministic dependency
|
||||
scheduler`). Verified with `go test ./internal/engine -run
|
||||
'^TestScheduleDependencies' -count=1`, `./scripts/check.sh`, and
|
||||
`git diff --check`. Loading behavior remains unchanged; Stage 2 onward remains.
|
||||
|
||||
## Stage 2 — Separate inheritance graph analysis from application
|
||||
|
||||
In `inheritance.go`, extract direct-sibling relationship analysis from the
|
||||
existing stack-driven application loop. Build keyed mapping nodes and edges;
|
||||
retain null controls, missing/non-mapping parent diagnostics, exact directive
|
||||
sites, and non-mapping siblings. Add a scheduled template-application helper
|
||||
reusing current merge hooks, origins, retained-node accounting, and unresolved
|
||||
nested template behavior. Keep the currently wired resolver unchanged until
|
||||
Stage 3; do not expose an alternate public path.
|
||||
|
||||
Acceptance: private scope tests demonstrate the shared schedule, ordinary/null
|
||||
relationships, real cycle sites, and preserved template values/ownership.
|
||||
|
||||
Completion: implemented in commit `2718fbb` (`Separate inheritance graph
|
||||
analysis and application`). Verified with
|
||||
`go test ./internal/engine -run 'Inheritance|Inherited' -count=1` and
|
||||
`./scripts/check.sh`. The wired resolver retains parent-first ordering; Stage 3
|
||||
will integrate the shared dependency-ready schedule.
|
||||
|
||||
## Stage 3 — Integrate scheduled inheritance
|
||||
|
||||
Replace the old sibling-frame resolution in the loading path with Stage 2's
|
||||
analysis and application. Remove superseded stack/state machinery. Retain
|
||||
`inheritance_walk.go`'s enclosing-before-nested scope discovery, sequence
|
||||
traversal, final projection budgets, and immutable retained evidence.
|
||||
|
||||
Review inheritance history/control/transfer ordering against the selected
|
||||
schedule, not the old DFS order. Add one unequal-branch integration case and
|
||||
reuse existing nested destination, cancellation, sensitivity, version ownership,
|
||||
and budget tests. Update the owning inheritance contract and internal overview
|
||||
to describe implemented scheduling, linking ADR 0015.
|
||||
|
||||
Acceptance: inheritance and facade inheritance suites pass; final values,
|
||||
presence, origins, and explanation agree; no alternate resolver remains.
|
||||
|
||||
Completion: implemented in commit `6e40c16` (`Schedule sibling inheritance by
|
||||
dependency readiness`). Verified with engine inheritance and facade inheritance
|
||||
tests, `./scripts/check.sh`, and `git diff --check`. Nested scope discovery,
|
||||
projection budgets, and retained evidence remain in their existing owners.
|
||||
|
||||
## Stage 4 — Retain the import inclusion graph during acquisition
|
||||
|
||||
Refactor `imports.go` and `load_input.go` so acquired contributions carry stable
|
||||
source identity, role, inclusion edges, and explicit scope ownership, rather
|
||||
than treating their slice position as both acquisition and application order.
|
||||
Separate item discovery from immediate item composition. Preserve parsed
|
||||
payloads and source-version processing results without rereading sources.
|
||||
|
||||
Keep existing wired application order for this extraction stage. Introduce the
|
||||
stable import keys and include-cycle integration with the shared cycle finder;
|
||||
keep non-active repeats invalid, including aliases and different inclusion
|
||||
routes. Do not expand successful shared dependency support for imports.
|
||||
|
||||
Acceptance: current ordinary/item import behavior tests pass, including a cycle
|
||||
through a nested item scope, repeated imports, source IDs, and handle closure.
|
||||
|
||||
Completion: implemented in commit `fbe9ae6` (`Retain import inclusion graph
|
||||
during acquisition`). Verified with `go test ./internal/engine -count=1`,
|
||||
`./scripts/check.sh`, and `git diff --check`. Import acquisition retains stable
|
||||
source keys, roles, inclusion edges, and explicit scope ownership; item
|
||||
scopes are composed after discovery in the existing contribution order.
|
||||
Non-active repeats remain invalid. Stage 5 onward remains.
|
||||
|
||||
## Stage 5 — Schedule ordinary main-tier contributions
|
||||
|
||||
Use the shared scheduler for ordinary mapping contributions in the main tier.
|
||||
Plan before composing. The main body follows its import dependencies; independent
|
||||
ready branches follow stable keys. Reuse `composeDocumentsWithPolicy` and the
|
||||
existing merge primitives. Update the corresponding order tests deliberately.
|
||||
|
||||
Replace assumptions that main is document/contribution zero in
|
||||
`importedVersionClaim`, source-root ownership, contexts, and history. Carry an
|
||||
explicit main source ID/role and locate its scheduled contribution. Keep source
|
||||
enumeration and version callbacks in acquisition order. Default-mode effective
|
||||
version checking still happens after main-tier assembly.
|
||||
|
||||
Acceptance: ordinary import values, registered sequence order, conflict sites,
|
||||
main-version protection, source records, and authored resource contexts agree.
|
||||
Document this implemented ordering boundary in the import contract.
|
||||
|
||||
Completion: implemented in commit `5ed0f55` (`Schedule ordinary import
|
||||
contributions by dependency`). Verified with `go test ./internal/engine
|
||||
-count=1`, `go test . -run
|
||||
'^TestFacadeSourceVersionEvidenceFollowsAssembledSequenceRanges$' -count=1`,
|
||||
`./scripts/check.sh`, and `git diff --check`. Acquisition records and version
|
||||
callbacks remain in traversal order; ordinary contributions use the shared
|
||||
dependency-ready schedule, with main-source ownership and projected evidence
|
||||
following the scheduled contributions. Item-scope ordering remains for later
|
||||
work.
|
||||
|
||||
## Stage 6 — Assemble item-local scopes under the scheduler
|
||||
|
||||
Integrate item nodes and local schedules for imported and selected main items.
|
||||
Complete helpers before local item assembly and outer placement. Reuse existing
|
||||
`projectNestedEvidence`, destination projection, sequence ranges, and diagnostic
|
||||
projection, correcting any assumptions about contribution order or index zero.
|
||||
Do not register application collection policies inside helper mapping scopes.
|
||||
|
||||
Keep local version checking/consumption before assembly and project retained
|
||||
source evidence through the actual assigned indexes. Failed-load source paths
|
||||
retain the existing unresolved-container semantics until indexes are known;
|
||||
do not invent an exact index to simplify a failure.
|
||||
|
||||
Acceptance: nested items, helper-supplied IDs, main items, multiple unequal
|
||||
branches, physical coordinates, sensitivity, history chronology, and item
|
||||
conflict paths pass engine and public tests. Run the focused race suite.
|
||||
|
||||
Completion: implemented in commit `af43d6f` (`Schedule item-local import
|
||||
scopes`). Verified with `go test ./internal/engine -count=1`,
|
||||
`go test -race ./internal/engine`, `go test . -count=1`, `go vet ./...`,
|
||||
`go build ./...`, the local documentation-link test, and `git diff --check`.
|
||||
`./scripts/check.sh` reaches the consumer examples and currently fails only on
|
||||
prompt-catalog ordering expectations assigned to Stage 18. Stage 7 onward
|
||||
remains.
|
||||
|
||||
## Stage 7 — Schedule underlay forests and preserve tier boundaries
|
||||
|
||||
In `load_request.go`, acquire all roots in a strict underlay group before
|
||||
scheduled group assembly. Combine their retained scope graphs rather than
|
||||
concatenating already composed per-input orders. Assign namespace anchors from
|
||||
explicit input positions. Include inputs without imports and item roots.
|
||||
|
||||
Keep per-tier repeated-source state, accounting, main acquisition timing, and
|
||||
underlay precedence unchanged. Independent groups never enter one scheduler;
|
||||
overlays/CLI remain later precedence operations. Maintain source layers,
|
||||
underlay indexes, global source IDs, and group-wide item path projections.
|
||||
|
||||
Acceptance: a group with interleaving dependency branches across input roots
|
||||
uses the expected schedule; separate tiers still replace sequences wholesale.
|
||||
Test same-label distinct native namespaces and copies of one namespace. NoSource
|
||||
still contributes no source or explicit empty root.
|
||||
|
||||
Completion: implemented in commit `014b995` (`Schedule underlay groups as
|
||||
dependency forests`). Verified with `go test ./internal/engine -count=1`,
|
||||
`go test . -count=1`, `go test -race ./internal/engine`, `go vet ./...`,
|
||||
`go build ./...`, the local documentation-link test, and `git diff --check`.
|
||||
`./scripts/check.sh` still reaches only the two prompt-catalog ordering
|
||||
expectations assigned to Stage 18. Stage 8 onward remains.
|
||||
|
||||
## Stage 8 — Define private defaults and the final declaration parser
|
||||
|
||||
Add the private defaults snapshot/validation and resolved-policy helpers. Add
|
||||
the final declaration parser as an unwired private helper until Stage 14; retain
|
||||
the current wired parser only until that cutover. Store declaration paths and
|
||||
field locations for exact diagnostics and later no-selection evidence.
|
||||
|
||||
Cover the roadmap's selector/option matrix: scalar/list normalization, exactly
|
||||
one selector, unknown fields including removed `optional`, null/wrong-type
|
||||
members, empty lists, duplicates, wildcard validation, literal destinations,
|
||||
placement, and effective-recursion/depth applicability. Parse all declarations
|
||||
before selecting candidates; invalid inactive fields still fail.
|
||||
|
||||
Acceptance: parser/policy tests cover omitted, explicit false, explicit empty,
|
||||
application defaults, and invalid default patterns without source access. No
|
||||
new YAML syntax is accepted by public loading yet.
|
||||
|
||||
Completion: implemented in commit `b498e5b` (`Define private import defaults
|
||||
and declaration parsing`). Verified with focused parser/policy tests, the full
|
||||
engine suite, `go vet ./...`, `go build ./...`, a focused engine race run, and
|
||||
`git diff --check`. The everyday check still reaches the two known consumer
|
||||
sequence-order expectation failures scheduled for Stage 18; public loading
|
||||
continues to use the existing parser. Stage 9 onward remains.
|
||||
|
||||
## Stage 9 — Implement named-file selection and exclusions
|
||||
|
||||
Build a private selector result containing normalized lexical candidates and
|
||||
selection outcome. Implement string/list `path`, local candidate deduplication,
|
||||
basename exclusions, and optional/required presence semantics. Check excluded
|
||||
required target existence without opening content. Do not treat permission,
|
||||
containment, regular-file admission, or parser failures as optional absence.
|
||||
|
||||
Keep selector results distinct from source-admission identities; no source IDs
|
||||
or read counts are assigned merely by selection. Preserve current validity rules
|
||||
for explicit paths and existing source-access behavior for named symlinks and
|
||||
hidden targets. Do not apply search-only controls to this selector.
|
||||
|
||||
Acceptance: temporary-file tests cover partial missing lists, required/excluded
|
||||
missing targets, explicit empty lists, alias candidates, and unread excluded
|
||||
content. No fake source records are created for absence.
|
||||
|
||||
Completion: implemented in commit `5f33e1e` (`Select named imports before source
|
||||
admission`). Verified with named-selection tests using temporary host files and
|
||||
a stat-only native filesystem, the full engine suite, a focused race run,
|
||||
`go vet ./...`, `go build ./...`, and `git diff --check`. The everyday check
|
||||
continues to fail only on the two consumer sequence-order expectations assigned
|
||||
to Stage 18. Stage 10 onward remains.
|
||||
|
||||
## Stage 10 — Implement shallow directory selection
|
||||
|
||||
Add literal directory string/list search, scalar/list basename patterns, and
|
||||
exclusions through the same candidate-selection result. Start with nonrecursive
|
||||
selection. Missing roots contribute nothing; an inaccessible or invalid supplied
|
||||
root is not absence. Directories are traversal containers, not documents.
|
||||
|
||||
Implement set union and difference, normalized candidate-path deduplication,
|
||||
and requiredness across the complete selected union. Preserve nil/unrestricted
|
||||
versus explicit-empty patterns and alias spelling during filtering. Test multiple
|
||||
roots, overlapping roots/patterns, extensionless and hidden filenames, and a
|
||||
missing root alongside one that supplies the required match.
|
||||
|
||||
Acceptance: shallow directory tests pass with actual host files and a native
|
||||
filesystem; no per-root/per-pattern requiredness is introduced.
|
||||
|
||||
Completion: implemented in commit `55a59de` (`Select shallow directory import
|
||||
sources`). Verified with shallow-selection tests on host and native filesystems,
|
||||
the full engine suite, a focused race run, `go vet ./...`, `go build ./...`, and
|
||||
`git diff --check`. The everyday check still reaches only the consumer
|
||||
sequence-order expectations reserved for Stage 18. Stage 11 onward remains.
|
||||
|
||||
## Stage 11 — Add recursive directory traversal and depth accounting
|
||||
|
||||
Extend the walker with explicit traversal frames, effective recursion, and the
|
||||
settled depth convention. Apply filename patterns only to files, never to prune
|
||||
directories. Share bounded directory-reading and entry accounting with glob
|
||||
discovery. Do not rescan a root per pattern or use the import-edge depth limit
|
||||
as a directory-recursion limit.
|
||||
|
||||
Acceptance: direct files are depth 1, depth 2 includes one descendant level,
|
||||
zero/omission are unlimited subject to existing budgets, and depth requires
|
||||
effective recursion. Cover application-enabled recursion, explicit false,
|
||||
large depth integers, empty searches, and limits using test-controlled budgets.
|
||||
|
||||
Completion: implemented in commit `77d9729` (`Add recursive directory search
|
||||
with depth limits`). Verified with recursive depth/limit tests, existing glob
|
||||
tests, the full engine suite and race suite, `go vet ./...`, `go build ./...`,
|
||||
and `git diff --check`. The everyday check still reaches only the consumer
|
||||
sequence-order expectations reserved for Stage 18. Stage 12 onward remains.
|
||||
|
||||
## Stage 12 — Add discovery symlink and hidden controls
|
||||
|
||||
Implement `follow`/`hidden` for directory discovery using capability-aware
|
||||
filesystem helpers. Explicit roots keep existing access semantics; discovered
|
||||
hidden directories prune when hidden is false, discovered symlink directories
|
||||
do not descend when follow is false, and analogous files are not selected.
|
||||
|
||||
When following links, track active-path directory identities and fail actual
|
||||
loops without treating earlier completed branches as cycles. Preserve namespace
|
||||
containment and candidate spellings. Cover contained links, escapes, filename
|
||||
filtering on aliases, self/ancestor loops, and separate noncyclic aliases that
|
||||
later retain ordinary repeated-source admission behavior.
|
||||
|
||||
Acceptance: host filesystem behavior and a focused link-aware native fixture
|
||||
pass; plain native filesystems still load without new capability requirements.
|
||||
All success/failure paths close library-owned directory and root handles.
|
||||
|
||||
Completion: implemented in commit `9b99463` (`Follow links safely during
|
||||
directory discovery`). Verified with directory-selection tests, the full engine
|
||||
suite and race suite, native resource-close tests, `go vet ./...`, `go build
|
||||
./...`, and `git diff --check`. Host traversal uses root-relative link inspection;
|
||||
native traversal uses `fs.ReadLinkFS` when available and leaves plain native
|
||||
filesystems usable. `./scripts/check.sh` still fails only on prompt-catalog
|
||||
sequence-order expectations reserved for Stage 18. Stage 13 onward remains.
|
||||
|
||||
## Stage 13 — Integrate glob filtering and final candidate accounting
|
||||
|
||||
Adapt `glob.go` to the common selection pipeline. Keep path wildcards and
|
||||
literal components distinct so follow/hidden filters affect only wildcard-
|
||||
discovered entries, not explicitly named components. Apply basename exclusions
|
||||
to final candidates. Preserve regular-file/source validity behavior rather than
|
||||
silently accepting invalid glob-selected sources.
|
||||
|
||||
Move source-allowance checking after exclusions and declaration-local
|
||||
deduplication. Intermediate traversal candidates are bounded by entry visits,
|
||||
not by the final source allowance. Charge bytes only when selected content is
|
||||
read; never read excluded content. A tiny private source limit with many excluded
|
||||
positive matches must succeed when its final selected set fits.
|
||||
|
||||
Acceptance: existing glob syntax, containment, alias, and visit-budget tests
|
||||
pass alongside discovery-control and post-filter source-limit regressions.
|
||||
|
||||
Completion: implemented in commit `e4abff4` (`Filter and account final glob
|
||||
candidates`). Verified with focused glob tests, the full engine suite and race
|
||||
suite, `go vet ./...`, `go build ./...`, and `git diff --check`. Glob expansion
|
||||
now uses bounded shared directory reads, preserves wildcard candidate spellings,
|
||||
applies discovery controls only to wildcard components, and filters exclusions
|
||||
before source admission. `./scripts/check.sh` still fails only on prompt-catalog
|
||||
sequence-order expectations reserved for Stage 18. Stage 14 onward remains.
|
||||
|
||||
## Stage 14 — Wire final syntax and load-wide defaults through the engine
|
||||
|
||||
Add the private defaults value to ordinary/grouped requests, snapshot/validate
|
||||
it before acquisition, and pass it to every enabled main/underlay/import
|
||||
declaration. Wire Stage 8's parser and Stages 9–13's selection pipeline into
|
||||
graph acquisition. Remove the old declaration parser and `optional` field;
|
||||
there is one final parser and one absence policy.
|
||||
|
||||
Include `source_version.go`'s stored import declarations in this wiring: syntax
|
||||
parsing stays independent of source-version interpretation, and every stored
|
||||
declaration receives the same effective policy before discovery. Do not bypass
|
||||
validation when a consumed header or empty payload leaves no application fields.
|
||||
|
||||
Keep imports disabled unless already enabled by the consumer. Selected overlays
|
||||
still cannot enable their own imports; defaults do not change that boundary.
|
||||
Validate effective placement/depth and declaration values before relying on a
|
||||
missing target or empty selection to skip work. Declarations do not inherit
|
||||
parent control values.
|
||||
|
||||
Migrate existing tests and maintained fixture declarations from `optional` to
|
||||
the intended `required` behavior. Tests expecting a missing import to fail must
|
||||
request requiredness, not weaken their checks. Update the owning configuration
|
||||
contract to describe the now-implemented final YAML interface and breaking
|
||||
absence/order changes. Keep exact public Go defaults documentation for Stage 16.
|
||||
|
||||
Acceptance: all engine imports, glob, destinations, source-version, and layered
|
||||
tests pass under final syntax. An explicit regression rejects `optional`;
|
||||
another proves missing imports are optional without setup. Run everyday checks.
|
||||
|
||||
Completion: implemented in commit `1e5bffc` (`Wire final import declarations
|
||||
through loading`). Verified with `go test ./internal/engine -count=1`,
|
||||
`go test -race ./internal/engine`, `go vet ./...`, `go build ./...`, the local
|
||||
documentation-link test, and `git diff --check`. `./scripts/check.sh` reaches
|
||||
only the two prompt-catalog sequence-order expectations reserved for Stage 18.
|
||||
Stage 15 onward remains.
|
||||
|
||||
## Stage 15 — Retain no-selection evidence across load outcomes
|
||||
|
||||
Add the private records from the settled inspection design to declaring source
|
||||
records and their clone functions. Record only completed, optional, whole-
|
||||
declaration empty outcomes. Preserve evidence for main and imported declarations
|
||||
and underlay inputs through composition, nested item placement, overlays,
|
||||
resolution entry state, final publication, and failed-load accumulation.
|
||||
|
||||
Use source IDs to attach outcomes, not scheduled positions. Reuse source-record
|
||||
transport; do not add synthetic imported sources or warnings. Keep evidence
|
||||
source-local and detached even when its source destination remains unresolved.
|
||||
|
||||
Acceptance: tests cover no matches, missing paths, empty lists, exclusions,
|
||||
partial path-list omission (no record), and required empty paths (no record).
|
||||
An error after a prior empty optional declaration retains that established
|
||||
evidence, but does not fabricate later outcomes. Mutating query results cannot
|
||||
alter subsequent snapshots or another load.
|
||||
|
||||
Completion: implemented in commit `e433988` (`Retain empty optional import
|
||||
selection evidence`). Verified with focused evidence tests,
|
||||
`go test ./internal/engine -count=1`, `go test -race ./internal/engine`,
|
||||
`go test . -count=1`, `go vet ./...`, `go build ./...`, and
|
||||
`git diff --check`. Source-local evidence survives composition, nested item
|
||||
placement, underlays, overlays, resolution entry, final publication, and failed
|
||||
loads; the known prompt-catalog ordering expectation updates remain assigned to
|
||||
Stage 18. Stages 16–19 remain.
|
||||
|
||||
## Stage 16 — Expose defaults and inspection through the public facade
|
||||
|
||||
Add root-owned declarations from this plan, `ImportDefaults` fields on ordinary
|
||||
and layered options, and `SourceInfo.ImportSelections`. Add explicit converters
|
||||
and engine transport types/helpers as needed, following existing root-wrapper
|
||||
ownership. Snapshot pointers/slices and preserve nil versus non-nil empty
|
||||
patterns. Do not add defaults on individual inputs or a new result query API.
|
||||
|
||||
Exercise final behavior through `Load` and `LoadLayered`, including defaults
|
||||
applied to transitive and underlay imports, explicit declaration exceptions,
|
||||
invalid default preflight without reader consumption, and result/error/resolver
|
||||
source inspection ownership. Update declaration comments and the owning Go API
|
||||
guidance when the surface becomes available.
|
||||
|
||||
Acceptance: public facade/import-boundary tests and representative engine/public
|
||||
equivalence cases pass; no internal type leaks through exported signatures.
|
||||
|
||||
Completion: implemented in commit `2413161` (`Expose import defaults and
|
||||
selection evidence`). Verified with full root and engine tests, public facade,
|
||||
dependency boundary, source-equivalence, and documentation-link tests,
|
||||
`go vet ./...`, `go build ./...`, formatting, and `git diff --check`. The
|
||||
repository-wide `./scripts/check.sh` remains blocked only by the two
|
||||
prompt-catalog sequence-order expectations assigned to Stage 18. Stages 17–19
|
||||
remain.
|
||||
|
||||
## Stage 17 — Audit cross-feature lifecycle and ownership
|
||||
|
||||
Review and fill meaningful gaps rather than adding a full Cartesian-product
|
||||
suite. Reuse existing regressions for version protection, consumed-header
|
||||
sensitivity, scope diagnostics, NoSource, and historical removed descendants.
|
||||
|
||||
Add representative combined cases for scheduled discovered items with helper-
|
||||
supplied IDs; nested destinations and actual assigned indexes; required versus
|
||||
if-present source versions; main-version ownership after main moves later in a
|
||||
schedule; inherited relative resources; superseded relationships; and final
|
||||
configuration/metadata/explanation agreement. Ensure consumed headers cannot
|
||||
silently supply record IDs and secret values remain absent from diagnostics
|
||||
and inspection. Preserve callback phases and source-local callback ordering.
|
||||
|
||||
Compare an equivalent keyed graph through import and inheritance integrations:
|
||||
both use the scheduler's eligible-node rule despite different domain operations.
|
||||
Keep acquisition-source records stable when only graph construction order is
|
||||
permuted in private scheduler/integration fixtures; do not promise unchanged IDs
|
||||
when authored discovery order itself changes.
|
||||
|
||||
Acceptance: run `go test -race ./internal/engine`, focused root facade tests,
|
||||
and everyday checks. Record any genuine unresolved gap before marking complete.
|
||||
|
||||
Completion: implemented in commit `55abcf8` (`Audit scheduled import and
|
||||
inheritance lifecycles`). Verified with full root and engine tests,
|
||||
`go test -race ./internal/engine`, `go vet ./...`, `go build ./...`, formatting,
|
||||
and `git diff --check`. Added combined coverage for discovered items with helper-
|
||||
supplied IDs, nested placement, inherited relative-resource provenance,
|
||||
consumed secret headers, and scheduler parity under graph-construction
|
||||
permutations. The everyday `./scripts/check.sh` still fails only on the two
|
||||
prompt-catalog order expectations assigned to Stage 18. Stages 18–19 remain.
|
||||
|
||||
## Stage 18 — Maintain simple and complex consumer examples
|
||||
|
||||
Update the prompt-catalog example for directory search, multiple patterns,
|
||||
exclusions, and helper-supplied IDs while retaining explicit registration and
|
||||
application final validation. Include application import defaults and a
|
||||
declaration exception, plus inspection of a deliberately empty optional search.
|
||||
Keep a short simple single-file/required-file example so adoption does not
|
||||
require understanding all discovery controls.
|
||||
|
||||
Update fixture expectations for scheduled sequence positions. Verify mounted
|
||||
resource origins, whole-sequence precedence, and quiet absence through executable
|
||||
examples. Do not add a consumer-side discovery, merge, scheduling, or provenance
|
||||
reconstruction pass. All fixtures remain local, deterministic, and secret-free.
|
||||
|
||||
Acceptance: consumer tests and executable public examples pass; the consumer
|
||||
guide links complete copyable examples rather than duplicating them.
|
||||
|
||||
Completion: implemented in commit `de3666a` (`Expand prompt catalog discovery
|
||||
example`). Verified with the full consumer suite, executable public examples,
|
||||
`./scripts/check.sh`, local documentation links, formatting, and
|
||||
`git diff --check`. The maintained catalog now demonstrates load-wide multiple
|
||||
patterns, a declaration exception, exclusions, empty optional-search evidence,
|
||||
helper-provided IDs, scheduled sequence positions, resource-origin lookup, and
|
||||
whole-sequence precedence. A simple required-file example remains available.
|
||||
Stage 19 remains.
|
||||
|
||||
## Stage 19 — Finalize documentation and validate the completed unit
|
||||
|
||||
Align README orientation, `docs/config.md`, public declaration comments,
|
||||
`docs/internal/overview.md`, relevant focused internal docs, and the development
|
||||
reading/test map. Document Kahn's algorithm, set union/difference, and active-path
|
||||
cycle detection beside their owners. State acquisition versus resolution order,
|
||||
explicit defaults precedence, no-selection evidence, and source namespace/key
|
||||
construction without duplicating canonical contracts.
|
||||
|
||||
Update architecture policy to recognize implemented explicit directory discovery
|
||||
and shared scheduling while retaining the boundary against unrequested implicit
|
||||
application-path discovery. Update only ADR status/navigation as needed to link
|
||||
implemented contracts; accepted decision text stays unchanged. Leave release
|
||||
notes/version choice and roadmap retirement for separately authorized release
|
||||
preparation. Do not introduce release gates, CI waiting, consumer validation
|
||||
requirements, dependencies, or unrelated defaults/security changes.
|
||||
|
||||
Run the everyday `./scripts/check.sh`, `go test -race ./...`, consumer/examples
|
||||
tests, local documentation links, and `git diff --check`; reuse checks already
|
||||
covering unchanged final content instead of repeating them gratuitously. Review
|
||||
the baseline-to-final public and YAML diff for removed `optional`, changed
|
||||
requiredness and sequence ordering, and accurately documented native filesystem
|
||||
capability limits. Inspect source and history indexes for accidental quadratic
|
||||
rescanning; no timing thresholds or busy-host benchmark evidence are required.
|
||||
|
||||
Acceptance: all stages are complete, the final implemented behavior matches the
|
||||
roadmap and ADRs, validations are recorded here, and the implementation workspace
|
||||
is committed and clean. No release is published by this plan.
|
||||
|
||||
Completion: implemented in commit `f6a6b01` (`Finalize import discovery
|
||||
documentation and audits`). Verified with `./scripts/check.sh`,
|
||||
`go test -race ./...`, focused import/scheduler/inheritance/source-version and
|
||||
public-defaults tests, consumer examples, local documentation links, formatting,
|
||||
and `git diff --check`. Reviewed baseline-to-final public and YAML changes for
|
||||
the `required` syntax cutover, requiredness defaults, and scheduled sequence
|
||||
ordering; reviewed native filesystem capability wording and source/history
|
||||
indexes. Replaced the source-version evidence nested scan with one source-ID
|
||||
index. No release was published. The implementation workspace is committed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None required to begin implementation. The API, evidence shape, ordering keys,
|
||||
scope boundaries, and cycle treatment are selected above. Private helper names
|
||||
and file splits may be adjusted for clarity without changing those decisions.
|
||||
If implementation reveals a genuine conflict between an accepted requirement
|
||||
and existing contracts, report it rather than silently strengthening or weakening
|
||||
either one.
|
||||
@@ -1,765 +0,0 @@
|
||||
# Import Ergonomics And Permissive Defaults
|
||||
|
||||
Status: accepted design; implementation pending. This document captures planned
|
||||
behavior and requirements. The [implementation plan](implementation.md) owns
|
||||
delivery sequencing and the concrete API and integration choices.
|
||||
The selected direction separates directory searches, filename matching,
|
||||
recursion, and absence handling. Optional-by-default imports are the intended
|
||||
direction. The consumer-directed and Ansible-aligned decisions below are settled;
|
||||
API spelling and implementation details are specified by the implementation plan.
|
||||
All new syntax below is illustrative and unimplemented.
|
||||
|
||||
The durable architectural decisions are recorded in
|
||||
[ADR 0015: Share Dependency-Ready Ordering Across Imports and Inheritance](../adr/0015-shared-dependency-ready-ordering.md)
|
||||
and
|
||||
[ADR 0016: Separate Import Selection, Traversal, and Presence Requirements](../adr/0016-independent-import-controls.md).
|
||||
This roadmap owns implementation status and the detailed requirements for
|
||||
carrying out those accepted decisions.
|
||||
|
||||
## Guiding Philosophy
|
||||
|
||||
ConfigKit should provide predictable configuration mechanisms and avoid imposing
|
||||
application policy. Its defaults should permit optional contributions to be
|
||||
absent, with explicit options for increased strictness when needed. That choice
|
||||
may come from an operator or the consuming application; authored import
|
||||
declarations can also express requirements on contributions.
|
||||
|
||||
Permissiveness should distinguish absence from invalid supplied configuration:
|
||||
|
||||
> Optional contributions may be absent by default. Consumers can explicitly
|
||||
> require them. Supplied configuration must still be structurally valid and
|
||||
> interpretable under the selected composition rules.
|
||||
|
||||
This is a general design direction worth making more explicit and consistent
|
||||
across ConfigKit. It is not blanket authorization to weaken existing contracts,
|
||||
ignore errors, or add a global permissiveness switch. Other defaults should be
|
||||
evaluated individually against this principle.
|
||||
|
||||
Discovery should accept hidden entries and follow symlinks by default, with
|
||||
options to narrow either behavior. An omitted filename filter should not impose
|
||||
an implicit extension restriction. These defaults do not waive containment,
|
||||
source validity, composition, or resource-limit rules. Recursion remains a
|
||||
separate choice about search scope, not a measure of permissiveness.
|
||||
|
||||
This is a hard-breaking pre-1.0 change. Do not build temporary aliases,
|
||||
translation layers, compatibility modes, or staged deprecation support. Evaluate
|
||||
the final interface on its own merits, without retaining syntax solely to ease
|
||||
upgrades.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
The [current import contract](../config.md#explicit-imports) uses `path` or
|
||||
`glob`, with required imports by default and `optional: true` to permit absence.
|
||||
Glob matching uses slash-based Go `path.Match` syntax. Recursive `**` syntax is
|
||||
not supported; authors can match particular depths with patterns such as
|
||||
`prompts/*.yaml` and `prompts/*/*.yaml`.
|
||||
|
||||
Current imports expand depth-first in declaration order, with each ordinary
|
||||
source contributing its body before its imports. The existing
|
||||
[sibling inheritance resolver](../config.md#sibling-inheritance) instead follows
|
||||
dependency chains before resolving dependents, using sorted sibling names and
|
||||
an explicit stack. Neither implements the shared ordering selected below.
|
||||
|
||||
The proposed changes concern naming, discovery, absence handling, and uniform
|
||||
dependency ordering for imports and `extends`. Import destinations, strict
|
||||
composition, inheritance overlay semantics, tier precedence, and resource
|
||||
identity retain their distinct roles.
|
||||
|
||||
## Proposed Import Vocabulary
|
||||
|
||||
Use direct-file, directory-search, and path-glob forms. Directory searches
|
||||
separate literal search roots from filename filters; path globs additionally
|
||||
support wildcard directory segments and selection at particular depths:
|
||||
|
||||
```yaml
|
||||
configkit:
|
||||
imports:
|
||||
- dir: prompts
|
||||
pattern: '*.yaml'
|
||||
recursive: true
|
||||
destination: [prompts]
|
||||
placement: sequence-item
|
||||
- path: defaults.yaml
|
||||
required: true
|
||||
```
|
||||
|
||||
The first declaration searches `prompts` and its descendants for matching
|
||||
filenames, placing each source mapping as one prompt item, and permits no
|
||||
matches. The second explicitly requires its target file. Sequence composition
|
||||
still requires the consumer's existing registration, as explained below.
|
||||
|
||||
The fields have distinct jobs:
|
||||
|
||||
| Field | Proposed meaning |
|
||||
| --- | --- |
|
||||
| `path` | Import a named file or list of named files, relative to the declaring file |
|
||||
| `dir` | Search a literal directory or list of literal directories, relative to the declaring file |
|
||||
| `pattern` | A filename wildcard or list of wildcards; select files matching any effective pattern, using the default precedence below |
|
||||
| `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file |
|
||||
| `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results |
|
||||
| `recursive` | Library default false, overridable by an application default; include descendant directories when effectively true |
|
||||
| `depth` | Bound directory-search depth; supplying it requires effective recursion to be enabled; 1 means direct files, 0 or omission means unlimited when recursive |
|
||||
| `follow` | Follow symlinks by default; false disables following within the discovery scope defined below |
|
||||
| `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below |
|
||||
| `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search |
|
||||
| `destination`, `placement` | Apply the existing placement rules to each selected source |
|
||||
|
||||
This follows the familiar separation of traversal and filtering in Ansible's
|
||||
[find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html)
|
||||
and [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html)
|
||||
interfaces, without adopting all their spellings, defaults, or matching modes.
|
||||
|
||||
## Decisions Resolved By Project Direction
|
||||
|
||||
- **No transition support.** Use the final interface immediately in the selected
|
||||
pre-1.0 minor release. Remove `optional` in favor of `required`; do not accept
|
||||
it as an alias or normalize old declarations at runtime.
|
||||
- **Optional contributions by default.** `required: true` opts into failure on
|
||||
absence. Supplied invalid configuration continues to fail.
|
||||
- **No library-imposed filename restriction.** Omitting `pattern` selects all
|
||||
filenames unless the application supplies a default pattern. Authors can
|
||||
override that default explicitly. There is no library default YAML extension
|
||||
list; extensionless sources remain discoverable.
|
||||
- **Follow symlinks by default, with an opt-out.** Provide `follow: false` for
|
||||
callers who want discovery without symlink following. Applicable containment
|
||||
and filesystem contracts remain in force.
|
||||
- **Include hidden entries by default, with an opt-out.** Provide `hidden: false`
|
||||
to exclude hidden files and prune hidden directories from discovery.
|
||||
- **One dependency-ordering algorithm for imports and `extends`.** Both use the
|
||||
shared dependency-ready scheduler defined below. This is settled independently
|
||||
of the existing implementations and includes refactoring both resolvers.
|
||||
|
||||
OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-3
|
||||
(directive validation), OD-4 (discovery boundaries and cycles), OD-5 (policy
|
||||
precedence), and OD-6 (no-selection evidence) are settled in the sections below.
|
||||
Supplying `depth` requires effective recursion to be enabled, including when
|
||||
enabled through an application default. This supersedes the earlier direction
|
||||
to accept but ignore depth when recursion is disabled.
|
||||
|
||||
## Decisions Resolved From Ansible
|
||||
|
||||
The comparison covers `ansible.builtin.find`, `ansible.builtin.include_vars`,
|
||||
`ansible.builtin.fileglob`, and `ansible.windows.win_find`. A consistent approach
|
||||
means agreement among the relevant interfaces that offer the capability, not
|
||||
that every Ansible module exposes it. Existing ConfigKit contracts and the
|
||||
already selected optional-by-default direction remain authoritative; this is
|
||||
not a general Ansible compatibility mode.
|
||||
|
||||
The following choices are settled:
|
||||
|
||||
- **Separate traversal from filename filtering.** Directory roots and filename
|
||||
filters have separate roles; filters operate on basenames in visited
|
||||
directories. A directory failing the filename filter does not prune its
|
||||
descendants. This agrees with the directory-search interfaces above.
|
||||
See [find's parameters](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters)
|
||||
and the [include_vars implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py).
|
||||
- **Include an optional integer `depth` control.** All three directory-search
|
||||
interfaces use that spelling. Direct files are at depth 1, and each additional
|
||||
directory adds one. Omission permits unlimited depth when recursion is
|
||||
enabled. See [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html#parameters)
|
||||
and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters).
|
||||
- **Treat `depth: 0` as unlimited.** This follows `include_vars` and agrees with
|
||||
the current [find implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py)
|
||||
and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1).
|
||||
The latter two document unlimited omission more clearly than explicit zero;
|
||||
this choice uses source-confirmed behavior, not a claimed universal documented
|
||||
Ansible guarantee. Negative values are invalid under the settled directive
|
||||
validation rules below.
|
||||
- **Directory searches select files, not directory documents.** Directories are
|
||||
traversal containers, even when their names match `pattern`. This agrees with
|
||||
the default file selection in the find modules and the file-oriented
|
||||
[fileglob lookup](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/fileglob_lookup.html).
|
||||
Existing regular-file and containment rules still govern opening selected
|
||||
sources, together with the selected discovery controls.
|
||||
|
||||
The accepted `recursive` spelling is retained. The formerly tentative
|
||||
`max_depth` spelling becomes `depth`. Resource budgets remain separate from
|
||||
depth, and the current limits still apply to an unlimited-depth search.
|
||||
|
||||
## Shared Dependency-Ready Ordering
|
||||
|
||||
**Status: settled for imports and `extends`; not yet implemented.** Use
|
||||
deterministic dependency-ready ordering: Kahn's topological-sort algorithm with
|
||||
a min-heap of eligible nodes. Both resolvers must use the same algorithm in the
|
||||
same way, through one shared internal scheduling implementation, common
|
||||
tie-breaking rules, and common cycle-path logic.
|
||||
|
||||
### Scheduling Rule
|
||||
|
||||
A dependency edge means that the dependent cannot complete until its dependency
|
||||
has completed. For each dependency graph:
|
||||
|
||||
1. Count each node's outstanding dependencies and identify initially eligible
|
||||
nodes: those with none.
|
||||
2. Choose the eligible node with the smallest stable ordering key.
|
||||
3. Append it to the schedule, decrement its dependents' outstanding counts, and
|
||||
immediately make newly eligible nodes available for the next selection.
|
||||
4. Continue until all nodes are scheduled. If unfinished nodes remain with no
|
||||
eligible node, reject the graph and identify an actual dependency cycle.
|
||||
|
||||
Use stable source or definition identities to construct ordering keys, with
|
||||
the same deterministic comparison rule in both resolvers. Map iteration order
|
||||
and discovery-assigned source IDs must not supply the tie-breaker. Specify key
|
||||
construction for source namespaces and logical definition paths during
|
||||
implementation planning; it must distinguish nodes within each scheduled graph.
|
||||
|
||||
Construct and validate a complete schedule for the available graph before
|
||||
applying its resolution operations. A blocked dependent is not necessarily a
|
||||
cycle member: diagnostics should identify the actual cycle using shared,
|
||||
deterministic cycle-path logic. Source discovery and parsing may already have
|
||||
occurred; this rule is not a promise to avoid reading invalid configurations.
|
||||
|
||||
### Scope And Integration
|
||||
|
||||
Discovery and resolution are separate operations. An importing source must be
|
||||
read to discover its imports, but its contribution need not be assembled first.
|
||||
Discovery order is not a substitute for the selected resolution schedule.
|
||||
|
||||
Both resolvers construct dependency graphs and apply resolver-specific operations
|
||||
through the shared scheduler. Imports retain strict composition; `extends`
|
||||
retains inheritance overlay behavior. Scheduling does not authorize replacement
|
||||
within a strict composition tier or change precedence between tiers. Shared
|
||||
dependencies can be scheduled once where valid; this does not relax the existing
|
||||
prohibition on repeated source inclusion for imports.
|
||||
|
||||
Preserve item-local helper scopes and enclosing-before-nested inheritance
|
||||
requirements. An imported item completes its helper contributions before being
|
||||
placed as an item. Enclosing inheritance establishes the context in which nested
|
||||
relationships are discovered and resolved. Use the same scheduler at those
|
||||
scope boundaries, with the actual prerequisites represented; do not flatten raw
|
||||
files or mappings in a way that loses their scope. Inheritance still resolves
|
||||
after source assembly and precedence have established the effective document.
|
||||
|
||||
The integration must account for resulting sequence positions, provenance,
|
||||
history, and diagnostic ordering. It must distinguish source acquisition records
|
||||
from assembly order rather than assuming they are the same. Review both
|
||||
resolvers' contracts and behavioral tests when implementing the new schedule;
|
||||
preserving the old traversal is not a design objective.
|
||||
|
||||
### Rationale And Validation
|
||||
|
||||
Dependency-ready ordering selects the smallest currently eligible node without
|
||||
adding branch preference, batch barriers, or distance-from-root priority. With
|
||||
a fixed total ordering key, it gives the lexicographically smallest valid
|
||||
dependency-first schedule. It can interleave independent branches; branch
|
||||
contiguity is not part of the chosen ordering contract.
|
||||
|
||||
The comparison considered deterministic depth-first postorder, dependency
|
||||
rounds, and descending longest-path depth as well. All can satisfy dependency
|
||||
correctness, but their additional ordering policies did not justify choosing
|
||||
them over the uniform eligibility rule. This decision gives no preference to
|
||||
the existing implementation or to minimizing pre-1.0 refactoring.
|
||||
|
||||
The scheduler requires O(E + V log V) time and O(V + E) memory for V nodes and
|
||||
E dependency edges, assuming constant-time key comparisons. Source access,
|
||||
parsing, merging, and provenance work are separate costs.
|
||||
|
||||
Validate independent branches, unequal chains, branches with multiple
|
||||
dependencies, shared dependencies at equal and unequal depths, cycles with
|
||||
blocked non-cycle dependents, and nested scopes. Equivalent valid graphs and
|
||||
ordering keys must produce identical schedules through the import and
|
||||
inheritance integrations. Verify independence from graph insertion order and
|
||||
preserve each resolver's domain-specific admissibility and combination rules.
|
||||
|
||||
## Discovery Semantics
|
||||
|
||||
### Filename Matching
|
||||
|
||||
`pattern` applies to a file's basename, not its full relative path. For example,
|
||||
`*.yaml` matches both `prompts/one.yaml` and
|
||||
`prompts/weather/daily/summary.yaml` when the latter's directory is visited.
|
||||
Recursion changes which directories are visited; it does not change what `*`
|
||||
means. A nonrecursive search considers only files directly inside `dir`.
|
||||
|
||||
Filename filters use the candidate name supplied by the declaration or
|
||||
encountered during discovery, not the canonical symlink target's basename.
|
||||
For example, a symlink named `welcome.draft.yaml` matches
|
||||
`exclude: '*.draft.yaml'` even if its target is named `welcome.yaml`. This
|
||||
selection rule does not change canonical source identity or containment checks.
|
||||
|
||||
**OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard
|
||||
grammar for filename filters. When neither the declaration nor the application
|
||||
supplies `pattern`, select all filenames, with no implicit extension restriction.
|
||||
Do not add regular expressions, automatic dialect detection, or recursive `**`
|
||||
syntax as part of this work.
|
||||
|
||||
Retain `glob` as a permanent, separate selector using the existing slash-based
|
||||
path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects
|
||||
files in a particular directory layout without selecting all descendants.
|
||||
Literal `dir` roots and basename filters do not express that same search.
|
||||
This capability justifies retaining `glob` independently of compatibility;
|
||||
it uses the same new absence policy as other selectors. `glob` remains a
|
||||
single string; list support is being added to `path`, `dir`, and the filename
|
||||
filters.
|
||||
|
||||
### Selector Lists And Set Operations
|
||||
|
||||
Accept a string or a list of strings for `path`, `dir`, and `pattern`. Normalize
|
||||
a scalar to a singleton list so both forms use the same selection logic.
|
||||
Listed paths and directory roots resolve relative to the declaring file.
|
||||
List order does not introduce precedence or replace the shared dependency-ready
|
||||
resolution schedule.
|
||||
|
||||
For directory roots D and filename patterns P, search the **Cartesian product**
|
||||
D × P and take the **union** of the matching candidate files:
|
||||
|
||||
`S = ⋃ {matches(d, p) | (d, p) ∈ D × P}`
|
||||
|
||||
Each root uses the declaration's recursion, depth, and applicable discovery
|
||||
controls. Resolve an omitted `pattern` from the application default first;
|
||||
if neither supplies patterns, select all filenames in the visited directories
|
||||
instead. This describes selection semantics, not a requirement to
|
||||
materialize the product or traverse a directory again for each pattern.
|
||||
|
||||
Overlapping patterns or directory searches within one declaration select the
|
||||
same candidate once. Repeated identical entries in one `path` list likewise
|
||||
select that candidate once; duplicate patterns or exclusions have no additional
|
||||
effect. This local deduplication is not global source deduplication:
|
||||
existing repeated-source and alias checks still apply to distinct inclusion
|
||||
routes. In particular, it does not authorize silently merging separate alias
|
||||
paths to an imported source.
|
||||
|
||||
For declaration-local deduplication, identify a candidate by its normalized path
|
||||
within its source namespace, before resolving symlinks. Compare paths relative
|
||||
to the same namespace rather than relative to each search root. Overlapping
|
||||
searches that encounter `prompts/welcome.yaml` therefore select it once, while
|
||||
distinct symlink paths remain distinct candidates subject to the existing
|
||||
canonical-source repeat checks. This candidate identity is separate from the
|
||||
canonical identity used when accepting sources.
|
||||
|
||||
For `path`, `required: true` requires every listed file to be present. For `dir`,
|
||||
it requires at least one selected file across the complete union after
|
||||
exclusions; it does not require a match per directory, pattern, or pair. A
|
||||
missing directory can therefore contribute nothing while another supplies the
|
||||
required match. The equivalent `glob` requirement is at least one selected file
|
||||
after exclusions. Invalid supplied configuration still fails under the existing
|
||||
rules.
|
||||
|
||||
### Exclusions
|
||||
|
||||
Accept `exclude` as a string or list of strings with the same case-sensitive
|
||||
basename wildcard grammar as `pattern`. Apply it to results from `path`, `dir`,
|
||||
and `glob`. A candidate matching any exclusion is removed before its content is
|
||||
loaded or parsed. An omitted exclusion removes nothing.
|
||||
|
||||
This is **set difference**: for candidate set S and its subset X matching any
|
||||
exclusion, the selected set is `S ∖ X`. Exclusions win over positive matches.
|
||||
They filter filenames, not directory paths, and do not prune traversal into a
|
||||
directory whose name happens to match. For example, `exclude: '*.draft.yaml'`
|
||||
removes matching files at every visited depth, including results of a path glob.
|
||||
|
||||
For searches, evaluate `required` on the selected set after exclusions. For an
|
||||
explicit `path` list, exclusions filter what is loaded but do not waive the
|
||||
settled requirement that every listed file be present when `required: true`.
|
||||
Excluding a candidate is not permission to suppress discovery, containment,
|
||||
or resource-limit errors. Exclusions are local to the declaration; they do not
|
||||
prohibit another declaration from explicitly selecting the same filename.
|
||||
|
||||
### Resource Accounting
|
||||
|
||||
Retain the existing resource limits, with an explicit adjustment to candidate
|
||||
accounting for the new selection operations:
|
||||
|
||||
- Every examined directory entry consumes the discovery visit budget, including
|
||||
nonmatches and excluded entries. A small selected set does not authorize an
|
||||
unbounded scan.
|
||||
- Check distinct selected candidates against the remaining source allowance
|
||||
after declaration-local deduplication and exclusions. Overlapping patterns
|
||||
and search roots do not charge the same selected candidate more than once
|
||||
within a declaration; excluded candidates do not consume source allowance.
|
||||
- Excluded files are not read for content and consume no content-reading
|
||||
allowance. Existing byte limits continue to govern selected sources.
|
||||
|
||||
The current glob implementation checks candidate counts against the remaining
|
||||
source allowance during expansion. Update that accounting to use the filtered,
|
||||
deduplicated selection; do not reject an otherwise valid search solely because
|
||||
its positive matches exceed that allowance before exclusions are applied.
|
||||
Discovery limits still apply throughout the search. This is not global source
|
||||
deduplication and does not suppress repeated-source errors across declarations
|
||||
or distinct aliases.
|
||||
|
||||
### Recursive Discovery And Depth Limits
|
||||
|
||||
Enable recursion explicitly or through an application default for recursive
|
||||
discovery, optionally bounded by `depth`.
|
||||
The motivation is to let authors organize definitions in subdirectories without
|
||||
updating import declarations for every depth. Directory organization should not
|
||||
change resource identity or the effective configuration destination.
|
||||
|
||||
The selected model uses an explicit search root and recursion flag, rather than
|
||||
introducing recursive `**` syntax or inferring a root from a wildcard prefix.
|
||||
|
||||
**OD-1: settled.** The library default for `recursive` is false: search only
|
||||
direct files. An application may supply a different default; an explicit
|
||||
declaration value overrides it, including `recursive: false`. Recursion remains
|
||||
independent of the permissive defaults for absence, symlinks, and hidden entries.
|
||||
|
||||
The settled depth convention is:
|
||||
|
||||
- Files directly inside the search root are at depth `1`.
|
||||
- Each additional directory increases depth by one.
|
||||
- Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit
|
||||
when recursion is enabled; existing loader resource limits still apply.
|
||||
- Supplying `depth` requires effective `recursive: true`. If recursion is false
|
||||
after resolving declaration, application, and library defaults, any supplied
|
||||
`depth` is an error, including `0` or `1`. Depth never enables recursion itself.
|
||||
- With recursion disabled and `depth` omitted, select only direct files.
|
||||
- Depth controls discovery only, not namespaces, identities, or destinations.
|
||||
|
||||
Resolve the effective recursion value before checking depth's applicability.
|
||||
For example, `depth: 2` with no explicit `recursive` is valid if the application
|
||||
default enables recursion, but invalid with the library default. Explicit
|
||||
`recursive: false` overrides an application default of true and makes any
|
||||
supplied depth invalid. Depth must also have the valid type and range defined
|
||||
under directive validation; enabling recursion does not excuse malformed depth.
|
||||
|
||||
Recursive discovery must retain the existing filesystem containment and
|
||||
repeated-source contracts and follow the resource accounting defined above.
|
||||
The discovery visit budget remains separate from the caller-selected depth
|
||||
bound and still applies to unlimited-depth searches.
|
||||
|
||||
### Symlinks And Hidden Entries
|
||||
|
||||
**OD-4: settled.** Discovery controls are local to the declaration. Directory
|
||||
searches default to `follow: true` and `hidden: true`, subject to the application
|
||||
defaults described below. Authors may disable either behavior independently of
|
||||
recursion, depth, matching, or absence handling.
|
||||
|
||||
Define hidden entries by a leading dot in the portable source namespace.
|
||||
`hidden: false` skips discovered hidden files and prunes discovered hidden
|
||||
directories. `follow: false` skips discovered symlink files and prevents descent
|
||||
through discovered symlink directories. Explicitly named roots and separate
|
||||
direct-file imports retain their existing source-access semantics. The controls
|
||||
do not implicitly propagate to declarations inside imported files.
|
||||
|
||||
These are discovery-selection controls, not a load-wide access prohibition.
|
||||
On `glob`, filtering applies to wildcard-selected entries; explicitly named
|
||||
path components remain explicit access.
|
||||
|
||||
When following links, detect a canonical directory repeated on the active
|
||||
traversal path and fail with a cycle diagnostic. Do not silently prune the loop
|
||||
and continue. Existing dependency-cycle and repeated-source errors remain in
|
||||
force; discovered aliases do not authorize silently deduplicating imported
|
||||
sources. Directory-cycle handling retains the existing containment and
|
||||
native-filesystem capabilities.
|
||||
|
||||
These controls narrow selection rather than authorize ignoring errors in a
|
||||
selected source. `required: true` applies after discovery filters: if the search
|
||||
selects no file, it fails. Native filesystems retain their existing capabilities;
|
||||
this feature does not promise link inspection where a supplied filesystem does
|
||||
not expose it.
|
||||
|
||||
## Illustrative Examples
|
||||
|
||||
These examples describe the proposed interface, not configuration accepted by
|
||||
the current release. They assume the consumer has enabled imports and has not
|
||||
changed the library's policy defaults. File paths and directory roots resolve
|
||||
from the declaring file; destinations use the
|
||||
existing relative-placement rules.
|
||||
|
||||
### Import One File
|
||||
|
||||
```yaml
|
||||
configkit:
|
||||
imports:
|
||||
- path: defaults.yaml
|
||||
```
|
||||
|
||||
Import `defaults.yaml` into the declaring source's mapping scope. Under the
|
||||
proposed default, a missing file contributes nothing. Add `required: true` when
|
||||
its presence is necessary. A present but invalid file still fails loading.
|
||||
|
||||
### Require Several Named Files
|
||||
|
||||
```yaml
|
||||
configkit:
|
||||
imports:
|
||||
- path: [defaults.yaml, site.yaml]
|
||||
required: true
|
||||
```
|
||||
|
||||
Both files must be present. Listing them together shares the declaration's
|
||||
options; their textual order does not establish overlay precedence. Their
|
||||
contributions follow the shared dependency-ready schedule and strict
|
||||
composition rules.
|
||||
|
||||
### Recursively Discover Mapping Fragments
|
||||
|
||||
```yaml
|
||||
configkit:
|
||||
imports:
|
||||
- dir: services
|
||||
pattern: '*.yaml'
|
||||
recursive: true
|
||||
destination: [services]
|
||||
```
|
||||
|
||||
This discovers files such as `services/api.yaml` and
|
||||
`services/regional/worker.yaml`, but not `services/README.md`. Each file contains
|
||||
named service entries, for example `api: {port: 8080}` or
|
||||
`worker: {concurrency: 4}`. They compose beneath `services` under the existing
|
||||
strict composition rules. Directory names do not create additional mapping
|
||||
keys. Missing directories or no matching files contribute nothing.
|
||||
|
||||
Adding `depth: 2` limits discovery to direct files and files one directory below
|
||||
`services`: it still selects `services/regional/worker.yaml`, but not
|
||||
`services/regional/east/worker.yaml`. Switching to `recursive: false` requires
|
||||
omitting `depth`; leaving `depth: 2` present is a declaration error.
|
||||
|
||||
Hidden entries and contained symlinks participate by default. To narrow the
|
||||
search, the declaration can also include:
|
||||
|
||||
```yaml
|
||||
follow: false
|
||||
hidden: false
|
||||
```
|
||||
|
||||
These controls narrow discovery below the explicitly named root; the root
|
||||
continues to resolve under the existing source-access rules.
|
||||
|
||||
### Search Several Prompt Collections And Filename Patterns
|
||||
|
||||
```yaml
|
||||
configkit:
|
||||
imports:
|
||||
- dir: [prompts, shared-prompts]
|
||||
pattern: ['*.prompt.yaml', '*.prompt.yml']
|
||||
exclude: ['*.draft.prompt.yaml', '*.draft.prompt.yml']
|
||||
recursive: true
|
||||
required: true
|
||||
destination: [prompts]
|
||||
placement: sequence-item
|
||||
```
|
||||
|
||||
The two directories and two patterns form four Cartesian-product pairs; the
|
||||
selected candidates are the union of their matches, minus excluded filenames.
|
||||
This selects `prompts/welcome.prompt.yaml`,
|
||||
`prompts/weather/daily/summary.prompt.yaml`, and
|
||||
`shared-prompts/greeting.prompt.yml`, but not `prompts/weather/helpers.yaml`
|
||||
or `shared-prompts/greeting.draft.prompt.yml`.
|
||||
Each selected file contains one prompt mapping,
|
||||
such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the
|
||||
`prompts` sequence with identity field `id` through the existing
|
||||
`Definition.SequenceCompositions` API. Distinct IDs compose into one sequence;
|
||||
duplicate IDs fail strict composition. Overlays continue replacing sequences.
|
||||
|
||||
Here `required: true` means at least one file must remain across the entire
|
||||
search after exclusions, not that every directory or pattern must produce a
|
||||
match. One missing root is acceptable if the other supplies a selected file. The
|
||||
filename convention keeps helper files out of this discovery declaration; it
|
||||
does not prevent a selected file from explicitly importing a helper. Existing
|
||||
repeated-source rules still apply if declarations select overlapping sources.
|
||||
|
||||
### Select A Directory Layout With A Glob And Exclusions
|
||||
|
||||
```yaml
|
||||
configkit:
|
||||
imports:
|
||||
- glob: 'collections/*/prompts/*.yaml'
|
||||
exclude: '*.draft.yaml'
|
||||
required: true
|
||||
destination: [prompts]
|
||||
placement: sequence-item
|
||||
```
|
||||
|
||||
This selects files such as `collections/support/prompts/welcome.yaml`, excluding
|
||||
`collections/support/prompts/welcome.draft.yaml`. It does not select deeper
|
||||
files such as `collections/support/prompts/internal/help.yaml`. At least one
|
||||
file must remain after exclusions. The same prompt-item composition setup as
|
||||
the preceding example applies. Exclusion patterns match basenames even though
|
||||
the positive glob matches paths.
|
||||
|
||||
## Optional Imports By Default
|
||||
|
||||
The selected direction is optional-by-default for explicit files, directory
|
||||
searches, and globs, with `required: true` opting into failure on absence. This
|
||||
deliberately supersedes the earlier suggestion to retain required-by-default
|
||||
imports.
|
||||
|
||||
The table assumes the library's default absence policy. Applications may select
|
||||
a different default using the settled precedence rules below.
|
||||
|
||||
| Situation | Proposed behavior |
|
||||
| --- | --- |
|
||||
| Missing `path` entry; `required` omitted or false | That entry contributes nothing; other listed files may still contribute |
|
||||
| Zero selected files across a complete search; `required` omitted or false | Contribute nothing |
|
||||
| Any listed `path` file missing with `required: true` | Fail with an import diagnostic |
|
||||
| Zero selected files across a complete `dir` or `glob` search after exclusions with `required: true` | Fail with an import diagnostic |
|
||||
| One `dir` root missing, but another supplies a selected file with `required: true` | Satisfy the declaration's presence requirement |
|
||||
| Invalid pattern or malformed import declaration | Fail |
|
||||
| Permission failure during discovery, unreadable selected file, or malformed YAML in a selected file | Fail |
|
||||
| Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules |
|
||||
|
||||
An optional declaration is permission for absence, not permission to ignore a
|
||||
selected contribution that cannot be read or interpreted. Optionality must not
|
||||
turn processing failures into apparently successful partial configuration.
|
||||
|
||||
This default supports installations with different optional components without
|
||||
requiring ceremony for every potentially absent directory or file. Its tradeoff
|
||||
is that a misspelled path or pattern can also contribute nothing. Authors and
|
||||
applications that need an import to exist can state that requirement explicitly;
|
||||
applications remain responsible for validating their necessary final values.
|
||||
Final validation does not automatically detect every unintentionally omitted
|
||||
contribution.
|
||||
|
||||
## Who Chooses Strictness?
|
||||
|
||||
**OD-5: settled.** Provide load-level defaults for `required`, `follow`, `hidden`,
|
||||
`recursive`, and `pattern`. Resolve each applicable field independently using
|
||||
this precedence:
|
||||
|
||||
1. Explicit value on the import declaration.
|
||||
2. Application-selected default, when supplied.
|
||||
3. Library default: false for `required` and `recursive`, true for `follow` and
|
||||
`hidden`, and no filename restriction for `pattern`.
|
||||
|
||||
An explicit `pattern` replaces the application default rather than combining
|
||||
with it. In particular, `pattern: []` selects nothing, even when the application
|
||||
supplies patterns. An application can default to `['*.yml', '*.yaml']`; without
|
||||
either an application default or an explicit pattern, a directory search
|
||||
attempts to load all files in its search scope. Application defaults apply only
|
||||
to selectors that support the field; they do not add inapplicable options to
|
||||
other selectors.
|
||||
|
||||
Apply this rule to transitive declarations too. A declaration does not inherit
|
||||
these field values from its importing parent. Applications may expose the
|
||||
defaults to operators through their own interfaces; simple consumers need no
|
||||
additional setup. Exact Go API spelling belongs to implementation planning.
|
||||
|
||||
Explicit declarations remain authoritative, permitting intentional exceptions
|
||||
to stricter application defaults. Defaults are not enforcement controls. Do not
|
||||
add options that override explicit declarations as part of this work without a
|
||||
separate concrete requirement and scope decision. The fields' applicability to
|
||||
each selector is defined under directive validation.
|
||||
|
||||
## Evidence For Optional Declarations With No Selection
|
||||
|
||||
**OD-6: settled.** Record minimal structured inspection evidence when an
|
||||
optional declaration selects nothing: its declaring source/location, selector,
|
||||
and no-selection outcome. Emit no warning by default and do not fabricate a
|
||||
loaded-source record. This keeps optional contributions quiet during ordinary
|
||||
use while allowing applications to explain omissions during troubleshooting.
|
||||
|
||||
No selection does not necessarily mean a source was missing. It may result from
|
||||
missing sources or no positive matches, an explicit empty list, or exclusions
|
||||
removing every candidate. The inspection record must not describe deliberate
|
||||
exclusion or an empty list as missing files.
|
||||
|
||||
This evidence is scoped to the whole declaration. An optional `path` list with
|
||||
one selected file and one missing file does not produce this no-selection
|
||||
record. Per-entry evidence for partial omissions is outside this feature's
|
||||
scope; no additional exclusion audit is required.
|
||||
|
||||
Decide the exact inspection record shape during implementation planning,
|
||||
reusing existing structures where appropriate and retaining existing inspection
|
||||
contracts. Include ownership tests for the new evidence. This addition does
|
||||
not require applications to reconstruct declaration-level no-selection evidence
|
||||
themselves.
|
||||
|
||||
## Release Boundary
|
||||
|
||||
Ship one final interface with a hard break in a pre-1.0 minor release, following
|
||||
the [release policy](../policy/release.md#versioning-and-compatibility). There is
|
||||
no transition release, legacy parser path, compatibility switch, or temporary
|
||||
`optional` alias. Release notes should describe the changed behavior and final
|
||||
syntax; this documentation does not imply migration machinery.
|
||||
|
||||
Retaining `glob` is a settled capability decision, as described under filename
|
||||
matching. It does not preserve the old required-by-default behavior or promise
|
||||
compatibility with every old declaration or use case.
|
||||
|
||||
## Named Algorithms And Implementation Documentation
|
||||
|
||||
Use the common names below in implementation comments and maintained
|
||||
documentation when the feature lands. Keep them beside the relevant logic and
|
||||
in the appropriate owning documents so human readers can recognize the design
|
||||
without reconstructing it from code. This does not call for new public API
|
||||
terminology or duplicate contracts across documents.
|
||||
|
||||
| Operation | Name and intended use |
|
||||
| --- | --- |
|
||||
| Shared import and `extends` scheduling | **Kahn's algorithm with a stable ordering key**, using a min-heap for the ready set; the same dependency-ready topological sort serves both resolvers |
|
||||
| Multiple directory roots and filename patterns | **Cartesian product** of roots and patterns, followed by **set union** of matching candidates within the declaration |
|
||||
| Exclusion filtering | **Set difference** between positive candidates and those matching any exclusion |
|
||||
| Directory symlink cycles | **Active-path cycle detection**, using canonical directories on the current traversal path rather than treating every previously visited directory as a cycle |
|
||||
|
||||
Scalar-to-singleton normalization lets scalar and list forms share selection
|
||||
logic. Set operations describe membership, not resolution order; the shared
|
||||
stable topological schedule supplies that order. Preserve the scheduler's
|
||||
complexity explanation and distinction between cycles and blocked dependents
|
||||
in its implementation documentation. Do not label implementation-specific
|
||||
matching or cycle-path helpers with an algorithm name until that algorithm is
|
||||
actually chosen.
|
||||
|
||||
## Directive Validation
|
||||
|
||||
**OD-3: settled.** Validate declaration shapes and values independently of
|
||||
whether discovery selects any files. Invalid declarations fail even when a
|
||||
search root is missing or a list is empty. This does not require opening or
|
||||
parsing excluded files or establish a new first-error ordering guarantee.
|
||||
|
||||
### Selectors And Applicable Options
|
||||
|
||||
Require exactly one selector per declaration: `path`, `dir`, or `glob`. A list
|
||||
inside `path` or `dir` still counts as one selector. Use separate declarations
|
||||
when combining different selector kinds. Diagnose unknown fields and options
|
||||
that have no meaning for the chosen selector. The removed `optional` spelling
|
||||
is not accepted.
|
||||
|
||||
| Option | `path` | `dir` | `glob` |
|
||||
| --- | --- | --- | --- |
|
||||
| `required`, `exclude` | Yes | Yes | Yes |
|
||||
| `destination`, `placement` | Yes | Yes | Yes |
|
||||
| `pattern` | No | Yes | No |
|
||||
| `recursive`, `depth` | No | Yes | No |
|
||||
| `follow`, `hidden` | No | Yes | Yes |
|
||||
|
||||
### Types And Depth Applicability
|
||||
|
||||
- `path`, `dir`, `pattern`, and `exclude` accept strings or flat lists of strings.
|
||||
- `glob` accepts a string.
|
||||
- `required`, `recursive`, `follow`, and `hidden` accept booleans.
|
||||
- `depth` accepts nonnegative integers under ConfigKit's existing YAML scalar
|
||||
rules. Zero means unlimited when recursion is enabled; negative, string,
|
||||
boolean, and fractional values are invalid. Do not introduce an arbitrary
|
||||
maximum depth.
|
||||
- Explicit null is invalid for these fields, including null list members.
|
||||
Do not coerce non-string members to strings or flatten nested lists.
|
||||
- Existing `destination` and `placement` validation remains unchanged.
|
||||
|
||||
A supplied `depth` additionally requires effective recursion to be enabled,
|
||||
using the default precedence defined above. Valid type and range alone do not
|
||||
make depth applicable. This rule applies even to empty directory or pattern
|
||||
lists. With recursion enabled, malformed depth values still fail validation.
|
||||
|
||||
### Empty Lists, Empty Strings, And Duplicates
|
||||
|
||||
Empty lists are valid and retain their set meanings:
|
||||
|
||||
| Declaration | Meaning | Effect of `required: true` |
|
||||
| --- | --- | --- |
|
||||
| `path: []` | No named files | Presence requirement satisfied: no listed file is missing |
|
||||
| `dir: []` | No search roots | Fail: no selected file |
|
||||
| `dir: fragments` with `pattern: []` | No positive patterns; select nothing | Fail: no selected file |
|
||||
| `exclude: []` | Exclude nothing | Otherwise unchanged |
|
||||
|
||||
An empty pattern list differs from omission: omission uses the application
|
||||
default if supplied, otherwise all filenames. An explicit empty list selects
|
||||
nothing and overrides any application default.
|
||||
|
||||
Empty strings are invalid for `path`, `dir`, and `glob`, including empty entries
|
||||
in path or directory lists. Use `.` to explicitly name the declaring directory
|
||||
as a search root. Empty strings in `pattern` and `exclude` are valid wildcard
|
||||
expressions that match no actual filename. Other source-path and wildcard
|
||||
validation rules still apply.
|
||||
|
||||
Repeated identical path entries within one declaration select that candidate
|
||||
once. Repeated patterns or exclusions have no additional effect, and directory
|
||||
search overlaps follow the settled set-union rule. These rules do not suppress
|
||||
existing repeated-source errors across separate declarations or collapse
|
||||
separate symlink aliases.
|
||||
|
||||
Validate every supplied wildcard expression even if it is inactive because
|
||||
another list is empty or discovery finds no files. Likewise, malformed list
|
||||
members remain errors even when other entries are valid. Do not silently drop
|
||||
invalid members or treat malformed filters as absent.
|
||||
@@ -1,242 +0,0 @@
|
||||
# Import Discovery and Scheduling Review Findings
|
||||
|
||||
Status: complete; all six findings addressed in separate validated commits.
|
||||
|
||||
Reviewed implementation: `aa93aed..15943e2`, ending at
|
||||
`15943e233ca3cd1286c03c0e7085871db344cd4b`, against the accepted
|
||||
[roadmap](imports.md), [implementation plan](implementation.md),
|
||||
[ADR 0015](../adr/0015-shared-dependency-ready-ordering.md), and
|
||||
[ADR 0016](../adr/0016-independent-import-controls.md).
|
||||
|
||||
The shared scheduler, scoped assembly, independent defaults, and detached
|
||||
inspection surface are present. The findings below concern their integration
|
||||
and discovery edge cases, not a proposal to change the accepted architecture.
|
||||
Implement fixes with focused regressions, preserving the selected semantics
|
||||
and existing resource limits. This review made no implementation changes.
|
||||
|
||||
## 1. [x] P1 — Preserve native source identity and relative import bases
|
||||
|
||||
Owners: [directory selection](../../internal/engine/import_selection.go),
|
||||
especially assignment of `target.identity` to a file candidate's `accessPath`
|
||||
around lines 184–194; [glob selection](../../internal/engine/glob.go), around
|
||||
lines 81–90; and source admission in `visitPath`.
|
||||
|
||||
Directory and wildcard-glob selection use resolved symlink targets as native
|
||||
source-admission paths. Direct `path` selection retains the supplied filesystem's
|
||||
namespace path. The same native source therefore has different identity,
|
||||
relative import behavior, and resource context depending on its selector.
|
||||
The plan explicitly separates traversal identity from native admission identity.
|
||||
|
||||
Public reproduction with `NewFilesystem("alias", os.DirFS(root))`:
|
||||
|
||||
- `aliases/item.yml` links to `../data/item.yml`.
|
||||
- That item's YAML imports `helper.yml` and contributes `value: 1`.
|
||||
- `aliases/helper.yml` contributes `owner: alias`; `data/helper.yml` contributes
|
||||
`owner: target`.
|
||||
- A root main document selecting `path: aliases/item.yml` yields
|
||||
`SourceInfo.Path == "aliases/item.yml"` and `owner == "alias"`.
|
||||
- Selecting `dir: aliases` with `pattern: item.yml`, or
|
||||
`glob: 'aliases/ite*.yml'`, instead yields `SourceInfo.Path == "data/item.yml"`
|
||||
and `owner == "target"`.
|
||||
|
||||
The prior implementation at `aa93aed` yields the alias path and alias helper
|
||||
for the same wildcard glob. This is a regression that silently changes final
|
||||
configuration, not merely different reporting. It can also change repeated-source
|
||||
admission and the stable scheduling key.
|
||||
|
||||
Remediation: retain separate candidate spelling, traversal/cycle identity, and
|
||||
native admission path. Do not silently apply host-style canonical source identity
|
||||
to native filesystems. Verify path, directory, and glob equivalence for a file
|
||||
alias and an alias directory, including nested imports, source context, origins,
|
||||
and existing repeated-source behavior. Keep host canonicalization unchanged.
|
||||
|
||||
Completed: native directory and glob discovery retain namespace admission paths,
|
||||
while traversal cycle identities remain resolved. Added public regressions for
|
||||
file and directory aliases across all three selectors, helper/resource origins,
|
||||
source contexts, repeated paths, and distinct alias admission. The regression
|
||||
failed before the fix. Focused discovery/resource tests and `./scripts/check.sh`
|
||||
passed with Go 1.26.9 and offline dependency selection.
|
||||
|
||||
## 2. [x] P1 — Scope main-version protection to the actual effective path
|
||||
|
||||
Owner: [composePendingItems](../../internal/engine/imports.go), around lines
|
||||
694–695, and its call to `importedVersionClaim`.
|
||||
|
||||
Pending item assembly passes the global protected version rule into an item-local
|
||||
mapping. Consequently, an ordinary item field named `version` is treated as if
|
||||
it occupied the protected main-root `/version` path. Imported item bodies have
|
||||
no `importRoleMain` exemption in that local scope, so their own domain fields
|
||||
are rejected as well as helper contributions.
|
||||
|
||||
Public reproduction: the main document declares `version: 1` and imports an
|
||||
item with `destination: [prompts]` and `placement: sequence-item`. The item is
|
||||
`{id: prompt, version: domain-version}`. Load with imports enabled, default
|
||||
source-version mode, and a `VersionRule` at `Keys("version")` whose checker
|
||||
accepts the main header. Current loading fails with `import.version` at
|
||||
`/version`; the identical input succeeds at `aa93aed`.
|
||||
|
||||
The item's effective field is `/prompts/0/version`, not `/version`. This
|
||||
unintentionally strengthens main-version ownership into a field-name restriction
|
||||
inside item scopes, contrary to the requested preservation of version semantics.
|
||||
|
||||
Remediation: use correctly projected/scoped protection, or defer the check to a
|
||||
boundary with the actual effective item coordinates. Preserve protection where
|
||||
the effective paths really overlap. Cover imported/nested items, item helpers,
|
||||
main-item loading, underlays, and enabled source-version modes without adding
|
||||
blanket exemptions or banning ordinary domain version fields.
|
||||
|
||||
Completed: removed the global-rule check against item-local field names; enclosing
|
||||
composition checks retain effective-path protection, including the existing
|
||||
main-item scoped check. New public regressions cover item bodies, helpers, nested
|
||||
items, underlays, and selected main items, and failed before the change. The
|
||||
full everyday check passed, including existing default/enabled source-version
|
||||
and assigned-index ownership regressions.
|
||||
|
||||
## 3. [x] P2 — Accept valid multi-component native filesystem roots
|
||||
|
||||
Owner: [resolveNativeImportPath](../../internal/engine/filesystem_links.go),
|
||||
especially lines 170–181.
|
||||
|
||||
The resolver splits a namespace-relative path from its first component and
|
||||
requires every intermediate prefix to be inside the configured root. For root
|
||||
`nested/root`, the first inspected prefix `nested` necessarily fails containment,
|
||||
even though the complete selected path lies within `nested/root`.
|
||||
|
||||
Public reproduction: use a link-aware `os.DirFS` namespace containing
|
||||
`nested/root/main.yml` and `nested/root/parts/value.yml`. Main imports `dir: parts`
|
||||
with `pattern: '*.yml'`. Supply `FilesystemContext{FS: &filesystem,
|
||||
Root: "nested/root"}`. Loading fails with `import.outside_root` on the directory
|
||||
selector despite all inputs being contained and no symlink being involved.
|
||||
|
||||
Remediation: resolve relative to the selected native root, or distinguish root
|
||||
prefix traversal from an actual escape. Preserve containment on selected and
|
||||
resolved targets. Test directory and glob discovery under multi-component roots,
|
||||
with ordinary entries and supported symlinks. Do not require consumers to flatten
|
||||
their roots or remove optional link-inspection capability.
|
||||
|
||||
Completed: native link resolution allows ancestor prefixes while resolving a
|
||||
contained multi-component path; selected paths and resolved link targets still
|
||||
undergo their existing containment checks. Public directory/glob regressions
|
||||
cover ordinary and aliased roots plus escaping links. All four valid cases
|
||||
failed before the fix. The full everyday check passed.
|
||||
|
||||
## 4. [x] P2 — Distinguish finite symlink reuse from actual cycles
|
||||
|
||||
Owners: [resolveHostImportPath and resolveNativeImportPath](../../internal/engine/filesystem_links.go),
|
||||
especially the `activeLinks` sets around lines 115–134 and 172–194.
|
||||
|
||||
These sets retain every symlink encountered during resolution of a whole path;
|
||||
entries are never removed after an expansion completes. A finite pathname that
|
||||
uses the same link twice is therefore rejected as a cycle. This is distinct from
|
||||
the correctly scoped active-directory set in recursive directory traversal.
|
||||
|
||||
Public host reproduction: create `parts/up -> ..` inside the configuration root,
|
||||
and a regular `parts/value.yml`. Import
|
||||
`dir: parts/up/parts/up/parts` with `pattern: '*.yml'` and default nonrecursive
|
||||
discovery. That finite explicit root resolves to `parts`; loading instead fails
|
||||
with `import.cycle`. No recursive directory walk or infinite link expansion is
|
||||
requested.
|
||||
|
||||
Remediation: identify actual active symlink expansion cycles, not any repeated
|
||||
link occurrence. Preserve ordinary explicit-root access semantics and the
|
||||
separate active-path directory-cycle rule. Add finite-reuse and real-loop
|
||||
regressions for host and link-aware native access; do not replace this defect
|
||||
with an arbitrary new link or path-depth restriction.
|
||||
|
||||
Completed: host and native resolvers now release active links when each target
|
||||
expansion completes, using explicit frames rather than retaining a whole-path
|
||||
visited set. Added resolver regressions for finite reuse, self loops, and mutual
|
||||
loops, plus public host/native directory/glob loading coverage. Finite cases
|
||||
failed before the change; all pass now alongside existing traversal-cycle and
|
||||
containment tests. The full everyday check passed; no new limit was added.
|
||||
|
||||
## 5. [x] P2 — Validate the complete depth token before saturating its value
|
||||
|
||||
Owner: [parseImportDepth](../../internal/engine/import_spec.go), around lines
|
||||
423–430.
|
||||
|
||||
On integer overflow, the parser immediately returns the saturated bound and
|
||||
success. It never validates the remainder of the numeric token, so a sufficiently
|
||||
large fractional or exponent-form number is accepted as an integer depth.
|
||||
|
||||
Public reproduction: load a main document declaring an optional missing
|
||||
directory, `recursive: true`, and either
|
||||
`depth: 999999999999999999999999.5` or
|
||||
`depth: 999999999999999999999999e1`. Both loads succeed. Both declarations should
|
||||
fail `import.invalid`, independently of whether discovery selects files.
|
||||
|
||||
Remediation: validate all characters while saturating the numeric accumulator,
|
||||
or validate integer grammar before conversion. Continue accepting arbitrarily
|
||||
large valid nonnegative integers and numeric `-0`; do not introduce a machine-size
|
||||
maximum. Cover invalid suffixes after the overflow point as well as ordinary
|
||||
invalid depths and valid large integers.
|
||||
|
||||
Completed: depth parsing saturates its accumulator without stopping token
|
||||
validation. Public absent-directory regressions now reject fractional/exponent
|
||||
suffixes after overflow and retain valid huge integers and `-0`. The two invalid
|
||||
cases failed before the fix. Existing depth/specification coverage and the full
|
||||
everyday check passed; the accepted integer range was not narrowed.
|
||||
|
||||
## 6. [x] P2 — Apply the active import-depth limit to actual inclusion edges
|
||||
|
||||
Owner: [walkDeclarations](../../internal/engine/imports.go), around lines 403–405.
|
||||
|
||||
The active-edge limit is checked for every declaration before candidate selection.
|
||||
An empty list or absent optional search at the depth boundary is therefore
|
||||
treated as an additional active import edge, even though it loads no source.
|
||||
This strengthens the established active-edge budget into a declaration-depth
|
||||
restriction and conflicts with the selected empty/absent contribution semantics.
|
||||
|
||||
Public reproduction: a main file followed by 64 imported files loads successfully
|
||||
when the final file contributes `end: true` and no imports. Adding either
|
||||
`path: []` or an optional `glob: 'missing/*.yml'` to that last file produces
|
||||
`import.limit_exceeded`. The same 64 actual inclusion edges are present in all
|
||||
three cases.
|
||||
|
||||
Remediation: enforce the existing depth bound when a selected source would add
|
||||
an inclusion edge. Continue validating declaration syntax, policy applicability,
|
||||
and selection resource budgets even when no source is selected. At the boundary,
|
||||
empty lists and optional zero selections must succeed, while a real next source
|
||||
must still fail. Keep the existing numeric resource limit unchanged.
|
||||
|
||||
Completed: the depth bound is checked for an existing selected target at source
|
||||
inclusion, rather than for every declaration. A test-controlled one-edge budget
|
||||
covers empty/required-empty lists, absent path/directory/glob selections, actual
|
||||
next sources, and invalid declarations. The original 64-edge public examples
|
||||
now succeed with empty optional contributions; a real next edge still fails.
|
||||
The full everyday and race checks passed without changing the resource limit.
|
||||
|
||||
## Review validation
|
||||
|
||||
Passed on the reviewed revision with Go `go1.26.9 linux/amd64`:
|
||||
|
||||
- `./scripts/check.sh` with the installed Go binary directory first on `PATH`,
|
||||
`GOTOOLCHAIN=local`, `GOPROXY=off`, and a build cache outside the repository.
|
||||
- `go test -race ./...` with the same offline toolchain selection.
|
||||
- `git diff --check aa93aed..15943e2`.
|
||||
|
||||
All six findings were independently reproduced through public APIs using a
|
||||
temporary Go program, in-memory filesystems, and disposable real filesystem
|
||||
fixtures outside the repository. The native wildcard-alias and ordinary item-
|
||||
version regressions were also compared with an archived `aa93aed` checkout
|
||||
outside the repository. No code, dependencies, release procedure, consumer
|
||||
repository, or accepted ADR decision text was changed during the review.
|
||||
|
||||
## Remediation validation
|
||||
|
||||
All six findings have completion evidence above. Final validation passed with
|
||||
Go `go1.26.9 linux/amd64`, local toolchain selection, offline dependencies, and
|
||||
build caches outside the repository:
|
||||
|
||||
- `./scripts/check.sh` (formatting, whitespace, full tests, vet, and build).
|
||||
- `go test -race ./...`.
|
||||
- Local documentation-link checks and `git diff --check`.
|
||||
|
||||
The original external public-API reproductions were rerun against the completed
|
||||
fixes. Native path/directory/glob aliases now retain the same helper origin and
|
||||
resource context; ordinary item versions load; nested native roots and finite
|
||||
symlink reuse load; invalid depth suffixes fail; and empty contributions do not
|
||||
consume active inclusion depth. Focused regressions also retain actual version
|
||||
ownership, real cycle/escape errors, invalid-declaration handling, and the depth
|
||||
budget for real sources. No dependencies, public controls, release gates, or
|
||||
downstream repositories were changed.
|
||||
Reference in New Issue
Block a user