diff --git a/docs/development.md b/docs/development.md index 68cd70c..2770d18 100644 --- a/docs/development.md +++ b/docs/development.md @@ -150,6 +150,16 @@ git show 008d2ba:docs/roadmap/implementation.md git show 008d2ba:docs/roadmap/remediation.md ``` +The completed import-discovery roadmap, implementation plan, and resolved +remediation record are preserved in Git history at `5747ec4` under +`docs/roadmap/`. For example: + +```sh +git show 5747ec4:docs/roadmap/imports.md +git show 5747ec4:docs/roadmap/implementation.md +git show 5747ec4:docs/roadmap/remediation.md +``` + Potential enhancements are tracked in [future features](roadmap/future.md). Install the Go version required by `go.mod` or a compatible newer toolchain. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 926228e..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,708 +0,0 @@ -# Implementation Plan: Import Discovery and Dependency-Ready Ordering - -Status: All 19 implementation stages complete. - -## Authority, baseline, and execution - -Implement the accepted [import roadmap](imports.md), -[ADR 0015](../adr/0015-shared-dependency-ready-ordering.md), and -[ADR 0016](../adr/0016-independent-import-controls.md), starting at -`94381a67fd39635f448a5225e38adb2a7f43cc9f`. The roadmap owns target behavior and -scope; this plan settles delivery order and the remaining implementation choices. -The accepted ADRs already exist; do not draft replacements or rewrite their -decision text. - -The implementer is a Luna coding agent. Execute one numbered stage per focused -prompt, in order. Each stage should end with a validated, nonempty commit and a -completion entry here identifying the commit, checks, and any remaining work. -Do not mark a stage complete while its acceptance conditions remain unmet. -Preserve unrelated work. This planning pass changes documentation only. - -Start with [development](../development.md) and read the applicable -[agent](../policy/agents.md), [architecture](../policy/architecture.md), -[testing](../policy/testing.md), and [documentation](../policy/documentation.md) -policies. Prefer the code graph for discovery, then inspect current source and -tests; indexed snippets can have stale line ranges. Use the actual working tree -when an indexed snippet is incomplete or mismatched. - -Keep each stage's public documentation accurate to its implemented boundary. -Private groundwork may precede wiring, but do not expose options that silently -do nothing. The final syntax cutover is Stage 14; there is no consumer-facing -compatibility mode, alias, or alternate scheduler. Intermediate private helpers -are scaffolding, not a second supported interface. Release publication and -consumer dependency updates are outside this plan. - -## Settled implementation choices - -### Public defaults - -Add a root-owned `ImportDefaults` struct and an `ImportDefaults ImportDefaults` -field to both `Options[T]` and `LayeredOptions[T]`: - -```go -type ImportDefaults struct { - Required *bool - Follow *bool - Hidden *bool - Recursive *bool - Pattern []string -} -``` - -Nil boolean pointers mean unspecified. Copy pointed-to values when loading -begins. Nil `Pattern` means unspecified; non-nil empty means select nothing for -directory searches. Snapshot the slice without collapsing this distinction. -No new defaults field belongs on `Input`, `Underlay`, or `Definition`: one -load-level selection applies to main and all underlay/transitive import -declarations. Existing per-input `Imports` booleans still enable imports. - -Resolve each applicable declaration field independently: explicit declaration, -then application default, then library default. Library defaults are required -false, recursive false, follow true, hidden true, and unrestricted filenames. -No parent-declaration inheritance, enforced application override, default -exclusion list, or application depth setting is added. `pattern` replaces rather -than augments the default. Validate supplied default patterns once at option -selection, reporting `options.invalid` before source reads or callbacks. -Defaults do not make unsupported fields applicable to a selector. - -Use a private presence-aware declaration representation, distinct from its -resolved discovery policy. Normalize scalar/list selectors and filters once. -Keep existing import path validity rules, with `.` accepted for literal directory -roots. Match basenames with case-sensitive `path.Match`; path globs retain their -existing grammar, including rejection of recursive `**`. A filename filter has -no directory-recursion interpretation. Validate all expressions even in empty -or otherwise inactive selections. Follow the roadmap's option applicability, -empty-list, null, and depth rules exactly. - -`depth` has no arbitrary machine-integer maximum. Accept nonnegative YAML -integers under the existing scalar rules, including bounds larger than can be -reached under the visit budget; an implementation may saturate its internal -bound at the largest representable traversal depth without rejecting that input. -Numeric `-0` denotes zero. Do not accept fractional/exponent numbers as integer -depth or coerce strings. - -### No-selection inspection - -Extend root-owned `SourceInfo` with `ImportSelections []ImportSelection`. Add -root-owned string types and constants: - -- `ImportSelector`: `ImportSelectorPath = "path"`, `ImportSelectorDirectory = - "dir"`, and `ImportSelectorGlob = "glob"`. -- `ImportSelectionOutcome`: `ImportSelectionNone = "none"`. -- `ImportSelection` fields: `Path Path`, `Location Location`, - `Selector ImportSelector`, and `Outcome ImportSelectionOutcome`. - -`Path` is the source-local declaration path, such as -`/configkit/imports/0`; `Location` identifies the declaring mapping and carries -its actual SourceID. Attach one record to that declaring source for each -effectively optional declaration whose entire selected set is empty. Keep -records in declaration order. Do not record partial omissions, candidate lists, -raw selector values, inferred missing-file reasons, warnings, or fake sources. -Required empty `path: []` succeeds but gets no optional-declaration record. - -Reuse `Result.Sources`, `LoadError.Sources`, and callback source snapshots rather -than adding another query surface. Snapshots contain only outcomes established -by that point; no complete discovery record is promised after an earlier error. -Records remain source-local through mounting and item placement. Deep-copy paths -and slices through engine cloning and facade conversion. Existing version-text -redaction remains independent and unchanged. - -### Scheduling and identities - -Keep the shared scheduler private to `internal/engine`, in a focused -`dependency_order.go` owner; no new public scheduling API or dependency is needed. -It accepts immutable node keys and dependency edges and returns a complete order -or an actual closed cycle. Use Kahn's algorithm and `container/heap`; deduplicate -edges before counting dependencies. Compute keys once. Compare key tuples by Go -bytewise string order component by component, shorter tuple first on a prefix. - -Use these key constructions in both integrations: - -- Import nodes: `(namespace-kind, namespace-anchor, resource-kind, resource-path)`. - Kind is `host` or `native`; resource kind is `file` or `synthetic`. Host paths - are canonical absolute paths rendered with slash separators; native paths use - their existing namespace-relative identity. Synthetic roots use their explicit - input anchor, not a label or allocated token address. -- Derive namespace anchors from explicit request positions, never discovery. - Positions are main `(0,0)` and underlay/input `(group+1,input+1)`. Encode each - numeric component in fixed-width decimal sufficient for a Go index. All host - accesses within one scheduled tier share its host anchor; each native token - uses the smallest explicit input position selecting that namespace in that - tier. Same-label, separately constructed filesystems remain distinct. Use - tokens to identify equality, not to order nodes. -- Inheritance nodes: `("definition", scope-pointer, definition-pointer)` using - canonical escaped semantic pointers. Within a sibling scope these distinguish - exact names, including empty or escaped keys. No source ID participates. - -The tuple comparator is shared; resolver-specific key construction is not a -second ordering algorithm. Unique keys are an internal construction requirement, -not a new user configuration restriction. Imports already reject repeated -canonical inclusion; separate synthetic roots have distinct explicit anchors. - -For a stalled graph, deterministically traverse residual dependencies using -explicit DFS state, visiting nodes and neighbors in key order. Return an actual -stack cycle, rotated to begin at its smallest key, with the start repeated at -the end. Exclude blocked non-cycle dependents. Both resolvers translate these -node/edge identities to existing domain diagnostic codes and authored sites; -do not expose inheritance names or payload values in messages. Preserve import -cycle-site evidence. Build deterministic adjacency from precomputed node ranks -in linear edge work, rather than repeatedly sorting the ready set or scanning -all nodes for each completion. Scheduling costs O(E + V log V), O(V + E) memory, -apart from key construction/comparison and domain operations. - -### Discovery, scopes, and assembly - -Keep acquisition deterministic and separate from application. Enumerate -declarations in authored order; normalize/deduplicate a declaration's selected -candidate paths and visit them in namespace-path lexical order. Keep main-first -acquisition records, IDs, source-local version checking, and physical import -resolution. This discovery order does not determine sequence assembly order. -There is no new first-error or no-read-before-error guarantee. - -Represent dependencies while acquiring files. Ordinary source-body nodes depend -on their imported contributions; importing bodies therefore assemble after -their dependencies. Record source roles explicitly: the main body must not be -identified as contribution index zero after scheduling. Keep all source IDs and -authored locations attached to contributions when reordering them. - -Each main tier or underlay group has one outer strict-scope graph, including all -its independent input roots and ordinary imports. An item contribution is one -outer node keyed by its owning source. It owns a nested strict mapping-scope -graph containing that source body, ordinary helper contributions, and further -item nodes. Prevalidate the available scope schedules before applying their -operations. When the outer schedule selects an item node, execute its validated -local schedule, compose the item mapping, then place the completed item into -the outer scope. Do not globally flatten helper bodies into the collection. -Resolve local paths and project final indexes using that actual scheduled -contribution order. Main-item and underlay-item roots use the same model. - -Retain the acquisition/inclusion graph independently of these composition scope -boundaries so cycles crossing item boundaries remain detectable. On an active -canonical repeat, retain the closing edge without rereading the source and use -the shared cycle logic on the available inclusion graph. Non-active repeated -inclusion still fails as `import.repeated`; no successful diamond import or -global source deduplication is introduced. - -Within inheritance, construct the graph of direct mapping siblings and their -string parent edges, validate it, then produce templates in scheduled order. -Null cancels a relationship without an edge. Keep enclosing-before-nested -destination-local traversal and unresolved nested templates, as in ADR 0009. -Do not reorder output mappings merely to mirror scheduling. - -Directory discovery keeps candidate spellings separate from canonical access. -One normalized candidate path per namespace is selected once per declaration; -aliases retain separate candidates and existing admission rules. Directory -filters run on discovered names. Exclusions never read content, but do not hide -discovery, containment, or resource failures. For required named paths, verify -every listed target exists even if excluded, without parsing excluded content. - -Use existing bounded directory reads. For distinct roots, traverse each search -with its applicable depth once and test the union of filename patterns per -file; do not walk once per root/pattern pair. Overlapping roots may revisit -entries under the existing visit accounting, but selected candidates deduplicate. -Do not use global canonical-directory visitation to silently suppress aliases. -Charge selected-source allowance after exclusions and local deduplication; -discovery entries still count throughout, and excluded content costs no bytes. - -Use host `Lstat`/root-relative operations for discovery controls and the standard -native `fs.ReadLinkFS` capability where available. Follow the supplied filesystem's -capabilities; do not require every `fs.FS` to implement link inspection or turn -opaque native behavior into a universal sandbox guarantee. Host directory cycle -keys are canonical paths; link-aware native traversal uses the corresponding -resolved namespace directory identity. This traversal identity must not silently -change the existing native source-admission identity. Track canonical directories -on the active traversal path, not all previously visited directories. Distinguish -directory traversal cycles from import dependency cycles; use `import.cycle` -with the declaring site and a safe directory-cycle message for the former. - -## Stage 1 — Build the shared scheduler - -Create the private scheduler, tuple comparator, min-heap, and deterministic -cycle finder specified above. Keep it independent of documents, filesystem -handles, diagnostics, and callbacks. Document Kahn's algorithm, actual-cycle -identification, and complexity beside its owner. - -Tests own observable schedules: independent branches, unequal chains, multiple -dependencies, shared dependencies at equal/unequal depths, a newly ready small -key preceding an older large key, duplicate edges, self/multi-node cycles, and -blocked non-cycle dependents. Permute insertion order while retaining keys and -assert equal schedules/cycles. Do not assert heap internals. - -Acceptance: focused scheduler tests pass; no loading behavior changes. - -Completion: implemented in commit `167fcd4` (`Add deterministic dependency -scheduler`). Verified with `go test ./internal/engine -run -'^TestScheduleDependencies' -count=1`, `./scripts/check.sh`, and -`git diff --check`. Loading behavior remains unchanged; Stage 2 onward remains. - -## Stage 2 — Separate inheritance graph analysis from application - -In `inheritance.go`, extract direct-sibling relationship analysis from the -existing stack-driven application loop. Build keyed mapping nodes and edges; -retain null controls, missing/non-mapping parent diagnostics, exact directive -sites, and non-mapping siblings. Add a scheduled template-application helper -reusing current merge hooks, origins, retained-node accounting, and unresolved -nested template behavior. Keep the currently wired resolver unchanged until -Stage 3; do not expose an alternate public path. - -Acceptance: private scope tests demonstrate the shared schedule, ordinary/null -relationships, real cycle sites, and preserved template values/ownership. - -Completion: implemented in commit `2718fbb` (`Separate inheritance graph -analysis and application`). Verified with -`go test ./internal/engine -run 'Inheritance|Inherited' -count=1` and -`./scripts/check.sh`. The wired resolver retains parent-first ordering; Stage 3 -will integrate the shared dependency-ready schedule. - -## Stage 3 — Integrate scheduled inheritance - -Replace the old sibling-frame resolution in the loading path with Stage 2's -analysis and application. Remove superseded stack/state machinery. Retain -`inheritance_walk.go`'s enclosing-before-nested scope discovery, sequence -traversal, final projection budgets, and immutable retained evidence. - -Review inheritance history/control/transfer ordering against the selected -schedule, not the old DFS order. Add one unequal-branch integration case and -reuse existing nested destination, cancellation, sensitivity, version ownership, -and budget tests. Update the owning inheritance contract and internal overview -to describe implemented scheduling, linking ADR 0015. - -Acceptance: inheritance and facade inheritance suites pass; final values, -presence, origins, and explanation agree; no alternate resolver remains. - -Completion: implemented in commit `6e40c16` (`Schedule sibling inheritance by -dependency readiness`). Verified with engine inheritance and facade inheritance -tests, `./scripts/check.sh`, and `git diff --check`. Nested scope discovery, -projection budgets, and retained evidence remain in their existing owners. - -## Stage 4 — Retain the import inclusion graph during acquisition - -Refactor `imports.go` and `load_input.go` so acquired contributions carry stable -source identity, role, inclusion edges, and explicit scope ownership, rather -than treating their slice position as both acquisition and application order. -Separate item discovery from immediate item composition. Preserve parsed -payloads and source-version processing results without rereading sources. - -Keep existing wired application order for this extraction stage. Introduce the -stable import keys and include-cycle integration with the shared cycle finder; -keep non-active repeats invalid, including aliases and different inclusion -routes. Do not expand successful shared dependency support for imports. - -Acceptance: current ordinary/item import behavior tests pass, including a cycle -through a nested item scope, repeated imports, source IDs, and handle closure. - -Completion: implemented in commit `fbe9ae6` (`Retain import inclusion graph -during acquisition`). Verified with `go test ./internal/engine -count=1`, -`./scripts/check.sh`, and `git diff --check`. Import acquisition retains stable -source keys, roles, inclusion edges, and explicit scope ownership; item -scopes are composed after discovery in the existing contribution order. -Non-active repeats remain invalid. Stage 5 onward remains. - -## Stage 5 — Schedule ordinary main-tier contributions - -Use the shared scheduler for ordinary mapping contributions in the main tier. -Plan before composing. The main body follows its import dependencies; independent -ready branches follow stable keys. Reuse `composeDocumentsWithPolicy` and the -existing merge primitives. Update the corresponding order tests deliberately. - -Replace assumptions that main is document/contribution zero in -`importedVersionClaim`, source-root ownership, contexts, and history. Carry an -explicit main source ID/role and locate its scheduled contribution. Keep source -enumeration and version callbacks in acquisition order. Default-mode effective -version checking still happens after main-tier assembly. - -Acceptance: ordinary import values, registered sequence order, conflict sites, -main-version protection, source records, and authored resource contexts agree. -Document this implemented ordering boundary in the import contract. - -Completion: implemented in commit `5ed0f55` (`Schedule ordinary import -contributions by dependency`). Verified with `go test ./internal/engine --count=1`, `go test . -run -'^TestFacadeSourceVersionEvidenceFollowsAssembledSequenceRanges$' -count=1`, -`./scripts/check.sh`, and `git diff --check`. Acquisition records and version -callbacks remain in traversal order; ordinary contributions use the shared -dependency-ready schedule, with main-source ownership and projected evidence -following the scheduled contributions. Item-scope ordering remains for later -work. - -## Stage 6 — Assemble item-local scopes under the scheduler - -Integrate item nodes and local schedules for imported and selected main items. -Complete helpers before local item assembly and outer placement. Reuse existing -`projectNestedEvidence`, destination projection, sequence ranges, and diagnostic -projection, correcting any assumptions about contribution order or index zero. -Do not register application collection policies inside helper mapping scopes. - -Keep local version checking/consumption before assembly and project retained -source evidence through the actual assigned indexes. Failed-load source paths -retain the existing unresolved-container semantics until indexes are known; -do not invent an exact index to simplify a failure. - -Acceptance: nested items, helper-supplied IDs, main items, multiple unequal -branches, physical coordinates, sensitivity, history chronology, and item -conflict paths pass engine and public tests. Run the focused race suite. - -Completion: implemented in commit `af43d6f` (`Schedule item-local import -scopes`). Verified with `go test ./internal/engine -count=1`, -`go test -race ./internal/engine`, `go test . -count=1`, `go vet ./...`, -`go build ./...`, the local documentation-link test, and `git diff --check`. -`./scripts/check.sh` reaches the consumer examples and currently fails only on -prompt-catalog ordering expectations assigned to Stage 18. Stage 7 onward -remains. - -## Stage 7 — Schedule underlay forests and preserve tier boundaries - -In `load_request.go`, acquire all roots in a strict underlay group before -scheduled group assembly. Combine their retained scope graphs rather than -concatenating already composed per-input orders. Assign namespace anchors from -explicit input positions. Include inputs without imports and item roots. - -Keep per-tier repeated-source state, accounting, main acquisition timing, and -underlay precedence unchanged. Independent groups never enter one scheduler; -overlays/CLI remain later precedence operations. Maintain source layers, -underlay indexes, global source IDs, and group-wide item path projections. - -Acceptance: a group with interleaving dependency branches across input roots -uses the expected schedule; separate tiers still replace sequences wholesale. -Test same-label distinct native namespaces and copies of one namespace. NoSource -still contributes no source or explicit empty root. - -Completion: implemented in commit `014b995` (`Schedule underlay groups as -dependency forests`). Verified with `go test ./internal/engine -count=1`, -`go test . -count=1`, `go test -race ./internal/engine`, `go vet ./...`, -`go build ./...`, the local documentation-link test, and `git diff --check`. -`./scripts/check.sh` still reaches only the two prompt-catalog ordering -expectations assigned to Stage 18. Stage 8 onward remains. - -## Stage 8 — Define private defaults and the final declaration parser - -Add the private defaults snapshot/validation and resolved-policy helpers. Add -the final declaration parser as an unwired private helper until Stage 14; retain -the current wired parser only until that cutover. Store declaration paths and -field locations for exact diagnostics and later no-selection evidence. - -Cover the roadmap's selector/option matrix: scalar/list normalization, exactly -one selector, unknown fields including removed `optional`, null/wrong-type -members, empty lists, duplicates, wildcard validation, literal destinations, -placement, and effective-recursion/depth applicability. Parse all declarations -before selecting candidates; invalid inactive fields still fail. - -Acceptance: parser/policy tests cover omitted, explicit false, explicit empty, -application defaults, and invalid default patterns without source access. No -new YAML syntax is accepted by public loading yet. - -Completion: implemented in commit `b498e5b` (`Define private import defaults -and declaration parsing`). Verified with focused parser/policy tests, the full -engine suite, `go vet ./...`, `go build ./...`, a focused engine race run, and -`git diff --check`. The everyday check still reaches the two known consumer -sequence-order expectation failures scheduled for Stage 18; public loading -continues to use the existing parser. Stage 9 onward remains. - -## Stage 9 — Implement named-file selection and exclusions - -Build a private selector result containing normalized lexical candidates and -selection outcome. Implement string/list `path`, local candidate deduplication, -basename exclusions, and optional/required presence semantics. Check excluded -required target existence without opening content. Do not treat permission, -containment, regular-file admission, or parser failures as optional absence. - -Keep selector results distinct from source-admission identities; no source IDs -or read counts are assigned merely by selection. Preserve current validity rules -for explicit paths and existing source-access behavior for named symlinks and -hidden targets. Do not apply search-only controls to this selector. - -Acceptance: temporary-file tests cover partial missing lists, required/excluded -missing targets, explicit empty lists, alias candidates, and unread excluded -content. No fake source records are created for absence. - -Completion: implemented in commit `5f33e1e` (`Select named imports before source -admission`). Verified with named-selection tests using temporary host files and -a stat-only native filesystem, the full engine suite, a focused race run, -`go vet ./...`, `go build ./...`, and `git diff --check`. The everyday check -continues to fail only on the two consumer sequence-order expectations assigned -to Stage 18. Stage 10 onward remains. - -## Stage 10 — Implement shallow directory selection - -Add literal directory string/list search, scalar/list basename patterns, and -exclusions through the same candidate-selection result. Start with nonrecursive -selection. Missing roots contribute nothing; an inaccessible or invalid supplied -root is not absence. Directories are traversal containers, not documents. - -Implement set union and difference, normalized candidate-path deduplication, -and requiredness across the complete selected union. Preserve nil/unrestricted -versus explicit-empty patterns and alias spelling during filtering. Test multiple -roots, overlapping roots/patterns, extensionless and hidden filenames, and a -missing root alongside one that supplies the required match. - -Acceptance: shallow directory tests pass with actual host files and a native -filesystem; no per-root/per-pattern requiredness is introduced. - -Completion: implemented in commit `55a59de` (`Select shallow directory import -sources`). Verified with shallow-selection tests on host and native filesystems, -the full engine suite, a focused race run, `go vet ./...`, `go build ./...`, and -`git diff --check`. The everyday check still reaches only the consumer -sequence-order expectations reserved for Stage 18. Stage 11 onward remains. - -## Stage 11 — Add recursive directory traversal and depth accounting - -Extend the walker with explicit traversal frames, effective recursion, and the -settled depth convention. Apply filename patterns only to files, never to prune -directories. Share bounded directory-reading and entry accounting with glob -discovery. Do not rescan a root per pattern or use the import-edge depth limit -as a directory-recursion limit. - -Acceptance: direct files are depth 1, depth 2 includes one descendant level, -zero/omission are unlimited subject to existing budgets, and depth requires -effective recursion. Cover application-enabled recursion, explicit false, -large depth integers, empty searches, and limits using test-controlled budgets. - -Completion: implemented in commit `77d9729` (`Add recursive directory search -with depth limits`). Verified with recursive depth/limit tests, existing glob -tests, the full engine suite and race suite, `go vet ./...`, `go build ./...`, -and `git diff --check`. The everyday check still reaches only the consumer -sequence-order expectations reserved for Stage 18. Stage 12 onward remains. - -## Stage 12 — Add discovery symlink and hidden controls - -Implement `follow`/`hidden` for directory discovery using capability-aware -filesystem helpers. Explicit roots keep existing access semantics; discovered -hidden directories prune when hidden is false, discovered symlink directories -do not descend when follow is false, and analogous files are not selected. - -When following links, track active-path directory identities and fail actual -loops without treating earlier completed branches as cycles. Preserve namespace -containment and candidate spellings. Cover contained links, escapes, filename -filtering on aliases, self/ancestor loops, and separate noncyclic aliases that -later retain ordinary repeated-source admission behavior. - -Acceptance: host filesystem behavior and a focused link-aware native fixture -pass; plain native filesystems still load without new capability requirements. -All success/failure paths close library-owned directory and root handles. - -Completion: implemented in commit `9b99463` (`Follow links safely during -directory discovery`). Verified with directory-selection tests, the full engine -suite and race suite, native resource-close tests, `go vet ./...`, `go build -./...`, and `git diff --check`. Host traversal uses root-relative link inspection; -native traversal uses `fs.ReadLinkFS` when available and leaves plain native -filesystems usable. `./scripts/check.sh` still fails only on prompt-catalog -sequence-order expectations reserved for Stage 18. Stage 13 onward remains. - -## Stage 13 — Integrate glob filtering and final candidate accounting - -Adapt `glob.go` to the common selection pipeline. Keep path wildcards and -literal components distinct so follow/hidden filters affect only wildcard- -discovered entries, not explicitly named components. Apply basename exclusions -to final candidates. Preserve regular-file/source validity behavior rather than -silently accepting invalid glob-selected sources. - -Move source-allowance checking after exclusions and declaration-local -deduplication. Intermediate traversal candidates are bounded by entry visits, -not by the final source allowance. Charge bytes only when selected content is -read; never read excluded content. A tiny private source limit with many excluded -positive matches must succeed when its final selected set fits. - -Acceptance: existing glob syntax, containment, alias, and visit-budget tests -pass alongside discovery-control and post-filter source-limit regressions. - -Completion: implemented in commit `e4abff4` (`Filter and account final glob -candidates`). Verified with focused glob tests, the full engine suite and race -suite, `go vet ./...`, `go build ./...`, and `git diff --check`. Glob expansion -now uses bounded shared directory reads, preserves wildcard candidate spellings, -applies discovery controls only to wildcard components, and filters exclusions -before source admission. `./scripts/check.sh` still fails only on prompt-catalog -sequence-order expectations reserved for Stage 18. Stage 14 onward remains. - -## Stage 14 — Wire final syntax and load-wide defaults through the engine - -Add the private defaults value to ordinary/grouped requests, snapshot/validate -it before acquisition, and pass it to every enabled main/underlay/import -declaration. Wire Stage 8's parser and Stages 9–13's selection pipeline into -graph acquisition. Remove the old declaration parser and `optional` field; -there is one final parser and one absence policy. - -Include `source_version.go`'s stored import declarations in this wiring: syntax -parsing stays independent of source-version interpretation, and every stored -declaration receives the same effective policy before discovery. Do not bypass -validation when a consumed header or empty payload leaves no application fields. - -Keep imports disabled unless already enabled by the consumer. Selected overlays -still cannot enable their own imports; defaults do not change that boundary. -Validate effective placement/depth and declaration values before relying on a -missing target or empty selection to skip work. Declarations do not inherit -parent control values. - -Migrate existing tests and maintained fixture declarations from `optional` to -the intended `required` behavior. Tests expecting a missing import to fail must -request requiredness, not weaken their checks. Update the owning configuration -contract to describe the now-implemented final YAML interface and breaking -absence/order changes. Keep exact public Go defaults documentation for Stage 16. - -Acceptance: all engine imports, glob, destinations, source-version, and layered -tests pass under final syntax. An explicit regression rejects `optional`; -another proves missing imports are optional without setup. Run everyday checks. - -Completion: implemented in commit `1e5bffc` (`Wire final import declarations -through loading`). Verified with `go test ./internal/engine -count=1`, -`go test -race ./internal/engine`, `go vet ./...`, `go build ./...`, the local -documentation-link test, and `git diff --check`. `./scripts/check.sh` reaches -only the two prompt-catalog sequence-order expectations reserved for Stage 18. -Stage 15 onward remains. - -## Stage 15 — Retain no-selection evidence across load outcomes - -Add the private records from the settled inspection design to declaring source -records and their clone functions. Record only completed, optional, whole- -declaration empty outcomes. Preserve evidence for main and imported declarations -and underlay inputs through composition, nested item placement, overlays, -resolution entry state, final publication, and failed-load accumulation. - -Use source IDs to attach outcomes, not scheduled positions. Reuse source-record -transport; do not add synthetic imported sources or warnings. Keep evidence -source-local and detached even when its source destination remains unresolved. - -Acceptance: tests cover no matches, missing paths, empty lists, exclusions, -partial path-list omission (no record), and required empty paths (no record). -An error after a prior empty optional declaration retains that established -evidence, but does not fabricate later outcomes. Mutating query results cannot -alter subsequent snapshots or another load. - -Completion: implemented in commit `e433988` (`Retain empty optional import -selection evidence`). Verified with focused evidence tests, -`go test ./internal/engine -count=1`, `go test -race ./internal/engine`, -`go test . -count=1`, `go vet ./...`, `go build ./...`, and -`git diff --check`. Source-local evidence survives composition, nested item -placement, underlays, overlays, resolution entry, final publication, and failed -loads; the known prompt-catalog ordering expectation updates remain assigned to -Stage 18. Stages 16–19 remain. - -## Stage 16 — Expose defaults and inspection through the public facade - -Add root-owned declarations from this plan, `ImportDefaults` fields on ordinary -and layered options, and `SourceInfo.ImportSelections`. Add explicit converters -and engine transport types/helpers as needed, following existing root-wrapper -ownership. Snapshot pointers/slices and preserve nil versus non-nil empty -patterns. Do not add defaults on individual inputs or a new result query API. - -Exercise final behavior through `Load` and `LoadLayered`, including defaults -applied to transitive and underlay imports, explicit declaration exceptions, -invalid default preflight without reader consumption, and result/error/resolver -source inspection ownership. Update declaration comments and the owning Go API -guidance when the surface becomes available. - -Acceptance: public facade/import-boundary tests and representative engine/public -equivalence cases pass; no internal type leaks through exported signatures. - -Completion: implemented in commit `2413161` (`Expose import defaults and -selection evidence`). Verified with full root and engine tests, public facade, -dependency boundary, source-equivalence, and documentation-link tests, -`go vet ./...`, `go build ./...`, formatting, and `git diff --check`. The -repository-wide `./scripts/check.sh` remains blocked only by the two -prompt-catalog sequence-order expectations assigned to Stage 18. Stages 17–19 -remain. - -## Stage 17 — Audit cross-feature lifecycle and ownership - -Review and fill meaningful gaps rather than adding a full Cartesian-product -suite. Reuse existing regressions for version protection, consumed-header -sensitivity, scope diagnostics, NoSource, and historical removed descendants. - -Add representative combined cases for scheduled discovered items with helper- -supplied IDs; nested destinations and actual assigned indexes; required versus -if-present source versions; main-version ownership after main moves later in a -schedule; inherited relative resources; superseded relationships; and final -configuration/metadata/explanation agreement. Ensure consumed headers cannot -silently supply record IDs and secret values remain absent from diagnostics -and inspection. Preserve callback phases and source-local callback ordering. - -Compare an equivalent keyed graph through import and inheritance integrations: -both use the scheduler's eligible-node rule despite different domain operations. -Keep acquisition-source records stable when only graph construction order is -permuted in private scheduler/integration fixtures; do not promise unchanged IDs -when authored discovery order itself changes. - -Acceptance: run `go test -race ./internal/engine`, focused root facade tests, -and everyday checks. Record any genuine unresolved gap before marking complete. - -Completion: implemented in commit `55abcf8` (`Audit scheduled import and -inheritance lifecycles`). Verified with full root and engine tests, -`go test -race ./internal/engine`, `go vet ./...`, `go build ./...`, formatting, -and `git diff --check`. Added combined coverage for discovered items with helper- -supplied IDs, nested placement, inherited relative-resource provenance, -consumed secret headers, and scheduler parity under graph-construction -permutations. The everyday `./scripts/check.sh` still fails only on the two -prompt-catalog order expectations assigned to Stage 18. Stages 18–19 remain. - -## Stage 18 — Maintain simple and complex consumer examples - -Update the prompt-catalog example for directory search, multiple patterns, -exclusions, and helper-supplied IDs while retaining explicit registration and -application final validation. Include application import defaults and a -declaration exception, plus inspection of a deliberately empty optional search. -Keep a short simple single-file/required-file example so adoption does not -require understanding all discovery controls. - -Update fixture expectations for scheduled sequence positions. Verify mounted -resource origins, whole-sequence precedence, and quiet absence through executable -examples. Do not add a consumer-side discovery, merge, scheduling, or provenance -reconstruction pass. All fixtures remain local, deterministic, and secret-free. - -Acceptance: consumer tests and executable public examples pass; the consumer -guide links complete copyable examples rather than duplicating them. - -Completion: implemented in commit `de3666a` (`Expand prompt catalog discovery -example`). Verified with the full consumer suite, executable public examples, -`./scripts/check.sh`, local documentation links, formatting, and -`git diff --check`. The maintained catalog now demonstrates load-wide multiple -patterns, a declaration exception, exclusions, empty optional-search evidence, -helper-provided IDs, scheduled sequence positions, resource-origin lookup, and -whole-sequence precedence. A simple required-file example remains available. -Stage 19 remains. - -## Stage 19 — Finalize documentation and validate the completed unit - -Align README orientation, `docs/config.md`, public declaration comments, -`docs/internal/overview.md`, relevant focused internal docs, and the development -reading/test map. Document Kahn's algorithm, set union/difference, and active-path -cycle detection beside their owners. State acquisition versus resolution order, -explicit defaults precedence, no-selection evidence, and source namespace/key -construction without duplicating canonical contracts. - -Update architecture policy to recognize implemented explicit directory discovery -and shared scheduling while retaining the boundary against unrequested implicit -application-path discovery. Update only ADR status/navigation as needed to link -implemented contracts; accepted decision text stays unchanged. Leave release -notes/version choice and roadmap retirement for separately authorized release -preparation. Do not introduce release gates, CI waiting, consumer validation -requirements, dependencies, or unrelated defaults/security changes. - -Run the everyday `./scripts/check.sh`, `go test -race ./...`, consumer/examples -tests, local documentation links, and `git diff --check`; reuse checks already -covering unchanged final content instead of repeating them gratuitously. Review -the baseline-to-final public and YAML diff for removed `optional`, changed -requiredness and sequence ordering, and accurately documented native filesystem -capability limits. Inspect source and history indexes for accidental quadratic -rescanning; no timing thresholds or busy-host benchmark evidence are required. - -Acceptance: all stages are complete, the final implemented behavior matches the -roadmap and ADRs, validations are recorded here, and the implementation workspace -is committed and clean. No release is published by this plan. - -Completion: implemented in commit `f6a6b01` (`Finalize import discovery -documentation and audits`). Verified with `./scripts/check.sh`, -`go test -race ./...`, focused import/scheduler/inheritance/source-version and -public-defaults tests, consumer examples, local documentation links, formatting, -and `git diff --check`. Reviewed baseline-to-final public and YAML changes for -the `required` syntax cutover, requiredness defaults, and scheduled sequence -ordering; reviewed native filesystem capability wording and source/history -indexes. Replaced the source-version evidence nested scan with one source-ID -index. No release was published. The implementation workspace is committed. - -## Open Questions - -None required to begin implementation. The API, evidence shape, ordering keys, -scope boundaries, and cycle treatment are selected above. Private helper names -and file splits may be adjusted for clarity without changing those decisions. -If implementation reveals a genuine conflict between an accepted requirement -and existing contracts, report it rather than silently strengthening or weakening -either one. diff --git a/docs/roadmap/imports.md b/docs/roadmap/imports.md deleted file mode 100644 index 3d9355c..0000000 --- a/docs/roadmap/imports.md +++ /dev/null @@ -1,765 +0,0 @@ -# Import Ergonomics And Permissive Defaults - -Status: accepted design; implementation pending. This document captures planned -behavior and requirements. The [implementation plan](implementation.md) owns -delivery sequencing and the concrete API and integration choices. -The selected direction separates directory searches, filename matching, -recursion, and absence handling. Optional-by-default imports are the intended -direction. The consumer-directed and Ansible-aligned decisions below are settled; -API spelling and implementation details are specified by the implementation plan. -All new syntax below is illustrative and unimplemented. - -The durable architectural decisions are recorded in -[ADR 0015: Share Dependency-Ready Ordering Across Imports and Inheritance](../adr/0015-shared-dependency-ready-ordering.md) -and -[ADR 0016: Separate Import Selection, Traversal, and Presence Requirements](../adr/0016-independent-import-controls.md). -This roadmap owns implementation status and the detailed requirements for -carrying out those accepted decisions. - -## Guiding Philosophy - -ConfigKit should provide predictable configuration mechanisms and avoid imposing -application policy. Its defaults should permit optional contributions to be -absent, with explicit options for increased strictness when needed. That choice -may come from an operator or the consuming application; authored import -declarations can also express requirements on contributions. - -Permissiveness should distinguish absence from invalid supplied configuration: - -> Optional contributions may be absent by default. Consumers can explicitly -> require them. Supplied configuration must still be structurally valid and -> interpretable under the selected composition rules. - -This is a general design direction worth making more explicit and consistent -across ConfigKit. It is not blanket authorization to weaken existing contracts, -ignore errors, or add a global permissiveness switch. Other defaults should be -evaluated individually against this principle. - -Discovery should accept hidden entries and follow symlinks by default, with -options to narrow either behavior. An omitted filename filter should not impose -an implicit extension restriction. These defaults do not waive containment, -source validity, composition, or resource-limit rules. Recursion remains a -separate choice about search scope, not a measure of permissiveness. - -This is a hard-breaking pre-1.0 change. Do not build temporary aliases, -translation layers, compatibility modes, or staged deprecation support. Evaluate -the final interface on its own merits, without retaining syntax solely to ease -upgrades. - -## Current Baseline - -The [current import contract](../config.md#explicit-imports) uses `path` or -`glob`, with required imports by default and `optional: true` to permit absence. -Glob matching uses slash-based Go `path.Match` syntax. Recursive `**` syntax is -not supported; authors can match particular depths with patterns such as -`prompts/*.yaml` and `prompts/*/*.yaml`. - -Current imports expand depth-first in declaration order, with each ordinary -source contributing its body before its imports. The existing -[sibling inheritance resolver](../config.md#sibling-inheritance) instead follows -dependency chains before resolving dependents, using sorted sibling names and -an explicit stack. Neither implements the shared ordering selected below. - -The proposed changes concern naming, discovery, absence handling, and uniform -dependency ordering for imports and `extends`. Import destinations, strict -composition, inheritance overlay semantics, tier precedence, and resource -identity retain their distinct roles. - -## Proposed Import Vocabulary - -Use direct-file, directory-search, and path-glob forms. Directory searches -separate literal search roots from filename filters; path globs additionally -support wildcard directory segments and selection at particular depths: - -```yaml -configkit: - imports: - - dir: prompts - pattern: '*.yaml' - recursive: true - destination: [prompts] - placement: sequence-item - - path: defaults.yaml - required: true -``` - -The first declaration searches `prompts` and its descendants for matching -filenames, placing each source mapping as one prompt item, and permits no -matches. The second explicitly requires its target file. Sequence composition -still requires the consumer's existing registration, as explained below. - -The fields have distinct jobs: - -| Field | Proposed meaning | -| --- | --- | -| `path` | Import a named file or list of named files, relative to the declaring file | -| `dir` | Search a literal directory or list of literal directories, relative to the declaring file | -| `pattern` | A filename wildcard or list of wildcards; select files matching any effective pattern, using the default precedence below | -| `glob` | Select files with one path wildcard, including wildcard directory segments, relative to the declaring file | -| `exclude` | A filename wildcard or list of wildcards; remove matching files from any selector's results | -| `recursive` | Library default false, overridable by an application default; include descendant directories when effectively true | -| `depth` | Bound directory-search depth; supplying it requires effective recursion to be enabled; 1 means direct files, 0 or omission means unlimited when recursive | -| `follow` | Follow symlinks by default; false disables following within the discovery scope defined below | -| `hidden` | Include hidden entries by default; false excludes hidden files and prunes hidden directories within the discovery scope defined below | -| `required` | For `path`, require every listed file to be present; for `dir` or `glob`, require at least one selected file after exclusions across the complete search | -| `destination`, `placement` | Apply the existing placement rules to each selected source | - -This follows the familiar separation of traversal and filtering in Ansible's -[find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html) -and [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html) -interfaces, without adopting all their spellings, defaults, or matching modes. - -## Decisions Resolved By Project Direction - -- **No transition support.** Use the final interface immediately in the selected - pre-1.0 minor release. Remove `optional` in favor of `required`; do not accept - it as an alias or normalize old declarations at runtime. -- **Optional contributions by default.** `required: true` opts into failure on - absence. Supplied invalid configuration continues to fail. -- **No library-imposed filename restriction.** Omitting `pattern` selects all - filenames unless the application supplies a default pattern. Authors can - override that default explicitly. There is no library default YAML extension - list; extensionless sources remain discoverable. -- **Follow symlinks by default, with an opt-out.** Provide `follow: false` for - callers who want discovery without symlink following. Applicable containment - and filesystem contracts remain in force. -- **Include hidden entries by default, with an opt-out.** Provide `hidden: false` - to exclude hidden files and prune hidden directories from discovery. -- **One dependency-ordering algorithm for imports and `extends`.** Both use the - shared dependency-ready scheduler defined below. This is settled independently - of the existing implementations and includes refactoring both resolvers. - -OD-1 (recursion default), OD-2 (matching and selector expressiveness), OD-3 -(directive validation), OD-4 (discovery boundaries and cycles), OD-5 (policy -precedence), and OD-6 (no-selection evidence) are settled in the sections below. -Supplying `depth` requires effective recursion to be enabled, including when -enabled through an application default. This supersedes the earlier direction -to accept but ignore depth when recursion is disabled. - -## Decisions Resolved From Ansible - -The comparison covers `ansible.builtin.find`, `ansible.builtin.include_vars`, -`ansible.builtin.fileglob`, and `ansible.windows.win_find`. A consistent approach -means agreement among the relevant interfaces that offer the capability, not -that every Ansible module exposes it. Existing ConfigKit contracts and the -already selected optional-by-default direction remain authoritative; this is -not a general Ansible compatibility mode. - -The following choices are settled: - -- **Separate traversal from filename filtering.** Directory roots and filename - filters have separate roles; filters operate on basenames in visited - directories. A directory failing the filename filter does not prune its - descendants. This agrees with the directory-search interfaces above. - See [find's parameters](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/find_module.html#parameters) - and the [include_vars implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/plugins/action/include_vars.py). -- **Include an optional integer `depth` control.** All three directory-search - interfaces use that spelling. Direct files are at depth 1, and each additional - directory adds one. Omission permits unlimited depth when recursion is - enabled. See [include_vars](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/include_vars_module.html#parameters) - and [win_find](https://docs.ansible.com/projects/ansible/latest/collections/ansible/windows/win_find_module.html#parameters). -- **Treat `depth: 0` as unlimited.** This follows `include_vars` and agrees with - the current [find implementation](https://github.com/ansible/ansible/blob/stable-2.20/lib/ansible/modules/find.py) - and [win_find implementation](https://github.com/ansible-collections/ansible.windows/blob/main/plugins/modules/win_find.ps1). - The latter two document unlimited omission more clearly than explicit zero; - this choice uses source-confirmed behavior, not a claimed universal documented - Ansible guarantee. Negative values are invalid under the settled directive - validation rules below. -- **Directory searches select files, not directory documents.** Directories are - traversal containers, even when their names match `pattern`. This agrees with - the default file selection in the find modules and the file-oriented - [fileglob lookup](https://docs.ansible.com/projects/ansible/latest/collections/ansible/builtin/fileglob_lookup.html). - Existing regular-file and containment rules still govern opening selected - sources, together with the selected discovery controls. - -The accepted `recursive` spelling is retained. The formerly tentative -`max_depth` spelling becomes `depth`. Resource budgets remain separate from -depth, and the current limits still apply to an unlimited-depth search. - -## Shared Dependency-Ready Ordering - -**Status: settled for imports and `extends`; not yet implemented.** Use -deterministic dependency-ready ordering: Kahn's topological-sort algorithm with -a min-heap of eligible nodes. Both resolvers must use the same algorithm in the -same way, through one shared internal scheduling implementation, common -tie-breaking rules, and common cycle-path logic. - -### Scheduling Rule - -A dependency edge means that the dependent cannot complete until its dependency -has completed. For each dependency graph: - -1. Count each node's outstanding dependencies and identify initially eligible - nodes: those with none. -2. Choose the eligible node with the smallest stable ordering key. -3. Append it to the schedule, decrement its dependents' outstanding counts, and - immediately make newly eligible nodes available for the next selection. -4. Continue until all nodes are scheduled. If unfinished nodes remain with no - eligible node, reject the graph and identify an actual dependency cycle. - -Use stable source or definition identities to construct ordering keys, with -the same deterministic comparison rule in both resolvers. Map iteration order -and discovery-assigned source IDs must not supply the tie-breaker. Specify key -construction for source namespaces and logical definition paths during -implementation planning; it must distinguish nodes within each scheduled graph. - -Construct and validate a complete schedule for the available graph before -applying its resolution operations. A blocked dependent is not necessarily a -cycle member: diagnostics should identify the actual cycle using shared, -deterministic cycle-path logic. Source discovery and parsing may already have -occurred; this rule is not a promise to avoid reading invalid configurations. - -### Scope And Integration - -Discovery and resolution are separate operations. An importing source must be -read to discover its imports, but its contribution need not be assembled first. -Discovery order is not a substitute for the selected resolution schedule. - -Both resolvers construct dependency graphs and apply resolver-specific operations -through the shared scheduler. Imports retain strict composition; `extends` -retains inheritance overlay behavior. Scheduling does not authorize replacement -within a strict composition tier or change precedence between tiers. Shared -dependencies can be scheduled once where valid; this does not relax the existing -prohibition on repeated source inclusion for imports. - -Preserve item-local helper scopes and enclosing-before-nested inheritance -requirements. An imported item completes its helper contributions before being -placed as an item. Enclosing inheritance establishes the context in which nested -relationships are discovered and resolved. Use the same scheduler at those -scope boundaries, with the actual prerequisites represented; do not flatten raw -files or mappings in a way that loses their scope. Inheritance still resolves -after source assembly and precedence have established the effective document. - -The integration must account for resulting sequence positions, provenance, -history, and diagnostic ordering. It must distinguish source acquisition records -from assembly order rather than assuming they are the same. Review both -resolvers' contracts and behavioral tests when implementing the new schedule; -preserving the old traversal is not a design objective. - -### Rationale And Validation - -Dependency-ready ordering selects the smallest currently eligible node without -adding branch preference, batch barriers, or distance-from-root priority. With -a fixed total ordering key, it gives the lexicographically smallest valid -dependency-first schedule. It can interleave independent branches; branch -contiguity is not part of the chosen ordering contract. - -The comparison considered deterministic depth-first postorder, dependency -rounds, and descending longest-path depth as well. All can satisfy dependency -correctness, but their additional ordering policies did not justify choosing -them over the uniform eligibility rule. This decision gives no preference to -the existing implementation or to minimizing pre-1.0 refactoring. - -The scheduler requires O(E + V log V) time and O(V + E) memory for V nodes and -E dependency edges, assuming constant-time key comparisons. Source access, -parsing, merging, and provenance work are separate costs. - -Validate independent branches, unequal chains, branches with multiple -dependencies, shared dependencies at equal and unequal depths, cycles with -blocked non-cycle dependents, and nested scopes. Equivalent valid graphs and -ordering keys must produce identical schedules through the import and -inheritance integrations. Verify independence from graph insertion order and -preserve each resolver's domain-specific admissibility and combination rules. - -## Discovery Semantics - -### Filename Matching - -`pattern` applies to a file's basename, not its full relative path. For example, -`*.yaml` matches both `prompts/one.yaml` and -`prompts/weather/daily/summary.yaml` when the latter's directory is visited. -Recursion changes which directories are visited; it does not change what `*` -means. A nonrecursive search considers only files directly inside `dir`. - -Filename filters use the candidate name supplied by the declaration or -encountered during discovery, not the canonical symlink target's basename. -For example, a symlink named `welcome.draft.yaml` matches -`exclude: '*.draft.yaml'` even if its target is named `welcome.yaml`. This -selection rule does not change canonical source identity or containment checks. - -**OD-2: settled.** Use the existing case-sensitive Go `path.Match` wildcard -grammar for filename filters. When neither the declaration nor the application -supplies `pattern`, select all filenames, with no implicit extension restriction. -Do not add regular expressions, automatic dialect detection, or recursive `**` -syntax as part of this work. - -Retain `glob` as a permanent, separate selector using the existing slash-based -path wildcard grammar. For example, `collections/*/prompts/*.yaml` selects -files in a particular directory layout without selecting all descendants. -Literal `dir` roots and basename filters do not express that same search. -This capability justifies retaining `glob` independently of compatibility; -it uses the same new absence policy as other selectors. `glob` remains a -single string; list support is being added to `path`, `dir`, and the filename -filters. - -### Selector Lists And Set Operations - -Accept a string or a list of strings for `path`, `dir`, and `pattern`. Normalize -a scalar to a singleton list so both forms use the same selection logic. -Listed paths and directory roots resolve relative to the declaring file. -List order does not introduce precedence or replace the shared dependency-ready -resolution schedule. - -For directory roots D and filename patterns P, search the **Cartesian product** -D × P and take the **union** of the matching candidate files: - -`S = ⋃ {matches(d, p) | (d, p) ∈ D × P}` - -Each root uses the declaration's recursion, depth, and applicable discovery -controls. Resolve an omitted `pattern` from the application default first; -if neither supplies patterns, select all filenames in the visited directories -instead. This describes selection semantics, not a requirement to -materialize the product or traverse a directory again for each pattern. - -Overlapping patterns or directory searches within one declaration select the -same candidate once. Repeated identical entries in one `path` list likewise -select that candidate once; duplicate patterns or exclusions have no additional -effect. This local deduplication is not global source deduplication: -existing repeated-source and alias checks still apply to distinct inclusion -routes. In particular, it does not authorize silently merging separate alias -paths to an imported source. - -For declaration-local deduplication, identify a candidate by its normalized path -within its source namespace, before resolving symlinks. Compare paths relative -to the same namespace rather than relative to each search root. Overlapping -searches that encounter `prompts/welcome.yaml` therefore select it once, while -distinct symlink paths remain distinct candidates subject to the existing -canonical-source repeat checks. This candidate identity is separate from the -canonical identity used when accepting sources. - -For `path`, `required: true` requires every listed file to be present. For `dir`, -it requires at least one selected file across the complete union after -exclusions; it does not require a match per directory, pattern, or pair. A -missing directory can therefore contribute nothing while another supplies the -required match. The equivalent `glob` requirement is at least one selected file -after exclusions. Invalid supplied configuration still fails under the existing -rules. - -### Exclusions - -Accept `exclude` as a string or list of strings with the same case-sensitive -basename wildcard grammar as `pattern`. Apply it to results from `path`, `dir`, -and `glob`. A candidate matching any exclusion is removed before its content is -loaded or parsed. An omitted exclusion removes nothing. - -This is **set difference**: for candidate set S and its subset X matching any -exclusion, the selected set is `S ∖ X`. Exclusions win over positive matches. -They filter filenames, not directory paths, and do not prune traversal into a -directory whose name happens to match. For example, `exclude: '*.draft.yaml'` -removes matching files at every visited depth, including results of a path glob. - -For searches, evaluate `required` on the selected set after exclusions. For an -explicit `path` list, exclusions filter what is loaded but do not waive the -settled requirement that every listed file be present when `required: true`. -Excluding a candidate is not permission to suppress discovery, containment, -or resource-limit errors. Exclusions are local to the declaration; they do not -prohibit another declaration from explicitly selecting the same filename. - -### Resource Accounting - -Retain the existing resource limits, with an explicit adjustment to candidate -accounting for the new selection operations: - -- Every examined directory entry consumes the discovery visit budget, including - nonmatches and excluded entries. A small selected set does not authorize an - unbounded scan. -- Check distinct selected candidates against the remaining source allowance - after declaration-local deduplication and exclusions. Overlapping patterns - and search roots do not charge the same selected candidate more than once - within a declaration; excluded candidates do not consume source allowance. -- Excluded files are not read for content and consume no content-reading - allowance. Existing byte limits continue to govern selected sources. - -The current glob implementation checks candidate counts against the remaining -source allowance during expansion. Update that accounting to use the filtered, -deduplicated selection; do not reject an otherwise valid search solely because -its positive matches exceed that allowance before exclusions are applied. -Discovery limits still apply throughout the search. This is not global source -deduplication and does not suppress repeated-source errors across declarations -or distinct aliases. - -### Recursive Discovery And Depth Limits - -Enable recursion explicitly or through an application default for recursive -discovery, optionally bounded by `depth`. -The motivation is to let authors organize definitions in subdirectories without -updating import declarations for every depth. Directory organization should not -change resource identity or the effective configuration destination. - -The selected model uses an explicit search root and recursion flag, rather than -introducing recursive `**` syntax or inferring a root from a wildcard prefix. - -**OD-1: settled.** The library default for `recursive` is false: search only -direct files. An application may supply a different default; an explicit -declaration value overrides it, including `recursive: false`. Recursion remains -independent of the permissive defaults for absence, symlinks, and hidden entries. - -The settled depth convention is: - -- Files directly inside the search root are at depth `1`. -- Each additional directory increases depth by one. -- Omitting `depth`, or setting it to 0, imposes no caller-selected depth limit - when recursion is enabled; existing loader resource limits still apply. -- Supplying `depth` requires effective `recursive: true`. If recursion is false - after resolving declaration, application, and library defaults, any supplied - `depth` is an error, including `0` or `1`. Depth never enables recursion itself. -- With recursion disabled and `depth` omitted, select only direct files. -- Depth controls discovery only, not namespaces, identities, or destinations. - -Resolve the effective recursion value before checking depth's applicability. -For example, `depth: 2` with no explicit `recursive` is valid if the application -default enables recursion, but invalid with the library default. Explicit -`recursive: false` overrides an application default of true and makes any -supplied depth invalid. Depth must also have the valid type and range defined -under directive validation; enabling recursion does not excuse malformed depth. - -Recursive discovery must retain the existing filesystem containment and -repeated-source contracts and follow the resource accounting defined above. -The discovery visit budget remains separate from the caller-selected depth -bound and still applies to unlimited-depth searches. - -### Symlinks And Hidden Entries - -**OD-4: settled.** Discovery controls are local to the declaration. Directory -searches default to `follow: true` and `hidden: true`, subject to the application -defaults described below. Authors may disable either behavior independently of -recursion, depth, matching, or absence handling. - -Define hidden entries by a leading dot in the portable source namespace. -`hidden: false` skips discovered hidden files and prunes discovered hidden -directories. `follow: false` skips discovered symlink files and prevents descent -through discovered symlink directories. Explicitly named roots and separate -direct-file imports retain their existing source-access semantics. The controls -do not implicitly propagate to declarations inside imported files. - -These are discovery-selection controls, not a load-wide access prohibition. -On `glob`, filtering applies to wildcard-selected entries; explicitly named -path components remain explicit access. - -When following links, detect a canonical directory repeated on the active -traversal path and fail with a cycle diagnostic. Do not silently prune the loop -and continue. Existing dependency-cycle and repeated-source errors remain in -force; discovered aliases do not authorize silently deduplicating imported -sources. Directory-cycle handling retains the existing containment and -native-filesystem capabilities. - -These controls narrow selection rather than authorize ignoring errors in a -selected source. `required: true` applies after discovery filters: if the search -selects no file, it fails. Native filesystems retain their existing capabilities; -this feature does not promise link inspection where a supplied filesystem does -not expose it. - -## Illustrative Examples - -These examples describe the proposed interface, not configuration accepted by -the current release. They assume the consumer has enabled imports and has not -changed the library's policy defaults. File paths and directory roots resolve -from the declaring file; destinations use the -existing relative-placement rules. - -### Import One File - -```yaml -configkit: - imports: - - path: defaults.yaml -``` - -Import `defaults.yaml` into the declaring source's mapping scope. Under the -proposed default, a missing file contributes nothing. Add `required: true` when -its presence is necessary. A present but invalid file still fails loading. - -### Require Several Named Files - -```yaml -configkit: - imports: - - path: [defaults.yaml, site.yaml] - required: true -``` - -Both files must be present. Listing them together shares the declaration's -options; their textual order does not establish overlay precedence. Their -contributions follow the shared dependency-ready schedule and strict -composition rules. - -### Recursively Discover Mapping Fragments - -```yaml -configkit: - imports: - - dir: services - pattern: '*.yaml' - recursive: true - destination: [services] -``` - -This discovers files such as `services/api.yaml` and -`services/regional/worker.yaml`, but not `services/README.md`. Each file contains -named service entries, for example `api: {port: 8080}` or -`worker: {concurrency: 4}`. They compose beneath `services` under the existing -strict composition rules. Directory names do not create additional mapping -keys. Missing directories or no matching files contribute nothing. - -Adding `depth: 2` limits discovery to direct files and files one directory below -`services`: it still selects `services/regional/worker.yaml`, but not -`services/regional/east/worker.yaml`. Switching to `recursive: false` requires -omitting `depth`; leaving `depth: 2` present is a declaration error. - -Hidden entries and contained symlinks participate by default. To narrow the -search, the declaration can also include: - -```yaml - follow: false - hidden: false -``` - -These controls narrow discovery below the explicitly named root; the root -continues to resolve under the existing source-access rules. - -### Search Several Prompt Collections And Filename Patterns - -```yaml -configkit: - imports: - - dir: [prompts, shared-prompts] - pattern: ['*.prompt.yaml', '*.prompt.yml'] - exclude: ['*.draft.prompt.yaml', '*.draft.prompt.yml'] - recursive: true - required: true - destination: [prompts] - placement: sequence-item -``` - -The two directories and two patterns form four Cartesian-product pairs; the -selected candidates are the union of their matches, minus excluded filenames. -This selects `prompts/welcome.prompt.yaml`, -`prompts/weather/daily/summary.prompt.yaml`, and -`shared-prompts/greeting.prompt.yml`, but not `prompts/weather/helpers.yaml` -or `shared-prompts/greeting.draft.prompt.yml`. -Each selected file contains one prompt mapping, -such as `id: prompt.welcome` and `text: Welcome`. The consumer registers the -`prompts` sequence with identity field `id` through the existing -`Definition.SequenceCompositions` API. Distinct IDs compose into one sequence; -duplicate IDs fail strict composition. Overlays continue replacing sequences. - -Here `required: true` means at least one file must remain across the entire -search after exclusions, not that every directory or pattern must produce a -match. One missing root is acceptable if the other supplies a selected file. The -filename convention keeps helper files out of this discovery declaration; it -does not prevent a selected file from explicitly importing a helper. Existing -repeated-source rules still apply if declarations select overlapping sources. - -### Select A Directory Layout With A Glob And Exclusions - -```yaml -configkit: - imports: - - glob: 'collections/*/prompts/*.yaml' - exclude: '*.draft.yaml' - required: true - destination: [prompts] - placement: sequence-item -``` - -This selects files such as `collections/support/prompts/welcome.yaml`, excluding -`collections/support/prompts/welcome.draft.yaml`. It does not select deeper -files such as `collections/support/prompts/internal/help.yaml`. At least one -file must remain after exclusions. The same prompt-item composition setup as -the preceding example applies. Exclusion patterns match basenames even though -the positive glob matches paths. - -## Optional Imports By Default - -The selected direction is optional-by-default for explicit files, directory -searches, and globs, with `required: true` opting into failure on absence. This -deliberately supersedes the earlier suggestion to retain required-by-default -imports. - -The table assumes the library's default absence policy. Applications may select -a different default using the settled precedence rules below. - -| Situation | Proposed behavior | -| --- | --- | -| Missing `path` entry; `required` omitted or false | That entry contributes nothing; other listed files may still contribute | -| Zero selected files across a complete search; `required` omitted or false | Contribute nothing | -| Any listed `path` file missing with `required: true` | Fail with an import diagnostic | -| Zero selected files across a complete `dir` or `glob` search after exclusions with `required: true` | Fail with an import diagnostic | -| One `dir` root missing, but another supplies a selected file with `required: true` | Satisfy the declaration's presence requirement | -| Invalid pattern or malformed import declaration | Fail | -| Permission failure during discovery, unreadable selected file, or malformed YAML in a selected file | Fail | -| Composition conflict or other invalid supplied configuration | Fail under the applicable existing rules | - -An optional declaration is permission for absence, not permission to ignore a -selected contribution that cannot be read or interpreted. Optionality must not -turn processing failures into apparently successful partial configuration. - -This default supports installations with different optional components without -requiring ceremony for every potentially absent directory or file. Its tradeoff -is that a misspelled path or pattern can also contribute nothing. Authors and -applications that need an import to exist can state that requirement explicitly; -applications remain responsible for validating their necessary final values. -Final validation does not automatically detect every unintentionally omitted -contribution. - -## Who Chooses Strictness? - -**OD-5: settled.** Provide load-level defaults for `required`, `follow`, `hidden`, -`recursive`, and `pattern`. Resolve each applicable field independently using -this precedence: - -1. Explicit value on the import declaration. -2. Application-selected default, when supplied. -3. Library default: false for `required` and `recursive`, true for `follow` and - `hidden`, and no filename restriction for `pattern`. - -An explicit `pattern` replaces the application default rather than combining -with it. In particular, `pattern: []` selects nothing, even when the application -supplies patterns. An application can default to `['*.yml', '*.yaml']`; without -either an application default or an explicit pattern, a directory search -attempts to load all files in its search scope. Application defaults apply only -to selectors that support the field; they do not add inapplicable options to -other selectors. - -Apply this rule to transitive declarations too. A declaration does not inherit -these field values from its importing parent. Applications may expose the -defaults to operators through their own interfaces; simple consumers need no -additional setup. Exact Go API spelling belongs to implementation planning. - -Explicit declarations remain authoritative, permitting intentional exceptions -to stricter application defaults. Defaults are not enforcement controls. Do not -add options that override explicit declarations as part of this work without a -separate concrete requirement and scope decision. The fields' applicability to -each selector is defined under directive validation. - -## Evidence For Optional Declarations With No Selection - -**OD-6: settled.** Record minimal structured inspection evidence when an -optional declaration selects nothing: its declaring source/location, selector, -and no-selection outcome. Emit no warning by default and do not fabricate a -loaded-source record. This keeps optional contributions quiet during ordinary -use while allowing applications to explain omissions during troubleshooting. - -No selection does not necessarily mean a source was missing. It may result from -missing sources or no positive matches, an explicit empty list, or exclusions -removing every candidate. The inspection record must not describe deliberate -exclusion or an empty list as missing files. - -This evidence is scoped to the whole declaration. An optional `path` list with -one selected file and one missing file does not produce this no-selection -record. Per-entry evidence for partial omissions is outside this feature's -scope; no additional exclusion audit is required. - -Decide the exact inspection record shape during implementation planning, -reusing existing structures where appropriate and retaining existing inspection -contracts. Include ownership tests for the new evidence. This addition does -not require applications to reconstruct declaration-level no-selection evidence -themselves. - -## Release Boundary - -Ship one final interface with a hard break in a pre-1.0 minor release, following -the [release policy](../policy/release.md#versioning-and-compatibility). There is -no transition release, legacy parser path, compatibility switch, or temporary -`optional` alias. Release notes should describe the changed behavior and final -syntax; this documentation does not imply migration machinery. - -Retaining `glob` is a settled capability decision, as described under filename -matching. It does not preserve the old required-by-default behavior or promise -compatibility with every old declaration or use case. - -## Named Algorithms And Implementation Documentation - -Use the common names below in implementation comments and maintained -documentation when the feature lands. Keep them beside the relevant logic and -in the appropriate owning documents so human readers can recognize the design -without reconstructing it from code. This does not call for new public API -terminology or duplicate contracts across documents. - -| Operation | Name and intended use | -| --- | --- | -| Shared import and `extends` scheduling | **Kahn's algorithm with a stable ordering key**, using a min-heap for the ready set; the same dependency-ready topological sort serves both resolvers | -| Multiple directory roots and filename patterns | **Cartesian product** of roots and patterns, followed by **set union** of matching candidates within the declaration | -| Exclusion filtering | **Set difference** between positive candidates and those matching any exclusion | -| Directory symlink cycles | **Active-path cycle detection**, using canonical directories on the current traversal path rather than treating every previously visited directory as a cycle | - -Scalar-to-singleton normalization lets scalar and list forms share selection -logic. Set operations describe membership, not resolution order; the shared -stable topological schedule supplies that order. Preserve the scheduler's -complexity explanation and distinction between cycles and blocked dependents -in its implementation documentation. Do not label implementation-specific -matching or cycle-path helpers with an algorithm name until that algorithm is -actually chosen. - -## Directive Validation - -**OD-3: settled.** Validate declaration shapes and values independently of -whether discovery selects any files. Invalid declarations fail even when a -search root is missing or a list is empty. This does not require opening or -parsing excluded files or establish a new first-error ordering guarantee. - -### Selectors And Applicable Options - -Require exactly one selector per declaration: `path`, `dir`, or `glob`. A list -inside `path` or `dir` still counts as one selector. Use separate declarations -when combining different selector kinds. Diagnose unknown fields and options -that have no meaning for the chosen selector. The removed `optional` spelling -is not accepted. - -| Option | `path` | `dir` | `glob` | -| --- | --- | --- | --- | -| `required`, `exclude` | Yes | Yes | Yes | -| `destination`, `placement` | Yes | Yes | Yes | -| `pattern` | No | Yes | No | -| `recursive`, `depth` | No | Yes | No | -| `follow`, `hidden` | No | Yes | Yes | - -### Types And Depth Applicability - -- `path`, `dir`, `pattern`, and `exclude` accept strings or flat lists of strings. -- `glob` accepts a string. -- `required`, `recursive`, `follow`, and `hidden` accept booleans. -- `depth` accepts nonnegative integers under ConfigKit's existing YAML scalar - rules. Zero means unlimited when recursion is enabled; negative, string, - boolean, and fractional values are invalid. Do not introduce an arbitrary - maximum depth. -- Explicit null is invalid for these fields, including null list members. - Do not coerce non-string members to strings or flatten nested lists. -- Existing `destination` and `placement` validation remains unchanged. - -A supplied `depth` additionally requires effective recursion to be enabled, -using the default precedence defined above. Valid type and range alone do not -make depth applicable. This rule applies even to empty directory or pattern -lists. With recursion enabled, malformed depth values still fail validation. - -### Empty Lists, Empty Strings, And Duplicates - -Empty lists are valid and retain their set meanings: - -| Declaration | Meaning | Effect of `required: true` | -| --- | --- | --- | -| `path: []` | No named files | Presence requirement satisfied: no listed file is missing | -| `dir: []` | No search roots | Fail: no selected file | -| `dir: fragments` with `pattern: []` | No positive patterns; select nothing | Fail: no selected file | -| `exclude: []` | Exclude nothing | Otherwise unchanged | - -An empty pattern list differs from omission: omission uses the application -default if supplied, otherwise all filenames. An explicit empty list selects -nothing and overrides any application default. - -Empty strings are invalid for `path`, `dir`, and `glob`, including empty entries -in path or directory lists. Use `.` to explicitly name the declaring directory -as a search root. Empty strings in `pattern` and `exclude` are valid wildcard -expressions that match no actual filename. Other source-path and wildcard -validation rules still apply. - -Repeated identical path entries within one declaration select that candidate -once. Repeated patterns or exclusions have no additional effect, and directory -search overlaps follow the settled set-union rule. These rules do not suppress -existing repeated-source errors across separate declarations or collapse -separate symlink aliases. - -Validate every supplied wildcard expression even if it is inactive because -another list is empty or discovery finds no files. Likewise, malformed list -members remain errors even when other entries are valid. Do not silently drop -invalid members or treat malformed filters as absent. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index bf61636..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,242 +0,0 @@ -# Import Discovery and Scheduling Review Findings - -Status: complete; all six findings addressed in separate validated commits. - -Reviewed implementation: `aa93aed..15943e2`, ending at -`15943e233ca3cd1286c03c0e7085871db344cd4b`, against the accepted -[roadmap](imports.md), [implementation plan](implementation.md), -[ADR 0015](../adr/0015-shared-dependency-ready-ordering.md), and -[ADR 0016](../adr/0016-independent-import-controls.md). - -The shared scheduler, scoped assembly, independent defaults, and detached -inspection surface are present. The findings below concern their integration -and discovery edge cases, not a proposal to change the accepted architecture. -Implement fixes with focused regressions, preserving the selected semantics -and existing resource limits. This review made no implementation changes. - -## 1. [x] P1 — Preserve native source identity and relative import bases - -Owners: [directory selection](../../internal/engine/import_selection.go), -especially assignment of `target.identity` to a file candidate's `accessPath` -around lines 184–194; [glob selection](../../internal/engine/glob.go), around -lines 81–90; and source admission in `visitPath`. - -Directory and wildcard-glob selection use resolved symlink targets as native -source-admission paths. Direct `path` selection retains the supplied filesystem's -namespace path. The same native source therefore has different identity, -relative import behavior, and resource context depending on its selector. -The plan explicitly separates traversal identity from native admission identity. - -Public reproduction with `NewFilesystem("alias", os.DirFS(root))`: - -- `aliases/item.yml` links to `../data/item.yml`. -- That item's YAML imports `helper.yml` and contributes `value: 1`. -- `aliases/helper.yml` contributes `owner: alias`; `data/helper.yml` contributes - `owner: target`. -- A root main document selecting `path: aliases/item.yml` yields - `SourceInfo.Path == "aliases/item.yml"` and `owner == "alias"`. -- Selecting `dir: aliases` with `pattern: item.yml`, or - `glob: 'aliases/ite*.yml'`, instead yields `SourceInfo.Path == "data/item.yml"` - and `owner == "target"`. - -The prior implementation at `aa93aed` yields the alias path and alias helper -for the same wildcard glob. This is a regression that silently changes final -configuration, not merely different reporting. It can also change repeated-source -admission and the stable scheduling key. - -Remediation: retain separate candidate spelling, traversal/cycle identity, and -native admission path. Do not silently apply host-style canonical source identity -to native filesystems. Verify path, directory, and glob equivalence for a file -alias and an alias directory, including nested imports, source context, origins, -and existing repeated-source behavior. Keep host canonicalization unchanged. - -Completed: native directory and glob discovery retain namespace admission paths, -while traversal cycle identities remain resolved. Added public regressions for -file and directory aliases across all three selectors, helper/resource origins, -source contexts, repeated paths, and distinct alias admission. The regression -failed before the fix. Focused discovery/resource tests and `./scripts/check.sh` -passed with Go 1.26.9 and offline dependency selection. - -## 2. [x] P1 — Scope main-version protection to the actual effective path - -Owner: [composePendingItems](../../internal/engine/imports.go), around lines -694–695, and its call to `importedVersionClaim`. - -Pending item assembly passes the global protected version rule into an item-local -mapping. Consequently, an ordinary item field named `version` is treated as if -it occupied the protected main-root `/version` path. Imported item bodies have -no `importRoleMain` exemption in that local scope, so their own domain fields -are rejected as well as helper contributions. - -Public reproduction: the main document declares `version: 1` and imports an -item with `destination: [prompts]` and `placement: sequence-item`. The item is -`{id: prompt, version: domain-version}`. Load with imports enabled, default -source-version mode, and a `VersionRule` at `Keys("version")` whose checker -accepts the main header. Current loading fails with `import.version` at -`/version`; the identical input succeeds at `aa93aed`. - -The item's effective field is `/prompts/0/version`, not `/version`. This -unintentionally strengthens main-version ownership into a field-name restriction -inside item scopes, contrary to the requested preservation of version semantics. - -Remediation: use correctly projected/scoped protection, or defer the check to a -boundary with the actual effective item coordinates. Preserve protection where -the effective paths really overlap. Cover imported/nested items, item helpers, -main-item loading, underlays, and enabled source-version modes without adding -blanket exemptions or banning ordinary domain version fields. - -Completed: removed the global-rule check against item-local field names; enclosing -composition checks retain effective-path protection, including the existing -main-item scoped check. New public regressions cover item bodies, helpers, nested -items, underlays, and selected main items, and failed before the change. The -full everyday check passed, including existing default/enabled source-version -and assigned-index ownership regressions. - -## 3. [x] P2 — Accept valid multi-component native filesystem roots - -Owner: [resolveNativeImportPath](../../internal/engine/filesystem_links.go), -especially lines 170–181. - -The resolver splits a namespace-relative path from its first component and -requires every intermediate prefix to be inside the configured root. For root -`nested/root`, the first inspected prefix `nested` necessarily fails containment, -even though the complete selected path lies within `nested/root`. - -Public reproduction: use a link-aware `os.DirFS` namespace containing -`nested/root/main.yml` and `nested/root/parts/value.yml`. Main imports `dir: parts` -with `pattern: '*.yml'`. Supply `FilesystemContext{FS: &filesystem, -Root: "nested/root"}`. Loading fails with `import.outside_root` on the directory -selector despite all inputs being contained and no symlink being involved. - -Remediation: resolve relative to the selected native root, or distinguish root -prefix traversal from an actual escape. Preserve containment on selected and -resolved targets. Test directory and glob discovery under multi-component roots, -with ordinary entries and supported symlinks. Do not require consumers to flatten -their roots or remove optional link-inspection capability. - -Completed: native link resolution allows ancestor prefixes while resolving a -contained multi-component path; selected paths and resolved link targets still -undergo their existing containment checks. Public directory/glob regressions -cover ordinary and aliased roots plus escaping links. All four valid cases -failed before the fix. The full everyday check passed. - -## 4. [x] P2 — Distinguish finite symlink reuse from actual cycles - -Owners: [resolveHostImportPath and resolveNativeImportPath](../../internal/engine/filesystem_links.go), -especially the `activeLinks` sets around lines 115–134 and 172–194. - -These sets retain every symlink encountered during resolution of a whole path; -entries are never removed after an expansion completes. A finite pathname that -uses the same link twice is therefore rejected as a cycle. This is distinct from -the correctly scoped active-directory set in recursive directory traversal. - -Public host reproduction: create `parts/up -> ..` inside the configuration root, -and a regular `parts/value.yml`. Import -`dir: parts/up/parts/up/parts` with `pattern: '*.yml'` and default nonrecursive -discovery. That finite explicit root resolves to `parts`; loading instead fails -with `import.cycle`. No recursive directory walk or infinite link expansion is -requested. - -Remediation: identify actual active symlink expansion cycles, not any repeated -link occurrence. Preserve ordinary explicit-root access semantics and the -separate active-path directory-cycle rule. Add finite-reuse and real-loop -regressions for host and link-aware native access; do not replace this defect -with an arbitrary new link or path-depth restriction. - -Completed: host and native resolvers now release active links when each target -expansion completes, using explicit frames rather than retaining a whole-path -visited set. Added resolver regressions for finite reuse, self loops, and mutual -loops, plus public host/native directory/glob loading coverage. Finite cases -failed before the change; all pass now alongside existing traversal-cycle and -containment tests. The full everyday check passed; no new limit was added. - -## 5. [x] P2 — Validate the complete depth token before saturating its value - -Owner: [parseImportDepth](../../internal/engine/import_spec.go), around lines -423–430. - -On integer overflow, the parser immediately returns the saturated bound and -success. It never validates the remainder of the numeric token, so a sufficiently -large fractional or exponent-form number is accepted as an integer depth. - -Public reproduction: load a main document declaring an optional missing -directory, `recursive: true`, and either -`depth: 999999999999999999999999.5` or -`depth: 999999999999999999999999e1`. Both loads succeed. Both declarations should -fail `import.invalid`, independently of whether discovery selects files. - -Remediation: validate all characters while saturating the numeric accumulator, -or validate integer grammar before conversion. Continue accepting arbitrarily -large valid nonnegative integers and numeric `-0`; do not introduce a machine-size -maximum. Cover invalid suffixes after the overflow point as well as ordinary -invalid depths and valid large integers. - -Completed: depth parsing saturates its accumulator without stopping token -validation. Public absent-directory regressions now reject fractional/exponent -suffixes after overflow and retain valid huge integers and `-0`. The two invalid -cases failed before the fix. Existing depth/specification coverage and the full -everyday check passed; the accepted integer range was not narrowed. - -## 6. [x] P2 — Apply the active import-depth limit to actual inclusion edges - -Owner: [walkDeclarations](../../internal/engine/imports.go), around lines 403–405. - -The active-edge limit is checked for every declaration before candidate selection. -An empty list or absent optional search at the depth boundary is therefore -treated as an additional active import edge, even though it loads no source. -This strengthens the established active-edge budget into a declaration-depth -restriction and conflicts with the selected empty/absent contribution semantics. - -Public reproduction: a main file followed by 64 imported files loads successfully -when the final file contributes `end: true` and no imports. Adding either -`path: []` or an optional `glob: 'missing/*.yml'` to that last file produces -`import.limit_exceeded`. The same 64 actual inclusion edges are present in all -three cases. - -Remediation: enforce the existing depth bound when a selected source would add -an inclusion edge. Continue validating declaration syntax, policy applicability, -and selection resource budgets even when no source is selected. At the boundary, -empty lists and optional zero selections must succeed, while a real next source -must still fail. Keep the existing numeric resource limit unchanged. - -Completed: the depth bound is checked for an existing selected target at source -inclusion, rather than for every declaration. A test-controlled one-edge budget -covers empty/required-empty lists, absent path/directory/glob selections, actual -next sources, and invalid declarations. The original 64-edge public examples -now succeed with empty optional contributions; a real next edge still fails. -The full everyday and race checks passed without changing the resource limit. - -## Review validation - -Passed on the reviewed revision with Go `go1.26.9 linux/amd64`: - -- `./scripts/check.sh` with the installed Go binary directory first on `PATH`, - `GOTOOLCHAIN=local`, `GOPROXY=off`, and a build cache outside the repository. -- `go test -race ./...` with the same offline toolchain selection. -- `git diff --check aa93aed..15943e2`. - -All six findings were independently reproduced through public APIs using a -temporary Go program, in-memory filesystems, and disposable real filesystem -fixtures outside the repository. The native wildcard-alias and ordinary item- -version regressions were also compared with an archived `aa93aed` checkout -outside the repository. No code, dependencies, release procedure, consumer -repository, or accepted ADR decision text was changed during the review. - -## Remediation validation - -All six findings have completion evidence above. Final validation passed with -Go `go1.26.9 linux/amd64`, local toolchain selection, offline dependencies, and -build caches outside the repository: - -- `./scripts/check.sh` (formatting, whitespace, full tests, vet, and build). -- `go test -race ./...`. -- Local documentation-link checks and `git diff --check`. - -The original external public-API reproductions were rerun against the completed -fixes. Native path/directory/glob aliases now retain the same helper origin and -resource context; ordinary item versions load; nested native roots and finite -symlink reuse load; invalid depth suffixes fail; and empty contributions do not -consume active inclusion depth. Focused regressions also retain actual version -ownership, real cycle/escape errors, invalid-declaration handling, and the depth -budget for real sources. No dependencies, public controls, release gates, or -downstream repositories were changed.