641 lines
36 KiB
Markdown
641 lines
36 KiB
Markdown
# Implementation Plan: Import Discovery and Dependency-Ready Ordering
|
||
|
||
Status: Stages 1–11 complete; 8 implementation stages remain.
|
||
|
||
## Authority, baseline, and execution
|
||
|
||
Implement the accepted [import roadmap](imports.md),
|
||
[ADR 0015](../adr/0015-shared-dependency-ready-ordering.md), and
|
||
[ADR 0016](../adr/0016-independent-import-controls.md), starting at
|
||
`94381a67fd39635f448a5225e38adb2a7f43cc9f`. The roadmap owns target behavior and
|
||
scope; this plan settles delivery order and the remaining implementation choices.
|
||
The accepted ADRs already exist; do not draft replacements or rewrite their
|
||
decision text.
|
||
|
||
The implementer is a Luna coding agent. Execute one numbered stage per focused
|
||
prompt, in order. Each stage should end with a validated, nonempty commit and a
|
||
completion entry here identifying the commit, checks, and any remaining work.
|
||
Do not mark a stage complete while its acceptance conditions remain unmet.
|
||
Preserve unrelated work. This planning pass changes documentation only.
|
||
|
||
Start with [development](../development.md) and read the applicable
|
||
[agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||
[testing](../policy/testing.md), and [documentation](../policy/documentation.md)
|
||
policies. Prefer the code graph for discovery, then inspect current source and
|
||
tests; indexed snippets can have stale line ranges. Use the actual working tree
|
||
when an indexed snippet is incomplete or mismatched.
|
||
|
||
Keep each stage's public documentation accurate to its implemented boundary.
|
||
Private groundwork may precede wiring, but do not expose options that silently
|
||
do nothing. The final syntax cutover is Stage 14; there is no consumer-facing
|
||
compatibility mode, alias, or alternate scheduler. Intermediate private helpers
|
||
are scaffolding, not a second supported interface. Release publication and
|
||
consumer dependency updates are outside this plan.
|
||
|
||
## Settled implementation choices
|
||
|
||
### Public defaults
|
||
|
||
Add a root-owned `ImportDefaults` struct and an `ImportDefaults ImportDefaults`
|
||
field to both `Options[T]` and `LayeredOptions[T]`:
|
||
|
||
```go
|
||
type ImportDefaults struct {
|
||
Required *bool
|
||
Follow *bool
|
||
Hidden *bool
|
||
Recursive *bool
|
||
Pattern []string
|
||
}
|
||
```
|
||
|
||
Nil boolean pointers mean unspecified. Copy pointed-to values when loading
|
||
begins. Nil `Pattern` means unspecified; non-nil empty means select nothing for
|
||
directory searches. Snapshot the slice without collapsing this distinction.
|
||
No new defaults field belongs on `Input`, `Underlay`, or `Definition`: one
|
||
load-level selection applies to main and all underlay/transitive import
|
||
declarations. Existing per-input `Imports` booleans still enable imports.
|
||
|
||
Resolve each applicable declaration field independently: explicit declaration,
|
||
then application default, then library default. Library defaults are required
|
||
false, recursive false, follow true, hidden true, and unrestricted filenames.
|
||
No parent-declaration inheritance, enforced application override, default
|
||
exclusion list, or application depth setting is added. `pattern` replaces rather
|
||
than augments the default. Validate supplied default patterns once at option
|
||
selection, reporting `options.invalid` before source reads or callbacks.
|
||
Defaults do not make unsupported fields applicable to a selector.
|
||
|
||
Use a private presence-aware declaration representation, distinct from its
|
||
resolved discovery policy. Normalize scalar/list selectors and filters once.
|
||
Keep existing import path validity rules, with `.` accepted for literal directory
|
||
roots. Match basenames with case-sensitive `path.Match`; path globs retain their
|
||
existing grammar, including rejection of recursive `**`. A filename filter has
|
||
no directory-recursion interpretation. Validate all expressions even in empty
|
||
or otherwise inactive selections. Follow the roadmap's option applicability,
|
||
empty-list, null, and depth rules exactly.
|
||
|
||
`depth` has no arbitrary machine-integer maximum. Accept nonnegative YAML
|
||
integers under the existing scalar rules, including bounds larger than can be
|
||
reached under the visit budget; an implementation may saturate its internal
|
||
bound at the largest representable traversal depth without rejecting that input.
|
||
Numeric `-0` denotes zero. Do not accept fractional/exponent numbers as integer
|
||
depth or coerce strings.
|
||
|
||
### No-selection inspection
|
||
|
||
Extend root-owned `SourceInfo` with `ImportSelections []ImportSelection`. Add
|
||
root-owned string types and constants:
|
||
|
||
- `ImportSelector`: `ImportSelectorPath = "path"`, `ImportSelectorDirectory =
|
||
"dir"`, and `ImportSelectorGlob = "glob"`.
|
||
- `ImportSelectionOutcome`: `ImportSelectionNone = "none"`.
|
||
- `ImportSelection` fields: `Path Path`, `Location Location`,
|
||
`Selector ImportSelector`, and `Outcome ImportSelectionOutcome`.
|
||
|
||
`Path` is the source-local declaration path, such as
|
||
`/configkit/imports/0`; `Location` identifies the declaring mapping and carries
|
||
its actual SourceID. Attach one record to that declaring source for each
|
||
effectively optional declaration whose entire selected set is empty. Keep
|
||
records in declaration order. Do not record partial omissions, candidate lists,
|
||
raw selector values, inferred missing-file reasons, warnings, or fake sources.
|
||
Required empty `path: []` succeeds but gets no optional-declaration record.
|
||
|
||
Reuse `Result.Sources`, `LoadError.Sources`, and callback source snapshots rather
|
||
than adding another query surface. Snapshots contain only outcomes established
|
||
by that point; no complete discovery record is promised after an earlier error.
|
||
Records remain source-local through mounting and item placement. Deep-copy paths
|
||
and slices through engine cloning and facade conversion. Existing version-text
|
||
redaction remains independent and unchanged.
|
||
|
||
### Scheduling and identities
|
||
|
||
Keep the shared scheduler private to `internal/engine`, in a focused
|
||
`dependency_order.go` owner; no new public scheduling API or dependency is needed.
|
||
It accepts immutable node keys and dependency edges and returns a complete order
|
||
or an actual closed cycle. Use Kahn's algorithm and `container/heap`; deduplicate
|
||
edges before counting dependencies. Compute keys once. Compare key tuples by Go
|
||
bytewise string order component by component, shorter tuple first on a prefix.
|
||
|
||
Use these key constructions in both integrations:
|
||
|
||
- Import nodes: `(namespace-kind, namespace-anchor, resource-kind, resource-path)`.
|
||
Kind is `host` or `native`; resource kind is `file` or `synthetic`. Host paths
|
||
are canonical absolute paths rendered with slash separators; native paths use
|
||
their existing namespace-relative identity. Synthetic roots use their explicit
|
||
input anchor, not a label or allocated token address.
|
||
- Derive namespace anchors from explicit request positions, never discovery.
|
||
Positions are main `(0,0)` and underlay/input `(group+1,input+1)`. Encode each
|
||
numeric component in fixed-width decimal sufficient for a Go index. All host
|
||
accesses within one scheduled tier share its host anchor; each native token
|
||
uses the smallest explicit input position selecting that namespace in that
|
||
tier. Same-label, separately constructed filesystems remain distinct. Use
|
||
tokens to identify equality, not to order nodes.
|
||
- Inheritance nodes: `("definition", scope-pointer, definition-pointer)` using
|
||
canonical escaped semantic pointers. Within a sibling scope these distinguish
|
||
exact names, including empty or escaped keys. No source ID participates.
|
||
|
||
The tuple comparator is shared; resolver-specific key construction is not a
|
||
second ordering algorithm. Unique keys are an internal construction requirement,
|
||
not a new user configuration restriction. Imports already reject repeated
|
||
canonical inclusion; separate synthetic roots have distinct explicit anchors.
|
||
|
||
For a stalled graph, deterministically traverse residual dependencies using
|
||
explicit DFS state, visiting nodes and neighbors in key order. Return an actual
|
||
stack cycle, rotated to begin at its smallest key, with the start repeated at
|
||
the end. Exclude blocked non-cycle dependents. Both resolvers translate these
|
||
node/edge identities to existing domain diagnostic codes and authored sites;
|
||
do not expose inheritance names or payload values in messages. Preserve import
|
||
cycle-site evidence. Build deterministic adjacency from precomputed node ranks
|
||
in linear edge work, rather than repeatedly sorting the ready set or scanning
|
||
all nodes for each completion. Scheduling costs O(E + V log V), O(V + E) memory,
|
||
apart from key construction/comparison and domain operations.
|
||
|
||
### Discovery, scopes, and assembly
|
||
|
||
Keep acquisition deterministic and separate from application. Enumerate
|
||
declarations in authored order; normalize/deduplicate a declaration's selected
|
||
candidate paths and visit them in namespace-path lexical order. Keep main-first
|
||
acquisition records, IDs, source-local version checking, and physical import
|
||
resolution. This discovery order does not determine sequence assembly order.
|
||
There is no new first-error or no-read-before-error guarantee.
|
||
|
||
Represent dependencies while acquiring files. Ordinary source-body nodes depend
|
||
on their imported contributions; importing bodies therefore assemble after
|
||
their dependencies. Record source roles explicitly: the main body must not be
|
||
identified as contribution index zero after scheduling. Keep all source IDs and
|
||
authored locations attached to contributions when reordering them.
|
||
|
||
Each main tier or underlay group has one outer strict-scope graph, including all
|
||
its independent input roots and ordinary imports. An item contribution is one
|
||
outer node keyed by its owning source. It owns a nested strict mapping-scope
|
||
graph containing that source body, ordinary helper contributions, and further
|
||
item nodes. Prevalidate the available scope schedules before applying their
|
||
operations. When the outer schedule selects an item node, execute its validated
|
||
local schedule, compose the item mapping, then place the completed item into
|
||
the outer scope. Do not globally flatten helper bodies into the collection.
|
||
Resolve local paths and project final indexes using that actual scheduled
|
||
contribution order. Main-item and underlay-item roots use the same model.
|
||
|
||
Retain the acquisition/inclusion graph independently of these composition scope
|
||
boundaries so cycles crossing item boundaries remain detectable. On an active
|
||
canonical repeat, retain the closing edge without rereading the source and use
|
||
the shared cycle logic on the available inclusion graph. Non-active repeated
|
||
inclusion still fails as `import.repeated`; no successful diamond import or
|
||
global source deduplication is introduced.
|
||
|
||
Within inheritance, construct the graph of direct mapping siblings and their
|
||
string parent edges, validate it, then produce templates in scheduled order.
|
||
Null cancels a relationship without an edge. Keep enclosing-before-nested
|
||
destination-local traversal and unresolved nested templates, as in ADR 0009.
|
||
Do not reorder output mappings merely to mirror scheduling.
|
||
|
||
Directory discovery keeps candidate spellings separate from canonical access.
|
||
One normalized candidate path per namespace is selected once per declaration;
|
||
aliases retain separate candidates and existing admission rules. Directory
|
||
filters run on discovered names. Exclusions never read content, but do not hide
|
||
discovery, containment, or resource failures. For required named paths, verify
|
||
every listed target exists even if excluded, without parsing excluded content.
|
||
|
||
Use existing bounded directory reads. For distinct roots, traverse each search
|
||
with its applicable depth once and test the union of filename patterns per
|
||
file; do not walk once per root/pattern pair. Overlapping roots may revisit
|
||
entries under the existing visit accounting, but selected candidates deduplicate.
|
||
Do not use global canonical-directory visitation to silently suppress aliases.
|
||
Charge selected-source allowance after exclusions and local deduplication;
|
||
discovery entries still count throughout, and excluded content costs no bytes.
|
||
|
||
Use host `Lstat`/root-relative operations for discovery controls and the standard
|
||
native `fs.ReadLinkFS` capability where available. Follow the supplied filesystem's
|
||
capabilities; do not require every `fs.FS` to implement link inspection or turn
|
||
opaque native behavior into a universal sandbox guarantee. Host directory cycle
|
||
keys are canonical paths; link-aware native traversal uses the corresponding
|
||
resolved namespace directory identity. This traversal identity must not silently
|
||
change the existing native source-admission identity. Track canonical directories
|
||
on the active traversal path, not all previously visited directories. Distinguish
|
||
directory traversal cycles from import dependency cycles; use `import.cycle`
|
||
with the declaring site and a safe directory-cycle message for the former.
|
||
|
||
## Stage 1 — Build the shared scheduler
|
||
|
||
Create the private scheduler, tuple comparator, min-heap, and deterministic
|
||
cycle finder specified above. Keep it independent of documents, filesystem
|
||
handles, diagnostics, and callbacks. Document Kahn's algorithm, actual-cycle
|
||
identification, and complexity beside its owner.
|
||
|
||
Tests own observable schedules: independent branches, unequal chains, multiple
|
||
dependencies, shared dependencies at equal/unequal depths, a newly ready small
|
||
key preceding an older large key, duplicate edges, self/multi-node cycles, and
|
||
blocked non-cycle dependents. Permute insertion order while retaining keys and
|
||
assert equal schedules/cycles. Do not assert heap internals.
|
||
|
||
Acceptance: focused scheduler tests pass; no loading behavior changes.
|
||
|
||
Completion: implemented in commit `167fcd4` (`Add deterministic dependency
|
||
scheduler`). Verified with `go test ./internal/engine -run
|
||
'^TestScheduleDependencies' -count=1`, `./scripts/check.sh`, and
|
||
`git diff --check`. Loading behavior remains unchanged; Stage 2 onward remains.
|
||
|
||
## Stage 2 — Separate inheritance graph analysis from application
|
||
|
||
In `inheritance.go`, extract direct-sibling relationship analysis from the
|
||
existing stack-driven application loop. Build keyed mapping nodes and edges;
|
||
retain null controls, missing/non-mapping parent diagnostics, exact directive
|
||
sites, and non-mapping siblings. Add a scheduled template-application helper
|
||
reusing current merge hooks, origins, retained-node accounting, and unresolved
|
||
nested template behavior. Keep the currently wired resolver unchanged until
|
||
Stage 3; do not expose an alternate public path.
|
||
|
||
Acceptance: private scope tests demonstrate the shared schedule, ordinary/null
|
||
relationships, real cycle sites, and preserved template values/ownership.
|
||
|
||
Completion: implemented in commit `2718fbb` (`Separate inheritance graph
|
||
analysis and application`). Verified with
|
||
`go test ./internal/engine -run 'Inheritance|Inherited' -count=1` and
|
||
`./scripts/check.sh`. The wired resolver retains parent-first ordering; Stage 3
|
||
will integrate the shared dependency-ready schedule.
|
||
|
||
## Stage 3 — Integrate scheduled inheritance
|
||
|
||
Replace the old sibling-frame resolution in the loading path with Stage 2's
|
||
analysis and application. Remove superseded stack/state machinery. Retain
|
||
`inheritance_walk.go`'s enclosing-before-nested scope discovery, sequence
|
||
traversal, final projection budgets, and immutable retained evidence.
|
||
|
||
Review inheritance history/control/transfer ordering against the selected
|
||
schedule, not the old DFS order. Add one unequal-branch integration case and
|
||
reuse existing nested destination, cancellation, sensitivity, version ownership,
|
||
and budget tests. Update the owning inheritance contract and internal overview
|
||
to describe implemented scheduling, linking ADR 0015.
|
||
|
||
Acceptance: inheritance and facade inheritance suites pass; final values,
|
||
presence, origins, and explanation agree; no alternate resolver remains.
|
||
|
||
Completion: implemented in commit `6e40c16` (`Schedule sibling inheritance by
|
||
dependency readiness`). Verified with engine inheritance and facade inheritance
|
||
tests, `./scripts/check.sh`, and `git diff --check`. Nested scope discovery,
|
||
projection budgets, and retained evidence remain in their existing owners.
|
||
|
||
## Stage 4 — Retain the import inclusion graph during acquisition
|
||
|
||
Refactor `imports.go` and `load_input.go` so acquired contributions carry stable
|
||
source identity, role, inclusion edges, and explicit scope ownership, rather
|
||
than treating their slice position as both acquisition and application order.
|
||
Separate item discovery from immediate item composition. Preserve parsed
|
||
payloads and source-version processing results without rereading sources.
|
||
|
||
Keep existing wired application order for this extraction stage. Introduce the
|
||
stable import keys and include-cycle integration with the shared cycle finder;
|
||
keep non-active repeats invalid, including aliases and different inclusion
|
||
routes. Do not expand successful shared dependency support for imports.
|
||
|
||
Acceptance: current ordinary/item import behavior tests pass, including a cycle
|
||
through a nested item scope, repeated imports, source IDs, and handle closure.
|
||
|
||
Completion: implemented in commit `fbe9ae6` (`Retain import inclusion graph
|
||
during acquisition`). Verified with `go test ./internal/engine -count=1`,
|
||
`./scripts/check.sh`, and `git diff --check`. Import acquisition retains stable
|
||
source keys, roles, inclusion edges, and explicit scope ownership; item
|
||
scopes are composed after discovery in the existing contribution order.
|
||
Non-active repeats remain invalid. Stage 5 onward remains.
|
||
|
||
## Stage 5 — Schedule ordinary main-tier contributions
|
||
|
||
Use the shared scheduler for ordinary mapping contributions in the main tier.
|
||
Plan before composing. The main body follows its import dependencies; independent
|
||
ready branches follow stable keys. Reuse `composeDocumentsWithPolicy` and the
|
||
existing merge primitives. Update the corresponding order tests deliberately.
|
||
|
||
Replace assumptions that main is document/contribution zero in
|
||
`importedVersionClaim`, source-root ownership, contexts, and history. Carry an
|
||
explicit main source ID/role and locate its scheduled contribution. Keep source
|
||
enumeration and version callbacks in acquisition order. Default-mode effective
|
||
version checking still happens after main-tier assembly.
|
||
|
||
Acceptance: ordinary import values, registered sequence order, conflict sites,
|
||
main-version protection, source records, and authored resource contexts agree.
|
||
Document this implemented ordering boundary in the import contract.
|
||
|
||
Completion: implemented in commit `5ed0f55` (`Schedule ordinary import
|
||
contributions by dependency`). Verified with `go test ./internal/engine
|
||
-count=1`, `go test . -run
|
||
'^TestFacadeSourceVersionEvidenceFollowsAssembledSequenceRanges$' -count=1`,
|
||
`./scripts/check.sh`, and `git diff --check`. Acquisition records and version
|
||
callbacks remain in traversal order; ordinary contributions use the shared
|
||
dependency-ready schedule, with main-source ownership and projected evidence
|
||
following the scheduled contributions. Item-scope ordering remains for later
|
||
work.
|
||
|
||
## Stage 6 — Assemble item-local scopes under the scheduler
|
||
|
||
Integrate item nodes and local schedules for imported and selected main items.
|
||
Complete helpers before local item assembly and outer placement. Reuse existing
|
||
`projectNestedEvidence`, destination projection, sequence ranges, and diagnostic
|
||
projection, correcting any assumptions about contribution order or index zero.
|
||
Do not register application collection policies inside helper mapping scopes.
|
||
|
||
Keep local version checking/consumption before assembly and project retained
|
||
source evidence through the actual assigned indexes. Failed-load source paths
|
||
retain the existing unresolved-container semantics until indexes are known;
|
||
do not invent an exact index to simplify a failure.
|
||
|
||
Acceptance: nested items, helper-supplied IDs, main items, multiple unequal
|
||
branches, physical coordinates, sensitivity, history chronology, and item
|
||
conflict paths pass engine and public tests. Run the focused race suite.
|
||
|
||
Completion: implemented in commit `af43d6f` (`Schedule item-local import
|
||
scopes`). Verified with `go test ./internal/engine -count=1`,
|
||
`go test -race ./internal/engine`, `go test . -count=1`, `go vet ./...`,
|
||
`go build ./...`, the local documentation-link test, and `git diff --check`.
|
||
`./scripts/check.sh` reaches the consumer examples and currently fails only on
|
||
prompt-catalog ordering expectations assigned to Stage 18. Stage 7 onward
|
||
remains.
|
||
|
||
## Stage 7 — Schedule underlay forests and preserve tier boundaries
|
||
|
||
In `load_request.go`, acquire all roots in a strict underlay group before
|
||
scheduled group assembly. Combine their retained scope graphs rather than
|
||
concatenating already composed per-input orders. Assign namespace anchors from
|
||
explicit input positions. Include inputs without imports and item roots.
|
||
|
||
Keep per-tier repeated-source state, accounting, main acquisition timing, and
|
||
underlay precedence unchanged. Independent groups never enter one scheduler;
|
||
overlays/CLI remain later precedence operations. Maintain source layers,
|
||
underlay indexes, global source IDs, and group-wide item path projections.
|
||
|
||
Acceptance: a group with interleaving dependency branches across input roots
|
||
uses the expected schedule; separate tiers still replace sequences wholesale.
|
||
Test same-label distinct native namespaces and copies of one namespace. NoSource
|
||
still contributes no source or explicit empty root.
|
||
|
||
Completion: implemented in commit `014b995` (`Schedule underlay groups as
|
||
dependency forests`). Verified with `go test ./internal/engine -count=1`,
|
||
`go test . -count=1`, `go test -race ./internal/engine`, `go vet ./...`,
|
||
`go build ./...`, the local documentation-link test, and `git diff --check`.
|
||
`./scripts/check.sh` still reaches only the two prompt-catalog ordering
|
||
expectations assigned to Stage 18. Stage 8 onward remains.
|
||
|
||
## Stage 8 — Define private defaults and the final declaration parser
|
||
|
||
Add the private defaults snapshot/validation and resolved-policy helpers. Add
|
||
the final declaration parser as an unwired private helper until Stage 14; retain
|
||
the current wired parser only until that cutover. Store declaration paths and
|
||
field locations for exact diagnostics and later no-selection evidence.
|
||
|
||
Cover the roadmap's selector/option matrix: scalar/list normalization, exactly
|
||
one selector, unknown fields including removed `optional`, null/wrong-type
|
||
members, empty lists, duplicates, wildcard validation, literal destinations,
|
||
placement, and effective-recursion/depth applicability. Parse all declarations
|
||
before selecting candidates; invalid inactive fields still fail.
|
||
|
||
Acceptance: parser/policy tests cover omitted, explicit false, explicit empty,
|
||
application defaults, and invalid default patterns without source access. No
|
||
new YAML syntax is accepted by public loading yet.
|
||
|
||
Completion: implemented in commit `b498e5b` (`Define private import defaults
|
||
and declaration parsing`). Verified with focused parser/policy tests, the full
|
||
engine suite, `go vet ./...`, `go build ./...`, a focused engine race run, and
|
||
`git diff --check`. The everyday check still reaches the two known consumer
|
||
sequence-order expectation failures scheduled for Stage 18; public loading
|
||
continues to use the existing parser. Stage 9 onward remains.
|
||
|
||
## Stage 9 — Implement named-file selection and exclusions
|
||
|
||
Build a private selector result containing normalized lexical candidates and
|
||
selection outcome. Implement string/list `path`, local candidate deduplication,
|
||
basename exclusions, and optional/required presence semantics. Check excluded
|
||
required target existence without opening content. Do not treat permission,
|
||
containment, regular-file admission, or parser failures as optional absence.
|
||
|
||
Keep selector results distinct from source-admission identities; no source IDs
|
||
or read counts are assigned merely by selection. Preserve current validity rules
|
||
for explicit paths and existing source-access behavior for named symlinks and
|
||
hidden targets. Do not apply search-only controls to this selector.
|
||
|
||
Acceptance: temporary-file tests cover partial missing lists, required/excluded
|
||
missing targets, explicit empty lists, alias candidates, and unread excluded
|
||
content. No fake source records are created for absence.
|
||
|
||
Completion: implemented in commit `5f33e1e` (`Select named imports before source
|
||
admission`). Verified with named-selection tests using temporary host files and
|
||
a stat-only native filesystem, the full engine suite, a focused race run,
|
||
`go vet ./...`, `go build ./...`, and `git diff --check`. The everyday check
|
||
continues to fail only on the two consumer sequence-order expectations assigned
|
||
to Stage 18. Stage 10 onward remains.
|
||
|
||
## Stage 10 — Implement shallow directory selection
|
||
|
||
Add literal directory string/list search, scalar/list basename patterns, and
|
||
exclusions through the same candidate-selection result. Start with nonrecursive
|
||
selection. Missing roots contribute nothing; an inaccessible or invalid supplied
|
||
root is not absence. Directories are traversal containers, not documents.
|
||
|
||
Implement set union and difference, normalized candidate-path deduplication,
|
||
and requiredness across the complete selected union. Preserve nil/unrestricted
|
||
versus explicit-empty patterns and alias spelling during filtering. Test multiple
|
||
roots, overlapping roots/patterns, extensionless and hidden filenames, and a
|
||
missing root alongside one that supplies the required match.
|
||
|
||
Acceptance: shallow directory tests pass with actual host files and a native
|
||
filesystem; no per-root/per-pattern requiredness is introduced.
|
||
|
||
Completion: implemented in commit `55a59de` (`Select shallow directory import
|
||
sources`). Verified with shallow-selection tests on host and native filesystems,
|
||
the full engine suite, a focused race run, `go vet ./...`, `go build ./...`, and
|
||
`git diff --check`. The everyday check still reaches only the consumer
|
||
sequence-order expectations reserved for Stage 18. Stage 11 onward remains.
|
||
|
||
## Stage 11 — Add recursive directory traversal and depth accounting
|
||
|
||
Extend the walker with explicit traversal frames, effective recursion, and the
|
||
settled depth convention. Apply filename patterns only to files, never to prune
|
||
directories. Share bounded directory-reading and entry accounting with glob
|
||
discovery. Do not rescan a root per pattern or use the import-edge depth limit
|
||
as a directory-recursion limit.
|
||
|
||
Acceptance: direct files are depth 1, depth 2 includes one descendant level,
|
||
zero/omission are unlimited subject to existing budgets, and depth requires
|
||
effective recursion. Cover application-enabled recursion, explicit false,
|
||
large depth integers, empty searches, and limits using test-controlled budgets.
|
||
|
||
Completion: implemented in commit `77d9729` (`Add recursive directory search
|
||
with depth limits`). Verified with recursive depth/limit tests, existing glob
|
||
tests, the full engine suite and race suite, `go vet ./...`, `go build ./...`,
|
||
and `git diff --check`. The everyday check still reaches only the consumer
|
||
sequence-order expectations reserved for Stage 18. Stage 12 onward remains.
|
||
|
||
## Stage 12 — Add discovery symlink and hidden controls
|
||
|
||
Implement `follow`/`hidden` for directory discovery using capability-aware
|
||
filesystem helpers. Explicit roots keep existing access semantics; discovered
|
||
hidden directories prune when hidden is false, discovered symlink directories
|
||
do not descend when follow is false, and analogous files are not selected.
|
||
|
||
When following links, track active-path directory identities and fail actual
|
||
loops without treating earlier completed branches as cycles. Preserve namespace
|
||
containment and candidate spellings. Cover contained links, escapes, filename
|
||
filtering on aliases, self/ancestor loops, and separate noncyclic aliases that
|
||
later retain ordinary repeated-source admission behavior.
|
||
|
||
Acceptance: host filesystem behavior and a focused link-aware native fixture
|
||
pass; plain native filesystems still load without new capability requirements.
|
||
All success/failure paths close library-owned directory and root handles.
|
||
|
||
## Stage 13 — Integrate glob filtering and final candidate accounting
|
||
|
||
Adapt `glob.go` to the common selection pipeline. Keep path wildcards and
|
||
literal components distinct so follow/hidden filters affect only wildcard-
|
||
discovered entries, not explicitly named components. Apply basename exclusions
|
||
to final candidates. Preserve regular-file/source validity behavior rather than
|
||
silently accepting invalid glob-selected sources.
|
||
|
||
Move source-allowance checking after exclusions and declaration-local
|
||
deduplication. Intermediate traversal candidates are bounded by entry visits,
|
||
not by the final source allowance. Charge bytes only when selected content is
|
||
read; never read excluded content. A tiny private source limit with many excluded
|
||
positive matches must succeed when its final selected set fits.
|
||
|
||
Acceptance: existing glob syntax, containment, alias, and visit-budget tests
|
||
pass alongside discovery-control and post-filter source-limit regressions.
|
||
|
||
## Stage 14 — Wire final syntax and load-wide defaults through the engine
|
||
|
||
Add the private defaults value to ordinary/grouped requests, snapshot/validate
|
||
it before acquisition, and pass it to every enabled main/underlay/import
|
||
declaration. Wire Stage 8's parser and Stages 9–13's selection pipeline into
|
||
graph acquisition. Remove the old declaration parser and `optional` field;
|
||
there is one final parser and one absence policy.
|
||
|
||
Include `source_version.go`'s stored import declarations in this wiring: syntax
|
||
parsing stays independent of source-version interpretation, and every stored
|
||
declaration receives the same effective policy before discovery. Do not bypass
|
||
validation when a consumed header or empty payload leaves no application fields.
|
||
|
||
Keep imports disabled unless already enabled by the consumer. Selected overlays
|
||
still cannot enable their own imports; defaults do not change that boundary.
|
||
Validate effective placement/depth and declaration values before relying on a
|
||
missing target or empty selection to skip work. Declarations do not inherit
|
||
parent control values.
|
||
|
||
Migrate existing tests and maintained fixture declarations from `optional` to
|
||
the intended `required` behavior. Tests expecting a missing import to fail must
|
||
request requiredness, not weaken their checks. Update the owning configuration
|
||
contract to describe the now-implemented final YAML interface and breaking
|
||
absence/order changes. Keep exact public Go defaults documentation for Stage 16.
|
||
|
||
Acceptance: all engine imports, glob, destinations, source-version, and layered
|
||
tests pass under final syntax. An explicit regression rejects `optional`;
|
||
another proves missing imports are optional without setup. Run everyday checks.
|
||
|
||
## Stage 15 — Retain no-selection evidence across load outcomes
|
||
|
||
Add the private records from the settled inspection design to declaring source
|
||
records and their clone functions. Record only completed, optional, whole-
|
||
declaration empty outcomes. Preserve evidence for main and imported declarations
|
||
and underlay inputs through composition, nested item placement, overlays,
|
||
resolution entry state, final publication, and failed-load accumulation.
|
||
|
||
Use source IDs to attach outcomes, not scheduled positions. Reuse source-record
|
||
transport; do not add synthetic imported sources or warnings. Keep evidence
|
||
source-local and detached even when its source destination remains unresolved.
|
||
|
||
Acceptance: tests cover no matches, missing paths, empty lists, exclusions,
|
||
partial path-list omission (no record), and required empty paths (no record).
|
||
An error after a prior empty optional declaration retains that established
|
||
evidence, but does not fabricate later outcomes. Mutating query results cannot
|
||
alter subsequent snapshots or another load.
|
||
|
||
## Stage 16 — Expose defaults and inspection through the public facade
|
||
|
||
Add root-owned declarations from this plan, `ImportDefaults` fields on ordinary
|
||
and layered options, and `SourceInfo.ImportSelections`. Add explicit converters
|
||
and engine transport types/helpers as needed, following existing root-wrapper
|
||
ownership. Snapshot pointers/slices and preserve nil versus non-nil empty
|
||
patterns. Do not add defaults on individual inputs or a new result query API.
|
||
|
||
Exercise final behavior through `Load` and `LoadLayered`, including defaults
|
||
applied to transitive and underlay imports, explicit declaration exceptions,
|
||
invalid default preflight without reader consumption, and result/error/resolver
|
||
source inspection ownership. Update declaration comments and the owning Go API
|
||
guidance when the surface becomes available.
|
||
|
||
Acceptance: public facade/import-boundary tests and representative engine/public
|
||
equivalence cases pass; no internal type leaks through exported signatures.
|
||
|
||
## Stage 17 — Audit cross-feature lifecycle and ownership
|
||
|
||
Review and fill meaningful gaps rather than adding a full Cartesian-product
|
||
suite. Reuse existing regressions for version protection, consumed-header
|
||
sensitivity, scope diagnostics, NoSource, and historical removed descendants.
|
||
|
||
Add representative combined cases for scheduled discovered items with helper-
|
||
supplied IDs; nested destinations and actual assigned indexes; required versus
|
||
if-present source versions; main-version ownership after main moves later in a
|
||
schedule; inherited relative resources; superseded relationships; and final
|
||
configuration/metadata/explanation agreement. Ensure consumed headers cannot
|
||
silently supply record IDs and secret values remain absent from diagnostics
|
||
and inspection. Preserve callback phases and source-local callback ordering.
|
||
|
||
Compare an equivalent keyed graph through import and inheritance integrations:
|
||
both use the scheduler's eligible-node rule despite different domain operations.
|
||
Keep acquisition-source records stable when only graph construction order is
|
||
permuted in private scheduler/integration fixtures; do not promise unchanged IDs
|
||
when authored discovery order itself changes.
|
||
|
||
Acceptance: run `go test -race ./internal/engine`, focused root facade tests,
|
||
and everyday checks. Record any genuine unresolved gap before marking complete.
|
||
|
||
## Stage 18 — Maintain simple and complex consumer examples
|
||
|
||
Update the prompt-catalog example for directory search, multiple patterns,
|
||
exclusions, and helper-supplied IDs while retaining explicit registration and
|
||
application final validation. Include application import defaults and a
|
||
declaration exception, plus inspection of a deliberately empty optional search.
|
||
Keep a short simple single-file/required-file example so adoption does not
|
||
require understanding all discovery controls.
|
||
|
||
Update fixture expectations for scheduled sequence positions. Verify mounted
|
||
resource origins, whole-sequence precedence, and quiet absence through executable
|
||
examples. Do not add a consumer-side discovery, merge, scheduling, or provenance
|
||
reconstruction pass. All fixtures remain local, deterministic, and secret-free.
|
||
|
||
Acceptance: consumer tests and executable public examples pass; the consumer
|
||
guide links complete copyable examples rather than duplicating them.
|
||
|
||
## Stage 19 — Finalize documentation and validate the completed unit
|
||
|
||
Align README orientation, `docs/config.md`, public declaration comments,
|
||
`docs/internal/overview.md`, relevant focused internal docs, and the development
|
||
reading/test map. Document Kahn's algorithm, set union/difference, and active-path
|
||
cycle detection beside their owners. State acquisition versus resolution order,
|
||
explicit defaults precedence, no-selection evidence, and source namespace/key
|
||
construction without duplicating canonical contracts.
|
||
|
||
Update architecture policy to recognize implemented explicit directory discovery
|
||
and shared scheduling while retaining the boundary against unrequested implicit
|
||
application-path discovery. Update only ADR status/navigation as needed to link
|
||
implemented contracts; accepted decision text stays unchanged. Leave release
|
||
notes/version choice and roadmap retirement for separately authorized release
|
||
preparation. Do not introduce release gates, CI waiting, consumer validation
|
||
requirements, dependencies, or unrelated defaults/security changes.
|
||
|
||
Run the everyday `./scripts/check.sh`, `go test -race ./...`, consumer/examples
|
||
tests, local documentation links, and `git diff --check`; reuse checks already
|
||
covering unchanged final content instead of repeating them gratuitously. Review
|
||
the baseline-to-final public and YAML diff for removed `optional`, changed
|
||
requiredness and sequence ordering, and accurately documented native filesystem
|
||
capability limits. Inspect source and history indexes for accidental quadratic
|
||
rescanning; no timing thresholds or busy-host benchmark evidence are required.
|
||
|
||
Acceptance: all stages are complete, the final implemented behavior matches the
|
||
roadmap and ADRs, validations are recorded here, and the implementation workspace
|
||
is committed and clean. No release is published by this plan.
|
||
|
||
## Open Questions
|
||
|
||
None required to begin implementation. The API, evidence shape, ordering keys,
|
||
scope boundaries, and cycle treatment are selected above. Private helper names
|
||
and file splits may be adjusted for clarity without changing those decisions.
|
||
If implementation reveals a genuine conflict between an accepted requirement
|
||
and existing contracts, report it rather than silently strengthening or weakening
|
||
either one.
|