diff --git a/docs/roadmap/main-relative-inputs.md b/docs/roadmap/main-relative-inputs.md new file mode 100644 index 0000000..00f42c4 --- /dev/null +++ b/docs/roadmap/main-relative-inputs.md @@ -0,0 +1,286 @@ +# Main-relative underlays and overlays + +Status: accepted direction; API and filesystem-policy decisions remain open. +Not implemented. Planning baseline: `191015e`. + +## Purpose + +Allow an application to select its main configuration through ConfigKit's +existing discovery descriptions, then load explicitly declared underlay files +and a selected overlay relative to that main file. Applications should not need +to repeat discovery themselves or run a preliminary configuration load to learn +the selected directory. + +The application continues to choose its search conventions, underlay tiers, and +overlay name. ConfigKit resolves their filesystem relationship within the same +load. Independently located inputs remain useful and can be combined with +main-relative inputs. + +## Current baseline + +The [discovery contract](../config.md#configuration-discovery) resolves a +`SearchConfig` source inside loading. It selects one host file or absent input; +errors processing the selected file do not resume the search. The public +constructor returns a source description, not a selected filename. + +Ordinary `Load` already derives an overlay context from a selected file. Without +an explicit root override, selecting `/etc/example/config.yml` permits overlay +`production` to load from `/etc/example/overlays/production.yml`. Optional search +exhaustion supplies no inferred root; ordinary loading then needs an explicit +context for an active overlay. + +The [layered API](../config.md#layered-loading) differs: + +- An active overlay requires `LayeredOptions.OverlayFilesystem` independently of + the main source. Grouped preflight rejects its absence before discovery runs. +- `Underlay.Inputs` contains independently specified sources and contexts. A + file source can establish its own context, but no declaration binds an + underlay filename to the main file that discovery will select. +- Main acquisition already precedes underlay acquisition. Underlay groups are + subsequently composed from lowest to highest priority, beneath the main tier. +- Ordinary relative imports are peers of their declaring input. They do not + express lower-priority underlay tiers. + +The extension can therefore reuse the selected main's acquired filesystem +information and the existing grouped lifecycle. It needs a way to express the +relationship publicly and bind it before dependent acquisition. + +## Targeted end state + +### Application declarations + +An application can declare, before loading: + +1. A main `Input` using `SearchConfig`, with any supported combination of search + presets and custom candidates. +2. One or more underlay files whose paths are relative to the selected main + file's directory, grouped and ordered through the existing underlay API. +3. A selected overlay whose context is derived from the selected main file. + +The relationship is explicit and declarative. It does not require a new +application callback, a second load, or a public intermediate configuration. +Exact API spelling and context selection are subject to OD-1. Missing-main +behavior, filesystem policy, and eligible main sources are subject to OD-2 +through OD-4; the examples below assume a successfully selected host file. + +Existing input destinations, placement, import opt-in, and load-wide policies +remain applicable to underlay inputs. The new relationship determines where an +input is acquired; it does not change its precedence or composition rules. +An underlay's own relative imports continue to resolve from that underlay's +declaring directory. + +### Examples + +These describe intended behavior, not implemented Go syntax. Assume there is no +explicit configuration-root override and that the selected main file is: + +```text +/etc/example/config.yml +``` + +| Application declaration | Selected input | +| --- | --- | +| Main-relative underlay `defaults/base.yml` | `/etc/example/defaults/base.yml` | +| Main-relative underlay `defaults/site.yml` | `/etc/example/defaults/site.yml` | +| Main-relative overlay context, overlay name `production` | `/etc/example/overlays/production.yml` | + +If the underlays are placed in separate groups in that order, effective priority +is typed defaults, base underlay, site underlay, main, overlay, then CLI. +Putting both underlay inputs in one group instead composes them strictly as +peers. A declaration of `extras.yml` inside `defaults/base.yml` resolves beside +that file, at `/etc/example/defaults/extras.yml`. + +If discovery instead selects a file in the user's configuration directory, the +same relative declarations follow that file's directory. An embedded or otherwise +independent underlay may coexist with these inputs and retains its own namespace +and context. No directory is searched for underlays merely because it exists. + +### Loading and inspection + +Resolve the main-relative descriptions using the selected main's acquired +filesystem information, before acquiring the inputs that depend on it. Keep +discovery within the existing single loading lifecycle and preserve main-first +acquisition, underlay tier order, version processing, selected-overlay timing, +and the existing defaults, resolution, secret, and validation phases. + +Static declaration errors can still be checked before source acquisition; +checks that need the selected directory must run after it is available. The +implementation plan should identify this division without imposing a new global +ordering guarantee for unrelated diagnostics or filesystem checks. + +After binding, use ordinary source acquisition, parsing, import handling, and +overlay loading. A missing required input, unreadable selected file, or invalid +configuration follows the applicable existing failure rules; it does not cause +main discovery to try another candidate. This feature does not itself introduce +optional-file syntax, underlay globbing, additional overlay filenames, or a new +overlay stack. + +Acquired files retain their actual source identity, path, source-local locations, +layer, underlay index, and passive resource context. Do not create an extra +source or precedence tier for the relative declaration itself. Preserve existing +namespace identity, repetition scope, shared acquisition budgets, detached +inspection, and resource cleanup. Use ordinary diagnostic classifications where +they fit; decide whether any new classification is needed during API design. + +### Compatibility and scope + +Preserve successful existing ordinary and layered loading behavior. Main-relative +selection should be additive and opt-in, with independently supplied sources and +contexts continuing to work. Preserve existing named-overlay lookup semantics; +OD-3 determines the root used by the new context selection. + +The work does not add general resource-path resolution, changes to process working +directory, remote acquisition, or a generic source-provider callback framework. +These are scope boundaries for this feature, not new architectural exclusions. +There is no new filesystem race-protection or sandbox guarantee. + +The design follows [ADR 0019](../adr/0019-explicit-configuration-discovery.md): +applications select conventions while ConfigKit performs selection and loading. +Review whether the finalized context-ownership decision needs an additional ADR; +do not rewrite an accepted ADR's decision content. No release version or new +release gate is selected by this roadmap. + +## Work from the baseline + +1. **Settle the public declaration and filesystem policies.** Resolve the open + decisions below, document combinations with explicit contexts, and prepare + representative API examples. Keep public types owned by the root package. +2. **Represent the relationship through the facade and request preparation.** + Update the relevant declarations and conversions in `layered.go`, + `lifecycle.go`, or `source.go` as the selected API requires. Carry the intent + through the private request without prematurely interpreting paths against + the working directory. Preserve reusable-description and input-ownership + conventions. +3. **Bind dependent inputs during acquisition.** Extend the request preparation + and acquisition boundary in `internal/engine/load_request.go`, + `load_input.go`, and `acquisition.go`. Reuse discovery's selected concrete + source and filesystem-context resolution. Establish which checks remain in + preflight and which require resolved main information. +4. **Integrate underlays and overlays.** Resolve relative underlay files into + ordinary acquisitions with their own import bases; derive the selected + overlay context and reuse its loader. Preserve independent input contexts, + namespace handling, cleanup on failure, and source/error evidence. +5. **Verify observable behavior and update documentation.** Add focused public + coverage and maintained consumer examples, then update the discovery, + layered-loading, and selected-overlay contracts, declaration comments, and + relevant internal ownership documentation. Explain the difference between a + file's directory and the configured root when they differ. + +This is dependency order for a later implementation plan, not a required commit +layout or a requirement to expose intermediate stages to consumers. Reuse the +existing lifecycle; a new discovery pass or a second configuration load is not +needed. + +## Validation and completion + +Exercise the behavior through the public API, supported by focused engine tests: + +- Main discovery selects an early or later candidate; the dependent files follow + the actual selection, including when it is outside the working directory. +- Relative and independent underlays coexist; multiple groups retain precedence, + and peers in one group retain strict-composition conflicts. +- Underlay imports use their own file's directory. Placement and source-version + policies remain effective after binding. +- Active and inactive overlays, explicit contexts, optional search exhaustion, + and required-search failures follow the settled policies. +- Symlinked main files, explicit root overrides, parent-relative paths, and + namespace boundaries follow OD-3 and OD-4 without broadening existing access + guarantees. +- Selected-file failures and dependent acquisition failures are terminal; no + repeated full load or duplicate defaults/resolution/validation callbacks are + introduced. +- Source records, underlay indexes, origins, passive contexts, and failed-load + evidence identify the actual acquired inputs. Existing handle cleanup and + budget accounting still cover dependent acquisitions. +- Existing ordinary discovery and overlay loading, independent layered inputs, + and reuse of descriptions across loads retain their behavior. + +Run focused checks during implementation and the repository's applicable normal +checks before completion. No benchmark campaign or additional release gate is +required for this feature. Completion means applications can demonstrate the +discovery-plus-relative-input workflow in one load, with documented public +declarations and verified failure behavior. + +## Open Decisions + +### OD-1: Public declaration shape and explicit-context combinations + +**Recommendation:** add declarative, explicit choices for a main-relative +underlay file and a main-derived overlay context. A dedicated underlay source +constructor and a small overlay context-selection option are plausible shapes; +names remain to be chosen. Keep the existing independent-source/context path as +the default. For overlays, an explicitly supplied `OverlayFilesystem` should +take precedence over a request to derive context, allowing application overrides +without changing shared setup. Specify analogous precedence for any permitted +underlay context overrides. This keeps common use short without making nil +contexts change meaning or introducing callback lifetime concerns. + +**Alternative:** add a general deferred filesystem-context descriptor shared by +inputs and overlays. This could give context selection one uniform representation +and accommodate later relationships, but risks expanding a small feature into +a general context API. It is attractive if concrete API sketches show less +duplication than separate declarations. Rejecting simultaneous explicit and +derived overlay choices is also viable: it catches ambiguous setup, but requires +callers to clear the derived choice when supplying an override. + +### OD-2: No main file is selected + +**Recommendation:** when an active dependent declaration needs a main-file base +and none exists, fail with a clear diagnostic. Do not use an attempted candidate +directory or the working directory as an implicit substitute. This makes an +unresolvable relationship explicit without making discovery globally required. +Applications can require the main search or use independent inputs and an +explicit overlay context when absent-main operation is intended. Inactive +overlays do not require a derived context. + +**Alternative:** treat main-relative inputs as conditional on main presence and +skip them after optional exhaustion. This supports defaults-only startup and is +consistent with a permissive search, but could silently omit an explicitly +selected overlay or expected underlay. A selectable skip policy could support +that use case, at the cost of another option; adopting it would require defining +its effect on tier evidence and distinguishing absent main from a missing file +beside an existing main. + +### OD-3: Relative base, configuration root, and symlinks + +**Recommendation:** anchor underlay filenames to the canonical selected main +file's directory, consistent with existing file/import behavior. Derive overlay +context from the main's resolved context, so overlay lookup remains relative to +its configuration root. When an explicit main root is wider than the main +directory, underlay filenames still use the directory as their base and overlays +use that root. Inherit the main's root for dependent underlay acquisition and +imports unless an explicitly supported context override selects another root. +Each acquired underlay supplies its own import base. Apply existing canonical +containment checks; allow parent-relative traversal when it stays within the +selected root rather than adding a blanket prohibition on `..`. + +This distinguishes lookup from containment, reuses existing machinery, and +supports sibling directories under an explicitly wider root. It also means a +symlinked main selects resources beside its target, which must be documented. + +**Alternative:** always create a fresh root at the main file's directory for +both kinds of input. This makes every main-relative lookup visually local, +including with a wider main root, but discards intentional root configuration +and prevents some sibling-directory layouts. Using the main file's lexical +symlink directory instead is another viable convention for deployment bundles; +it would differ from current canonical import behavior and require separate +handling for lookup and containment. + +### OD-4: Eligible main sources and namespaces + +**Recommendation:** support discovered and explicit host files, and native +`Filesystem.File` mains, using each namespace's existing path rules and resource +ownership. Discovery currently selects host files; a native file should not +require discovery merely to provide the same directory relationship. Bind paths +within the selected namespace and reuse its identity. For this initial feature, +byte, reader, mapping, and absent mains do not supply a main-file directory; +independently specified contexts remain available for those workflows. + +**Alternative:** derive from any resolved main filesystem context, including an +explicit `BaseDir` for non-file inputs. This is more uniform for context-based +loading and avoids an artificial file requirement when a base is already known, +but changes the concept from relative-to-main-file to relative-to-main-context. +It needs explicit rules for absent mains and root-only contexts, where root and +base are not interchangeable. Limiting the first implementation to host files +is also viable and smaller, but would leave native-filesystem consumers with +the same coordination gap.