ConfigKit/docs/roadmap/discovery.md

# Composable Configuration Discovery Roadmap

Status: implemented. This document preserves the design rationale and agreed
direction. The [configuration contract](../config.md#configuration-discovery)
owns current behavior, [ADR 0019](../adr/0019-explicit-configuration-discovery.md)
records the decision, and the [implementation plan](implementation.md) records
the completed work sequence.

## Purpose And Ownership

Make ordinary file-based loading concise for applications that want conventional
configuration locations. ConfigKit supplies useful built-in candidate lists and
an ordered selection operation. Consumers continue to own main-file search
conventions: they choose which lists apply, their order, custom candidates, and
whether a configuration file is required.

A consumer can select one preset for a simple application or compose several
presets and custom path lists. The application supplies an explicit slug where
needed to construct conventional paths. ConfigKit does not infer application
identity from the executable name or enable discovery for existing explicit
sources.

Discovery selects one source for the existing loading lifecycle. It does not
create additional configuration tiers or merge every matching file. Existing
layered loading remains available when consumers intentionally combine sources.

## Baseline Before Implementation

Before this work, sources included explicit host files, native filesystem files,
bytes, readers, mappings, and `NoSource()`, with no conventional main-file search.
That baseline also restricted `NoSource()` with imports and overlays. The
implemented discovery behavior now lets an exhausted optional search follow the
absent-main lifecycle, including enabled imports and an active overlay when its
required explicit context is supplied. The current [source contract](../config.md#sources),
[single-source loading](../config.md#single-source-loading), and
[layered loading](../config.md#layered-loading) sections define the behavior;
this baseline is retained only to explain the original design problem.

Applications still choose search conventions and ordering. The architecture
policy now distinguishes those consumer choices from explicitly selected search
descriptions resolved during loading, while continuing to exclude implicit
discovery and network acquisition.

## Agreed Direction

### Filename Groups, Directory Searches, And Search Presets

Organize discovery around three composable descriptions. These are ordered lists:
their order determines precedence.

**Filename groups** provide reusable, ordered filename templates. The built-in
groups are:

| Group | Filenames, in order |
| --- | --- |
| Named | `<appname>.yml`, `<appname>.yaml` |
| Generic | `config.yml`, `config.yaml`, `main.yml`, `main.yaml` |

`<appname>` is the application slug supplied by the consumer.

**Directory searches** pair a directory with an ordered list of filename groups.
The default selection is `[Named, Generic]`, producing the six filenames above
in that order. A directory search may select different groups or reverse their
precedence, such as `[Generic, Named]`, while retaining each group's internal
order. Consumers can also supply custom filenames, subject to the literal
single-component rules under OD-5 below.

**Search presets** bundle ordered directory searches. Consumers can select a
complete preset for the simple case or compose presets, custom directory searches,
and explicit file candidates in their desired order. Explicit file candidates
are literal paths and receive no filename expansion.

### Ordered Expansion And Concatenation

The selection model is **ordered expansion and concatenation**, followed by
first-match selection:

1. Expand the selected presets into directory searches in their supplied order.
2. For each directory search, concatenate its selected filename groups in order,
   substitute the application slug where needed, and join each filename to that
   directory.
3. Concatenate the resulting candidate lists, preserving the positions of custom
   directory searches and explicit file candidates.
4. Try candidates in order, stopping at the first selected file or an error as
   described below.

Directory priority comes before filename priority: exhaust one directory's
filenames before trying the next directory. When several directories share a
filename list, their expansion is a Cartesian product traversed directory first.
Different directories may select different lists, so ordered expansion and
concatenation describes the complete model. Set union does not express its
precedence semantics.

The [configuration contract](../config.md#configuration-discovery) explains this
model using ordered lists, ordered concatenation, and the directory-first
Cartesian product where a shared filename list applies. These terms describe
ordering without requiring eager construction of every candidate.

### Unix System Preset

The agreed Unix system preset is:

| Directory, in search order | Filename groups, in order |
| --- | --- |
| `/usr/local/etc/<appname>` | `[Named, Generic]` |
| `/etc/<appname>` | `[Named, Generic]` |
| `/etc` | `[Named]` |

This produces 14 candidates: six in each application-specific directory, then
`/etc/<appname>.yml` and `/etc/<appname>.yaml`. For example,
`/usr/local/etc/<appname>/main.yaml` precedes
`/etc/<appname>/<appname>.yml`. Selecting only `Named` for `/etc` is an ordinary
directory-search description; the selection algorithm needs no special exception.

### Additional Presets (OD-1 — Settled)

Provide the following independently selectable presets alongside Unix system.
Within each preset, search directories in the order shown:

| Preset | Directory | Filename groups, in order |
| --- | --- | --- |
| User configuration | `<user-config-dir>/<appname>` | `[Named, Generic]` |
| User configuration | `<user-config-dir>` | `[Named]` |
| Home directory | `<home>/.<appname>` | `[Named, Generic]` |
| Home directory | `<home>` | `[Named]` |
| Working directory | Current working directory | `[Named]` |

Resolve the user-configuration base with
[`os.UserConfigDir`](https://pkg.go.dev/os#UserConfigDir), the home base with
[`os.UserHomeDir`](https://pkg.go.dev/os#UserHomeDir), and the working-directory
base with [`os.Getwd`](https://pkg.go.dev/os#Getwd). Adopt the Go helpers'
platform-specific directory and environment behavior, including their errors;
the settled OD-2 rules below govern how discovery handles resolution errors.

The goal is to support familiar conventions through idiomatic Go. This feature
does not add a separate XDG compatibility layer or XDG system preset. A selected
user-configuration base has no additional hard-coded `<home>/.config` fallback.
The Home directory preset is a separate, explicit choice.

The working-directory preset searches only that directory and uses application-named
files; it does not walk ancestor directories. The consumer chooses precedence
between all presets. Selecting one does not automatically select any others.

### Illustrative Consumer API

Names below are illustrative and are not implemented Go API. The
[implementation plan](implementation.md#public-api) fixes the concrete signatures
and the `Searches` constructor for concatenating several searches:

```go
// A simple system-wide application.
source := configkit.SearchConfig("myapp", configkit.UnixSystemDirectories())
```

```go
// A consumer-selected working-directory, user, then system search.
source := configkit.SearchConfig(
    "myapp",
    configkit.Searches(
        configkit.WorkingDirectory(),
        configkit.UserConfigDirectory(),
        configkit.UnixSystemDirectories(),
    ),
)
```

```go
// Explicit custom candidates followed by a selected conventional fallback.
source := configkit.SearchConfig(
    "myapp",
    configkit.Searches(
        configkit.Paths("/run/myapp/config.yml", "/opt/myapp/config.yaml"),
        configkit.UnixSystemDirectories(),
    ),
)
```

Use the resulting source with ordinary loading or as a layered main input. Custom
searches and file lists participate at their specified position; they do not
silently append unselected presets. Reordering presets or their directory
searches changes search priority.

### Selection And Failure Behavior

| Situation | Selected behavior |
| --- | --- |
| Candidate is absent | Continue to the next candidate. |
| A candidate file is selected | Stop searching and load that file. |
| Candidate is inaccessible, unreadable, or not an accepted file type | Fail using normal source diagnostics. |
| Selected file fails parsing, version checks, composition, decoding, or validation | Fail without trying another candidate. |
| Entire search has no match | Absence is permitted by default; follow the shared absent-main lifecycle settled under OD-3 below. |
| Entire search has no match and is required | Fail with a diagnostic explaining the unsuccessful search. |
| Caller explicitly uses `File(path)` | Retain existing exclusive-file behavior, including errors for a missing file. |

Requiredness applies to the complete search, not to each preset or directory.
Permitting an absent file does not waive application version, schema, or domain
requirements; the remaining lifecycle may still reject an incomplete configuration.

Perform discovery before parsing the chosen main file. Do not implement fallback
by repeatedly running complete loads. Defaults, application resolution, secret
lookup, and final validation retain their existing lifecycle execution rules.
Once a file has been selected, later failure does not restart discovery.

### Construction And Filesystem Behavior

Constructors describe the search without filesystem access. Resolve dynamic base
directories and access candidate files during loading, so reusable descriptions
can observe changes on subsequent loads. Do not change the process working
directory or environment.

Reuse the host-file acquisition rules for selected files, including regular-file
checks, symlink handling, resource ownership, and source limits. The feature does
not require a new filesystem traversal engine, recursive directory search, or
new file-size limits. Candidate filenames and groups are bounded by their
explicit descriptions, rather than by scanning unrelated directories.

The exact Go names, constructors, and representation of search-wide options
belong in the implementation plan. Keep the public surface root-owned and keep
the single-preset case concise.

### Unavailable Bases And Platform Applicability (OD-2 — Settled)

Resolve directory bases only as the ordered search reaches them. Once an earlier
candidate is selected, later searches do not resolve their bases. Static invalid
search arguments still fail ordinary option validation before source acquisition.
An explicitly empty filename selection skips directory resolution as specified
under OD-4; it has no candidates to locate.

If a selected directory's base cannot be resolved, stop and report the error.
A missing directory or candidate file beneath a resolved base allows the search
to continue; a directory-resolution error does not. This distinction also applies
to optional searches: permitting no match does not suppress errors encountered
while searching. Preserve the agreed errors for inaccessible or unreadable
candidates.

Treat using the Unix system preset on an unsupported platform as an explicit
configuration error when that preset is reached. Specify its supported platforms
in the implementation plan. Do not translate the Unix preset into a different
platform's convention. User, home, and working-directory presets use their native
Go directory helpers.

This keeps search priority predictable when directory resolution fails. Consumers
select the presets appropriate to their application and environment; there is no
option to skip unavailable bases or unsupported presets.

### No Match, Imports, And Selected Overlays (OD-3 — Settled)

Use a shared absent-main lifecycle for optional no-match and explicit `NoSource()`.
Allow imports to be enabled when the main file is absent: there are no main-file
import declarations to process, and enablement does not introduce automatic
directory scanning. Other actual sources retain their normal import behavior.
In layered loads, process underlays normally and retain the existing explicit
`OverlayFilesystem` requirement for an active overlay.

For ordinary loads with an active overlay and no main file, require an explicit
filesystem context to locate the overlay. Do not infer an overlay root from the
first or last searched directory. A missing selected overlay remains an error;
absence of a main file does not silently disable a requested overlay.

For an absent main input, a host filesystem context specifying only `Root` is
sufficient to locate the overlay; no main-file `BaseDir` is needed. If discovery
selects a host file, retain the normal file-context rules: the selected file
determines its base directory, an explicit `BaseDir` is rejected, and the file
must be contained within any supplied root. This allows one root-only context
to work with both selected and absent main inputs, subject to those existing
root and containment checks. An invalid context is an error, not a reason to
resume discovery.

Defaults, CLI overrides, application resolution, and validation continue through
the normal lifecycle. Main-version checking receives an absent main input and
may reject it under the application's existing policy. Create no synthetic empty
document or source record for the absent main file.

This deliberately relaxes the former `NoSource()` restrictions on imports and
on overlays with explicit context. Both ways of obtaining an absent main input
have the same behavior, as defined in the [configuration contract](../config.md#configuration-discovery).

### Empty Composition And Implicit Preset Selection (OD-4 — Settled)

Require explicit preset or search selection to activate a convention. Supplying
no presets or custom searches produces no candidates and activates no implicit
fallback. Empty custom path lists and explicitly empty filename-group selections
also contribute no candidates. An empty component does not prevent later
components from contributing candidates.

A directory search with an explicitly empty filename selection does not resolve
its directory base. Static argument validation still applies. This gives empty
selections their no-candidate meaning without introducing directory-resolution
errors for a search that cannot select a file.

If the complete search is empty, an optional search follows the absent-main
lifecycle settled under OD-3; a required search fails as exhausted. Empty lists
are valid, supporting consumers that construct their searches programmatically.

Distinguish omission from an explicitly empty filename selection for a directory
search. Omitting the selection uses the agreed `[Named, Generic]` default;
explicitly selecting an empty list means no filenames. The eventual API must
represent both intentions clearly. Presets retain their specified filename groups.

Selecting a single preset keeps the simple case concise. Search-wide requiredness
provides the check for consumers that must find a configuration file.

### Slugs And Custom Path Interpretation (OD-5 — Settled)

Require a nonempty single-component application slug whenever a selected search
uses it to construct a directory or filename. Reject path separators, `.` and
`..`, and forms unusable as that component on the target platform. Preserve
spelling and case; do not trim, lowercase, or impose a kebab-case naming scheme.
Validation keeps the slug from changing the intended directory structure without
imposing a separate naming style.

Searches that do not use the slug do not require one. Examples include explicit
file candidates and a custom directory searched with only Generic filenames.

Treat custom paths as literal host paths. Resolve relative paths against the
working directory at load time. Do not expand shell variables, `~`, or wildcard
syntax. Consumers can construct expanded paths themselves; User and Home presets
provide the common directory-resolution conveniences. These rules keep custom
paths consistent with explicit file loading.

Custom filenames within directory searches are literal single filename
components. They cannot be absolute paths or contain subdirectory components;
use custom directory searches or explicit file candidates for those layouts.
They receive no application-name placeholder expansion. The Named group supplies
application-name expansion, while consumers may construct custom filenames
themselves before supplying them.

### Placement, Provenance, And Existing Features

The selected file participates as the actual file source. Preserve its physical
identity, effective placement, source origin, and passive resource context.
Relative imports and relative application resources use the selected file's
normal context. Discovery does not relocate overlay or CLI coordinates.

Missing candidates are not acquired sources and do not create source IDs,
configuration history, or explicit input presence. An entirely absent search
contributes no synthetic empty configuration file. Diagnostics should distinguish
an exhausted required search from a failed candidate. Existing selected-source
inspection remains the foundation; a new public per-attempt trace is not part of
the implemented API.

Preserve collection policies, inheritance, source-version rules, CLI ordering,
final validation, and frozen inspection after selection. Search-list order does
not change precedence among underlays, the main input, an overlay, and CLI values.
An explicitly selected overlay is not another main-file fallback candidate.

## Target Integration And Documentation

Discovery belongs at the source-acquisition boundary. A selected file retains
the existing file-loading behavior and optional exhaustion shares the absent-main
lifecycle above. Public search descriptions remain root-owned while private
implementation handles selection and loading. The feature's implemented
configuration contract and maintained consumer examples will explain the
three-part discovery model, ordering, and context behavior.

The architecture policy should recognize explicitly consumer-selected discovery
conventions while preserving consumer ownership of search policy. An ADR records
that durable responsibility boundary; preset filenames and API mechanics belong
in their canonical contracts once implemented. Execution steps and validation
coverage are defined in the [implementation plan](implementation.md).

This roadmap does not change implementation, policy, or ADRs by itself. The
released v0.9.0 code, including the completed sequence integration and remediation
work, provides the implementation baseline.